Installation guide

Important notice

You don’t need to install MateCat in order to use it: just visit the page http://www.matecat.com to translate with MateCat.
This guide is meant for users who want to install and administer the open source version on their own machines.

Table of content

Overview

MateCat is a client-server web application. Most of the interaction runs in the user browser (client), while all calculation along with user data permanently resides on the server side (server).
To run MateCat as a end-user connected to a remote server you only need:

  1. Google Chrome browser
  2. A machine with a few GB of RAM
  3. Internet connection

To run MateCat server on your personal machine (keeping all data on your machine and bearing all the load), you also need administration privileges to install Apache, PHP and MySQL. Running MateCat locally on your machine does not require an Internet connection.
Go to top

Hardware requirements

If you just want to test MateCat on a few million words you will need 1 server with:

  • Storage: 50GB
  • CPU: 2 core x 2.5 Ghz
  • RAM: 4GB

To manage 50M words per year of translation capacity, you will need 1 server with:

  • Storage: 1TB
  • CPU: 6 core x 2.5 Ghz
  • RAM: 32GB

If you need to manage up to 200M words per year, you can do it using 4TB of storage, 12 CPU core, and 64GB of RAM and placing the MySQL database on a separate machine (easy to do with the MateCat config file). For handling more than 200M words per year you will need to add a load balancer in front of the web servers and databases. Databases will need a master-master replica schema. This will require an experienced LAMP and scalability engineer.
Go to top

Advanced Manual Setup

OS

This guide will assume you are on Ubuntu 18 Linux distribution or higher.
You will be guided in the installation of all the necessary dependencies,
if there is any package that you have already installed on your server you can skip the step.
Go to top

Install Web Server: Apache2

sudo su
apt-get update
apt-get install apache2

Go to top

Install Redis

Differently from ActiveMQ, Redis will automatically start on boot.

apt-get install redis-server

Make redis listen on all ports

sed -i 's/bind 127.0.0.1/bind 0.0.0.0/g' /etc/redis/redis.conf

Done! You can launch it by typing

service redis-server restart

Go to top

Install MySQL

apt-get install mysql-server mysql-client

Go to top

Install PHP5.6 additional packages

sudo apt-get install software-properties-common
sudo add-apt-repository ppa:ondrej/php
sudo apt-get update
sudo apt-get install -y php5.6
apt-get install php5.6-mysql libapache2-mod-php5.6 php5.6-curl php5.6-json php5.6-simplexml php5.6-mbstring php5.6-mcrypt php5.6-xdebug php5.6-mysql php5.6-xml php5.6-dev php5.6-redis php5.6-zip

Go to top

Install other packages

apt install zip unzip screen

Go to top

Enable some modules

Apache web server needs to be fine tuned for MateCat to work properly. Enter these commands to enable required Apache submodules

a2enmod rewrite filter deflate headers expires proxy_http ssl php5.6
phpenmod mcrypt
apache2ctl restart

Go to top

Configure MySql

Save your current configuration

cp /etc/mysql/mysql.conf.d/mysqld.cnf /etc/mysql/mysql.conf.d/mysqld.cnf.bkp

Paste the new configuration

vi /etc/mysql/mysql.conf.d/mysqld.cnf
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
 
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
 
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
skip-name-resolve

init_connect='SET NAMES utf8'
skip-character-set-client-handshake
character-set-server=utf8

bind-address = 0.0.0.0

key_buffer_size = 32M
max_allowed_packet=64M
thread_stack = 192K
thread_cache_size       = 50
myisam-recover-options = BACKUP
max_connections        = 500
open_files_limit               = 65535
table_definition_cache         = 1024
table_open_cache               = 2048
query_cache_limit = 1M
query_cache_size        = 16M
slow-query-log = 1
slow-query-log-file = /var/log/mysql/mysql-slow.log
long_query_time = 1000000
log-queries-not-using-indexes=1
log_error = /var/log/mysql/error.log

expire_logs_days        = 10
max_binlog_size         = 100M

gtid_mode = on
enforce_gtid_consistency = true
sql-mode="NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
explicit_defaults_for_timestamp=true

innodb_use_native_aio = 0
innodb-flush-method            = O_DIRECT
innodb_file_per_table = 1
innodb-buffer-pool-size = 1G
innodb_read_io_threads=64
innodb-log-files-in-group      = 2
innodb_write_io_threads=64
innodb-flush-log-at-trx-commit = 2
innodb_log_file_size=256M

[mysqldump]
quick
quote-names
max_allowed_packet = 32M
 
[mysql]
 
[isamchk]
key_buffer = 16M
!includedir /etc/mysql/conf.d/

Restart mysql service

service mysql restart

Go to top

Install Active MQ

ActiveMQ requires an installed version of Java. If you don’t have java installed just type:

apt-get install openjdk-8-jre

Download ActiveMQ 5.11

wget https://archive.apache.org/dist/activemq/5.11.3/apache-activemq-5.11.3-bin.tar.gz

Extract the tarball

tar xzf apache-activemq-5.11.3-bin.tar.gz

Move it to the /opt folder

mv apache-activemq-5.11.3 /opt

Create a soft link to the folder for ease of next steps

ln -sf /opt/apache-activemq-5.11.3/ /opt/activemq

Create an “activemq” user and allow it to execute bash scripts

adduser -system activemq
sed -i 's#activemq:/bin/false#activemq:/bin/bash#g' /etc/passwd

Change ownership of activemq folders

chown -R activemq: /opt/apache-activemq-5.11.3/

Sym-link the init script provided by Active MQ to /etc/init.d/activemq

ln -sf /opt/activemq/bin/activemq /etc/init.d/

Create a default configuration file with ActiveMQ, change ownership and group for this file and make it accessible only by root

/etc/init.d/activemq create /etc/default/activemq
chown root:nogroup /etc/default/activemq
chmod 600 /etc/default/activemq

Enable ActiveMQ connector for the web interface

sed -i 's/Context createConnector='false'/Context createConnector='true'/g' /etc/default/activemq/conf/activemq.xml

Sym-link ActiveMQ into /bin/bash so that it can be easily called into terminal

ln -s /etc/init.d/activemq /usr/bin/activemq

Done! You can now start ActiveMQ by typing

activemq start

Optionally, ActiveMQ tar.gz file downloaded at the beginning is not necessary, and can be removed:

rm apache-activemq-5.11.3-bin.tar.gz

Go to top

Install Node 9.11.2

Node.js is necessary to have a realtime chat within matecat. Install it with:

apt-get install python2.7
sudo ln -s /usr/bin/python2.7 /usr/bin/python2 && sudo ln -s /usr/bin/python2 /usr/bin/python
sudo apt-get install build-essential libssl-dev curl git-core
apt-get install libicu-dev
apt install pkg-config
apt install make
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.31.3/install.sh | bash
vi ~/.zshrc

Edit:

export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"  # This loads nvm bash_completion

source ~/.nvm/nvm.sh
nvm install 9.11.2

Go to top

Tune PHP configuration

echo "short_open_tag = On" >> /etc/php/5.6/cli/php.ini
echo "memory_limit = 1024M" >> /etc/php/5.6/cli/php.ini
apache2ctl restart

Go to top

Insall MateCat

This section assumes you’ll install MateCat under the home of user matecat (/home/matecat).
In order to create that user, use the following:

adduser --disabled-password --gecos "" matecat
apt-get install git
su - matecat

Then clone the repository in /home/matecat:

git clone https://github.com/matecat/MateCat.git cattool
exit
cd /home/matecat/cattool/INSTALL
mysql -u root -p < matecat.sql
mysql -u root -p

Feel free to specify the privileges you want, with the next two instructions you will be able to access only from localhost

delete from mysql.user where user = 'matecat' and host = '%';
grant all privileges on matecat.* to 'matecat'@'localhost' identified by 'matecat01';
FLUSH PRIVILEGES;
exit

Go to top

Create the virtualhost

Navigate to the directory

cd /etc/apache2/sites-available

Remove other virtualhosts:

rm *

Paste the configuration in the default virtualhost:

vi default-vhost.conf
<VirtualHost *:80>
    RewriteEngine On
    RewriteCond %{HTTPS} !=on
    RewriteCond %{SERVER_PORT} 80
    RewriteCond %{HTTP_HOST} ^(.*)$
    RewriteRule ^/(.*)$ https://%1/$1 [R]
</VirtualHost>

Paste the configuration in the Matecat virtualhost:

vi 443-matecat.conf
<VirtualHost *:443>

    ServerAdmin webmaster@localhost
    ServerName dev.matecat.com (your_domain)

    DocumentRoot /home/matecat/cattool
    DirectoryIndex index.php index.php3 index.html index.htm index.shtml
    <Directory />
        Options FollowSymLinks
        AllowOverride None
    </Directory>

    SSLEngine On
    SSLProtocol all -SSLv3 -SSLv2

    SSLProxyEngine On

    SSLCertificateFile /etc/apache2/ssl-cert/my_matecat.crt
    SSLCertificateKeyFile /etc/apache2/ssl-cert/my_matecat.key


    <Directory /home/matecat/cattool/ >

            Options All
            AllowOverride All
	Require all granted

            ExpiresActive On
            ExpiresByType text/html "access plus 1 minute"
            ExpiresByType text/css "access plus 1 minute"
            ExpiresByType text/javascript "access plus 1 minute"
            ExpiresByType image/gif "access plus 1 week"
            ExpiresByType image/jpeg "access plus 1 week"
            ExpiresByType image/jpg "access plus 1 week"
            ExpiresByType image/png "access plus 1 week"
            ExpiresByType image/vnd.microsoft.icon "access plus 1 week"
            ExpiresByType image/ico "access plus 1 week"
            ExpiresByType application/x-shockwave-flash "access plus 1 week"

    </Directory>

    php_flag register_globals off
    php_flag magic_quotes_gpc off
    ErrorLog ${APACHE_LOG_DIR}/matecat/error.log
    CustomLog ${APACHE_LOG_DIR}/matecat/access.log combined
    ServerSignature Off

    <Location /sse/ >
        ProxyPass http://localhost:7788/
        ProxyPassReverse http://localhost:7788/
    </Location>

</VirtualHost>

<VirtualHost *:443>

    ServerAdmin webmaster@localhost
    ServerName 0.ajax.your_domain.com
    ServerAlias 1..ajax.your_domain.com
    ServerAlias 2.ajax.your_domain.com
    ServerAlias 3.ajax.your_domain.com

    DocumentRoot /home/matecat/cattool
    DirectoryIndex index.php index.php3 index.html index.htm index.shtml
    <Directory />
        Options FollowSymLinks
        AllowOverride None
    </Directory>

    SSLEngine On
    SSLProtocol all -SSLv3 -SSLv2

    SSLProxyEngine On

    SSLCertificateFile /etc/apache2/ssl-cert/my_matecat.crt
    SSLCertificateKeyFile /etc/apache2/ssl-cert/my_matecat.key

    <Directory /home/matecat/cattool/>

            Options All
            AllowOverride All
            Order allow,deny
            allow from all

            ExpiresActive On
            ExpiresByType text/html "access plus 1 minute"
            ExpiresByType text/css "access plus 1 minute"
            ExpiresByType text/javascript "access plus 1 minute"
            ExpiresByType image/gif "access plus 1 week"
            ExpiresByType image/jpeg "access plus 1 week"
            ExpiresByType image/jpg "access plus 1 week"
            ExpiresByType image/png "access plus 1 week"
            ExpiresByType image/vnd.microsoft.icon "access plus 1 week"
            ExpiresByType image/ico "access plus 1 week"
            ExpiresByType application/x-shockwave-flash "access plus 1 week"

    </Directory>

    php_flag register_globals off
    php_flag magic_quotes_gpc off
    ErrorLog ${APACHE_LOG_DIR}/matecat/error.log
    CustomLog ${APACHE_LOG_DIR}/matecat/access.log combined
    ServerSignature Off

    <Location /sse/ >
        ProxyPass http://localhost:7788/
        ProxyPassReverse http://localhost:7788/
    </Location>

</VirtualHost>

Go to top

Create Certificate

Create a self-signed certificate (if you have a registered domain and already have a certificate, use your certificates)

cd ~
openssl req -x509 -newkey rsa:4096 -keyout my_matecat.key -out my_matecat.crt -days 365 -nodes                    (insert Country Name and Organization Name=Matecat)
mkdir /etc/apache2/ssl-cert
mv my_matecat.* /etc/apache2/ssl-cert/
mkdir /var/log/apache2/matecat

Enable virtualhosts and restart apache2

a2ensite 443-matecat.conf
a2ensite default-vhost.conf
service apache2 restart

(If you want enable multidomain, you must create another certificate for “*.ajax.your_domain.com”)
Go to top

Configure MateCat

Go to your matecat root folder in order to install PHP composer packages:

cd /home/matecat/cattool
php -r "readfile('https://getcomposer.org/installer');" | php
php composer.phar --no-dev install
su - matecat
cd ~/cattool/inc
cp oauth_config.ini.sample oauth_config.ini
cp task_manager_config.ini.sample task_manager_config.ini
sed -i "s/\/home\/matecat\/storage/\/home\/matecat\/cattool\/storage/g" config.ini
cd ~/cattool/nodejs

Paste the configuration in nodejs/config.ini file:

vi /home/matecat/cattool/nodejs/config.ini
[server]
address = 0.0.0.0
port = 7788
path = /channel/updates

[queue]
name = /topic/matecat_sse_notifications
host = localhost

port = 61613
login = admin
passcode = admin

[log]
file = log/server.log
level = debug

Go to top

Compile JS code

exit

Move to the Grunt configuration folder:

cd /home/matecat/cattool/support_scripts/grunt

Prepare the environment and deploy the source code

npm config set user 0
npm config set unsafe-perm true
npm install --save-dev @babel/core
npm install --save-dev webpack@2
rm -rf node_modules
npm install
npm install grunt
npm install -g grunt-cli
grunt development

Go to top

Config File

Paste the configuration in the Matecat config.ini file:

vi /home/matecat/cattool/inc/config.ini
ENV = "production"
CHECK_FS = no
[production]
CLI_HTTP_HOST = "https://your_domain.com"
COOKIE_DOMAIN = "your_domain.com"
ENABLE_MULTI_DOMAIN_API = no
DB_SERVER = "localhost"
DB_DATABASE = "matecat"
DB_USER = "matecat"
DB_PASS = "matecat01"
REDIS_SERVERS = "tcp://localhost:6379"
QUEUE_BROKER_ADDRESS = "tcp://localhost:61613"
QUEUE_DQF_ADDRESS = "tcp://localhost:61613"
QUEUE_JMX_ADDRESS = "http://localhost:8161"
QUEUE_CREDENTIALS = "admin:admin"
SSE_BASE_URL      = "your_domain.com/sse"         (your_domain)
STORAGE_DIR = "/home/matecat/cattool/storage"         ;provide an absolute path
SMTP_HOST = 'localhost'
SMTP_PORT = '25'
SMTP_SENDER = 'no-reply@matecat.loc'
SMTP_HOSTNAME = 'localhost'
; Filters Configuration
FILTERS_ADDRESS = 'xxx'
FILTERS_MASHAPE_KEY = 'xxx'
; THIS SHOULD BE YOUR LEXIQA LICENSE, Request your license key at
; @see http://www.lexiqa.net
LXQ_LICENSE = false
LXQ_PARTNERID = false
FILE_STORAGE_METHOD = 'fs'
#TRACKING_CODES_VIEW_PATH = "internal_scripts/tracking_codes/"

Change owner files:

chown -R matecat:matecat /home/matecat/cattool/*

Go to top

Run Daemons

Launch analysis daemon:

cd /home/matecat/cattool/daemons
/bin/bash /home/matecat/cattool/daemons/restartAnalysis.sh
chown -R www-data /home/matecat/cattool/storage/
service apache2 restart
cd /home/matecat/cattool/nodejs/
npm install sse-channel
npm install

Run Node.js server in a screen with the following command:

screen -d -m -S 'node' node /home/matecat/cattool/nodejs/server.js

Go to top

Door

Make sure this server can be seen from the outside, in particular that door 80 and 443 are open

HTTP	TCP	80	0.0.0.0/0
HTTPS	TCP	443	0.0.0.0/0

Go to top

Set Hosts

If your domain is not registered, add the ip to your hosts file

vi /etc/hosts

Add a new line with the ip and domain:

ip_server your_domain.com

Go to top