Important notice
Table of content
- Overview
- Hardware requirements
- Advanced Manual Setup
- OS
- Install Web Server: Apache2
- Install Redis
- Install MySQL
- Install PHP5.6 additional packages
- Install other packages
- Enable some modules
- Configure MySql
- Install ActiveMQ
- Install Node 9.11.2
- Tune PHP configuration
- Install MateCat
- Create the virtualhost
- Create Certificate
- Configure MateCat
- Compile JS code
- Config File
- Run Daemons
- Door
- Set Hosts
Overview
MateCat is a client-server web application. Most of the interaction runs in the user browser (client), while all calculation along with user data permanently resides on the server side (server).
To run MateCat as a end-user connected to a remote server you only need:
- Google Chrome browser
- A machine with a few GB of RAM
- Internet connection
To run MateCat server on your personal machine (keeping all data on your machine and bearing all the load), you also need administration privileges to install Apache, PHP and MySQL. Running MateCat locally on your machine does not require an Internet connection.
Go to top
Hardware requirements
If you just want to test MateCat on a few million words you will need 1 server with:
- Storage: 50GB
- CPU: 2 core x 2.5 Ghz
- RAM: 4GB
To manage 50M words per year of translation capacity, you will need 1 server with:
- Storage: 1TB
- CPU: 6 core x 2.5 Ghz
- RAM: 32GB
If you need to manage up to 200M words per year, you can do it using 4TB of storage, 12 CPU core, and 64GB of RAM and placing the MySQL database on a separate machine (easy to do with the MateCat config file). For handling more than 200M words per year you will need to add a load balancer in front of the web servers and databases. Databases will need a master-master replica schema. This will require an experienced LAMP and scalability engineer.
Go to top
Advanced Manual Setup
OS
This guide will assume you are on Ubuntu 18 Linux distribution or higher.
You will be guided in the installation of all the necessary dependencies,
if there is any package that you have already installed on your server you can skip the step.
Go to top
Install Web Server: Apache2
sudo su
apt-get update
apt-get install apache2
Install Redis
Differently from ActiveMQ, Redis will automatically start on boot.
apt-get install redis-server
Make redis listen on all ports
sed -i 's/bind 127.0.0.1/bind 0.0.0.0/g' /etc/redis/redis.conf
Done! You can launch it by typing
service redis-server restart
Install MySQL
apt-get install mysql-server mysql-client
Install PHP5.6 additional packages
sudo apt-get install software-properties-common
sudo add-apt-repository ppa:ondrej/php
sudo apt-get update
sudo apt-get install -y php5.6
apt-get install php5.6-mysql libapache2-mod-php5.6 php5.6-curl php5.6-json php5.6-simplexml php5.6-mbstring php5.6-mcrypt php5.6-xdebug php5.6-mysql php5.6-xml php5.6-dev php5.6-redis php5.6-zip
Install other packages
apt install zip unzip screen
Enable some modules
Apache web server needs to be fine tuned for MateCat to work properly. Enter these commands to enable required Apache submodules
a2enmod rewrite filter deflate headers expires proxy_http ssl php5.6
phpenmod mcrypt
apache2ctl restart
Configure MySql
Save your current configuration
cp /etc/mysql/mysql.conf.d/mysqld.cnf /etc/mysql/mysql.conf.d/mysqld.cnf.bkp
Paste the new configuration
vi /etc/mysql/mysql.conf.d/mysqld.cnf
[client] port = 3306 socket = /var/run/mysqld/mysqld.sock [mysqld_safe] socket = /var/run/mysqld/mysqld.sock nice = 0 [mysqld] user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock port = 3306 basedir = /usr datadir = /var/lib/mysql tmpdir = /tmp lc-messages-dir = /usr/share/mysql skip-external-locking skip-name-resolve init_connect='SET NAMES utf8' skip-character-set-client-handshake character-set-server=utf8 bind-address = 0.0.0.0 key_buffer_size = 32M max_allowed_packet=64M thread_stack = 192K thread_cache_size = 50 myisam-recover-options = BACKUP max_connections = 500 open_files_limit = 65535 table_definition_cache = 1024 table_open_cache = 2048 query_cache_limit = 1M query_cache_size = 16M slow-query-log = 1 slow-query-log-file = /var/log/mysql/mysql-slow.log long_query_time = 1000000 log-queries-not-using-indexes=1 log_error = /var/log/mysql/error.log expire_logs_days = 10 max_binlog_size = 100M gtid_mode = on enforce_gtid_consistency = true sql-mode="NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION" explicit_defaults_for_timestamp=true innodb_use_native_aio = 0 innodb-flush-method = O_DIRECT innodb_file_per_table = 1 innodb-buffer-pool-size = 1G innodb_read_io_threads=64 innodb-log-files-in-group = 2 innodb_write_io_threads=64 innodb-flush-log-at-trx-commit = 2 innodb_log_file_size=256M [mysqldump] quick quote-names max_allowed_packet = 32M [mysql] [isamchk] key_buffer = 16M !includedir /etc/mysql/conf.d/
Restart mysql service
service mysql restart
Install Active MQ
ActiveMQ requires an installed version of Java. If you don’t have java installed just type:
apt-get install openjdk-8-jre
Download ActiveMQ 5.11
wget https://archive.apache.org/dist/activemq/5.11.3/apache-activemq-5.11.3-bin.tar.gz
Extract the tarball
tar xzf apache-activemq-5.11.3-bin.tar.gz
Move it to the /opt folder
mv apache-activemq-5.11.3 /opt
Create a soft link to the folder for ease of next steps
ln -sf /opt/apache-activemq-5.11.3/ /opt/activemq
Create an “activemq” user and allow it to execute bash scripts
adduser -system activemq
sed -i 's#activemq:/bin/false#activemq:/bin/bash#g' /etc/passwd
Change ownership of activemq folders
chown -R activemq: /opt/apache-activemq-5.11.3/
Sym-link the init script provided by Active MQ to /etc/init.d/activemq
ln -sf /opt/activemq/bin/activemq /etc/init.d/
Create a default configuration file with ActiveMQ, change ownership and group for this file and make it accessible only by root
/etc/init.d/activemq create /etc/default/activemq
chown root:nogroup /etc/default/activemq
chmod 600 /etc/default/activemq
Enable ActiveMQ connector for the web interface
sed -i 's/Context createConnector='false'/Context createConnector='true'/g' /etc/default/activemq/conf/activemq.xml
Sym-link ActiveMQ into /bin/bash so that it can be easily called into terminal
ln -s /etc/init.d/activemq /usr/bin/activemq
Done! You can now start ActiveMQ by typing
activemq start
Optionally, ActiveMQ tar.gz file downloaded at the beginning is not necessary, and can be removed:
rm apache-activemq-5.11.3-bin.tar.gz
Install Node 9.11.2
Node.js is necessary to have a realtime chat within matecat. Install it with:
apt-get install python2.7
sudo ln -s /usr/bin/python2.7 /usr/bin/python2 && sudo ln -s /usr/bin/python2 /usr/bin/python
sudo apt-get install build-essential libssl-dev curl git-core
apt-get install libicu-dev
apt install pkg-config
apt install make
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.31.3/install.sh | bash
vi ~/.zshrc
Edit:
export NVM_DIR="$HOME/.nvm" [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" # This loads nvm [ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion" # This loads nvm bash_completion
source ~/.nvm/nvm.sh
nvm install 9.11.2
Tune PHP configuration
echo "short_open_tag = On" >> /etc/php/5.6/cli/php.ini
echo "memory_limit = 1024M" >> /etc/php/5.6/cli/php.ini
apache2ctl restart
Insall MateCat
This section assumes you’ll install MateCat under the home of user matecat (/home/matecat).
In order to create that user, use the following:
adduser --disabled-password --gecos "" matecat
apt-get install git
su - matecat
Then clone the repository in /home/matecat:
git clone https://github.com/matecat/MateCat.git cattool
exit
cd /home/matecat/cattool/INSTALL
mysql -u root -p < matecat.sql
mysql -u root -p
Feel free to specify the privileges you want, with the next two instructions you will be able to access only from localhost
delete from mysql.user where user = 'matecat' and host = '%';
grant all privileges on matecat.* to 'matecat'@'localhost' identified by 'matecat01';
FLUSH PRIVILEGES;
exit
Create the virtualhost
Navigate to the directory
cd /etc/apache2/sites-available
Remove other virtualhosts:
rm *
Paste the configuration in the default virtualhost:
vi default-vhost.conf
<VirtualHost *:80> RewriteEngine On RewriteCond %{HTTPS} !=on RewriteCond %{SERVER_PORT} 80 RewriteCond %{HTTP_HOST} ^(.*)$ RewriteRule ^/(.*)$ https://%1/$1 [R] </VirtualHost>
Paste the configuration in the Matecat virtualhost:
vi 443-matecat.conf
<VirtualHost *:443> ServerAdmin webmaster@localhost ServerName dev.matecat.com (your_domain) DocumentRoot /home/matecat/cattool DirectoryIndex index.php index.php3 index.html index.htm index.shtml <Directory /> Options FollowSymLinks AllowOverride None </Directory> SSLEngine On SSLProtocol all -SSLv3 -SSLv2 SSLProxyEngine On SSLCertificateFile /etc/apache2/ssl-cert/my_matecat.crt SSLCertificateKeyFile /etc/apache2/ssl-cert/my_matecat.key <Directory /home/matecat/cattool/ > Options All AllowOverride All Require all granted ExpiresActive On ExpiresByType text/html "access plus 1 minute" ExpiresByType text/css "access plus 1 minute" ExpiresByType text/javascript "access plus 1 minute" ExpiresByType image/gif "access plus 1 week" ExpiresByType image/jpeg "access plus 1 week" ExpiresByType image/jpg "access plus 1 week" ExpiresByType image/png "access plus 1 week" ExpiresByType image/vnd.microsoft.icon "access plus 1 week" ExpiresByType image/ico "access plus 1 week" ExpiresByType application/x-shockwave-flash "access plus 1 week" </Directory> php_flag register_globals off php_flag magic_quotes_gpc off ErrorLog ${APACHE_LOG_DIR}/matecat/error.log CustomLog ${APACHE_LOG_DIR}/matecat/access.log combined ServerSignature Off <Location /sse/ > ProxyPass http://localhost:7788/ ProxyPassReverse http://localhost:7788/ </Location> </VirtualHost> <VirtualHost *:443> ServerAdmin webmaster@localhost ServerName 0.ajax.your_domain.com ServerAlias 1..ajax.your_domain.com ServerAlias 2.ajax.your_domain.com ServerAlias 3.ajax.your_domain.com DocumentRoot /home/matecat/cattool DirectoryIndex index.php index.php3 index.html index.htm index.shtml <Directory /> Options FollowSymLinks AllowOverride None </Directory> SSLEngine On SSLProtocol all -SSLv3 -SSLv2 SSLProxyEngine On SSLCertificateFile /etc/apache2/ssl-cert/my_matecat.crt SSLCertificateKeyFile /etc/apache2/ssl-cert/my_matecat.key <Directory /home/matecat/cattool/> Options All AllowOverride All Order allow,deny allow from all ExpiresActive On ExpiresByType text/html "access plus 1 minute" ExpiresByType text/css "access plus 1 minute" ExpiresByType text/javascript "access plus 1 minute" ExpiresByType image/gif "access plus 1 week" ExpiresByType image/jpeg "access plus 1 week" ExpiresByType image/jpg "access plus 1 week" ExpiresByType image/png "access plus 1 week" ExpiresByType image/vnd.microsoft.icon "access plus 1 week" ExpiresByType image/ico "access plus 1 week" ExpiresByType application/x-shockwave-flash "access plus 1 week" </Directory> php_flag register_globals off php_flag magic_quotes_gpc off ErrorLog ${APACHE_LOG_DIR}/matecat/error.log CustomLog ${APACHE_LOG_DIR}/matecat/access.log combined ServerSignature Off <Location /sse/ > ProxyPass http://localhost:7788/ ProxyPassReverse http://localhost:7788/ </Location> </VirtualHost>
Create Certificate
Create a self-signed certificate (if you have a registered domain and already have a certificate, use your certificates)
cd ~
openssl req -x509 -newkey rsa:4096 -keyout my_matecat.key -out my_matecat.crt -days 365 -nodes (insert Country Name and Organization Name=Matecat)
mkdir /etc/apache2/ssl-cert
mv my_matecat.* /etc/apache2/ssl-cert/
mkdir /var/log/apache2/matecat
Enable virtualhosts and restart apache2
a2ensite 443-matecat.conf
a2ensite default-vhost.conf
service apache2 restart
(If you want enable multidomain, you must create another certificate for “*.ajax.your_domain.com”)
Go to top
Configure MateCat
Go to your matecat root folder in order to install PHP composer packages:
cd /home/matecat/cattool
php -r "readfile('https://getcomposer.org/installer');" | php
php composer.phar --no-dev install
su - matecat
cd ~/cattool/inc
cp oauth_config.ini.sample oauth_config.ini
cp task_manager_config.ini.sample task_manager_config.ini
sed -i "s/\/home\/matecat\/storage/\/home\/matecat\/cattool\/storage/g" config.ini
cd ~/cattool/nodejs
Paste the configuration in nodejs/config.ini file:
vi /home/matecat/cattool/nodejs/config.ini
[server] address = 0.0.0.0 port = 7788 path = /channel/updates [queue] name = /topic/matecat_sse_notifications host = localhost port = 61613 login = admin passcode = admin [log] file = log/server.log level = debug
Compile JS code
exit
Move to the Grunt configuration folder:
cd /home/matecat/cattool/support_scripts/grunt
Prepare the environment and deploy the source code
npm config set user 0
npm config set unsafe-perm true
npm install --save-dev @babel/core
npm install --save-dev webpack@2
rm -rf node_modules
npm install
npm install grunt
npm install -g grunt-cli
grunt development
Config File
Paste the configuration in the Matecat config.ini file:
vi /home/matecat/cattool/inc/config.ini
ENV = "production" CHECK_FS = no [production] CLI_HTTP_HOST = "https://your_domain.com" COOKIE_DOMAIN = "your_domain.com" ENABLE_MULTI_DOMAIN_API = no DB_SERVER = "localhost" DB_DATABASE = "matecat" DB_USER = "matecat" DB_PASS = "matecat01" REDIS_SERVERS = "tcp://localhost:6379" QUEUE_BROKER_ADDRESS = "tcp://localhost:61613" QUEUE_DQF_ADDRESS = "tcp://localhost:61613" QUEUE_JMX_ADDRESS = "http://localhost:8161" QUEUE_CREDENTIALS = "admin:admin" SSE_BASE_URL = "your_domain.com/sse" (your_domain) STORAGE_DIR = "/home/matecat/cattool/storage" ;provide an absolute path SMTP_HOST = 'localhost' SMTP_PORT = '25' SMTP_SENDER = 'no-reply@matecat.loc' SMTP_HOSTNAME = 'localhost' ; Filters Configuration FILTERS_ADDRESS = 'xxx' FILTERS_MASHAPE_KEY = 'xxx' ; THIS SHOULD BE YOUR LEXIQA LICENSE, Request your license key at ; @see http://www.lexiqa.net LXQ_LICENSE = false LXQ_PARTNERID = false FILE_STORAGE_METHOD = 'fs' #TRACKING_CODES_VIEW_PATH = "internal_scripts/tracking_codes/"
Change owner files:
chown -R matecat:matecat /home/matecat/cattool/*
Run Daemons
Launch analysis daemon:
cd /home/matecat/cattool/daemons
/bin/bash /home/matecat/cattool/daemons/restartAnalysis.sh
chown -R www-data /home/matecat/cattool/storage/
service apache2 restart
cd /home/matecat/cattool/nodejs/
npm install sse-channel
npm install
Run Node.js server in a screen with the following command:
screen -d -m -S 'node' node /home/matecat/cattool/nodejs/server.js
Door
Make sure this server can be seen from the outside, in particular that door 80 and 443 are open
HTTP TCP 80 0.0.0.0/0 HTTPS TCP 443 0.0.0.0/0
Set Hosts
If your domain is not registered, add the ip to your hosts file
vi /etc/hosts
Add a new line with the ip and domain:
ip_server your_domain.com