Important notice
Table of content
- Overview
- Hardware requirements
-
Advanced Manual Setup
- OS
- Install Web Server: Apache2
- Install Redis
- Install MySQL
- Install PHP5.6 additional packages
- Install other packages
- Enable some modules
- Configure MySql
- Install ActiveMQ
- Install Node 14
- Tune PHP configuration
- Install MateCat
- Create the virtualhost
- Create Certificate
- Configure MateCat
- MateCat Filters API
- Config File
- Compile JS code
- Run Daemons
- Port
- Set Hosts
Overview
MateCat is a client-server web application. Most of the interaction runs in the user browser (client), while all calculation along with user data permanently resides on the server side (server).
To run MateCat as a end-user connected to a remote server you only need:
- Google Chrome browser
- A machine with a few GB of RAM
- Internet connection
To run MateCat server on your personal machine (keeping all data on your machine and bearing all the load), you also need administration privileges to install Apache, PHP and MySQL. Running MateCat locally on your machine does not require an Internet connection.
Go to top
Hardware requirements
If you just want to test MateCat on a few million words you will need 1 server with:
- Storage: 50GB
- CPU: 2 core x 2.5 Ghz
- RAM: 4GB
To manage 50M words per year of translation capacity, you will need 1 server with:
- Storage: 1TB
- CPU: 6 core x 2.5 Ghz
- RAM: 32GB
If you need to manage up to 200M words per year, you can do it using 4TB of storage, 12 CPU core, and 64GB of RAM and placing the MySQL database on a separate machine (easy to do with the MateCat config file). For handling more than 200M words per year you will need to add a load balancer in front of the web servers and databases. Databases will need a master-master replica schema. This will require an experienced LAMP and scalability engineer.
Go to top
Advanced Manual Setup
OS
This guide will assume you are on Ubuntu 18 Linux distribution or higher.
You will be guided in the installation of all the necessary dependencies,
if there is any package that you have already installed on your server you can skip the step.
Go to top
Install Web Server: Apache2
sudo su apt-get update apt-get install apache2
Install Redis
Different from ActiveMQ, Redis will automatically start on boot.
apt-get install redis-server
Make redis listen on all ports
sed -i 's/bind 127.0.0.1/bind 0.0.0.0/g' /etc/redis/redis.conf
Done! You can launch it by typing
service redis-server restart
If you cannot start Redis on the default port you can change it:
redis-server –port 6400
Install MySQL
This command installs mysql v5.7.36:
apt-get install mysql-server mysql-client
Install PHP5.6 additional packages
Use this command if add-apt-repository is not installed:
apt-get install software-properties-common
Then:
add-apt-repository ppa:ondrej/php
add-apt-repository ppa:ondrej/apache2
apt-get update
apt-get -y --fix-missing install libapache2-mod-php5.6 \ php5.6 php5.6-json php5.6-curl php5.6-mysql php5.6-xml php5.6-mbstring php5.6-dev php5.6-mcrypt php5.6-redis \ php5.6-zip
Install other packages
apt install zip unzip screen
Enable some modules
Apache web server needs to be fine tuned for MateCat to work properly. Enter these commands to enable required Apache submodules
a2enmod rewrite filter deflate headers expires proxy_http ssl php5.6
phpenmod mcrypt
service apache2 restart
Configure MySql
Save your current configuration
cp /etc/mysql/mysql.conf.d/mysqld.cnf /etc/mysql/mysql.conf.d/mysqld.cnf.bkp
Paste the new configuration
vi /etc/mysql/mysql.conf.d/mysqld.cnf
[client] port = 3306 socket = /var/run/mysqld/mysqld.sock [mysqld_safe] socket = /var/run/mysqld/mysqld.sock nice = 0 [mysqld] user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock port = 3306 basedir = /usr datadir = /var/lib/mysql tmpdir = /tmp lc-messages-dir = /usr/share/mysql skip-external-locking skip-name-resolve init_connect='SET NAMES utf8' skip-character-set-client-handshake character-set-server=utf8 bind-address = 0.0.0.0 key_buffer_size = 32M max_allowed_packet=64M thread_stack = 192K thread_cache_size = 50 myisam-recover-options = BACKUP max_connections = 500 open_files_limit = 65535 table_definition_cache = 1024 table_open_cache = 2048 query_cache_limit = 1M query_cache_size = 16M slow-query-log = 1 slow-query-log-file = /var/log/mysql/mysql-slow.log long_query_time = 1000000 log-queries-not-using-indexes=1 log_error = /var/log/mysql/error.log expire_logs_days = 10 max_binlog_size = 100M gtid_mode = on enforce_gtid_consistency = true sql-mode="NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION" explicit_defaults_for_timestamp=true innodb_use_native_aio = 0 innodb-flush-method = O_DIRECT innodb_file_per_table = 1 innodb-buffer-pool-size = 1G innodb_read_io_threads=64 innodb-log-files-in-group = 2 innodb_write_io_threads=64 innodb-flush-log-at-trx-commit = 2 innodb_log_file_size=256M [mysqldump] quick quote-names max_allowed_packet = 32M [mysql] [isamchk] key_buffer = 16M !includedir /etc/mysql/conf.d/
Restart mysql service
service mysql restart
Install Active MQ
ActiveMQ requires an installed version of Java. If you don't have java installed just type:
apt-get install openjdk-11-jre -y
Download ActiveMQ 5.15
wget http://archive.apache.org/dist/activemq/5.15.8/apache-activemq-5.15.8-bin.tar.gz
Extract the tarball
tar -xvzf apache-activemq-5.15.8-bin.tar.gz
Move it to the /opt folder
mv apache-activemq-5.15.8 /opt/activemq
Create an “activemq” user and allow it to execute bash scripts
adduser -system activemq
Change ownership of activemq folders
chown -R activemq: /opt/activemq/
Sym-link the init script provided by Active MQ to /etc/init.d/activemq
ln -sf /opt/activemq/bin/activemq /etc/init.d/
Create a default configuration file with ActiveMQ, change ownership and group for this file and make it accessible only by root
/etc/init.d/activemq create /etc/default/activemq
chown root:nogroup /etc/default/activemq
chmod 600 /etc/default/activemq
Enable ActiveMQ connector for the web interface
sed -i 's/Context createConnector='false'/Context createConnector='true'/g' /etc/default/activemq/conf/activemq.xml
Sym-link ActiveMQ into /bin/bash so that it can be easily called into the terminal
ln -s /etc/init.d/activemq /usr/bin/activemq
Done! You can now start ActiveMQ by typing
activemq start
Install Node 14
Node.js is necessary to have a real-time chat within Matecat. Install it with:
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.0/install.sh | bash
vi ~/.zshrc
Edit this file:
export NVM_DIR="$HOME/.nvm" [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" [ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"
source ~/.nvm/nvm.sh
Finally, install node 14:
nvm install 14.15.5
Tune PHP configuration
echo "short_open_tag = On" >> /etc/php/5.6/cli/php.ini
echo "memory_limit = 1024M" >> /etc/php/5.6/cli/php.ini
service apache2 restart
Install MateCat
This section assumes you'll install MateCat under the home of user matecat (/home/matecat).
In order to create that user, use the following:
adduser --disabled-password --gecos "" matecat
apt-get install git
su - matecat
Then clone the repository in /home/matecat/matecat (feel free to change folder name):
git clone https://github.com/matecat/MateCat.git matecat
exit
cd /home/matecat/matecat/INSTALL
mysql -u root -p < matecat.sql
mysql -u root
Feel free to specify the privileges you want, with the next two instructions you will be able to access only from localhost
delete from mysql.user where user = 'matecat' and host = '%';
grant all privileges on matecat.* to 'matecat'@'localhost' identified by 'matecat01';FLUSH PRIVILEGES;exit
Create the virtualhost
Navigate to the directory
cd /etc/apache2/sites-available
Remove other virtualhosts:
rm *
Paste the configuration in the default virtualhost:
vi default-vhost.conf
<VirtualHost *:80> RewriteEngine On RewriteCond %{HTTPS} !=on RewriteCond %{SERVER_PORT} 80 RewriteCond %{HTTP_HOST} ^(.*)$ RewriteRule ^/(.*)$ https://%1/$1 [R] </VirtualHost>
Paste the configuration in the Matecat virtualhost:
vi 443-matecat.conf
<VirtualHost *:443> ServerAdmin webmaster@localhost ServerName your_domain.com DocumentRoot /home/matecat/matecat DirectoryIndex index.php index.php3 index.html index.htm index.shtml <Directory /> Options FollowSymLinks AllowOverride None </Directory> SSLEngine On SSLProtocol all -SSLv3 -SSLv2 SSLProxyEngine On SSLCertificateFile /etc/apache2/ssl-cert/my_matecat.crt SSLCertificateKeyFile /etc/apache2/ssl-cert/my_matecat.key <Directory /home/matecat/matecat/ > Options All AllowOverride All Require all granted ExpiresActive On ExpiresByType text/html "access plus 1 minute" ExpiresByType text/css "access plus 1 minute" ExpiresByType text/javascript "access plus 1 minute" ExpiresByType image/gif "access plus 1 week" ExpiresByType image/jpeg "access plus 1 week" ExpiresByType image/jpg "access plus 1 week" ExpiresByType image/png "access plus 1 week" ExpiresByType image/vnd.microsoft.icon "access plus 1 week" ExpiresByType image/ico "access plus 1 week" ExpiresByType application/x-shockwave-flash "access plus 1 week" </Directory> php_flag register_globals off php_flag magic_quotes_gpc off ErrorLog ${APACHE_LOG_DIR}/matecat/error.log CustomLog ${APACHE_LOG_DIR}/matecat/access.log combined ServerSignature Off <Location /sse/ > ProxyPass http://localhost:7788/ ProxyPassReverse http://localhost:7788/ </Location> </VirtualHost> <VirtualHost *:443> ServerAdmin webmaster@localhost ServerName 0.ajax.your_domain.com ServerAlias 1.ajax.your_domain.com ServerAlias 2.ajax.your_domain.com ServerAlias 3.ajax.your_domain.com DocumentRoot /home/matecat/matecat DirectoryIndex index.php index.php3 index.html index.htm index.shtml <Directory /> Options FollowSymLinks AllowOverride None </Directory> SSLEngine On SSLProtocol all -SSLv3 -SSLv2 SSLProxyEngine On SSLCertificateFile /etc/apache2/ssl-cert/my_matecat.crt SSLCertificateKeyFile /etc/apache2/ssl-cert/my_matecat.key <Directory /home/matecat/matecat/> Options All AllowOverride All Order allow,deny allow from all ExpiresActive On ExpiresByType text/html "access plus 1 minute" ExpiresByType text/css "access plus 1 minute" ExpiresByType text/javascript "access plus 1 minute" ExpiresByType image/gif "access plus 1 week" ExpiresByType image/jpeg "access plus 1 week" ExpiresByType image/jpg "access plus 1 week" ExpiresByType image/png "access plus 1 week" ExpiresByType image/vnd.microsoft.icon "access plus 1 week" ExpiresByType image/ico "access plus 1 week" ExpiresByType application/x-shockwave-flash "access plus 1 week" </Directory> php_flag register_globals off php_flag magic_quotes_gpc off ErrorLog ${APACHE_LOG_DIR}/matecat/error.log CustomLog ${APACHE_LOG_DIR}/matecat/access.log combined ServerSignature Off <Location /sse/ > ProxyPass http://localhost:7788/ ProxyPassReverse http://localhost:7788/ </Location> </VirtualHost>
Please pay attention to your domain name. For example, if you want to install Matecat on your localhost replace your_domain.com with localhost.
Create Certificate
Create a self-signed certificate (if you have a registered domain and already have a certificate, use your certificates)
cd ~
openssl req -x509 -newkey rsa:4096 -keyout my_matecat.key -out my_matecat.crt -days 365 -nodes (insert Country Name and Organization Name=Matecat)
mkdir /etc/apache2/ssl-cert
mv my_matecat.* /etc/apache2/ssl-cert/
mkdir /var/log/apache2/matecat
Enable virtual hosts and restart apache2
a2ensite 443-matecat.conf
a2ensite default-vhost.confservice apache2 reload
service apache2 restart
(If you want to enable multidomain, you must create another certificate for “*.ajax.your_domain.com”)
Go to top
Configure MateCat
Go to your matecat root folder in order to install PHP composer packages:
cd /home/matecat/matecat
php -r "readfile('https://getcomposer.org/installer');" | php
php composer.phar --no-dev install
su - matecat
cd ~/matecat/inc
cp oauth_config.ini.sample oauth_config.ini
cp task_manager_config.ini.sample task_manager_config.ini
cp Error_Mail_List.ini.sample Error_Mail_List.ini
cd ~/matecat/nodejs
Paste the configuration in nodejs/config.ini file:
vi /home/matecat/matecat/nodejs/config.ini
[server] address = 0.0.0.0 port = 7788 path = /channel/updates [queue] name = /topic/matecat_sse_notifications host = localhost port = 61613 login = admin passcode = admin [log] file = log/server.log level = debug [cors] allowedOrigins[] = *
Please note that in this example there is a section dedicated to CORS, please feel free to ignore it if you don't need it.
MateCat Filters API
Matecat Filters is a Translated API to extract translatable text from documents in several formats as XLIFF and to generate the translated document in the same format from the translated XLIFF. The API is made available through the RapidAPI marketplace.
To make your Matecat instance use the Matecat Filters API you should sign-up on RapidAPI at https://rapidapi.com/.
The Matecat Filters API is available on RapidAPI Hub at https://rapidapi.com/translated/api/matecat-filters/ (you can also search for matecat).
Once you are registered and logged in you should obtain your RapidAPI key.
When you get your API key you can use it in the config.ini file to fill the FILTERS_RAPIDAPI_KEY
value (see below).
Config File
Paste the configuration in the Matecat config.ini file:
vi /home/matecat/matecat/inc/config.ini
ENV = "production" CHECK_FS = no [production] CLI_HTTP_HOST = "https://your_domain.com" COOKIE_DOMAIN = "your_domain.com" ENABLE_MULTI_DOMAIN_API = no DB_SERVER = "localhost" DB_DATABASE = "matecat" DB_USER = "matecat" DB_PASS = "matecat01" REDIS_SERVERS = "tcp://localhost:6379" QUEUE_BROKER_ADDRESS = "tcp://localhost:61613" QUEUE_DQF_ADDRESS = "tcp://localhost:61613" QUEUE_JMX_ADDRESS = "http://localhost:8161" QUEUE_CREDENTIALS = "admin:admin" SSE_BASE_URL = "your_domain.com/sse" STORAGE_DIR = "/home/matecat/matecat/storage" SMTP_HOST = 'localhost' SMTP_PORT = '25' SMTP_SENDER = 'no-reply@matecat.loc' SMTP_HOSTNAME = 'localhost' ; Filters Configuration FILTERS_ADDRESS = 'https://translated-matecat-filters-v1.p.rapidapi.com' FILTERS_RAPIDAPI_KEY = "your RapidAPI Key" ; THIS SHOULD BE YOUR LEXIQA LICENSE, Request your license key at ; @see http://www.lexiqa.net LXQ_LICENSE = false LXQ_PARTNERID = false FILE_STORAGE_METHOD = 'fs' #TRACKING_CODES_VIEW_PATH = "internal_scripts/tracking_codes/"
Please pay particular attention to your domain URL, this MUST match your domain.
Then:
sed -i "s/\/home\/matecat\/storage/\/home\/matecat\/matecat\/storage/g" config.iniexit
Change owner files:
chown -R matecat:matecat /home/matecat/matecat/*
Compile JS code
Install yarn:
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt-get update
sudo apt-get install yarn
Install vendors and compile assets:
yarn install
yarn build:dev
Open your browser and check if you see the homepage running.
Check also if there is file called secret_login.dat in /home/matecat/matecat/inc folder.
Run Daemons
Launch analysis daemon:
cd /home/matecat/matecat/daemons
/bin/bash /home/matecat/matecat/daemons/restartAnalysis.sh
chown -R www-data /home/matecat/matecat/storage/
service apache2 restart
cd /home/matecat/matecat/nodejs/
npm install sse-channel
npm install
Run Node.js server in a screen with the following command:
screen -d -m -S 'node' node /home/matecat/matecat/nodejs/server.js
Port
Make sure this server can be seen from the outside, in particular that ports 80 and 443 are open
HTTP TCP 80 0.0.0.0/0 HTTPS TCP 443 0.0.0.0/0
Set Hosts
If your domain is not registered, add the ip to your hosts file
vi /etc/hosts
Add a new line with the ip and domain:
ip_server your_domain.com
Example:
127.0.0.1 localhost dev.matecat.com