Installing Visulox Cluster

1. Introduction

This guide describes the process of provisioning and validating a highly available Visulox Cluster consisting of Master, Gateway, and Worker nodes. The setup follows a strict order, where each component depends on configuration and credentials created in earlier steps

2. Definitions

There are three types of server nodes in a Visulox Cluster, all served by NGINX installed on them:

Master

• Contains: Core, Portal, Connector

• Must be installed first

• Some Core logic runs only on the Master

• There can be only one Master

Gateway

• Contains: Connector

• Responsible for load balancing

• There can be multiple Gateways

Worker

• Contains: Core, Portal, Connector

• There can be multiple Workers

Core

• Core means the Visulox Core application. That's the amitego legacy app written in TCL language.

Portal

• Portal is the newly developed Visulox application written in Go. It includes several APIs and serves as the user contact point for both the UI4 panel and the dashboard panel. It communicates with Visulox Core via the Connector

Connector

• Connector is a Golang project that interfaces between the portal and core. Visulox core uses various protocols, including Socket, HTTP, and SOAP. The Connector project wraps all core APIs, regardless of protocol, into an HTTP REST API.

3. Dependencies

Visulox requires:

• PostgreSQL

• KeyDB

These can:

• Be installed separately before the Master installation

• OR be installed automatically during Master setup

4. Overview

Setting up a Visulox Cluster is a step-by-step process that builds a distributed system consisting of Master, Gateway, and Worker nodes. The setup must follow a strict order, as each stage depends on configuration and credentials created in earlier steps .

4.1 Conceptual Flow of the System

The cluster is composed of three roles:

• Master Node → Central control (Core, Portal, Connector)

• Gateway Node(s) → Entry point and traffic routing

• Worker Node(s) → Execution and scaling layer

Together, they form a system where the Master manages configuration, the Gateway exposes access, and Workers handle workload execution. This separation enables scalability and high availability.

4.2 Setup Process

The setup follows a top-down approach, where components are added and connected progressively.

Step 1 — Environment Preparation

Before installation:

• Ensure system time is synchronized

• Prepare at least three servers

• Verify required dependencies (PostgreSQL, KeyDB, networking)

This step is critical, as issues here affect the entire setup.

Step 2 — Master Installation

The Master is installed first and initializes the cluster.

• Sets up Core and Portal services

• Stores configuration and cluster state

• Defines the Gateway address

It also generates the credentials required for other nodes to join.

Step 3 — Gateway Setup

The Gateway is installed next and connects to the Master.

• Registers itself in the cluster

• Establishes secure communication

• Enables external access and request routing

At this stage, the system becomes accessible but not yet scalable.

Step 4 — Worker Deployment

Workers are added to provide execution capacity.

• Install required components

• Synchronize keys with the Master

• Attach to the cluster

This enables distributed execution and allows horizontal scaling.

Step 5 — Worker Registration & Integration

Each Worker is registered in the Portal layer and fully integrated.

• Makes Workers visible to the system

• Enables proper request routing

• Finalizes communication with the Master

5. Installation Order (Mandatory)

1. Master

2. Gateway

3. Worker

⚠️ During Master installation:

• The gateway hostname and IP must be provided

• The master does not work properly before the Gateway installation is completed

6. Prerequisites

6.1 Servers

• At least 3 servers needed for clustering:

  • 1 Master

  • 1 Gateway

  • 1 Worker

6.2 Time Synchronization (Mandatory)

⚠️ Critical Requirement

The server system time must be correct and synchronized before running:

• visulox start

• setup-master

• Any Docker-based service initialization (e.g., PostgreSQL container)

An incorrect system time can cause:

• TLS certificate validation failures

• Docker image pull errors

• PostgreSQL container startup failure

• setup-master interruptions

Example failure:

tls: failed to verify certificate: x509: certificate has expired or is not yet valid
current time 2026-02-16T14:04:39+01:00 is before 2026-02-19T20:57:44Z

Fix command:

REMOTE_TIME=$(curl -kfsSI https://www.google.com | awk -F': ' '/^[Dd]ate:/{print $2}' | tr -d '\r')
echo "$REMOTE_TIME"
sudo date -s "$REMOTE_TIME"
sudo hwclock --systohc

6.3 Installing & Using tmux (Optional)

Some installation operations can take a while; session disconnections or network issues may cause the process to fail. It is recommended to use tools like tmux, which allow running better work/SSH session management.

7. Master Setup

7.1 Minimum Requirements

• 2 Core CPU

• 4 GB RAM (8 GB recommended)

7.2 Installation

7.2.1 Setting up the Master server in cluster mode

The installation process is the same for single-instance mode (Installation Guide) with one exception: in setup-master, you must provide the Gateway hostname.

Required Version

Latest available Visulox 5 version.

When Master setup is completed, you should be able to browse UI4, Dashboard and login.

7.2.2 Generating Connector Secret for Master

A Connector should be defined on the Master server. The Main server generates a secret key for this Connector. The Gateway needs this key to connect to the Master Connector.

To create the Gateway connector, run this on Master:

sudo /opt/visulox-portal/bin/vlx-portald create-maingateway-connector -c /opt/visulox-portal/configs/config.json --output json

Sample output format would be like:

{"secret":"63301a03-3c1b-4ec5-9590-a84961304ba2"}

Parameters:

• Use --output json or --output secret to get the key in JSON or plain format.

• -c /opt/visulox-portal/configs/config.json provide the config file of the Portal server. It contains Portal database credentials which is needed to create Connector in Portal database.

sudo /opt/visulox-portal/bin/vlx-portald create-maingateway-connector -c /opt/visulox-portal/configs/config.json --output json --name gateway-connector-02

8. Gateway Setup

8.1 Installing Main Gateway (or Connector) RPM

Remove the old version, if it exists:

sudo dnf remove vlx-connector vlx-portal vlx-portal-rte -y || true
sudo rm -rf /opt/visulox-portal* || true
sudo rm -rf /var/vlx-portal || true
sudo rm -f /opt/visulox-portal/.setup_done || true
sudo systemctl daemon-reload || true

Which package should I install?

• vlx-main-gateway (recommended) contains only the components required for a Main Gateway deployment.

• vlx-connector contains Main Gateway functionality plus additional Connector-related components and dependencies.

For most deployments, vlx-main-gateway is sufficient and recommended. It is the newer, smaller package and avoids installing components that are not required for a Gateway-only setup.

Install the RPM on the target server:

cd /tmp/vlx-portal
sudo dnf install vlx-main-gateway-*.rpm -y

Alternatively, you can install the full Connector package:

cd /tmp/vlx-portal
sudo dnf install vlx-connector-*.rpm -y

Note that vlx-connector requires additional server preparation steps and dependencies.
Before installing it, complete the prerequisites described in the Server Preparation section of the Visulox Installation Guide: Installing Visulox (Server preparation).

8.2 Gateway Setup (setup-maingateway)

After installing the RPM, run the Gateway setup wizard.

The setup wizard guides you through the required configuration and generates the Gateway configuration files.

The setup can be used in two modes:

Interactive Setup

On the Gateway server, run:

sudo /opt/visulox-portal-connector/bin/vlx-connectord setup-maingateway

Regarding the Connector Secret prompt, use the secret of the Gateway Connector that was created in Section 7.2.2.

During the setup, you will be asked:

Enter XPRA Route Type [safe/performant] (default safe):

XPRA Route Type

safe (recommended):

• All XPRA traffic is routed through the Gateway over HTTPS (port 443).

• XPRA daemon ports do not need to be exposed externally.

• Simpler firewall configuration.

• Better security posture.

• Suitable for most deployments.

performant:

• The Gateway connects directly to the XPRA daemon ports.

• Eliminates one proxy hop and may provide slightly better performance.

• Requires multiple XPRA-related ports to be reachable between clients and the Gateway.

• Increases the number of exposed network ports and firewall rules.

In most cases, safe should be used unless there is a specific performance requirement that justifies exposing additional ports.

Non-Interactive Setup

In some situations, such as automated deployments or scripting, you may want to run the setup without interactive prompts.

Generate a template file:

sudo /opt/visulox-portal-connector/bin/vlx-connectord setup-maingateway --get-template /tmp/vlx-portal

This command creates:

/tmp/vlx-portal/setup_maingateway_template.json
/tmp/vlx-portal/setup_maingateway_template.json_README.txt

A default-generated template looks similar to:

{
  "connector_secret": "",
  "gateway_host": "",
  "gateway_port": "8180",
  "main_server_host": "",
  "main_server_port": "8080",
  "dns_resolver": "172.17.3.1",
  "xpra_route_type": "safe"
}

Input definitions:

connector_secret — UUID of the Main Gateway connector from the Portal database (required).

gateway_host     — Public URL used by clients to access the Gateway
                   (for example: https://access.example.com) (required).

gateway_port     — HTTPS listen port used by NGINX on this host.
                   Default: 8180.

main_server_host — Portal hostname or IP address (required).

main_server_port — Portal REST API port.
                   Default: 8080.

dns_resolver     — Optional comma-separated DNS resolver IPs used by NGINX.
                   If empty, system DNS resolvers are used.

xpra_route_type  — Optional.
                   Values: "safe" or "performant".
                   Default: "safe".

After filling the template file, run the setup using the JSON file:

sudo /opt/visulox-portal-connector/bin/vlx-connectord setup-maingateway \
  --file /tmp/vlx-portal/setup_maingateway_template.json

Multi-gateway:

For mulit-gateway setup, we need to add --name {gateway_name} at the end of create-maingateway-connector command:

sudo /opt/visulox-portal/bin/vlx-portald create-maingateway-connector -c /opt/visulox-portal/configs/config.json --output json --name {gateway_name}

8.3 Verifying Installation

cat /etc/nginx/includes/visulox-gateway-main-upstream-servers.conf

Expected: Master server IP is listed.

8.4 Verifying Setup

Visit:

https://$gateway_domain:8180/ui4/login

Test both NativeClient and WebClient.

8.5 Verifying Network Accessibility (Troubleshooting)

This step is only required if sessions fail to start or applications become stuck during launch.

The required network connectivity depends on the selected XPRA Route Type:

Safe Mode

In safe mode, all XPRA traffic is routed through the Gateway using HTTPS. XPRA daemon ports are not exposed and do not need to be reachable from clients.

Verify that:

• the Gateway can reach the Master and Worker Portal services.

• HTTPS communication between Gateway, Master, and Worker nodes is working.

• DNS resolution is functioning correctly between cluster nodes.

Performant Mode

In performant mode, the Gateway connects directly to XPRA daemon ports on the Master and Worker nodes. Therefore, the Gateway must be able to reach the XPRA port range:

14600-15600/tcp

To verify connectivity, select a test port from this range.

On the Master or Worker server:

nc -l -p <PORT_NUMBER>

On the Gateway server:

nc -w 3 <SERVER_HOSTNAME_OR_IP> <PORT_NUMBER>

After the connection is established, type a test message such as:

Hello

If the message appears on the receiving server, connectivity is working correctly.

If the connection fails, traffic may be blocked by:

• the Gateway server firewall

• the Master or Worker server firewall

• a firewall between the servers

• network routing restrictions

9. Worker Setup

9.1 Core and Portal RPM Installation

Only Core and Portal RPMs should be installed on Worker as explained in Installing Visulox from 1st step to Step 2.
So, it means that no setup-master is needed at this point

9.2 Starting and Attaching Core

Then start Core service and attach it:

# Start services
sudo visulox start
sudo visulox portal attach -proxy

9.3 Key synchronization

Both Visulox Core modules must have the same key; if their keys are not the same, they cannot be configured as a cluster.

Find the key on Master in this path:

/opt/visulox/etc/key

From the Master server, copy the VISULOX key to the Worker server:

sudo scp /opt/visulox/etc/key <SERVER_USER>@<WORKER_HOST>:/tmp/vlx-master-key

On the Worker server, replace the existing key with the one copied from the Master:

sudo rm -f /opt/visulox/etc/key
sudo cp /tmp/vlx-master-key /opt/visulox/etc/key
sudo chown vlx:vlxgroup /opt/visulox/etc/key
sudo chmod 644 /opt/visulox/etc/key

Restart Worker Visulox when key is copied from Master:

sudo visulox restart

9.4 Firewall Configuration

Visulox Core nodes communicate on specific port ranges. Required firewall rules depend on the selected XPRA Route Type.

Safe Mode

In safe mode, XPRA traffic is routed through the Gateway over HTTPS. XPRA daemon ports do not need to be exposed.

Open these ports on Master and Worker servers:

sudo touch /usr/lib/firewalld/services/vlx.xml

Content:

<service>
  <short>VISULOX Firewall Rules</short>
  <description>The following rules are needed for the VISULOX Service and the VISULOX Components.</description>
  <port protocol="tcp" port="6000-6500"/>
  <port protocol="tcp" port="7001-7010"/>
  <port protocol="tcp" port="21"/>
  <port protocol="tcp" port="1443"/>
  <port protocol="tcp" port="8114"/>
  <port protocol="tcp" port="8115"/>
</service>

Apply:

sudo firewall-cmd --permanent --zone=public --add-service=vlx
sudo firewall-cmd --reload

Performant Mode

In performant mode, XPRA daemon ports must also be reachable. Use this mode only when direct XPRA connectivity is required.

Open these ports on Master and Worker servers:

sudo touch /usr/lib/firewalld/services/vlx.xml

Content:

<service>
  <short>VISULOX Firewall Rules</short>
  <description>The following rules are needed for the VISULOX Service and the VISULOX Components.</description>
  <port protocol="tcp" port="6000-6500"/>
  <port protocol="tcp" port="7001-7010"/>
  <port protocol="tcp" port="21"/>
  <port protocol="tcp" port="1443"/>
  <port protocol="tcp" port="8114"/>
  <port protocol="tcp" port="8115"/>
  <port protocol="tcp" port="14600-15600"/>
</service>

Apply:

sudo firewall-cmd --permanent --zone=public --add-service=vlx
sudo firewall-cmd --reload

9.5 Attaching Worker to Master

At the final stage, we attach the Visulox Core of the Master server to that of the worker server(s).

Run on Master:

sudo -s
cd ~
visulox attach $domain_of_worker_server

Expected output:

attached node vSFTESTWORKERSERVER06 at location 'STANDARD' zone 'STANDARD'
etcd cluster: added vSFTESTMASTERSERVER06
etcd client started

Then check:

visulox status

Expected Output (Example)

---------------------------------------------------------------------------------------------------------------------
|                  node | location |     zone | nodeinfo | status | archive |    servicestarttime |         version |
---------------------------------------------------------------------------------------------------------------------
| vSFTESTMASTERSERVER05 | STANDARD | STANDARD | STANDARD |     up |         | 2026-03-05 12:47:39 | 5.alpha-vlxxpra |
| vSFTESTWORKERSERVER05 | STANDARD | STANDARD | STANDARD |     up |         | 2026-03-05 12:46:34 | 5.alpha-vlxxpra |
---------------------------------------------------------------------------------------------------------------------

9.6 Troubleshooting

Run these commands on Worker:

visulox integrity
journalctl -u visulox -n 100 -f

visulox log -l info
visulox log -l debug
visulox log -l error

Example visulox integrity output:

integrityXPRA:******************** no jet done ***************** 
Integrity-Check on vW1DEMO : VISULOX PRIVILEGED ACCESS MANAGEMENT EVALUATION / 5.0.2 / 2026-07-13 07:54:23 UTC
    ---------------------------------------------------------------------------------------------------------------
    | option   | cat                               | info                                            | returnCode |
    ---------------------------------------------------------------------------------------------------------------
    | -license | License                           | time limited in use                             | WARNING(2) |
    | -db      | connect                           | VISULOX cannot run from : /home/develadmin      | FAILURE(1) |
    | -db      | status                            | VISULOX cannot run from : /home/develadmin      | FAILURE(1) |
    | -vlx     | Cluster                           | offline                                         | WARNING(2) |
    | -sys     | Key vlxkey                        | perms not 00640                                 | FAILURE(1) |
    | -sys     | Size of /opt/visulox/var/filearea | has 25.88GB. need 80.0GB                        | WARNING(2) |
    | -sys     | timedatectl                       | not synchronised                                | WARNING(2) |
    | -sys     | FQDN:w1demo.tbsol.de              | check DNS: No Reserse lookup answer for ::1     | FAILURE(1) |
    | -store   | Filestore                         | filestore warning (at 25.88GB) threshold 50.0GB | WARNING(2) |
    | -mail    | Mail                              | localhost:25 not reachable                      | WARNING(2) |
    ---------------------------------------------------------------------------------------------------------------

ExitCode: FAILURE: One or more tests failed! For more information use <visulox log>

10. Registering Worker in Master

You should insert a Portal server in the database for the Worker server you want to attach to the cluster.

Run create-worker-portal-server and keep the Portal server secret that is displayed.

Note: Portal server secret will be shown to you once only.

There are three modes to run this command:

Plain text mode:

In the Master server, run the following command with the domain and IP address of the worker server:

sudo /opt/visulox-portal/bin/vlx-portald create-worker-portal-server -c /opt/visulox-portal/configs/config.json --domain $WORKER_SERVER_HOSTNAME --server-ip $WORKER_SERVER_IP

You can also pass --server-ip to provide the IP of the worker server. Note that if DNS is not configured, or has no record linking $WORKER_SERVER_DOMAIN to $WORKER_SERVER_IP, you will receive an error.

JSON mode:

If you want to receive the command result in a JSON format, you can add the --output json argument. For example:

sudo /opt/visulox-portal/bin/vlx-portald create-worker-portal-server -c /opt/visulox-portal/configs/config.json --domain <hostname.domain> --server-ip xxx.xxx.xxx.xxx --output json
{"server_name":"vSFTESTWORKERSERVER10","domain":"<hostname.domain>","server_ip":"xxx.xxx.xxx.xxx","secret":"<xxxxxxxxx>"}

Only secret mode:

If you only want to receive the portal secret by running the above command, you can add the --output secret switch to the first command. For example:

sudo /opt/visulox-portal/bin/vlx-portald create-worker-portal-server -c /opt/visulox-portal/configs/config.json --domain <hostname.domain> --server-ip xxx.xxx.xxx.xxx --output secret
<xxxxxxxxx>

⚠️ Write down the secret of the created portal server, as it is only shown once.

11. Installing Portal On Worker

11.1 Portal Installation

Download the desired RTE, Portal, and Connector RPMs to /tmp/vlx-portal/.

Then run this from Worker:

cd /tmp/vlx-portal

sudo dnf install vlx-portal-rte-*.rpm -y
sudo dnf install vlx-portal-*.rpm -y
sudo dnf install vlx-connector-*.rpm -y

11.2 Portal Setup

From Worker run setup-node. You can run it in two modes:

Interactive

sudo /opt/visulox-portal/bin/vlx-portald setup-node

This command will ask for the portal secret. You should use the secret you received in last step.

Non-interactive

In some cases, you may not want to enter the inputs of the setup-node command interactively. For example, if you are writing a script, you need to run this command in a non-interactive mode to have control over the inputs.
To do so, you need to create template files for the setup command to run. You need to run the following command:

/opt/visulox-portal/bin/vlx-portald setup-node --get-template /tmp/vlx-portal

After running the above command, a setup_node_template.json template file and a setup_node_template_README.txt file will be created in the /tmp/vlx-portal directory.

An example of a default-generated template file looks like this:

{
  "master_ip": "",
  "master_port": "8080",
  "secret": "",
  "dns_resolver_ips": [
    "172.17.3.1"
  ],
  "keydb_tls": {
    "ca_cert_path": "",
    "client_cert_path": "",
    "client_key_path": ""
  },
  "postgresql_tls": {
    "ca_cert_path": "",
    "client_cert_path": "",
    "client_key_path": ""
  }
}

You need to fill in the required items of this template file by reading the README file.
Then, consider you have prepared a setup JSON file using the template file named setup-node.json; You can run the setup node command non-interactively as follows:

sudo /opt/visulox-portal/bin/vlx-portald setup-node --file /path/to/setup-node.json

⚠️ You should only set values for keydb_ca_cert_path, keydb_client_cert_path, and keydb_client_key_path fields in the setup-node.json file if you haven't installed KeyDB automatically via the setup-master command, and you use an external KeyDB that has a "self-signed" certificate.

11.3 Verifying network accessibility

If you have issues with setting up sessions, you can check what we did in step 8.5 and check the range of 14600-15600/tcp.

12. Setting Up Additional Workers

• Recommended: at least 3 nodes (odd number)

• Repeat worker setup

13. Final Testing

• Browse using Gateway address:

https://$gateway_domain:8180/ui4/login

• Run some apps using Vispra and WebClient.

• Run health-check command from all servers