🏢 Distributed Multi-Server Cluster Installation Guide

Enterprise step-by-step installation guide for deploying a high-availability, horizontally scalable Ring2All PBX cluster on Debian 13 (Trixie).

---

🏛️ Architecture Overview

In a distributed deployment, every platform layer is decoupled across dedicated nodes to eliminate single points of failure, provide high availability, and support horizontal scaling up to 100,000+ extensions and 15,000+ concurrent calls.

flowchart TD
    subgraph Clients["Clients & Edge Network"]
        SIP[SIP Endpoints & Hardphones]
        WebRTC[WebRTC Softphones]
        Trunks[Carrier SIP Trunks]
    end

    subgraph Edge["Edge Perimeter"]
        SBC["Ring2All SBC Gateway<br/>(Kamailio 6.1 + RTPEngine)"]
    end

    subgraph LoadBalancer["Internal DB Routing & Dynamic Proxy"]
        HAP["Local HAProxy Proxy (:5000 write / :5001 read)<br/>(Auto-tracks Patroni Leader)"]
    end

    subgraph TelephonyCore["Telephony Nodes (N+1 FreeSWITCH Cluster)"]
        FS1["Ring2All PBX Node 01<br/>192.168.10.41"]
        FS2["Ring2All PBX Node 02<br/>192.168.10.42"]
        FSN["Ring2All PBX Node N<br/>192.168.10.4x"]
    end

    subgraph WebApps["Web & API Layer"]
        Admin["Admin Server (:443)<br/>192.168.10.40"]
        API["Platform API (:3001) & Monitoring (:3500)"]
        Portal["User Portal (:443/portal)"]
        Switchboard["Switchboard (:443/switchboard)"]
    end

    subgraph DataStore["High-Availability Data & Storage Layer"]
        DB["PostgreSQL 17 HA Cluster<br/>(3 Nodes + Patroni + Etcd)"]
        Storage["GlusterFS / S3 Cluster<br/>(3 Nodes Replicated Storage)"]
    end

    SIP --> SBC
    WebRTC --> SBC
    Trunks --> SBC

    SBC -->|Encrypted WireGuard Mesh / SIP| FS1
    SBC -->|Encrypted WireGuard Mesh / SIP| FS2
    SBC -->|Encrypted WireGuard Mesh / SIP| FSN

    Admin --> API
    Portal --> API
    Switchboard --> API

    API --> HAP
    FS1 --> HAP
    FS2 --> HAP
    FSN --> HAP

    HAP -->|Port 5000 (Write)| DB
    HAP -->|Port 5001 (Read)| DB

    FS1 --> Storage
    FS2 --> Storage
    FSN --> Storage
    Admin --> Storage

---

🖥️ Server Roles & Lab Topology

The following reference topology assumes a private management network on 192.168.10.0/24:

Role	Hostname	IP Address	Target Packages	Hardware Sizing
DB Node 1	pg-node-01	192.168.10.34	PostgreSQL 17 + Patroni + Etcd	4 vCPU, 16 GB RAM, 250 GB NVMe
DB Node 2	pg-node-02	192.168.10.35	PostgreSQL 17 + Patroni + Etcd	4 vCPU, 16 GB RAM, 250 GB NVMe
DB Node 3	pg-node-03	192.168.10.36	PostgreSQL 17 + Patroni + Etcd	4 vCPU, 16 GB RAM, 250 GB NVMe
Storage Node 1	fs-node-01	192.168.10.37	glusterfs-server	2 vCPU, 4 GB RAM, 1+ TB HDD/SSD
Storage Node 2	fs-node-02	192.168.10.38	glusterfs-server	2 vCPU, 4 GB RAM, 1+ TB HDD/SSD
Storage Node 3	fs-node-03	192.168.10.39	glusterfs-server	2 vCPU, 4 GB RAM, 1+ TB HDD/SSD
Admin & API	admin	192.168.10.40	softswitch-admin, softswitch-api, softswitch-monitoring-api	4 vCPU, 8 GB RAM, 100 GB SSD
Telephony Node 1	fs-01	192.168.10.41	softswitch-telephony (FreeSWITCH 1.11+)	8-16 vCPU, 16-32 GB RAM, 100 GB SSD
Telephony Node 2	fs-02	192.168.10.42	softswitch-telephony (FreeSWITCH 1.11+)	8-16 vCPU, 16-32 GB RAM, 100 GB SSD
Telephony Node N	fs-0N	192.168.10.4x	softswitch-telephony (N+1 expansion)	8-16 vCPU, 16-32 GB RAM, 100 GB SSD
Portal Node (Optional)	portal	192.168.10.51	softswitch-portal, nginx	2 vCPU, 2 GB RAM, 30 GB SSD
Switchboard Node (Optional)	switchboard	192.168.10.52	softswitch-switchboard, nginx	2 vCPU, 2 GB RAM, 30 GB SSD

---

🛠️ Step-by-Step Installation Phases

Phase 1: High-Availability Database Cluster (Patroni + Etcd)

Instead of relying on a single database host, we deploy a 3-node PostgreSQL 17 cluster managed by Patroni and Etcd for Raft consensus.

1.1 Hostname & Resolution Mapping

On all three DB nodes (pg-node-01, pg-node-02, pg-node-03), configure /etc/hosts:

cat << 'EOF' >> /etc/hosts
192.168.10.34 pg-node-01
192.168.10.35 pg-node-02
192.168.10.36 pg-node-03
EOF

1.2 Install PostgreSQL 17, Etcd, and Patroni

Run on each DB node:

apt-get update
apt-get install -y curl gnupg2 lsb-release postgresql-17 patroni etcd-server etcd-client
systemctl stop postgresql
systemctl disable postgresql

1.3 Configure Etcd Cluster

On each node, configure /etc/default/etcd with its respective IP and peer list, then start the service:

systemctl enable --now etcd
etcdctl endpoint health

1.4 Configure Patroni & Initialize Schema

Create /etc/patroni/config.yml on each node specifying the Etcd endpoints and replication slots. Start Patroni on the primary node (pg-node-01), allow it to bootstrap the cluster, and start Patroni on standby nodes.

Once the leader is elected, install softswitch-db on the primary node to create the required Ring2All schemas:

# Add Ring2All APT repository
curl -fsSL https://repo.softswitchone.com/apt/setup_repo | bash

# Install database schemas and seed data
apt-get install -y softswitch-db

Verify cluster status on pg-node-01:

patronictl -c /etc/patroni/config.yml list

---

Phase 2: High-Availability Shared Storage (GlusterFS)

To share voicemail greetings, tenant call recordings, custom music-on-hold, and uploaded branding assets across all telephony and web nodes, deploy a 3-way replicated GlusterFS volume pool.

2.1 Prepare Storage Nodes

On fs-node-01, fs-node-02, and fs-node-03:

apt-get update && apt-get install -y glusterfs-server
systemctl enable --now glusterd

2.2 Peer Probe & Volume Creation

From fs-node-01:

gluster peer probe 192.168.10.38
gluster peer probe 192.168.10.39
gluster peer status

# Create replicated storage volumes
mkdir -p /data/glusterfs/brick1/{recordings,uploads,music}

gluster volume create ss-recordings replica 3 \
  fs-node-01:/data/glusterfs/brick1/recordings \
  fs-node-02:/data/glusterfs/brick1/recordings \
  fs-node-03:/data/glusterfs/brick1/recordings force

gluster volume create ss-uploads replica 3 \
  fs-node-01:/data/glusterfs/brick1/uploads \
  fs-node-02:/data/glusterfs/brick1/uploads \
  fs-node-03:/data/glusterfs/brick1/uploads force

gluster volume create ss-music replica 3 \
  fs-node-01:/data/glusterfs/brick1/music \
  fs-node-02:/data/glusterfs/brick1/music \
  fs-node-03:/data/glusterfs/brick1/music force

# Start volumes
gluster volume start ss-recordings
gluster volume start ss-uploads
gluster volume start ss-music
gluster volume info

---

Phase 3: Admin & API Server Setup

Log in to the Admin Server (192.168.10.40).

3.1 Install & Configure Local HAProxy

To route SQL queries dynamically to the active Patroni leader without hardcoding IPs, install HAProxy locally:

apt-get update
apt-get install -y haproxy make curl gnupg2 wget sudo
systemctl enable haproxy

Write /etc/haproxy/haproxy.cfg:

global
    log /dev/log local0
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

defaults
    log global
    mode tcp
    option tcplog
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

# Port 5000: Write queries dynamically routed to active Patroni Leader
frontend pg_write
    bind 127.0.0.1:5000
    default_backend pg_primary

backend pg_primary
    mode tcp
    option httpchk GET /primary
    http-check expect status 200
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server pg-node-01 192.168.10.34:5432 maxconn 100 maxqueue 10 check port 8008
    server pg-node-02 192.168.10.35:5432 maxconn 100 maxqueue 10 check port 8008
    server pg-node-03 192.168.10.36:5432 maxconn 100 maxqueue 10 check port 8008

# Port 5001: Read queries load balanced across Standby replicas
frontend pg_read
    bind 127.0.0.1:5001
    default_backend pg_replicas

backend pg_replicas
    mode tcp
    balance roundrobin
    option httpchk GET /replica
    http-check expect status 200
    default-server inter 3s fall 3 rise 2
    server pg-node-01 192.168.10.34:5432 check port 8008
    server pg-node-02 192.168.10.35:5432 check port 8008
    server pg-node-03 192.168.10.36:5432 check port 8008

Restart and verify:

haproxy -c -f /etc/haproxy/haproxy.cfg
systemctl restart haproxy
ss -ltn | grep -E '5000|5001'

3.2 Sync Database Credentials

mkdir -p /etc/softswitch
scp root@192.168.10.34:/etc/softswitch/db-credentials /etc/softswitch/db-credentials
chmod 600 /etc/softswitch/db-credentials

3.3 Register Repositories & Install API Packages

curl -fsSL https://repo.softswitchone.com/apt/setup_repo | bash

apt-get update
apt-get install -y nodejs build-essential python3 postgresql-client

# Install Ring2All API & telemetry daemons
apt-get install -y -o Dpkg::Options::="--force-overwrite" softswitch-api softswitch-monitoring-api

3.4 Mount Shared Storage Volumes

apt-get install -y glusterfs-client
mkdir -p /var/www/softswitch/uploads

# Mount with backup failover servers
mount -t glusterfs -o backup-volfile-servers=fs-node-02:fs-node-03 fs-node-01:/ss-uploads /var/www/softswitch/uploads

# Add fstab entry
cat << 'EOF' >> /etc/fstab
fs-node-01:/ss-uploads /var/www/softswitch/uploads glusterfs defaults,_netdev,backup-volfile-servers=fs-node-02:fs-node-03 0 0
EOF

3.5 Install Frontend Admin Dashboard

apt-get install -y -o Dpkg::Options::="--force-overwrite" softswitch-admin
nginx -t && systemctl reload nginx

---

Phase 4: User Portal & Switchboard Setup

You can host the User Portal and Switchboard on the Admin Server or on dedicated frontend hosts.

Option A: Co-located on the Admin Server

apt-get install -y -o Dpkg::Options::="--force-overwrite" softswitch-portal softswitch-switchboard
nginx -t && systemctl reload nginx

The portals will be accessible at https://admin.example.com/portal and https://admin.example.com/switchboard.

Option B: Dedicated Servers (e.g., portal on 192.168.10.51)

On the dedicated server:

apt-get update && apt-get install -y nginx
curl -fsSL https://repo.softswitchone.com/apt/setup_repo | bash
apt-get install -y softswitch-portal

Configure Nginx reverse proxy at /etc/nginx/sites-available/softswitch-portal:

server {
    listen 80;
    server_name portal.example.com;

    root /var/www/softswitch/portal;
    index index.html;

    location / {
        try_files $uri $uri/ /index.html;
    }

    location /api {
        proxy_pass http://192.168.10.40:3001;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    location /uploads/ {
        proxy_pass http://192.168.10.40/uploads/;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        expires 7d;
    }
}

Enable and reload:

ln -sf /etc/nginx/sites-available/softswitch-portal /etc/nginx/sites-enabled/
nginx -t && systemctl reload nginx

---

Phase 5: Telephony Nodes Setup (FreeSWITCH N+1 Cluster)

Repeat these steps on each Telephony node (fs-01, fs-02, …, fs-0N).

5.1 Base Setup & Local HAProxy

apt-get update && apt-get install -y curl gnupg2 wget make sudo haproxy
curl -fsSL https://repo.softswitchone.com/apt/setup_repo | bash

Deploy /etc/haproxy/haproxy.cfg (identical to the Admin node HAProxy configuration in Phase 3.1) so FreeSWITCH queries 127.0.0.1:5000 for writes and 127.0.0.1:5001 for reads.

systemctl restart haproxy
ss -ltn | grep 5000

5.2 Copy Database Credentials

mkdir -p /etc/softswitch
scp root@192.168.10.34:/etc/softswitch/db-credentials /etc/softswitch/db-credentials
chmod 600 /etc/softswitch/db-credentials

5.3 (Optional High-Scale) Deploy PgBouncer Connection Pooler (:6432)

For nodes handling >500 concurrent calls or high CPS bursts, deploy PgBouncer to multiplex thousands of Lua/ODBC queries into 20–30 persistent PostgreSQL connections:

apt-get install -y pgbouncer

Configure /etc/pgbouncer/pgbouncer.ini:

[databases]
ss_telephony = host=127.0.0.1 port=5000 dbname=ss_telephony
ring2all     = host=127.0.0.1 port=5000 dbname=ring2all
kamailio     = host=127.0.0.1 port=5000 dbname=kamailio

[pgbouncer]
logfile = /var/log/postgresql/pgbouncer.log
pidfile = /var/run/postgresql/pgbouncer.pid
listen_addr = 127.0.0.1
listen_port = 6432
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt
admin_users = postgres, ss_db_user

pool_mode = transaction
max_client_conn = 5000
default_pool_size = 25
reserve_pool_size = 5
ignore_startup_parameters = extra_float_digits, search_path, application_name

Create /etc/pgbouncer/userlist.txt with credentials and start:

source /etc/softswitch/db-credentials
echo "\"ss_db_user\" \"$DB_PASSWORD\"" > /etc/pgbouncer/userlist.txt
chown postgres:postgres /etc/pgbouncer/userlist.txt && chmod 640 /etc/pgbouncer/userlist.txt
systemctl enable --now pgbouncer
ss -ltn | grep 6432

5.4 Mount Shared Recordings & Music

apt-get install -y glusterfs-client
mkdir -p /var/lib/freeswitch/recordings
mkdir -p /usr/share/freeswitch/sounds/music

mount -t glusterfs -o backup-volfile-servers=fs-node-02:fs-node-03 fs-node-01:/ss-recordings /var/lib/freeswitch/recordings
mount -t glusterfs -o backup-volfile-servers=fs-node-02:fs-node-03 fs-node-01:/ss-music /usr/share/freeswitch/sounds/music

cat << 'EOF' >> /etc/fstab
fs-node-01:/ss-recordings /var/lib/freeswitch/recordings glusterfs defaults,_netdev,backup-volfile-servers=fs-node-02:fs-node-03 0 0
fs-node-01:/ss-music /usr/share/freeswitch/sounds/music glusterfs defaults,_netdev,backup-volfile-servers=fs-node-02:fs-node-03 0 0
EOF

5.5 Install FreeSWITCH & Ring2All Telephony Engine

apt-get update
apt-get install -y -o Dpkg::Options::="--force-overwrite" softswitch-telephony

The post-install script automatically detects the local HAProxy/PgBouncer port and configures /etc/odbc.ini and /etc/freeswitch/autoload_configs/switch.conf.xml.

Verify telephony service:

systemctl status freeswitch
fs_cli -x "sofia status"

5.6 Register Telephony Node in Admin Web UI

In the Admin Dashboard (https://admin.example.com/admin), navigate to Telephony > Telephony Servers and click + Add Telephony Node:

• Node Name: FS-01
• Internal IP: 192.168.10.41
• ESL Port: 8021
• Capacity: 10,000 extensions / 1,500 concurrent calls.

---

Phase 6: Automated Credentials Distribution Script

To simplify cluster maintenance and credential rotation, run this helper script from the DB Leader node:

cat << 'EOF' > /root/distribute-credentials.sh
#!/bin/bash
set -euo pipefail

# Nodes requiring synced db-credentials
NODES=(
    "192.168.10.40"  # Admin API Node
    "192.168.10.41"  # Telephony Node 01
    "192.168.10.42"  # Telephony Node 02
)

echo "Distributing /etc/softswitch/db-credentials across cluster..."
for node in "${NODES[@]}"; do
    echo "Syncing to $node..."
    ssh root@"$node" "mkdir -p /etc/softswitch"
    scp /etc/softswitch/db-credentials root@"$node":/etc/softswitch/db-credentials
    ssh root@"$node" "chmod 600 /etc/softswitch/db-credentials"
done
echo "Credential sync complete."
EOF

chmod +x /root/distribute-credentials.sh

---

🔍 Cluster Verification Checklist

Layer	Verification Command	Expected Result
Patroni DB	patronictl -c /etc/patroni/config.yml list	1 Leader (Running), 2 Replicas (Running)
Local HAProxy	ss -ltn | grep -E '5000|5001'	Ports 5000 and 5001 listening on 127.0.0.1
Shared Storage	gluster volume status	All bricks reported online and active
Recordings Sync	touch /var/lib/freeswitch/recordings/test.txt	File appears instantly on other nodes
Telephony Core	fs_cli -x "sofia status"	Internal & External profiles RUNNING
Platform API	curl -s http://127.0.0.1:3001/health	{"status":"ok"}
ESL Monitoring	systemctl status softswitch-monitoring-api	active (running) connected to all nodes

---

🔧 Production Troubleshooting

1. FreeSWITCH fails with “CORE DATABASE INITIALIZATION FAILURE”

• Cause: FreeSWITCH cannot reach the database on 127.0.0.1:5000 or /etc/odbc.ini credentials mismatch.
• Solution: Test ODBC directly:

  isql -v ss_telephony ss_db_user "$(grep DB_PASSWORD /etc/softswitch/db-credentials | cut -d= -f2)"

  Ensure local HAProxy is active (systemctl restart haproxy).

2. GlusterFS volume reports “Transport endpoint is not connected”

• Cause: Network interruption or one of the brick processes restarted.
• Solution: Remount with failover parameters:

  mount -a
  gluster volume heal ss-recordings

---

🚀 Next Steps

• Web Cluster & Load Balancing Guide: Scale multiple Web Admin and API nodes behind Cloudflare.
• Ring2All SBC Deployment: Deploy the perimeter SIP gateway with Kamailio 6.1 and RTPEngine.
• Ring2All BSS Deployment: Connect carrier billing, real-time OCS, and the customer store.