--- title: "🏢 Distributed Multi-Server Cluster Installation Guide" description: "Step-by-step enterprise guide for deploying Ring2All PBX across a high-availability multi-server cluster on Debian 13" --- > Enterprise step-by-step installation guide for deploying a high-availability, horizontally scalable Ring2All PBX cluster on Debian 13 (Trixie). --- ## 🏛️ Architecture Overview In a distributed deployment, every platform layer is decoupled across dedicated nodes to eliminate single points of failure, provide high availability, and support horizontal scaling up to **100,000+ extensions** and **15,000+ concurrent calls**. ```mermaid flowchart TD subgraph Clients["Clients & Edge Network"] SIP[SIP Endpoints & Hardphones] WebRTC[WebRTC Softphones] Trunks[Carrier SIP Trunks] end subgraph Edge["Edge Perimeter"] SBC["Ring2All SBC Gateway
(Kamailio 6.1 + RTPEngine)"] end subgraph LoadBalancer["Internal DB Routing & Dynamic Proxy"] HAP["Local HAProxy Proxy (:5000 write / :5001 read)
(Auto-tracks Patroni Leader)"] end subgraph TelephonyCore["Telephony Nodes (N+1 FreeSWITCH Cluster)"] FS1["Ring2All PBX Node 01
192.168.10.41"] FS2["Ring2All PBX Node 02
192.168.10.42"] FSN["Ring2All PBX Node N
192.168.10.4x"] end subgraph WebApps["Web & API Layer"] Admin["Admin Server (:443)
192.168.10.40"] API["Platform API (:3001) & Monitoring (:3500)"] Portal["User Portal (:443/portal)"] Switchboard["Switchboard (:443/switchboard)"] end subgraph DataStore["High-Availability Data & Storage Layer"] DB["PostgreSQL 17 HA Cluster
(3 Nodes + Patroni + Etcd)"] Storage["GlusterFS / S3 Cluster
(3 Nodes Replicated Storage)"] end SIP --> SBC WebRTC --> SBC Trunks --> SBC SBC -->|Encrypted WireGuard Mesh / SIP| FS1 SBC -->|Encrypted WireGuard Mesh / SIP| FS2 SBC -->|Encrypted WireGuard Mesh / SIP| FSN Admin --> API Portal --> API Switchboard --> API API --> HAP FS1 --> HAP FS2 --> HAP FSN --> HAP HAP -->|Port 5000 (Write)| DB HAP -->|Port 5001 (Read)| DB FS1 --> Storage FS2 --> Storage FSN --> Storage Admin --> Storage ``` --- ## 🖥️ Server Roles & Lab Topology The following reference topology assumes a private management network on `192.168.10.0/24`: | Role | Hostname | IP Address | Target Packages | Hardware Sizing | | :--- | :--- | :--- | :--- | :--- | | **DB Node 1** | `pg-node-01` | `192.168.10.34` | PostgreSQL 17 + Patroni + Etcd | 4 vCPU, 16 GB RAM, 250 GB NVMe | | **DB Node 2** | `pg-node-02` | `192.168.10.35` | PostgreSQL 17 + Patroni + Etcd | 4 vCPU, 16 GB RAM, 250 GB NVMe | | **DB Node 3** | `pg-node-03` | `192.168.10.36` | PostgreSQL 17 + Patroni + Etcd | 4 vCPU, 16 GB RAM, 250 GB NVMe | | **Storage Node 1** | `fs-node-01` | `192.168.10.37` | `glusterfs-server` | 2 vCPU, 4 GB RAM, 1+ TB HDD/SSD | | **Storage Node 2** | `fs-node-02` | `192.168.10.38` | `glusterfs-server` | 2 vCPU, 4 GB RAM, 1+ TB HDD/SSD | | **Storage Node 3** | `fs-node-03` | `192.168.10.39` | `glusterfs-server` | 2 vCPU, 4 GB RAM, 1+ TB HDD/SSD | | **Admin & API** | `admin` | `192.168.10.40` | `softswitch-admin`, `softswitch-api`, `softswitch-monitoring-api` | 4 vCPU, 8 GB RAM, 100 GB SSD | | **Telephony Node 1** | `fs-01` | `192.168.10.41` | `softswitch-telephony` (FreeSWITCH 1.11+) | 8-16 vCPU, 16-32 GB RAM, 100 GB SSD | | **Telephony Node 2** | `fs-02` | `192.168.10.42` | `softswitch-telephony` (FreeSWITCH 1.11+) | 8-16 vCPU, 16-32 GB RAM, 100 GB SSD | | **Telephony Node N** | `fs-0N` | `192.168.10.4x` | `softswitch-telephony` (N+1 expansion) | 8-16 vCPU, 16-32 GB RAM, 100 GB SSD | | **Portal Node** *(Optional)* | `portal` | `192.168.10.51` | `softswitch-portal`, `nginx` | 2 vCPU, 2 GB RAM, 30 GB SSD | | **Switchboard Node** *(Optional)* | `switchboard` | `192.168.10.52` | `softswitch-switchboard`, `nginx` | 2 vCPU, 2 GB RAM, 30 GB SSD | --- ## 🛠️ Step-by-Step Installation Phases ### Phase 1: High-Availability Database Cluster (Patroni + Etcd) Instead of relying on a single database host, we deploy a 3-node PostgreSQL 17 cluster managed by Patroni and Etcd for Raft consensus. #### 1.1 Hostname & Resolution Mapping On all three DB nodes (`pg-node-01`, `pg-node-02`, `pg-node-03`), configure `/etc/hosts`: ```bash cat << 'EOF' >> /etc/hosts 192.168.10.34 pg-node-01 192.168.10.35 pg-node-02 192.168.10.36 pg-node-03 EOF ``` #### 1.2 Install PostgreSQL 17, Etcd, and Patroni Run on each DB node: ```bash apt-get update apt-get install -y curl gnupg2 lsb-release postgresql-17 patroni etcd-server etcd-client systemctl stop postgresql systemctl disable postgresql ``` #### 1.3 Configure Etcd Cluster On each node, configure `/etc/default/etcd` with its respective IP and peer list, then start the service: ```bash systemctl enable --now etcd etcdctl endpoint health ``` #### 1.4 Configure Patroni & Initialize Schema Create `/etc/patroni/config.yml` on each node specifying the Etcd endpoints and replication slots. Start Patroni on the primary node (`pg-node-01`), allow it to bootstrap the cluster, and start Patroni on standby nodes. Once the leader is elected, install `softswitch-db` on the primary node to create the required Ring2All schemas: ```bash # Add Ring2All APT repository curl -fsSL https://repo.softswitchone.com/apt/setup_repo | bash # Install database schemas and seed data apt-get install -y softswitch-db ``` Verify cluster status on `pg-node-01`: ```bash patronictl -c /etc/patroni/config.yml list ``` --- ### Phase 2: High-Availability Shared Storage (GlusterFS) To share voicemail greetings, tenant call recordings, custom music-on-hold, and uploaded branding assets across all telephony and web nodes, deploy a 3-way replicated GlusterFS volume pool. #### 2.1 Prepare Storage Nodes On `fs-node-01`, `fs-node-02`, and `fs-node-03`: ```bash apt-get update && apt-get install -y glusterfs-server systemctl enable --now glusterd ``` #### 2.2 Peer Probe & Volume Creation From `fs-node-01`: ```bash gluster peer probe 192.168.10.38 gluster peer probe 192.168.10.39 gluster peer status # Create replicated storage volumes mkdir -p /data/glusterfs/brick1/{recordings,uploads,music} gluster volume create ss-recordings replica 3 \ fs-node-01:/data/glusterfs/brick1/recordings \ fs-node-02:/data/glusterfs/brick1/recordings \ fs-node-03:/data/glusterfs/brick1/recordings force gluster volume create ss-uploads replica 3 \ fs-node-01:/data/glusterfs/brick1/uploads \ fs-node-02:/data/glusterfs/brick1/uploads \ fs-node-03:/data/glusterfs/brick1/uploads force gluster volume create ss-music replica 3 \ fs-node-01:/data/glusterfs/brick1/music \ fs-node-02:/data/glusterfs/brick1/music \ fs-node-03:/data/glusterfs/brick1/music force # Start volumes gluster volume start ss-recordings gluster volume start ss-uploads gluster volume start ss-music gluster volume info ``` --- ### Phase 3: Admin & API Server Setup Log in to the Admin Server (`192.168.10.40`). #### 3.1 Install & Configure Local HAProxy To route SQL queries dynamically to the active Patroni leader without hardcoding IPs, install HAProxy locally: ```bash apt-get update apt-get install -y haproxy make curl gnupg2 wget sudo systemctl enable haproxy ``` Write `/etc/haproxy/haproxy.cfg`: ```haproxy global log /dev/log local0 chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin stats timeout 30s user haproxy group haproxy daemon defaults log global mode tcp option tcplog timeout connect 5000ms timeout client 50000ms timeout server 50000ms # Port 5000: Write queries dynamically routed to active Patroni Leader frontend pg_write bind 127.0.0.1:5000 default_backend pg_primary backend pg_primary mode tcp option httpchk GET /primary http-check expect status 200 default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions server pg-node-01 192.168.10.34:5432 maxconn 100 maxqueue 10 check port 8008 server pg-node-02 192.168.10.35:5432 maxconn 100 maxqueue 10 check port 8008 server pg-node-03 192.168.10.36:5432 maxconn 100 maxqueue 10 check port 8008 # Port 5001: Read queries load balanced across Standby replicas frontend pg_read bind 127.0.0.1:5001 default_backend pg_replicas backend pg_replicas mode tcp balance roundrobin option httpchk GET /replica http-check expect status 200 default-server inter 3s fall 3 rise 2 server pg-node-01 192.168.10.34:5432 check port 8008 server pg-node-02 192.168.10.35:5432 check port 8008 server pg-node-03 192.168.10.36:5432 check port 8008 ``` Restart and verify: ```bash haproxy -c -f /etc/haproxy/haproxy.cfg systemctl restart haproxy ss -ltn | grep -E '5000|5001' ``` #### 3.2 Sync Database Credentials ```bash mkdir -p /etc/softswitch scp root@192.168.10.34:/etc/softswitch/db-credentials /etc/softswitch/db-credentials chmod 600 /etc/softswitch/db-credentials ``` #### 3.3 Register Repositories & Install API Packages ```bash curl -fsSL https://repo.softswitchone.com/apt/setup_repo | bash apt-get update apt-get install -y nodejs build-essential python3 postgresql-client # Install Ring2All API & telemetry daemons apt-get install -y -o Dpkg::Options::="--force-overwrite" softswitch-api softswitch-monitoring-api ``` #### 3.4 Mount Shared Storage Volumes ```bash apt-get install -y glusterfs-client mkdir -p /var/www/softswitch/uploads # Mount with backup failover servers mount -t glusterfs -o backup-volfile-servers=fs-node-02:fs-node-03 fs-node-01:/ss-uploads /var/www/softswitch/uploads # Add fstab entry cat << 'EOF' >> /etc/fstab fs-node-01:/ss-uploads /var/www/softswitch/uploads glusterfs defaults,_netdev,backup-volfile-servers=fs-node-02:fs-node-03 0 0 EOF ``` #### 3.5 Install Frontend Admin Dashboard ```bash apt-get install -y -o Dpkg::Options::="--force-overwrite" softswitch-admin nginx -t && systemctl reload nginx ``` --- ### Phase 4: User Portal & Switchboard Setup You can host the User Portal and Switchboard on the Admin Server or on dedicated frontend hosts. #### Option A: Co-located on the Admin Server ```bash apt-get install -y -o Dpkg::Options::="--force-overwrite" softswitch-portal softswitch-switchboard nginx -t && systemctl reload nginx ``` The portals will be accessible at `https://admin.example.com/portal` and `https://admin.example.com/switchboard`. #### Option B: Dedicated Servers (e.g., `portal` on `192.168.10.51`) On the dedicated server: ```bash apt-get update && apt-get install -y nginx curl -fsSL https://repo.softswitchone.com/apt/setup_repo | bash apt-get install -y softswitch-portal ``` Configure Nginx reverse proxy at `/etc/nginx/sites-available/softswitch-portal`: ```nginx server { listen 80; server_name portal.example.com; root /var/www/softswitch/portal; index index.html; location / { try_files $uri $uri/ /index.html; } location /api { proxy_pass http://192.168.10.40:3001; proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } location /uploads/ { proxy_pass http://192.168.10.40/uploads/; proxy_http_version 1.1; proxy_set_header Host $host; expires 7d; } } ``` Enable and reload: ```bash ln -sf /etc/nginx/sites-available/softswitch-portal /etc/nginx/sites-enabled/ nginx -t && systemctl reload nginx ``` --- ### Phase 5: Telephony Nodes Setup (FreeSWITCH N+1 Cluster) Repeat these steps on each Telephony node (`fs-01`, `fs-02`, ..., `fs-0N`). #### 5.1 Base Setup & Local HAProxy ```bash apt-get update && apt-get install -y curl gnupg2 wget make sudo haproxy curl -fsSL https://repo.softswitchone.com/apt/setup_repo | bash ``` Deploy `/etc/haproxy/haproxy.cfg` (identical to the Admin node HAProxy configuration in Phase 3.1) so FreeSWITCH queries `127.0.0.1:5000` for writes and `127.0.0.1:5001` for reads. ```bash systemctl restart haproxy ss -ltn | grep 5000 ``` #### 5.2 Copy Database Credentials ```bash mkdir -p /etc/softswitch scp root@192.168.10.34:/etc/softswitch/db-credentials /etc/softswitch/db-credentials chmod 600 /etc/softswitch/db-credentials ``` #### 5.3 (Optional High-Scale) Deploy PgBouncer Connection Pooler (:6432) For nodes handling **>500 concurrent calls** or high CPS bursts, deploy PgBouncer to multiplex thousands of Lua/ODBC queries into 20–30 persistent PostgreSQL connections: ```bash apt-get install -y pgbouncer ``` Configure `/etc/pgbouncer/pgbouncer.ini`: ```ini [databases] ss_telephony = host=127.0.0.1 port=5000 dbname=ss_telephony ring2all = host=127.0.0.1 port=5000 dbname=ring2all kamailio = host=127.0.0.1 port=5000 dbname=kamailio [pgbouncer] logfile = /var/log/postgresql/pgbouncer.log pidfile = /var/run/postgresql/pgbouncer.pid listen_addr = 127.0.0.1 listen_port = 6432 auth_type = md5 auth_file = /etc/pgbouncer/userlist.txt admin_users = postgres, ss_db_user pool_mode = transaction max_client_conn = 5000 default_pool_size = 25 reserve_pool_size = 5 ignore_startup_parameters = extra_float_digits, search_path, application_name ``` Create `/etc/pgbouncer/userlist.txt` with credentials and start: ```bash source /etc/softswitch/db-credentials echo "\"ss_db_user\" \"$DB_PASSWORD\"" > /etc/pgbouncer/userlist.txt chown postgres:postgres /etc/pgbouncer/userlist.txt && chmod 640 /etc/pgbouncer/userlist.txt systemctl enable --now pgbouncer ss -ltn | grep 6432 ``` #### 5.4 Mount Shared Recordings & Music ```bash apt-get install -y glusterfs-client mkdir -p /var/lib/freeswitch/recordings mkdir -p /usr/share/freeswitch/sounds/music mount -t glusterfs -o backup-volfile-servers=fs-node-02:fs-node-03 fs-node-01:/ss-recordings /var/lib/freeswitch/recordings mount -t glusterfs -o backup-volfile-servers=fs-node-02:fs-node-03 fs-node-01:/ss-music /usr/share/freeswitch/sounds/music cat << 'EOF' >> /etc/fstab fs-node-01:/ss-recordings /var/lib/freeswitch/recordings glusterfs defaults,_netdev,backup-volfile-servers=fs-node-02:fs-node-03 0 0 fs-node-01:/ss-music /usr/share/freeswitch/sounds/music glusterfs defaults,_netdev,backup-volfile-servers=fs-node-02:fs-node-03 0 0 EOF ``` #### 5.5 Install FreeSWITCH & Ring2All Telephony Engine ```bash apt-get update apt-get install -y -o Dpkg::Options::="--force-overwrite" softswitch-telephony ``` The post-install script automatically detects the local HAProxy/PgBouncer port and configures `/etc/odbc.ini` and `/etc/freeswitch/autoload_configs/switch.conf.xml`. Verify telephony service: ```bash systemctl status freeswitch fs_cli -x "sofia status" ``` #### 5.6 Register Telephony Node in Admin Web UI In the Admin Dashboard (`https://admin.example.com/admin`), navigate to **Telephony > Telephony Servers** and click **+ Add Telephony Node**: - **Node Name**: `FS-01` - **Internal IP**: `192.168.10.41` - **ESL Port**: `8021` - **Capacity**: `10,000` extensions / `1,500` concurrent calls. --- ### Phase 6: Automated Credentials Distribution Script To simplify cluster maintenance and credential rotation, run this helper script from the DB Leader node: ```bash cat << 'EOF' > /root/distribute-credentials.sh #!/bin/bash set -euo pipefail # Nodes requiring synced db-credentials NODES=( "192.168.10.40" # Admin API Node "192.168.10.41" # Telephony Node 01 "192.168.10.42" # Telephony Node 02 ) echo "Distributing /etc/softswitch/db-credentials across cluster..." for node in "${NODES[@]}"; do echo "Syncing to $node..." ssh root@"$node" "mkdir -p /etc/softswitch" scp /etc/softswitch/db-credentials root@"$node":/etc/softswitch/db-credentials ssh root@"$node" "chmod 600 /etc/softswitch/db-credentials" done echo "Credential sync complete." EOF chmod +x /root/distribute-credentials.sh ``` --- ## 🔍 Cluster Verification Checklist | Layer | Verification Command | Expected Result | | :--- | :--- | :--- | | **Patroni DB** | `patronictl -c /etc/patroni/config.yml list` | 1 Leader (Running), 2 Replicas (Running) | | **Local HAProxy** | `ss -ltn \| grep -E '5000\|5001'` | Ports 5000 and 5001 listening on `127.0.0.1` | | **Shared Storage** | `gluster volume status` | All bricks reported online and active | | **Recordings Sync** | `touch /var/lib/freeswitch/recordings/test.txt` | File appears instantly on other nodes | | **Telephony Core** | `fs_cli -x "sofia status"` | Internal & External profiles `RUNNING` | | **Platform API** | `curl -s http://127.0.0.1:3001/health` | `{"status":"ok"}` | | **ESL Monitoring** | `systemctl status softswitch-monitoring-api` | `active (running)` connected to all nodes | --- ## 🔧 Production Troubleshooting ### 1. FreeSWITCH fails with "CORE DATABASE INITIALIZATION FAILURE" - **Cause**: FreeSWITCH cannot reach the database on `127.0.0.1:5000` or `/etc/odbc.ini` credentials mismatch. - **Solution**: Test ODBC directly: ```bash isql -v ss_telephony ss_db_user "$(grep DB_PASSWORD /etc/softswitch/db-credentials | cut -d= -f2)" ``` Ensure local HAProxy is active (`systemctl restart haproxy`). ### 2. GlusterFS volume reports "Transport endpoint is not connected" - **Cause**: Network interruption or one of the brick processes restarted. - **Solution**: Remount with failover parameters: ```bash mount -a gluster volume heal ss-recordings ``` --- ## 🚀 Next Steps - **[Web Cluster & Load Balancing Guide](web-cluster-load-balancing.md)**: Scale multiple Web Admin and API nodes behind Cloudflare. - **[Ring2All SBC Deployment](sbc-deployment.md)**: Deploy the perimeter SIP gateway with Kamailio 6.1 and RTPEngine. - **[Ring2All BSS Deployment](bss-deployment.md)**: Connect carrier billing, real-time OCS, and the customer store.