🏢 Part 8: Enterprise-Grade Distributed Deployment on Debian 13

Welcome to the eighth and final installment of our “Debian 13 Clustering & Distribution” series. In the previous articles, we packaged Telephony Server, Kamailio, and RTPEngine, and explored individual high-availability storage and database clustering configurations. In this final guide, we will unify these concepts into a production-ready, multi-server enterprise environment. We will walk through setting up a complete distributed topology on Debian 13 (Trixie). Instead of repeating base setups, we will integrate the self-healing PostgreSQL 17 cluster managed by Patroni & Etcd configured in Part 4, and the high-availability GlusterFS storage network deployed in Part 5. We will configure local HAProxy proxies on application nodes to dynamically route database requests based on Patroni’s active roles. We will also install our application APIs and frontend portals, deploy dedicated Telephony Server telephony nodes connecting via ODBC and mounting shared media files, detail credentials distribution, and establish troubleshooting and validation checks for cross-service connectivity.

---

🏗️ Architecture Overview

In a distributed deployment, each component is decoupled and assigned to dedicated servers to ensure high availability, redundancy, and maximum horizontal scalability.

┌─────────────────────────────────────────────────────────────────────────────────┐
│                           DISTRIBUTED ARCHITECTURE                               │
├─────────────────────────────────────────────────────────────────────────────────┤
│                                                                                  │
│   ┌─────────────────────────────────────────────────────────────────────────┐   │
│   │                        LOAD BALANCER (HAProxy/Nginx)                     │   │
│   │                         Public IP: lb.example.com                        │   │
│   └─────────────────────────────────────────────────────────────────────────┘   │
│                    │              │              │              │                │
│          ┌─────────┴──────┐ ┌────┴────┐ ┌──────┴──────┐ ┌─────┴─────┐          │
│          │                │ │         │ │             │ │           │          │
│          ▼                ▼ ▼         ▼ ▼             ▼ ▼           ▼          │
│   ┌────────────┐   ┌────────────┐   ┌────────────┐   ┌────────────┐            │
│   │   ADMIN    │   │   PORTAL   │   │ SWITCHBOARD│   │   API(s)   │            │
│   │  Server    │   │   Server   │   │   Server   │   │  Server(s) │            │
│   └────────────┘   └────────────┘   └────────────┘   └────────────┘            │
│          │                │              │                  │                   │
│          └────────────────┴──────────────┴──────────────────┘                   │
│                                     │                                            │
│                                     ▼                                            │
│   ┌─────────────────────────────────────────────────────────────────────────┐   │
│   │                     PRIVATE NETWORK (192.168.10.0/24)                    │   │
│   └─────────────────────────────────────────────────────────────────────────┘   │
│          │                                                      │                │
│          ▼                                                      ▼                │
│   ┌──────────────────────────────────────────────┐    ┌───────────────────────┐ │
│   │      PostgreSQL 17 CLUSTER (3 Nodes)          │    │   TELEPHONY CLUSTER   │ │
│   │        Managed by Patroni & Etcd             │    │                       │ │
│   │  ┌───────────────┐  ┌──────────┐  ┌────────┐ │    │  ┌────────┐ ┌────────┐ │ │
│   │  │  pg-node-01   │  │pg-node-02│  │pg-node-│ │    │  │  FS-01 │ │  FS-02 │ │ │
│   │  │192.168.10.34  │  │192.168.  │  │   03   │ │    │  │        │ │        │ │ │
│   │  │  Active Leader│  │  10.35   │  │192.168.│ │    │  └────────┘ └────────┘ │ │
│   │  │  (Read/Write)  │  │ Replica  │  │ 10.36  │ │    │      + FS-N (N+1)      │ │
│   │  └───────────────┘  └──────────┘  └────────┘ │    └───────────────────────┘ │
│   │    PostgreSQL 17 · HAProxy :5000 (write API)   │                             │
│   │                   · HAProxy :5001 (read LB)    │                             │
│   └──────────────────────────────────────────────┘                              │
│                                                                                  │
└─────────────────────────────────────────────────────────────────────────────────┘

---

🖥️ Server Roles & Requirements

To set up a production-ready lab, we allocate dedicated servers with the following configurations:

Role	Hostname	Example IP	Target Package	Minimum Spec
DB Node 1	pg-node-01	192.168.10.34	PostgreSQL 17 + Patroni + Etcd	4 vCPU, 16GB RAM, SSD
DB Node 2	pg-node-02	192.168.10.35	PostgreSQL 17 + Patroni + Etcd	4 vCPU, 16GB RAM, SSD
DB Node 3	pg-node-03	192.168.10.36	PostgreSQL 17 + Patroni + Etcd	4 vCPU, 16GB RAM, SSD
File Server 1	fs-node-01	192.168.10.37	glusterfs-server	2 vCPU, 4GB RAM, Storage
File Server 2	fs-node-02	192.168.10.38	glusterfs-server	2 vCPU, 4GB RAM, Storage
File Server 3	fs-node-03	192.168.10.39	glusterfs-server	2 vCPU, 4GB RAM, Storage
Admin Server	admin	192.168.10.40	softswitch-admin, softswitch-api	2 vCPU, 4GB RAM, SSD
Portal Server (Option B)	portal	192.168.10.51	softswitch-portal	2 vCPU, 2GB RAM, SSD
Switchboard Server (Option B)	switchboard	192.168.10.52	softswitch-switchboard	2 vCPU, 2GB RAM, SSD
Telephony Nodes 1-5	fs-01 to fs-05	192.168.10.41 - 192.168.10.45	softswitch-telephony	4 vCPU, 8GB RAM, SSD

---

🛠️ Step-by-Step Deployment Guide

Phase 1: High-Availability Database Cluster Setup

Instead of manual replication, we deploy a High-Availability PostgreSQL 17 cluster managed by Patroni and Etcd.

1. Configure hostnames (pg-node-01, pg-node-02, and pg-node-03) and name resolutions in /etc/hosts across all nodes.
2. Follow the detailed steps in Part 4: PostgreSQL High-Availability Clustering with Patroni and Etcd to install etcd, clean PostgreSQL data paths, configure /etc/default/etcd and /etc/patroni/config.yml on each server, start the services, and install the softswitch-db package to initialize database schemas and generate system credentials.
3. Verify that the Patroni cluster status is healthy on the primary node:

   patronictl -c /etc/patroni/config.yml list

---

Phase 2: GlusterFS Shared Storage Cluster Setup

We deploy a split-brain safe active-active storage pool to store telephony assets, call recordings, music, and portal uploads.

1. Prepare host resolutions on /etc/hosts mapping fs-node-01 (192.168.10.37), fs-node-02 (192.168.10.38), and fs-node-03 (192.168.10.39).
2. Follow Part 5: Distributed Storage with GlusterFS on Debian 13 to peer-probe nodes, configure brick directories, and create three-way replicated volumes.
3. Verify that the storage pool and volumes are active:

   gluster volume info

   Ensure the following volumes are created and running: ss-recordings, ss-uploads, and ss-music.

---

Phase 3: Admin Server Setup

Log in to the Admin Server (192.168.10.40).

3.1 Install and Configure Local HAProxy

To support dynamic database failover transparently, we deploy HAProxy locally on this node before installing any Ring2All softswitch packages.

[!NOTE] Installer Auto-Detection: The Ring2All installer scripts (softswitch-api, softswitch-telephony, etc.) are designed to automatically detect if a local HAProxy instance is installed. If detected, the scripts assume a distributed, high-availability topology and automatically configure database profiles, system DSNs, and /etc/odbc.ini to route traffic through the local proxy on loopback ports 5000 (writes) and 5001 (reads). This eliminates the need for manual connection profiling.

HAProxy acts as a local proxy, querying Patroni’s REST API on port 8008 to check which node holds the active leadership lock.

Install HAProxy:

apt-get update
apt-get install -y haproxy make curl gnupg2 wget sudo
systemctl start haproxy
systemctl enable haproxy

Write the following configuration into /etc/haproxy/haproxy.cfg:

cat << 'EOF' > /etc/haproxy/haproxy.cfg
global
    log /dev/log local0
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

defaults
    log global
    mode tcp
    option tcplog
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

# Port 5000: Write transactions routed dynamically to Patroni Leader
frontend pg_write
    bind 127.0.0.1:5000
    default_backend pg_primary

backend pg_primary
    mode tcp
    option httpchk GET /primary
    http-check expect status 200
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server pg-node-01 192.168.10.34:5432 maxconn 100 maxqueue 10 check port 8008
    server pg-node-02 192.168.10.35:5432 maxconn 100 maxqueue 10 check port 8008
    server pg-node-03 192.168.10.36:5432 maxconn 100 maxqueue 10 check port 8008

# Port 5001: Read transactions load balanced across all Standby nodes
frontend pg_read
    bind 127.0.0.1:5001
    default_backend pg_replicas

backend pg_replicas
    mode tcp
    balance roundrobin
    option httpchk GET /replica
    http-check expect status 200
    default-server inter 3s fall 3 rise 2
    server pg-node-01 192.168.10.34:5432 check port 8008
    server pg-node-02 192.168.10.35:5432 check port 8008
    server pg-node-03 192.168.10.36:5432 check port 8008
EOF

Validate and restart the service, then verify that port 5000 is listening locally:

haproxy -c -f /etc/haproxy/haproxy.cfg
systemctl restart haproxy
ss -ltn | grep 5000

3.2 Copy Database Credentials

Create the config path and copy the /etc/softswitch/db-credentials file from the DB Leader node:

mkdir -p /etc/softswitch
# Run on Admin node:
scp root@192.168.10.34:/etc/softswitch/db-credentials /etc/softswitch/db-credentials

3.3 Register Repositories

Configure repository sources and signatures for Ring2All and NodeSource (Node.js 22) using the unified setup script:

# Clean up any conflicting NodeSource repositories from previous attempts
rm -f /etc/apt/sources.list.d/nodesource*
rm -f /usr/share/keyrings/nodesource*
rm -f /etc/apt/keyrings/nodesource*

# Download and run the unified repository setup script
# (This automatically configures both Ring2All and NodeSource repositories)
curl -fsSL https://repo.softswitchone.com/apt/setup_repo | bash

3.4 Install Node.js and API Packages

Install Node.js (which includes npm), build tools, postgresql-client, and the REST API and telemetry modules. The post-installation script detects local HAProxy port 5000 and automatically links and starts the services:

# Install Node.js, build dependencies, and postgresql-client (do NOT install 'npm' separately)
apt-get update
apt-get install -y nodejs build-essential python3 postgresql-client

# Install Ring2All API packages
apt-get install -y -o Dpkg::Options::="--force-overwrite" softswitch-api softswitch-monitoring-api

3.5 Mount Shared Storage Volumes

Configure host resolutions in /etc/hosts and mount the shared storage uploads directory using backup-volfile-servers to protect against node outages:

# On the Admin Server:
# Add storage hosts to /etc/hosts (update IPs if they differ in your lab)
cat << 'EOF' >> /etc/hosts
192.168.10.37 fs-node-01
192.168.10.38 fs-node-02
192.168.10.39 fs-node-03
EOF

apt-get install -y glusterfs-client
mkdir -p /var/www/softswitch/uploads

# Mount volume
mount -t glusterfs -o backup-volfile-servers=fs-node-02:fs-node-03 fs-node-01:/ss-uploads /var/www/softswitch/uploads

# Add fstab entry with backup mount configurations
cat << 'EOF' >> /etc/fstab
fs-node-01:/ss-uploads /var/www/softswitch/uploads glusterfs defaults,_netdev,backup-volfile-servers=fs-node-02:fs-node-03 0 0
EOF

3.6 Install Frontend Admin Dashboard

apt-get install -y -o Dpkg::Options::="--force-overwrite" softswitch-admin
nginx -t && systemctl reload nginx

---

Phase 4: Portal Frontend Setup

You can deploy the Portal frontend on the same Admin server (using virtual directories under /portal) or split it onto a dedicated host.

Option A: Deployment on the Admin Server

# Run on the Admin server (192.168.10.40):
apt-get install -y -o Dpkg::Options::="--force-overwrite" softswitch-portal

The package registers static directories inside /var/www/softswitch/ and hooks into the Nginx configuration block automatically.

Option B: Dedicated Server (e.g. Portal on 192.168.10.51)

Add Nginx and repository settings on the target server, then install the package:

apt-get install -y -o Dpkg::Options::="--force-overwrite" nginx softswitch-portal

Create a proxy configuration at /etc/nginx/sites-available/softswitch-portal to forward API requests to the Admin API:

cat << 'EOF' > /etc/nginx/sites-available/softswitch-portal
server {
    listen 80;
    server_name portal.example.com;

    root /var/www/softswitch/portal;
    index index.html;

    location / {
        try_files $uri $uri/ /index.html;
    }

    # Reverse Proxy to the main Admin API Server
    location /api {
        proxy_pass http://192.168.10.40:3001;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    # Centralized Branding Assets & Logos Reverse Proxy
    location /uploads/ {
        proxy_pass http://192.168.10.40/uploads/;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_cache_valid 200 7d;
        expires 7d;
        add_header Cache-Control "public, no-transform";
    }
}
EOF

[!TIP] Alternative Shared Storage Mount: Instead of reverse proxying /uploads/ over HTTP, you can mount the GlusterFS ss-uploads volume directly on this server to /var/www/softswitch/uploads and use standard local Nginx aliases (alias /var/www/softswitch/uploads/;).

Link and activate the virtual host block:

ln -sf /etc/nginx/sites-available/softswitch-portal /etc/nginx/sites-enabled/
systemctl reload nginx

---

Phase 5: Switchboard Frontend Setup

You can deploy the Switchboard frontend on the same Admin server (using virtual directories under /switchboard) or split it onto a dedicated host.

Option A: Deployment on the Admin Server

# Run on the Admin server (192.168.10.40):
apt-get install -y -o Dpkg::Options::="--force-overwrite" softswitch-switchboard

The package registers static directories inside /var/www/softswitch/ and hooks into the Nginx configuration block automatically.

Option B: Dedicated Server (e.g. Switchboard on 192.168.10.52)

Add Nginx and repository settings on the target server, then install the package:

apt-get install -y -o Dpkg::Options::="--force-overwrite" nginx softswitch-switchboard

Create a proxy configuration at /etc/nginx/sites-available/softswitch-switchboard to forward API requests and static assets to the Admin API:

cat << 'EOF' > /etc/nginx/sites-available/softswitch-switchboard
server {
    listen 80;
    server_name switchboard.example.com;

    root /var/www/softswitch/switchboard;
    index index.html;

    location / {
        try_files $uri $uri/ /index.html;
    }

    # Reverse Proxy to the main Admin API Server
    location /api {
        proxy_pass http://192.168.10.40:3001;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    # Centralized Branding Assets & Logos Reverse Proxy
    location /uploads/ {
        proxy_pass http://192.168.10.40/uploads/;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_cache_valid 200 7d;
        expires 7d;
        add_header Cache-Control "public, no-transform";
    }
}
EOF

Link and activate the virtual host block:

ln -sf /etc/nginx/sites-available/softswitch-switchboard /etc/nginx/sites-enabled/
systemctl reload nginx

---

Phase 6: Telephony Server Setup (Telephony Node)

Log in to the Telephony Server (192.168.10.41).

6.1 Register Repositories and Install Base Dependencies

apt-get update && apt-get install -y curl gnupg2 wget make sudo
wget -qO- https://repo.softswitchone.com/apt/setup_repo | bash
apt-get update

6.2 Install and Configure Local HAProxy

To support dynamic database failover transparently, we deploy HAProxy locally on this node before installing any Ring2All softswitch packages.

HAProxy acts as a local proxy, querying Patroni’s REST API on port 8008 to check which node holds the active leadership lock.

Install HAProxy:

apt-get update
apt-get install -y haproxy
systemctl start haproxy
systemctl enable haproxy

Write the following configuration into /etc/haproxy/haproxy.cfg:

cat << 'EOF' > /etc/haproxy/haproxy.cfg
global
    log /dev/log local0
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

defaults
    log global
    mode tcp
    option tcplog
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

# Port 5000: Write transactions routed dynamically to Patroni Leader
frontend pg_write
    bind 127.0.0.1:5000
    default_backend pg_primary

backend pg_primary
    mode tcp
    option httpchk GET /primary
    http-check expect status 200
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server pg-node-01 192.168.10.34:5432 maxconn 100 maxqueue 10 check port 8008
    server pg-node-02 192.168.10.35:5432 maxconn 100 maxqueue 10 check port 8008
    server pg-node-03 192.168.10.36:5432 maxconn 100 maxqueue 10 check port 8008

# Port 5001: Read transactions load balanced across all Standby nodes
frontend pg_read
    bind 127.0.0.1:5001
    default_backend pg_replicas

backend pg_replicas
    mode tcp
    balance roundrobin
    option httpchk GET /replica
    http-check expect status 200
    default-server inter 3s fall 3 rise 2
    server pg-node-01 192.168.10.34:5432 check port 8008
    server pg-node-02 192.168.10.35:5432 check port 8008
    server pg-node-03 192.168.10.36:5432 check port 8008
EOF

Validate and restart the service, then verify that port 5000 is listening locally:

haproxy -c -f /etc/haproxy/haproxy.cfg
systemctl restart haproxy
ss -ltn | grep 5000

6.3 Fetch DB Credentials File

Copy the credentials from the DB Leader node:

mkdir -p /etc/softswitch
scp root@192.168.10.34:/etc/softswitch/db-credentials /etc/softswitch/db-credentials

6.4 (Optional High-Scale) Deploy Local PgBouncer Connection Pooler (Port 6432)

While local HAProxy provides seamless Layer 4 TCP routing and automatic leader failover, it operates on a 1:1 connection model: every connection opened by Telephony Server, Kamailio, and background APIs is passed directly to the PostgreSQL backend.

In PostgreSQL, each client connection spawns a dedicated operating system process consuming approximately 10MB of RAM, along with CPU context-switching overhead.

Under high call volumes (e.g., 500–5,000+ concurrent calls, high Call Per Second / CPS bursts, or multi-tenant PBX workloads across multiple telephony nodes), direct connections can quickly exhaust PostgreSQL’s max_connections limit and throttle database CPU performance.

🚀 What PgBouncer Does

PgBouncer is an ultra-lightweight Layer 7 connection pooler designed specifically for PostgreSQL. When configured in pool_mode = transaction:

• Multiplexes Connections: Thousands of active telephony client queries share a tiny, reusable pool of 20–30 persistent backend connections.
• Instant Connection Lifecycle: It attaches a real database connection only for the milliseconds required to execute a SQL query or transaction, releasing it immediately back to the pool.
• Sub-Millisecond Latency: It drops query overhead from ~15ms to < 0.5ms, eliminating TCP connection handshakes and preventing connection spikes during high CPS bursts or cluster failovers.

📊 When Should You Add PgBouncer?

• Standard Deployments (< 300 concurrent calls): Direct local HAProxy on port 5000 is fully sufficient.
• Carrier & High-Volume Deployments (> 500–1,000+ concurrent calls, Multi-Node Telephony Server / Kamailio clusters, or high-frequency CDR logging): Strongly recommended to prevent database connection exhaustion and keep memory usage flat.

🗺️ High-Scale Traffic Flow:

[ Telephony Core (ODBC :6432) ] ──┐
[ Kamailio (db_postgres)  ] ──┼─► [ Local PgBouncer :6432 ] ──► [ Local HAProxy :5000 ] ──► [ Patroni Leader :5432 ]
[ Node.js REST API        ] ──┘   (Transaction Pooling)         (Failover & Healthcheck)    (PostgreSQL Cluster)

⚙️ Step-by-Step Installation & Configuration:

1. Install PgBouncer on the Telephony / Application Node:

apt-get install -y pgbouncer

2. Configure /etc/pgbouncer/pgbouncer.ini to route database traffic locally to HAProxy on port 5000:

cat << 'EOF' > /etc/pgbouncer/pgbouncer.ini
[databases]
ss_telephony = host=127.0.0.1 port=5000 dbname=ss_telephony
ring2all     = host=127.0.0.1 port=5000 dbname=ring2all
kamailio     = host=127.0.0.1 port=5000 dbname=kamailio
sbc_admin    = host=127.0.0.1 port=5000 dbname=sbc_admin

[pgbouncer]
logfile = /var/log/postgresql/pgbouncer.log
pidfile = /var/run/postgresql/pgbouncer.pid
listen_addr = 127.0.0.1
listen_port = 6432
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt
admin_users = postgres, ss_db_user

# Transaction pooling for carrier-grade throughput
pool_mode = transaction
max_client_conn = 5000
default_pool_size = 25
reserve_pool_size = 5
max_prepared_statements = 100

# Critical for Node.js (Kysely/pg) and Telephony Server ODBC compatibility
ignore_startup_parameters = extra_float_digits, search_path, application_name
EOF

3. Generate /etc/pgbouncer/userlist.txt using the synced database credentials:

source /etc/softswitch/db-credentials
cat << EOF > /etc/pgbouncer/userlist.txt
"ss_db_user" "$DB_PASSWORD"
"postgres" "$DB_PASSWORD"
EOF
chmod 640 /etc/pgbouncer/userlist.txt
chown postgres:postgres /etc/pgbouncer/userlist.txt

4. Enable and start PgBouncer, then verify that port 6432 is listening:

systemctl enable --now pgbouncer
ss -ltn | grep 6432

5. When PgBouncer is active, adjust /etc/odbc.ini in Telephony Server to point to port 6432 instead of 5000:

[ring2all]
Driver   = PostgreSQL ANSI
Database = ss_telephony
Servername = 127.0.0.1
Port     = 6432
UserName = ss_db_user
Password = YOUR_PASSWORD

---

6.5 Mount Shared Storage Volumes

Configure host resolutions in /etc/hosts and mount the recordings and music volumes using backup-volfile-servers for redundancy:

# On Telephony Nodes (Telephony Server):
# Add storage hosts to /etc/hosts (update IPs if they differ in your lab)
cat << 'EOF' >> /etc/hosts
192.168.10.37 fs-node-01
192.168.10.38 fs-node-02
192.168.10.39 fs-node-03
EOF

apt-get install -y glusterfs-client

mkdir -p /var/lib/freeswitch/recordings
mkdir -p /usr/share/freeswitch/sounds/music

# Mount volumes
mount -t glusterfs -o backup-volfile-servers=fs-node-02:fs-node-03 fs-node-01:/ss-recordings /var/lib/freeswitch/recordings
mount -t glusterfs -o backup-volfile-servers=fs-node-02:fs-node-03 fs-node-01:/ss-music       /usr/share/freeswitch/sounds/music

# Configure persistent mounting in fstab
cat >> /etc/fstab << 'EOF'
fs-node-01:/ss-recordings /var/lib/freeswitch/recordings glusterfs defaults,_netdev,backup-volfile-servers=fs-node-02:fs-node-03 0 0
fs-node-01:/ss-music       /usr/share/freeswitch/sounds/music         glusterfs defaults,_netdev,backup-volfile-servers=fs-node-02:fs-node-03 0 0
EOF

6.6 Install Telephony Modules

Install the custom softswitch-telephony configurations. The post-installation script will detect the local database proxy on port 5000 (or PgBouncer on port 6432 if configured) and write matching /etc/odbc.ini profiles automatically:

apt-get install -y -o Dpkg::Options::="--force-overwrite" softswitch-telephony

6.7 Link Telephony Node to the Admin Monitoring Service (Optional CLI Method)

[!NOTE] Recommended Method (Web UI): In production multi-node environments, Telephony Nodes are added and managed directly from the Admin Web Portal under Telephony > Telephony Servers. Registering nodes in the Web UI automatically configures real-time monitoring across all active telephony cluster nodes.

For automated scripted deployments, you can optionally set the initial Telephony node IP (192.168.10.41) directly via SQL:

# On Admin Server (192.168.10.40):
source /etc/softswitch/db-credentials
PGPASSWORD="$DB_PASSWORD" psql -h 127.0.0.1 -p 5000 -U ss_db_user -d ss_telephony \
  -c "UPDATE esl_users SET listen_ip = '192.168.10.41' WHERE username = 'system_monitoring';"

# Restart monitoring service
systemctl restart softswitch-monitoring-api

---

🔑 Secure Credentials Distribution

To distribute generated keys securely across the cluster network, create and run this helper script on the DB Leader node:

distribute-credentials.sh

cat << 'EOF' > distribute-credentials.sh
#!/bin/bash
# Load database credentials
source /etc/softswitch/db-credentials

# Define client IP list
CLIENTS=(
    "192.168.10.40"  # Admin API Server
    "192.168.10.41"  # Telephony Server 1
    "192.168.10.42"  # Telephony Server 2
    "192.168.10.43"  # Telephony Server 3
    "192.168.10.44"  # Telephony Server 4
    "192.168.10.45"  # Telephony Server 5
)

for target in "${CLIENTS[@]}"; do
    echo "🔑 Secure copying credentials to: ${target}..."
    ssh root@${target} "mkdir -p /etc/softswitch"
    scp /etc/softswitch/db-credentials root@${target}:/etc/softswitch/db-credentials
done
EOF

chmod +x distribute-credentials.sh
./distribute-credentials.sh

---

🔍 Validation Checklist

Review this checklist to ensure all distributed platform modules are connected:

1. Database Clustering

• Patroni reports cluster health correctly (patronictl -c /etc/patroni/config.yml list).
• Read requests are distributed across all standby replicas via port 5001.
• Write requests are successfully routed to the Patroni leader node via port 5000.

2. Storage Clustering

• GlusterFS volumes report healthy and synchronized brick status (gluster volume status).
• Files written to /var/lib/freeswitch/recordings synchronize to all storage bricks.

3. Telephony Connectivity

• Telephony Server successfully connects to database schemas via ODBC.
• The fs_cli -x "sofia status" command reports the internal and external profiles as RUNNING.
• SIP devices can register and route calls using any telephony node IP.

---

📈 Capacity & Scaling Guidelines

Refer to this guide to plan your cluster deployment architecture:

Concurrent Calls	Telephony Nodes	Database Configuration	Storage Type
< 100	1	Single Local DB	Local Filesystem
100–500	2	Single Dedicated DB	Replicated GlusterFS (2 Nodes)
500–1,000	3	Replicated Primary + Replica	Replicated GlusterFS (3 Nodes)
1,000–5,000	4–8	Patroni Cluster (3+ Nodes)	Distributed-Replicated GlusterFS
> 5,000	8+	High-Availability Cluster	Dedicated Storage Array / SAN

---

🔍 Troubleshooting

1. Database connection is refused from clients

• Cause: The Patroni node has not finished bootstrapping, or nftables firewalls are blocking port 5432/8008.
• Fix: Check status using patronictl:

  patronictl -c /etc/patroni/config.yml list

  Run a TCP socket check from the client to the database node:

  nc -zv 192.168.10.34 5432

2. Telephony Server logs “CORE DATABASE INITIALIZATION FAILURE”

• Cause: The core system-level DSN connection is misaligned in /etc/freeswitch/autoload_configs/switch.conf.xml.
• Fix: Check core-db-dsn. If the parameters are pointing directly to the database IP or port 5432, change the endpoint to the local HAProxy write interface:

  <param name="core-db-dsn" value="pgsql://host=127.0.0.1 port=5000 dbname=freeswitch user=ss_db_user password=YOUR_PASSWORD" />

  Restart Telephony Server to reinitialize the SQL engine:

  systemctl reset-failed freeswitch && systemctl start freeswitch

3. Web portal returns “502 Bad Gateway” on API calls or dashboard widgets

• Cause: The API services are not running or failed to initialize their dependencies.
• Fix: Verify systemd status:

  systemctl status softswitch-api
  systemctl status softswitch-monitoring-api

  If dependencies are missing or compilation failed, run a manual npm installation:

  cd /var/www/softswitch
  npm install
  systemctl restart softswitch-api
  systemctl restart softswitch-monitoring-api

---

This concludes our 8-part Debian 13 Clustering and Distribution series. You now have a complete, production-ready, enterprise-grade softswitch platform deployed on Debian 13.