🏢 Part 8: Enterprise-Grade Distributed Deployment on Debian 13

Welcome to the eighth and final installment of our "Debian 13 Clustering & Distribution" series. In the previous articles, we packaged FreeSWITCH, Kamailio, and RTPEngine, and explored individual high-availability storage and database clustering configurations. In this final guide, we will unify these concepts into a production-ready, multi-server enterprise environment. We will walk through setting up a complete distributed topology on Debian 13 (Trixie). We will configure a replicated PostgreSQL 17 cluster, a high-availability GlusterFS storage network, and local HAProxy proxy configurations on application nodes for transparent database routing. We will also install our application APIs and frontend portals, deploy dedicated FreeSWITCH telephony nodes connecting via ODBC and mounting shared media files, detail credentials distribution, and establish troubleshooting and validation checks for cross-service connectivity.

🏗️ Architecture Overview

In a distributed deployment, each component is decoupled and assigned to dedicated servers to ensure high availability, redundancy, and maximum horizontal scalability.

Plaintext

┌─────────────────────────────────────────────────────────────────────────────────┐
│                           DISTRIBUTED ARCHITECTURE                               │
├─────────────────────────────────────────────────────────────────────────────────┤
│                                                                                  │
│   ┌─────────────────────────────────────────────────────────────────────────┐   │
│   │                        LOAD BALANCER (HAProxy/Nginx)                     │   │
│   │                         Public IP: lb.example.com                        │   │
│   └─────────────────────────────────────────────────────────────────────────┘   │
│                    │              │              │              │                │
│          ┌─────────┴──────┐ ┌────┴────┐ ┌──────┴──────┐ ┌─────┴─────┐          │
│          │                │ │         │ │             │ │           │          │
│          ▼                ▼ ▼         ▼ ▼             ▼ ▼           ▼          │
│   ┌────────────┐   ┌────────────┐   ┌────────────┐   ┌────────────┐            │
│   │   ADMIN    │   │   PORTAL   │   │ SWITCHBOARD│   │   API(s)   │            │
│   │  Server    │   │   Server   │   │   Server   │   │  Server(s) │            │
│   └────────────┘   └────────────┘   └────────────┘   └────────────┘            │
│          │                │              │                  │                   │
│          └────────────────┴──────────────┴──────────────────┘                   │
│                                     │                                            │
│                                     ▼                                            │
│   ┌─────────────────────────────────────────────────────────────────────────┐   │
│   │                     PRIVATE NETWORK (192.168.10.0/24)                    │   │
│   └─────────────────────────────────────────────────────────────────────────┘   │
│          │                                                      │                │
│          ▼                                                      ▼                │
│   ┌──────────────────────────────────────────────┐    ┌───────────────────────┐ │
│   │      PostgreSQL 17 CLUSTER (3 Nodes)          │    │   TELEPHONY CLUSTER   │ │
│   │                                               │    │                       │ │
│   │  ┌───────────────┐  ┌──────────┐  ┌────────┐ │    │  ┌────────┐ ┌────────┐ │ │
│   │  │    PRIMARY    │  │REPLICA 1 │  │REPLICA2│ │    │  │  FS-01 │ │  FS-02 │ │ │
│   │  │192.168.10.34  │  │192.168.  │  │192.168.│ │    │  │        │ │        │ │ │
│   │  │  Read/Write   │  │  10.35   │  │ 10.36  │ │    │  └────────┘ └────────┘ │ │
│   │  │               │  │ Read Only│  │ReadOnly│ │    │      + FS-N (N+1)      │ │
│   │  └───────────────┘  └──────────┘  └────────┘ │    └───────────────────────┘ │
│   │    PostgreSQL 17 · HAProxy :5000 (write)       │                             │
│   │                   · HAProxy :5001 (read LB)    │                             │
│   └──────────────────────────────────────────────┘                              │
│                                                                                  │
└─────────────────────────────────────────────────────────────────────────────────┘

🖥️ Server Roles & Requirements

To set up a production-ready lab, we allocate dedicated servers with the following configurations:

Role	Hostname	Example IP	Target Package	Minimum Spec
DB Primary	`db-primary`	`192.168.10.34`	`softswitch-db`	4 vCPU, 16GB RAM, SSD
DB Replica 1	`db-replica1`	`192.168.10.35`	PostgreSQL 17 (Manual)	4 vCPU, 16GB RAM, SSD
DB Replica 2	`db-replica2`	`192.168.10.36`	PostgreSQL 17 (Manual)	4 vCPU, 16GB RAM, SSD
GlusterFS 1	`fs01`	`192.168.10.37`	`glusterfs-server`	2 vCPU, 4GB RAM, Storage
GlusterFS 2	`fs02`	`192.168.10.38`	`glusterfs-server`	2 vCPU, 4GB RAM, Storage
GlusterFS 3	`fs03`	`192.168.10.39`	`glusterfs-server`	2 vCPU, 4GB RAM, Storage
Admin Server	`admin`	`192.168.10.40`	`softswitch-admin`, `softswitch-api`	2 vCPU, 4GB RAM, SSD
Portal Server	`portal`	`192.168.10.41`	`softswitch-portal`	2 vCPU, 2GB RAM, SSD
Telephony 1	`fs-01`	`192.168.10.30`	`softswitch-telephony`	4 vCPU, 8GB RAM, SSD

---

🛠️ Step-by-Step Deployment Guide

Phase 0: Install PostgreSQL 17 on Database Servers

Execute these tasks on all three database nodes (192.168.10.34, 192.168.10.35, and 192.168.10.36) to build the database engine base before configuring the cluster layers.

WARNING

Do not install the Ring2All package repositories on the database replica servers. They only require PostgreSQL 17. The repository configuration and softswitch-db schemas should be registered solely on the DB Primary node.

0.1 Install PostgreSQL 17 Base

PostgreSQL 17 is available natively in the Debian 13 (Trixie) main repository:

Bash

apt-get update
apt-get install -y postgresql-17 postgresql-client-17

0.2 Verify Cluster Directories

If pg_lsclusters prints an "Invalid data directory" error, append the directory path to the config file manually:

Bash

pg_lsclusters 2>&1 | grep -q "Invalid" && \
    echo "data_directory = '/var/lib/postgresql/17/main'" >> /etc/postgresql/17/main/postgresql.conf

# Start and enable the service
systemctl enable postgresql
systemctl start postgresql

Verify service execution:

Bash

# Check version (Expected: 17.x) and state (Expected: active)
psql --version
systemctl is-active postgresql

Phase 1: Database Cluster Setup (DB Primary)

1.1 Register Repository and Install Database Package

Bash

apt-get install -y sudo curl gpg

# Register signing key
curl -fsSL https://repo.softswitchone.com/apt/gpgkey/ring2all.gpg | gpg --dearmor -o /etc/apt/keyrings/ring2all.gpg

# Add repositories
echo "deb [signed-by=/etc/apt/keyrings/ring2all.gpg] https://repo.softswitchone.com/apt/base stable main" > /etc/apt/sources.list.d/ring2all.list
echo "deb [signed-by=/etc/apt/keyrings/ring2all.gpg] https://repo.softswitchone.com/apt/core stable main" >> /etc/apt/sources.list.d/ring2all.list
apt-get update

# Install softswitch database schemas
apt-get install -y softswitch-db

Verify the generated DB credentials. Note down the password:

Bash

cat /etc/softswitch/db-credentials

1.2 Configure PostgreSQL for Streaming Replication

Replace /etc/postgresql/17/main/postgresql.conf on the Primary server with the following settings:

Ini

data_directory = '/var/lib/postgresql/17/main'
listen_addresses = '*'
port = 5432

# Resource Configuration
shared_buffers = 4GB
effective_cache_size = 12GB
work_mem = 64MB
maintenance_work_mem = 1GB
max_connections = 500

# Streaming Replication Settings
wal_level = replica
max_wal_senders = 10
wal_keep_size = 2GB
hot_standby = on

1.3 Configure Client and Replication Authentication

Add the network access blocks to the end of /etc/postgresql/17/main/pg_hba.conf:

Plaintext

# Allow local Unix socket connections
local   all             postgres                                peer
local   all             all                                     md5

# Allow application servers in the private subnet
host    all             ss_db_user      192.168.10.0/24         scram-sha-256

# Allow replication nodes
host    replication     replicator      192.168.10.35/32        scram-sha-256
host    replication     replicator      192.168.10.36/32        scram-sha-256

1.4 Create Replication User

Bash

sudo -u postgres psql -c "CREATE USER replicator WITH REPLICATION ENCRYPTED PASSWORD 'STRONG_REPLICATION_PASSWORD';"

1.5 Restart PostgreSQL Primary

Bash

systemctl restart postgresql

# Verify that the primary is running in write mode (Expected: f)
sudo -u postgres psql -c "SELECT pg_is_in_recovery();"

Phase 2: Configure Database Replica Nodes

Run these steps on the DB Replicas (192.168.10.35 and 192.168.10.36).

IMPORTANT

The performance configuration variables (shared_buffers, max_connections, etc.) in postgresql.conf on the Replicas must be equal to or higher than those on the Primary. Otherwise, replication will fail to initialize.

2.1 Write Configuration

Deploy the replication configuration file:

Bash

tee /etc/postgresql/17/main/postgresql.conf << 'EOF'
data_directory = '/var/lib/postgresql/17/main'
listen_addresses = '*'
port = 5432

shared_buffers = 4GB
effective_cache_size = 12GB
work_mem = 64MB
maintenance_work_mem = 1GB
max_connections = 500

wal_level = replica
max_wal_senders = 10
wal_keep_size = 2GB
hot_standby = on
EOF

2.2 Rebuild Data Directory

Stop the local database service and wipe the default database file system:

Bash

systemctl stop postgresql
rm -rf /var/lib/postgresql/17/main/*

2.3 Bootstrap Directory from Primary

Sync the database files from the Primary database node using pg_basebackup:

Bash

pg_basebackup \
    -h 192.168.10.34 \
    -U replicator \
    -P -R \
    -D /var/lib/postgresql/17/main \
    -Fp -Xs

Enter the replication user password when prompted.

2.4 Set Directory Owner Permissions

Because pg_basebackup runs as root, update permissions to the postgres user:

Bash

chown -R postgres:postgres /var/lib/postgresql/17/main
systemctl start postgresql

2.5 Confirm Replication Link

Check if the server successfully started in recovery/replica mode:

Bash

# Expected: t (true)
sudo -u postgres psql -c "SELECT pg_is_in_recovery();"

Check the active replica streams on the Primary database node (192.168.10.34):

Bash

sudo -u postgres psql -c "SELECT client_addr, state, sent_lsn, replay_lsn FROM pg_stat_replication;"

Expected: Two stream rows corresponding to replication nodes (192.168.10.35 and 192.168.10.36) in the streaming state.

Phase 3: GlusterFS Shared Storage Cluster Setup

GlusterFS provides a distributed file system so all client servers can access recordings, uploads, and media folders concurrently.

3.1 Install GlusterFS Server on Storage Nodes

Run these commands on the three storage nodes (192.168.10.37, 192.168.10.38, and 192.168.10.39):

Bash

apt-get update
apt-get install -y glusterfs-server
systemctl enable --now glusterd

3.2 Update Network Hosts Resolutions

Add name resolutions on all three storage nodes:

Bash

cat >> /etc/hosts << 'EOF'
192.168.10.37 fs01
192.168.10.38 fs02
192.168.10.39 fs03
EOF

3.3 Peer Cluster Probe (Only from fs01)

Establish the trusted storage pool by probing peers from fs01 (192.168.10.37):

Bash

gluster peer probe fs02
gluster peer probe fs03

# Verify cluster peers status
gluster peer status

3.4 Create Data Brick Directories

Create brick paths on all three storage nodes:

Bash

mkdir -p /gluster/data/recordings
mkdir -p /gluster/data/uploads
mkdir -p /gluster/data/music

3.5 Create Replicated Volumes (Only from fs01)

Configure three-way replication volumes to ensure data remains online if a storage node fails:

Bash

# Recordings volume
gluster volume create ss-recordings replica 3 \
    fs01:/gluster/data/recordings \
    fs02:/gluster/data/recordings \
    fs03:/gluster/data/recordings \
    force

# Uploads volume
gluster volume create ss-uploads replica 3 \
    fs01:/gluster/data/uploads \
    fs02:/gluster/data/uploads \
    fs03:/gluster/data/uploads \
    force

# Music on hold volume
gluster volume create ss-music replica 3 \
    fs01:/gluster/data/music \
    fs02:/gluster/data/music \
    fs03:/gluster/data/music \
    force

# Start the volumes
gluster volume start ss-recordings
gluster volume start ss-uploads
gluster volume start ss-music

Phase 4: Local HAProxy Configurations on Client Nodes

To handle database primary-replica routing transparently, we deploy HAProxy locally on every application and telephony node (e.g., Admin API, Telephony servers).

All local applications configure their database connection string to point to 127.0.0.1. Local HAProxy routes write requests on port 5000 to the Primary database node, and load balances read requests on port 5001 across the standby replicas.

Bash

apt-get install -y haproxy
systemctl enable haproxy

Write the configuration into /etc/haproxy/haproxy.cfg:

Haproxy

global
    log /dev/log local0
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

defaults
    log global
    mode tcp
    option tcplog
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

# Port 5000: Write transactions routed to Primary database node
frontend pg_write
    bind 127.0.0.1:5000
    default_backend pg_primary

backend pg_primary
    server db-primary 192.168.10.34:5432 check

# Port 5001: Read transactions load balanced across all nodes
frontend pg_read
    bind 127.0.0.1:5001
    default_backend pg_replicas

backend pg_replicas
    balance roundrobin
    server db-primary  192.168.10.34:5432 check
    server db-replica1 192.168.10.35:5432 check
    server db-replica2 192.168.10.36:5432 check

Validate and restart the HAProxy service:

Bash

haproxy -c -f /etc/haproxy/haproxy.cfg
systemctl restart haproxy

Phase 5: Admin Server Setup

5.1 System Pre-requisites and Repositories

Configure repositories as shown in Phase 1. Install prerequisites:

Bash

apt-get install -y build-essential python3 nginx
curl -fsSL https://deb.nodesource.com/setup_22.x | bash -
apt-get install -y nodejs

5.2 Configure HAProxy

Configure a local HAProxy proxy on this node following Phase 4. Verify that port 5000 is listening locally:

Bash

ss -ltn | grep 5000

5.3 Copy Database Credentials

Create the config path and copy the /etc/softswitch/db-credentials file from the DB Primary node:

Bash

mkdir -p /etc/softswitch
# Run on Admin node:
scp root@192.168.10.34:/etc/softswitch/db-credentials /etc/softswitch/db-credentials

5.4 Install API Packages

Install the REST API and telemetry modules. The post-installation script detects local HAProxy port 5000 and automatically links the configurations:

Bash

apt-get install -y softswitch-api softswitch-monitoring-api

Install packages dependencies for the monitoring service:

Bash

cd /var/www/softswitch/apps/monitoring-api
npm install
systemctl restart softswitch-monitoring-api

5.5 Mount Shared Storage Volumes

Install the client tools and mount the shared storage uploads directory:

Bash

apt-get install -y glusterfs-client
mkdir -p /var/www/softswitch/uploads

# Mount volume
mount -t glusterfs 192.168.10.37:/ss-uploads /var/www/softswitch/uploads

# Add fstab entry with backup mount configurations
echo "192.168.10.37:/ss-uploads /var/www/softswitch/uploads glusterfs defaults,_netdev,backupvolfile-server=192.168.10.38 0 0" >> /etc/fstab

5.6 Install Frontend Admin Dashboard

Bash

apt-get install -y softswitch-admin
nginx -t && systemctl reload nginx

Phase 6: Portal & Switchboard Frontends Setup

You can deploy the Portal and Switchboard frontends on the same Admin server (using virtual directories under /portal and /switchboard) or split them onto separate hosts.

Option A: Deployment on the Admin Server

Bash

# Run on the Admin server (192.168.10.40):
apt-get install -y softswitch-portal softswitch-switchboard

The packages register static directories inside /var/www/softswitch/ and hook into the Nginx configuration block automatically.

Option B: Dedicated Servers (e.g. Portal on 192.168.10.41)

Add Nginx and repository settings on the target server, then install the package:

Bash

apt-get install -y nginx softswitch-portal

Create a proxy configuration at /etc/nginx/sites-available/softswitch-portal to forward API requests to the Admin API:

Nginx

server {
    listen 80;
    server_name portal.example.com;

    root /var/www/softswitch/portal;
    index index.html;

    location / {
        try_files $uri $uri/ /index.html;
    }

    # Reverse Proxy to the main Admin API Server
    location /api {
        proxy_pass http://192.168.10.40:3001;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Link and activate the virtual host block:

Bash

ln -sf /etc/nginx/sites-available/softswitch-portal /etc/nginx/sites-enabled/
systemctl reload nginx

Phase 7: Telephony Server Setup (FreeSWITCH Client Node)

7.1 Register Repositories and Install Base Dependencies

Bash

apt-get update && apt-get install -y curl gnupg2 wget
wget -qO- https://repo.softswitchone.com/apt/setup_repo | bash
apt-get update

7.2 Install local HAProxy Proxy

Follow Phase 4 to install and configure local HAProxy routing on the telephony node.

7.3 Fetch DB Credentials File

Copy the credentials from the DB Primary node:

Bash

mkdir -p /etc/softswitch
scp root@192.168.10.34:/etc/softswitch/db-credentials /etc/softswitch/db-credentials

7.4 Mount Shared Storage Volumes

Mount the recordings and music volumes:

Bash

apt-get install -y glusterfs-client

mkdir -p /var/lib/freeswitch/recordings
mkdir -p /usr/share/freeswitch/sounds/music/default

# Mount volumes
mount -t glusterfs 192.168.10.37:/ss-recordings /var/lib/freeswitch/recordings
mount -t glusterfs 192.168.10.37:/ss-music       /usr/share/freeswitch/sounds/music/default

# Configure persistent mounting in fstab
cat >> /etc/fstab << 'EOF'
192.168.10.37:/ss-recordings /var/lib/freeswitch/recordings glusterfs defaults,_netdev,backupvolfile-server=192.168.10.38 0 0
192.168.10.37:/ss-music       /usr/share/freeswitch/sounds/music/default glusterfs defaults,_netdev,backupvolfile-server=192.168.10.38 0 0
EOF

7.5 Install Telephony Modules

Install the custom softswitch-telephony configurations. The post-installation script will detect the local database proxy on port 5000 and write matching /etc/odbc.ini profiles automatically:

Bash

apt-get install -y softswitch-telephony

7.6 Telephony Domain Aliasing

If the internal SIP profile fails to start or report domain errors, configure a domain IP alias using SQL over the database proxy:

Bash

DB_PASS=$(grep DB_PASSWORD /etc/softswitch/db-credentials | cut -d= -f2)

# Retrieve the Primary Domain ID
DOMAIN_ID=$(PGPASSWORD=$DB_PASS psql -h 127.0.0.1 -p 5000 -U ss_db_user -d ss_telephony \
  -tAc "SELECT id FROM domains WHERE tenant_id=1 AND is_primary=TRUE LIMIT 1")

SERVER_IP=$(hostname -I | awk '{print $1}')

# Register this telephony server IP as an alias of the domain
PGPASSWORD=$DB_PASS psql -h 127.0.0.1 -p 5000 -U ss_db_user -d ss_telephony << EOF
INSERT INTO public.domain_aliases (domain_id, alias, alias_type, description, enabled, created_at)
VALUES ($DOMAIN_ID, '$SERVER_IP', 'ip', 'Telephony Server IP', TRUE, NOW())
ON CONFLICT (alias) DO NOTHING;
EOF

# Restart the FreeSWITCH engine and start profiles
systemctl restart freeswitch
fs_cli -x "sofia profile internal start"

7.7 Map Telephony node to the Admin Monitoring API

Configure the telemetry API on the Admin server (192.168.10.40) to hook into this FreeSWITCH node ESL socket:

Bash

# On Admin Server (192.168.10.40):
nano /etc/softswitch/monitoring.env

Update parameters to match the telephony server IP and default ClueCon secret:

Env

ESL_HOST=192.168.10.30
ESL_PORT=8021
ESL_PASSWORD=ClueCon
ESL_ENABLED=true

Restart the service to initialize connections:

Bash

systemctl restart softswitch-monitoring-api

🔑 Secure Credentials Distribution

To distribute generated keys securely across the cluster network, run this wrapper helper script from the DB Primary node:

Bash

#!/bin/bash
# distribute-credentials.sh

# Load database credentials
source /etc/softswitch/db-credentials

# Define client IP list
CLIENTS=(
    "192.168.10.40"  # Admin API Server
    "192.168.10.30"  # Telephony Server 1
)

for target in "${CLIENTS[@]}"; do
    echo "🔑 Secure copying credentials to: ${target}..."
    ssh root@${target} "mkdir -p /etc/softswitch"
    scp /etc/softswitch/db-credentials root@${target}:/etc/softswitch/db-credentials
done

🔍 Validation Checklist

Review this checklist to ensure all distributed platform modules are connected:

1. Database Clustering

[ ] Database replication shows a delay close to zero.
[ ] Read requests are distributed across all three replication nodes via port 5001.
[ ] Write requests are successfully routed to the Primary database node via port 5000.

2. Storage Clustering

[ ] GlusterFS volumes report healthy and synchronized brick status (gluster volume status).
[ ] Files written to /var/lib/freeswitch/recordings synchronize to all storage bricks.

3. Telephony Connectivity

[ ] FreeSWITCH successfully connects to database schemas via ODBC.
[ ] The fs_cli -x "sofia status" command reports the internal and external profiles as RUNNING.
[ ] SIP devices can register and route calls using any telephony node IP.

📈 Capacity & Scaling Guidelines

Refer to this guide to plan your cluster deployment architecture:

Concurrent Calls	Telephony Nodes	Database Configuration	Storage Type
< 100	1	Single Local DB	Local Filesystem
100–500	2	Single Dedicated DB	Replicated GlusterFS (2 Nodes)
500–1,000	3	Replicated Primary + Replica	Replicated GlusterFS (3 Nodes)
1,000–5,000	4–8	Patroni Cluster (3+ Nodes)	Distributed-Replicated GlusterFS
> 5,000	8+	High-Availability Cluster	Dedicated Storage Array / SAN

---

🔍 Troubleshooting

1. Database connection is refused from clients

Cause: The /etc/postgresql/17/main/pg_hba.conf file on the target database nodes is missing client IP permissions, or local firewalls are blocking port 5432.
Fix: Verify client subnet matching:

Bash

  host all ss_db_user 192.168.10.0/24 scram-sha-256

Run a TCP socket check from the client to the database node:

Bash

  nc -zv 192.168.10.34 5432

2. FreeSWITCH logs "CORE DATABASE INITIALIZATION FAILURE"

Cause: The core system-level DSN connection is misaligned in /etc/freeswitch/autoload_configs/switch.conf.xml.
Fix: Check core-db-dsn. If the parameters are pointing directly to the database IP or port 5432, change the endpoint to the local HAProxy write interface:

Xml

  <param name="core-db-dsn" value="pgsql://host=127.0.0.1 port=5000 dbname=freeswitch user=ss_db_user password=YOUR_PASSWORD" />

Restart FreeSWITCH to reinitialize the SQL engine:

Bash

  systemctl reset-failed freeswitch && systemctl start freeswitch

3. Web portal returns "502 Bad Gateway" on dashboard widgets

Cause: The softswitch-monitoring-api package was not installed or lacks required npm library dependencies.
Fix: Verify systemd status:

Bash

  systemctl status softswitch-monitoring-api

If dependencies are missing, run a manual npm build step:

Bash

  cd /var/www/softswitch/apps/monitoring-api
  npm install
  systemctl restart softswitch-monitoring-api

This concludes our 8-part Debian 13 Clustering and Distribution series. You now have a complete, production-ready, enterprise-grade softswitch platform deployed on Debian 13.