---
title: "🏢 Part 8: Enterprise-Grade Distributed Deployment on Debian 13"
description: "Documentation for 8. Distributed Setup"
---

*Welcome to the eighth and final installment of our "Debian 13 Clustering & Distribution" series. In the previous articles, we packaged Telephony Server, Kamailio, and RTPEngine, and explored individual high-availability storage and database clustering configurations. In this final guide, we will unify these concepts into a production-ready, multi-server enterprise environment. We will walk through setting up a complete distributed topology on Debian 13 (Trixie). Instead of repeating base setups, we will integrate the self-healing PostgreSQL 17 cluster managed by Patroni & Etcd configured in Part 4, and the high-availability GlusterFS storage network deployed in Part 5. We will configure local HAProxy proxies on application nodes to dynamically route database requests based on Patroni's active roles. We will also install our application APIs and frontend portals, deploy dedicated Telephony Server telephony nodes connecting via ODBC and mounting shared media files, detail credentials distribution, and establish troubleshooting and validation checks for cross-service connectivity.*

---

## 🏗️ Architecture Overview

In a distributed deployment, each component is decoupled and assigned to dedicated servers to ensure high availability, redundancy, and maximum horizontal scalability. 

```
┌─────────────────────────────────────────────────────────────────────────────────┐
│                           DISTRIBUTED ARCHITECTURE                               │
├─────────────────────────────────────────────────────────────────────────────────┤
│                                                                                  │
│   ┌─────────────────────────────────────────────────────────────────────────┐   │
│   │                        LOAD BALANCER (HAProxy/Nginx)                     │   │
│   │                         Public IP: lb.example.com                        │   │
│   └─────────────────────────────────────────────────────────────────────────┘   │
│                    │              │              │              │                │
│          ┌─────────┴──────┐ ┌────┴────┐ ┌──────┴──────┐ ┌─────┴─────┐          │
│          │                │ │         │ │             │ │           │          │
│          ▼                ▼ ▼         ▼ ▼             ▼ ▼           ▼          │
│   ┌────────────┐   ┌────────────┐   ┌────────────┐   ┌────────────┐            │
│   │   ADMIN    │   │   PORTAL   │   │ SWITCHBOARD│   │   API(s)   │            │
│   │  Server    │   │   Server   │   │   Server   │   │  Server(s) │            │
│   └────────────┘   └────────────┘   └────────────┘   └────────────┘            │
│          │                │              │                  │                   │
│          └────────────────┴──────────────┴──────────────────┘                   │
│                                     │                                            │
│                                     ▼                                            │
│   ┌─────────────────────────────────────────────────────────────────────────┐   │
│   │                     PRIVATE NETWORK (192.168.10.0/24)                    │   │
│   └─────────────────────────────────────────────────────────────────────────┘   │
│          │                                                      │                │
│          ▼                                                      ▼                │
│   ┌──────────────────────────────────────────────┐    ┌───────────────────────┐ │
│   │      PostgreSQL 17 CLUSTER (3 Nodes)          │    │   TELEPHONY CLUSTER   │ │
│   │        Managed by Patroni & Etcd             │    │                       │ │
│   │  ┌───────────────┐  ┌──────────┐  ┌────────┐ │    │  ┌────────┐ ┌────────┐ │ │
│   │  │  pg-node-01   │  │pg-node-02│  │pg-node-│ │    │  │  FS-01 │ │  FS-02 │ │ │
│   │  │192.168.10.34  │  │192.168.  │  │   03   │ │    │  │        │ │        │ │ │
│   │  │  Active Leader│  │  10.35   │  │192.168.│ │    │  └────────┘ └────────┘ │ │
│   │  │  (Read/Write)  │  │ Replica  │  │ 10.36  │ │    │      + FS-N (N+1)      │ │
│   │  └───────────────┘  └──────────┘  └────────┘ │    └───────────────────────┘ │
│   │    PostgreSQL 17 · HAProxy :5000 (write API)   │                             │
│   │                   · HAProxy :5001 (read LB)    │                             │
│   └──────────────────────────────────────────────┘                              │
│                                                                                  │
└─────────────────────────────────────────────────────────────────────────────────┘
```

---

## 🖥️ Server Roles & Requirements

To set up a production-ready lab, we allocate dedicated servers with the following configurations:

| Role | Hostname | Example IP | Target Package | Minimum Spec |
|------|----------|------------|----------------|--------------|
| **DB Node 1** | `pg-node-01` | `192.168.10.34` | PostgreSQL 17 + Patroni + Etcd | 4 vCPU, 16GB RAM, SSD |
| **DB Node 2** | `pg-node-02` | `192.168.10.35` | PostgreSQL 17 + Patroni + Etcd | 4 vCPU, 16GB RAM, SSD |
| **DB Node 3** | `pg-node-03` | `192.168.10.36` | PostgreSQL 17 + Patroni + Etcd | 4 vCPU, 16GB RAM, SSD |
| **File Server 1** | `fs-node-01` | `192.168.10.37` | `glusterfs-server` | 2 vCPU, 4GB RAM, Storage |
| **File Server 2** | `fs-node-02` | `192.168.10.38` | `glusterfs-server` | 2 vCPU, 4GB RAM, Storage |
| **File Server 3** | `fs-node-03` | `192.168.10.39` | `glusterfs-server` | 2 vCPU, 4GB RAM, Storage |
| **Admin Server** | `admin` | `192.168.10.40` | `softswitch-admin`, `softswitch-api` | 2 vCPU, 4GB RAM, SSD |
| **Portal Server** (Option B) | `portal` | `192.168.10.51` | `softswitch-portal` | 2 vCPU, 2GB RAM, SSD |
| **Switchboard Server** (Option B) | `switchboard` | `192.168.10.52` | `softswitch-switchboard` | 2 vCPU, 2GB RAM, SSD |
| **Telephony Nodes 1-5** | `fs-01` to `fs-05` | `192.168.10.41 - 192.168.10.45` | `softswitch-telephony` | 4 vCPU, 8GB RAM, SSD |

---

## 🛠️ Step-by-Step Deployment Guide

### Phase 1: High-Availability Database Cluster Setup

Instead of manual replication, we deploy a High-Availability PostgreSQL 17 cluster managed by Patroni and Etcd.

1. Configure hostnames (`pg-node-01`, `pg-node-02`, and `pg-node-03`) and name resolutions in `/etc/hosts` across all nodes.
2. Follow the detailed steps in [Part 4: PostgreSQL High-Availability Clustering with Patroni and Etcd](04-postgresql-ha-cluster-debian-13.md) to install etcd, clean PostgreSQL data paths, configure `/etc/default/etcd` and `/etc/patroni/config.yml` on each server, start the services, and install the `softswitch-db` package to initialize database schemas and generate system credentials.
3. Verify that the Patroni cluster status is healthy on the primary node:
   ```bash
   patronictl -c /etc/patroni/config.yml list
   ```

---

### Phase 2: GlusterFS Shared Storage Cluster Setup

We deploy a split-brain safe active-active storage pool to store telephony assets, call recordings, music, and portal uploads.

1. Prepare host resolutions on `/etc/hosts` mapping `fs-node-01` (`192.168.10.37`), `fs-node-02` (`192.168.10.38`), and `fs-node-03` (`192.168.10.39`).
2. Follow [Part 5: Distributed Storage with GlusterFS on Debian 13](05-glusterfs-file-server-cluster-debian-13.md) to peer-probe nodes, configure brick directories, and create three-way replicated volumes.
3. Verify that the storage pool and volumes are active:
   ```bash
   gluster volume info
   ```
   Ensure the following volumes are created and running: `ss-recordings`, `ss-uploads`, and `ss-music`.

---

### Phase 3: Admin Server Setup

Log in to the Admin Server (`192.168.10.40`).

#### 3.1 Install and Configure Local HAProxy
To support dynamic database failover transparently, we deploy HAProxy locally on this node before installing any Ring2All softswitch packages.

> [!NOTE]
> **Installer Auto-Detection**:
> The Ring2All installer scripts (`softswitch-api`, `softswitch-telephony`, etc.) are designed to automatically detect if a local HAProxy instance is installed. If detected, the scripts assume a distributed, high-availability topology and automatically configure database profiles, system DSNs, and `/etc/odbc.ini` to route traffic through the local proxy on loopback ports `5000` (writes) and `5001` (reads). This eliminates the need for manual connection profiling.

HAProxy acts as a local proxy, querying Patroni's REST API on port `8008` to check which node holds the active leadership lock.

Install HAProxy:
```bash
apt-get update
apt-get install -y haproxy make curl gnupg2 wget sudo
systemctl start haproxy
systemctl enable haproxy
```

Write the following configuration into `/etc/haproxy/haproxy.cfg`:
```bash
cat << 'EOF' > /etc/haproxy/haproxy.cfg
global
    log /dev/log local0
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

defaults
    log global
    mode tcp
    option tcplog
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

# Port 5000: Write transactions routed dynamically to Patroni Leader
frontend pg_write
    bind 127.0.0.1:5000
    default_backend pg_primary

backend pg_primary
    mode tcp
    option httpchk GET /primary
    http-check expect status 200
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server pg-node-01 192.168.10.34:5432 maxconn 100 maxqueue 10 check port 8008
    server pg-node-02 192.168.10.35:5432 maxconn 100 maxqueue 10 check port 8008
    server pg-node-03 192.168.10.36:5432 maxconn 100 maxqueue 10 check port 8008

# Port 5001: Read transactions load balanced across all Standby nodes
frontend pg_read
    bind 127.0.0.1:5001
    default_backend pg_replicas

backend pg_replicas
    mode tcp
    balance roundrobin
    option httpchk GET /replica
    http-check expect status 200
    default-server inter 3s fall 3 rise 2
    server pg-node-01 192.168.10.34:5432 check port 8008
    server pg-node-02 192.168.10.35:5432 check port 8008
    server pg-node-03 192.168.10.36:5432 check port 8008
EOF
```

Validate and restart the service, then verify that port 5000 is listening locally:
```bash
haproxy -c -f /etc/haproxy/haproxy.cfg
systemctl restart haproxy
ss -ltn | grep 5000
```

#### 3.2 Copy Database Credentials
Create the config path and copy the `/etc/softswitch/db-credentials` file from the DB Leader node:
```bash
mkdir -p /etc/softswitch
# Run on Admin node:
scp root@192.168.10.34:/etc/softswitch/db-credentials /etc/softswitch/db-credentials
```

#### 3.3 Register Repositories
Configure repository sources and signatures for Ring2All and NodeSource (Node.js 22) using the unified setup script:
```bash
# Clean up any conflicting NodeSource repositories from previous attempts
rm -f /etc/apt/sources.list.d/nodesource*
rm -f /usr/share/keyrings/nodesource*
rm -f /etc/apt/keyrings/nodesource*

# Download and run the unified repository setup script
# (This automatically configures both Ring2All and NodeSource repositories)
curl -fsSL https://repo.softswitchone.com/apt/setup_repo | bash
```

#### 3.4 Install Node.js and API Packages
Install Node.js (which includes npm), build tools, postgresql-client, and the REST API and telemetry modules. The post-installation script detects local HAProxy port 5000 and automatically links and starts the services:
```bash
# Install Node.js, build dependencies, and postgresql-client (do NOT install 'npm' separately)
apt-get update
apt-get install -y nodejs build-essential python3 postgresql-client

# Install Ring2All API packages
apt-get install -y -o Dpkg::Options::="--force-overwrite" softswitch-api softswitch-monitoring-api
```

#### 3.5 Mount Shared Storage Volumes
Configure host resolutions in `/etc/hosts` and mount the shared storage uploads directory using `backup-volfile-servers` to protect against node outages:
```bash
# On the Admin Server:
# Add storage hosts to /etc/hosts (update IPs if they differ in your lab)
cat << 'EOF' >> /etc/hosts
192.168.10.37 fs-node-01
192.168.10.38 fs-node-02
192.168.10.39 fs-node-03
EOF

apt-get install -y glusterfs-client
mkdir -p /var/www/softswitch/uploads

# Mount volume
mount -t glusterfs -o backup-volfile-servers=fs-node-02:fs-node-03 fs-node-01:/ss-uploads /var/www/softswitch/uploads

# Add fstab entry with backup mount configurations
cat << 'EOF' >> /etc/fstab
fs-node-01:/ss-uploads /var/www/softswitch/uploads glusterfs defaults,_netdev,backup-volfile-servers=fs-node-02:fs-node-03 0 0
EOF
```

#### 3.6 Install Frontend Admin Dashboard
```bash
apt-get install -y -o Dpkg::Options::="--force-overwrite" softswitch-admin
nginx -t && systemctl reload nginx
```

---

### Phase 4: Portal Frontend Setup

You can deploy the Portal frontend on the same Admin server (using virtual directories under `/portal`) or split it onto a dedicated host.

#### Option A: Deployment on the Admin Server
```bash
# Run on the Admin server (192.168.10.40):
apt-get install -y -o Dpkg::Options::="--force-overwrite" softswitch-portal
```
The package registers static directories inside `/var/www/softswitch/` and hooks into the Nginx configuration block automatically.

#### Option B: Dedicated Server (e.g. Portal on 192.168.10.51)
Add Nginx and repository settings on the target server, then install the package:
```bash
apt-get install -y -o Dpkg::Options::="--force-overwrite" nginx softswitch-portal
```
Create a proxy configuration at `/etc/nginx/sites-available/softswitch-portal` to forward API requests to the Admin API:
```bash
cat << 'EOF' > /etc/nginx/sites-available/softswitch-portal
server {
    listen 80;
    server_name portal.example.com;

    root /var/www/softswitch/portal;
    index index.html;

    location / {
        try_files $uri $uri/ /index.html;
    }

    # Reverse Proxy to the main Admin API Server
    location /api {
        proxy_pass http://192.168.10.40:3001;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    # Centralized Branding Assets & Logos Reverse Proxy
    location /uploads/ {
        proxy_pass http://192.168.10.40/uploads/;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_cache_valid 200 7d;
        expires 7d;
        add_header Cache-Control "public, no-transform";
    }
}
EOF
```
> [!TIP]
> **Alternative Shared Storage Mount**:
> Instead of reverse proxying `/uploads/` over HTTP, you can mount the GlusterFS `ss-uploads` volume directly on this server to `/var/www/softswitch/uploads` and use standard local Nginx aliases (`alias /var/www/softswitch/uploads/;`).

Link and activate the virtual host block:
```bash
ln -sf /etc/nginx/sites-available/softswitch-portal /etc/nginx/sites-enabled/
systemctl reload nginx
```

---

### Phase 5: Switchboard Frontend Setup

You can deploy the Switchboard frontend on the same Admin server (using virtual directories under `/switchboard`) or split it onto a dedicated host.

#### Option A: Deployment on the Admin Server
```bash
# Run on the Admin server (192.168.10.40):
apt-get install -y -o Dpkg::Options::="--force-overwrite" softswitch-switchboard
```
The package registers static directories inside `/var/www/softswitch/` and hooks into the Nginx configuration block automatically.

#### Option B: Dedicated Server (e.g. Switchboard on 192.168.10.52)
Add Nginx and repository settings on the target server, then install the package:
```bash
apt-get install -y -o Dpkg::Options::="--force-overwrite" nginx softswitch-switchboard
```
Create a proxy configuration at `/etc/nginx/sites-available/softswitch-switchboard` to forward API requests and static assets to the Admin API:
```bash
cat << 'EOF' > /etc/nginx/sites-available/softswitch-switchboard
server {
    listen 80;
    server_name switchboard.example.com;

    root /var/www/softswitch/switchboard;
    index index.html;

    location / {
        try_files $uri $uri/ /index.html;
    }

    # Reverse Proxy to the main Admin API Server
    location /api {
        proxy_pass http://192.168.10.40:3001;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    # Centralized Branding Assets & Logos Reverse Proxy
    location /uploads/ {
        proxy_pass http://192.168.10.40/uploads/;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_cache_valid 200 7d;
        expires 7d;
        add_header Cache-Control "public, no-transform";
    }
}
EOF
```
Link and activate the virtual host block:
```bash
ln -sf /etc/nginx/sites-available/softswitch-switchboard /etc/nginx/sites-enabled/
systemctl reload nginx
```

---

### Phase 6: Telephony Server Setup (Telephony Node)

Log in to the Telephony Server (`192.168.10.41`).

#### 6.1 Register Repositories and Install Base Dependencies
```bash
apt-get update && apt-get install -y curl gnupg2 wget make sudo
wget -qO- https://repo.softswitchone.com/apt/setup_repo | bash
apt-get update
```

#### 6.2 Install and Configure Local HAProxy
To support dynamic database failover transparently, we deploy HAProxy locally on this node before installing any Ring2All softswitch packages.

HAProxy acts as a local proxy, querying Patroni's REST API on port `8008` to check which node holds the active leadership lock.

Install HAProxy:
```bash
apt-get update
apt-get install -y haproxy
systemctl start haproxy
systemctl enable haproxy
```

Write the following configuration into `/etc/haproxy/haproxy.cfg`:
```bash
cat << 'EOF' > /etc/haproxy/haproxy.cfg
global
    log /dev/log local0
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

defaults
    log global
    mode tcp
    option tcplog
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

# Port 5000: Write transactions routed dynamically to Patroni Leader
frontend pg_write
    bind 127.0.0.1:5000
    default_backend pg_primary

backend pg_primary
    mode tcp
    option httpchk GET /primary
    http-check expect status 200
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server pg-node-01 192.168.10.34:5432 maxconn 100 maxqueue 10 check port 8008
    server pg-node-02 192.168.10.35:5432 maxconn 100 maxqueue 10 check port 8008
    server pg-node-03 192.168.10.36:5432 maxconn 100 maxqueue 10 check port 8008

# Port 5001: Read transactions load balanced across all Standby nodes
frontend pg_read
    bind 127.0.0.1:5001
    default_backend pg_replicas

backend pg_replicas
    mode tcp
    balance roundrobin
    option httpchk GET /replica
    http-check expect status 200
    default-server inter 3s fall 3 rise 2
    server pg-node-01 192.168.10.34:5432 check port 8008
    server pg-node-02 192.168.10.35:5432 check port 8008
    server pg-node-03 192.168.10.36:5432 check port 8008
EOF
```

Validate and restart the service, then verify that port 5000 is listening locally:
```bash
haproxy -c -f /etc/haproxy/haproxy.cfg
systemctl restart haproxy
ss -ltn | grep 5000
```

#### 6.3 Fetch DB Credentials File
Copy the credentials from the DB Leader node:
```bash
mkdir -p /etc/softswitch
scp root@192.168.10.34:/etc/softswitch/db-credentials /etc/softswitch/db-credentials
```

#### 6.4 (Optional High-Scale) Deploy Local PgBouncer Connection Pooler (Port 6432)

While local HAProxy provides seamless Layer 4 TCP routing and automatic leader failover, it operates on a 1:1 connection model: every connection opened by Telephony Server, Kamailio, and background APIs is passed directly to the PostgreSQL backend. 

In PostgreSQL, each client connection spawns a dedicated operating system process consuming approximately 10MB of RAM, along with CPU context-switching overhead.

Under high call volumes (e.g., 500–5,000+ concurrent calls, high Call Per Second / CPS bursts, or multi-tenant PBX workloads across multiple telephony nodes), direct connections can quickly exhaust PostgreSQL's `max_connections` limit and throttle database CPU performance.

##### 🚀 What PgBouncer Does
**PgBouncer** is an ultra-lightweight Layer 7 connection pooler designed specifically for PostgreSQL. When configured in `pool_mode = transaction`:
- **Multiplexes Connections**: Thousands of active telephony client queries share a tiny, reusable pool of **20–30 persistent backend connections**.
- **Instant Connection Lifecycle**: It attaches a real database connection only for the milliseconds required to execute a SQL query or transaction, releasing it immediately back to the pool.
- **Sub-Millisecond Latency**: It drops query overhead from ~15ms to **< 0.5ms**, eliminating TCP connection handshakes and preventing connection spikes during high CPS bursts or cluster failovers.

##### 📊 When Should You Add PgBouncer?
- **Standard Deployments (< 300 concurrent calls)**: Direct local HAProxy on port 5000 is fully sufficient.
- **Carrier & High-Volume Deployments (> 500–1,000+ concurrent calls, Multi-Node Telephony Server / Kamailio clusters, or high-frequency CDR logging)**: Strongly recommended to prevent database connection exhaustion and keep memory usage flat.

##### 🗺️ High-Scale Traffic Flow:
```
[ Telephony Core (ODBC :6432) ] ──┐
[ Kamailio (db_postgres)  ] ──┼─► [ Local PgBouncer :6432 ] ──► [ Local HAProxy :5000 ] ──► [ Patroni Leader :5432 ]
[ Node.js REST API        ] ──┘   (Transaction Pooling)         (Failover & Healthcheck)    (PostgreSQL Cluster)
```

##### ⚙️ Step-by-Step Installation & Configuration:

1. Install PgBouncer on the Telephony / Application Node:
```bash
apt-get install -y pgbouncer
```

2. Configure `/etc/pgbouncer/pgbouncer.ini` to route database traffic locally to HAProxy on port 5000:
```bash
cat << 'EOF' > /etc/pgbouncer/pgbouncer.ini
[databases]
ss_telephony = host=127.0.0.1 port=5000 dbname=ss_telephony
ring2all     = host=127.0.0.1 port=5000 dbname=ring2all
kamailio     = host=127.0.0.1 port=5000 dbname=kamailio
sbc_admin    = host=127.0.0.1 port=5000 dbname=sbc_admin

[pgbouncer]
logfile = /var/log/postgresql/pgbouncer.log
pidfile = /var/run/postgresql/pgbouncer.pid
listen_addr = 127.0.0.1
listen_port = 6432
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt
admin_users = postgres, ss_db_user

# Transaction pooling for carrier-grade throughput
pool_mode = transaction
max_client_conn = 5000
default_pool_size = 25
reserve_pool_size = 5
max_prepared_statements = 100

# Critical for Node.js (Kysely/pg) and Telephony Server ODBC compatibility
ignore_startup_parameters = extra_float_digits, search_path, application_name
EOF
```

3. Generate `/etc/pgbouncer/userlist.txt` using the synced database credentials:
```bash
source /etc/softswitch/db-credentials
cat << EOF > /etc/pgbouncer/userlist.txt
"ss_db_user" "$DB_PASSWORD"
"postgres" "$DB_PASSWORD"
EOF
chmod 640 /etc/pgbouncer/userlist.txt
chown postgres:postgres /etc/pgbouncer/userlist.txt
```

4. Enable and start PgBouncer, then verify that port 6432 is listening:
```bash
systemctl enable --now pgbouncer
ss -ltn | grep 6432
```

5. When PgBouncer is active, adjust `/etc/odbc.ini` in Telephony Server to point to port `6432` instead of `5000`:
```ini
[ring2all]
Driver   = PostgreSQL ANSI
Database = ss_telephony
Servername = 127.0.0.1
Port     = 6432
UserName = ss_db_user
Password = YOUR_PASSWORD
```

---

#### 6.5 Mount Shared Storage Volumes
Configure host resolutions in `/etc/hosts` and mount the recordings and music volumes using `backup-volfile-servers` for redundancy:
```bash
# On Telephony Nodes (Telephony Server):
# Add storage hosts to /etc/hosts (update IPs if they differ in your lab)
cat << 'EOF' >> /etc/hosts
192.168.10.37 fs-node-01
192.168.10.38 fs-node-02
192.168.10.39 fs-node-03
EOF

apt-get install -y glusterfs-client

mkdir -p /var/lib/freeswitch/recordings
mkdir -p /usr/share/freeswitch/sounds/music

# Mount volumes
mount -t glusterfs -o backup-volfile-servers=fs-node-02:fs-node-03 fs-node-01:/ss-recordings /var/lib/freeswitch/recordings
mount -t glusterfs -o backup-volfile-servers=fs-node-02:fs-node-03 fs-node-01:/ss-music       /usr/share/freeswitch/sounds/music

# Configure persistent mounting in fstab
cat >> /etc/fstab << 'EOF'
fs-node-01:/ss-recordings /var/lib/freeswitch/recordings glusterfs defaults,_netdev,backup-volfile-servers=fs-node-02:fs-node-03 0 0
fs-node-01:/ss-music       /usr/share/freeswitch/sounds/music         glusterfs defaults,_netdev,backup-volfile-servers=fs-node-02:fs-node-03 0 0
EOF
```

#### 6.6 Install Telephony Modules
Install the custom `softswitch-telephony` configurations. The post-installation script will detect the local database proxy on port 5000 (or PgBouncer on port 6432 if configured) and write matching `/etc/odbc.ini` profiles automatically:
```bash
apt-get install -y -o Dpkg::Options::="--force-overwrite" softswitch-telephony
```

#### 6.7 Link Telephony Node to the Admin Monitoring Service (Optional CLI Method)

> [!NOTE]
> **Recommended Method (Web UI)**:
> In production multi-node environments, Telephony Nodes are added and managed directly from the Admin Web Portal under **Telephony > Telephony Servers**. Registering nodes in the Web UI automatically configures real-time monitoring across all active telephony cluster nodes.

For automated scripted deployments, you can optionally set the initial Telephony node IP (`192.168.10.41`) directly via SQL:

```bash
# On Admin Server (192.168.10.40):
source /etc/softswitch/db-credentials
PGPASSWORD="$DB_PASSWORD" psql -h 127.0.0.1 -p 5000 -U ss_db_user -d ss_telephony \
  -c "UPDATE esl_users SET listen_ip = '192.168.10.41' WHERE username = 'system_monitoring';"

# Restart monitoring service
systemctl restart softswitch-monitoring-api
```


---

## 🔑 Secure Credentials Distribution

To distribute generated keys securely across the cluster network, create and run this helper script on the DB Leader node:

```bash
cat << 'EOF' > distribute-credentials.sh
#!/bin/bash
# distribute-credentials.sh

# Load database credentials
source /etc/softswitch/db-credentials

# Define client IP list
CLIENTS=(
    "192.168.10.40"  # Admin API Server
    "192.168.10.41"  # Telephony Server 1
    "192.168.10.42"  # Telephony Server 2
    "192.168.10.43"  # Telephony Server 3
    "192.168.10.44"  # Telephony Server 4
    "192.168.10.45"  # Telephony Server 5
)

for target in "${CLIENTS[@]}"; do
    echo "🔑 Secure copying credentials to: ${target}..."
    ssh root@${target} "mkdir -p /etc/softswitch"
    scp /etc/softswitch/db-credentials root@${target}:/etc/softswitch/db-credentials
done
EOF

chmod +x distribute-credentials.sh
./distribute-credentials.sh
```

---

## 🔍 Validation Checklist

Review this checklist to ensure all distributed platform modules are connected:

### 1. Database Clustering
- [ ] Patroni reports cluster health correctly (`patronictl -c /etc/patroni/config.yml list`).
- [ ] Read requests are distributed across all standby replicas via port 5001.
- [ ] Write requests are successfully routed to the Patroni leader node via port 5000.

### 2. Storage Clustering
- [ ] GlusterFS volumes report healthy and synchronized brick status (`gluster volume status`).
- [ ] Files written to `/var/lib/freeswitch/recordings` synchronize to all storage bricks.

### 3. Telephony Connectivity
- [ ] Telephony Server successfully connects to database schemas via ODBC.
- [ ] The `fs_cli -x "sofia status"` command reports the internal and external profiles as RUNNING.
- [ ] SIP devices can register and route calls using any telephony node IP.

---

## 📈 Capacity & Scaling Guidelines

Refer to this guide to plan your cluster deployment architecture:

| Concurrent Calls | Telephony Nodes | Database Configuration | Storage Type |
|------------------|-----------------|------------------------|--------------|
| **< 100** | 1 | Single Local DB | Local Filesystem |
| **100–500** | 2 | Single Dedicated DB | Replicated GlusterFS (2 Nodes) |
| **500–1,000** | 3 | Replicated Primary + Replica | Replicated GlusterFS (3 Nodes) |
| **1,000–5,000** | 4–8 | Patroni Cluster (3+ Nodes) | Distributed-Replicated GlusterFS |
| **> 5,000** | 8+ | High-Availability Cluster | Dedicated Storage Array / SAN |

---

## 🔍 Troubleshooting

### 1. Database connection is refused from clients
- **Cause**: The Patroni node has not finished bootstrapping, or nftables firewalls are blocking port `5432`/`8008`.
- **Fix**: Check status using `patronictl`:
  ```bash
  patronictl -c /etc/patroni/config.yml list
  ```
  Run a TCP socket check from the client to the database node:
  ```bash
  nc -zv 192.168.10.34 5432
  ```

### 2. Telephony Server logs "CORE DATABASE INITIALIZATION FAILURE"
- **Cause**: The core system-level DSN connection is misaligned in `/etc/freeswitch/autoload_configs/switch.conf.xml`.
- **Fix**: Check `core-db-dsn`. If the parameters are pointing directly to the database IP or port 5432, change the endpoint to the local HAProxy write interface:
  ```xml
  <param name="core-db-dsn" value="pgsql://host=127.0.0.1 port=5000 dbname=freeswitch user=ss_db_user password=YOUR_PASSWORD" />
  ```
  Restart Telephony Server to reinitialize the SQL engine:
  ```bash
  systemctl reset-failed freeswitch && systemctl start freeswitch
  ```

### 3. Web portal returns "502 Bad Gateway" on API calls or dashboard widgets
- **Cause**: The API services are not running or failed to initialize their dependencies.
- **Fix**: Verify systemd status:
  ```bash
  systemctl status softswitch-api
  systemctl status softswitch-monitoring-api
  ```
  If dependencies are missing or compilation failed, run a manual npm installation:
  ```bash
  cd /var/www/softswitch
  npm install
  systemctl restart softswitch-api
  systemctl restart softswitch-monitoring-api
  ```

---

*This concludes our 8-part Debian 13 Clustering and Distribution series. You now have a complete, production-ready, enterprise-grade softswitch platform deployed on Debian 13.*