---
title: "🛡️ Part 13: High-Availability Multi-Master Ring2All SBC Cluster on Debian 13"
description: "Documentation for 13. Ring2All SBC Multi-Master HA Cluster"
---

*Welcome to the thirteenth installment of our "Debian 13 Clustering & Distribution" series. In previous guides, we covered the standalone deployment of Ring2All SBC (Part 9), PostgreSQL 17 high-availability clustering with Patroni & Etcd (Part 4), and enterprise distributed telephony architectures (Part 8). In this comprehensive guide, we elevate our boundary defense to a carrier-grade **Multi-Node Session Border Controller (SBC) Cluster** powered by **Ring2All SBC (Kamailio 6.x + Sipwise RTPEngine)** on Debian 13 (Trixie).*

*In this architecture, every SBC node actively processes inbound and outbound SIP traffic simultaneously. In-memory telephony states—including SIP user registrations (`usrloc`), active calls and dialogs (`dialog`), security rate limits, and dynamic blacklists (`htable`)—are replicated across all cluster nodes in real time using Kamailio's Distributed Message Queue (DMQ) engine. Furthermore, to prevent database bottlenecks under heavy call traffic, every SBC node employs a local high-performance database proxy stack: **HAProxy (Layer 4 failover routing on port 5000)** and **PgBouncer (Layer 7 transaction pooling on port 6432)** connected to our self-healing PostgreSQL 17 Patroni cluster.*

---

## 🏗️ Architecture & Deployment Topologies

Ring2All SBC is built on a modular package architecture (`softswitch-sbc-db`, `softswitch-sbc-telephony`, `softswitch-sbc-api`, `softswitch-sbc-admin`), allowing enterprises to deploy either of two proven production topologies:

---

### Topology A: Central Control Plane + Headless Telephony Edge Nodes (Aligned with Ring2All PBX)

Just like in **Ring2All PBX (Part 8)**—where the Admin Web UI and Fastify API reside on a central control plane server while the telephony engines run on dedicated, lightweight, headless nodes—Ring2All SBC supports complete separation of the management interface from the high-throughput SIP boundary routers:

```
┌─────────────────────────────────────────────────────────────────────────────────────────┐
│              TOPOLOGY A: CENTRAL CONTROL PLANE & HEADLESS SBC EDGE ENGINES              │
├─────────────────────────────────────────────────────────────────────────────────────────┤
│                                                                                         │
│                               ADMINISTRATORS & NOC OPERATORS                            │
│                                              │                                          │
│                                              ▼                                          │
│                        ┌───────────────────────────────────────────┐                    │
│                        │       CENTRAL RING2ALL SBC MANAGEMENT          │                    │
│                        │       sbc-mgmt-01 (192.168.10.30)         │                    │
│                        │       - softswitch-sbc-admin (React UI)   │                    │
│                        │       - softswitch-sbc-api (Fastify API)  │                    │
│                        │       (No Kamailio / No RTPEngine)        │                    │
│                        └─────────────────────┬─────────────────────┘                    │
│                                              │                                          │
│                      ┌───────────────────────┴───────────────────────┐                  │
│                      │                                               │                  │
│                      ▼                                               ▼                  │
│       ┌───────────────────────────────┐               ┌───────────────────────────────┐ │
│       │    SBC TELEPHONY EDGE 01      │               │    SBC TELEPHONY EDGE 02      │ │
│       │    sbc-edge-01 (192.168.10.32)│               │    sbc-edge-02 (192.168.10.33)│ │
│       │ ┌───────────────────────────┐ │  DMQ (SIP RAM)│ ┌───────────────────────────┐ │ │
│       │ │ Kamailio 6.x (SIP Engine) │◄├───────────────┤►│ Kamailio 6.x (SIP Engine) │ │ │
│       │ │ - dmq_usrloc (Regs Sync)  │ │   Port 5090   │ │ - dmq_usrloc (Regs Sync)  │ │ │
│       │ │ - dmq_dialog (Calls Sync) │ │               │ │ - dmq_dialog (Calls Sync) │ │ │
│       │ │ - dmq_htable (Anti-Flood) │ │               │ │ - dmq_htable (Anti-Flood) │ │ │
│       │ └─────────────┬─────────────┘ │               │ └─────────────┬─────────────┘ │ │
│       │ ┌─────────────▼─────────────┐ │               │ ┌─────────────▼─────────────┐ │ │
│       │ │ RTPEngine (Media Relay)   │ │               │ │ RTPEngine (Media Relay)   │ │ │
│       │ └─────────────┬─────────────┘ │               │ └─────────────┬─────────────┘ │ │
│       │ ┌─────────────▼─────────────┐ │               │ ┌─────────────▼─────────────┐ │ │
│       │ │ PgBouncer (:6432 Pooler)  │ │               │ │ PgBouncer (:6432 Pooler)  │ │ │
│       │ └─────────────┬─────────────┘ │               │ └─────────────┬─────────────┘ │ │
│       │ ┌─────────────▼─────────────┐ │               │ ┌─────────────▼─────────────┐ │ │
│       │ │ HAProxy (:5000 DB Router) │ │               │ │ HAProxy (:5000 DB Router) │ │ │
│       │ └───────────────────────────┘ │               │ └───────────────────────────┘ │ │
│       │ (Headless: No Web UI/Node.js) │               │ (Headless: No Web UI/Node.js) │ │
│       └───────────────┬───────────────┘               └───────────────┬───────────────┘ │
│                       │                                               │                 │
│                       └───────────────────────┬───────────────────────┘                 │
│                                               ▼                                         │
│                               ┌───────────────────────────────┐                         │
│                               │   PostgreSQL 17 HA CLUSTER    │                         │
│                               │   (Patroni + Etcd Leader:5432)│                         │
│                               │   - softswitch-sbc-db         │                         │
│                               └───────────────────────────────┘                         │
│                                                                                         │
└─────────────────────────────────────────────────────────────────────────────────────────┘
```

---

### Topology B: Autonomous Multi-Master Active-Active Mesh

In this carrier-grade topology, each SBC node is a complete, self-contained edge stack running Telephony + API + Web UI. Operators can log into any node (`https://sbc-node-01/` or `https://sbc-node-02/`), and any configuration or reload is automatically propagated across all active nodes via the **Cluster Broadcast Service** and Kamailio DMQ.

```
                                  PUBLIC SIP TRAFFIC & CARRIERS
                                 (DNS SRV / Anycast / Round-Robin)
                                                │
                        ┌───────────────────────┴───────────────────────┐
                        │                                               │
                        ▼                                               ▼
        ┌───────────────────────────────┐               ┌───────────────────────────────┐
        │       SBC NODE 01 (Full Mesh) │               │       SBC NODE 02 (Full Mesh) │
        │       IP: 192.168.10.32       │               │       IP: 192.168.10.33       │
        │ ┌───────────────────────────┐ │  DMQ (SIP RAM)│ ┌───────────────────────────┐ │
        │ │ Kamailio 6.x (SIP Engine) │◄├───────────────┤►│ Kamailio 6.x (SIP Engine) │ │
        │ │ - dmq_usrloc (Regs Sync)  │ │   Port 5090   │ │ - dmq_usrloc (Regs Sync)  │ │
        │ │ - dmq_dialog (Calls Sync) │ │               │ │ - dmq_dialog (Calls Sync) │ │
        │ │ - dmq_htable (Anti-Flood) │ │               │ │ - dmq_htable (Anti-Flood) │ │
        │ └─────────────┬─────────────┘ │               │ └─────────────┬─────────────┘ │
        │ ┌─────────────▼─────────────┐ │               │ ┌─────────────▼─────────────┐ │
        │ │ RTPEngine (Media Relay)   │ │               │ │ RTPEngine (Media Relay)   │ │
        │ └─────────────┬─────────────┘ │               │ └─────────────┬─────────────┘ │
        │ ┌─────────────▼─────────────┐ │  Cluster API  │ ┌─────────────▼─────────────┐ │
        │ │ sbc-api + Web UI (Nginx)  │◄├───────────────┤►│ sbc-api + Web UI (Nginx)  │ │
        │ └─────────────┬─────────────┘ │   Port 3003   │ └─────────────┬─────────────┘ │
        │ ┌─────────────▼─────────────┐ │               │ ┌─────────────▼─────────────┐ │
        │ │ PgBouncer (:6432 Pooler)  │ │               │ │ PgBouncer (:6432 Pooler)  │ │
        │ └─────────────┬─────────────┘ │               │ └─────────────┬─────────────┘ │
        │ ┌─────────────▼─────────────┐ │               │ ┌─────────────▼─────────────┐ │
        │ │ HAProxy (:5000 DB Router) │ │               │ │ HAProxy (:5000 DB Router) │ │
        │ └───────────────────────────┘ │               │ └───────────────────────────┘ │
        └───────────────┬───────────────┘               └───────────────┬───────────────┘
                        │                                               │
                        ├───────────────────────┬───────────────────────┤
                        │                       │                       │
                        ▼                       ▼                       ▼
        ┌───────────────────────────────┐       │       ┌───────────────────────────────┐
        │   PostgreSQL 17 HA CLUSTER    │       │       │   TELEPHONY CORE (Telephony Server) │
        │   (Patroni + Etcd Leader:5432)│       │       │   fs-01, fs-02, fs-03         │
        └───────────────────────────────┘       │       └───────────────────────────────┘
                                                ▼
                                ┌───────────────────────────────┐
                                │ Redis Pub/Sub (Media Session) │
                                └───────────────────────────────┘
```

---

## 📊 Topology Comparison Matrix

| Feature | Topology A: Decoupled Central Control (Ring2All PBX Style) | Topology B: Autonomous Multi-Master Mesh |
| :--- | :--- | :--- |
| **Recommended Use Case** | **Enterprise, Large PBX Networks, Centralized NOC** | **Carriers, Multi-DataCenter, Geo-Redundant Clouds** |
| **Management Point** | **Single Portal**: `https://sbc-mgmt-01/` | **Any Edge Node**: `https://sbc-01/` or `https://sbc-02/` |
| **Edge Server Footprint** | **Minimal**: Only Kamailio + RTPEngine (Headless) | **Full**: Kamailio + RTPEngine + Node.js API + Web |
| **Node Packages (Edge)** | `softswitch-sbc-telephony` | `softswitch-sbc-telephony`, `softswitch-sbc-api`, `softswitch-sbc-admin` |
| **State Replication** | Real-time RAM via Kamailio DMQ (Port 5090) | Real-time RAM via Kamailio DMQ + HTTP Cluster API |
| **DB Access on Edge** | Local PgBouncer (:6432) → HAProxy (:5000) | Local PgBouncer (:6432) → HAProxy (:5000) |

---

## 🖥️ Server Roles & Lab Requirements

For our reference deployment, we allocate dedicated servers across our cluster:

| Hostname | Example IP | Role & Installed Packages | Minimum Specifications |
| :--- | :--- | :--- | :--- |
| **sbc-mgmt-01** *(Top. A)* | `192.168.10.30` | Central Management Server (`softswitch-sbc-admin`, `softswitch-sbc-api`) | 2 vCPU, 4GB RAM, SSD |
| **sbc-edge-01** | `192.168.10.32` | SBC Telephony Edge Node 1 (`softswitch-sbc-telephony` + local PgBouncer/HAProxy) | 4 vCPU, 8GB RAM, SSD |
| **sbc-edge-02** | `192.168.10.33` | SBC Telephony Edge Node 2 (`softswitch-sbc-telephony` + local PgBouncer/HAProxy) | 4 vCPU, 8GB RAM, SSD |
| **pg-node-01..03** | `192.168.10.34..36` | PostgreSQL 17 Patroni Cluster (`softswitch-sbc-db`) | *From Part 4* |
| **fs-node-01..03** | `192.168.10.41..43` | Telephony Server Telephony Core Engines | *From Part 8* |

---

## 🛠️ Step-by-Step Deployment Guide

---

### Phase 1: Database Initialization & Credentials

Ring2All SBC requires two PostgreSQL databases:
1. `sbc_admin`: Stores administrative users, RBAC roles, SSL certificates, Apple/Google push profiles, and cluster settings.
2. `kamailio`: Stores runtime SIP routing tables, dispatchers, subscribers, domain aliases, and access control lists.

Log in to the **Active Database Leader** (`pg-node-01` / `192.168.10.34`):

```bash
# Option A: One-Touch DB Initialization (Recommended)
wget -O- https://repo.softswitchone.com/apt/install-sbc.sh | bash -s -- --db-only

# Option B: Manual APT Installation
wget -qO- https://repo.softswitchone.com/apt/setup_repo | bash
apt-get update
apt-get install -y softswitch-sbc-db
```

Verify that credentials file `/etc/softswitch/db-credentials` (or `/etc/softswitch/sbc-db-credentials`) exists on the DB leader.

---

### Phase 2: Base System Preparation (On All SBC Nodes)

Execute these steps on **all SBC nodes** (`sbc-mgmt-01`, `sbc-edge-01`, `sbc-edge-02`):

```bash
# 1. Configure Hostname and Hosts Resolution
cat << 'EOF' >> /etc/hosts
192.168.10.30 sbc-mgmt-01
192.168.10.32 sbc-edge-01
192.168.10.33 sbc-edge-02
192.168.10.34 pg-node-01
192.168.10.35 pg-node-02
192.168.10.36 pg-node-03
192.168.10.41 fs-node-01
192.168.10.42 fs-node-02
192.168.10.43 fs-node-03
EOF

# 2. Install base utilities
apt-get update
apt-get install -y curl wget gnupg2 sudo lsb-release net-tools htop

# 3. Fetch cluster database credentials from the DB leader node
mkdir -p /etc/softswitch
scp root@192.168.10.34:/etc/softswitch/db-credentials /etc/softswitch/db-credentials
chmod 600 /etc/softswitch/db-credentials
```

---

### Phase 3: Deploy Local HAProxy Database Proxy (On SBC Edge Nodes)

To enable zero-downtime database failover, install HAProxy locally on each SBC telephony node. HAProxy continuously monitors Patroni's HTTP healthcheck endpoint (`:8008/primary`) and routes write traffic dynamically to the active PostgreSQL leader.

On **both `sbc-edge-01` and `sbc-edge-02`**:

```bash
apt-get install -y haproxy

cat << 'EOF' > /etc/haproxy/haproxy.cfg
global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    user haproxy
    group haproxy
    daemon

defaults
    log global
    mode tcp
    option tcplog
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

# Port 5000: Write transactions routed to Patroni Leader
frontend pg_write
    bind 127.0.0.1:5000
    default_backend pg_primary

backend pg_primary
    mode tcp
    option httpchk GET /primary
    http-check expect status 200
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server pg-node-01 192.168.10.34:5432 maxconn 100 check port 8008
    server pg-node-02 192.168.10.35:5432 maxconn 100 check port 8008
    server pg-node-03 192.168.10.36:5432 maxconn 100 check port 8008

# Port 5001: Read transactions load-balanced across replicas
frontend pg_read
    bind 127.0.0.1:5001
    default_backend pg_replicas

backend pg_replicas
    mode tcp
    balance roundrobin
    option httpchk GET /replica
    http-check expect status 200
    default-server inter 3s fall 3 rise 2
    server pg-node-01 192.168.10.34:5432 check port 8008
    server pg-node-02 192.168.10.35:5432 check port 8008
    server pg-node-03 192.168.10.36:5432 check port 8008
EOF

# Validate and restart HAProxy
haproxy -c -f /etc/haproxy/haproxy.cfg
systemctl restart haproxy
systemctl enable haproxy
ss -ltn | grep 5000
```

---

### Phase 4: Deploy Local PgBouncer Connection Pooler (On SBC Edge Nodes)

Each active Kamailio worker thread and database connection multiplexer connects to PostgreSQL. With multiple SBC nodes processing hundreds of CPS, opening direct PostgreSQL connections can cause connection thrashing and CPU saturation. 

By layering **PgBouncer** in `pool_mode = transaction` on port `6432`:
- Up to **5,000 client connections** are multiplexed into **20–30 persistent connections** to HAProxy (`127.0.0.1:5000`).
- Query latency is reduced to **< 0.5ms**.

On **both `sbc-edge-01` and `sbc-edge-02`**:

```bash
apt-get install -y pgbouncer

# 1. Configure PgBouncer
cat << 'EOF' > /etc/pgbouncer/pgbouncer.ini
[databases]
sbc_admin = host=127.0.0.1 port=5000 dbname=sbc_admin
kamailio  = host=127.0.0.1 port=5000 dbname=kamailio

[pgbouncer]
logfile = /var/log/postgresql/pgbouncer.log
pidfile = /var/run/postgresql/pgbouncer.pid
listen_addr = 127.0.0.1
listen_port = 6432
auth_type = trust
auth_file = /etc/pgbouncer/userlist.txt
admin_users = postgres, ss_db_user

# Transaction pooling optimization
pool_mode = transaction
max_client_conn = 5000
default_pool_size = 30
reserve_pool_size = 5
max_prepared_statements = 100

# Critical parameters for Node.js (Kysely/pg) and Kamailio db_postgres
ignore_startup_parameters = extra_float_digits, search_path, application_name
EOF

# 2. Build authentication userlist
source /etc/softswitch/db-credentials
cat << EOF > /etc/pgbouncer/userlist.txt
"ss_db_user" "$DB_PASSWORD"
"postgres" "$DB_PASSWORD"
"kamailio" "$DB_PASSWORD"
EOF
chmod 640 /etc/pgbouncer/userlist.txt
chown postgres:postgres /etc/pgbouncer/userlist.txt

# 3. Enable and start PgBouncer
systemctl enable --now pgbouncer
ss -ltn | grep 6432
```

---

### Phase 5: Install Modular Softswitch SBC Software

Select your deployment topology:

---

#### 🅰️ Deployment for Topology A: Decoupled Central Control (Ring2All PBX Style)

##### 1. On Central Management Server (`sbc-mgmt-01` / `192.168.10.30`):
Install the Administrative API and Web Portal:
```bash
# Option A: One-Touch Installer
wget -O- https://repo.softswitchone.com/apt/install-sbc.sh | bash -s -- --api-only
wget -O- https://repo.softswitchone.com/apt/install-sbc.sh | bash -s -- --admin-only

# Option B: Manual APT Installation
wget -qO- https://repo.softswitchone.com/apt/setup_repo | bash
apt-get update
apt-get install -y softswitch-sbc-api softswitch-sbc-admin
```

##### 2. On Headless Telephony Edge Nodes (`sbc-edge-01` & `sbc-edge-02`):
Install **strictly the SIP & Media engine** without web or node overhead:
```bash
# Option A: One-Touch Installer
wget -O- https://repo.softswitchone.com/apt/install-sbc.sh | bash -s -- --telephony-only

# Option B: Manual APT Installation
wget -qO- https://repo.softswitchone.com/apt/setup_repo | bash
apt-get update
apt-get install -y softswitch-sbc-telephony
```

---

#### 🅱️ Deployment for Topology B: Autonomous Multi-Master Mesh

On **both `sbc-edge-01` and `sbc-edge-02`**:
```bash
# Option A: One-Touch Edge Stack Installer
wget -O- https://repo.softswitchone.com/apt/install-sbc.sh | bash -s -- --sbc-edge

# Option B: Manual APT Installation
wget -qO- https://repo.softswitchone.com/apt/setup_repo | bash
apt-get update
apt-get install -y softswitch-sbc-telephony softswitch-sbc-api softswitch-sbc-admin
```

---

### Phase 6: Configure Kamailio Distributed Message Queue (DMQ) Engine

To enable real-time state synchronization between `sbc-edge-01` and `sbc-edge-02`, we configure the Kamailio DMQ modules:
- **`dmq`**: Core clustering communication bus over internal SIP UDP.
- **`dmq_usrloc`**: Broadcasts SIP endpoint registrations instantly across nodes.
- **`dmq_dialog`**: Broadcasts active call state (dialogs) so in-flight calls survive node restarts.
- **`dmq_htable`**: Synchronizes anti-flood rate limit counters and dynamic IP blacklists in RAM.

#### A. Configure DMQ on `sbc-edge-01` (`192.168.10.32`)
Edit `/etc/kamailio/kamailio.cfg` (or `/etc/kamailio/kamailio-local.cfg`):

```c
#!KAMAILIO

####### Defined Values #########
#!define DBURL "postgres://ss_db_user:TU_CLAVE@127.0.0.1:6432/kamailio"
#!define LOCAL_IP "192.168.10.32"
#!define PEER_IP "192.168.10.33"
#!define DMQ_PORT 5090

####### Global Parameters #########
listen=udp:LOCAL_IP:5060
listen=tcp:LOCAL_IP:5060
listen=udp:LOCAL_IP:DMQ_PORT

####### Modules Section #########
loadmodule "db_postgres.so"
loadmodule "sl.so"
loadmodule "tm.so"
loadmodule "rr.so"
loadmodule "pv.so"
loadmodule "usrloc.so"
loadmodule "dialog.so"
loadmodule "htable.so"
loadmodule "dmq.so"
loadmodule "dmq_usrloc.so"
loadmodule "dmq_dialog.so"

# ---------- DMQ Core Configuration ----------
modparam("dmq", "server_address", "sip:LOCAL_IP:5090")
modparam("dmq", "notification_address", "sip:PEER_IP:5090")
modparam("dmq", "multi_notify", 1)
modparam("dmq", "ping_interval", 15)

# ---------- UsrLoc & DMQ UsrLoc ----------
modparam("usrloc", "db_url", DBURL)
modparam("usrloc", "db_mode", 2) # Write-Back DB mode
modparam("dmq_usrloc", "enable", 1)
modparam("dmq_usrloc", "sync", 1)

# ---------- Dialog & DMQ Dialog ----------
modparam("dialog", "db_url", DBURL)
modparam("dialog", "db_mode", 1) # Realtime DB mode
modparam("dialog", "enable_stats", 1)
modparam("dmq_dialog", "enable", 1)

# ---------- HTable (Anti-Flood & DMQ Sync) ----------
modparam("htable", "htable", "ipban=>size=16;autoexpire=3600;dmqreplicate=1;")
modparam("htable", "htable", "ratelimit=>size=16;autoexpire=60;dmqreplicate=1;")
modparam("htable", "dmq_replicate_intervals", 1)

####### Routing Logic #########
route {
    # Process DMQ Cluster Messages Internally
    if (is_method("KDMQ")) {
        dmq_handle_message();
        exit;
    }

    # Standard SIP Processing Logic
    route(REQINIT);
    route(AUTH);
    route(REGISTRAR);
    route(RELAY);
}
```

#### B. Configure DMQ on `sbc-edge-02` (`192.168.10.33`)
Invert `LOCAL_IP` and `PEER_IP` in `/etc/kamailio/kamailio.cfg`:
```c
#!define LOCAL_IP "192.168.10.33"
#!define PEER_IP "192.168.10.32"
```

Restart and verify Kamailio on both nodes:
```bash
kamailio -c
systemctl restart kamailio
```

---

### Phase 7: Configure Ring2All SBC API & Web UI

Configure the administrative REST API on your Management Server (Topology A) or on both Edge nodes (Topology B) via `/etc/softswitch/sbc-api.env`:

```bash
source /etc/softswitch/db-credentials

cat << EOF > /etc/softswitch/sbc-api.env
NODE_ENV=production
PORT=3003
HOST=127.0.0.1

# Point API to Database (Local PgBouncer :6432 or Cluster Proxy)
DATABASE_URL=postgresql://ss_db_user:${DB_PASSWORD}@127.0.0.1:6432/sbc_admin
KAM_DATABASE_URL=postgresql://ss_db_user:${DB_PASSWORD}@127.0.0.1:6432/kamailio

# Cluster HA Topology Setting
HA_MODE=active_active_mesh
DMQ_SERVER_ADDRESS=sip:127.0.0.1:5090
DMQ_PING_INTERVAL=15
EOF

chmod 600 /etc/softswitch/sbc-api.env
systemctl restart sbc-api
systemctl restart nginx
```

---

### Phase 8: Cluster Broadcast & Multi-Node Centralized Management

In Ring2All SBC, you never have to log into each SBC node individually to execute reloads or replicate satellite drop-in files. The **Cluster Broadcast Service** (`ClusterBroadcastService`) provides a unified single pane of glass:

1. **Register Cluster Nodes in Web UI**:
   - Navigate to **High Availability & Cluster → Nodes**.
   - Add all active SBC edge nodes (`192.168.10.32`, `192.168.10.33`) with their internal API secret tokens.
2. **Atomic RPC Broadcasting**:
   - Whenever an administrator modifies Outbound Routes, Carrier Dispatchers, TLS Certificates, or MS Teams Direct Routing, the API automatically broadcasts `binrpc` execution (`kamcmd drouting.reload`, `kamcmd dispatcher.reload`, `kamcmd tls.reload`) across **all registered cluster nodes simultaneously** via parallel promises with circuit breaker timeouts.
3. **Satellite Configuration Layer (SMR / `conf.d/`)**:
   - Static drop-in configurations (such as Teams routing `.cfg` or bespoke dialplan snippets) are replicated to `/etc/kamailio/conf.d/*.cfg` across all edge nodes using the internal cluster endpoint `/internal/ha-cluster/sync-dropin-cfg`.

---

## 🔍 Validation & Cluster Failover Testing

After configuring all nodes, run the following verification procedures to ensure full cluster synchrony:

### 1. Verify DMQ Cluster Mesh State
Execute on `sbc-edge-01`:
```bash
kamctl rpc dmq.list_nodes
```
*Expected Output:*
```json
{
  "jsonrpc": "2.0",
  "result": [
    {
      "host": "192.168.10.32",
      "port": 5090,
      "status": "active",
      "last_ping": "now"
    },
    {
      "host": "192.168.10.33",
      "port": 5090,
      "status": "active",
      "last_ping": "now"
    }
  ]
}
```

### 2. Test Real-Time SIP Registration (`UsrLoc`) Replication
1. Register a SIP endpoint (e.g., Extension `1001`) against **SBC Edge 1** (`192.168.10.32:5060`).
2. Query the user location table directly in memory on **SBC Edge 2** (`192.168.10.33`):
   ```bash
   kamcli ul show 1001
   ```
*Verification*: Extension `1001` appears in Edge 2's memory immediately with identical Contact URIs and socket metadata.

### 3. Test Dynamic Blacklist & Rate Limit Replication
Block an offending IP on `sbc-edge-01`:
```bash
kamcli htable seti ipban 198.51.100.25 1
```
Check the hash table on `sbc-edge-02`:
```bash
kamcli htable get ipban 198.51.100.25
```
*Verification*: Value `1` is retrieved instantly on Edge 2 without querying PostgreSQL.

### 4. Test Zero-Downtime Node Failover
1. Establish an active call through `sbc-edge-01`.
2. Inspect active dialogs on `sbc-edge-02`:
   ```bash
   kamcli dialog show
   ```
3. Abruptly stop Kamailio on `sbc-edge-01` (`systemctl stop kamailio`).
4. Route the subsequent in-dialog `BYE` or `re-INVITE` to `sbc-edge-02`.
*Verification*: Edge 2 recognizes the call dialog ID, processes the request, and tears down the media session cleanly with zero dropped packets.

### 5. Verify PgBouncer Connection Pooling
Verify that thousands of potential calls are pooled cleanly on port 6432:
```bash
psql -p 6432 -h 127.0.0.1 -U ss_db_user -d sbc_admin -c "SHOW POOLS;"
```
*Verification*: `cl_active` (client connections) multiplexes into a compact, steady `sv_active` (server connections) to HAProxy.

---

## 📈 Day-2 Operations & Diagnostic Cheat Sheet

| Task | Command |
| :--- | :--- |
| **Check DMQ Node Health** | `kamctl rpc dmq.list_nodes` |
| **Trigger Full DMQ Resync** | `kamctl rpc dmq.request_sync` |
| **View Replicated Endpoints** | `kamcli ul show` |
| **View Active Shared Dialogs** | `kamcli dialog show` |
| **Inspect Replicated HTables** | `kamctl rpc htable.dump ipban` |
| **Reload Dispatchers dynamically** | `kamcli dispatcher reload` |
| **Inspect PgBouncer Client Pools** | `psql -p 6432 -h 127.0.0.1 -U postgres pgbouncer -c "SHOW CLIENTS;"` |
| **Inspect HAProxy Backend States** | `echo "show stat" | socat stdio /var/lib/haproxy/stats` |

---

## 🏁 Conclusion

You now have a production-ready, enterprise-grade **Multi-Node Ring2All SBC Cluster** on Debian 13 (Trixie). 

Whether you deploy in a **Decoupled Central Control Plane topology** (matching Ring2All PBX) or an **Autonomous Multi-Master Mesh**, the union of **Kamailio 6.x DMQ real-time in-memory replication**, **RTPEngine media distribution**, **HAProxy automatic failover routing**, and **PgBouncer transaction-level connection pooling** guarantees maximum throughput, horizontal scalability, and zero downtime across your telephony perimeter.