📂 Part 5: Distributed Storage with GlusterFS on Debian 13 (Trixie)

🏗️ Why Use Distributed Storage?

In a multi-server PBX deployment, sharing files over a single server via standard NFS creates a Single Point of Failure (SPOF) and a severe I/O bottleneck.

A distributed file system like GlusterFS solves this by offering:

1. Redundancy (Replica 3): Media files are written simultaneously to three nodes. If a storage node goes offline, files remain fully accessible.
2. Active-Active Access: Client nodes (like FreeSWITCH application servers) can read and write to any storage node in the pool.
3. Self-Healing: If a failed node is restored, GlusterFS automatically synchronizes missing block differences.

🏛️ Storage Cluster Architecture

Our cluster will use 3 dedicated storage nodes to run a split-brain safe Replica 3 volume.

🛠️ Step 1: Environment Preparation (All 3 Storage Nodes)

Log into each storage node to perform the base installation.

1.1 Configure Hostnames

Assign hostnames to distinguish nodes:

# On fs-node-01
hostnamectl set-hostname fs-node-01

# On fs-node-02
hostnamectl set-hostname fs-node-02

# On fs-node-03
hostnamectl set-hostname fs-node-03

1.2 Resolution IP Mappings (/etc/hosts)

Add this mapping block to /etc/hosts on all nodes (including your VoIP/FreeSWITCH clients):

192.168.10.34 fs-node-01
192.168.10.35 fs-node-02
192.168.10.36 fs-node-03

1.3 Install GlusterFS Daemon

Debian 13 includes GlusterFS Server in its native repositories. Run the following command on all storage nodes:

apt update && apt upgrade -y
apt install -y glusterfs-server
systemctl enable --now glusterd
systemctl status glusterd

1.4 Firewall Configuration (nftables)

GlusterFS dynamically maps ports, but the main broker daemon listens on port 24007. To ensure simple and secure communications, allow total access between storage cluster nodes, and restrict client mount traffic.

Add these rules to /etc/nftables.conf on all 3 nodes:

cat <<EOF > /etc/nftables.conf
#!/usr/sbin/nft -f

flush ruleset

table inet filter {
    chain input {
        type filter hook input priority 0; policy drop;
        iif "lo" accept
        ct state established,related accept
        tcp dport 22 accept
        ip protocol icmp accept
        
        # Internal Storage Traffic: Total access between cluster nodes
        ip saddr { 192.168.10.34, 192.168.10.35, 192.168.10.36 } accept
        
        # Client Mount Traffic: Allow VoIP subnet to access daemon and brick port ranges
        # ip saddr { 192.168.10.0/24 } tcp dport { 24007, 24008, 49152-49251 } accept
    }
    chain forward { type filter hook forward priority 0; policy drop; }
    chain output { type filter hook output priority 0; policy accept; }
}
EOF

systemctl enable --now nftables

🤝 Step 2: Establish the Trusted Storage Pool

Next, we link our cluster nodes together. Run these peer commands from fs-node-01 ONLY:

gluster peer probe fs-node-02
gluster peer probe fs-node-03

Verify that both nodes are successfully connected:

gluster peer status

Number of Peers: 2

Hostname: fs-node-02
Uuid: 5e612cb7-8547-4ad3-a9d1-3e4b104928db
State: Peer in Cluster (Connected)

Hostname: fs-node-03
Uuid: 8cb019ef-b328-40ef-bb4b-cf109cde8e11
State: Peer in Cluster (Connected)

🧱 Step 3: Create the Distributed Volume

With our storage pool established, we define a replicated volume named gv_recordings.

3.1 Create Brick Directories (All 3 Nodes)

Create a directory path where GlusterFS will write block files:

mkdir -p /gluster/brick1/gv_recordings

3.2 Initialize the Volume (fs-node-01 only)

Define the replicated volume structure:

gluster volume create gv_recordings replica 3 \
  fs-node-01:/gluster/brick1/gv_recordings \
  fs-node-02:/gluster/brick1/gv_recordings \
  fs-node-03:/gluster/brick1/gv_recordings \
  force

3.3 Start the Replicated Volume

gluster volume start gv_recordings

Verify that the volume is started:

gluster volume info
# Status: Started

🔌 Step 4: Client Configuration (Mounting & High Availability)

To access the distributed storage pool, telephony and application servers must mount the gv_recordings volume locally.

4.1 Install the Native FUSE Client

Run the following command on all client servers (such as FreeSWITCH or API nodes):

apt install -y glusterfs-client

4.2 Create Target Mount Point

Create the directories matching FreeSWITCH's expected layout:

mkdir -p /var/lib/freeswitch/recordings

4.3 Configure Automount in /etc/fstab with Failover

To ensure the storage mount survives server restarts and automatically handles node failures, configure the /etc/fstab table using the backup-volfile-servers flag.

If the primary node fs-node-01 goes offline, the client will immediately switch to fs-node-02 or fs-node-03 to keep reads/writes alive without interrupting calls:

fs-node-01:/gv_recordings /var/lib/freeswitch/recordings glusterfs defaults,_netdev,backup-volfile-servers=fs-node-02:fs-node-03 0 0

4.4 Mount and Verify

Trigger the mounting process and inspect the active storage size:

mount -a
df -h | grep recordings

4.5 Alternative: NAS / NFS Mount Option

If your organization prefers using a pre-existing physical NAS appliance instead of building a software cluster, configure NFS client mounting on your nodes:

# Install NFS client
apt install -y nfs-common

# Create mount point
mkdir -p /var/lib/freeswitch/recordings

# Add configuration to fstab
echo "nas.example.com:/recordings /var/lib/freeswitch/recordings nfs defaults,_netdev 0 0" >> /etc/fstab

# Mount
mount -a

🔄 Step 5: Advanced Operations & Health Checks

Self-Healing Synchronization

If a storage node crashes and is later restored, the cluster repairs block differences automatically in the background. To monitor progress, run:

gluster volume heal gv_recordings info

If the output displays 0 entries, your distributed data is fully synchronized.

🏁 Series Wrap-Up

Congratulations! You have completed the Debian 13 Clustering & Distribution guide series.

We have successfully covered:

1. Part 1: Setting up a secure, signed public APT repository.
2. Part 2: Compiling and packaging FreeSWITCH from source.
3. Part 3: Packaging Kamailio and deploying it securely.
4. Part 4: Implementing a self-healing PostgreSQL cluster using Patroni and Etcd consensus.
5. Part 5: Building a replicated storage layer using GlusterFS for media file distribution.

Your infrastructure is now secure, modular, highly available, and ready for enterprise scale.