📂 Part 5: Distributed Storage with GlusterFS on Debian 13 (Trixie)

Welcome to the fifth and final installment of our “Debian 13 Clustering & Distribution” series. In the previous guide, we deployed a self-healing PostgreSQL cluster. Now, we will establish a highly available, active-active distributed file storage system for telephony media assets (such as call recordings, voicemail logs, and system music). In this tutorial, we will configure GlusterFS (Replica 3) on Debian 13, build a trusted storage pool, create replication volumes, and configure target clients with automatic failover mounts.

---

🏗️ Why Use Distributed Storage?

In a multi-server PBX deployment, sharing files over a single server via standard NFS creates a Single Point of Failure (SPOF) and a severe I/O bottleneck.

A distributed file system like GlusterFS solves this by offering:

1. Redundancy (Replica 3): Media files are written simultaneously to three nodes. If a storage node goes offline, files remain fully accessible.
2. Active-Active Access: Client nodes (like Telephony application servers) can read and write to any storage node in the pool.
3. Self-Healing: If a failed node is restored, GlusterFS automatically synchronizes missing block differences.

---

🏛️ Storage Cluster Architecture

Our cluster will use 3 dedicated storage nodes to run a split-brain safe Replica 3 volume.

Role	Hostname	IP	Function
File Server 1	fs-node-01	192.168.10.37	GlusterFS Brick Node 1
File Server 2	fs-node-02	192.168.10.38	GlusterFS Brick Node 2
File Server 3	fs-node-03	192.168.10.39	GlusterFS Brick Node 3

---

🛠️ Step 1: Environment Preparation (All 3 Storage Nodes)

Log into each storage node to perform the base installation.

1.1 Configure Hostnames

Assign hostnames to distinguish nodes:

# On fs-node-01
hostnamectl set-hostname fs-node-01

# On fs-node-02
hostnamectl set-hostname fs-node-02

# On fs-node-03
hostnamectl set-hostname fs-node-03

1.2 Resolution IP Mappings (/etc/hosts)

Add this mapping block to /etc/hosts on all nodes (including your VoIP/FreeSWITCH clients):

cat << 'EOF' >> /etc/hosts
192.168.10.37 fs-node-01
192.168.10.38 fs-node-02
192.168.10.39 fs-node-03
EOF

1.3 Install GlusterFS Daemon

Debian 13 includes GlusterFS Server in its native repositories. Run the following command on all storage nodes:

apt update && apt upgrade -y
apt install -y glusterfs-server

1.4 Firewall Configuration (nftables)

GlusterFS dynamically maps ports, but the main broker daemon listens on port 24007. To ensure simple and secure communications, allow total access between storage cluster nodes, and restrict client mount traffic.

Add these rules to /etc/nftables.conf on all 3 nodes:

cat <<EOF > /etc/nftables.conf
#!/usr/sbin/nft -f

flush ruleset

table inet filter {
    chain input {
        type filter hook input priority 0; policy drop;
        iif "lo" accept
        ct state established,related accept
        tcp dport 22 accept
        ip protocol icmp accept

        # Internal Storage Traffic: Total access between cluster nodes
        ip saddr { 192.168.10.37, 192.168.10.38, 192.168.10.39 } accept

        # Client Mount Traffic: Allow VoIP subnet to access daemon and brick port ranges
        ip saddr { 192.168.10.0/24 } tcp dport { 24007, 24008, 49152-65535 } accept
    }
    chain forward { type filter hook forward priority 0; policy drop; }
    chain output { type filter hook output priority 0; policy accept; }
}
EOF

systemctl start nftables
systemctl enable nftables

---

🤝 Step 2: Establish the Trusted Storage Pool

Next, we link our cluster nodes together.

2.1 Start and Enable glusterd (All 3 Nodes)

Start and enable the glusterd daemon across all three storage nodes before joining them to the pool:

systemctl start glusterd
systemctl enable glusterd
systemctl status glusterd

2.2 Probe Peers (fs-node-01 only)

Run these peer commands from fs-node-01 ONLY to register the other nodes:

gluster peer probe fs-node-02
gluster peer probe fs-node-03

Verify that both nodes are successfully connected:

gluster peer status

Expected Output:

Number of Peers: 2

Hostname: fs-node-02
Uuid: 5e612cb7-8547-4ad3-a9d1-3e4b104928db
State: Peer in Cluster (Connected)

Hostname: fs-node-03
Uuid: 8cb019ef-b328-40ef-bb4b-cf109cde8e11
State: Peer in Cluster (Connected)

---

🧱 Step 3: Create the Distributed Volumes

With our storage pool established, we define the three replicated volumes required for the Ring2All platform: ss-recordings (for call recordings), ss-uploads (for portal branding and uploads), and ss-music (for music on hold).

3.1 Create Brick Directories (All 3 Nodes)

Create directory paths where GlusterFS will write block files:

# Run on fs-node-01, fs-node-02, and fs-node-03:
mkdir -p /gluster/brick1/ss-recordings
mkdir -p /gluster/brick1/ss-uploads
mkdir -p /gluster/brick1/ss-music

[!TIP] Production Best Practice: In production, do not create bricks on the server OS root partition. Instead, attach a dedicated physical storage disk (e.g., /dev/sdb), format it with an XFS filesystem, and mount it to your brick path to guarantee I/O isolation.

3.2 Initialize the Volumes (fs-node-01 only)

Define the replicated volume structures:

# Initialize the recordings volume
gluster volume create ss-recordings replica 3 \
  fs-node-01:/gluster/brick1/ss-recordings \
  fs-node-02:/gluster/brick1/ss-recordings \
  fs-node-03:/gluster/brick1/ss-recordings \
  force

# Initialize the uploads volume
gluster volume create ss-uploads replica 3 \
  fs-node-01:/gluster/brick1/ss-uploads \
  fs-node-02:/gluster/brick1/ss-uploads \
  fs-node-03:/gluster/brick1/ss-uploads \
  force

# Initialize the music volume
gluster volume create ss-music replica 3 \
  fs-node-01:/gluster/brick1/ss-music \
  fs-node-02:/gluster/brick1/ss-music \
  fs-node-03:/gluster/brick1/ss-music \
  force

(We use the force flag here because we are writing to the root filesystem block loop).

3.3 Start the Replicated Volumes

gluster volume start ss-recordings
gluster volume start ss-uploads
gluster volume start ss-music

Verify that all three volumes are started:

gluster volume info

---

🔌 Step 4: Client Configuration (Mounting & High Availability)

To access the distributed storage pool, telephony and application servers must mount the corresponding volumes locally.

4.1 Install the Native FUSE Client

Run the following command on all client servers (such as Telephony Server or API nodes):

apt install -y glusterfs-client

4.2 Create Target Mount Points

Create the directories matching each node type’s expected layout:

# On Telephony Nodes (Telephony Server):
mkdir -p /var/lib/freeswitch/recordings
mkdir -p /usr/share/freeswitch/sounds/music

# On the Admin API Node:
mkdir -p /var/www/softswitch/uploads

4.3 Configure Automount in /etc/fstab with Failover

To ensure the storage mounts survive server restarts and automatically handle node failures, configure the /etc/fstab table using the backup-volfile-servers flag.

If the primary node fs-node-01 goes offline, the client will immediately switch to fs-node-02 or fs-node-03 to keep reads/writes alive without interrupting active services:

# On Telephony Nodes (Telephony Server):
cat << 'EOF' >> /etc/fstab
fs-node-01:/ss-recordings /var/lib/freeswitch/recordings glusterfs defaults,_netdev,backup-volfile-servers=fs-node-02:fs-node-03 0 0
fs-node-01:/ss-music       /usr/share/freeswitch/sounds/music         glusterfs defaults,_netdev,backup-volfile-servers=fs-node-02:fs-node-03 0 0
EOF

# On the Admin API Node:
cat << 'EOF' >> /etc/fstab
fs-node-01:/ss-uploads /var/www/softswitch/uploads glusterfs defaults,_netdev,backup-volfile-servers=fs-node-02:fs-node-03 0 0
EOF

4.4 Mount and Verify

Trigger the mounting process and inspect the active storage:

mount -a
df -h | grep -E 'recordings|music|uploads'

4.5 Alternative: NAS / NFS Mount Option

If your organization prefers using a pre-existing physical NAS appliance instead of building a software cluster, configure NFS client mounting on your nodes:

# Install NFS client
apt install -y nfs-common

# Create mount points
mkdir -p /var/lib/freeswitch/recordings
mkdir -p /usr/share/freeswitch/sounds/music
mkdir -p /var/www/softswitch/uploads

# Add configuration to fstab
cat << 'EOF' >> /etc/fstab
nas.example.com:/recordings /var/lib/freeswitch/recordings nfs defaults,_netdev 0 0
nas.example.com:/music /usr/share/freeswitch/sounds/music         nfs defaults,_netdev 0 0
nas.example.com:/uploads /var/www/softswitch/uploads nfs defaults,_netdev 0 0
EOF

# Mount
mount -a

[!WARNING] Single Point of Failure (SPOF): While simpler to deploy, a single NFS NAS represents a single point of failure. Ensure your NAS architecture features redundant controllers and multi-path networking for production.

---

🔄 Step 5: Advanced Operations & Health Checks

Self-Healing Synchronization

If a storage node crashes and is later restored, the cluster repairs block differences automatically in the background. To monitor progress, run:

gluster volume heal gv_recordings info

If the output displays 0 entries, your distributed data is fully synchronized.

---

🏁 Series Wrap-Up

Congratulations! You have completed the Debian 13 Clustering & Distribution guide series.

We have successfully covered:

1. Part 1: Setting up a secure, signed public APT repository.
2. Part 2: Compiling and packaging Telephony Server from source.
3. Part 3: Packaging Kamailio and deploying it securely.
4. Part 4: Implementing a self-healing PostgreSQL cluster using Patroni and Etcd consensus.
5. Part 5: Building a replicated storage layer using GlusterFS for media file distribution.

Your infrastructure is now secure, modular, highly available, and ready for enterprise scale.