--- title: "📂 Part 5: Distributed Storage with GlusterFS on Debian 13 (Trixie)" description: "Documentation for 5. GlusterFS File Server" --- *Welcome to the fifth and final installment of our "Debian 13 Clustering & Distribution" series. In the previous guide, we deployed a self-healing PostgreSQL cluster. Now, we will establish a highly available, active-active distributed file storage system for telephony media assets (such as call recordings, voicemail logs, and system music). In this tutorial, we will configure GlusterFS (Replica 3) on Debian 13, build a trusted storage pool, create replication volumes, and configure target clients with automatic failover mounts.* --- ## 🏗️ Why Use Distributed Storage? In a multi-server PBX deployment, sharing files over a single server via standard NFS creates a Single Point of Failure (SPOF) and a severe I/O bottleneck. A distributed file system like **GlusterFS** solves this by offering: 1. **Redundancy (Replica 3)**: Media files are written simultaneously to three nodes. If a storage node goes offline, files remain fully accessible. 2. **Active-Active Access**: Client nodes (like Telephony application servers) can read and write to any storage node in the pool. 3. **Self-Healing**: If a failed node is restored, GlusterFS automatically synchronizes missing block differences. --- ## 🏛️ Storage Cluster Architecture Our cluster will use **3 dedicated storage nodes** to run a split-brain safe Replica 3 volume. | Role | Hostname | IP | Function | |------|----------|----|---------| | **File Server 1** | `fs-node-01` | `192.168.10.37` | GlusterFS Brick Node 1 | | **File Server 2** | `fs-node-02` | `192.168.10.38` | GlusterFS Brick Node 2 | | **File Server 3** | `fs-node-03` | `192.168.10.39` | GlusterFS Brick Node 3 | --- ## 🛠️ Step 1: Environment Preparation (All 3 Storage Nodes) Log into each storage node to perform the base installation. ### 1.1 Configure Hostnames Assign hostnames to distinguish nodes: ```bash # On fs-node-01 hostnamectl set-hostname fs-node-01 # On fs-node-02 hostnamectl set-hostname fs-node-02 # On fs-node-03 hostnamectl set-hostname fs-node-03 ``` ### 1.2 Resolution IP Mappings (`/etc/hosts`) Add this mapping block to `/etc/hosts` on **all nodes** (including your VoIP/FreeSWITCH clients): ```bash cat << 'EOF' >> /etc/hosts 192.168.10.37 fs-node-01 192.168.10.38 fs-node-02 192.168.10.39 fs-node-03 EOF ``` ### 1.3 Install GlusterFS Daemon Debian 13 includes GlusterFS Server in its native repositories. Run the following command on all storage nodes: ```bash apt update && apt upgrade -y apt install -y glusterfs-server ``` ### 1.4 Firewall Configuration (`nftables`) GlusterFS dynamically maps ports, but the main broker daemon listens on port **24007**. To ensure simple and secure communications, allow total access between storage cluster nodes, and restrict client mount traffic. Add these rules to `/etc/nftables.conf` on **all 3 nodes**: ```bash cat < /etc/nftables.conf #!/usr/sbin/nft -f flush ruleset table inet filter { chain input { type filter hook input priority 0; policy drop; iif "lo" accept ct state established,related accept tcp dport 22 accept ip protocol icmp accept # Internal Storage Traffic: Total access between cluster nodes ip saddr { 192.168.10.37, 192.168.10.38, 192.168.10.39 } accept # Client Mount Traffic: Allow VoIP subnet to access daemon and brick port ranges ip saddr { 192.168.10.0/24 } tcp dport { 24007, 24008, 49152-65535 } accept } chain forward { type filter hook forward priority 0; policy drop; } chain output { type filter hook output priority 0; policy accept; } } EOF systemctl start nftables systemctl enable nftables ``` --- ## 🤝 Step 2: Establish the Trusted Storage Pool Next, we link our cluster nodes together. ### 2.1 Start and Enable glusterd (All 3 Nodes) Start and enable the `glusterd` daemon across all three storage nodes before joining them to the pool: ```bash systemctl start glusterd systemctl enable glusterd systemctl status glusterd ``` ### 2.2 Probe Peers (fs-node-01 only) Run these peer commands **from `fs-node-01` ONLY** to register the other nodes: ```bash gluster peer probe fs-node-02 gluster peer probe fs-node-03 ``` Verify that both nodes are successfully connected: ```bash gluster peer status ``` *Expected Output:* ```text Number of Peers: 2 Hostname: fs-node-02 Uuid: 5e612cb7-8547-4ad3-a9d1-3e4b104928db State: Peer in Cluster (Connected) Hostname: fs-node-03 Uuid: 8cb019ef-b328-40ef-bb4b-cf109cde8e11 State: Peer in Cluster (Connected) ``` --- ## 🧱 Step 3: Create the Distributed Volumes With our storage pool established, we define the three replicated volumes required for the Ring2All platform: `ss-recordings` (for call recordings), `ss-uploads` (for portal branding and uploads), and `ss-music` (for music on hold). ### 3.1 Create Brick Directories (All 3 Nodes) Create directory paths where GlusterFS will write block files: ```bash # Run on fs-node-01, fs-node-02, and fs-node-03: mkdir -p /gluster/brick1/ss-recordings mkdir -p /gluster/brick1/ss-uploads mkdir -p /gluster/brick1/ss-music ``` > [!TIP] > **Production Best Practice**: In production, do not create bricks on the server OS root partition. Instead, attach a dedicated physical storage disk (e.g., `/dev/sdb`), format it with an XFS filesystem, and mount it to your brick path to guarantee I/O isolation. ### 3.2 Initialize the Volumes (fs-node-01 only) Define the replicated volume structures: ```bash # Initialize the recordings volume gluster volume create ss-recordings replica 3 \ fs-node-01:/gluster/brick1/ss-recordings \ fs-node-02:/gluster/brick1/ss-recordings \ fs-node-03:/gluster/brick1/ss-recordings \ force # Initialize the uploads volume gluster volume create ss-uploads replica 3 \ fs-node-01:/gluster/brick1/ss-uploads \ fs-node-02:/gluster/brick1/ss-uploads \ fs-node-03:/gluster/brick1/ss-uploads \ force # Initialize the music volume gluster volume create ss-music replica 3 \ fs-node-01:/gluster/brick1/ss-music \ fs-node-02:/gluster/brick1/ss-music \ fs-node-03:/gluster/brick1/ss-music \ force ``` *(We use the `force` flag here because we are writing to the root filesystem block loop).* ### 3.3 Start the Replicated Volumes ```bash gluster volume start ss-recordings gluster volume start ss-uploads gluster volume start ss-music ``` Verify that all three volumes are started: ```bash gluster volume info ``` --- ## 🔌 Step 4: Client Configuration (Mounting & High Availability) To access the distributed storage pool, telephony and application servers must mount the corresponding volumes locally. ### 4.1 Install the Native FUSE Client Run the following command on all client servers (such as Telephony Server or API nodes): ```bash apt install -y glusterfs-client ``` ### 4.2 Create Target Mount Points Create the directories matching each node type's expected layout: ```bash # On Telephony Nodes (Telephony Server): mkdir -p /var/lib/freeswitch/recordings mkdir -p /usr/share/freeswitch/sounds/music # On the Admin API Node: mkdir -p /var/www/softswitch/uploads ``` ### 4.3 Configure Automount in `/etc/fstab` with Failover To ensure the storage mounts survive server restarts and automatically handle node failures, configure the `/etc/fstab` table using the `backup-volfile-servers` flag. If the primary node `fs-node-01` goes offline, the client will immediately switch to `fs-node-02` or `fs-node-03` to keep reads/writes alive without interrupting active services: ```bash # On Telephony Nodes (Telephony Server): cat << 'EOF' >> /etc/fstab fs-node-01:/ss-recordings /var/lib/freeswitch/recordings glusterfs defaults,_netdev,backup-volfile-servers=fs-node-02:fs-node-03 0 0 fs-node-01:/ss-music /usr/share/freeswitch/sounds/music glusterfs defaults,_netdev,backup-volfile-servers=fs-node-02:fs-node-03 0 0 EOF # On the Admin API Node: cat << 'EOF' >> /etc/fstab fs-node-01:/ss-uploads /var/www/softswitch/uploads glusterfs defaults,_netdev,backup-volfile-servers=fs-node-02:fs-node-03 0 0 EOF ``` ### 4.4 Mount and Verify Trigger the mounting process and inspect the active storage: ```bash mount -a df -h | grep -E 'recordings|music|uploads' ``` ### 4.5 Alternative: NAS / NFS Mount Option If your organization prefers using a pre-existing physical NAS appliance instead of building a software cluster, configure NFS client mounting on your nodes: ```bash # Install NFS client apt install -y nfs-common # Create mount points mkdir -p /var/lib/freeswitch/recordings mkdir -p /usr/share/freeswitch/sounds/music mkdir -p /var/www/softswitch/uploads # Add configuration to fstab cat << 'EOF' >> /etc/fstab nas.example.com:/recordings /var/lib/freeswitch/recordings nfs defaults,_netdev 0 0 nas.example.com:/music /usr/share/freeswitch/sounds/music nfs defaults,_netdev 0 0 nas.example.com:/uploads /var/www/softswitch/uploads nfs defaults,_netdev 0 0 EOF # Mount mount -a ``` > [!WARNING] > **Single Point of Failure (SPOF)**: While simpler to deploy, a single NFS NAS represents a single point of failure. Ensure your NAS architecture features redundant controllers and multi-path networking for production. --- ## 🔄 Step 5: Advanced Operations & Health Checks ### Self-Healing Synchronization If a storage node crashes and is later restored, the cluster repairs block differences automatically in the background. To monitor progress, run: ```bash gluster volume heal gv_recordings info ``` If the output displays `0` entries, your distributed data is fully synchronized. --- ## 🏁 Series Wrap-Up Congratulations! You have completed the **Debian 13 Clustering & Distribution** guide series. We have successfully covered: 1. **Part 1**: Setting up a secure, signed public APT repository. 2. **Part 2**: Compiling and packaging Telephony Server from source. 3. **Part 3**: Packaging Kamailio and deploying it securely. 4. **Part 4**: Implementing a self-healing PostgreSQL cluster using Patroni and Etcd consensus. 5. **Part 5**: Building a replicated storage layer using GlusterFS for media file distribution. Your infrastructure is now secure, modular, highly available, and ready for enterprise scale.