How to Configure an NFS Matrix for Scalable Storage
Scalable storage using an NFS matrix (a coordinated set of NFS servers and export configurations designed for growth and performance) requires planning across architecture, storage backend, networking, authentication, tuning, and monitoring. This guide gives a prescriptive, step-by-step configuration you can apply for small-to-large deployments.
Assumptions & environment
- Linux servers (e.g., Ubuntu 22.04 / RHEL 8+) for NFS servers and clients.
- Storage backends: local disks, RAID, SAN (iSCSI/Fc) or clustered storage (Ceph, GlusterFS).
- Management network and dedicated storage network(s).
- Authentication via Kerberos (optional) or rootsquash/ID mapping for simpler setups.
- Goal: scale capacity and throughput while maintaining reliability and manageable metadata performance.
1) Design the NFS matrix architecture
-
Define roles:
- Metadata controllers (MDCs): handle namespace/exports and metadata-heavy operations (if using clustered storage).
- Data servers: serve file content.
- Load balancers / VIPs: present single endpoints to clients and distribute connections.
- Monitoring & config servers: run Prometheus/Grafana, logging, and config management (Ansible).
-
Choose topology:
- Small deployments: two NFS servers behind an active/passive VIP.
- Medium-to-large: multiple active NFS servers behind TCP/UDP load balancers with shared clustered storage (CephFS, Gluster) or replicated backends.
- High metadata load: cluster with dedicated metadata nodes (CephFS, Lustre).
-
Plan for scaling: capacity (add OSDs or bricks), throughput (more data servers), and availability (replication, erasure coding).
2) Prepare storage backends
- Local RAID or SAN: provision LVM volumes or filesystems (XFS recommended for NFS).
- Clustered backends: deploy and tune Ceph/Gluster/Lustre per vendor guides. Use SSDs for metadata/DB where supported.
- Filesystem settings: XFS with reflink off for high concurrency; tune inode sizes and noatime. Example mkfs.xfs flags:
Code
mkfs.xfs -f -m crc=1 -i size=512 /dev/sdX
3) Install and configure NFS server components
-
Install packages (Ubuntu):
Code
sudo apt update sudo apt install nfs-kernel-server rpcbind(RHEL/CentOS: nfs-utils.)
-
Enable services:
Code
sudo systemctl enable –now rpcbind nfs-server -
Export directories: create export mountpoints on your storage backend (e.g., /export/data). Add entries to /etc/exports with appropriate options:
- For performance and scalability:
- rw — read/write
- async — faster (accept only when safe for your use case)
- no_subtree_check — avoids performance penalties
- crossmnt — allow crossing mount points
- no_root_squash or rootsquash depending on trust model
- fsid=0 for pseudo-root in clustered setups Example:
Code
/export/data 10.0.0.0/24(rw,async,no_subtreecheck,crossmnt) - For performance and scalability:
-
Refresh exports:
Code
sudo exportfs -ra
4) Networking and load balancing
- Separate networks: isolate client, replication, and management traffic. Use jumbo frames on storage networks if supported.
- Load balancer / VIP: use HAProxy, keepalived, or dedicated LB to distribute client mounts across servers. Configure TCP (NFSv3 over TCP) session persistence based on client IP or use DNS round-robin for simple cases.
- Firewall rules: allow rpcbind (portmap), NFS ports (2049), mount
Leave a Reply