PNFS block server setup
From Linux NFS
| (One intermediate revision not shown) | |||
| Line 7: | Line 7: | ||
* a shared block device that the server and clients can all access. | * a shared block device that the server and clients can all access. | ||
| - | Format the block device with an xfs filesystem, export it with the "pnfs" export option, and start the nfs server. | + | Format the block device with an xfs filesystem, export it with the "pnfs" export option, and start the nfs server, and create an /sbin/nfsd-recall-failed as described in the second WARNING below. |
| - | On the client: start the blkmap daemon. (On fedora: systemctl enable nfs-blkmap and systemctl start nfs-blkmap). Then mount with nfs | + | On the client: start the blkmap daemon. (On fedora: systemctl enable nfs-blkmap and systemctl start nfs-blkmap). Then mount with nfs version at least 4.1. |
The client will then perform reads and writes to regular files by reading or writing directly to the block device rather than by sending NFS reads and writes to the server. Things are probably working if you can see LAYOUTGET calls in /proc/self/mountstats. | The client will then perform reads and writes to regular files by reading or writing directly to the block device rather than by sending NFS reads and writes to the server. Things are probably working if you can see LAYOUTGET calls in /proc/self/mountstats. | ||
Latest revision as of 21:54, 7 April 2015
A block-layout-based pNFS server was recently merged into the Linux kernel. It should be used carefully--see WARNINGS below.
To use it, you need:
- a kernel version at least 4.0 built with CONFIG_NFSD_PNFS,
- similarly recent nfs-utils on the server (not officially released as of this writing; need at least c08f1382e5609bc686c3df95ff1e267804b37a61 from steved's git tree), and
- a shared block device that the server and clients can all access.
Format the block device with an xfs filesystem, export it with the "pnfs" export option, and start the nfs server, and create an /sbin/nfsd-recall-failed as described in the second WARNING below.
On the client: start the blkmap daemon. (On fedora: systemctl enable nfs-blkmap and systemctl start nfs-blkmap). Then mount with nfs version at least 4.1.
The client will then perform reads and writes to regular files by reading or writing directly to the block device rather than by sending NFS reads and writes to the server. Things are probably working if you can see LAYOUTGET calls in /proc/self/mountstats.
WARNINGS:
- The client figures out which block device to write to by looking at the content of the block device. It may choose incorrectly, for example, if the client also has access to snapshots of the same filesystem. THIS MAY CORRUPT YOUR DATA.
- The server needs to be able to revoke client's direct access to data in cases where, for example, multiple clients need simultaneous access. If the client is unresponsive to requests to do this by normal NFS callbacks, the server must be able to forcibly fence the client's access. To make this work you must provide a /sbin/nfsd-recall-failed script which knows how to fence clients. See Documentation/filesystems/nfs/pnfs-block-server.txt for details. Failure to do this may again CORRUPT YOUR DATA.
