NfsRdmaClient/MemRegModes
From Linux NFS
Chucklever (Talk | contribs) (Clarify a MTHCAFMR limitation) |
Chucklever (Talk | contribs) (→Long-term Plans) |
||
Line 235: | Line 235: | ||
|- | |- | ||
! Mode | ! Mode | ||
+ | ! Safe | ||
+ | ! Fast | ||
! Long-term | ! Long-term | ||
! Comments | ! Comments | ||
|- | |- | ||
| RPCRDMA_BOUNCEBUFFERS | | RPCRDMA_BOUNCEBUFFERS | ||
+ | | yes | ||
+ | | no | ||
| Deprecate then remove | | Deprecate then remove | ||
| Never intended for production | | Never intended for production | ||
|- | |- | ||
| RPCRDMA_REGISTER | | RPCRDMA_REGISTER | ||
+ | | yes | ||
+ | | no | ||
| Deprecate then remove | | Deprecate then remove | ||
| Safe, but not fast | | Safe, but not fast | ||
|- | |- | ||
| RPCRDMA_MEMWINDOWS | | RPCRDMA_MEMWINDOWS | ||
+ | | yes | ||
+ | | no | ||
| Deprecate then remove | | Deprecate then remove | ||
| No value to keeping this | | No value to keeping this | ||
|- | |- | ||
| RPCRDMA_MEMWINDOWS_ASYNC | | RPCRDMA_MEMWINDOWS_ASYNC | ||
+ | | no | ||
+ | | yes | ||
| Deprecate then remove | | Deprecate then remove | ||
| Generally unsafe | | Generally unsafe | ||
|- | |- | ||
| RPCRDMA_MTHCAFMR | | RPCRDMA_MTHCAFMR | ||
+ | | no | ||
+ | | yes | ||
| Keep | | Keep | ||
| Known user; RDMA protection on page boundaries only | | Known user; RDMA protection on page boundaries only | ||
|- | |- | ||
| FRMR | | FRMR | ||
+ | | yes | ||
+ | | yes | ||
| Keep | | Keep | ||
| Generally fast and safe, should remain our default | | Generally fast and safe, should remain our default | ||
|- | |- | ||
| ALLPHYSICAL | | ALLPHYSICAL | ||
+ | | no | ||
+ | | yes | ||
| Keep | | Keep | ||
| Known user | | Known user | ||
|- | |- | ||
|} | |} |
Revision as of 01:32, 4 March 2014
Contents |
Simplifying NFS/RDMA Client Memory Registration Modes
The RPC/RDMA transport uses a variety of memory registration strategies to mark areas of the NFS client's memory that are eligible for RDMA. At mount time, one registration mode is chosen based on the provider and HCA used to connect to the NFS server. By default, RPC/RDMA attempts to use FRMR because it is fast and generally safe.
However, not all providers and HCAs support FRMR. Thus the RPC/RDMA client transport code maintains 7 different memory registration modes, which introduces a lot of complexity to the code base and our test matrix. Over time we would like to remove some of these modes to simplify the code base and testing requirements.
Matrix of Registration Mode Support
The following table is based on a code audit of RDMA providers in the 3.13 kernel and one (the Soft RoCE driver, rxe) that is only in OFED.
memreg mode/provider | BOUNCEBUFFERS | REGISTER (reg_phys_mr verb present) | MEMWINDOWS (device mem_window flag set) | MEMWINDOWS_ASYNC (device mem_window flag set) | MTHCAFMR (alloc_fmr verb present) | FRMR (device mem_mgt_ext and loc_dma_key flags set) | ALLPHYSICAL |
---|---|---|---|---|---|---|---|
amso1100 | yes | yes | yes | yes | no | no | yes |
cxgb3 | yes | yes | yes | yes | no | yes | yes |
cxgb4 | yes | yes | yes | yes | no | yes | yes |
ehca | yes | yes | no | no | yes? | no | yes |
ipath | yes | yes | no | no | yes | no | yes |
mlx4 | yes | no | yes | yes | yes | yes | yes |
mlx5 | yes | no | no | no | no | yes | yes |
mthca | yes | yes | no | no | yes | no | yes |
nes | yes | yes | yes | yes | no | yes | yes |
ocrdma | yes | yes | no | no | no | yes | yes |
qib | yes | yes | no | no | yes | no | yes |
rxe | yes | yes | no | no | yes | no | yes |
Summary of memory registration modes
From Tom Talpey.
Registration mode | Pros | Cons |
---|---|---|
RPCRDMA_BOUNCEBUFFERS |
|
|
RPCRDMA_REGISTER |
|
|
RPCRDMA_MEMWINDOWS |
|
|
RPCRDMA_MEMWINDOWS_ASYNC |
|
|
RPCRDMA_MTHCAFMR |
|
Strongly suggest not relying on mthca except for purpose of supporting antique mellanox adapters |
RPCRDMA_FRMR |
|
|
RPCRDMA_ALLPHYSICAL |
|
|
Long-term Plans
After some discussion on linux-rdma and linux-nfs, we might be able to justify the following changes.
Mode | Safe | Fast | Long-term | Comments |
---|---|---|---|---|
RPCRDMA_BOUNCEBUFFERS | yes | no | Deprecate then remove | Never intended for production |
RPCRDMA_REGISTER | yes | no | Deprecate then remove | Safe, but not fast |
RPCRDMA_MEMWINDOWS | yes | no | Deprecate then remove | No value to keeping this |
RPCRDMA_MEMWINDOWS_ASYNC | no | yes | Deprecate then remove | Generally unsafe |
RPCRDMA_MTHCAFMR | no | yes | Keep | Known user; RDMA protection on page boundaries only |
FRMR | yes | yes | Keep | Generally fast and safe, should remain our default |
ALLPHYSICAL | no | yes | Keep | Known user |