Testing forecast roadmap
From Linux NFS
(→Milestone 0: Developer Testing Support) |
(→Milestone 4: Enterprise Readiness) |
||
(5 intermediate revisions not shown) | |||
Line 21: | Line 21: | ||
Functionality: | Functionality: | ||
- | I.B.1 [ ] [OSDL] Automate running applicable existing functionality | + | I.B.1 [X] [OSDL] Automate running applicable existing functionality |
tests on codebase periodically: cthon, pynfs, | tests on codebase periodically: cthon, pynfs, | ||
iozone | iozone | ||
Line 49: | Line 49: | ||
IV.A.2 [ ] [Bull] Time to perform sequence of cacheable read/write | IV.A.2 [ ] [Bull] Time to perform sequence of cacheable read/write | ||
operations | operations | ||
- | IV.F.4 [ ] [ | + | IV.F.4 [ ] [Bull] Measure memory/network/CPU efficiency of client for |
fixed workload | fixed workload | ||
IV.G.3 [ ] [Bull] Measure effects of scaling up number of | IV.G.3 [ ] [Bull] Measure effects of scaling up number of | ||
- | connections. Measure number of mounts per second | + | connections with exportfs/mount. Measure number of |
- | + | mounts per second on client and server. | |
IV.J.2 [ ] [Bull] Measure performance when operating with Kerberos 5 | IV.J.2 [ ] [Bull] Measure performance when operating with Kerberos 5 | ||
integrity and privacy | integrity and privacy | ||
Line 68: | Line 68: | ||
Misc: | Misc: | ||
- | * [ ] [OSDL] Establish bug tracking process for NFSv4 | + | * [X] [OSDL] Establish bug tracking process for NFSv4 |
==Milestone 1: Distro Quality Assurance== | ==Milestone 1: Distro Quality Assurance== | ||
Line 92: | Line 92: | ||
II.E.2 [ ] [ ] Verify features of the Ext3 file system work under | II.E.2 [ ] [ ] Verify features of the Ext3 file system work under | ||
NFSv4 | NFSv4 | ||
- | III.A.1 [ ] [ | + | III.A.1 [ ] [Bull] Run iozone for 2 wks on basic client/server |
operations | operations | ||
III.A.2 [ ] [ ] Run automounter use case for 2 wks on amd, autofs, | III.A.2 [ ] [ ] Run automounter use case for 2 wks on amd, autofs, | ||
Line 106: | Line 106: | ||
V.E.1 [ ] [ ] Ensure a functionality test sufficiently tests | V.E.1 [ ] [ ] Ensure a functionality test sufficiently tests | ||
Authentication/ACL | Authentication/ACL | ||
- | |||
==Milestone 2: Drill Down for One Use Case== | ==Milestone 2: Drill Down for One Use Case== | ||
Line 128: | Line 127: | ||
IV.E.1 [ ] [Bull] Test performance with large numbers of small (<4k) | IV.E.1 [ ] [Bull] Test performance with large numbers of small (<4k) | ||
files | files | ||
- | IV.E.2 [ ] [ | + | IV.E.2 [ ] [Bull] Test performance with a few very large (>1G) files |
IV.F.4 [ ] [Bull] Measure memory/network/CPU efficiency of client | IV.F.4 [ ] [Bull] Measure memory/network/CPU efficiency of client | ||
for fixed workload | for fixed workload | ||
Line 198: | Line 197: | ||
III.D.1 [ ] [ ] Find maximum number of connections to Linux IA-32 | III.D.1 [ ] [ ] Find maximum number of connections to Linux IA-32 | ||
server | server | ||
- | III.D.4 [ ] [ | + | III.D.4 [ ] [Bull] Find maximum number of mounted file systems on |
client | client | ||
III.E.4 [ ] [ ] Test multiple clients using, locking, etc. same | III.E.4 [ ] [ ] Test multiple clients using, locking, etc. same | ||
Line 212: | Line 211: | ||
III.G.2 [ ] [ ] Test concurrent access tests for races in | III.G.2 [ ] [ ] Test concurrent access tests for races in | ||
automounter | automounter | ||
- | IV.G.4 [ ] [ | + | IV.G.4 [ ] [Bull] Measure effects of increasing number of files |
- | IV.G.5 [ ] [ | + | IV.G.5 [ ] [Bull] Measure effects of increasing file size |
(with/without cache) | (with/without cache) | ||
- | IV.G.6 [ ] [ | + | IV.G.6 [ ] [Bull] Measure effects when increasing size of |
on-the-wire NFS read or write operations | on-the-wire NFS read or write operations | ||
- | V.J.1 [ ] [ | + | V.J.1 [ ] [Bull] Identify security issues assuming attack from |
client-side | client-side |
Latest revision as of 22:52, 4 October 2005
Linux NFSv4 Testing Roadmap
As of May 2005, we've completed a detailed itemization of testing work necessary for NFSv4, in a spreadsheet "Test Matrix", identifying items as High, Medium, and Low.
This document seeks to divide the high priority work into achievable chunks that we as a community can strive to attain.
I tried to prioritize for things that look reasonably easy to complete in the near term, that have owners, and/or are most likely to reveal issues that developers want to know about.
Contents |
Milestone 0: Developer Testing Support
These testing tasks seek to put into place tools and practices that will help support development activities (including testing) on an ongoing basis.
Functionality:
I.B.1 [X] [OSDL] Automate running applicable existing functionality tests on codebase periodically: cthon, pynfs, iozone I.B.3 [X] [OSDL] Automate cross-compile testing for major platforms on each NFSv4 patchset I.C.2 [ ] [ ] Test install on Fedora Core of NFSv4 server & client, krb5, ldap, et al I.E.9 [ ] [ ] Verify ability to force operations (close files, change states, unmount) I.G.5 [ ] [Bull] Test compilation and functionality on IA-64 I.G.7 [ ] [Bull] Test compilation and functionality on PPC-64 I.K.2 [ ] [ ] Check that NFS HOWTO is updated with sufficient nfsv4 info
Interop:
II.A.1 [ ] [Bull] Kerberos (MIT) - verify basic functionality. Mount w/ krb5, etc.
Robustness:
III.A.3 [ ] [OSDL] Run NFS server for 2 wks with random configuration changes III.C.1 [ ] [Bull] Run LTP NFS fstress in a std config on each release
Performance:
IV.A.1 [ ] [Bull] Time to perform sequence of unique read/write operations IV.A.2 [ ] [Bull] Time to perform sequence of cacheable read/write operations IV.F.4 [ ] [Bull] Measure memory/network/CPU efficiency of client for fixed workload IV.G.3 [ ] [Bull] Measure effects of scaling up number of connections with exportfs/mount. Measure number of mounts per second on client and server. IV.J.2 [ ] [Bull] Measure performance when operating with Kerberos 5 integrity and privacy
Security:
V.A.6 [ ] [ ] Audit the mountd code V.B.4 [ ] [OSDL] Run sparse as regression test periodically V.L.1 [ ] [ ] Ensure there is high level design documentation of NFSv4 security V.L.2 [ ] [ ] Ensure there is inline documentation for security related code in kernel V.L.3 [ ] [ ] Ensure there is inline documentation for GSS API (libgssapi, librpcsecgss)
Misc:
* [X] [OSDL] Establish bug tracking process for NFSv4
Milestone 1: Distro Quality Assurance
This milestone strives to attain a "Top Ten" list of testing that is felt will emphasize that NFSv4 is stable enough to be included in all enterprise distros. The focus is robustness, and items that would cause distros the most headaches if they're busted.
I.A.5 [ ] [ ] Protocol implementation interoperability between Linux server and client I.C.5 [ ] [ ] Test install on SLES of NFSv4 server & client, krb5, ldap, et al I.C.6 [ ] [ ] Test install on RHEL of NFSv4 server & client, krb5, ldap, et al I.H.1 [ ] [ ] Verify compatibility with glibc I.H.9 [ ] [ IBM] Verify compatibility with automounter I.H.11 [ ] [ ] Verify compatibility with basic system tools (file utils, core utils, util-linux, mount, sar, iostat, etc.) I.K.4 [ ] [ ] Check that Network Admin Guide at tldp.org is updated for nfsv4 II.E.2 [ ] [ ] Verify features of the Ext3 file system work under NFSv4 III.A.1 [ ] [Bull] Run iozone for 2 wks on basic client/server operations III.A.2 [ ] [ ] Run automounter use case for 2 wks on amd, autofs, and autong III.A.4 [ ] [ ] Run connectathon locking tests against NFS server for 2 weeks III.E.1 [ ] [OSDL] Test short & long term local network failure (unplugged cable, ifdown eth0, etc.) III.E.2 [ ] [OSDL] Test short & long duration remote network partition III.E.3 [ ] [OSDL] Test behavior during crash/reboot of server with clients holding various states V.D.1 [ ] [ ] Review Authentication/ACL feature design V.E.1 [ ] [ ] Ensure a functionality test sufficiently tests Authentication/ACL
Milestone 2: Drill Down for One Use Case
This milestone strives to push NFSv4 through one complete use case, to give it some thorough exercise and scaling, and to establish a "case study" we can use as an example of NFSv4's value.
I.E.5 [ ] [ ] Verify sufficient monitoring functionality within Ethereal I.F.1 [ ] [ ] Client notification to server of locking, write, read, etc. I.P.1 [ ] [OSDL] Database functionality on NFS I.L.1 [ ] [ ] Test compatibility with TCP protocol IV.A.7 [ ] [ ] NFSv3 vs. v4: Metadata - open/close intensive workload IV.A.8 [ ] [ ] NFSv3 vs. v4: Metadata - directory scanning IV.A.9 [ ] [ ] NFSv3 vs. v4: Metadata - create/delete IV.A.10 [ ] [ ] NFSv3 vs. v4: Metadata - changing attributes (chown, chmod) while dir scanning IV.E.1 [ ] [Bull] Test performance with large numbers of small (<4k) files IV.E.2 [ ] [Bull] Test performance with a few very large (>1G) files IV.F.4 [ ] [Bull] Measure memory/network/CPU efficiency of client for fixed workload IV.G.1 [ ] [ ] Verify server scalability with clients generating various basic requests (ACCESS, GETATTR, et al) IV.G.2 [ ] [ ] Verify server scalability with clients using compound requests V.D.6 [ ] [ ] Penetration testing for client callback implementation V.E.6 [ ] [ ] Ensure a functionality test sufficiently covers penetration testing for client callback implementation
Milestone 3: Enterprise Usability
This milestone focuses on testing that verifies "basic" usability of NFSv4 in data centers. It strives to establish that NFSv4 is able to perform basic functions in enterprise-class deployments.
I.F.2 [ ] [ ] Reboot recovery I.F.4 [ ] [ ] Open with shares / deny I.F.5 [ ] [ ] Bumping a sequence ID I.F.6 [ ] [ ] Network partition recovery I.F.8 [ ] [ ] Locking (Non-blocking locks) I.H.3 [ ] [ ] Verify compatibility with Kerberos I.H.4 [ ] [ ] Verify compatibility with Ipsec I.H.5 [ ] [ ] Verify compatibility with POSIX ACLs I.H.6 [ ] [ ] Verify compatibility with NFS ACLs II.B.1 [ ] [Bull] Interoperability between 32-bit and 64-bit for client and server: IA32/AIX-PPC, IA32/Linux-PPC II.B.2 [ ] [Bull] Interoperability between little endian and big endian II.C.1 [ ] [Bull] Interoperability for Linux IA-32 client ? Solaris 10 server II.C.4 [ ] [Bull] Interoperability for Linux IA-32 client ? AIX 5.3 server III.D.7 [ ] [ ] Test correctness of NFS client when backed by a large (>100GB) cachefs IV.E.5 [ ] [ ] NFS "Cluster" scenario with 1000+ clients and several servers V.A.7 [ ] [ ] Audit the RPC authentication code (gssd, authsys, etc.) V.D.2 [ ] [ ] Review each security flavor feature design: Krb5, Spkm V.E.2 [ ] [ ] Ensure a functionality test sufficiently covers each security flavor
Milestone 4: Enterprise Readiness
This milestone builds on the previous one by ensuring that NFSv4 will behave well and perform robustly under typical enterprise-class loads.
III.B.2 [ ] [ ] Test stability of client in out of memory situation III.B.3 [ ] [ ] Test stability of client in out of disk space on server situation III.B.4 [ ] [ ] Test stability of client in out of inode situation III.B.5 [ ] [ ] Test stability of client in out of swap space situation III.B.6 [ ] [ ] Test stability of server in out of pid situation III.B.7 [ ] [ ] Test stability of server in out of memory situation III.B.8 [ ] [ ] Test stability of server in out of disk space situation III.B.9 [ ] [ ] Test stability of server in out of inode situation III.B.10 [ ] [ ] Test stability of server in out of swap space situation III.D.1 [ ] [ ] Find maximum number of connections to Linux IA-32 server III.D.4 [ ] [Bull] Find maximum number of mounted file systems on client III.E.4 [ ] [ ] Test multiple clients using, locking, etc. same files III.E.5 [ ] [ ] Test behavior of server with failed storage device III.E.6 [ ] [ ] Test behavior during crash of client with open delegations and locks III.E.7 [ ] [ ] Test recovery from denied permission III.E.8 [ ] [ ] Test recovery from JUKEBOX/DELAY III.E.9 [ ] [ ] Test recovery from ESTALE III.E.10 [ ] [ ] Test server callback mechanism (c.f. III.E.1, III.E.2) III.G.2 [ ] [ ] Test concurrent access tests for races in automounter IV.G.4 [ ] [Bull] Measure effects of increasing number of files IV.G.5 [ ] [Bull] Measure effects of increasing file size (with/without cache) IV.G.6 [ ] [Bull] Measure effects when increasing size of on-the-wire NFS read or write operations V.J.1 [ ] [Bull] Identify security issues assuming attack from client-side