Linux NFS client prioritized to-do list

This document outlines work in-progress and remaining work on the Linux kernel's NFS client. Related work on user-level tools and libraries and automounter support may be described in other documents.

Several tables below describe specific tasks we'd like to accomplish for the Linux NFS client. All tasks in one table are of the same priority (H, M, L), and are ordered within each table roughly by their priority relative to other tasks in the same table. Each task includes a description, an estimate of resources required (in developer months), and a list of interested parties. A set of test plans for each task will be provided later.

In addition, a number of tasks are captured under "unscheduled." These are controversial or low-priority issues that may or may not be promoted onto the of scheduled work. A table of "ongoing" tasks tracks housekeeping issues.

Finally, we include sections that describe medium-term issues and long term projects. Medium-term issues are tasks that may require more extensive changes or changes that involve several other parts of the kernel. These projects were originally targeted for the next development branch. Long term projects are important areas of work that are not ready to broken into specific tasks.

High priority tasks

Medium priority tasks

Low priority tasks

Ongoing tasks

Unscheduled tasks

Long-term projects

In this section we capture projects whose scope is broad enough that we don't yet have clearly specifiable tasks that can be laid out in a roadmap.

Linux NFS client quality assurance

The Bull Open Source group has begun assembling a list of NFSv4 tests. Much of the Linux NFSv4 implementation shares code with the version 2 and 3 support, so these tests will exercise and improve all versions of the NFS code in many cases.

However, there are still some things missing.

• We need an organized test strategy for using these tests. Some of these tests (such as connect-a-thon) are appropriate as "check-in" tests, but some important tests take a very long time to run (openly available application certification tests, or POSIX conformance testing, for example). We must be careful about choosing tests that exercise specific client issues, versus server tests which may not test the client. Finally we have to plan for the difference between testing the community source base versus testing distributions, which often add many patches, potentially changing behavior and performance.
• We need a commitment of hardware and human resources to carry out large-scale tests. Especially server scalability testing will require significant resources.
• Unit testing is necessarily absent from the list of tests provided by Bull. Developers must compose unit test plans as they add new features and fix bugs. Some developers may not have experience composing test plans, so some developer education is required.
• We must identify who will sponsor specific interoperability testing.
• Other tests are still missing: code coverage, sparse, destructive testing, security review, among others.

The open source community has a tradition of managing testing requirements differently than more product-oriented development agencies. Naturally we must be sensitive to the traditions and attitudes of the community as we introduce a greater level of rigorousness to software testing.

Improving customer support

In the following section, I refer to "customers" although this is an ill-defined term in the open source community. Customer support continues to be an issue not just for Linux distributors, but also for the open source community in general, and for commercial interests.

Linux NFS client customer support issues are changing in nature. As the Linux NFS client improves, there are many more customers who find that the NFS client "just works" when it is deployed in their environment. Basic implementation and interoperability issues have, for the most part, been resolved. On the other hand, the client is now stable enough to deploy in more sophisticated environments under considerably heavier workloads. Customer problem events are becoming fewer but each takes longer to resolve.

Here are the issues:

• As we've built support and engineering relationships with the major Linux distributors, we've managed to cut down the time it takes for fixes and features to get into the product source bases. There is still improvement that can be made, but we have climbed the largest hurdles here.
• Improvements in out-of-the-box configuration (easier, and less of it) are continuing. The latest clients default to using TCP and large rsize and wsize, meaning performance and stability are generally good by default and are less dependent on prevailing network conditions. The client now supports application programming interfaces to control some NFS behavior, so applications and administrators no longer need to depend on getting the mount options right.
• Helping customers help themselves is still a big problem.
  • Weak facilities for providing performance metric information
  • Poor tracing messages
  • Poor error messages
  • Recovery behavior is often untested (WAN simulation, destructive testing, and so on)
• The resources required for distributor pre-release testing and reproducing customer problems "in-house" are increasing.
• Auxiliary protocols and services (LDAP, NIS, automounter, NLM/NSM) are becoming more critical. Reliability and performance problems in these areas are exposed as the NFS client itself becomes more robust. We must bolster community efforts to improve these components similar to how we have improved the NFS client.
• Customers are demanding security features such as Kerberos, IPsec, and ACLs. These new features are complicated to configure and deploy, and in some cases, will remain unfinished for a long time. Here is where we will see a lot of customer interaction.
• Because Linux is appearing in the enterprise, demands for support for legacy versions of Linux (eg RH7.2, RHEL AS 2.1) will increase. The support lifetime of enterprise Linux products is five to seven years.
• A bug database will require some administrative effort, and discipline on the part of developers, to maintain. Some developers are unconvinced that having a database will be helpful in their everyday work, but will simply be busy work for them.
• Scaling the open source community's ability to handle problem reports will be critical as the difficulty of issues increases. The traditional mechanism of using a mailing list to ask questions of open source developers directly will become an issue both for problem tracking and developer efficiency.

Future protocol issues

There are several new directions for NFS version 4. Generally, advanced development on Linux NFS is funded by specific parties, rather than provided via traditional open source mechanisms. It is difficult to integrate some of these features into the Linux NFS client long before they are available in other implementations.

• Making the most of NFSv4 compound RPC: metadata read-ahead, large read and write compounds, and so on
• Parallel NFS
• NFSv4.x -- CCM, directory notifications, and so on
• Sessions (only supported by Linux and NetApp prototypes at the moment)
• Final integration of TOE and RDMA support
• Internationalization

NFS is being used more and more in cluster environments. There are a number of questions we have about how NFS will fit in with the increasing number of cluster file system solutions. First, what cluster semantics are desirable in NFS that it doesn't have? How can we improve on the traditional NFS client cache coherency model to meet the needs of clusters? How will pNFS fit in? What are the transport protocol implications for thousands of client nodes on a single LAN? What are the GRID's requirements -- security, WAN performance, etc? How well does NFS work on a large NUMA SMP client?

Advanced Transports

The networking world is certainly not standing still. Let's start considering some of these new transports and how we can integrate them into the NFS client. We also need to track user requirements for these new features and facilities.

• BIC-TCP: how useful is it for NFS?
• TOE: probably won't have network stack support; will need RDMA/DAPL
• IPv6, SCTP, and beyond: new network transports
• WAN simulation: need organization, testing facilities, and expertise
• 10GB Ethernet and IPoIB: need to begin planning resources for testing

Database on Linux NFS

There are several specific areas where we can improve the Linux NFS client for use with large-scale and clustered database applications. This effort is already ongoing, but includes:

• NFS O_DIRECT scalability (# of CPUs, processes, NICs, memory); more underlying parallelism; fewer operations on the wire
• Enhancing NFS support for aio
• NFS/RDMA
• SMP scalability, including spin lock removal
• More rigorous testing of pre-alpha kernels using database test suites
• Performance metrics
• Servicability enhancements
• Issues with umount hangs
• Make lock recovery seamless and automatic
• NFS client stability; more thorough destructive testing

Many of these items already appear above in our task list, but it is important to understand why they are there.

Last modified: Fri Mar 11 11:55:50 EDT 2005