[nfsv4] Performance drop in iozone for reading large files

[Bryce Harrington]

On Fri, Aug 25, 2006 at 05:13:23PM -0400, J. Bruce Fields wrote:
> On Fri, Aug 25, 2006 at 02:01:22PM -0700, Bryce Harrington wrote:
> > Does anyone have an idea why the dropoff is occuring?  Is the memory
> > size expected to affect NFS performance this way?
> 
> Yes, definitely; it can answer the reads out of the client's cache, as
> long as the file is small enough to fit in the client's memory.
> 
> (What exactly is it testing in this case?  I don't really know what
> iozone does.  Bad me.)

For each file size and record length, it does the following:

  * Write a file of that size to the server
  * Re-write it
  * Read a file of that size from the server
  * Re-read it

Here's two cycles from the raw log:

              KB  reclen   write rewrite    read    reread  
          131072     512   31048   33882    64771    65143
         1048576     512   31620   14321    18157    25666

The units for the first two columns is kB, the rest are kB/sec.

In the first case, with a file 10% the size of memory, write and
rewrite are nearly identical, as are read and reread.  

In the second case, where the file is about the size of available
memory, write performance is unaffected but rewrite performance drops by
over half (shouldn't it be going up, not down in this case if it can
cache?)  Read performance drops by more than two thirds, although we do
see some benefit (presumably from caching?) in the reread case.

As far as I understand from how iozone works, the read and write cases
should not be subject to caching, but I could be wrong...  We could ask
Don Capps for interpretation if it might help.

Bryce