Performance drop in iozone for reading large files

[Bryce Harrington]

There is a performance issue in READ operations we've noticed in iozone
runs for a while now.  If you look at this plot, of NFSv4 with 1G mem:

   http://crucible.osdl.org/runs/1604/test_output/iozone.sys.log.png

You can see that we're getting fairly steady performance up until the
1048576kb case, where read performance drops to about a third.

Is anyone else experiencing similar effects?

Testing against NFSv3 shows similar behavior. 

Also, from our tests it appears to be memory-related.  We booted with
less memory and see the issue occuring for lower file sizes:

 NFSv3 w/ 256M mem:
    http://crucible.osdl.org/runs/1607/test_output/iozone.nfsv3.log.png
    -> Drops off at the 131072k file size

 NFSv3 w/ 512M mem:
    http://crucible.osdl.org/runs/1612/test_output/iozone.nfsv3.log.png
    -> Drops off at the 262144k file size

On our 64-bit x86 systems with 4G RAM, we are not seeing the issue,
presumably due to the larger memory size.

 NFSv4 w/ 4G mem:
    http://crucible.osdl.org/runs/1511/test_output/iozone.sys.log.png
    -> No drop off

This issue has been present for a while now, but other issues had hidden
it in recent iozone runs.

Does anyone have an idea why the dropoff is occuring?  Is the memory
size expected to affect NFS performance this way?  Is anyone else
experiencing similar effects?

Thanks,
Bryce