[nfsv4] Performance drop in iozone for reading large files

[Benny Halevy]

Assuming the graphs plot iozone read and write (vs. re-read and re-write)
you should always go to disk first time you read the file so the performance
drop is more likely to be related to readahead, not to the mere size of
your cache.

Benny

Dean Hildebrand wrote:
> So your server only has 1 GB of RAM in the original case?  If your
> server only has 1 GB of RAM, then reading a file greater than 1 GB will
> definitely reduce performance as the server is now going to disk.  Since
> the server operating system also requires memory, it would be normal to
> see a performance drop off much earlier, maybe around 600-800 MB.
>
> Does this make sense, or do the original clients/servers have more than
> 1 GB of RAM?
> Dean
>
> Bryce Harrington wrote:
>> The client and server for these tests are identical hardware, and we're
>> booting them with the same kernel args, so we reduced the RAM the same
>> on both client and server.
>>
>> We didn't try a 4GB file on the 4GB ram system yet.  I suppose we could
>> also try reducing the memory on that system to <1GB and see what 
>> happens.
>>
>> Bryce
>>
>> On Fri, Aug 25, 2006 at 07:15:10PM -0400, Dean Hildebrand wrote:
>>  
>>> Just to clarify, are you referring to client memory below?  Does the 
>>> amount of server memory affect the situation?  Did you try a 4GB 
>>> file on the 4GB ram system?
>>>
>>> Dean
>>>
>>> Bryce Harrington wrote:
>>>    
>>>> There is a performance issue in READ operations we've noticed in 
>>>> iozone
>>>> runs for a while now.  If you look at this plot, of NFSv4 with 1G mem:
>>>>
>>>>   http://crucible.osdl.org/runs/1604/test_output/iozone.sys.log.png
>>>>
>>>> You can see that we're getting fairly steady performance up until the
>>>> 1048576kb case, where read performance drops to about a third.
>>>>
>>>> Is anyone else experiencing similar effects?
>>>>
>>>> Testing against NFSv3 shows similar behavior.
>>>> Also, from our tests it appears to be memory-related.  We booted with
>>>> less memory and see the issue occuring for lower file sizes:
>>>>
>>>> NFSv3 w/ 256M mem:
>>>>    http://crucible.osdl.org/runs/1607/test_output/iozone.nfsv3.log.png
>>>>    -> Drops off at the 131072k file size
>>>>
>>>> NFSv3 w/ 512M mem:
>>>>    http://crucible.osdl.org/runs/1612/test_output/iozone.nfsv3.log.png
>>>>    -> Drops off at the 262144k file size
>>>>
>>>> On our 64-bit x86 systems with 4G RAM, we are not seeing the issue,
>>>> presumably due to the larger memory size.
>>>>
>>>> NFSv4 w/ 4G mem:
>>>>    http://crucible.osdl.org/runs/1511/test_output/iozone.sys.log.png
>>>>    -> No drop off
>>>>
>>>> This issue has been present for a while now, but other issues had 
>>>> hidden
>>>> it in recent iozone runs.
>>>>
>>>> Does anyone have an idea why the dropoff is occuring?  Is the memory
>>>> size expected to affect NFS performance this way?  Is anyone else
>>>> experiencing similar effects?
>>>>
>>>> Thanks,
>>>> Bryce
>>>> _______________________________________________
>>>> NFSv4 mailing list
>>>> NFSv4 at linux-nfs.org
>>>> http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4
>>>>  
>>>>       
>>> -- 
>>> Dean Hildebrand
>>> Ph.D. Candidate
>>> University of Michigan
>>>     
>