[nfsv4] Performance drop in iozone for reading large files
Le Rouzic
aime.le-rouzic at bull.net
Thu Sep 7 10:24:40 EDT 2006
nfsv4-request at linux-nfs.org a écrit :
>
> 4. Re: [nfsv4] Performance drop in iozone for reading large
> files (Dean Hildebrand)
>
>------------------------------
>
>Message: 4
>Date: Wed, 06 Sep 2006 10:59:48 -0400
>From: Dean Hildebrand <dhildebz at eecs.umich.edu>
>Subject: Re: [nfsv4] Performance drop in iozone for reading large
> files
>To: Trond Myklebust <trond.myklebust at fys.uio.no>
>Cc: nfsv4 at linux-nfs.org
>Message-ID: <44FEE264.7080204 at eecs.umich.edu>
>Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
>To prove/disprove the sever caching theory, here are a couple *echm* fun
>things you can try:
>1) While writing and reading the file, watch the memory on the server
>using 'top'. You should see it rise up an amount roughly equal to the
>size of the file. Once the file is deleted by iozone, the memory will
>drop back down again.
>2) After writing the file do not delete it (use -w I believe). Umount
>the server, reboot the server, then mount the server and read the file.
>This will read the file directly from disk. Compare this performance to
>your original read performance (which used the server cache).
>
>Dean
>
>
Hi,
Running the last linux-2.6.18-rc5-CITI_NFS4_ALL-1.diff between two Intel
Xeon
as client and server with 2GB RAM each , I get the same drop off when
processing a 2GB file.
./iozone -ace -r 64 -g 2G -i 0 -i 1 -f /mnt/nosec/file1 -U /mnt/nosec
Using vmstat on the server we can see there is no io/reads until
the case of the 2GB file is running.
Here is the vmstat result when the 2GB file case is running:
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
0 9 120 93916 6992 1617480 0 0 0 44524 18092 520 0
27 0 73
2 9 120 51616 6256 1658580 0 0 0 48850 15083 463 0
23 0 77
4 9 120 49604 6272 1658856 0 0 0 41764 16932 553 0
26 0 74
0 9 120 49712 6280 1657472 0 0 0 47084 17302 597 0
28 0 72
2 9 120 51800 6296 1655580 0 0 0 39104 20281 598 0
32 0 67
0 9 120 51860 6460 1659676 0 0 2 44170 38985 1227 0
67 3 30
0 9 120 51448 6480 1659160 0 0 0 42612 30544 748 0
46 0 54
0 8 120 51700 6496 1658564 0 0 0 46276 13790 525 0
23 0 77
0 9 120 51700 6496 1658912 0 0 20 42512 642 190 0
2 0 98
0 9 120 51776 6496 1659740 0 0 1150 37336 1481 772 0
3 0 97
0 9 120 57804 6496 1659520 0 0 630 31330 1150 496 0
4 0 96
0 3 120 57696 6504 1660608 0 0 590 54126 945 417 0 2
19 79
0 6 120 51976 6504 1667268 0 0 3296 28006 3563 1829 0
5 0 95
0 2 120 53728 6512 1670672 0 0 1684 7832 2012 994 0 4
22 74
0 2 120 52920 6520 1672068 0 0 8526 8 8460 4518 0 10
28 62
0 1 120 51696 6520 1673048 0 0 9056 0 9002 4783 0 10
46 44
0 5 120 51920 6520 1672096 0 0 5694 17320 5884 3064 0 7
41 52
0 3 120 51464 6520 1672768 0 0 4200 6092 4456 2299 0 5
48 47
0 4 120 52348 6524 1672528 0 0 8194 4162 8661 4356 0 9
37 54
The cache size still continues to progress to reach 1930524 before
coming back to 16620
when the run is over.
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
0 8 120 54764 2348 1928060 0 0 11684 8 5602 1005 0 4
25 71
0 8 120 58296 2368 1924276 0 0 11760 0 5565 896 0 4
26 71
0 8 120 53804 2376 1929096 0 0 10394 0 5037 858 0 3
40 56
0 8 120 52304 2380 1930524 0 0 8568 8 6564 1039 0 3
29 68
0 8 120 57636 2380 1924780 0 0 9082 0 6526 1012 0 4
28 67
2 0 120 473696 2420 1509920 0 0 1638 8 31522 3589 0
34 51 15
0 0 120 1988176 2620 16620 0 0 102 0 294 113 0 20
76 5
0 0 120 1988176 2620 16620 0 0 0 12 268 71 0 0
100 0
0 0 120 1988300 2632 16620 0 0 0 70 265 73 0 0
100 0
0 0 120 1981356 2672 24708 0 0 0 0 15152 1991 0
16 84 0
The created file is 2147483648 bytes long.
Best regards
>Trond Myklebust wrote:
>
>
>>On Tue, 2006-09-05 at 15:55 -0700, Bryce Harrington wrote:
>>
>>
>>
>>>>Is the readahead algorithm working correctly? Could you please try
>>>>applying the following patch, and then check your syslog for warnings?
>>>>
>>>>
>>>>
>>>I just want to follow up on this one and make sure I understand it
>>>properly.
>>>
>>>Dean says this is due to hitting the memory limit of server-side
>>>caching, which seems to explain it. Our server has 1G memory, and when
>>>shrinking it to lower ram, the issue does occur earlier.
>>>
>>>
>>>
>>Do you see the same dropoff when running iozone locally on the server?
>>Personally, I would expect that the server itself should be doing
>>readahead too in order to mitigate these effects.
>>
>>Cheers,
>> Trond
>>
>>_______________________________________________
>>NFSv4 mailing list
>>NFSv4 at linux-nfs.org
>>http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4
>>
>>
>>
>
>
>
--
-----------------------------------------------------------------
Company : Bull, Architect of an Open World TM (www.bull.com)
Name : Aime Le Rouzic
Mail : Bull - BP 208 - 38432 Echirolles Cedex - France
E-Mail : aime.le-rouzic at bull.net
Phone : 33 (4) 76.29.75.51
Fax : 33 (4) 76.29.75.18
-----------------------------------------------------------------
More information about the NFSv4
mailing list