Regarding testing of NFS over RDMA patch on AMMASO RNIC.
Sivakumar Subramani
Sivakumar.Subramani at neterion.com
Tue Dec 12 11:55:52 EST 2006
Hi James,
Thanks for the reply. Sorry for the confusion in the previous mail.
<<<<James>>>>
This was a mount over TCP or UDP, correct?
<<<<<<Siva>>>>>
No it is NFS over RDMA. I am able to do NFS mount over RDMA with debug option enabled. Normal NFS mount over TCP works with out any issue.
Here the output of the NFS mount over RDMA.....
#########<<<Intially I have not enable the Debug option on Server and tried nfs mount over RDMA. It failed >>> ##############
On Client Machine:
[root at tyan nfsrdmamount]# ./nfsrdmamount -o rdma=192.168.69.148 192.168.68.148:/home/ssk /mnt
Doing nfs/rdma mount to 192.168.69.148, mount protocol to 192.168.68.148
nfsmount: Input/output error
[root at tyan nfsrdmamount]# ./nfsrdmamount -o rdma=192.168.69.148 192.168.68.148:/home/ssk /mnt
Doing nfs/rdma mount to 192.168.69.148, mount protocol to 192.168.68.148
nfsmount: Input/output error
######<<< Then On the server machine I enabled the debug option and restarted (Stop/start) the nfs server on server machine.>>>> #######
root at Nemo ~]# echo 32767 > /proc/sys/sunrpc/nfsd_debug
[root at Nemo ~]#
[root at Nemo ~]#
[root at Nemo ~]# /sbin/service nfs stop
Shutting down NFS mountd: [ OK ]
Shutting down NFS daemon:
[ OK ]
Shutting down NFS quotas: [ OK ]
Shutting down NFS services: [ OK ]
[root at Nemo ~]#
[root at Nemo ~]# /sbin/service nfs start
Starting NFS services: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS daemon: [ OK ]
Starting NFS mountd: [ OK ]
#######<<< Then I tried nfs mount over RDMA on Client machine. It worked. I am able to see the /home/ssk directory of the server system on client system. Here I used NFS over RDMA. Here is the output.>>>>##########
[root at tyan nfsrdmamount]# ./nfsrdmamount -o rdma=192.168.69.148 192.168.68.148:/home/ssk /mnt
Doing nfs/rdma mount to 192.168.69.148, mount protocol to 192.168.68.148
[root at tyan nfsrdmamount]# cd /mnt
[root at tyan mnt]# ls
backup dev_kma kernel load_mod ogc_amso server.txt
Desktop ip.conf kma nfsrdma20060804 ogc_amso.tar tools
[root at tyan mnt]#
So the issue is that I am able to do NFS mount over RDMA only by enabling debug options on the Server system. I hope it is clear now. Please let me know if you have any queries.
Thanks,
~Siva
-----Original Message-----
From: nfsv4-bounces at linux-nfs.org on behalf of James Lentini
Sent: Tue 12/12/2006 10:17 AM
To: Sivakumar Subramani
Cc: nfsv4 at linux-nfs.org; Leonid Grossman; Tom Tucker; Sriram Rapuru
Subject: RE: Regarding testing of NFS over RDMA patch on AMMASO RNIC.
Hi Siva,
Replies below.
On Tue, 12 Dec 2006, Sivakumar Subramani wrote:
> It seems that I need to restart the nfs server after setting the
> debug value in the proc files. I was able to see the debug statement
> on server side only when I restart the server after copying the
> value to the corresponding files in the proc file system.
That is incorrect. The server does not need to be restarted after
changing the debug values.
A couple of things could be happening.
The server may have executed the prints statements before you
turned on debugging.
Another possibility is that the server really wasn't doing anything.
The client can cache data. Therefore, a client operation (ls, cat,
etc.) is not guaranteed to result in NFS requests to the server.
> Today I was able to nfs mount my home directory only if I enable the
> following option on server side:
>
> echo 32767 > /proc/sys/sunrpc/nfsd_debug
This was a mount over TCP or UDP, correct?
Something is really wrong with your setup if a mount over TCP or UDP
only works when debugging is on. Turning debugging on slows things
down which could mask a race condition in either the client or the
server. However I don't know of any such race conditions in the latest
Linux kernels.
You need to fix this problem before attempting to use RDMA.
What Linux kernel version are you using?
I suggest you use 2.6.18.5 for two reasons: (1) this kernel is
compatible with the NFS-RDMA release 7 and (2) there was an issue in
the 2.6.18 client that was fixed in the 2.6.18.1 client. The fix is
here:
<http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;\
a=commitdiff_plain;h=bd4c8ce41a2e2f0c5bf54343ab54e8e09faec021;\
hp=d025c9db7f31fc0554ce7fb2dfc78d35a77f3487>
In my testing this bug did not prevent a mount from succeeding, so I
don't think it is related to the problems you are having. You'll want
it fixed when you come to actually transmitting data though :)
Once you think you have NFS working reliably over TCP/UDP, I
suggest you do some sanity testing. Try mounting and unmounting
hundreds of times in a row, run some io tests (iozone, etc) on the
mounted filesytem, and try running the NFS connectathon test suite:
http://www.connectathon.org/nfstests.html
That should iron out the basic NFS over TCP/UDP issues. Once that is
done, download NFS-RDMA release 7 (posted two weeks ago) and let me
know if you have any problems setting up RDMA.
<snip>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>James<<<<<<<<<<<<<<<<<<<<<<<<<
> <<<<<<<<
> In fs/nfsd/nfssvc.c:nfsd_svc(), there should be a call to
> svc_makesock() under a CONFIG_NFSD_RDMA guard. Can you send that line of
> code?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>/James<<<<<<<<<<<<<<<<<<<<<<<<<
> <<<<<<<
>
> #ifdef CONFIG_NFSD_RDMA
> error = svc_makesock(nfsd_serv, IPPROTO_MAX, port);
> if (error < 0)
> goto failure;
> #endif
That looks good. Once we get the NFS over TCP/UDP problem fixed we
can return to the RDMA issues.
_______________________________________________
NFSv4 mailing list
NFSv4 at linux-nfs.org
http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://linux-nfs.org/pipermail/nfsv4/attachments/20061212/160cc5db/attachment-0005.htm
More information about the NFSv4
mailing list