Regarding testing of NFS over RDMA patch on AMMASO RNIC.

[James Lentini]

Hi Siva,

Replies below.

On Tue, 12 Dec 2006, Sivakumar Subramani wrote:

> It seems that I need to restart the nfs server after setting the 
> debug value in the proc files. I was able to see the debug statement 
> on server side only when I restart the server after copying the 
> value to the corresponding files in the proc file system.

That is incorrect. The server does not need to be restarted after 
changing the debug values. 

A couple of things could be happening. 

The server may have executed the prints statements before you 
turned on debugging. 

Another possibility is that the server really wasn't doing anything. 
The client can cache data. Therefore, a client operation (ls, cat, 
etc.) is not guaranteed to result in NFS requests to the server.

> Today I was able to nfs mount my home directory only if I enable the
> following option on server side:
> 
> echo 32767 > /proc/sys/sunrpc/nfsd_debug

This was a mount over TCP or UDP, correct? 

Something is really wrong with your setup if a mount over TCP or UDP 
only works when debugging is on. Turning debugging on slows things 
down which could mask a race condition in either the client or the 
server. However I don't know of any such race conditions in the latest 
Linux kernels.

You need to fix this problem before attempting to use RDMA.

What Linux kernel version are you using? 

I suggest you use 2.6.18.5 for two reasons: (1) this kernel is 
compatible with the NFS-RDMA release 7 and (2) there was an issue in 
the 2.6.18 client that was fixed in the 2.6.18.1 client. The fix is 
here:
  
<http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;\
   a=commitdiff_plain;h=bd4c8ce41a2e2f0c5bf54343ab54e8e09faec021;\
   hp=d025c9db7f31fc0554ce7fb2dfc78d35a77f3487>

In my testing this bug did not prevent a mount from succeeding, so I 
don't think it is related to the problems you are having. You'll want 
it fixed when you come to actually transmitting data though :)

Once you think you have NFS working reliably over TCP/UDP, I 
suggest you do some sanity testing. Try mounting and unmounting 
hundreds of times in a row, run some io tests (iozone, etc) on the 
mounted filesytem, and try running the NFS connectathon test suite:

http://www.connectathon.org/nfstests.html

That should iron out the basic NFS over TCP/UDP issues. Once that is 
done, download NFS-RDMA release 7 (posted two weeks ago) and let me 
know if you have any problems setting up RDMA.

<snip>

> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>James<<<<<<<<<<<<<<<<<<<<<<<<<
> <<<<<<<<
> In fs/nfsd/nfssvc.c:nfsd_svc(), there should be a call to
> svc_makesock() under a CONFIG_NFSD_RDMA guard. Can you send that line of
> code?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>/James<<<<<<<<<<<<<<<<<<<<<<<<<
> <<<<<<<
> 
> #ifdef CONFIG_NFSD_RDMA
>                 error = svc_makesock(nfsd_serv, IPPROTO_MAX, port);
>                 if (error < 0)
>                         goto failure;
> #endif

That looks good. Once we get the NFS over TCP/UDP problem fixed we 
can return to the RDMA issues.