Regarding testing of NFS over RDMA patch on AMMASO RNIC.

[Sivakumar Subramani]

Hi James,

Thanks for the reply.

It seems that I need to restart the nfs server after setting the debug
value in the proc files. I was able to see the debug statement on server
side only when I restart the server after copying the value to the
corresponding files in the proc file system.

Today I was able to nfs mount my home directory only if I enable the
following option on server side:

echo 32767 > /proc/sys/sunrpc/nfsd_debug

After this I restart the nfs server and tried nfsmount on the client
side, It worked.

[root at tyan nfsrdmamount]# ./nfsrdmamount -o rdma=192.168.69.148
192.168.68.148:/home/ssk /mnt
Doing nfs/rdma mount to 192.168.69.148, mount protocol to 192.168.68.148
[root at tyan nfsrdmamount]# cd /mnt
[root at tyan mnt]# ls
backup   dev_kma  kernel  load_mod         ogc_amso      server.txt
Desktop  ip.conf  kma     nfsrdma20060804  ogc_amso.tar  tools
[root at tyan mnt]#


Then I restart (Stop/Start) nfs  server on the server system and with
out setting debug option, I tried nfs mount on the client machine, It
failed.

Server system:
===========
[root at Nemo ssk]# /sbin/service nfs stop
Shutting down NFS mountd:                                  [  OK  ]
Shutting down NFS daemon:                                  [  OK  ]
Shutting down NFS quotas:                                  [  OK  ]
Shutting down NFS services:                                [  OK  ]
[root at Nemo ssk]#
[root at Nemo ssk]# /sbin/service nfs start
Starting NFS services:                                     [  OK  ]
Starting NFS quotas:                                       [  OK  ]
Starting NFS daemon:                                       [  OK  ]
Starting NFS mountd:                                       [  OK  ]
[root at Nemo ssk]#

[root at tyan nfsrdmamount]# ./nfsrdmamount -o rdma=192.168.69.148
192.168.68.148:/home/ssk /mnt
Doing nfs/rdma mount to 192.168.69.148, mount protocol to 192.168.68.148
nfsmount: Input/output error

Any suggestion? Please let me know if you need any log files.


>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>James<<<<<<<<<<<<<<<<<<<<<<<<<
<<<<<<<<
In fs/nfsd/nfssvc.c:nfsd_svc(), there should be a call to
svc_makesock() under a CONFIG_NFSD_RDMA guard. Can you send that line of
code?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>/James<<<<<<<<<<<<<<<<<<<<<<<<<
<<<<<<<

#ifdef CONFIG_NFSD_RDMA
                error = svc_makesock(nfsd_serv, IPPROTO_MAX, port);
                if (error < 0)
                        goto failure;
#endif

Thanks,
~Siva
 
-----Original Message-----
From: James Lentini [mailto:jlentini at netapp.com] 
Sent: Monday, December 11, 2006 9:00 PM
To: Sivakumar Subramani
Cc: Talpey, Thomas; nfsv4 at linux-nfs.org; Tom Tucker; Leonid Grossman;
Sriram Rapuru
Subject: RE: Regarding testing of NFS over RDMA patch on AMMASO RNIC.


Thank you for the logs Siva.

In fs/nfsd/nfssvc.c:nfsd_svc(), there should be a call to
svc_makesock() under a CONFIG_NFSD_RDMA guard. Can you send that line of
code?

I suspect that your client is attempting to connect to port 2049 and the
server is listening on port 2050.

james

On Thu, 7 Dec 2006, Sivakumar Subramani wrote:

> 
> Hi James,
> 
> <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<JAMES>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> I suggest you do the following:
> 
> 1. On the client
> 
>  dmesg -c
>  echo 32767 > /proc/sys/sunrpc/rpc_debug  echo 65535 > 
> /proc/sys/sunrpc/nfs_debug
> 
> 2. On the server
> 
>  dmesg -c
>  echo 32767 > /proc/sys/sunrpc/rpc_debug  echo 32767 > 
> /proc/sys/sunrpc/nfsd_debug
> 
> 3. Attempt a mount.
> 
> 4. On the client
> 
>    dmesg > /tmp/clnt.txt
> 
> 5. On the server
> 
>    dmesg > /tmp/srv.txt
> 
> Please send the the two log files and we will try to get to the bottom

> of this.
> 
> <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<</JAMES>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> 
> <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<SIVA>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Now I am able to do normal TCP nfs mount. As per 
> your previous mail, I have collected the log files on server and
client systems. The files are attached with this mail.
> I could see some error messages in the client log. Please let me know
if you find any issues.
> 
> [root at tyan ~]# mount -t nfs 10.114.52.99:/home/ssk /mnt [root at tyan ~]#

> cd /mnt [root at tyan mnt]# ls
> backup   dev_kma  kernel  load_mod         ogc_amso      server.txt
> Desktop  ip.conf  kma     nfsrdma20060804  ogc_amso.tar  tools
> [root at tyan mnt]# cd ..
> [root at tyan /]# umount /mnt
> 
> 
> [root at tyan nfsrdmamount]# ./nfsrdmamount -o rdma=192.168.69.148 
> 192.168.68.148:/tmp /mnt Doing nfs/rdma mount to 192.168.69.148, mount

> protocol to 192.168.68.148
> 
> nfsmount: Input/output error
> [root at tyan nfsrdmamount]#
> 
> <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<</SIVA>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> Thanks,
> ~Siva
> 
> -----Original Message-----
> From: James Lentini [mailto:jlentini at netapp.com]
> Sent: Wed 12/6/2006 6:36 PM
> To: Sivakumar Subramani
> Cc: Talpey, Thomas; nfsv4 at linux-nfs.org; Tom Tucker; Leonid Grossman; 
> Sriram Rapuru
> Subject: RE: Regarding testing of NFS over RDMA patch on AMMASO RNIC.
>  
> 
> 
> On Tue, 5 Dec 2006, Sivakumar Subramani wrote:
> 
> > Sorry for the delay. Currently I am facing different problem. I am 
> > getting "Permission Denied" error when I try nfs mount. I am getting

> > this error even with normal TCP nfs mount.
> 
> Hi Siva,
> 
> I don't see any obvious errors in the information you provided. To 
> configure your systems, I suggest you follow these instructions:
> 
>  http://nfs.sourceforge.net/nfs-howto/
> 
> You definitely need NFS TCP/UDP mounts working before attempting to 
> setup NFS-RDMA.
> 
> james
> 
> 
>