Regarding testing of NFS over RDMA patch on AMMASO RNIC.
Sivakumar Subramani
Sivakumar.Subramani at neterion.com
Tue Dec 5 01:34:37 EST 2006
Hi James,
Sorry for the delay. Currently I am facing different problem. I am
getting "Permission Denied" error when I try nfs mount. I am getting
this error even with normal TCP nfs mount.
I have given all of my configuration details on both server and client
machine. Also I have give the error of the nfs mount. Currently I am
trying to resolve this "Permission Denied" error. Any help on this
current issue. I am sure this a configuration error. But I could not get
what step I am missing.
Server Machine configuration:
=============================
Lsmod Output:
Module Size Used by
nfsd 248304 16
nfs 224856 0
nfs_acl 5376 1 nfs
lockd 65296 3 nfsd,nfs
exportfs 7808 1 nfsd
s2io 85604 0
parport_pc 31208 1
lp 16464 0
parport 42508 2 parport_pc,lp
autofs4 25352 0
dm_mirror 23424 0
dm_multipath 21136 0
dm_mod 60368 2 dm_mirror,dm_multipath
button 9376 0
battery 12168 0
ac 7176 0
ohci_hcd 22660 0
i2c_amd8111 8064 0
i2c_amd756 9476 0
i2c_core 25344 2 i2c_amd8111,i2c_amd756
shpchp 41192 0
iw_c2 60928 0
xprtrdma 39044 0
sunrpc 213360 20 nfsd,nfs,nfs_acl,lockd,xprtrdma
ipv6 268000 21 sunrpc
rdma_krping 19984 0
rdma_ucm 15232 0
rdma_cm 26760 4 xprtrdma,sunrpc,rdma_krping,rdma_ucm
ib_local_sa 14096 2 rdma_ucm,rdma_cm
findex 6656 1 ib_local_sa
iw_cm 11528 1 rdma_cm
ib_addr 10120 1 rdma_cm
ib_umad 17968 0
ib_ucm 19336 0
ib_uverbs 40144 2 rdma_ucm,ib_ucm
ib_sa 17084 2 rdma_cm,ib_local_sa
ib_cm 34368 2 rdma_cm,ib_ucm
ib_mad 39096 4 ib_local_sa,ib_umad,ib_sa,ib_cm
ib_core 55424 13
iw_c2,xprtrdma,sunrpc,rdma_krping,rdma_cm,ib_local_sa,iw_cm,ib_umad,ib_u
cm,ib_uverbs,ib_sa,ib_cm,ib_mad
tg3 104068 0
floppy 68104 0
ext3 133520 1
jbd 60584 1 ext3
Ifconfig Output:
================
eth3 Link encap:Ethernet HWaddr 00:0D:B2:00:11:DE
inet addr:192.168.68.148 Bcast:192.168.68.255
Mask:255.255.255.0
inet6 addr: fe80::20d:b2ff:fe00:11de/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:33 errors:0 dropped:0 overruns:0 frame:0
TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:5848 (5.7 KiB) TX bytes:650 (650.0 b)
Interrupt:193
iw3 Link encap:Ethernet HWaddr 00:0D:B2:00:11:DF
inet addr:192.168.69.148 Mask:255.255.255.0
inet6 addr: fe80::20d:b2ff:fe00:11df/64 Scope:Link
UP RUNNING NOARP MTU:1500 Metric:1
RX packets:33 errors:0 dropped:0 overruns:0 frame:0
TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5848 (5.7 KiB) TX bytes:650 (650.0 b)
Exportfs Ouput:
===============
root at Nemo fs]# cat /etc/exports
/home/ssk 192.168.68.149(rw,async)
192.168.69.149(rw,async,insecure,no_root_squash)
/tmp 192.168.68.149(rw,async)
192.168.69.149(rw,async,insecure,no_root_squash)
/home/ssk 10.114.52.156(rw,async,no_root_squash,insecure)
/tmp 10.114.52.156(rw,async)
#/tmp 192.168.68.149(rw,async)
/home/ssk 10.114.52.154(rw,async,no_root_squash,insecure)
[root at Nemo fs]# exportfs -a
[root at Nemo fs]# exportfs -i
/home/ssk 192.168.68.149
/home/ssk 192.168.69.149
/home/ssk tyan
/home/ssk IA64
/tmp 192.168.68.149
/tmp 192.168.69.149
/tmp tyan
NFS server status:
==================
[root at Nemo fs]# /sbin/service nfs start
Starting NFS services: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS daemon: [ OK ]
Starting NFS mountd: [ OK ]
[root at Nemo fs]#
Client Machine Configuration:
=============================
[root at tyan fs]# lsmod
Module Size Used by
nfsd 242832 16
nfs 209976 0
nfs_acl 7168 2 nfsd,nfs
lockd 60816 3 nfsd,nfs
exportfs 8832 1 nfsd
iw_c2 57088 0
xprtrdma 36612 0
sunrpc 187376 20 nfsd,nfs,nfs_acl,lockd,xprtrdma
rdma_krping 19600 0
rdma_ucm 14464 0
rdma_cm 23048 4 xprtrdma,sunrpc,rdma_krping,rdma_ucm
ib_local_sa 13072 2 rdma_ucm,rdma_cm
findex 7936 1 ib_local_sa
iw_cm 11784 1 rdma_cm
ib_addr 10120 1 rdma_cm
ib_umad 16048 0
ib_ucm 17672 0
ib_uverbs 35504 2 rdma_ucm,ib_ucm
ib_sa 16012 2 rdma_cm,ib_local_sa
ib_cm 31632 2 rdma_cm,ib_ucm
ib_mad 36388 4 ib_local_sa,ib_umad,ib_sa,ib_cm
ib_core 47104 13
iw_c2,xprtrdma,sunrpc,rdma_krping,rdma_cm,ib_local_sa,iw_cm,ib_umad,ib_u
cm,ib_uverbs,ib_sa,ib_cm,ib_mad
s2io 63244 0
[root at tyan fs]#
Ifconfig configuration:
========================
eth4 Link encap:Ethernet HWaddr 00:0D:B2:00:11:D6
inet addr:192.168.68.149 Bcast:192.168.68.255
Mask:255.255.255.0
inet6 addr: fe80::20d:b2ff:fe00:11d6/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:7 errors:0 dropped:0 overruns:0 frame:0
TX packets:60 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:488 (488.0 b) TX bytes:8735 (8.5 KiB)
Interrupt:209
iw4 Link encap:Ethernet HWaddr 00:0D:B2:00:11:D7
inet addr:192.168.69.149 Mask:255.255.255.0
inet6 addr: fe80::20d:b2ff:fe00:11d7/64 Scope:Link
UP RUNNING NOARP MTU:1500 Metric:1
RX packets:7 errors:0 dropped:0 overruns:0 frame:0
TX packets:60 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:488 (488.0 b) TX bytes:8735 (8.5 KiB)
************************************************************************
*****
NFS RDMA Mount Output:
======================
[root at tyan nfsrdmamount]# ./nfsrdmamount -o rdma=192.168.69.148
192.168.68.148:/tmp /mnt
192.168.68.148:/tmp failed, reason given by server: Permission denied
[root at tyan nfsrdmamount]#
Normal NFS mount:
=================
[root at tyan nfsrdmamount]# mount -t nfs nemo:/tmp /mnt
mount: nemo:/tmp failed, reason given by server: Permission denied
[root at tyan nfsrdmamount]# mount -t nfs 10.114.52.99:/tmp /mnt
mount: 10.114.52.99:/tmp failed, reason given by server: Permission
denied
[root at tyan nfsrdmamount]#
************************************************************************
****
Dmesg on the Server side:
ec 5 12:03:10 Nemo rpc.mountd: authenticated mount request from
192.168.68.149:778 for /tmp (/tmp)
Dec 5 12:03:36 Nemo rpc.mountd: authenticated mount request from
192.168.68.149:787 for /tmp (/tmp)
Dec 5 12:04:07 Nemo rpc.mountd: authenticated mount request from
192.168.68.149:788 for /tmp (/tmp)
Dec 5 12:04:15 Nemo rpc.mountd: authenticated mount request from
192.168.68.149:790 for /tmp (/tmp)
Dec 5 12:04:24 Nemo rpc.mountd: authenticated mount request from
tyan:791 for /tmp (/tmp)
Dec 5 12:04:29 Nemo rpc.mountd: authenticated mount request from
192.168.68.149:792 for /tmp (/tmp)
Thanks,
~Siva
-----Original Message-----
From: James Lentini [mailto:jlentini at netapp.com]
Sent: Thursday, November 30, 2006 9:50 PM
To: Sivakumar Subramani
Cc: Talpey, Thomas; nfsv4 at linux-nfs.org; Tom Tucker; Leonid Grossman;
Sriram Rapuru
Subject: RE: Regarding testing of NFS over RDMA patch on AMMASO RNIC.
Siva,
> What are the netmasks you are using? What is the output of "ip route"
> on both the client and server?
> <<<<<<<<<<<<<<<<<<<<<<<<<<<<<SIVA>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
> Here is the output of ifconfig and route command:
>
> Server system:
> ==============
> eth2 Link encap:Ethernet HWaddr 00:0D:B2:00:11:DE
> inet addr:192.168.68.148 Bcast:192.168.68.255
Mask:255.255.255.0
> inet6 addr: fe80::20d:b2ff:fe00:11de/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:2529 errors:0 dropped:0 overruns:0 frame:0
> TX packets:368 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:527885 (515.5 KiB) TX bytes:53108 (51.8 KiB)
> Interrupt:185
>
> iw2 Link encap:Ethernet HWaddr 00:0D:B2:00:11:DF
> inet addr:192.168.69.148 Mask:255.255.255.0
> inet6 addr: fe80::20d:b2ff:fe00:11df/64 Scope:Link
> UP RUNNING NOARP MTU:1500 Metric:1
> RX packets:2529 errors:0 dropped:0 overruns:0 frame:0
> TX packets:368 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:527885 (515.5 KiB) TX bytes:53108 (51.8 KiB)
>
>
> [root at Nemo ~]# route
> Kernel IP routing table
> Destination Gateway Genmask Flags Metric Ref
Use Iface
> 192.168.69.0 * 255.255.255.0 U 0 0
0 iw2
> 192.168.68.0 * 255.255.255.0 U 0 0
0 eth2
> 10.114.52.0 * 255.255.255.0 U 0 0
0 eth0
> 169.254.0.0 * 255.255.0.0 U 0 0
0 eth0
> default 10.114.52.1 0.0.0.0 UG 0 0
0 eth0
> [root at Nemo ~]#
>
>
> Client System:
> ===============
> eth4 Link encap:Ethernet HWaddr 00:0D:B2:00:11:D6
> inet addr:192.168.68.149 Bcast:192.168.68.255
Mask:255.255.255.0
> inet6 addr: fe80::20d:b2ff:fe00:11d6/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:110 errors:0 dropped:0 overruns:0 frame:0
> TX packets:279 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:16716 (16.3 KiB) TX bytes:40535 (39.5 KiB)
> Interrupt:209
>
> iw4 Link encap:Ethernet HWaddr 00:0D:B2:00:11:D7
> inet addr:192.168.69.149 Mask:255.255.255.0
> inet6 addr: fe80::20d:b2ff:fe00:11d7/64 Scope:Link
> UP RUNNING NOARP MTU:1500 Metric:1
> RX packets:110 errors:0 dropped:0 overruns:0 frame:0
> TX packets:279 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:16716 (16.3 KiB) TX bytes:40535 (39.5 KiB)
>
>
> [root at tyan ~]# route
> Kernel IP routing table
> Destination Gateway Genmask Flags Metric Ref
Use Iface
> 192.168.69.0 * 255.255.255.0 U 0 0
0 iw4
> 192.168.68.0 * 255.255.255.0 U 0 0
0 eth4
> 10.114.52.0 * 255.255.255.0 U 0 0
0 eth0
> 169.254.0.0 * 255.255.0.0 U 0 0
0 eth0
> default 10.114.52.1 0.0.0.0 UG 0 0
0 eth0
> [root at tyan ~]#
> <<<<<<<<<<<<<<<<<<<<<<<<<<<<</SIVA>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >
This all looks good.
> You need to have an 2 entries: one for the ammasso adapter's eth
> interface and one for the iwarp interface.
>
> In the output below, it appears that you have two /etc/exports entries
> for /tmp 192.168.68.149. One of the entries has a typo. This should be
> fixed, but I doubt it is the source of your problems.
>
> <<<<<<<<<<<<<<<<<<<<<<<<<<<<<SIVA>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> I added the second entry just to verify whether I am able to do nfs
> mount with out rdma. Now I have removed it. The content of the
> /etc/exports on server machine is as follow:
>
> [root at Nemo ~]# cat /etc/exports
> /home/ssk 192.168.68.149(rw,async)
> 192.168.69.149(rw,async,insecure,no_root_squash)
> /tmp 192.168.68.149(rw,async)
> 192.168.69.149(rw,async,insecure,no_root_squash)
> /home/ssk 10.114.52.156(rw,async,no_root_squash,insecure)
> /tmp 10.114.52.156(rw,async)
> #/tmp 192.168.68.149(rw,async)
> /home/ssk 10.114.52.154(rw,async,no_root_squash,insecure)
> [root at Nemo ~]# exportfs
> /home/ssk 192.168.68.149
> /home/ssk 192.168.69.149
> /home/ssk tyan
> /home/ssk IA64
> /tmp 192.168.68.149
> /tmp 192.168.69.149
> /tmp tyan
> [root at Nemo ~]#
> <<<<<<<<<<<<<<<<<<<<<<<<<<<<</SIVA>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >
Looks good now.
> There are NFS and RPC debug options that will provide more
> information. If your answers to the above questions don't shed any
> light on this problem, I'll have you turn those on.
>
> <<<<<<<<<<<<<<<<<<<<<<<<<<<<<SIVA>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
> Still I am facing the problem. The output of the nfsrdmamount is as
follow:
>
> [root at tyan nfsrdmamount]# ./nfsrdmamount -o rdma=192.168.69.148
> 192.168.68.148:/tmp /mnt Doing nfs/rdma mount to 192.168.69.148, mount
> protocol to 192.168.68.148
> nfsmount: Input/output error
> [root at tyan nfsrdmamount]#
>
>
> Is this the debug option that you suggest?
> echo 32767 >/proc/sys/sunrpc/rpc_debug echo 32767
> >/proc/sys/sunrpc/nfs_debug echo 32767 >/proc/sys/sunrpc/nfsd_debug
>
> <<<<<<<<<<<<<<<<<<<<<<<<<<<<</SIVA>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >
I suggest you do the following:
1. On the client
dmesg -c
echo 32767 > /proc/sys/sunrpc/rpc_debug echo 65535 >
/proc/sys/sunrpc/nfs_debug
2. On the server
dmesg -c
echo 32767 > /proc/sys/sunrpc/rpc_debug echo 32767 >
/proc/sys/sunrpc/nfsd_debug
3. Attempt a mount.
4. On the client
dmesg > /tmp/clnt.txt
5. On the server
dmesg > /tmp/srv.txt
Please send the the two log files and we will try to get to the bottom
of this.
More information about the NFSv4
mailing list