NFSv4 clients freeze under certain conditions -- RHEL/DataONTAP

[Simon Vallet]

Hi,

yesterday we had to partially shut-down a Netapp cluster to remove some
extra hardware. Normally, this should not cause any problems, as the
other cluster member takes over the one that is shut down and the
clients aren't supposed to see the difference.

However every single Linux client that was NFSv4-mounting a
filesystem on the filer finally failed to the point we had to reboot 
(hard-reset) it, as ls/df etc were unresponsive and probably froze 
the entire system (no ssh access etc.)

from a RHEL AS4 -- kernel is rh-2.6.9-42 ; no idea which patches
they included :
[...]
Sep  5 10:38:01 www1 kernel: RPC: rpciod waiting on sync task!
Sep  5 12:20:14 www1 kernel: RPC: rpciod waiting on sync task!
Sep  5 13:21:19 www1 kernel: RPC: rpciod waiting on sync task!
Sep  5 13:28:34 www1 kernel: RPC: rpciod waiting on sync task!
Sep  5 13:32:46 www1 kernel: RPC: rpciod waiting on sync task!
Sep  5 13:32:46 www1 kernel: RPC: rpciod waiting on sync task!
Sep  5 17:04:01 www1 kernel: RPC: rpciod waiting on sync task!
Sep  5 17:12:52 www1 kernel: RPC: error 512 connecting to server nfs2
Sep  5 22:22:57 www1 kernel: RPC: rpciod waiting on sync task!
Sep  5 23:00:12 www1 kernel: RPC: rpciod waiting on sync task!
Sep  6 09:51:01 www1 kernel: nfs_statfs: statfs error = 512
Sep  6 10:26:38 www1 kernel: nfs_statfs: statfs error = 512
Sep  6 10:26:45 www1 kernel: nfs_statfs: statfs error = 512
[...]

nfs2, which is one of the cluster nodes, was shut-down at 17:12:39

>From another RHEL client, we don't observe the 512 error (what is this error, actually ?)
at all, but only the "rpciod waiting on sync task!"

Suse users seem to encounter similar problems : 
http://lists.suse.com/archive/suse-sles-e/2006-Jul/0055.html

Any idea what could be the cause of this ?

Simon

-- 
Simon Vallet
Ingénieur Systèmes/Réseaux
Genoscope / CNRG
Tél. : 01 60 87 36 06
E-mail : svallet at genoscope.cns.fr