rpc.gssd: Inexplicable "Unknown code" error

[Nathan Patwardhan]

Hello,

We recently tested NFSv4 in our dev environment and qualified it with
3 Linux distros (Debian Etch, Ubuntu Gutsy, SLES 10).  Our NFS server
is a NetApp and our KDC is Win2k3.  Anyhow, since it worked in
development, what's the worst thing that could happen when we tried to
make it work in production?  :-)

Under SLES-10, where we'd had things working in dev previously, I now
see these errors when I try to mount our production NFS server:

  Feb  6 01:29:28 prod-unix-shell03 rpc.gssd[11402]: rpcsec_gss:
gss_init_sec_context: (major) Miscellaneous failure - (minor)
  Unknown code
  Feb  6 01:29:28 prod-unix-shell03 rpc.gssd[11402]: in authgss_destroy()
  Feb  6 01:29:28 prod-unix-shell03 rpc.gssd[11402]: in
authgss_destroy_context()
  Feb  6 01:29:28 prod-unix-shell03 rpc.gssd[11402]: authgss_destroy:
freeing name 0x50b570
  Feb  6 01:29:28 prod-unix-shell03 rpc.gssd[11402]:
authgss_create_default: freeing name 0x51a760
  Feb  6 01:29:28 prod-unix-shell03 rpc.gssd[11402]: WARNING: Failed
to create krb5 context for user with uid 0 for server
  prod-fs-cc1a-pubnet.kendall.corp.my.domain
  Feb  6 01:29:28 prod-unix-shell03 rpc.gssd[11402]: WARNING: Failed
to create krb5 context for user with uid 0 with credentials
  cache FILE:/tmp/krb5cc_machine_CORP.MY.DOMAIN for server
prod-fs-cc1a-pubnet.kendall.corp.my.domain
  Feb  6 01:29:28 prod-unix-shell03 rpc.gssd[11402]: WARNING: Failed
to create krb5 context for user with uid 0 with any credentials cache
for server prod-fs-cc1a-pubnet.kendall.corp.my.domain

For whatever reason, in production, /tmp/krb5cc_machine_CORP.MY.DOMAIN
is missing a service principle for NFS:

 # klist /tmp/krb5cc_machine_CORP.MY.DOMAIN
Ticket cache: FILE:/tmp/krb5cc_machine_CORP.MY.DOMAIN
Default principal: nfs/prod-unix-shell03.kendall.corp.my.domain at CORP.MY.DOMAIN
Valid starting     Expires            Service principal
02/06/08 01:29:27  02/06/08 11:29:27  krbtgt/CORP.AKAMAI.COM at CORP.MY.DOMAIN
        renew until 02/06/08 11:29:27

Note that in our dev environment, we ARE NOT missing a service
principle for NFS here.  Obviously, this is the nature of our problem.
 But why might this be?

  - There's an SPN for our NFS server in our Windows KDC.
  - DNS seems to be correct for our NFS server.
  - keytab on the NFS server seems to be correct (verified service
principal w/klist).
  - Unknown error is shown, but unknown how?  I don't see any code here.

Any ideas as to what's going on?

-- Nate