kernel client BUGs during shutdown with recent kernels

[Bryce Harrington]

Hi Trond,

With the recent kernel patch we've started noticing frequent (but
intermittent) issues during shutdown.  We are seeing two different BUG
messages, although possibly they are just two symptoms of the same bug.
The issue was also present on both the client and server branches in
GIT, although we did not see the crash in the latest nfs-server-stable
checkin yesterday.  Are you aware of this one already?  I can provide
additional details if needed.

Thanks,
Bryce



----- Forwarded message from Jason Neighbors <jasonn at osdl.org> -----

Date: Fri, 14 Jul 2006 18:49:03 -0700
From: Jason Neighbors <jasonn at osdl.org>
To: bryce at osdl.org
Subject: kernel crashes: 2.6.18-rc1-g3efad9f-nfs-client-stable

Seems to happen on reboots frequently.  So far just on our client systems, but they do reboot quite a bit.

------------[ cut here ]------------
kernel BUG at fs/dcache.c:611!
invalid opcode: 0000 [#1]
PREEMPT SMP
Modules linked in:
CPU:    0
EIP:    0060:[<c0170042>]    Not tainted VLI
EFLAGS: 00010246   (2.6.18-rc1-g3efad9f-nfs-client-stable #1)
EIP is at shrink_dcache_for_umount_subtree+0x150/0x20b
eax: 0000005a   ebx: f43bc2a0   ecx: c043b2a4   edx: 00000001
esi: f3f5b448   edi: f765ce40   ebp: f6469d1c   esp: f6469cf4
ds: 007b   es: 007b   ss: 0068
Process 180-boot_kernel (pid: 9380, ti=f6469000 task=f4bc6550 task.ti=f6469000)
Stack: c03d7800 f43bc2a0 00000072 f43bc310 00000001 c0419e2c f6ee9f24 f6ee9d84
       c047e3a0 c04cc324 f6469d2c c017012d f765ce40 f6ee9d84 f6469d40 c0160f77
       f6ee9d84 00000011 c047e400 f6469d50 c01617c8 f6ee9d84 f6ee9d84 f6469d60
Call Trace:
 [<c010378e>] show_stack_log_lvl+0x8a/0x92
 [<c01038ef>] show_registers+0x11d/0x186
 [<c0103adb>] die+0x10c/0x1da
 [<c0103c17>] do_trap+0x6e/0x8a
 [<c0103e6e>] do_invalid_op+0x90/0x97
 [<c0103461>] error_code+0x39/0x40
 [<c017012d>] shrink_dcache_for_umount+0x30/0x54
 [<c0160f77>] generic_shutdown_super+0x1b/0xfd
 [<c01617c8>] kill_anon_super+0x10/0x3a
 [<c016180d>] kill_litter_super+0] mntput_no_expire+0x48/0x6a
 [<c01795f7>] simple_release_fs+0x42/0x4b
 [<c039af51>] rpc_put_mount+0x12/0x14
 [<c038dffd>] rpc_destroy_client+0xca/0xda
 [<c038df2b>] rpc_shutdown_client+0xf2/0xfa
 [<c01bd5ce>] nfs_free_client+0x99/0xe5
 [<c01bd686>] nfs_put_client+0x6c/0x72
 [<c01be3cf>] nfs_free_server+0x8d/0xc1
 [<c01c4c9b>] nfs_kill_super+0x19/0x1e
 [<c0160ecd>] deactiva3fc7>] mntput_no_expire+0x48/0x6a
 [<c011efb8>] do_exit+0x22f/0x404
 [<c011f241>] sys_exit_group+0x0/0x12
 [<c0127c2f>] get_signal_to_deliver+0x281/0x2aa
 [<c0102773>] do_signal+0x57/0x104
 [<c010284d>] do_notify_resume+0x2d/0x3c
 [<c0102a26>] work_notifysig+0x13/0x19
Code: d2 05 a0 01 00 00 50 8b 43 60 8b 40 1c ff 30 8b 03 50 ff 73 34 8b 43 1c 85 c0 74 03 8b 50 20 52 53 68  fe cb fa ff <0
EIP: [<c0170042>] shrink_dcache_for_umount_subtree+0x150/0x20b SS:ESP 0068:f6469cf4
 <1>Fixing recursive fault but reboot is needed!


------------[ cut here ]------------
kernel BUG at kernel/timer.c:396!
invalid opcode: 0000 [#1]
PREEMPT SMP
Modules linked in:
CPU:    0
EIP:    0060:[<c01247c3>]    Not tainted VLI
EFLAGS: 00010883   (2.6.18-rc1-g3efad9f-nfs-client-stable #1)
EIP is at cascade+0x40/0x65
eax: f7287418   ebx: c0478678   ecx: c0504f94   edx: c0552e00
esi: c0504f94   edi: 00000026   ebp: c0504fa8   esp: c0504f94
ds: 007b   es: 007b   ss: 0068
Process swapper (pid: 0, ti=c0504000 task=c0438820 task.ti=c04cd000)
Stack: f7287418 c19abad8 00000000 c0552e00 c04fc380 c0504fd8 c0125546 c0552e00
       c055361c 00000026 c02f9025 c0504fc0 c0504fc0 00000000 c04c2b08 00000001
       c04fc380 c0504ff8 c0121317 c04c2b08 00000000 0000000a c04cdf6c c04cd000
Call Trace:
 [<c010378e>] show_stack_log_lvl+0x8a/0x92
 [<c01038ef>] show_registers+0x11d/0x186
 [<c0103adb>] die+0x10c/0x1da
 [<c0103c17>] do_trap+0x6e/0x8a
 [<c0103e6e>] do_invalid_op+0x90/0x97
 [<c0103461>] error_code+0x39/0x40
 [<c0125546>] run_timer_softirq+0x64/0x180
 [<c0121317>] __do_softirq+0x5d/0xc6
 [<c0104c76>] do_softirq+0x5b/0xaa
 =======================
 [<c01213b9>] irq_exit+0x39/0x46
 [<c0110221>] smp_apic_timer_interrupt+0x5a/0x5f
 [<c01033af>] apic_timer_interrupt+0x1f/0x24
 [<c0100c3f>] cpu_idle+0xae/0xdb
 [<c01002d2>] _stext+0x3a/0x3c
 [<c04d27f4>] start_kernel+0x184/0x186
 [<c0100210>] 0xc0100210
Code: 00 00 03 45 0c 8b 10 89 55 ec 89 4a 04 8b 50 04 89 0a 89 55 f0 89 00 89 40 04 8b 45 ec 39 c8 8b 18 74 23 8b 55 08 39
EIP: [<c01247c3>] cascade+0x40/0x65 SS:ESP 0068:c0504f94
 <0>Kernel panic - not syncing: Fatal exception in interrupt


-- 
Jason Neighbors
x1939

----- End forwarded message -----

The bugs have occurred on the following test runs:


dcache.c:611
1023:	07/12 20:00-21:00, nfs13    linux-2.6.18-rc1-CITI_NFS4_ALL-1
1023:	07/12 21:00-22:00, nfs13    ""
1026:	07/13 01:00-02:00, nfs03    linux-2.6.18-rc1-gae9dce8-nfs-server-stable
1039:	07/13 20:00-21:00, nfs13    linux-2.6.18-rc1-g3efad9f-nfs-client-stable
1052:	07/14 22:00-23:00, nfs03    ""
1052:	07/15 00:00-01:00, nfs03    ""

timer.c:396
1027:	07/13 01:00-02:00, nfs13    linux-2.6.18-rc1-gae9dce8-nfs-server-stable
1039:	07/13 22:00-23:00, nfs13    linux-2.6.18-rc1-g3efad9f-nfs-client-stable
1052:	07/15 01:00-02:00, nfs03    ""

Bryce