I've designed a custom circuit board with an AT91SAM9RL processor. The circuit board has 64 MB of SDRAM memory. (Using test programs, I've verified that there is nothing wrong with the SDRAM memory on this particular circuit board). I've been able to use U-Boot to copy a new version of the Linux kernel into SDRAM memory. I can then use U-Boot to write the kernel to a UBI partition on NAND flash. U-boot is able to successfully boot this kernel.

I am able to mount an SD card connected to the MMC bus, and I am also able to copy files from the root file system to the SD card.

However, if I try to copy files from the SD card to the root file system, I receive a kernel oops. I've tried selecting SLAB and SLUB allocators in the kernel config, but nothing seems to work.

What might be going wrong here, and what could I do to fix it? Could anyone comment on this problem?

Here is the output of the kernel oops, including the Busybox "cp" command that causes the oops:

/media/sdcard # cp -R ./modules /lib/
Unable to handle kernel NULL pointer dereference at virtual address 00000000
pgd = c32e4000
[00000000] *pgd=232e2831, *pte=00000000, *ppte=00000000
Internal error: Oops: 80000005 [#1] PREEMPT
last sysfs file: /sys/devices/virtual/vc/vcsa1/dev
Modules linked in:
CPU: 0 Not tainted (2.6.39-rc7+ #2)
PC is at 0x0
LR is at kobject_uevent_env+0x3fc/0x460
pc : [<00000000>] lr : [<c0132a64>] psr: 20000013
sp : c32cdd64 ip : 00000020 fp : c32cdd88
r10: c32cdd8c r9 : 00000000 r8 : 00001000
r7 : c006873c r6 : c36870b4 r5 : 00026000 r4 : 00000000
r3 : 002b6008 r2 : 40000013 r1 : c0ad8020 r0 : 00001846
Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
Control: 0005317f Table: 232e4000 DAC: 00000015
Process cp (pid: 894, stack limit = 0xc32cc270)
Stack: (0xc32cdd64 to 0xc32ce000)
dd60: c32cc000 c3bf6c80 00001000 00000000 c020aa00 c32cdf40 00000001
dd80: 00000000 00001000 c0353980 c0330000 c0358050 00027000 00000000 00026000
dda0: 00000000 c3687000 00001000 00000000 00001000 c006a7c4 00026000 00000000
ddc0: c32cdf00 00001000 00000000 c01fd5fc 00000027 c32cdf00 c32cdf40 c32cdec8
dde0: c3bf6c80 c36870b4 00000000 00000001 c32cddf0 c32cddf0 c32d11e0 c32cde6c
de00: 00000000 00001000 00000000 c3687018 00000001 c32cdec8 c32cdf40 00000000
de20: c3bf6c80 00026000 00000000 c006a884 00001000 00000000 91827364 c32cde3c
de40: c32cde3c c32cde44 c32cde44 00000000 00026000 00000000 c3687000 c3816000
de60: c32cdec8 c32cdf40 00000001 c00f9510 00026000 00000000 c32cde78 c32cde78
de80: 00000000 00001000 00000000 bea3a548 00000000 00000000 c32cdec4 c32cdec8
dea0: c32cdec8 c3bf6c80 c32cdf40 c32cdf80 fffffdee c32cc000 00000000 c0093620
dec0: 00026000 00000000 c01fe80c 00000001 00000000 00000001 ffffffff c3bf6c80
dee0: 00000000 00000000 00000000 00000000 c3825ce0 00000000 00000000 00000000
df00: 00026000 00000000 c3b57030 00000000 00001000 c367fab8 00001000 00000001
df20: c3bf6208 00000001 c3683400 00000000 00000000 00000000 0000000a c3bf6208
df40: bea39548 00001000 c3bf6c80 00001000 bea39548 c32cdf80 bea39548 c0094050
df60: 00000000 00000000 00026000 00000000 c3bf6c80 00001000 bea39548 c0094194
df80: 00026000 00000000 00001000 00000000 00001000 bea39548 0000000a 00000004
dfa0: c0023144 c0022fc0 00001000 bea39548 0000000a bea39548 00001000 000a6910
dfc0: 00001000 bea39548 0000000a 00000004 00026000 00000000 0000000a 00000000
dfe0: 000a52f4 bea39508 0000e610 40100098 20000010 0000000a 00000000 00000000
[<c0132a64>] (kobject_uevent_env+0x3fc/0x460) from [<00026000>] (0x26000)
Code: bad PC value
---[ end trace 55d3f6b7614f1f39 ]---
note: cp[894] exited with preempt_count 2
BUG: scheduling while atomic: cp/894/0x40000003
Modules linked in:
[<c0026e98>] (unwind_backtrace+0x0/0xec) from [<c01fc878>] (schedule+0x58/0x41c)
[<c01fc878>] (schedule+0x58/0x41c) from [<c002dd60>] (__cond_resched+0x18/0x24)
[<c002dd60>] (__cond_resched+0x18/0x24) from [<c01fcdd0>] (_cond_resched+0x30/0x40)
[<c01fcdd0>] (_cond_resched+0x30/0x40) from [<c0081b6c>] (unmap_vmas+0x660/0x72c)
[<c0081b6c>] (unmap_vmas+0x660/0x72c) from [<c0084474>] (exit_mmap+0x134/0x2a0)
[<c0084474>] (exit_mmap+0x134/0x2a0) from [<c0032c5c>] (mmput+0x3c/0xf8)
[<c0032c5c>] (mmput+0x3c/0xf8) from [<c0036c30>] (exit_mm+0x168/0x178)
[<c0036c30>] (exit_mm+0x168/0x178) from [<c0038744>] (do_exit+0x1dc/0x6bc)
[<c0038744>] (do_exit+0x1dc/0x6bc) from [<c0025d18>] (die+0x1bc/0x1ec)
[<c0025d18>] (die+0x1bc/0x1ec) from [<c0027f04>] (__do_kernel_fault+0x64/0x84)
[<c0027f04>] (__do_kernel_fault+0x64/0x84) from [<c00280dc>] (do_page_fault+0x1b8/0x1d0)
[<c00280dc>] (do_page_fault+0x1b8/0x1d0) from [<c0022254>] (do_PrefetchAbort+0x34/0x94)
[<c0022254>] (do_PrefetchAbort+0x34/0x94) from [<c0022cf0>] (__pabt_svc+0x50/0x80)
Exception stack(0xc32cdd18 to 0xc32cdd60)
dd00: 00001846 c0ad8020
dd20: 40000013 002b6008 00000000 00026000 c36870b4 c006873c 00001000 00000000
dd40: c32cdd8c c32cdd88 00000020 c32cdd64 c0132a64 00000000 20000013 ffffffff
[<c0022cf0>] (__pabt_svc+0x50/0x80) from [<c0132a64>] (kobject_uevent_env+0x3fc/0x460)
[<c0132a64>] (kobject_uevent_env+0x3fc/0x460) from [<00026000>] (0x26000)
Segmentation fault
/media/sdcard # Internal error: Oops - undefined instruction: 0 [#2] PREEMPT
last sysfs file: /sys/devices/virtual/vc/vcsa1/dev
Modules linked in:
CPU: 0 Tainted: G D (2.6.39-rc7+ #2)
PC is at vol_cdev_write+0x1e0/0x370
LR is at nand_write_page_raw+0x20/0x3c
pc : [<c019ee94>] lr : [<c01927c4>] psr: 80000033
sp : c3bf3b20 ip : 000000e5 fp : c3880a30
r10: c392c000 r9 : 00000100 r8 : 00000003
r7 : 00000000 r6 : c3b52100 r5 : c388e5e5 r4 : c388e5e5
r3 : 000000e7 r2 : fffffff0 r1 : c3b52000 r0 : c5000014
Flags: Nzcv IRQs on FIQs on Mode SVC_32 ISA Thumb Segment kernel
Control: 0005317f Table: 232d8000 DAC: 00000017
Process flush-ubifs_1_0 (pid: 841, stack limit = 0xc3bf2270)
Stack: (0xc3bf3b20 to 0xc3bf4000)
3b20: c3880a30 c3880800 00000033 c0192950 c3b51000 c02bd200 c3880800 c3880800
3b40: c3880a30 c3b51000 00006b9a c3880800 00001000 c3880a30 c38809f8 c0192b6c
3b60: 00000000 c3880800 c3880a30 00001000 00001000 00000000 00006b9a c0193b94
3b80: 00000000 00000000 00000000 0000000c 0007ffff 00000000 00006b9a 0000003f
3ba0: 00000000 00000000 00000080 00000000 c3b51000 c3b51000 00000000 00001000
3bc0: c3880800 c3880a30 00000186 06b9a000 00000000 00001000 00000186 c0193d7c
3be0: c38809f8 c3880800 0619a000 00000000 00001000 0001a000 c3b51000 c018ccf4
3c00: 00001000 c3bf3c34 c3b51000 00000000 00000000 c38ad200 c384f800 c01a43fc
3c20: 00001000 c3bf3c34 c3b51000 c002db9c c38ad200 00019000 00000000 c38ad200
3c40: 00019000 00001000 00000022 c384f800 00000000 c01a15ec 00001000 c002db9c
3c60: 00000041 c02b00c8 c3ac94d8 00000008 00000000 00000000 00000000 00024250
3c80: c02c2288 00000000 c3bf2000 c3b51000 00000022 c38ad200 00000000 c02c1cb4
3ca0: c3b58750 00004250 00004250 00000000 c38ad200 00019000 00001000 00000022
3cc0: c3ac81e0 c384f800 c3b51000 c019f7b0 00019000 00001000 00000002 c3bf3ce8
3ce0: c00522d0 c3ac9520 c32e4000 c3816000 00000000 00001030 c3816000 00019000
3d00: 00001030 c0101248 00001000 00000002 00000000 00000000 00000000 c32e4000
3d20: c0355a20 c3ac9520 c32e4000 c3816000 00000000 00001030 00019000 00000030
3d40: c3bf3d90 c00f8e6c c3bf3d5c c0355a20 00000001 00000022 00001000 00000000
3d60: c3bf3de8 00001000 c0355a20 c366edc8 c3816000 00000001 00000001 ffffffff
3d80: c3bf3de8 c00fa56c 00001000 c366ee7c 000003ed 20000001 00000001 c366ee7c
3da0: 00000000 c3bf3ee0 00000001 c006fd40 c0355a20 c0070854 0000000e c3824000
3dc0: ffffffff c366ee7c c006fd2c c366ee7c 00000000 0000000e 00000000 00000000
3de0: 00000000 0000000e 00000000 c0355900 c0355a20 c0355a00 c0355a40 c0355b40
3e00: c0355b60 c0355b80 c0355ba0 c0355bc0 c0355be0 c0355c00 c0355c20 c0355c40
3e20: c0355c60 0000000e c002d828 c366edc8 c366ee7c c3bf3ee0 c3bf3ee0 c366ee7c
3e40: c3816098 00000000 c3816090 c0070994 91827364 c3bf3e54 c3bf3e54 c3bf3e5c
3e60: c3bf3e5c 00000000 c366edc8 c3bf3ee0 c366edc8 c00b3880 00000000 c3bf3ee0
3e80: c366edc8 c3bf2000 c3816058 c00b3fb0 c3687034 c3ac9000 c3816090 c3ac9040
3ea0: c3ac9000 c3816058 c3bf2000 c3bf3ee0 c3816090 c366edc8 c3816090 c00b464c
3ec0: 00000000 c3bf2000 00000400 00000000 c3816058 c3816098 c3bf3f4c c00b486c
3ee0: 00000000 c3bf3f1c ffffa7b2 000003ff 00000000 00000000 00000000 00000000
3f00: 00000000 00000000 00000024 00000000 c38b2fc0 00000017 00000000 ffff9bfa
3f20: c3bf3f5c c3bf3f4c 00001747 00000000 c3816058 00000000 00000000 c38160a0
3f40: 00000001 c00b4a80 c3816090 00001747 00000000 00000000 00000003 00000000
3f60: 00000000 00000000 ffffa7b2 c02c7380 c004093c 00000000 c3816058 c3bf2000
3f80: c3816008 c38160a0 c3816090 c3816098 c3816088 c00b4bb8 00000000 c381606c
3fa0: c3861f24 c3bf3fd4 c3861f24 c3816058 c00b4b38 00000000 00000000 00000000
3fc0: 00000000 c004e9f0 c002385c 00000000 c3816058 00000000 c3bf3fd8 c3bf3fd8
3fe0: 00000000 c3861f24 c004e970 c002385c 00000013 c002385c 23ffe831 23ffec31
[<c019ee94>] (vol_cdev_write+0x1e0/0x370) from [<00001000>] (0x1000)
Code: e1a0200b e3a0c003 e1a0000a e1a03009 (e98d1040)
---[ end trace 55d3f6b7614f1f3a ]---

About 91% into kobject_uevent_env() you are jumping to a subroutine identified in a structure, but the address is zero. Perhaps a timeout or callback.

ie foo->bar()

where foo is a defined structure, the field bar is NULL.

You should sanity check the value before calling, and then work back up through your code to identify why the field was not set.

if (!foo->bar)
kprintf("Bang!");
else
foo->bar(whatever);

Thanks for your insightful response, CptTitanic, this is greatly appreciated!

After much tinkering, I've been unable to track down exactly what is the issue. The problem always occurs after initiating a "cp -R" from the sdcard to NAND flash, or from a directory on the sdcard to another directory on the sdcard.

The problem also sporadically occurs when doing an "ls" of a directory contained on the root UBI partition of the NAND flash. This occurs only when the sdcard is mounted.

The kernel oops seems to be sporadic, and not associated with just one line of code. This mystifies me. Perhaps this problem could be caused by a compiler bug? I've tried compiling with the flag -fno-strict-aliasing as suggested in another post (http://www.cygwin.com/ml/crossgcc/2006-03/msg00114.html), but this creates another kernel oops (pasted below).

The problem never arises when performing "cp -R" from a directory on the UBI root file system to the sdcard.

I am compiling the kernel with the buildroot arm-linux-gcc compiler, but I've also tried the most recent compiler from CodeSourcery.

The UBI partitions have been created using U-Boot. Perhaps this could be a U-Boot related bug?

Here is the (rather long) output of another kernel oops when I've attempted to copy the "modules" directory on the sdcard to the root file system /home/ directory:

# cp -R ./modules /home/
Unable to handle kernel paging request at virtual address 645f679f
pgd = c3384000
[645f679f] *pgd=00000000
Internal error: Oops: 801 [#1] PREEMPT
last sysfs file: /sys/devices/virtual/vc/vcsa1/dev
Modules linked in:
CPU: 0 Not tainted (2.6.39-rc7+ #4)
PC is at fb_set_suspend+0x44/0x64
LR is at 0x6f640074
pc : [<c01462cc>] lr : [<6f640074>] psr: 40000013
sp : c335bd0c ip : 00000020 fp : 00000000
r10: c335bf40 r9 : 00000000 r8 : 74707572
r7 : 7265746e r6 : 695f6f6e r5 : 00617461 r4 : 645f6567
r3 : 72656d5f r2 : ffffff80 r1 : be9ec450 r0 : c2c01000
Flags: nZcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
Control: 0005317f Table: 23384000 DAC: 00000015
Process cp (pid: 853, stack limit = 0xc335a270)
Stack: (0xc335bd0c to 0xc335c000)
bd00: 00001000 00000000 c032c000 0001b000 c2c00000
bd20: 00001000 00000000 c335bd78 c006847c 00001000 00000000 00001000 00000000
bd40: 00000000 0001b000 c368f47c c0068640 00001000 00000000 c335bd8c c335bd88
bd60: 00001000 c335a000 c30ab4c0 00001000 00000000 c0207660 c335bf40 00000001
bd80: 00000000 00001000 c034f7e0 c032c000 c03540a0 0001c000 00000000 0001b000
bda0: 00000000 c368f3c8 00001000 00000000 00001000 c006a6c8 0001b000 00000000
bdc0: c335bf00 00001000 00000000 c01fa25c 00000030 c335bf00 c335bf40 c335bec8
bde0: c30ab4c0 c368f47c 00000000 00000001 c335bdf0 c335bdf0 c380da80 c335be6c
be00: 00000000 00001000 00000000 c368f3e0 00000001 c335bec8 c335bf40 00000000
be20: c30ab4c0 0001b000 00000000 c006a788 00001000 00000000 91827364 c335be3c
be40: c335be3c c335be44 c335be44 00000000 0001b000 00000000 c368f3c8 c3b25000
be60: c335bec8 c335bf40 00000001 c00f6abc 0001b000 00000000 c335be78 c335be78
be80: 00000000 00001000 00000000 be9ec458 00000000 00000000 c335bed4 c335bec8
bea0: c335bec8 c30ab4c0 c335bf40 c335bf80 fffffdee c335a000 00000000 c0090f28
bec0: 0001b000 00000000 c335beec c335bed8 00000000 00000001 ffffffff c30ab4c0
bee0: 00000000 00000000 00000000 00000000 c384bd60 00000000 00000000 00000000
bf00: 0001b000 00000000 00000001 c3694bc0 00001000 c335a000 00001000 00000001
bf20: c30abe48 00000001 c0091bec 00000000 00000000 00000000 c3bec000 c30abe48
bf40: be9eb458 00001000 c30ab4c0 00001000 be9eb458 c335bf80 be9eb458 c0091958
bf60: 00000000 00000000 0001b000 00000000 c30ab4c0 00001000 be9eb458 c0091a9c
bf80: 0001b000 00000000 be9eb458 00000000 00001000 be9eb458 0000000a 00000004
bfa0: c0023144 c0022fc0 00001000 be9eb458 0000000a be9eb458 00001000 000aa918
bfc0: 00001000 be9eb458 0000000a 00000004 0001b000 00000000 0000000a 00000000
bfe0: 000a92f8 be9eb418 0000ec88 401b3098 20000010 0000000a 00000000 00000000
[<c01462cc>] (fb_set_suspend+0x44/0x64) from [<00001000>] (0x1000)
Code: ebfffe25 e3a03001 e5843238 ea000003 (e5845238)
---[ end trace e9ade5ba914d9bfd ]---
note: cp[853] exited with preempt_count 2
BUG: scheduling while atomic: cp/853/0x40000003
Modules linked in:
[<c0026e8c>] (unwind_backtrace+0x0/0xec) from [<c01f94d8>] (schedule+0x58/0x41c)
[<c01f94d8>] (schedule+0x58/0x41c) from [<c002dd60>] (__cond_resched+0x18/0x24)
[<c002dd60>] (__cond_resched+0x18/0x24) from [<c01f9a30>] (_cond_resched+0x30/0x40)
[<c01f9a30>] (_cond_resched+0x30/0x40) from [<c0081a28>] (unmap_vmas+0x660/0x72c)
[<c0081a28>] (unmap_vmas+0x660/0x72c) from [<c0084330>] (exit_mmap+0x134/0x2a0)
[<c0084330>] (exit_mmap+0x134/0x2a0) from [<c0032c5c>] (mmput+0x3c/0xf8)
[<c0032c5c>] (mmput+0x3c/0xf8) from [<c0036c30>] (exit_mm+0x168/0x178)
[<c0036c30>] (exit_mm+0x168/0x178) from [<c0038744>] (do_exit+0x1dc/0x6bc)
[<c0038744>] (do_exit+0x1dc/0x6bc) from [<c0025d18>] (die+0x1bc/0x1ec)
[<c0025d18>] (die+0x1bc/0x1ec) from [<c0027f04>] (__do_kernel_fault+0x64/0x84)
[<c0027f04>] (__do_kernel_fault+0x64/0x84) from [<c0029d38>] (do_alignment+0x230/0x354)
[<c0029d38>] (do_alignment+0x230/0x354) from [<c00222e8>] (do_DataAbort+0x34/0x94)
[<c00222e8>] (do_DataAbort+0x34/0x94) from [<c0022b6c>] (__dabt_svc+0x4c/0x60)
Exception stack(0xc335bcc0 to 0xc335bd08)
bcc0: c2c01000 be9ec450 ffffff80 72656d5f 645f6567 00617461 695f6f6e 7265746e
bce0: 74707572 00000000 c335bf40 00000000 00000020 c335bd0c 6f640074 c01462cc
bd00: 40000013 ffffffff
[<c0022b6c>] (__dabt_svc+0x4c/0x60) from [<c01462cc>] (fb_set_suspend+0x44/0x64)
[<c01462cc>] (fb_set_suspend+0x44/0x64) from [<00001000>] (0x1000)
Unable to handle kernel NULL pointer dereference at virtual address 00000064
pgd = c0004000
[00000064] *pgd=00000000
Internal error: Oops: 817 [#2] PREEMPT
last sysfs file: /sys/devices/virtual/vc/vcsa1/dev
Modules linked in:
CPU: 0 Tainted: G D (2.6.39-rc7+ #4)
PC is at _clear_bit+0x4/0x38
LR is at schedule+0x38c/0x41c
pc : [<c012a860>] lr : [<c01f980c>] psr: a0000013
sp : c3bedf78 ip : 00000000 fp : c3bedfa4
r10: 00000000 r9 : c02aa520 r8 : c3bec000
r7 : c0000004 r6 : c30771e0 r5 : c384b2e0 r4 : c3077960
r3 : 00000000 r2 : 00000000 r1 : c307732c r0 : 00000000
Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel
Control: 0005317f Table: 23364000 DAC: 00000017
Process mmcqd/0 (pid: 788, stack limit = 0xc3bec270)
Stack: (0xc3bedf78 to 0xc3bee000)
df60: 00000001 00000000
df80: c3bec000 c3ad74a4 c3ad74ac c39323a8 00000001 00000000 00000000 c3bedfa8
dfa0: c01c12c8 c01f9490 c3bedfd4 c3829e0c c3ad74a4 c01c1204 00000000 00000000
dfc0: 00000000 c004e994 c002385c 00000000 c3ad74a4 00000000 c3bedfd8 c3bedfd8
dfe0: 00000000 c3829e0c c004e914 c002385c 00000013 c002385c 00000000 00000000
Code: 14e0646d e3a07079 e8bd726e e211726f (15cc0064)
---[ end trace e9ade5ba914d9bfe ]---
note: mmcqd/0[788] exited with preempt_count 1
Internal error: Oops - undefined instruction: 0 [#3] PREEMPT
last sysfs file: /sys/devices/virtual/vc/vcsa1/dev
Modules linked in:
CPU: 0 Tainted: G D (2.6.39-rc7+ #4)
PC is at _clear_bit+0x18/0x38
LR is at free_pidmap+0x20/0x40
pc : [<c012a874>] lr : [<c004b420>] psr: a0000093
sp : c3bedda8 ip : c3beddb0 fp : 00000150
r10: c3ad8060 r9 : 00000000 r8 : 00000000
r7 : c0000000 r6 : c012a87c r5 : c381c1e0 r4 : c02abf78
r3 : 00000000 r2 : 00000314 r1 : c380a000 r0 : 00000314
Flags: NzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment user
Control: 0005317f Table: 23364000 DAC: 00000015
Process mmcqd/0 (pid: 788, stack limit = 0xc3bec270)
Stack: (0xc3bedda8 to 0xc3bee000)
dda0: 00000001 c004b880 c3856a80 c384b2e0 c3856a80 c0037588
ddc0: 00000000 00000000 c3bedf30 ffffffff c3beddf4 c384b2e0 c3815d60 c384b3b8
dde0: c384b2d8 c384b3b8 00000001 c0038b80 c3bedf30 c3beddf4 c3beddf4 c384b400
de00: c3bedfa4 00000000 c384b2e0 c3bec000 c3bedf30 00000814 c3bedf30 c025bd1a
de20: c3bedfa4 c0025d18 c3bec270 0000000b c3bedfa4 00000064 c3bedf30 00000000
de40: 00000817 00000000 00000000 c0027f04 c02a88e8 00000064 c384b2e0 00000000
de60: 00000817 c00280dc 00000000 000000b5 c02a88e8 c3bedf64 00000817 c3bedf30
de80: 00000064 20000013 00000000 c00222e8 00000000 00000000 00000000 0000049d
dea0: 00000000 00000000 00000000 00000000 c3bede40 05f5e100 00000000 00000200
dec0: 000000cb 00000000 00000200 00019600 c3bede84 c3bede40 00000001 c3be4000
dee0: c02c5104 c3bedee8 c3bedee8 c3bedee8 00000000 0000000b 16a65700 00000000
df00: c3bec000 c384b310 c02aa520 00000001 c3ad74ac c384b2e0 ffffffff c3bedf64
df20: c30771e0 c0000004 c3bec000 c0022b6c 00000000 c307732c 00000000 00000000
df40: c3077960 c384b2e0 c30771e0 c0000004 c3bec000 c02aa520 00000000 c3bedfa4
df60: 00000000 c3bedf78 c01f980c c012a860 a0000013 ffffffff 00000001 00000000
df80: c3bec000 c3ad74a4 c3ad74ac c39323a8 00000001 00000000 00000000 c3bedfa8
dfa0: c01c12c8 c01f9490 c3bedfd4 c3829e0c c3ad74a4 c01c1204 00000000 00000000
dfc0: 00000000 c004e994 c002385c 00000000 c3ad74a4 00000001 c3bedfd8 c3bedfd8
dfe0: 00000000 c3829e0c c004e914 c002385c 00000013 c002385c 00000000 00000000
Code: e200676e e1a06574 e3a06e66 e1a0635f (e10f746e)
---[ end trace e9ade5ba914d9bff ]---
Fixing recursive fault but reboot is needed!
BUG: scheduling while atomic: klogd/840/0x00000004
Modules linked in:
[<c0026e8c>] (unwind_backtrace+0x0/0xec) from [<c01f94d8>] (schedule+0x58/0x41c)
[<c01f94d8>] (schedule+0x58/0x41c) from [<c003868c>] (do_exit+0x124/0x6bc)
[<c003868c>] (do_exit+0x124/0x6bc) from [<c0025d18>] (die+0x1bc/0x1ec)
[<c0025d18>] (die+0x1bc/0x1ec) from [<c0022200>] (do_undefinstr+0x154/0x174)
[<c0022200>] (do_undefinstr+0x154/0x174) from [<c0022c84>] (__und_svc+0x44/0x60)
Exception stack(0xc3bedd60 to 0xc3bedda8)
dd60: 00000314 c380a000 00000314 00000000 c02abf78 c381c1e0 c012a87c c0000000
dd80: 00000000 00000000 c3ad8060 00000150 c3beddb0 c3bedda8 c004b420 c012a874
dda0: a0000093 ffffffff
[<c0022c84>] (__und_svc+0x44/0x60) from [<c012a874>] (_clear_bit+0x18/0x38)
Unable to handle kernel NULL pointer dereference at virtual address 00000064
pgd = c0004000
[00000064] *pgd=00000000
Internal error: Oops: 817 [#4] PREEMPT
last sysfs file: /sys/devices/virtual/vc/vcsa1/dev
Modules linked in:
CPU: 0 Tainted: G D (2.6.39-rc7+ #4)
PC is at _clear_bit+0x4/0x38
LR is at schedule+0x38c/0x41c
pc : [<c012a860>] lr : [<c01f980c>] psr: a0000013
sp : c3bedc30 ip : 00000000 fp : c3bedc5c
r10: 00000000 r9 : c02aa520 r8 : c3bec000
r7 : c0000004 r6 : c3077960 r5 : c3839820 r4 : c30774e0
r3 : 00000000 r2 : 00000000 r1 : c3077aac r0 : 00000000
Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
Control: 0005317f Table: 23378000 DAC: 00000015
Process mmcqd/0 (pid: 788, stack limit = 0xc3bec270)
Stack: (0xc3bedc30 to 0xc3bee000)
dc20: c3bedd60 0000000b c384b2e0 c384b2e0
dc40: c3bedd60 00000000 c0022c84 c025b798 00000150 c3bedc60 c003868c c01f9490
dc60: c3bedd60 00000000 c0022c84 c02aaa94 00000150 00000000 c384b2e0 c3bec000
dc80: c3bedd60 00000000 c0022c84 c025b798 00000150 c0025d18 c3bec270 0000000b
dca0: 00000000 c3bedd60 60000093 00000000 e10f746e c012a874 c3ad8060 c0022200
dcc0: 00000006 00000001 00000004 00000000 00030001 c012a874 c02ac674 00000001
dce0: 00000003 00000001 00000000 00000001 00000003 00000002 00000000 c3bedd70
dd00: c3bedd7f 00000002 c02666b0 c3bedd8c c3bedd7f c3bedd20 c00a354c c002db9c
dd20: ff0a0210 ffffffff c384b2e0 c3bedd30 ff0a0210 00000558 c3bedd8c c3401698
dd40: c02c600c 00000001 ffffffff c3bedd94 c012a87c c0000000 00000000 c0022c84
dd60: 00000314 c380a000 00000314 00000000 c02abf78 c381c1e0 c012a87c c0000000
dd80: 00000000 00000000 c3ad8060 00000150 c3beddb0 c3bedda8 c004b420 c012a874
dda0: a0000093 ffffffff 00000001 c004b880 c3856a80 c384b2e0 c3856a80 c0037588
ddc0: 00000000 00000000 c3bedf30 ffffffff c3beddf4 c384b2e0 c3815d60 c384b3b8
dde0: c384b2d8 c384b3b8 00000001 c0038b80 c3bedf30 c3beddf4 c3beddf4 c384b400
de00: c3bedfa4 00000000 c384b2e0 c3bec000 c3bedf30 00000814 c3bedf30 c025bd1a
de20: c3bedfa4 c0025d18 c3bec270 0000000b c3bedfa4 00000064 c3bedf30 00000000
de40: 00000817 00000000 00000000 c0027f04 c02a88e8 00000064 c384b2e0 00000000
de60: 00000817 c00280dc 00000000 000000b5 c02a88e8 c3bedf64 00000817 c3bedf30
de80: 00000064 20000013 00000000 c00222e8 00000000 00000000 00000000 0000049d
dea0: 00000000 00000000 00000000 00000000 c3bede40 05f5e100 00000000 00000200
dec0: 000000cb 00000000 00000200 00019600 c3bede84 c3bede40 00000001 c3be4000
dee0: c02c5104 c3bedee8 c3bedee8 c3bedee8 00000000 0000000b 16a65700 00000000
df00: c3bec000 c384b310 c02aa520 00000001 c3ad74ac c384b2e0 ffffffff c3bedf64
df20: c30771e0 c0000004 c3bec000 c0022b6c 00000000 c307732c 00000000 00000000
df40: c3077960 c384b2e0 c30771e0 c0000004 c3bec000 c02aa520 00000000 c3bedfa4
df60: 00000000 c3bedf78 c01f980c c012a860 a0000013 ffffffff 00000001 00000000
df80: c3bec000 c3ad74a4 c3ad74ac c39323a8 00000001 00000000 00000000 c3bedfa8
dfa0: c01c12c8 c01f9490 c3bedfd4 c3829e0c c3ad74a4 c01c1204 00000000 00000000
dfc0: 00000000 c004e994 c002385c 00000000 c3ad74a4 00000001 c3bedfd8 c3bedfd8
dfe0: 00000000 c3829e0c c004e914 c002385c 00000013 c002385c 00000000 00000000
Code: 14e0646d e3a07079 e8bd726e e211726f (15cc0064)
---[ end trace e9ade5ba914d9c00 ]---
Fixing recursive fault but reboot is needed!
BUG: scheduling while atomic: tail/841/0x00000005
Modules linked in:
[<c0026e8c>] (unwind_backtrace+0x0/0xec) from [<c01f94d8>] (schedule+0x58/0x41c)
[<c01f94d8>] (schedule+0x58/0x41c) from [<c003868c>] (do_exit+0x124/0x6bc)
[<c003868c>] (do_exit+0x124/0x6bc) from [<c0025d18>] (die+0x1bc/0x1ec)
[<c0025d18>] (die+0x1bc/0x1ec) from [<c0027f04>] (__do_kernel_fault+0x64/0x84)
[<c0027f04>] (__do_kernel_fault+0x64/0x84) from [<c00280dc>] (do_page_fault+0x1b8/0x1d0)
[<c00280dc>] (do_page_fault+0x1b8/0x1d0) from [<c00222e8>] (do_DataAbort+0x34/0x94)
[<c00222e8>] (do_DataAbort+0x34/0x94) from [<c0022b6c>] (__dabt_svc+0x4c/0x60)
Exception stack(0xc3bedbe8 to 0xc3bedc30)
dbe0: 00000000 c3077aac 00000000 00000000 c30774e0 c3839820
dc00: c3077960 c0000004 c3bec000 c02aa520 00000000 c3bedc5c 00000000 c3bedc30
dc20: c01f980c c012a860 a0000013 ffffffff
[<c0022b6c>] (__dabt_svc+0x4c/0x60) from [<c012a860>] (_clear_bit+0x4/0x38)
Unable to handle kernel NULL pointer dereference at virtual address 00000064
pgd = c0004000
[00000064] *pgd=00000000
Internal error: Oops: 817 [#5] PREEMPT
last sysfs file: /sys/devices/virtual/vc/vcsa1/dev
Modules linked in:
CPU: 0 Tainted: G D (2.6.39-rc7+ #4)
PC is at _clear_bit+0x4/0x38
LR is at schedule+0x38c/0x41c
pc : [<c012a860>] lr : [<c01f980c>] psr: a0000013
sp : c382bf28 ip : 00000000 fp : c382bf54
r10: 00000000 r9 : c02aa520 r8 : c382a000
r7 : c0000004 r6 : c30774e0 r5 : c3815040 r4 : c30777e0
r3 : 00000000 r2 : 00000000 r1 : c307762c r0 : 00000000
Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel
Control: 0005317f Table: 23348000 DAC: 00000017
Process rcu_kthread (pid: 6, stack limit = 0xc382a270)
Stack: (0xc382bf28 to 0xc382c000)
bf20: c00537a4 ffff9e11 c02c3240 00000001 ffff9e10 c382bfa0
bf40: c004ebc4 00000000 00000000 c382bf58 c01fa148 c01f9490 00000000 00200200
bf60: ffff9e11 c02c3240 c00408fc c3815040 ffffffff a0000013 00000000 c3815040
bf80: c382a000 c02c5138 c382bf94 c0067958 c01f986c 00000000 c3815040 c004ebc4
bfa0: c382bfa0 c382bfa0 c382bfd4 c381bf74 00000000 c006788c 00000000 00000000
bfc0: 00000000 c004e994 c002385c 00000000 00000000 00000000 c382bfd8 c382bfd8
bfe0: 00000000 c381bf74 c004e914 c002385c 00000013 c002385c 33cc33cc 33cc33cc
Code: 14e0646d e3a07079 e8bd726e e211726f (15cc0064)
---[ end trace e9ade5ba914d9c01 ]---
note: rcu_kthread[6] exited with preempt_count 1
Internal error: Oops - undefined instruction: 0 [#6] PREEMPT
last sysfs file: /sys/devices/virtual/vc/vcsa1/dev
Modules linked in:
CPU: 0 Tainted: G D (2.6.39-rc7+ #4)
PC is at _clear_bit+0x18/0x38
LR is at free_pidmap+0x20/0x40
pc : [<c012a874>] lr : [<c004b420>] psr: a0000093
sp : c382bd58 ip : c382bd60 fp : 00000150
r10: c3825580 r9 : 00000000 r8 : 00000000
r7 : c0000000 r6 : c012a87c r5 : c381c3a0 r4 : c02abf78
r3 : 00000000 r2 : 00000006 r1 : c380a000 r0 : 00000006
Flags: NzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment user
Control: 0005317f Table: 23348000 DAC: 00000015
Process rcu_kthread (pid: 6, stack limit = 0xc382a270)
Stack: (0xc382bd58 to 0xc382c000)
bd40: 00000001 c004b880
bd60: c3819240 c3815040 c3819240 c0037588 00000000 00000000 c382bee0 ffffffff
bd80: c382bda4 c3815040 c3815d60 c3815118 c3815038 c3815118 00000001 c0038b80
bda0: c382bee0 c382bda4 c382bda4 c3815160 c382bf54 00000000 c3815040 c382a000
bdc0: c382bee0 00000806 c382bee0 c025bd1a c382bf54 c0025d18 c382a270 0000000b
bde0: c382bf54 00000064 c382bee0 00000000 00000817 00000000 00000000 c0027f04
be00: c02a88e8 00000064 c3815040 00000000 00000817 c00280dc 00000001 00000001
be20: c02a88e8 c382bf14 00000817 c382bee0 00000064 20000013 00000000 c00222e8
be40: 00000000 00000020 c02c5104 00000000 c382be7c c382be60 c002e368 c002db9c
be60: 00000000 c382be70 c3923800 00000001 00000001 c382be80 c01b6b00 c002e31c
be80: c38fe4a0 c01c1f88 00000104 00000000 ffdc0a00 00000001 c02ac674 c38fe4a0
bea0: c02ac9d4 00000000 00000001 0000000a 00000001 c02ac9d4 c382beec c382bec8
bec0: c002e82c c002db9c ffffffff c382bf14 c30774e0 c0000004 c382a000 c0022b6c
bee0: 00000000 c307762c 00000000 00000000 c30777e0 c3815040 c30774e0 c0000004
bf00: c382a000 c02aa520 00000000 c382bf54 00000000 c382bf28 c01f980c c012a860
bf20: a0000013 ffffffff c00537a4 ffff9e11 c02c3240 00000001 ffff9e10 c382bfa0
bf40: c004ebc4 00000000 00000000 c382bf58 c01fa148 c01f9490 00000000 00200200
bf60: ffff9e11 c02c3240 c00408fc c3815040 ffffffff a0000013 00000000 c3815040
bf80: c382a000 c02c5138 c382bf94 c0067958 c01f986c 00000000 c3815040 c004ebc4
bfa0: c382bfa0 c382bfa0 c382bfd4 c381bf74 00000000 c006788c 00000000 00000000
bfc0: 00000000 c004e994 c002385c 00000000 00000000 00000001 c382bfd8 c382bfd8
bfe0: 00000000 c381bf74 c004e914 c002385c 00000013 c002385c 33cc33cc 33cc33cc
Code: e200676e e1a06574 e3a06e66 e1a0635f (e10f746e)
---[ end trace e9ade5ba914d9c02 ]---
Fixing recursive fault but reboot is needed!
BUG: scheduling while atomic: sh/837/0x00000004
Modules linked in:
[<c0026e8c>] (unwind_backtrace+0x0/0xec) from [<c01f94d8>] (schedule+0x58/0x41c)
[<c01f94d8>] (schedule+0x58/0x41c) from [<c003868c>] (do_exit+0x124/0x6bc)
[<c003868c>] (do_exit+0x124/0x6bc) from [<c0025d18>] (die+0x1bc/0x1ec)
[<c0025d18>] (die+0x1bc/0x1ec) from [<c0022200>] (do_undefinstr+0x154/0x174)
[<c0022200>] (do_undefinstr+0x154/0x174) from [<c0022c84>] (__und_svc+0x44/0x60)
Exception stack(0xc382bd10 to 0xc382bd58)
bd00: 00000006 c380a000 00000006 00000000
bd20: c02abf78 c381c3a0 c012a87c c0000000 00000000 00000000 c3825580 00000150
bd40: c382bd60 c382bd58 c004b420 c012a874 a0000093 ffffffff
[<c0022c84>] (__und_svc+0x44/0x60) from [<c012a874>] (_clear_bit+0x18/0x38)
Internal error: Oops - undefined instruction: 0 [#7] PREEMPT
last sysfs file: /sys/devices/virtual/vc/vcsa1/dev
Modules linked in:
CPU: 0 Tainted: G D (2.6.39-rc7+ #4)
PC is at unbound_global_cwq+0x138/0x180
LR is at arm_syscall+0x48/0x314
pc : [<c02c43ec>] lr : [<c0025efc>] psr: a0000013
sp : c02c4fc8 ip : 00000000 fp : c382a000
r10: c02c3ffc r9 : c02c4000 r8 : 00000000
r7 : ffffffff r6 : c3815040 r5 : c02c50f0 r4 : ffffffff
r3 : c02c400c r2 : c02c4000 r1 : c02c50f0 r0 : ffffffff
Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
Control: 0005317f Table: 23348000 DAC: 00000015
Process ¬A,4A,4A,<A,À (pid: -1070841644, stack limit = 0xc02c4270)
Stack: (0xc02c4fc8 to 0xc02c600c)
4fc0: c02c4fc4 c02c4fcc c02c4fcc c02c4fd4 c02c4fd4 c02c4fdc
4fe0: c02c4fdc c02c4fe4 c02c4fe4 c02c4fec c02c4fec c02c4ff4 c02c4ff4 c02c4ffc
5000: c02c4ffc c02c5004 c02c5004 c02c500c c02c500c c02c5014 c02c5014 c02c501c
5020: c02c501c c02c5024 c02c5024 c02c502c c02c502c c02c5034 c02c5034 c02c503c
5040: c02c503c c02c5044 c02c5044 c02c504c c02c504c c02c5054 c02c5054 c02c505c
5060: c02c505c c02c5064 c02c5064 c02c506c c02c506c c02c5074 c02c5074 c02c507c
5080: c02c507c c02c5084 c02c5084 00000000 00000000 00000000 00000000 00000000
50a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
50c0: 00000000 00000000 00000000 00000000 00000000 c02c3240 c00408fc c3815040
50e0: ffffffff c002311c c02c3240 c00408fc 00000000 00000000 c382a000 00000000
5100: c02c3240 c00408fc c3815040 ffffffff a0000013 00000000 c3815040 c382a000
5120: c382bf60 be9eb418 0000ec88 c382c01c 60000013 00000000 00000001 00000000
5140: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000002
5160: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5180: 00000000 00000000 0000000d 00000000 00000002 00000000 00000000 00000000
51a0: 00000000 00000000 00000001 c02c51ac c02c51ac 00000000 00000000 00000000
51c0: 0000000d 00000000 000000fa 00000000 00000000 00000000 00000000 00000000
51e0: 00000001 c02c51e4 c02c51e4 00000000 00000000 00000000 00000003 00003f80
5200: 00000000 c380d760 c3811e60 00002eb1 00000005 0000008c 00000506 000000d0
5220: 00000000 00000000 0000008a 00000079 000005dc 000000f8 00000000 00000297
5240: 00000175 0000001e 00000022 00000000 00000000 00000000 00000000 00000000
5260: 00000000 00000007 000000fa 00000002 00000000 00000000 00000000 c3826920
5280: c3827580 c02c3b44 c39324a4 ffff9fee c02c3240 c007c9a8 00000000 ffffffff
52a0: c3401318 00000000 00000000 c3879260 00000000 c035e100 00000000 00000000
52c0: c02d4000 00004000 00004000 c4000000 00000000 00000000 00000000 00000000
52e0: 00000000 00000000 000001ac 00000000 c3800e40 c3800de0 00000000 c3868e80
5300: c38fecac c3be1ae0 0000001b c3be1aec 00003000 c4800000 00000001 fee00000
5320: 00004000 00000000 00004000 00000000 00000000 56e00000 04000000 00000001
5340: 00000080 c02c535c 00000002 00000080 c02c5764 00000000 00000001 20000000
5360: 04000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5380: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
53a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
53c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
53e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5400: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5420: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5440: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5460: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5480: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
54a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
54c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
54e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5500: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5520: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5540: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5560: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5580: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
55a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
55c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
55e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5600: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5620: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5640: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5660: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5680: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
56a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
56c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
56e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5700: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5720: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5740: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 9d74e35b
5760: 00000000 20004000 002cf9b4 23ffb000 00005000 00000000 00000000 00000000
5780: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
57a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
57c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
57e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5800: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5820: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5840: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5860: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5880: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
58a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
58c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
58e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5900: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5920: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5940: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5960: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5980: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
59a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
59c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
59e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5a00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5a20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5a40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5a60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5a80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5aa0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5ac0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5ae0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5b00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5b20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5b40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5b60: 00000000 9d74e35b 00000000 00000001 00000004 c3886dc4 c02b1868 00000000
5b80: 00000000 00000000 0000002a 00000000 00000000 c380ff60 c380f710 00000001
5ba0: 0000000d c3806620 0000000f 00000000 00000000 00000000 00000000 00000000
5bc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5be0: 00000000 00000000 00000000 00000000 00000000 00000000 c3804c00 00000000
5c00: c383a620 c3883ba0 c38db540 c383a1a0 c383a320 00000000 c383a0e0 00000000
5c20: 00000000 c383ab60 00000000 00000000 c383a980 00000000 00000000 00000000
5c40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5c60: 00000000 00000000 00000000 00000000 c383ac20 00000000 00000000 00000000
5c80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5ca0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5cc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5ce0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5d00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5d20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5d40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5d60: c3ad7aa0 c391ec00 00000000 00000000 00000000 00000000 00000000 00000000
5d80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5da0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5dc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5de0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 c3903e00
5e00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 c3903da0
5e20: 00000000 00000000 00000000 00000000 00000000 00000000 00000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5e60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5e80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5ea0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5ec0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5ee0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5f00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5f20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5f40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5f60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5f80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5fa0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5fe0: 00000000 00000000 c391e240 c391e360 c391e480 c383a920 00000000 00000000
6000: 00000000 00000000 00000000
[<c0025efc>] (arm_syscall+0x48/0x314) from [<c002311c>] (__sys_trace_return+0x0/0x24)
Code: 00000180 c02c3241 c0048e78 c02c42b4 (ffffffff)
---[ end trace e9ade5ba914d9c03 ]---
Kernel panic - not syncing: Fatal exception in interrupt
[<c0026e8c>] (unwind_backtrace+0x0/0xec) from [<c01f91dc>] (panic+0x54/0x188)
[<c01f91dc>] (panic+0x54/0x188) from [<c0025d04>] (die+0x1a8/0x1ec)
[<c0025d04>] (die+0x1a8/0x1ec) from [<c0022200>] (do_undefinstr+0x154/0x174)
[<c0022200>] (do_undefinstr+0x154/0x174) from [<c0022c84>] (__und_svc+0x44/0x60)
Exception stack(0xc02c4f80 to 0xc02c4fc8)
4f80: ffffffff c02c50f0 c02c4000 c02c400c ffffffff c02c50f0 c3815040 ffffffff
4fa0: 00000000 c02c4000 c02c3ffc c382a000 00000000 c02c4fc8 c0025efc c02c43ec
4fc0: a0000013 ffffffff
[<c0022c84>] (__und_svc+0x44/0x60) from [<c02c43ec>] (unbound_global_cwq+0x138/0x180)

It's usually not compiler bugs, but bad code that causes these problems.

Ok, I don't see a point in analyzing the whole log. It fails, and then subsequently crashes and burns in the aftermath. So lets look at what happen when it failed.

Unable to handle kernel paging request at virtual address 645f679f
pgd = c3384000
[645f679f] *pgd=00000000
Internal error: Oops: 801 [#1] PREEMPT
last sysfs file: /sys/devices/virtual/vc/vcsa1/dev
Modules linked in:
CPU: 0 Not tainted (2.6.39-rc7+ #4)
PC is at fb_set_suspend+0x44/0x64
LR is at 0x6f640074
pc : [<c01462cc>] lr : [<6f640074>] psr: 40000013
sp : c335bd0c ip : 00000020 fp : 00000000
r10: c335bf40 r9 : 00000000 r8 : 74707572
r7 : 7265746e r6 : 695f6f6e r5 : 00617461 r4 : 645f6567
r3 : 72656d5f r2 : ffffff80 r1 : be9ec450 r0 : c2c01000

00000000 EBFFFE25 bl sub_FFFFF89C
00000004 E3A03001 mov r3, #1 ; $1
00000008 E5943238 ldr r3, [r4, #568] ; FAILS HERE
0000000C EA000003 b loc_000020

R4 =0x645F6567 , looks like ASCII (ge_d) characters pulled from a buffer
R4 + 568 = 0x645F679F, looks like your failing access

ARM9's do not like to do un-aligned memory accesses, and trying to load R3 from the content of this bogus un-aligned address will cause it to fail.

So I'd guess you are corrupting your stack, or overflowing a buffer. R4 really should be some structure, presumably "info" in fb_set_suspend().

info->state = FBINFO_STATE_SUSPENDED; // FAILS HERE

Why this fails with CP, is probably just a consequence of your SD access trashing the memory used by the video driver.

Thanks again for your help, CptTitanic; this is very much appreciated!

To try and diagnose the problem, I went into the Linux kernel xconfig and switched off both "Support for Frame Buffer Devices" and "AT91/AT32 LCD Controller Support." So far, so good: after compiling the kernel and transferring it to the embedded system, the frame buffer kernel oops disappears.

However, another kernel oops now occurs when I try the copy operation. The mysterious thing about all of this is that the kernel oops is not the same every time I try the copy operation! Perhaps this might point to a bug in the RCU subsystem? Perhaps this is associated with IRQ handling, or maybe even the watchdog?

Could something be changed in the kernel xconfig to stop this from happening?

# cp -R ./modules/ ./test/
Unable to handle kernel NULL pointer dereference at virtual address 00000004
pgd = c0004000
[00000004] *pgd=00000000
Internal error: Oops: 817 [#1] PREEMPT
last sysfs file: /sys/devices/virtual/vc/vcsa1/dev
Modules linked in:
CPU: 0 Not tainted (2.6.39-rc7+ #5)
PC is at free_hot_cold_page+0x144/0x1a0
LR is at 0x0
pc : [<c006dc58>] lr : [<00000000>] psr: 60000093
sp : c382bf00 ip : c0332398 fp : 00100100
r10: c3805b70 r9 : 00200200 r8 : 00000000
r7 : 20000093 r6 : c02b373c r5 : c0332380 r4 : c0342000
r3 : c0342008 r2 : c034200c r1 : 00000000 r0 : 00000001
Flags: nZCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment kernel
Control: 0005317f Table: 23350000 DAC: 00000017
Process rcu_kthread (pid: 6, stack limit = 0xc382a270)
Stack: (0xc382bf00 to 0xc382c000)
bf00: c341c000 c3800960 c3800960 c38009ac 0000001b c008cab8 c382bf54 c3802fa0
bf20: 00000004 c008cbd8 c00527a4 00000000 0000001b c3805b60 c3800960 c382bfa0
bf40: c004dbc4 00000000 00000000 c008c908 c3800960 c3805b60 80000013 c3800960
bf60: c343d8c8 c008ca50 00000000 c343b1a0 c343d914 c02bac78 c382bf94 c0066874
bf80: c382a000 c3815040 c382a000 c0066948 c01f2974 00000000 c3815040 c004dbc4
bfa0: c382bfa0 c382bfa0 c382bfd4 c381bf74 00000000 c006688c 00000000 00000000
bfc0: 00000000 c004d994 c002285c 00000000 00000000 00000000 c382bfd8 c382bfd8
bfe0: 00000000 c381bf74 c004d914 c002285c 00000013 c002285c 33cc33cc 33cc33cc
[<c006dc58>] (free_hot_cold_page+0x144/0x1a0) from [<c008cab8>] (slab_destroy+0x48/0x74)
[<c008cab8>] (slab_destroy+0x48/0x74) from [<c008cbd8>] (free_block+0xf4/0x14c)
[<c008cbd8>] (free_block+0xf4/0x14c) from [<c008c908>] (cache_flusharray+0x78/0xc0)
[<c008c908>] (cache_flusharray+0x78/0xc0) from [<c008ca50>] (kmem_cache_free+0x44/0x64)
[<c008ca50>] (kmem_cache_free+0x44/0x64) from [<c0066874>] (rcu_process_callbacks+0x6c/0x84)
[<c0066874>] (rcu_process_callbacks+0x6c/0x84) from [<c0066948>] (rcu_kthread+0xbc/0xe4)
[<c0066948>] (rcu_kthread+0xbc/0xe4) from [<c004d994>] (kthread+0x80/0x88)
[<c004d994>] (kthread+0x80/0x88) from [<c002285c>] (kernel_thread_exit+0x0/0x8)
Code: ea000006 e0843180 e5931004 e2832004 (e581c004)
---[ end trace 655c579d4e82550d ]---
Kernel panic - not syncing: Fatal exception in interrupt
[<c0025e8c>] (unwind_backtrace+0x0/0xec) from [<c01f22e4>] (panic+0x54/0x188)
[<c01f22e4>] (panic+0x54/0x188) from [<c0024d04>] (die+0x1a8/0x1ec)
[<c0024d04>] (die+0x1a8/0x1ec) from [<c0026f04>] (__do_kernel_fault+0x64/0x84)
[<c0026f04>] (__do_kernel_fault+0x64/0x84) from [<c00270dc>] (do_page_fault+0x1b8/0x1d0)
[<c00270dc>] (do_page_fault+0x1b8/0x1d0) from [<c00212e8>] (do_DataAbort+0x34/0x94)
[<c00212e8>] (do_DataAbort+0x34/0x94) from [<c0021b6c>] (__dabt_svc+0x4c/0x60)
Exception stack(0xc382beb8 to 0xc382bf00)
bea0: 00000001 00000000
bec0: c034200c c0342008 c0342000 c0332380 c02b373c 20000093 00000000 00200200
bee0: c3805b70 00100100 c0332398 c382bf00 00000000 c006dc58 60000093 ffffffff
[<c0021b6c>] (__dabt_svc+0x4c/0x60) from [<c006dc58>] (free_hot_cold_page+0x144/0x1a0)
[<c006dc58>] (free_hot_cold_page+0x144/0x1a0) from [<c008cab8>] (slab_destroy+0x48/0x74)
[<c008cab8>] (slab_destroy+0x48/0x74) from [<c008cbd8>] (free_block+0xf4/0x14c)
[<c008cbd8>] (free_block+0xf4/0x14c) from [<c008c908>] (cache_flusharray+0x78/0xc0)
[<c008c908>] (cache_flusharray+0x78/0xc0) from [<c008ca50>] (kmem_cache_free+0x44/0x64)
[<c008ca50>] (kmem_cache_free+0x44/0x64) from [<c0066874>] (rcu_process_callbacks+0x6c/0x84)
[<c0066874>] (rcu_process_callbacks+0x6c/0x84) from [<c0066948>] (rcu_kthread+0xbc/0xe4)
[<c0066948>] (rcu_kthread+0xbc/0xe4) from [<c004d994>] (kthread+0x80/0x88)
[<c004d994>] (kthread+0x80/0x88) from [<c002285c>] (kernel_thread_exit+0x0/0x8)


Here is another kernel oops:

# cp -R ./modules/ ./test/
Unable to handle kernel paging request at virtual address 058d3024
Unable to handle kernel paging request at virtual address b0400000
pgd = c3bf4000
[b0400000] *pgd=00000000
Internal error: Oops: 5 [#1] PREEMPT
last sysfs file: /sys/devices/virtual/vc/vcsa1/dev
Modules linked in:
CPU: 0 Not tainted (2.6.39-rc7+ #5)
PC is at check_preempt_curr+0x58/0xa4
LR is at try_to_wake_up+0x74/0xec
pc : [<c002bafc>] lr : [<c002f9a4>] psr: 60000093
sp : c334b9f8 ip : c01f5618 fp : c334ba14
r10: 00000000 r9 : 00000001 r8 : c02a0780
r7 : 00000000 r6 : c3af0006 r5 : c02a0248 r4 : 0248c3af
r3 : b0400000 r2 : 60000093 r1 : 60000093 r0 : c02a0248
Flags: nZCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment user
Control: 0005317f Table: 23bf4000 DAC: 00000015
Process cp (pid: 848, stack limit = 0xc334a270)
Stack: (0xc334b9f8 to 0xc334c000)
b9e0: c002c828 00000001
ba00: c3afed60 c02a0248 c334ba3c c334ba18 c002f9a4 c002bab4 c029f2f8 20000093
ba20: c3099f6c c02a0774 00000000 00000001 c334ba74 c334ba40 c004dbd0 c002f940
ba40: c3099f6c c002bf14 00000000 c02a0780 00000001 00000001 00000000 20000093
ba60: c02bac44 00000000 c334baa4 c334ba78 c002d47c c002bee8 00000000 00000000
ba80: 00000000 00000000 00000000 c3afe820 c02b418c c029f2f8 00000001 c334baa8
baa0: c003f9c4 c002d440 00000001 c02b4190 c029f2f8 c005969c c39199e0 c3b23eb4
bac0: c334a000 c334bad0 c01bae14 00000001 c02b4190 c029f2f8 c02b418c c02a239c
bae0: c02bac44 00000000 00000001 c002afdc 00000001 c029f2c8 c02a239c 00000001
bb00: 00000000 c0063264 00000102 c3345a00 c334bb8c c029f2c8 c02a239c 00000001
bb20: 00000004 00000003 c334a000 00000049 00000000 c0063420 00000000 c02a239c
bb40: 00000000 c0065694 00000001 c0021078 00002835 ffffffff fefff000 c0021bc8
bb60: 00000001 00000003 c334a000 00000000 20000013 c02a07dc 00000043 00000046
bb80: 00000003 c02b880b 00000049 00000000 c334bba0 c334bba8 c0033f44 c0034d18
bba0: 60000013 ffffffff 00000000 20000013 c392a3a8 00000000 00000000 00000000
bbc0: 00000001 000001ff c3345a00 00000010 00000000 0000000f 00000000 c36a3500
bbe0: c3375000 c00e8040 00000001 00002e7d 00000000 058d3024 c334bd10 c3076060
bc00: 80000005 00000000 c334bd10 c334bf40 00000000 c01f2430 c3076060 c0026ee8
bc20: c0253172 c025314a 058d3024 c025314a c029e508 058d3024 c3afe820 c3076060
bc40: 80000005 c00270dc c0067eb8 c002cb9c c029e508 c334bd44 00000005 c334bd10
bc60: 058d3024 60000013 c334bf40 c0021254 c0322020 00000200 00000000 c00b79c4
bc80: 00000000 c00b827c c36bb848 00000001 c0322020 fffff000 c36a3500 00001000
bca0: 00000010 00000000 c36bb688 c334bcc4 c334a000 fffffe00 00001200 fffff000
bcc0: 00000000 c00685ec 000000d0 c36bb688 00001000 c0322020 00001000 00000000
bce0: 00000000 00001000 c36a35b4 00001000 00000000 c00b86bc ffffffff c334bd44
bd00: e5933024 e1530002 059d3024 c0021cf0 c2c02000 beba6448 ffffff80 00000001
bd20: e5923018 e5932004 e5933024 e1530002 059d3024 00000000 c334bf40 00000000
bd40: 00000020 c334bd5c 058d3024 058d3024 a0000013 ffffffff 058d3024 00001000
bd60: 00000000 c0322020 00001000 c2c01000 00001000 00000000 c334bdc8 c006747c
bd80: c00eeb1c 00000000 00001000 00000000 00000000 00001000 c36a35b4 c0067640
bda0: 00001000 00000000 c334bddc c334bdd8 c334a000 c334a000 c3374ce0 00001000
bdc0: 00000000 c01fcc54 c334bf40 00000001 00000000 00001000 c36a3500 c0322020
bde0: 00001000 00002000 00000000 00001000 00000000 c36a3500 00001000 00000000
be00: 00001000 c00696c8 00001000 00000000 c334bf00 00001000 00000000 c0069f2c
be20: 00000001 c334bf00 c334bf40 c334bec8 c3374ce0 c36a35b4 00000000 00000001
be40: c334bf00 c334a000 00000002 c3374ba0 c36a3660 00001000 00001000 c36a3518
be60: 00000001 c334bec8 c334bf40 00000000 c3374ce0 00001000 00000000 c0069788
be80: 00000000 00001000 91827364 c334be8c c334be8c c334be94 c334be94 00000000
bea0: c334bec8 c3374ce0 c334bf40 c334bf80 fffffdee c334a000 00000000 c008ff28
bec0: 00001000 00000000 c3b61032 00000001 00000000 00000001 ffffffff c3374ce0
bee0: 00000000 00000000 00000000 00000000 c3afe820 00000000 00000000 00000000
bf00: 00001000 00000000 c3b61032 00000000 00001000 c36a3d40 00001000 00000010
bf20: c3374b68 00000001 c0022144 c3afe820 c387fd98 00000002 c3b22000 ffffffff
bf40: beba5448 00001000 c3374ce0 00001000 beba5448 c334bf80 beba5448 c0090958
bf60: c3afe820 00000017 00001000 00000000 c3374ce0 00001000 beba5448 c0090a9c
bf80: 00001000 00000000 beba5448 00000000 00001000 beba5448 0000000a 00000004
bfa0: c0022144 c0021fc0 00001000 beba5448 0000000a beba5448 00001000 000aa918
bfc0: 00001000 beba5448 0000000a 00000004 00001000 00000000 0000000a 00000000
bfe0: 000a92f8 beba5408 0000ec88 40177098 20000010 0000000a 23ffe831 23ffec31
[<c002bafc>] (check_preempt_curr+0x58/0xa4) from [<c002f9a4>] (try_to_wake_up+0x74/0xec)
[<c002f9a4>] (try_to_wake_up+0x74/0xec) from [<c004dbd0>] (autoremove_wake_function+0xc/0x34)
[<c004dbd0>] (autoremove_wake_function+0xc/0x34) from [<c002bf14>] (__wake_up_common+0x3c/0x80)
[<c002bf14>] (__wake_up_common+0x3c/0x80) from [<c002d47c>] (__wake_up+0x4c/0x80)
[<c002d47c>] (__wake_up+0x4c/0x80) from [<c003f9c4>] (update_process_times+0x3c/0x4c)
[<c003f9c4>] (update_process_times+0x3c/0x4c) from [<c005969c>] (tick_handle_periodic+0x18/0x98)
[<c005969c>] (tick_handle_periodic+0x18/0x98) from [<c002afdc>] (at91sam926x_pit_interrupt+0x88/0xb8)
[<c002afdc>] (at91sam926x_pit_interrupt+0x88/0xb8) from [<c0063264>] (handle_irq_event_percpu+0x34/0x19c)
[<c0063264>] (handle_irq_event_percpu+0x34/0x19c) from [<c0063420>] (handle_irq_event+0x54/0x78)
[<c0063420>] (handle_irq_event+0x54/0x78) from [<c0065694>] (handle_level_irq+0xc0/0x108)
[<c0065694>] (handle_level_irq+0xc0/0x108) from [<c0021078>] (asm_do_IRQ+0x78/0xac)
[<c0021078>] (asm_do_IRQ+0x78/0xac) from [<c0021bc8>] (__irq_svc+0x48/0x8c)
Exception stack(0xc334bb60 to 0xc334bba8)
bb60: 00000001 00000003 c334a000 00000000 20000013 c02a07dc 00000043 00000046
bb80: 00000003 c02b880b 00000049 00000000 c334bba0 c334bba8 c0033f44 c0034d18
bba0: 60000013 ffffffff
[<c0021bc8>] (__irq_svc+0x48/0x8c) from [<c0034d18>] (vprintk+0x398/0x428)
[<c0034d18>] (vprintk+0x398/0x428) from [<c01f2430>] (printk+0x18/0x28)
[<c01f2430>] (printk+0x18/0x28) from [<c0026ee8>] (__do_kernel_fault+0x48/0x84)
[<c0026ee8>] (__do_kernel_fault+0x48/0x84) from [<c00270dc>] (do_page_fault+0x1b8/0x1d0)
[<c00270dc>] (do_page_fault+0x1b8/0x1d0) from [<c0021254>] (do_PrefetchAbort+0x34/0x94)
[<c0021254>] (do_PrefetchAbort+0x34/0x94) from [<c0021cf0>] (__pabt_svc+0x50/0x80)
Exception stack(0xc334bd10 to 0xc334bd58)
bd00: c2c02000 beba6448 ffffff80 00000001
bd20: e5923018 e5932004 e5933024 e1530002 059d3024 00000000 c334bf40 00000000
bd40: 00000020 c334bd5c 058d3024 058d3024 a0000013 ffffffff
[<c0021cf0>] (__pabt_svc+0x50/0x80) from [<058d3024>] (0x58d3024)
Code: e5963004 e10f1000 e3812080 e121f002 (e5932000)
---[ end trace 4d78efb396ecc74c ]---
Kernel panic - not syncing: Fatal exception in interrupt
[<c0025e8c>] (unwind_backtrace+0x0/0xec) from [<c01f22e4>] (panic+0x54/0x188)
[<c01f22e4>] (panic+0x54/0x188) from [<c0024d04>] (die+0x1a8/0x1ec)
[<c0024d04>] (die+0x1a8/0x1ec) from [<c0026f04>] (__do_kernel_fault+0x64/0x84)
[<c0026f04>] (__do_kernel_fault+0x64/0x84) from [<c00270dc>] (do_page_fault+0x1b8/0x1d0)
[<c00270dc>] (do_page_fault+0x1b8/0x1d0) from [<c00212e8>] (do_DataAbort+0x34/0x94)
[<c00212e8>] (do_DataAbort+0x34/0x94) from [<c0021b6c>] (__dabt_svc+0x4c/0x60)
Exception stack(0xc334b9b0 to 0xc334b9f8)
b9a0: c02a0248 60000093 60000093 b0400000
b9c0: 0248c3af c02a0248 c3af0006 00000000 c02a0780 00000001 00000000 c334ba14
b9e0: c01f5618 c334b9f8 c002f9a4 c002bafc 60000093 ffffffff
[<c0021b6c>] (__dabt_svc+0x4c/0x60) from [<c002bafc>] (check_preempt_curr+0x58/0xa4)
[<c002bafc>] (check_preempt_curr+0x58/0xa4) from [<c002f9a4>] (try_to_wake_up+0x74/0xec)
[<c002f9a4>] (try_to_wake_up+0x74/0xec) from [<c004dbd0>] (autoremove_wake_function+0xc/0x34)
[<c004dbd0>] (autoremove_wake_function+0xc/0x34) from [<c002bf14>] (__wake_up_common+0x3c/0x80)
[<c002bf14>] (__wake_up_common+0x3c/0x80) from [<c002d47c>] (__wake_up+0x4c/0x80)
[<c002d47c>] (__wake_up+0x4c/0x80) from [<c003f9c4>] (update_process_times+0x3c/0x4c)
[<c003f9c4>] (update_process_times+0x3c/0x4c) from [<c005969c>] (tick_handle_periodic+0x18/0x98)
[<c005969c>] (tick_handle_periodic+0x18/0x98) from [<c002afdc>] (at91sam926x_pit_interrupt+0x88/0xb8)
[<c002afdc>] (at91sam926x_pit_interrupt+0x88/0xb8) from [<c0063264>] (handle_irq_event_percpu+0x34/0x19c)
[<c0063264>] (handle_irq_event_percpu+0x34/0x19c) from [<c0063420>] (handle_irq_event+0x54/0x78)
[<c0063420>] (handle_irq_event+0x54/0x78) from [<c0065694>] (handle_level_irq+0xc0/0x108)
[<c0065694>] (handle_level_irq+0xc0/0x108) from [<c0021078>] (asm_do_IRQ+0x78/0xac)
[<c0021078>] (asm_do_IRQ+0x78/0xac) from [<c0021bc8>] (__irq_svc+0x48/0x8c)
Exception stack(0xc334bb60 to 0xc334bba8)
bb60: 00000001 00000003 c334a000 00000000 20000013 c02a07dc 00000043 00000046
bb80: 00000003 c02b880b 00000049 00000000 c334bba0 c334bba8 c0033f44 c0034d18
bba0: 60000013 ffffffff
[<c0021bc8>] (__irq_svc+0x48/0x8c) from [<c0034d18>] (vprintk+0x398/0x428)
[<c0034d18>] (vprintk+0x398/0x428) from [<c01f2430>] (printk+0x18/0x28)
[<c01f2430>] (printk+0x18/0x28) from [<c0026ee8>] (__do_kernel_fault+0x48/0x84)
[<c0026ee8>] (__do_kernel_fault+0x48/0x84) from [<c00270dc>] (do_page_fault+0x1b8/0x1d0)
[<c00270dc>] (do_page_fault+0x1b8/0x1d0) from [<c0021254>] (do_PrefetchAbort+0x34/0x94)
[<c0021254>] (do_PrefetchAbort+0x34/0x94) from [<c0021cf0>] (__pabt_svc+0x50/0x80)
Exception stack(0xc334bd10 to 0xc334bd58)
bd00: c2c02000 beba6448 ffffff80 00000001
bd20: e5923018 e5932004 e5933024 e1530002 059d3024 00000000 c334bf40 00000000
bd40: 00000020 c334bd5c 058d3024 058d3024 a0000013 ffffffff
[<c0021cf0>] (__pabt_svc+0x50/0x80) from [<058d3024>] (0x58d3024)


Here's another kernel oops:

# cp -R ./modules/ ./test/
Unable to handle kernel paging request at virtual address e1a05009
pgd = c3340000
[e1a05009] *pgd=00000000
Internal error: Oops: 801 [#1] PREEMPT
last sysfs file: /sys/devices/virtual/vc/vcsa1/dev
Modules linked in:
CPU: 0 Not tainted (2.6.39-rc7+ #5)
PC is at prandom32+0x48/0x5c
LR is at 0xe1a03083
pc : [<c0135934>] lr : [<e1a03083>] psr: 40000013
sp : c335bd5c ip : 00000020 fp : 00000000
r10: c335bf40 r9 : 00000000 r8 : e591a01c
r7 : e5d13018 r6 : ea000008 r5 : e1a05001 r4 : e59fa650
r3 : e59fa670 r2 : ffffff80 r1 : bebd9448 r0 : 2017d7b2
Flags: nZcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
Control: 0005317f Table: 23340000 DAC: 00000015
Process cp (pid: 847, stack limit = 0xc335a270)
Stack: (0xc335bd5c to 0xc335c000)
bd40: 00001000
bd60: 00000000 c0322000 00002000 c2c00000 00001000 00000000 c335bdc8 c006747c
bd80: c00eeb1c 00000000 00001000 00000000 00000000 00002000 c36a35b4 c0067640
bda0: 00001000 00000000 c335bddc c335bdd8 c335a000 c335a000 c3092440 00001000
bdc0: 00000000 c01fcc54 c335bf40 00000001 00000000 00001000 c36a3500 c0322000
bde0: 00002000 00003000 00000000 00002000 00000000 c36a3500 00001000 00000000
be00: 00001000 c00696c8 00002000 00000000 c335bf00 00001000 00000000 c0069f2c
be20: 00000001 c335bf00 c335bf40 c335bec8 c3092440 c36a35b4 00000000 00000001
be40: c335bf00 c335a000 00000003 c3092700 c36a3660 00001000 00001000 c36a3518
be60: 00000001 c335bec8 c335bf40 00000000 c3092440 00002000 00000000 c0069788
be80: 00000000 00001000 91827364 c335be8c c335be8c c335be94 c335be94 00000000
bea0: c335bec8 c3092440 c335bf40 c335bf80 fffffdee c335a000 00000000 c008ff28
bec0: 00002000 00000000 c3aed032 00000001 00000000 00000001 ffffffff c3092440
bee0: 00000000 00000000 00000000 00000000 c30ab040 00000000 00000000 00000000
bf00: 00002000 00000000 c3aed032 00000000 00001000 c36a3d40 00001000 00000001
bf20: c30926c8 00000001 c0022144 00000000 00000000 00000000 c3be8000 c30926c8
bf40: bebd8448 00001000 c3092440 00001000 bebd8448 c335bf80 bebd8448 c0090958
bf60: 00000000 00000000 00002000 00000000 c3092440 00001000 bebd8448 c0090a9c
bf80: 00002000 00000000 bebd8448 00000000 00001000 bebd8448 0000000a 00000004
bfa0: c0022144 c0021fc0 00001000 bebd8448 0000000a bebd8448 00001000 000aa918
bfc0: 00001000 bebd8448 0000000a 00000004 00002000 00000000 0000000a 00000000
bfe0: 000a92f8 bebd8408 0000ec88 4010a098 20000010 0000000a 00000000 00000000
[<c0135934>] (prandom32+0x48/0x5c) from [<c0322000>] (0xc0322000)
Code: e0211181 e1a00880 e18005a1 e024300c (e5850008)
---[ end trace 2202e91d1a551745 ]---
note: cp[847] exited with preempt_count 2
BUG: scheduling while atomic: cp/847/0x40000003
Modules linked in:
[<c0025e8c>] (unwind_backtrace+0x0/0xec) from [<c01f25e0>] (schedule+0x58/0x41c)
[<c01f25e0>] (schedule+0x58/0x41c) from [<c002cd60>] (__cond_resched+0x18/0x24)
[<c002cd60>] (__cond_resched+0x18/0x24) from [<c01f2b38>] (_cond_resched+0x30/0x40)
[<c01f2b38>] (_cond_resched+0x30/0x40) from [<c0080a28>] (unmap_vmas+0x660/0x72c)
[<c0080a28>] (unmap_vmas+0x660/0x72c) from [<c0083330>] (exit_mmap+0x134/0x2a0)
[<c0083330>] (exit_mmap+0x134/0x2a0) from [<c0031c5c>] (mmput+0x3c/0xf8)
[<c0031c5c>] (mmput+0x3c/0xf8) from [<c0035c30>] (exit_mm+0x168/0x178)
[<c0035c30>] (exit_mm+0x168/0x178) from [<c0037744>] (do_exit+0x1dc/0x6bc)
[<c0037744>] (do_exit+0x1dc/0x6bc) from [<c0024d18>] (die+0x1bc/0x1ec)
[<c0024d18>] (die+0x1bc/0x1ec) from [<c0026f04>] (__do_kernel_fault+0x64/0x84)
[<c0026f04>] (__do_kernel_fault+0x64/0x84) from [<c0028d38>] (do_alignment+0x230/0x354)
[<c0028d38>] (do_alignment+0x230/0x354) from [<c00212e8>] (do_DataAbort+0x34/0x94)
[<c00212e8>] (do_DataAbort+0x34/0x94) from [<c0021b6c>] (__dabt_svc+0x4c/0x60)
Exception stack(0xc335bd10 to 0xc335bd58)
bd00: 2017d7b2 bebd9448 ffffff80 e59fa670
bd20: e59fa650 e1a05001 ea000008 e5d13018 e591a01c 00000000 c335bf40 00000000
bd40: 00000020 c335bd5c e1a03083 c0135934 40000013 ffffffff
[<c0021b6c>] (__dabt_svc+0x4c/0x60) from [<c0135934>] (prandom32+0x48/0x5c)
[<c0135934>] (prandom32+0x48/0x5c) from [<c0322000>] (0xc0322000)
Unable to handle kernel NULL pointer dereference at virtual address 00000ff0
pgd = c0004000
[00000ff0] *pgd=00000000
Internal error: Oops: 817 [#2] PREEMPT
last sysfs file: /sys/devices/virtual/vc/vcsa1/dev
Modules linked in:
CPU: 0 Tainted: G D (2.6.39-rc7+ #5)
PC is at _clear_bit+0x4/0x38
LR is at schedule+0x38c/0x41c
pc : [<c0129860>] lr : [<c01f2914>] psr: 00000013
sp : c382bf60 ip : 00000000 fp : c382bf8c
r10: 00000000 r9 : c02a0248 r8 : c382a000
r7 : c30abd60 r6 : c30761e0 r5 : c3815040 r4 : c30764e0
r3 : 00000000 r2 : 00000000 r1 : c307632c r0 : 00000200
Flags: nzcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel
Control: 0005317f Table: 23374000 DAC: 00000017
Process rcu_kthread (pid: 6, stack limit = 0xc382a270)
Stack: (0xc382bf60 to 0xc382c000)
bf60: ffff9a1f c3815040 c382a000 c02bac78 c382bf94 c382bfa0 c004dbc4 00000000
bf80: 00000000 c382bf90 c0066910 c01f2598 c01f2974 00000000 c3815040 c004dbc4
bfa0: c02a6c28 c02a6c28 c382bfd4 c381bf74 00000000 c006688c 00000000 00000000
bfc0: 00000000 c004d994 c002285c 00000000 00000000 00000000 c382bfd8 c382bfd8
bfe0: 00000000 c381bf74 c004d914 c002285c 00000013 c002285c 33cc33cc 33cc33cc
Code: 14e00000 e3a00800 e8bd0174 e2110c02 (15cc4ff0)
---[ end trace 2202e91d1a551746 ]---
note: rcu_kthread[6] exited with preempt_count 1

The problem isn't in the Frame Buffer device, you are trashing the memory arena. Removing the module just moves the memory, and problem, around, it does not address the issue of your SD driver, or those stacked around it, from trashing the system.

You need to focus on the root cause of this, not the ghosts/damage it manifests as a result.

You have found a way to repeatably demonstrate the failure, now you have to drill down to the actual cause.

You many have confined it enough for a hardware trace to be useful, or you might need to confine it some more.

This might take several hours/days/weeks to nail down depending on the skills and tools you can bring to bear.

Yes, that is great advice, CptTitanic. I will look into this and see what I can find.

I do think it is very curious that I can "cp -R" a directory on the root file system into the sdcard, but I cannot copy a directory from the sdcard into the root file system. I am inclined to think that this is probably a software issue.

I'm not sure I would rate that as curious. You are exercising different code paths with reads/writes, and there is a bug some where in your driver stack. It's much more likely a coding error, than a compiler bug. Increase the debug/noise level of the involved drivers, and add some sanity checking to key structures and pointers.

Thanks, CptTitanic!

With a little more exploration, I think that I've managed to narrow the focus of the problem. I now strongly suspect that the kernel oops is related to something going wrong with memory allocation.

I've downloaded the memtest suite for ARM (http://www.arm.linux.org.uk/developer/stresstests.php) and built the binaries using the arm-linux-gcc cross-compiler from buildroot. I am cross-compiling the binaries on a 64-bit Ubuntu Linux system.

Running the memtest program on my embedded system always results in the following error. It looks like a crash is occurring due to an undefined instruction. What could be a possible cause of this oops, and what is really meant by an "undefined instruction"?

# ./mtest
Starting test run with 8 megabyte heap.
Setting up 2048 4096kB pages for test...Internal error: Oops - undefined instruction: 0 [#1] PREEMPT
last sysfs file: /sys/devices/virtual/vc/vcsa1/dev
Modules linked in:
CPU: 0 Not tainted (2.6.39-rc7+ #9)
PC is at v5tj_early_abort+0x0/0x38
LR is at __dabt_usr+0x38/0x60
pc : [<c00290a0>] lr : [<c0021d58>] psr: 00000093
sp : c3375fb0 ip : 00012008 fp : 00011664
r10: 00011638 r9 : 00011630 r8 : 00011644
r7 : 00011634 r6 : 00011678 r5 : 00011670 r4 : ffffffff
r3 : 80000010 r2 : 00008e78 r1 : 40150008 r0 : 00053177
Flags: nzcv IRQs off FIQs on Mode SVC_32 ISA ARM Segment user
Control: 0005317f Table: 23bfc000 DAC: 00000015
Process mtest (pid: 844, stack limit = 0xc3374270)
Stack: (0xc3375fb0 to 0xc3376000)
5fa0: 40150008 000002a6 002a6000 00001000
5fc0: 0001166c 00011670 00011678 00011634 00011644 00011630 00011638 00011664
5fe0: 00012008 bee22d10 4012ea64 00008e78 80000010 ffffffff 00000000 00000000
[<c00290a0>] (v5tj_early_abort+0x0/0x38) from [<00011664>] (0x11664)
Code: 00000000 00000000 00000000 00000000 (ee150000)
---[ end trace 7be324a20a10bf65 ]---

I would wonder if this undefined instruction behavior is caused by an improper use of compiler switches to compile the Linux kernel.

Using a non-Linux (bare-metal) ARM program loaded over JTAG, I've tested the SDRAM memory by reading and writing values. All values that I write into memory I can also read back successfully.

However, here is an example of the compiler switches being used for the Linux kernel compilation:

cmd_usr/initramfs_data.o := arm-linux-gcc -Wp,-MD,usr/.initramfs_data.o.d -nostdinc -isystem /media/RESEARCH/DEVICE-CODE/buildroot-heat-probe/buildroot-2011.02/output/host/usr/lib/gcc/arm-unknown-linux-uclibcgnueabi/4.3.5/include -I/media/RESEARCH/DEVICE-CODE/linux-kernel/linux-2.6-git/linux-2.6/arch/arm/include -Iinclude -include include/generated/autoconf.h -D__KERNEL__ -mlittle-endian -Iarch/arm/mach-at91/include -D__ASSEMBLY__ -mabi=aapcs-linux -mno-thumb-interwork -funwind-tables -D__LINUX_ARM_ARCH__=5 -march=armv5te -mtune=arm9tdmi -include asm/unified.h -msoft-float -gdwarf-2 -DINITRAMFS_IMAGE="usr/initramfs_data.cpio" -c -o usr/initramfs_data.o usr/initramfs_data.S