Hello All,

We have designed a custom board around AT91SAM9260 and were using NAND Flash from Micron which had support for 1 bit ECC.

Now that the device has reached EOL (End Of Life) we had to replace this device with a newer device. Newer devices from micron support "On Die 4 Bit ECC".

Since the MCU supports only 1 Bit ECC, I had to disable HW ECC and SW ECC and make use of the "On Die ECC feature".

I am trying to get the new Micron NAND flash with 4 bit On Die ECC to work and have made very little progress.

I could find an application note from Micron which points out the changes to be made to UBoot and Linux Kernel NAND Flash driver.

I have made all the changes as per the application note to bootstrap, UBoot and the kernel driver to enable "On Die ECC".

When I write to the NAND flash using UBoot commands, data is getting written correctly, but when I read back, I get an "On Die ECC Error".

I have an JFFS2 partition which I use for storing data. When I try to create a file or copy a file to the JFFS2 partition, I get Kernel Panic. Please find the log files attached to this post.

As an intermediate step, now I am trying to disable all ECC features and work with the flash. I have disabled HW ECC, SW ECC (Selected NO ECC option in Linux Kernel) and disabled On Die ECC too.

This build works fine on the boards which have the older die. But when I create files on the JFFS2 partition on the board with the newer die, I get kernel panic.

At this point I am kind of lost. Any input would be of great help.

Thanks & Regards,
Shakthi

******************************************************
Part of the Linux Kernel boot log for the new die:
******************************************************

NAND device: Manufacturer ID: 0x2c, Chip ID: 0xca (Micron NAND 256MiB 3,3V 16-bit)
AT91 NAND: 16-bit, No ECC
NAND_ECC_NONE selected by board driver. This is not recommended !!

******************************************************

Creating 2 MTD partitions on "NAND 256MiB 3,3V 16-bit":
0x000000000000-0x000001000000 : "Partition 1"
0x000001000000-0x000010000000 : "Partition 2"


******************************************************

Data CRC 22ee8d0e != calculated CRC 93e14ada for node at 0efd2544
kernel BUG at fs/jffs2/file.c:251!
Unable to handle kernel NULL pointer dereference at virtual address 00000000
pgd = c119c000
[00000000] *pgd=2119b031, *pte=00000000, *ppte=00000000
Internal error: Oops: 817 [#1]
Modules linked in:
CPU: 0 Not tainted (2.6.30 #55)
PC is at __bug+0x1c/0x28
LR is at __bug+0x18/0x28
pc : [<c00284c4>] lr : [<c00284c0>] psr: 60000013
sp : c1193d28 ip : fefff200 fp : c0fd7800
r10: 00000000 r9 : 00000000 r8 : c1192000
r7 : c0350d60 r6 : c1982c40 r5 : 00001000 r4 : 00001000
r3 : 00000000 r2 : 60000013 r1 : 00001cd0 r0 : 00000026
Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
Control: 0005317f Table: 2119c000 DAC: 00000015
Process ExtractFirmware (pid: 826, stack limit = 0xc1192268)
Stack: (0xc1193d28 to 0xc1194000)
3d20: 00000000 c011549c 00012000 00001000 c1193d4c 00000000
3d40: 00013000 00000000 00001000 00000000 c1192000 00001000 00001000 00013000
3d60: 00000000 c1192000 00000000 00000000 00013000 c0060e4c 00001000 00001000
3d80: c0350d60 c0fa0260 c02e97e0 c1193eb0 c0fa0260 c1982cd8 c022f9c0 c1982c40
3da0: c1982cd8 00001000 00000000 c022f9c0 c1193f40 00000001 00013000 001b39d0
3dc0: c0fa0260 c0350d60 00000006 c1982cac c1982c40 00000000 00000000 c1982c40
3de0: 00000000 00000000 00000000 c00615e0 00000000 00000000 c1193f00 001c69d0
3e00: 00000000 c00360dc c1193f00 c1193f40 c1193eb0 c0fa0260 c1982cd8 001c69d0
3e20: 00000000 00000001 c1193e4c c0038d48 c02e97e0 00000001 c02e97e0 001c69d0
3e40: 00000000 c1982cac c1982c40 c1193eb0 c1193f40 00000000 00000000 00000000
3e60: 00000000 c00618dc c1193e9c c004dab4 c1c23fb4 00000001 c0fa0260 c1982cd8
3e80: 00000100 c1193eb0 c0fa0260 c1193eb0 c1193f40 c1193f88 c1192000 fffffdee
3ea0: be9e9afc c00801ac 00000000 00000000 c030d568 c004a2b8 00000000 00000001
3ec0: ffffffff c0fa0260 00000000 00000000 00000000 00000000 c1cab6e0 00000100
3ee0: 00000000 00000000 00000000 c1cab6e0 c004daa8 c1193ef4 c1193ef4 c02ebb68
3f00: 00000000 00000000 00000000 00000001 c1c913a0 c1cab86c 001c69d0 00000000
3f20: 00000001 c0024060 00000000 ffffffff fefff000 c0024974 00000001 00000000
3f40: 40001008 001c69d0 c02e97e0 c0fa0260 40001008 c1193f88 001c69d0 c0024e88
3f60: 00000000 c0080a5c c1cab86c c00510a4 00000000 00000000 00000000 c0fa0260
3f80: 00000004 c0080f00 00000000 00000000 c1192000 00000000 00000000 00081a80
3fa0: 00000000 c0024ce0 00000000 00081a80 00000004 40001008 001c69d0 001c69d0
3fc0: 00000000 00081a80 00000000 00000004 00008ee0 00008e98 00000000 be9e9afc
3fe0: 00000000 be9e9878 00008548 00018c1c 60000010 00000004 00000000 00000000
[<c00284c4>] (__bug+0x1c/0x28) from [<c011549c>] (jffs2_write_end+0x2b8/0x2d0)
[<c011549c>] (jffs2_write_end+0x2b8/0x2d0) from [<c0060e4c>] (generic_file_buffered_write+0x178/0x2cc)
[<c0060e4c>] (generic_file_buffered_write+0x178/0x2cc) from [<c00615e0>] (__generic_file_aio_write_nolock+0x424/0x470)
[<c00615e0>] (__generic_file_aio_write_nolock+0x424/0x470) from [<c00618dc>] (generic_file_aio_write+0x6c/0xe8)
[<c00618dc>] (generic_file_aio_write+0x6c/0xe8) from [<c00801ac>] (do_sync_write+0xb4/0x104)
[<c00801ac>] (do_sync_write+0xb4/0x104) from [<c0080a5c>] (vfs_write+0xb0/0x140)
[<c0080a5c>] (vfs_write+0xb0/0x140) from [<c0080f00>] (sys_write+0x3c/0x68)
[<c0080f00>] (sys_write+0x3c/0x68) from [<c0024ce0>] (ret_fast_syscall+0x0/0x2c)
Code: e1a01000 e59f000c eb004ecb e3a03000 (e5833000)
---[ end trace ca3bbfac0d9ce05d ]---


******************************************************
Complete Log
******************************************************


### BEGIN LOG - DATE: 120118, TIME: 163818 ###

RomBOOT
>

U-Boot 1.3.4 (Jan 16 2012 - 19:19:17)

DRAM: 32 MB
NAND: NAND ECC Status: 6
NAND_ECC_NONE selected by board driver. This is not recommended !!
256 MiB
*** Warning - bad CRC or NAND, using default environment

In: serial
Out: serial
Err: serial
Net: macb0
MII
macb0: Starting autonegotiation...
macb0: Autonegotiation timed out (status=0x7849)
macb0: link down (status: 0x7849)
Hit any key to stop autoboot: 0

NAND read: device 0 offset 0x200000, size 0x200000
2097152 bytes read: OK

NAND read: device 0 offset 0x600000, size 0x600000
6291456 bytes read: OK
## Booting kernel from Legacy Image at 20a00000 ...
Image Name: Linux-2.6.30
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 1561112 Bytes = 1.5 MB
Load Address: 20008000
Entry Point: 20008000
Verifying Checksum ... OK
Loading Kernel Image ... OK
OK

Starting kernel ...

Uncompressing Linux.................................................................................................... done, booting the kernel.
Linux version 2.6.30 (shakthi@shakthi) (gcc version 4.2.0 20070413 (prerelease) (CodeSourcery Sourcery G++ Lite 2007q1-10)) #55 Wed Jan 18 12:25:29 IST 2012
CPU: ARM926EJ-S [41069265] revision 5 (ARMv5TEJ), cr=00053177
CPU: VIVT data cache, VIVT instruction cache
Machine: Atmel AT91SAM9260-EK
Memory policy: ECC disabled, Data cache writeback
Clocks: CPU 198 MHz, master 99 MHz, main 18.432 MHz
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 8128
Kernel command line: mem=32M console=ttyS0,115200 initrd=0x21100000,6291456 root=/dev/ram0 rw
NR_IRQS:192
AT91: 96 gpio irqs in 3 banks
PID hash table entries: 128 (order: 7, 512 bytes)
Console: colour dummy device 80x30
console [ttyS0] enabled
Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
Memory: 32MB = 32MB total
Memory: 23112KB available (2724K code, 255K data, 112K init, 0K highmem)
Calibrating delay loop... 98.91 BogoMIPS (lpj=494592)
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
net_namespace: 296 bytes
NET: Registered protocol family 16
bio: create slab <bio-0> at 0
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 1024 (order: 1, 8192 bytes)
TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
TCP: Hash tables configured (established 1024 bind 1024)
TCP reno registered
NET: Registered protocol family 1
Trying to unpack rootfs image as initramfs...
rootfs image is not initramfs (junk in compressed archive); looks like an initrd
Freeing initrd memory: 6144K
NetWinder Floating Point Emulator V0.97 (double precision)
JFFS2 version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
msgmni has been set to 57
alg: No test for stdrng (krng)
io scheduler noop registered
io scheduler anticipatory registered (default)
atmel_usart.0: ttyS0 at MMIO 0xfefff200 (irq = 1) is a ATMEL_SERIAL
atmel_usart.1: ttyS1 at MMIO 0xfffb0000 (irq = 6) is a ATMEL_SERIAL
atmel_usart.2: ttyS2 at MMIO 0xfffb4000 (irq = 7) is a ATMEL_SERIAL
atmel_usart.3: ttyS3 at MMIO 0xfffb8000 (irq = is a ATMEL_SERIAL
brd: module loaded
loop: module loaded
Driver 'sd' needs updating - please use bus_type methods
macb macb: invalid hw address, using random
MACB_mii_bus: probed
eth0: Atmel MACB at 0xfffc4000 irq 21 (8e:08:81:b4:ca:7e)
eth0: attached PHY driver [Generic PHY] (mii_bus:phy_addr=ffffffff:01, irq=-1)
NAND device: Manufacturer ID: 0x2c, Chip ID: 0xca (Micron NAND 256MiB 3,3V 16-bit)
AT91 NAND: 16-bit, No ECC
NAND_ECC_NONE selected by board driver. This is not recommended !!
Scanning device for bad blocks
Bad eraseblock 727 at 0x000005ae0000
Creating 2 MTD partitions on "NAND 256MiB 3,3V 16-bit":
0x000000000000-0x000001000000 : "Partition 1"
0x000001000000-0x000010000000 : "Partition 2"
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
at91_ohci at91_ohci: AT91 OHCI
at91_ohci at91_ohci: new USB bus registered, assigned bus number 1
at91_ohci at91_ohci: irq 20, io mem 0x00500000
usb usb1: New USB device found, idVendor=1d6b, idProduct=0001
usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: AT91 OHCI
usb usb1: Manufacturer: Linux 2.6.30 ohci_hcd
usb usb1: SerialNumber: at91
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
usbcore: registered new interface driver cdc_acm
cdc_acm: v0.26:USB Abstract Control Model driver for USB modems and ISDN adapters
Initializing USB Mass Storage driver...
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
usbcore: registered new interface driver usbserial
USB Serial support registered for generic
usbcore: registered new interface driver usbserial_generic
usbserial: USB Serial Driver core
udc: at91_udc version 3 May 2006
g_serial gadget: Gadget Serial v2.4
g_serial gadget: g_serial ready
mice: PS/2 mouse device common for all mice
AT91SAM9 Watchdog enabled (heartbeat=15 sec, nowayout=0)
TCP cubic registered
NET: Registered protocol family 17
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RAMDISK: ext2 filesystem found at block 0
RAMDISK: Loading 6144KiB [1 disk] into ram disk... done.
EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended
VFS: Mounted root (ext2 filesystem) on device 1:0.
Freeing init memory: 112K
Invoking script.sh
Inside script.sh
sh: can't open '/home/flash/ipsetup.sh'
# rm: cannot remove 'FirmWare': No such file or directory
ERROR IN OPENNING FILE
Error in Extracting Firmware.bin
FirmFile.Date: 20-09-2011
FirmFile.CVSTag: CVS_TAG_1
FirmFile.Signature: DEXCEL_1
FirmFile.MajorVer: 0
FirmFile.MinorVer: 10
FirmFile.Length: 1862096
FirmFile.CRC: 36913
ret: 1862096 FileCRC: 36913
Data CRC 22ee8d0e != calculated CRC 93e14ada for node at 0efd2544
kernel BUG at fs/jffs2/file.c:251!
Unable to handle kernel NULL pointer dereference at virtual address 00000000
pgd = c119c000
[00000000] *pgd=2119b031, *pte=00000000, *ppte=00000000
Internal error: Oops: 817 [#1]
Modules linked in:
CPU: 0 Not tainted (2.6.30 #55)
PC is at __bug+0x1c/0x28
LR is at __bug+0x18/0x28
pc : [<c00284c4>] lr : [<c00284c0>] psr: 60000013
sp : c1193d28 ip : fefff200 fp : c0fd7800
r10: 00000000 r9 : 00000000 r8 : c1192000
r7 : c0350d60 r6 : c1982c40 r5 : 00001000 r4 : 00001000
r3 : 00000000 r2 : 60000013 r1 : 00001cd0 r0 : 00000026
Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
Control: 0005317f Table: 2119c000 DAC: 00000015
Process ExtractFirmware (pid: 826, stack limit = 0xc1192268)
Stack: (0xc1193d28 to 0xc1194000)
3d20: 00000000 c011549c 00012000 00001000 c1193d4c 00000000
3d40: 00013000 00000000 00001000 00000000 c1192000 00001000 00001000 00013000
3d60: 00000000 c1192000 00000000 00000000 00013000 c0060e4c 00001000 00001000
3d80: c0350d60 c0fa0260 c02e97e0 c1193eb0 c0fa0260 c1982cd8 c022f9c0 c1982c40
3da0: c1982cd8 00001000 00000000 c022f9c0 c1193f40 00000001 00013000 001b39d0
3dc0: c0fa0260 c0350d60 00000006 c1982cac c1982c40 00000000 00000000 c1982c40
3de0: 00000000 00000000 00000000 c00615e0 00000000 00000000 c1193f00 001c69d0
3e00: 00000000 c00360dc c1193f00 c1193f40 c1193eb0 c0fa0260 c1982cd8 001c69d0
3e20: 00000000 00000001 c1193e4c c0038d48 c02e97e0 00000001 c02e97e0 001c69d0
3e40: 00000000 c1982cac c1982c40 c1193eb0 c1193f40 00000000 00000000 00000000
3e60: 00000000 c00618dc c1193e9c c004dab4 c1c23fb4 00000001 c0fa0260 c1982cd8
3e80: 00000100 c1193eb0 c0fa0260 c1193eb0 c1193f40 c1193f88 c1192000 fffffdee
3ea0: be9e9afc c00801ac 00000000 00000000 c030d568 c004a2b8 00000000 00000001
3ec0: ffffffff c0fa0260 00000000 00000000 00000000 00000000 c1cab6e0 00000100
3ee0: 00000000 00000000 00000000 c1cab6e0 c004daa8 c1193ef4 c1193ef4 c02ebb68
3f00: 00000000 00000000 00000000 00000001 c1c913a0 c1cab86c 001c69d0 00000000
3f20: 00000001 c0024060 00000000 ffffffff fefff000 c0024974 00000001 00000000
3f40: 40001008 001c69d0 c02e97e0 c0fa0260 40001008 c1193f88 001c69d0 c0024e88
3f60: 00000000 c0080a5c c1cab86c c00510a4 00000000 00000000 00000000 c0fa0260
3f80: 00000004 c0080f00 00000000 00000000 c1192000 00000000 00000000 00081a80
3fa0: 00000000 c0024ce0 00000000 00081a80 00000004 40001008 001c69d0 001c69d0
3fc0: 00000000 00081a80 00000000 00000004 00008ee0 00008e98 00000000 be9e9afc
3fe0: 00000000 be9e9878 00008548 00018c1c 60000010 00000004 00000000 00000000
[<c00284c4>] (__bug+0x1c/0x28) from [<c011549c>] (jffs2_write_end+0x2b8/0x2d0)
[<c011549c>] (jffs2_write_end+0x2b8/0x2d0) from [<c0060e4c>] (generic_file_buffered_write+0x178/0x2cc)
[<c0060e4c>] (generic_file_buffered_write+0x178/0x2cc) from [<c00615e0>] (__generic_file_aio_write_nolock+0x424/0x470)
[<c00615e0>] (__generic_file_aio_write_nolock+0x424/0x470) from [<c00618dc>] (generic_file_aio_write+0x6c/0xe8)
[<c00618dc>] (generic_file_aio_write+0x6c/0xe8) from [<c00801ac>] (do_sync_write+0xb4/0x104)
[<c00801ac>] (do_sync_write+0xb4/0x104) from [<c0080a5c>] (vfs_write+0xb0/0x140)
[<c0080a5c>] (vfs_write+0xb0/0x140) from [<c0080f00>] (sys_write+0x3c/0x68)
[<c0080f00>] (sys_write+0x3c/0x68) from [<c0024ce0>] (ret_fast_syscall+0x0/0x2c)
Code: e1a01000 e59f000c eb004ecb e3a03000 (e5833000)
---[ end trace ca3bbfac0d9ce05d ]---


### END LOG - DATE: 120118, TIME: 163905 ###

I brought this issue up almost a year ago.
http://www.at91.com/forum/viewtopic.php/f,9/t,19814/
Also posted it on the arm linux mail list. Did not get a lot of response on it.

Hello Tim,

I could resolve the kernel panic by using YAFFS2 with the default NAND Flash driver and SW ECC. Now the board boots up without any panic.

But when I copy files to my YAFFS2 partition and reboot, files get corrupted and on multiple reboots, files disappear.

I found multiple posts reporting the same issue in various forums, but none have a solution.

I tried enabling "On Die ECC" by following instructions given in an application note from micron "TN-29-56: Enabling On-Die ECC for OMAP3 on Linux/Android OS Enabling/Disabling On-Die ECC", but this too didn't help.

How about you?? Did you make any progress?? Please contact me at shakthi.pradeep@dexceldesigns.com

Thanks & Regards,
Shakthi

We just buy Toshiba or Samsung. They are still 1 bit ECC parts.

On further investigation we found the problem to be with READ from NAND flash in Linux.

Here's what I did:

1) Wrote 5 MB of 0xaaaa to the NAND flash (/dev/mtdblock1) using dd command.
2) Read back the contents using hexdump utility
3) 70% of the time I get 5 MB of 0xaaaa and 30% of the time I get 0xaaaa with 0xffff at 2 or 3 random offsets.
4) Then I go to UBoot and use nand dump 0x1000000 command to read the nand flash content and I dont see any corruption of data.

I tried reading the data in Linux multiple times with multiple reboots but the behavior is random.

We are using NAND flash from micron with 16 bit bus interface.

Below is the NAND flash dump:

# hexdump /dev/mtdblock1 -n 10485760
0000000 aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa
*
0a00000
# hexdump /dev/mtdblock1 -n 10485760
0000000 aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa
*
0435ff0 aaaa aaaa aaaa aaaa aaaa aaaa aaaa ffff
0436000 aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa
*
05b5ff0 aaaa aaaa aaaa aaaa aaaa aaaa aaaa ffff
05b6000 aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa
*
0a00000
# hexdump /dev/mtdblock1 -n 10485760
0000000 aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa
*
0a00000
# hexdump /dev/mtdblock1 -n 10485760
0000000 aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa
*
0a00000
# hexdump /dev/mtdblock1 -n 10485760
0000000 aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa
*
0a00000
# hexdump /dev/mtdblock1 -n 10485760
0000000 aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa
*
0515ff0 aaaa aaaa aaaa aaaa aaaa aaaa aaaa ffff
0516000 aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa
*
0a00000
# hexdump /dev/mtdblock1 -n 10485760
0000000 aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa
*
0a00000
# hexdump /dev/mtdblock1 -n 10485760
0000000 aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa
*
0a00000
# hexdump /dev/mtdblock1 -n 10485760
0000000 aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa
*
0975ff0 aaaa aaaa aaaa aaaa aaaa aaaa aaaa ffff
0976000 aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa
*
0a00000
# hexdump /dev/mtdblock1 -n 10485760
0000000 aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa
*
07d5ff0 aaaa aaaa aaaa aaaa aaaa aaaa aaaa ffff
07d6000 aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa
*
0a00000
# hexdump /dev/mtdblock1 -n 10485760
0000000 aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa
*
0455ff0 aaaa aaaa aaaa aaaa aaaa aaaa aaaa ffff
0456000 aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa
*
0a00000
#

Hello All,

We could resolve the issue by tweaking the Linux driver.

As per the Atmel reference design the software was enabling the NAND flash chip enable for the entire NAND page read duration.

When reading a NAND page from Linux there will be interruptions during this NAND page read and the processor may go to do other tasks and then come back and read the remaining data in that page.

But the NAND flash chip enable is enabled during this entire duration of NAND page read. We observed that whenever there is a large gap between two reads in a page with the chip enable active, the NAND flash was missing a word when we read it after this gap.

This was not happening with the older NAND chip.

So to resolve the issue we made the NAND flash chip enable hardware controlled and enabled only for one word instead of one full page.

Shakthi