Booting Linux From NAND Flash on a SAMA5D44 Custom Board

[methos]

Revision -10 of our product has successfully used an SD card for the past few months. Recently, our team has decided to transition from SD cards to NAND flash. I have been responsible for building the image up until this point. As a disclaimer, I am an application engineer and prior to this project I have not had a lot of experience working with building images for custom boards. Nonetheless, after some research I was able to use the Yocto project to successfully build an operating system for our device.

The only major changes I made were to the DTS to add some of our custom hardware, and I ended up creating an entirely new Yocto layer to hold all our patches and changes to the already existing Atmel bsp. I have become somewhat familiar with working with Yocto project as a result.

Revision -20 of our product now has NAND flash. I decided that a good first step was to use the SAMA5D44-Xplained demonstration as a starting point. This is where I initially discovered a problem. After flashing the NAND flash demo using SAM-BA, the only output I would get was RomBOOT.

Initially, I thought it may have been a hardware error. After consulting with our electrical engineering team, they confirmed via oscilloscope that the NAND flash was operating correctly, and that it was properly being flashed. I dumped all the data of the NAND flash as well, and everything I flashed was there at the expected memory addresses as outlined in the SAMA5D4 memory map on Linux4Sam.

It was around this time that I discovered PMECC. I quickly realized that our NAND flash was a different model compared to the SAMA5D44-Xplained reference board. More specifically, our model was the MT29F2G08ABAEAWP, while the reference board contained MT29F4G08ABAEA. After reviewing PMECC, and reviewing the datasheets for our NAND module and the SAMA5D44, I was able to modify the header in the QML script provided by the NAND demo.

I determined that the new PMECC header should be 0xC0902405. After changing the script to reflect this, the AT91Bootstrap was successfully loaded into memory, followed by u-boot. However, u-boot reported multiple ECC errors. After some research, I realized that u-boot encodes the PMECC settings separately. At this point I realized using the default demonstration would not work very well without modification, so I broke out my Yocto build environment.

The first step of the process was to modify u-boot such that I could successfully load the Linux Kernel and Device Tree into memory. After researching u-boot for sometime, I identified two areas that required modification. The first change I made was to the sama5d4_xplained.h file in the u-boot source code. I realize that because we have a custom board, I should create my own header, however for the time being I am simply attempting to hack together a bootable image.

sama5d4_xplained.h before:

Code: Select all

#elif CONFIG_NAND_BOOT
#define CONFIG_SPL_NAND_DRIVERS
#define CONFIG_SPL_NAND_BASE
#endif
#define CONFIG_SYS_NAND_U_BOOT_OFFS	0x40000
#define CONFIG_SYS_NAND_5_ADDR_CYCLE
#define CONFIG_SYS_NAND_PAGE_SIZE	0x1000
#define CONFIG_SYS_NAND_PAGE_COUNT	64
#define CONFIG_SYS_NAND_OOBSIZE		224
#define CONFIG_SYS_NAND_BLOCK_SIZE	0x40000
#define CONFIG_SYS_NAND_BAD_BLOCK_POS	0x0

sama5d4_xplained.h after:

Code: Select all

#elif CONFIG_NAND_BOOT
#define CONFIG_SPL_NAND_DRIVERS
#define CONFIG_SPL_NAND_BASE
#endif
#define CONFIG_SYS_NAND_U_BOOT_OFFS	0x40000
#define CONFIG_SYS_NAND_5_ADDR_CYCLE
#define CONFIG_SYS_NAND_PAGE_SIZE	0x800
#define CONFIG_SYS_NAND_PAGE_COUNT	64
#define CONFIG_SYS_NAND_OOBSIZE		64
#define CONFIG_SYS_NAND_BLOCK_SIZE	0x20000
#define CONFIG_SYS_NAND_BAD_BLOCK_POS	0x0
#endif

These changes reflect the difference in our NAND versus the reference board NAND. I also modified the sama5d4_xplained_nandflash_defconfig such that the field CONFIG_PMECC_CAP=4 instead of the default 8. After making these changes, and recompiling u-boot, AT91Bootstrap executed twice, but u-boot never began.

I quickly realized our SDRAM settings are also different then the reference board. It turns out that the memory settings on our board are more analogous to the SAMA5D3-Xplained in that we have a similar version of NAND flash with the same specifications and identical SDRAM. Using this information, I modified the header for the sama5d4_xplained again and made the following changes:

Code: Select all

#define CONFIG_SYS_SDRAM_BASE           0x20000000
#define CONFIG_SYS_SDRAM_SIZE		0x10000000

#ifdef CONFIG_SPL_BUILD
#define CONFIG_SYS_INIT_SP_ADDR		0x218000
#else
#define CONFIG_SYS_INIT_SP_ADDR \
	(CONFIG_SYS_SDRAM_BASE + 16 * 1024 - GENERATED_GBL_DATA_SIZE)
#endif

#define CONFIG_SYS_LOAD_ADDR		0x22000000 /* load address */

My first question stems from the above modification. After researching these configs, I only understand one of the changes. CONFIG_SYS_SDRAM_SIZE being set to 0x10000000 makes sense, it is analogous to our total SDRAM size. However, the CONFIG_SYS_SDRAM_BASE and CONFIG_SYS_LOAD_ADDR confuse me. Both are far outside the available SDRAM size, and yet even the SAMA5D3 used these settings with it's 256 MB of SDRAM. Are these virtualized addresses?

After making the above changes, I have some good progress. The below output is what I see when attempting to boot the board.

Code: Select all

RomBOOT


AT91Bootstrap 3.8.13 (Tue Oct 29 17:30:33 UTC 2019)

EEPROM: Loading AT24xx information ...
EEPROM: BoardName | [Revid] | VendorName

EEPROM: Using default information

EEPROM: Board sn: 0x1012420 revision: 0x680820

NAND: ONFI flash detected
NAND: Manufacturer ID: 0x2c Chip ID: 0xda
NAND: Page Bytes: 2048, Spare Bytes: 64
NAND: ECC Correctability Bits: 4, ECC Sector Bytes: 512
NAND: Disable On-Die ECC
NAND: Initialize PMECC params, cap: 4, sector: 512
NAND: Image: Copy 0xa0000 bytes from 0x40000 to 0x26f00000
NAND: Done to load image
<debug_uart> 

U-Boot 2019.04-linux4sam_6.1 (Oct 29 2019 - 21:11:15 +0000)

CPU: SAMA5D44
Crystal frequency:       12 MHz
CPU clock        :      600 MHz
Master clock     :      200 MHz
DRAM:  256 MiB
NAND:  256 MiB
MMC:   Atmel mci: 0
Loading Environment from NAND... *** Warning - bad CRC, using default environment

In:    serial@fc00c000
Out:   serial@fc00c000
Err:   serial@fc00c000
Net:   eth0: ethernet@f8020000
Hit any key to stop autoboot:  0 

NAND read: device 0 offset 0x180000, size 0x80000
 524288 bytes read: OK

NAND read: device 0 offset 0x200000, size 0x600000
 6291456 bytes read: OK
=> 

The first alarming thing from the above output is that there is a bad CRC. Upon researching this, I realized this is a checksum for the uboot.env file. I have hypothesized that this is perhaps because my uboot.env does not match my uboot.bin I compiled outside of the Yocto Project? I am not sure if this would cause the problem I still have.

The next thing I noticed is that the FIT image has been read successfully, however neither the device tree nor the Linux Kernel are starting.

The next oddity is that there should be nothing at offset 0x200000. The ROOT FS is loaded at 0x800000, and I suspect that u-boot should not be responsible for loading this into memory, but rather the monolithic FIT Image.

Now, I suspect that either my u-boot settings are still incorrect OR it is time to modify my DTS such that it conforms to the correct NAND profile. Perhaps the FIT Image is loaded correctly, but it does not start because it to needs to know what NAND environment it is on?

My question regarding all this is, what exactly must I modify in the Kernel and the DTS (more generally the FIT Image) assuming this is the likely culprit? More generally, where can I find resources for all of these settings? Googling has not given me much details to work with. Ultimately, I want to understand every component of this boot process such that I can easily modify the AT91Bootstrap, U-boot, FIT Image and ROOT FS in the future for any given compatible NAND flash module. Thank you for your time in reading all of this!

[blue_z]

CONFIG_SYS_SDRAM_SIZE being set to 0x10000000 makes sense, it is analogous to our total SDRAM size. However, the CONFIG_SYS_SDRAM_BASE and CONFIG_SYS_LOAD_ADDR confuse me. Both are far outside the available SDRAM size, and yet even the SAMA5D3 used these settings with it's 256 MB of SDRAM. Are these virtualized addresses?

CONFIG_SYS_SDRAM_SIZE is not "analogous" to the size; rather this number actually does specify the size of physical RAM installed.
Of course the processor memory address space starts at zero, but the physical RAM for an ARM processor typically does not start at address zero.
CONFIG_SYS_SDRAM_BASE is used to specify the physical memory address for the start of the installed (external) RAM.
CONFIG_SYS_LOAD_ADDR is a default address used to store data transfers (e.g. the tftpboot command).

I have hypothesized that this is perhaps because my uboot.env does not match my uboot.bin I compiled outside of the Yocto Project?

If you don't test your "hypothesis", then you're just guessing.
There should be no reason to "hypothesize".
All you have to do is google the salient error phrase to get explanations. There's even one in the U-Boot documentation (i.e. at denx.de).

Be aware that there are three (3) sets of environment variables in U-Boot.

The default environment is the set of environment variables that are defined in source code,

The saved environment is the set of environment variables that are retained in persistent storage.
The storage device is configurable, and could be a file or raw sectors/storage.
A CRC32 checkword is stored with the environment variables stored as text strings to validate the integrity.

The active environment is the set of environment variables that are held in RAM while U-Boot executes.
On startup U-Boot tries to populate the active environment with the saved environment, but only if the CRC32 checkword confirms the integrity of the saved environment.
When validation is successful, the environment variables from the persistent storage (silently) become the active environment.
When validation fails, a notice is displayed, "Warning: Bad CRC, using default environment", and the environment variables from the default environment become the active environment.

FYI
The boot steps that AT91Bootstrap performs are hardcoded. Where it retrieves the image that it reads is baked in by the configuration prior to compilation.
U-Boot tries to live up to its name ("Universal"), and the boot procedure that it can/will perform is fully parameterized in environment variables.
A default boot procedure can be defined, i.e. the CONFIG_BOOTCOMMAND definition in the source code.
But it's the bootcmd variable in the active environment that determines what happens.

The next thing I noticed is that the FIT image has been read successfully, however neither the device tree nor the Linux Kernel are starting.

The next oddity is that there should be nothing at offset 0x200000.
...
Now, I suspect that either my u-boot settings are still incorrect OR it is time to modify my DTS such that it conforms to the correct NAND profile. Perhaps the FIT Image is loaded correctly, but it does not start because it to needs to know what NAND environment it is on?

My question regarding all this is, what exactly must I modify in the Kernel and the DTS (more generally the FIT Image) assuming this is the likely culprit?

Your boot is using the default environment of U-Boot.
Microchip/Atmel developers have not updated the default environment of U-Boot to boot a FIT image.
The default environments for Atmel boards still boot the traditional kernel zImage file and Device Tree blob.

Suggest you review a recent Linux4SAM demo build (e.g. version 6.1 or 6.2) that uses Yocto.
Rather than modify the U-Boot source code (to boot a FIT image), the Yocto build procedure creates a stored environment image that does boot a FIT image.

More generally, where can I find resources for all of these settings?

AFAIK such comprehensive documentation (e.g. a book) does not exist.

Regards

[methos]

Blue_z, your response was incredibly helpful.

I have now created a comprehensive document to modify all requisite aspects to boot from different NAND modules.

First, I used the correct PMECC parameters, which is pre-appended to the AT91Bootstrap binary. I was able to calculate this PMECC header by referencing the NAND module datasheet and using the recommended solution on page 77 of the SAMA5D44 Datasheet.

I then wrote my own qml script to flash the board, using the examples scripts in sam-ba, there is a variable for the PMECC header near the top of this script.

Next, using a patch and Yocto project, I modified the configuration of AT91Bootstrap (PMECC Error Correction Bits, PMECC Sector Size). Note, you can modify this using menuconfig from within Yocto project:

Code: Select all

bitbake at91bootstrap -c menuconfig

It is advisable to create a bbappend file and patch the source files of at91bootstrap, this is the route I opted for.

Next, I patched u-boot in three places: sama5d4_xplained.h, at91-sama5_common.h, sama5d4_xplained_nandflash_defconfig.

For the sama5d4_xplained.h, the exact changes are to the following parameters:

Code: Select all

#define CONFIG_SYS_SDRAM_BASE           0x20000000 // address of SDRAM on MPU bus. Check the memory map in the SAMA5D44 datasheet.
#define CONFIG_SYS_SDRAM_SIZE       0x20000000 // Size of SDRAM installed 


#define CONFIG_SYS_NAND_U_BOOT_OFFS 0x40000 // Location of U-boot in NAND
#define CONFIG_SYS_NAND_5_ADDR_CYCLE
#define CONFIG_SYS_NAND_PAGE_SIZE   0x1000 // Size of your NAND page.
#define CONFIG_SYS_NAND_PAGE_COUNT  64 // Number of pages per block
#define CONFIG_SYS_NAND_OOBSIZE     224 // Spare size
#define CONFIG_SYS_NAND_BLOCK_SIZE  0x40000 // Size of block in bytes
#define CONFIG_SYS_NAND_BAD_BLOCK_POS   0x0 // Offset of Bad Block allocation in spare

For the at91-sama5_common.h header, I made a small change to:

Code: Select all

#define CONFIG_BOOTCOMMAND      "nand read 0x21000000 0x180000 0x80000;"    \
                    "nand read 0x22000000 0x200000 0x600000;"   \
                    "bootz 0x22000000 - 0x21000000"
                    

This is the default environment that is compiled into u-boot. I modified it to work with a FIT Image. Blue_z, thank you for pointing me in the right direction here.

Finally, I modified sama5d4_xplained_nandflash_defconfig,

I changed the Bootargs, PMECC_CAP and PMECC_SECTOR_SIZE to correspond with my NAND module.

Finally, I modified the UBIFS file system by creating my own machine based on the already existing machines in meta-atmel/conf/machines. The two arguments that correspond to the NAND environment are to:

Code: Select all

MKUBIFS_ARGS ?= " -e 0x1f000 -c 2048 -m 0x800  -x lzo"
UBINIZE_ARGS ?= " -m 0x800 -p 0x20000 -s 2048"

After making all of the above changes to match my NAND module, I was able to boot into the operating system with no issues. I am writing this post to help others.

All this being said, I have run into a new issue. The NAND Module I used had a page size of 4096. I attempted the same process with a larger NAND Module that has a page size of 8192. To be more specific, this is the model: MT29F32G08ABAAA.

Everything went smoothly until I reached u-boot, where I received the follow error:
NAND: atmel_nand: Fail to initialize #0 chip0 MiB

I was able to trace this error to the atmel_nand.c drivers. After some searching, I determined that other manufactures drivers (Texas Instruments) are not compatible with 8192 page sizes, but I could not confirm this for atmel-nand. Could this cause the failure to initalize seen above?

Code: Select all

if (nand->ecc.mode == NAND_ECC_HW) {
		/* ECC is calculated for the whole page (1 step) */
		nand->ecc.size = mtd->writesize;

		/* set ECC page size and oob layout */
		switch (mtd->writesize) {
		case 512:
			nand->ecc.layout = &atmel_oobinfo_small;
			ecc_writel(CONFIG_SYS_NAND_ECC_BASE, MR,
					ATMEL_ECC_PAGESIZE_528);
			break;
		case 1024:
			nand->ecc.layout = &atmel_oobinfo_large;
			ecc_writel(CONFIG_SYS_NAND_ECC_BASE, MR,
					ATMEL_ECC_PAGESIZE_1056);
			break;
		case 2048:
			nand->ecc.layout = &atmel_oobinfo_large;
			ecc_writel(CONFIG_SYS_NAND_ECC_BASE, MR,
					ATMEL_ECC_PAGESIZE_2112);
			break;
		case 4096:
			nand->ecc.layout = &atmel_oobinfo_large;
			ecc_writel(CONFIG_SYS_NAND_ECC_BASE, MR,
					ATMEL_ECC_PAGESIZE_4224);
			break;
		default:
			/* page size not handled by HW ECC */
			/* switching back to soft ECC */
			nand->ecc.mode = NAND_ECC_SOFT;
			nand->ecc.calculate = NULL;
			nand->ecc.correct = NULL;
			nand->ecc.hwctl = NULL;
			nand->ecc.read_page = NULL;
			nand->ecc.postpad = 0;
			nand->ecc.prepad = 0;
			nand->ecc.bytes = 0;
			break;
		}
	}
	

This code is from the atmel_nand.c driver, it looks like HW ECC support is only possible up to 4096. This does not necessary mean the module is not compatible, but I was hoping someone here would have more insight. In the meanwhile, we have decided to prototype the following module to our board: MT29F8G08ABACAWP-IT:C. It has identical settings to the reference boards NAND module, so I suspect I will not have the same issue.

[blue_z]

All this being said, I have run into a new issue. The NAND Module I used had a page size of 4096. I attempted the same process with a larger NAND Module that has a page size of 8192. To be more specific, this is the model: MT29F32G08ABAAA.

That's a NAND chip, not a "module".

Could this cause the failure to initalize seen above?

Your code analysis is likely wrong because atmel_hwecc_nand_init_param(), i.e. the routine that the code snippet is presumably from, always returns zero, and therefore not the origin of any failure.
There probably was salient text prior to that error message that you overlooked/ignored.

This code is from the atmel_nand.c driver, it looks like HW ECC support is only possible up to 4096. This does not necessary mean the module is not compatible, but I was hoping someone here would have more insight.

If you're going to quote source code, then clearly indicate which program, what version, the filename with path, and a procedure name.

What you have posted appears to be within conditional compilation using:

#ifdef CONFIG_ATMEL_NAND_HW_PMECC
...
#else
...
<the code snippet you posted>
...
#endif

For a SAMA5D4 SoC, you can enable CONFIG_ATMEL_NAND_HW_PMECC, which (in recent versions) does support a page size of 8192 bytes.
So how have you configured U-Boot?

Regards