LPC1768: getting information about hard faults

We have a hard fault in our app that is pretty hard to reproduce and it takes a fair bit of time to do. I would like to be able to get more data about the crash other than the register dump printed on the console at the time of the crash. The dump is:

    ++ MbedOS Fault Handler ++
    FaultType: HardFault
    Context:
    R0   : E04FF020
    R1   : 10002B00
    R2   : 00000000
    R3   : 00000471
    R4   : 2008078C
    R5   : 20080178
    R6   : 00000000
    R7   : 10005E70
    R8   : 000027CA
    R9   : 34A3F045
    R10  : 20080790
    R11  : 00000000
    R12  : 00044E45
    SP   : 100044D0
    LR   : 0003A7E7
    PC   : 0003A96A
    xPSR : 21000000
    PSP  : 100044B0
    MSP  : 10007FA0
    CPUID: 412FC230
    HFSR : 40000000
    MMFSR: 00000082
    BFSR : 00000000
    UFSR : 00000000
    DFSR : 00000000
    AFSR : 00000000
    MMFAR: 00000000
    Mode : Thread
    Priv : Privileged
    Stack: PSP
    -- MbedOS Fault Handler --

Based on data I have found for this dump it seems that MMFSR tells me;

  • DACCVIOL - Indicates that a data access triggered the MemManage fault.
  • MMARVALID - Indicates that the MemManage Fault Address Register ( MMFAR ), a 32 bit register located at 0xE000ED34 , holds the address which triggered the MemManage fault.

Also HFSR seems to tell me:

  • DEBUGEVT - Indicates that a debug event (i.e executing a breakpoint instruction) occurred while the debug subsystem was not enabled

But all these really tell me nothing more than that I apparently read data from a NULL pointer, as MMFAR is 0. So, to get more info I wanted to save the crash data and decode it during the next boot. For that I enabled the crash handling:

"target_overrides": {
    "*": {
        "platform.crash-capture-enabled": true,
        "platform.error-hist-enabled": true,
        "platform.fatal-error-auto-reboot-enabled": true,

That gave a linker error for the missing crash data region, so I added:

.crash_data_ram :
{
    . = ALIGN(8);
	__CRASH_DATA_RAM__ = .;
    __CRASH_DATA_RAM_START__ = .; /* Create a global symbol at data start */
    KEEP(*(.keep.crash_data_ram))
    *(.m_crash_data_ram)     /* This is a user defined section */
    . += M_CRASH_DATA_RAM_SIZE;
    . = ALIGN(8);
    __CRASH_DATA_RAM_END__ = .; /* Define a global symbol at data end */
} > RAM

to targets/TARGET_NXP/TARGET_LPC176X/device/TOOLCHAIN_GCC_ARM/LPC1768.ld right before the .data segment and my app links and runs just fine. To decode the crash data I added this snippet to my main file:

#if MBED_CONF_PLATFORM_CRASH_CAPTURE_ENABLED
mbed_error_ctx error_ctx;
mbed_error_status_t err_status;
mbed_fault_context_t fault_ctx;
static int reboot_error_happened = 0;

void mbed_error_reboot_callback(mbed_error_ctx *error_context) {
    printf("mbed_error_reboot_callback: System rebooting, reboot error callback received");
    reboot_error_happened = 1;
    err_status = error_context->error_status;
    mbed_get_reboot_error_info(&error_ctx);
    mbed_reset_reboot_error_info();
}

void printErrorReport() {
    if (err_status < 0) {
        log("\nSuccessfully read reboot info ==> \n");
        log("\n  error status:       0x%08lX", (uint32_t)error_ctx.error_status);
        log("\n  error value:        0x%08lX", (uint32_t)error_ctx.error_value);
        log("\n  error address:      0x%08lX", (uint32_t)error_ctx.error_address);
        log("\n  error reboot count: 0x%08lX", (uint32_t)error_ctx.error_reboot_count);
        log("\n  error crc:          0x%08lX\n", (uint32_t)error_ctx.crc_error_ctx);

        //Read fault context
        if(error_ctx.error_status == MBED_ERROR_HARDFAULT_EXCEPTION) {
            mbed_get_reboot_fault_context(&fault_ctx);

            log("\nCrash Report data captured:\n");
            for(int i=0;i<16;i++) {
                log("[%d]: 0x%08X\n", i, ((uint32_t *)(&fault_ctx))[i]);
            }
        }
    }
}
#endif

This is mostly straight from the docs. In main() I check the error state as soon as I’ve enabled the console for logging:

#if MBED_CONF_PLATFORM_CRASH_CAPTURE_ENABLED
    log("main: error state: %d", reboot_error_happened);
    // any errors?
    if ( reboot_error_happened == 0) {
        printErrorReport();
    }
#endif

To test I trigger a hard fault after a minute of running the app using the same code as in the docs:

void generateCrash() {
    SCB->CCR |= SCB_CCR_UNALIGN_TRP_Msk;
    uint32_t inv_addr = 0xAAA3;
    uint32_t val = *(uint32_t *)inv_addr;

    printf("\nval= %X", val);

    return;
}

However, I never seem to get any crash data. My mbed_error_reboot_callback is never called. So something in my setup is wrong, or this hardware simply can not handle storing crash reports. My guess is that the memory segment gets erased during the boot, but that’s just a somewhat uneducated guess. Googling for this only turns up millions of posts with regular crashes, not much about getting info about them.

Hello Jan,

These links might help:

Crash dump analysis.
Crash reporting in Mbed.

The crash log parser is located in the mbed-os/tools/debug_tools/crash_log_parser folder.

Good luck, Zoltan

Oh, the crash log parser tool was news to me. I haven’t seen it mentioned anywhere before. I will see what it does with my manually generated crash.

The tool doesn’t seem to work too well, or it expects something unexpected.

% python ./mbed-os/tools/debug_tools/crash_log_parser/crash_log_parser.py app.log app.elf app.map
Traceback (most recent call last):
  File "./mbed-os/tools/debug_tools/crash_log_parser/crash_log_parser.py", line 199, in <module>
    elfhelper = ElfHelper(args.elffile, args.mapfile)
  File "./mbed-os/tools/debug_tools/crash_log_parser/crash_log_parser.py", line 36, in __init__
    op = check_output([_NM_EXEC, _OPT, elf_file.name])
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 216, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

But the files are there:

% ll app.*
-rwxr-xr-x  1 chakie  staff  5971960 Aug 15 16:41 app.elf*
-rw-r--r--  1 chakie  staff      546 Aug 15 16:34 app.log
-rw-r--r--  1 chakie  staff  5391625 Aug 15 16:41 app.map

Probably something in the map file that it can’t find, but it would be nice to know what.

Ok, the script assumes PATH is set to include the directory of the cross compiler binaries. Now I got some result:

Crash Info:
	Crash location = generateCrash(unsigned long) [0x00030410] (based on PC value)
	Caller location = main [0x000307B5] (based on LR value)
	Stack Pointer at the time of crash = [10004830]
	Target and Fault Info:
		Processor Arch: ARM-V7M or above
		Processor Variant: C23
		Forced exception, a fault with configurable priority has been escalated to HardFault
		Unaligned access error has occurred

This should be more or less what I need (the generateCrashfunction name)!

Thank you!

Hi, I have been having an issue with the crash log parser, when I run it nothing seems to come up I can’t see any output to figure out what the issue is.
how did you make yours work?

I did basically as shown above. Interestingly, I haven’t had any non obvious crashes since that and haven’t needed to use the script. But once I set my PATH correctly it worked fine. How do you run it and do you have a valid crash log? I created a small Python script that reads all my app’s console output and extracts any crash dumps into own files. Unfortunately I just cleared out all old logs and crashes and can’t show what running the script looks like for me.

Hi @chakie thanks for the input, to avoid reiteration, how ive been doing it can be seen on this thread
I have no clue why I’m getting the crash log (although I suspect it might be a memory issue based on what ive read) and I have no idea how to fix it

all the best
George

I would make a copy of the script and modify it a bit. Add in some print statements here and there to see how far it gets and to see if you get any output at all. If a print directly where it starts doesn’t show up you have some environment issues. I don’t use Windows myself, so I’m not really an expert. Windows is always a mess for development.

okay good and bad news, i am now able to get an output rather than literally nothing happening. unfortunately that output is an error, fortunately the error appears to be the same as yours. I am unsure how you set the PATH to include the cross compiler binaries. unfortunately I dont really know python so fixing the program would be extremely difficult for me
many thanks
George

You can just add the directory where arm-none-eabi-nm can be found to your normal Windows PATH. It’s set somewhere in settings under Environment or Variables or similar. It’s found in the compiler toolchain (in my case gcc-arm-none-eabi-9-2019-q4-major). Once the script can find your arm-none-eabi-nm it should work better.

Hi, thanks for your help and patience so far unfortunately I will need you to bear with my cluelessness just a bit longer.
I have been looking for the location of arm-none-eabi-nm and can’t seem to find anything
the closes path I’ve found for what it seems like I need
C:\Users\georg.LAPTOP-MIF8IBGQ\Mbed Programs\SDMotors\mbed-os\connectivity\drivers\mbedtls\FEATURE_CRYPTOCELL310\binaries\TOOLCHAIN_GCC_ARM
however, I don’t see what I need there.
ive looked elsewhere but I have no clue where to find it

On my machine the path is C:\Program Files (x86)\GNU Tools ARM Embedded\9 2019-q4-major\bin\arm-none-eabi-nm.exe. It should be in Program Files somewhere.

Thank you for that!!
I cant seem to find anything like gnu tools… in either program files or program files x86
please let me know if I am just being a fool


You may need to install it from here: Downloads | GNU Arm Embedded Toolchain Downloads – Arm Developer. Just check “Add to PATH” in the installer and you should be good.

1 Like

THANK YOU IT WORKED! I am very apreciative of everyone who has helped so far!

Crash Info:
        Crash location = SHT$$INIT_ARRAY$$Limit [0xFFFFFFFF] (based on PC value)
        Caller location = SHT$$INIT_ARRAY$$Limit [0xFFFFFFFF] (based on LR value)
        Stack Pointer at the time of crash = [1FFFDBE4]
        Target and Fault Info:
                Processor Arch: ARM-V7M or above
                Processor Variant: C24
                Forced exception, a fault with configurable priority has been escalated to HardFault
                MPU or Execute Never (XN) default memory map access violation on an instruction fetch has occurred

(now just need to figure out how to actually fix the error lol)