ARMCC - FromELF multiple regions #5021

chrissnow · 2017-09-05T12:58:33Z

Description

ELF files containing multiple regions were being converted to a folder with each region as a separate file, this caused the build scripts to fail since the .bin was not created.

This was noticed when building LPC1768 with a bootloader as the CRP word is set as a seperate region.

mbed-os/targets/TARGET_NXP/TARGET_LPC176X/device/TOOLCHAIN_ARM_STD/startup_LPC17xx.S

Line 86 in 5bddd88

AREA |.ARM.__at_0x02FC|, CODE, READONLY

Status

READY

Migrations

NO

Related PRs

Todos

Deploy notes

Steps to test or reproduce

Enable bootloader support for LPC1768 and try and build

Link: mbed-os-example-bootloader-blinky_application
[DEBUG] Link: C:/Keil_v5/ARM/ARMCC\bin\armlink --via .\BUILD\lpc1768\arm\.link_files.txt
[DEBUG] Return: 0
Elf2Bin: mbed-os-example-bootloader-blinky_application
[DEBUG] FromELF: C:/Keil_v5/ARM/ARMCC\bin\fromelf --bin -o .\BUILD\lpc1768\arm\mbed-os-example-bootloader-blinky_application.bin .\BUILD\lpc1768\arm\mbed-os-example-bootloader-blinky_application.elf
[DEBUG] Return: 1
[DEBUG] Errors: Error: Q0147E: Failed to create Directory .\BUILD\lpc1768\arm\mbed-os-example-bootloader-blinky_application.bin\ER$$.ARM.__at_0x02FC: File exists
[DEBUG] Errors: Finished: 0 information, 0 warning and 1 error messages.
Traceback (most recent call last):
  File "c:\Code\Chris\mbed-os-example-bootloader-blinky\mbed-os\tools\make.py", line 296, in <module>
    stats_depth=options.stats_depth)
  File "c:\Code\Chris\mbed-os-example-bootloader-blinky\mbed-os\tools\build_api.py", line 544, in build_project
    res, _ = toolchain.link_program(resources, build_path, name + "_application")
  File "c:\Code\Chris\mbed-os-example-bootloader-blinky\mbed-os\tools\toolchains\__init__.py", line 1128, in link_program
    self.binary(r, elf, bin)
  File "c:\Code\Chris\mbed-os-example-bootloader-blinky\mbed-os\tools\hooks.py", line 51, in wrapper
    res = function(t_self, *args, **kwargs)
  File "c:\Code\Chris\mbed-os-example-bootloader-blinky\mbed-os\tools\toolchains\arm.py", line 229, in binary
    self.default_cmd(cmd)
  File "c:\Code\Chris\mbed-os-example-bootloader-blinky\mbed-os\tools\toolchains\__init__.py", line 1152, in default_cmd
    raise ToolException(_stderr)
ToolException: Error: Q0147E: Failed to create Directory .\BUILD\lpc1768\arm\mbed-os-example-bootloader-blinky_application.bin\ER$$.ARM.__at_0x02FC: File exists
Finished: 0 information, 0 warning and 1 error messages.

[mbed] ERROR: "c:\python27\python.exe" returned error code 1.
[mbed] ERROR: Command "c:\python27\python.exe -u c:\Code\Chris\mbed-os-example-bootloader-blinky\mbed-os\tools\make.py -t arm -m lpc1768 --source . --build .\BUILD\lpc1768\arm -v" in "c:\Code\Chris\mbed-os-example-bootloader-blinky"
---

theotherjimmy

Awesome diff. How backwards compatible do you think this is?

chrissnow · 2017-09-05T14:11:07Z

It shouldn't make any difference if there is a single region.
http://www.keil.com/support/man/docs/armutil/armutil_pge1362128886407.htm

--bincombined
Produces plain binary output. It generates one output file for an image containing multiple load regions.

chrissnow · 2017-09-05T14:13:59Z

http://www.keil.com/support/man/docs/armutil/armutil_pge1362128885938.htm

--bin
If you convert an ELF image containing multiple load regions to a binary format, fromelf creates an output directory named destination and generates one binary output file for each load region in the input image. fromelf places the output files in the destination directory.

theotherjimmy · 2017-09-05T14:15:53Z

Thanks for the clarification. I think that --bincombined is probably what we wanted in the first place anyway.

chrissnow · 2017-09-05T19:09:04Z

How can I find the reason for failure?

theotherjimmy · 2017-09-05T19:11:41Z

@chrissnow I'm looking for the error right now. Just a moment.

theotherjimmy · 2017-09-05T19:18:33Z

I think the logging borked for that job. @tommikas Could you verify? It looks very suspicious:

[eth-8023] Traceback (most recent call last):
[eth-8023]   File "/usr/lib/python2.7/logging/__init__.py", line 861, in emit
[eth-8023]     msg = self.format(record)
[eth-8023]   File "/usr/lib/python2.7/logging/__init__.py", line 734, in format
[eth-8023]     return fmt.format(record)
[eth-8023]   File "/usr/lib/python2.7/logging/__init__.py", line 465, in format
[eth-8023]     record.message = record.getMessage()
[eth-8023]   File "/usr/lib/python2.7/logging/__init__.py", line 329, in getMessage
[eth-8023]     msg = msg % self.args
[eth-8023] TypeError: not all arguments converted during string formatting
[eth-8023] Logged from file opentmi_client.py, line 189
[eth-8023] Traceback (most recent call last):
[eth-8023]   File "/usr/lib/python2.7/logging/__init__.py", line 861, in emit
[eth-8023]     msg = self.format(record)
[eth-8023]   File "/usr/lib/python2.7/logging/__init__.py", line 734, in format
[eth-8023]     return fmt.format(record)
[eth-8023]   File "/usr/lib/python2.7/logging/__init__.py", line 465, in format
[eth-8023]     record.message = record.getMessage()
[eth-8023]   File "/usr/lib/python2.7/logging/__init__.py", line 329, in getMessage
[eth-8023]     msg = msg % self.args
[eth-8023] TypeError: not all arguments converted during string formatting

theotherjimmy · 2017-09-05T19:19:02Z

@chrissnow I'm not sure why it failed. Maybe @tommikas can help us out with this one.

tommikas · 2017-09-06T05:31:11Z

@theotherjimmy Looks like some 6lowpan tests failed. A few other builds also failed for the same reason so it's probably nothing to do with this PR. I'll restart the builds.

chrissnow · 2017-09-06T10:36:48Z

@theotherjimmy it passed this time :-) , Thanks @tommikas

…ltiple regions are converted to a single binary and not a folder of regions

0xc0170 · 2017-09-20T10:12:02Z

cc @c1728p9

c1728p9 · 2017-09-20T14:37:19Z

@chrissnow, with this change are you able to get mbed-os-example-bootloader-blinky_application running, or does this only enable you to build a binary? I would imagine this change still does not create a bootable image.

Normally the CRP region (0x02FC) lies inside a load region as seen in the lpc1768 scatter file. If you are building blinky with a bootloader then the application is at a higher address, so the CRP (fixed at 0x02FC) no longer resides in in a load region.

A safer solution would be to put the CRP data in a section relative to the load region. That way when you build a program with a bootloader the CRP moves with the application. An example of this can be found in the K64F scatter file which has a dedicated execution region in the load region for flash settings.

chrissnow · 2017-09-20T14:51:00Z

@c1728p9 With this change alone no, see #5038, this just fixed the build tools should multiple regions exist for whatever reason. CRP is addressed in there (partly in line with your suggestion).

c1728p9 · 2017-09-20T15:11:47Z

@chrissnow, with the changes in #5038 do you still see two separate regions created?

chrissnow · 2017-09-20T15:16:20Z

@c1728p9 not as it stands, though that might not be the case if I add extra ER as suggested.

c1728p9 · 2017-09-20T15:26:24Z

@chrissnow I believe an extra binary will only be created if there are multiple LRs (Load Regions). Extra ERs (Execution Regions) shouldn't be a problem, as many targets already have several ERs.

Typically when there are multiple LRs and building creates a directory rather than a binary it indicates something is wrong in the scatter file. Adding this change may mask scatter file errors. Rather than having an obvious error at compile time, a binary will be created that does not work.

0xc0170 · 2017-09-21T11:23:29Z

@chrissnow I believe an extra binary will only be created if there are multiple LRs (Load Regions). Extra ERs (Execution Regions) shouldn't be a problem, as many targets already have several ERs.

From --bincombined: Produces plain binary output. It generates one output file for an image containing multiple load regions. That is correct.

I found few devices, LPC407X_8X, also REALTEK has 3. Is this regular use case to have multiple load regions?

theotherjimmy · 2017-09-21T14:18:02Z

/morph test

mbed-bot · 2017-09-22T10:16:04Z

Result: ABORTED

Your command has finished executing! Here's what you wrote!

/morph test

0xc0170 · 2017-09-22T10:16:23Z

This was aborted, needs an approval from @c1728p9

c1728p9 · 2017-09-22T14:23:24Z

It looks like the LPC4088_DM's post build hook makes use of multiple regions being created if the binary is big enough. It does this by manually combining the regions with little padding (up to 512K). With this PR padding is done automatically, but rather than 512K it pads the entire address space from LR_IROM1 to LR_IROM which is ~640MB.

chrissnow · 2017-09-22T14:33:35Z

Short of using hex files over bin there isn't much you can do about this.

If nothing else the build tools should deal with this gracefully, it's a bit cryptic as it stands.

0xc0170 · 2017-09-26T12:58:57Z

If nothing else the build tools should deal with this gracefully, it's a bit cryptic as it stands.

@theotherjimmy @c1728p9 What are you opinions for this?

c1728p9 · 2017-09-26T16:51:27Z

Maybe the mbed-os build tools should print a nicely formatted error message if this occurs and there isn't a post build hook for the platform. @theotherjimmy what do you think?

theotherjimmy · 2017-09-26T18:18:45Z

@c1728p9 That sounds fine. What would you like the tools to say?

0xc0170 · 2017-10-09T12:38:24Z

@theotherjimmy @c1728p9 bump

studavekar · 2017-10-11T03:03:49Z

/morph build

mbed-ci · 2017-10-11T05:16:12Z

Build : SUCCESS

Build number : 96
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/5021/

Triggering tests

/test mbed-os

mbed-ci · 2017-10-11T15:38:51Z

Test : SUCCESS

Build number : 34
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/5021/34

0xc0170 · 2017-10-13T10:44:18Z

Maybe the mbed-os build tools should print a nicely formatted error message if this occurs and there isn't a post build hook for the platform. @theotherjimmy what do you think?

I set this to needs: clarification. I would like to see this closed with a new tracking issue for tools

0xc0170 · 2017-11-20T16:21:21Z

Tracking issue created #5537. We can close this for now

@chrissnow Thanks for the effort!

theotherjimmy approved these changes Sep 5, 2017

View reviewed changes

0xc0170 added the needs: review label Sep 5, 2017

Change fromELF parameter bin to bincombined so that elf files with mu…

3bffb49

…ltiple regions are converted to a single binary and not a folder of regions

chrissnow force-pushed the ARMCC-FromELF branch from 08a544a to 3bffb49 Compare September 19, 2017 09:08

0xc0170 added needs: CI and removed needs: review labels Sep 20, 2017

0xc0170 approved these changes Sep 20, 2017

View reviewed changes

0xc0170 added needs: clarification and removed needs: CI labels Oct 13, 2017

adbridge removed the needs: clarification label Nov 9, 2017

theotherjimmy added the needs: work label Nov 13, 2017

0xc0170 closed this Nov 20, 2017

0xc0170 removed the needs: work label Nov 20, 2017

0xc0170 mentioned this pull request Nov 20, 2017

Tools: multiple region - tools error msg improvement #5537

Closed

0xc0170 mentioned this pull request Mar 13, 2018

armcc - remove fromelf output before regenerating one #6344

Merged

5 tasks

0xc0170 mentioned this pull request Jan 14, 2019

Add binary hook for ARMC6 which use --bincombined for .bin outputs #9320

Closed

ARMCC - FromELF multiple regions #5021

ARMCC - FromELF multiple regions #5021

Uh oh!

Conversation

chrissnow commented Sep 5, 2017

Description

Status

Migrations

Related PRs

Todos

Deploy notes

Steps to test or reproduce

Uh oh!

theotherjimmy left a comment

Choose a reason for hiding this comment

Uh oh!

chrissnow commented Sep 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chrissnow commented Sep 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

theotherjimmy commented Sep 5, 2017

Uh oh!

chrissnow commented Sep 5, 2017

Uh oh!

theotherjimmy commented Sep 5, 2017

Uh oh!

theotherjimmy commented Sep 5, 2017

Uh oh!

theotherjimmy commented Sep 5, 2017

Uh oh!

tommikas commented Sep 6, 2017

Uh oh!

chrissnow commented Sep 6, 2017

Uh oh!

0xc0170 commented Sep 20, 2017

Uh oh!

c1728p9 commented Sep 20, 2017

Uh oh!

chrissnow commented Sep 20, 2017

Uh oh!

c1728p9 commented Sep 20, 2017

Uh oh!

chrissnow commented Sep 20, 2017

Uh oh!

c1728p9 commented Sep 20, 2017

Uh oh!

0xc0170 commented Sep 21, 2017

Uh oh!

theotherjimmy commented Sep 21, 2017

Uh oh!

mbed-bot commented Sep 22, 2017

Result: ABORTED

Uh oh!

0xc0170 commented Sep 22, 2017

Uh oh!

c1728p9 commented Sep 22, 2017

Uh oh!

chrissnow commented Sep 22, 2017

Uh oh!

0xc0170 commented Sep 26, 2017

Uh oh!

c1728p9 commented Sep 26, 2017

Uh oh!

theotherjimmy commented Sep 26, 2017

Uh oh!

0xc0170 commented Oct 9, 2017

Uh oh!

studavekar commented Oct 11, 2017

Uh oh!

mbed-ci commented Oct 11, 2017

Build : SUCCESS

Triggering tests

Uh oh!

mbed-ci commented Oct 11, 2017

Test : SUCCESS

Uh oh!

0xc0170 commented Oct 13, 2017

Uh oh!

chrissnow commented Sep 5, 2017 •

edited

Loading

chrissnow commented Sep 5, 2017 •

edited

Loading