Reserve and Render header in managed BL mode #5950

theotherjimmy · 2018-01-26T16:44:49Z

Detailed Design

The header specification is a JSON ordered list of
JSON tuples in the target.header_format configu-
ration key. The tuple represents a single element of
the header format as first the name (a valid C identi-
fier), then a type, a subtype and finally a single argu-
ment. For example, the const type defines subtypes
for common sizes of constants, including the 32 sub-
type that indicates the constant will be represented
as 32 bits. The following is a list of all types and
subtypes that we plan to support.

const A constant value.
- 8be A value represented as 8 bits.
- 16be A value represented as 16 bits big endian.
- 32be A value represented as 32 bits big endian.
- 64be A value represented as 64 bits big endian.
- 8le A value represented as 8 bits.
- 16le A value represented as 16 bits little endian.
- 32le A value represented as 32 bits little endian.
- 64le A value represented as 64 bits little endian.
timestamp A time stamp value, in seconds from
epoch in the timezone GMT/UTC.
This type’s argument is always null.
- 64be A time stamp value truncated to 64 bits big endian.
- 64le A time stamp value truncated to 64 bits ilttle endian.
digest A digest of an image. All digests are
computed in the order they appear. A digest of the header
digests the header up to its start address.
- CRCITT32be A big endian 32bit checksum of an image.
- CRCITT32be A little endian 32bit checksum of an image.
- SHA256 A SHA-2 using a block-size of 256
  bits.
- SHA512 A SHA-2 using a block-size of 512
  bits.
size The size of a list of images, added together.
- 32be A size represented as 32 bits big endian.
- 64be A size represented as 64 bits big endain.
- 32le A size represented as 32 bits little endian.
- 64le A size represented as 64 bits little endain.

The addition of new types and new subtypes must
come in a feature release.

Items in the header are packed starting where
the previous field ended; Items in the header are
packed using the C “packed” struct semantics. The
presence of the target.header format field de-
fines two macros MBED_HEADER_START, which expands
to the start address of the firmware header, and
MBED_HEADER_SIZE, containing the size in bytes of
the firmware header.The region following the
application header starts at
MBED_HEADER_START + MBED_HEADER_SIZE
rounded up to a multiple of 8
bytes.

Example

I wrote up a mbed_lib.json that covers a firmware header I have seen.
The JSON description follows.

{                                                                                                        
    "name": "bootloader_images",                                                                         
    "target_overrides": {                                                                                
        "*": {                                                                                           
            "target.header_format": [                                                                    
                ["magic", "const", "32be", "0x5a51b3d4"],                                                  
                ["version", "const", "32be", "2"],                                                         
                ["firmwareVersion", "timestamp", "64be", null],                                            
                ["firmwareSize", "size", "64be", ["application"]],                                         
                ["firmwareHash", "digest", "SHA256", "application"],                                     
                ["hashpad", "const", "64be", "0"],                                                         
                ["hashpad", "const", "64be", "0"],                                                         
                ["hashpad", "const", "64be", "0"],                                                         
                ["hashpad", "const", "64be", "0"],                                                         
                ["campaign", "const", "64be", "0"],                                                        
                ["campaign", "const", "64be", "0"],                                                        
                ["firmwareSignatureSize", "const", "32be", "0"],                                           
                ["headerCRC", "digest", "CRCITT32be", "header"]                                            
            ]                                                                                            
        },                                                                                               
        "K64F": {                                                                                        
            "target.bootloader_img": "K64F.bin"                                                          
        },                                                                                               
        "NUCLEO_F429ZI": {                                                                               
            "target.bootloader_img": "NUCLEO_F429ZI.bin"                                                 
        },                                                                                               
        "UBLOX_EVK_ODIN_W2": {                                                                           
            "target.bootloader_img": "UBLOX_EVK_ODIN_W2.bin"                                             
        }                                                                                                
    }                                                                                                    
}

This produces a header as follows:

$ mbed compile -t gcc_arm -m k64f
[snip]
Using regions in this build:
  Region bootloader size 0x20000, offset 0x0
  Region header size 0x70, offset 0x20000
  Region application size 0xdff90, offset 0x20070
Merging Regions:
  Filling region bootloader with /home/jimmy/src/armmbed/mbed-os-example-bootloader-blinky/bootloader/K64F.bin
  Filling region header with ./BUILD/k64f/gcc_arm/mbed-os-example-bootloader-blinky_header.hex
  Filling region application with ./BUILD/k64f/gcc_arm/mbed-os-example-bootloader-blinky_application.bin
Space used after regions merged: 0x2c000

For reference the application compiled to 0xbf90 bytes. which makes application + header 0xc000 bytes.

The header's hex dump is as follows:

00020000: 5a51 b3d4 0000 0002 0000 0000 5a74 e345  ZQ..........Zt.E
00020010: 0000 0000 0005 d960 290e 9447 e293 673e  .......`)..G..g>
00020020: 67b2 5e2a e8e5 5443 2a13 0df1 01d6 f0e8  g.^*..TC*.......
00020030: 6ea9 194f 535f 226e 0000 0000 0000 0000  n..OS_"n........
00020040: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00020050: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00020060: 0000 0000 0000 0000 0000 0000 b22c 879e  .............,..

TODO

Testing

theotherjimmy · 2018-01-26T16:45:44Z

@thegecko @JoeTheGuitarist Could you tag a bunch of people on this? Update and Client teams need to test this for themselves.

JoeTheGuitarist · 2018-01-26T17:15:00Z

@seankinghan @yogpan01 @evedon

yogpan01 · 2018-01-30T09:15:53Z

@evedon @LiyouZhou Can you guide @teetak01 and @saheerb on how they can test this PR with client test setup ?

seankinghan · 2018-01-30T09:41:56Z

It's very elegant. I do have one question about the calculation of CRCs. At the moment we are calculating the CRC by generating it for all fields up to but not including the CRC field itself. Although there is currently no need to generate it by including the CRC field itself, all zeroed, is there a way to do this - it was done this way in the past, so might reappear for some reason.

seankinghan · 2018-01-30T09:43:25Z

Some minor things:

"and
MBED_HEADER_SIZ E, containing the size in bytes of
the firmware header."

and

    ["firmawerHash", "digest", "SHA256", "application"],

theotherjimmy · 2018-01-30T15:00:06Z

@seankinghan I could calculate any digests as "up to their start address", or "all uncomputed checksums are 0s" Which would you prefer?

seankinghan · 2018-01-30T15:08:41Z

In this particular case of the header CRC it needs to be "up to". Whether or not that applies to, for example, a version 1 header, I don't know. Let me get back to you.

seankinghan · 2018-01-30T15:21:52Z

In connection with validating this change, we are trying to create a build with this new capability, and having some issues. I think. @theotherjimmy would you mind taking a look at this PR and letting me know what we are doing wrong, if it is at all evident?

https://github.com/ARMmbed/mbed-cloud-client-example-internal/pull/280

theotherjimmy · 2018-01-30T15:51:11Z

@seankinghan Discussion has moved to the referenced thread, and is progressing there.

yogpan01 · 2018-02-20T12:46:01Z

What's the status with this PR ? We need this in mbed OS 5.8.
@theotherjimmy @evedon @LiyouZhou @seankinghan Why is this stalled ?
@jenia81 @shelib01 FYI

theotherjimmy · 2018-02-20T14:37:36Z

@yogpan01 This PR is stalled awaiting confirmation that it will resolve the requirements for using managed BL mode in Mbed Client / Update Client for the near future. See the appropriate prototype PRs in those repos for more details.

seankinghan · 2018-02-28T08:17:47Z

The original fields were restrict_size and bootloader_img.

In that case they are actually a lot more consistent and communicative than they were originally, good.

theotherjimmy · 2018-02-28T14:46:06Z

@sarahmarshy Review please.

theotherjimmy · 2018-02-28T17:28:02Z

/morph build

mbed-ci · 2018-02-28T17:52:38Z

Build : FAILURE

Build number : 1301
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/5950/

theotherjimmy · 2018-02-28T17:56:06Z

License failure
/morph build

mbed-ci · 2018-02-28T18:18:52Z

Build : SUCCESS

Build number : 1302
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/5950/

Triggering tests

/morph test
/morph uvisor-test
/morph export-build
/morph mbed2-build

sarahmarshy

Overall, LGTM. No glaring problems. My only feedback would be to add a few more comments.

sarahmarshy · 2018-02-28T18:51:10Z

tools/config/__init__.py

+        size = self._header_size(header_format)
+        region = Region("header", start, size, False, None)
+        start += size
+        start = ((start + 7) // 8) * 8


Could you explain this logic in a comment?

That's pretty standard "round up to a multiple of 8" logic. Do you want an explanation of why 8 or something else?

Yeah, I got that it is a round up to a multiple of 8, but I haven't seen it that often so had to logic it out myself. I think just a comment saying "round up to multiple of 8" would help people who haven't seen it before. Or you could add a comment for the entire function about the region it returns/its alignment.

I have a strong preference for commenting the entire function. I'll do that, if you don't mind.

mbed-ci · 2018-02-28T19:53:50Z

Exporter Build : SUCCESS

Build number : 962
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/5950/

mbed-ci · 2018-02-28T22:15:46Z

Test : SUCCESS

Build number : 1086
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/5950/1086

cmonr · 2018-03-01T00:53:18Z

Need a check mark from a reviewer.

cmonr · 2018-03-01T00:53:40Z

@seankinghan @yogpan01 @evedon

adbridge · 2018-03-01T11:38:01Z

CI has passed but this is blocked pending review approval from one of:
@seankinghan @yogpan01 @evedon

adbridge · 2018-03-01T16:23:28Z

Bogdan has approved this so I am also doing so to allow for merging

AGlass0fMilk · 2018-05-17T15:18:58Z

@theotherjimmy Is there documentation for this feature yet?

cmonr · 2018-05-18T03:07:50Z

@AGlass0fMilk Does this page not fit the bill?
https://os.mbed.com/docs/v5.8/tutorials/bootloader.html

AGlass0fMilk · 2018-05-18T03:35:22Z

Thanks @cmonr

I wasn’t sure what part of the handbook might already pertain to this.

theotherjimmy added needs: review needs: preceding PR labels Jan 26, 2018

theotherjimmy mentioned this pull request Jan 26, 2018

Let libraries, targets configure bootloader #5909

Merged

1 task

theotherjimmy force-pushed the header-config branch 2 times, most recently from 09ca7e7 to 17291bf Compare January 26, 2018 20:52

0xc0170 requested review from yogpan01, evedon and seankinghan January 29, 2018 12:50

theotherjimmy force-pushed the header-config branch from 17291bf to c0f1080 Compare January 29, 2018 16:48

theotherjimmy force-pushed the header-config branch from c0f1080 to 7726c49 Compare January 30, 2018 15:11

theotherjimmy force-pushed the header-config branch 3 times, most recently from a1d50df to 7c7171d Compare February 2, 2018 22:35

theotherjimmy added do not merge and removed needs: preceding PR needs: review labels Feb 2, 2018

theotherjimmy force-pushed the header-config branch from 19fcf0f to acb37cb Compare February 5, 2018 17:44

theotherjimmy force-pushed the header-config branch 2 times, most recently from 9b732fc to 4f4e645 Compare February 14, 2018 15:46

theotherjimmy added needs: CI and removed needs: review labels Feb 28, 2018

sarahmarshy approved these changes Feb 28, 2018

View reviewed changes

theotherjimmy added the release-version: 5.8.0-rc1 label Feb 28, 2018

cmonr added needs: review ready for merge and removed needs: CI labels Mar 1, 2018

adbridge removed the ready for merge label Mar 1, 2018

bogdanm approved these changes Mar 1, 2018

View reviewed changes

adbridge self-requested a review March 1, 2018 16:23

adbridge approved these changes Mar 1, 2018

View reviewed changes

adbridge added ready for merge and removed needs: review labels Mar 1, 2018

adbridge merged commit 8f5b857 into ARMmbed:master Mar 1, 2018

adbridge removed the ready for merge label Mar 1, 2018

Reserve and Render header in managed BL mode #5950

Reserve and Render header in managed BL mode #5950

Uh oh!

Conversation

theotherjimmy commented Jan 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Detailed Design

Example

TODO

Uh oh!

theotherjimmy commented Jan 26, 2018

Uh oh!

JoeTheGuitarist commented Jan 26, 2018

Uh oh!

yogpan01 commented Jan 30, 2018

Uh oh!

seankinghan commented Jan 30, 2018

Uh oh!

seankinghan commented Jan 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

theotherjimmy commented Jan 30, 2018

Uh oh!

seankinghan commented Jan 30, 2018

Uh oh!

seankinghan commented Jan 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

theotherjimmy commented Jan 30, 2018

Uh oh!

yogpan01 commented Feb 20, 2018

Uh oh!

theotherjimmy commented Feb 20, 2018

Uh oh!

seankinghan commented Feb 28, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

theotherjimmy commented Feb 28, 2018

Uh oh!

theotherjimmy commented Feb 28, 2018

Uh oh!

mbed-ci commented Feb 28, 2018

Build : FAILURE

Uh oh!

theotherjimmy commented Feb 28, 2018

Uh oh!

mbed-ci commented Feb 28, 2018

Build : SUCCESS

Triggering tests

Uh oh!

sarahmarshy left a comment

Choose a reason for hiding this comment

Uh oh!

sarahmarshy Feb 28, 2018

Choose a reason for hiding this comment

Uh oh!

theotherjimmy Feb 28, 2018

Choose a reason for hiding this comment

Uh oh!

sarahmarshy Feb 28, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

theotherjimmy Feb 28, 2018

Choose a reason for hiding this comment

Uh oh!

mbed-ci commented Feb 28, 2018

Exporter Build : SUCCESS

Uh oh!

mbed-ci commented Feb 28, 2018

Test : SUCCESS

Uh oh!

cmonr commented Mar 1, 2018

Uh oh!

cmonr commented Mar 1, 2018

Uh oh!

adbridge commented Mar 1, 2018

Uh oh!

adbridge commented Mar 1, 2018

Uh oh!

theotherjimmy commented Jan 26, 2018 •

edited

Loading

seankinghan commented Jan 30, 2018 •

edited

Loading

seankinghan commented Jan 30, 2018 •

edited

Loading

seankinghan commented Feb 28, 2018 •

edited

Loading

sarahmarshy Feb 28, 2018 •

edited

Loading