Skip to content

Adapt mbed-cmake's automatic uploading and debugging system #14010

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 7 commits into from

Conversation

multiplemonomials
Copy link
Contributor

Summary of changes

This PR ports over one of the best features of mbed-cmake (in my opinion at least): automatic uploading and debugging from the command line. In short, it provides two useful features: the ability to automatically flash code after building with a special make target (flash-<executable name>), and the ability to automatically start a debug server with the gdbserver make target. It supports a number of different upload tools (not just pyOCD or the USB drive interface), including OpenOCD and J-Link, so it's friendly to custom board users.

A more complete description of how to use it is available on the doc page.

I've refactored the original mbed-cmake implementation to clean it up and split out each upload method into its own file. This makes the upload methods easier to edit and even lets users add their own upload methods that can be called via our tool.

Possible future enhancements include additional upload methods (does the LPC-Link have a specific upload tool or GDB server?) and the ability to auto detect the correct USB drive of attached Mbed devices (can mbed-tools do this?). It also might be neat to create a preexisting config file for each Mbed board that is automatically used, though this will take a fair bit of work and testing.

Impact of changes

CMake build scripts will need to be updated to add the upload method config file as described in the documentation. If the config file isn't added, a notice will be printed and the new functionality will be disabled by the cmake scripts (UPLOAD_METHOD will be set to NONE).

Migration actions required

Documentation

I created a doc page here as part of the code. Eventually we should add this to the actual docs, but since there's no cmake stuff in there yet I held off on adding it.


Pull request type

[] Patch update (Bug fix / Target update / Docs update / Test update / Refactor)
[X] Feature update (New feature / Functionality change / New API)
[] Major update (Breaking change E.g. Return code change / API behaviour change)

Test results

[X] No Tests required for this change (E.g docs only update)
[] Covered by existing mbed-os tests (Greentea or Unittest)
[] Tests / results supplied as part of this PR

Reviewers

@0xc0170
@hugueskamba

@ciarmcom
Copy link
Member

ciarmcom commented Dec 8, 2020

@multiplemonomials, thank you for your changes.
@0xc0170 @hugueskamba @ARMmbed/mbed-os-tools @ARMmbed/mbed-os-maintainers please review.

@ghost
Copy link

ghost commented Dec 8, 2020

Ah, I've been wanting a way to do this for nrfproj and the thing adafruit uses for their uf2 based bootloaders.

Although it seems like few of these clearly support dfu.

@multiplemonomials
Copy link
Contributor Author

multiplemonomials commented Dec 9, 2020

@jrobeson Do you want to help me test an nrfjprog upload method? I'd definitely like to add support for it, and if you've got hardware that you can test with I'd appreciate your help. If so, can you send me the command you're using to flash your device as a starting point?

@ghost
Copy link

ghost commented Dec 9, 2020

I think i just followed the nrfproject commnad in the documentation. I've mostly been playing with adafruit's somewhat mbed equivalent like so

adafruit-nrfutil dfu genpkg --dev-type 0x0052 --application /path/to/app_name_application.hex dfu-package.zip
dafruit-nrfutil --verbose dfu serial --package dfu-package.zip -p /dev/ttyACM0 -b 115200 --singlebank --touch 1200

I've mostly been playing with stuff that will hopefully/eventually give me/us FOTA.

@AGlass0fMilk and others have finally gotten support for mbed os applications merged into https://github.com/mcu-tools/mcuboot as per https://github.com/AGlass0fMilk/mbed-mcuboot-demo so that's what i'll be trying next.

@AGlass0fMilk
Copy link
Member

I've mostly been playing with stuff that will hopefully/eventually give me/us FOTA.

@AGlass0fMilk and others have finally gotten support for mbed os applications merged into https://github.com/mcu-tools/mcuboot as per https://github.com/AGlass0fMilk/mbed-mcuboot-demo so that's what i'll be trying next.

If you're interested in FOTA you will want to check this out if you haven't already: ARMmbed/mbed-os-experimental-ble-services#13

I have a demo of FOTA working but I have yet to push up some changes/documentation... maybe tomorrow.

@ghost
Copy link

ghost commented Dec 10, 2020

wow @AGlass0fMilk. I really appreciate you pointing that out, and the repository generally.

@adbridge
Copy link
Contributor

adbridge commented Jan 7, 2021

@0xc0170 @hugueskamba please review

@multiplemonomials
Copy link
Contributor Author

@0xc0170 @hugueskamba Have you had a chance to look at this yet? Any comments?

@0xc0170
Copy link
Contributor

0xc0170 commented Jan 18, 2021

@multiplemonomials sorry for the delay reviewing this one. We should review this week.

Copy link
Contributor

@ladislas ladislas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First review, just for typos. A lot of double spaces everywhere that need fixing :)

I've spotted the first ones, but you'll be better using grep and fixing this locally.

Comment on lines 14 to 21
|Name|CMake Argument|Supports Uploading|Supports Debugging|Upload Speed|Supported On
|---|---|---|---|---|---|
|No upload method|`-DUPLOAD_METHOD=NONE`|❌|❌|N/A|N/A
|MBed USB|`-DUPLOAD_METHOD=MBED`|✔️|❌|Fast|All Mbed Boards
|J-Link|`-DUPLOAD_METHOD=JLINK`|✔️|✔️|Fast|Mbed boards with J-Link On-Board. Custom boards with a J-Link probe.
|pyOCD|`-DUPLOAD_METHOD=PYOCD`|✔️|✔️|Medium|Almost all Mbed boards. Custom boards with an ST-Link or DAPLink probe.
|OpenOCD|`-DUPLOAD_METHOD=OPENOCD`|✔️|✔️|(Varies)|A number of different debug probes, but requires configuration.
|NXP Bootloader|`-DUPLOAD_METHOD=NXPPROG`|✔️|❌|Slow|LPC1768 and LPC1114 on custom boards.
|STM32Cube|`-DUPLOAD_METHOD=STM32CUBE`|✔️|✔️|Fast|All STMicroelectronics Mbed boards, custom boards with ST_Link probes.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better rendering of the table

Suggested change
|Name|CMake Argument|Supports Uploading|Supports Debugging|Upload Speed|Supported On
|---|---|---|---|---|---|
|No upload method|`-DUPLOAD_METHOD=NONE`|❌|❌|N/A|N/A
|MBed USB|`-DUPLOAD_METHOD=MBED`|✔️|❌|Fast|All Mbed Boards
|J-Link|`-DUPLOAD_METHOD=JLINK`|✔️|✔️|Fast|Mbed boards with J-Link On-Board. Custom boards with a J-Link probe.
|pyOCD|`-DUPLOAD_METHOD=PYOCD`|✔️|✔️|Medium|Almost all Mbed boards. Custom boards with an ST-Link or DAPLink probe.
|OpenOCD|`-DUPLOAD_METHOD=OPENOCD`|✔️|✔️|(Varies)|A number of different debug probes, but requires configuration.
|NXP Bootloader|`-DUPLOAD_METHOD=NXPPROG`|✔️|❌|Slow|LPC1768 and LPC1114 on custom boards.
|STM32Cube|`-DUPLOAD_METHOD=STM32CUBE`|✔️|✔️|Fast|All STMicroelectronics Mbed boards, custom boards with ST_Link probes.
| Name | CMake Argument | Supports Uploading | Supports Debugging | Upload Speed | Supported On |
|------------------ |----------------------------- |-------------------- |-------------------- |-------------- |-------------------------------------------------------------------------- |
| No upload method | `-DUPLOAD_METHOD=NONE` ||| N/A | N/A |
| MBed USB | `-DUPLOAD_METHOD=MBED` | ✔️ || Fast | All Mbed Boards |
| J-Link | `-DUPLOAD_METHOD=JLINK` | ✔️ | ✔️ | Fast | Mbed boards with J-Link On-Board. Custom boards with a J-Link probe. |
| pyOCD | `-DUPLOAD_METHOD=PYOCD` | ✔️ | ✔️ | Medium | Almost all Mbed boards. Custom boards with an ST-Link or DAPLink probe. |
| OpenOCD | `-DUPLOAD_METHOD=OPENOCD` | ✔️ | ✔️ | (Varies) | A number of different debug probes, but requires configuration. |
| NXP Bootloader | `-DUPLOAD_METHOD=NXPPROG` | ✔️ || Slow | LPC1768 and LPC1114 on custom boards. |
| STM32Cube | `-DUPLOAD_METHOD=STM32CUBE` | ✔️ | ✔️ | Fast | All STMicroelectronics Mbed boards, custom boards with ST_Link probes. |

# Load the upload method that the user selects

if(NOT DEFINED UPLOAD_METHOD_DEFAULT)
message(STATUS "Note: UPLOAD_METHOD_DEFAULT not found. Please create an upload method config file if you wish to use CMake's auto uploading and debugging support.")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

double space

Suggested change
message(STATUS "Note: UPLOAD_METHOD_DEFAULT not found. Please create an upload method config file if you wish to use CMake's auto uploading and debugging support.")
message(STATUS "Note: UPLOAD_METHOD_DEFAULT not found. Please create an upload method config file if you wish to use CMake's auto uploading and debugging support.")

@@ -0,0 +1,214 @@
Mbed OS's CMake build system supports a number of upload methods to help get code onto your device. Some uploaders also support debugging, and for these CMake will help you by automatically running the needed commands to start a debugging session.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

double space

Suggested change
Mbed OS's CMake build system supports a number of upload methods to help get code onto your device. Some uploaders also support debugging, and for these CMake will help you by automatically running the needed commands to start a debugging session.
Mbed OS's CMake build system supports a number of upload methods to help get code onto your device. Some uploaders also support debugging, and for these CMake will help you by automatically running the needed commands to start a debugging session.

@@ -0,0 +1,214 @@
Mbed OS's CMake build system supports a number of upload methods to help get code onto your device. Some uploaders also support debugging, and for these CMake will help you by automatically running the needed commands to start a debugging session.

In order to enable uploading, you'll need to configure which upload methods your project supports. To do this, you'll need to create an UploadMethodConfig.cmake file and include it in your top-level CMakeLists.txt. You can start by using the template UploadMethodConfig-Sample.cmake (which is for Nucleo F429ZI) in this folder. You'll then edit it to control the upload methods that are available for your target, and set the parameters that they need to operate. THe different parameters and their values are explained in this document.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

double spaces

Suggested change
In order to enable uploading, you'll need to configure which upload methods your project supports. To do this, you'll need to create an UploadMethodConfig.cmake file and include it in your top-level CMakeLists.txt. You can start by using the template UploadMethodConfig-Sample.cmake (which is for Nucleo F429ZI) in this folder. You'll then edit it to control the upload methods that are available for your target, and set the parameters that they need to operate. THe different parameters and their values are explained in this document.
In order to enable uploading, you'll need to configure which upload methods your project supports. To do this, you'll need to create an UploadMethodConfig.cmake file and include it in your top-level CMakeLists.txt. You can start by using the template UploadMethodConfig-Sample.cmake (which is for Nucleo F429ZI) in this folder. You'll then edit it to control the upload methods that are available for your target, and set the parameters that they need to operate. THe different parameters and their values are explained in this document.


In order to enable uploading, you'll need to configure which upload methods your project supports. To do this, you'll need to create an UploadMethodConfig.cmake file and include it in your top-level CMakeLists.txt. You can start by using the template UploadMethodConfig-Sample.cmake (which is for Nucleo F429ZI) in this folder. You'll then edit it to control the upload methods that are available for your target, and set the parameters that they need to operate. THe different parameters and their values are explained in this document.

This page lists each upload method, its _parameters_ (variables that will have a fixed value) and its _options_ (variables that are different for different people and should be set on the command line when you run cmake). Since this config is set by standard CMake code, it's very easy to add your own additional logic (including command line options) to adjust these variables!
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

double space

Suggested change
This page lists each upload method, its _parameters_ (variables that will have a fixed value) and its _options_ (variables that are different for different people and should be set on the command line when you run cmake). Since this config is set by standard CMake code, it's very easy to add your own additional logic (including command line options) to adjust these variables!
This page lists each upload method, its _parameters_ (variables that will have a fixed value) and its _options_ (variables that are different for different people and should be set on the command line when you run cmake). Since this config is set by standard CMake code, it's very easy to add your own additional logic (including command line options) to adjust these variables!


**Type:** Integer

This controls the port that GDB servers will be run on when debugging. A value higher than 1024 is recommended to allow debugging without root on Linux/Mac.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

double space

Suggested change
This controls the port that GDB servers will be run on when debugging. A value higher than 1024 is recommended to allow debugging without root on Linux/Mac.
This controls the port that GDB servers will be run on when debugging. A value higher than 1024 is recommended to allow debugging without root on Linux/Mac.


## MBed USB Upload Method

This upload method interfaces with standard MBed boards which present themselves as USB drives. Given the folder where the USB drive is mounted, it first removes any other .bin files from the folder, then automatically copies the compiled program on.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

double space

Suggested change
This upload method interfaces with standard MBed boards which present themselves as USB drives. Given the folder where the USB drive is mounted, it first removes any other .bin files from the folder, then automatically copies the compiled program on.
This upload method interfaces with standard MBed boards which present themselves as USB drives. Given the folder where the USB drive is mounted, it first removes any other .bin files from the folder, then automatically copies the compiled program on.


**Type:** Folder Path

Path that the MBed device is mounted to on your machine. On Windows, this is usually a drive letter, such as `D:/`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

double space

Suggested change
Path that the MBed device is mounted to on your machine. On Windows, this is usually a drive letter, such as `D:/`.
Path that the MBed device is mounted to on your machine. On Windows, this is usually a drive letter, such as `D:/`.


## J-Link Upload Method

This upload method connects to your processor via a J-Link JTAG box and the J-Link command line tools. It supports both flashing and debugging. CMake will automatically locate the J-Link tools in their standard install locations on Windows, Mac, and Linux.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

double space

Suggested change
This upload method connects to your processor via a J-Link JTAG box and the J-Link command line tools. It supports both flashing and debugging. CMake will automatically locate the J-Link tools in their standard install locations on Windows, Mac, and Linux.
This upload method connects to your processor via a J-Link JTAG box and the J-Link command line tools. It supports both flashing and debugging. CMake will automatically locate the J-Link tools in their standard install locations on Windows, Mac, and Linux.


**Type:** String

The name that your processor is known by to J-link. These are listed [here](https://www.segger.com/downloads/supported-devices.php).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

double space

Suggested change
The name that your processor is known by to J-link. These are listed [here](https://www.segger.com/downloads/supported-devices.php).
The name that your processor is known by to J-link. These are listed [here](https://www.segger.com/downloads/supported-devices.php).

@multiplemonomials
Copy link
Contributor Author

Is it... bad to have double spaces? That's what my grammar school teacher taught me...

@ladislas
Copy link
Contributor

Is it... bad to have double spaces? That's what my grammar school teacher taught me...

Well I don't think you'll go to jail for that, but looking more into it, I found this: How Many Spaces Should Be After a Period? which taught me a lot.

Pandoc will remove them when parsing from markdown to CommonMark.

I'll let the mbed os team shine in, but I'd say it's better to stick with how the current documentation is written, and it uses one space.

@ciarmcom ciarmcom added the stale Stale Pull Request label Jan 23, 2021
@ciarmcom
Copy link
Member

This pull request has automatically been marked as stale because it has had no recent activity. @0xc0170, @hugueskamba, @ARMmbed/mbed-os-maintainers, @ARMmbed/mbed-os-tools, please complete review of the changes to move the PR forward. Thank you for your contributions.

Copy link
Contributor

@0xc0170 0xc0170 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First initial review. It took me a while to get back to this after the longer break.

We used to have something similar (wrappers for flashing) in yotta. I realized after seeing gdb implementation here. What we did back then was to use gdb server by default or export to IDE that has own debug server there. Two options I can think of: simple wrapper to start gdb and load the executable with configured target - the config came from target module. Or export to IDE that has the MCU configured and just flash and start debug session there.

The tricky part is that each of flashing methods need specific configuration that depends various settings either coming from a user (MCU name, speed),

We would like to support --flash argument that old tools provided - we do already , via cp binary to usb so basically copying it. We haven't yet looked at this subject to how it shall be done (CMake/Tools parts). I'll share this with the tools team and trigger a discussion this week.

Looking at this from the old tools perspective, I recall their support for flashing is either via usb mbed drive or jlink or stlink.

Thanks for sharing all this work, impressive set of support for flashing.

Update one day later: I also remembered we got cmsis-pack manager - fetching device data - like MCU type. We used it for exporting or getting flash algo. That could be useful to get some of the required data here like MCU name.


# set executable suffix (has to be done after enabling languages)
# Note: This is nice to have, but is also required because STM32Cube will only work on files with a .elf extension
set(CMAKE_EXECUTABLE_SUFFIX .elf)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recall other IDE also had this limitation (only able to find executable in their viewer if it has specific extension, it could have been .axf for uvision). However it should work (this shall be tested before we hit the problem like STM32Cube in this case).

set(JLINK_UPLOAD_ENABLED FALSE)

if(JLINK_UPLOAD_ENABLED)
set(JLINK_CPU_NAME STM32F429ZI)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this config contains couple of values that should be stored somewhere. The only place that comes to my mind is targets.json that would have this (MCU_ target there would contain jlink/pyocd or others).

When back then we had project generator in exporters, that one has this in yaml files - debugging configuration for various exporters, each one has specific value and they do not match unfortunately).

Copy link
Contributor

@0xc0170 0xc0170 Feb 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How was this used in mbed-cmake repo? This is still the main issue for this flashing bits. Based on your latest reply, would it make sense to simplify the flashing: to initially add support for mbed flashing via copy binaries and extend it with more advanced flashing like this jlink/pyOCD - as they will need further configuration. Or we just figure it out here now (it might take a while to get this configuration for targets).

@ciarmcom ciarmcom removed the stale Stale Pull Request label Jan 25, 2021
@multiplemonomials
Copy link
Contributor Author

Okay, I broke some of the pre-requisite functionality out into other PRs.

And yeah, thinking about what you said, I suppose we can remove the NXPPROG method for now. It's not useful except in a pretty narrow use case (helping people using an LPC1768 on a custom board flash code without a programmer). It will still be available to add but not part of the base code. FWIW, the script is basically being maintained by RPL right now (we forked it), but I don't know if I want to sign up for actually maintaining it going forward.

@multiplemonomials
Copy link
Contributor Author

Update: I looked into what you said about mbed-cli having flash support, and you're right -- it handles the mbed boards a lot better than the CMake implementation does. It can automatically detect where they're mounted, and even filter them by target name and UID.

The only annoying part is that I couldn't find any public script that calls the flashing code, other than mbed compile itself. So, I wrote one: https://github.com/USCRPL/mbed-cmake/blob/mbed-pathless/cmake/upload_methods/install_bin_file.py . This script also improves handling of multiple boards of the same type (it allows selecting one by UID, instead of the mbed-cli behavior of flashing all of them).

Once some people try it out, I'll merge that in here, and then CMake will be able to use (most of) the full power of mbed's upload tools.

@ciarmcom ciarmcom added the stale Stale Pull Request label Jan 29, 2021
@ciarmcom ciarmcom added stale Stale Pull Request and removed stale Stale Pull Request labels Aug 20, 2021
@ciarmcom ciarmcom added stale Stale Pull Request and removed stale Stale Pull Request labels Aug 27, 2021
@ciarmcom ciarmcom added stale Stale Pull Request and removed stale Stale Pull Request labels Sep 3, 2021
@ciarmcom ciarmcom added stale Stale Pull Request and removed stale Stale Pull Request labels Sep 12, 2021
@ciarmcom ciarmcom added stale Stale Pull Request and removed stale Stale Pull Request labels Sep 19, 2021
@ciarmcom ciarmcom added the stale Stale Pull Request label Sep 26, 2021
@0xc0170 0xc0170 removed the stale Stale Pull Request label Sep 29, 2021
@mergify mergify bot dismissed rwalton-arm’s stale review November 1, 2021 01:39

Pull request has been modified.

@multiplemonomials
Copy link
Contributor Author

@rwalton-arm sorry for the delay, ready for another look

@multiplemonomials
Copy link
Contributor Author

multiplemonomials commented Feb 15, 2022

@0xc0170 @rwalton-arm is this able to be looked at? It it doesn't seem like merging it is in the cards, I might just make it a standalone library.

@0xc0170
Copy link
Contributor

0xc0170 commented Feb 16, 2022

@0xc0170 @rwalton-arm is this able to be looked at? It it doesn't seem like merging it is in the cards, I might just make it a standalone library.

Yes, please, it will be better as standalone library and be referenced from the docs.
It is cool feature and it will be helpful for many users.

@0xc0170
Copy link
Contributor

0xc0170 commented Feb 16, 2022

I'll close this as result.

@0xc0170 0xc0170 closed this Feb 16, 2022
@multiplemonomials multiplemonomials deleted the cmake-upload-debug branch May 17, 2023 06:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants