Skip to content

Fix usage of custom targets in applications and tests #2196

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from

Conversation

bogdanm
Copy link
Contributor

@bogdanm bogdanm commented Jul 20, 2016

Commit 43e036d removed the ability to use custom targets (defined in
mbed_app.json) by restricting the type of the "target" argument to the
list of the known targets. The problem with this is that the
configuration system can dynamically add new custom targets to the list
of known targets. In order to do this, however, it needs the list of top
level source directories, so that it can look for a mbed_app.json file
in them. This commit fixes this problem by splitting argument parsing in
two steps:

  1. in step 1, only '--source' is parsed.
  2. the config object is created after this.
  3. the rest of the command line options are parsed.
  4. the config object created in step 2 is passed around to the various
    build functions (build_project, build_tests and so on) when needed.

Commit 43e036d removed the ability to use custom targets (defined in
mbed_app.json) by restricting the type of the "target" argument to the
list of the known targets. The problem with this is that the
configuration system can dynamically add new custom targets to the list
of known targets. In order to do this, however, it needs the list of top
level source directories, so that it can look for a mbed_app.json file
in them. This commit fixes this problem by splitting argument parsing in
two steps:

1. in step 1, only '--source' is parsed.
2. the config object is created after this.
3. the rest of the command line options are parsed.
4. the config object created in step 2 is passed around to the various
   build functions (build_project, build_tests and so on) when needed.
@bogdanm
Copy link
Contributor Author

bogdanm commented Jul 20, 2016

@0xc0170 @theotherjimmy

@bogdanm
Copy link
Contributor Author

bogdanm commented Jul 20, 2016

Cc @screamerbg

@AlessandroA
Copy link
Contributor

@bogdanm I can confirm that this is fixed both for apps and tests. Thanks for the fix!

@screamerbg @0xc0170 @theotherjimmy Please note that this issue is now a blocker for the public release of the uVisor example ARMmbed/mbed-os-example-uvisor, which uses a custom target.

@screamerbg
Copy link
Contributor

This creates a spaghetti dependencies:

  • make/build/test.py need to call the config system first, which actually initializes all targets
  • then verify the actual options
  • then call the build_api functionality, but pass the config otherwise it will be initialized twice
  • then again initialize the target and then initialize the toolchain

Why not simply remove the target verification from options and leave that to the toolchain object??

@bogdanm
Copy link
Contributor Author

bogdanm commented Jul 20, 2016

Because you need to know the full list of targets when you parse the arguments, so that you can display it to the user if he specifies a wrong target (or simply because he needs to know all available targets).

@screamerbg
Copy link
Contributor

@bogdanm I'm aware of the reasoning of the implementation. The question is "Why not simply remove the target verification from options and leave that to the toolchain object??"

@bogdanm
Copy link
Contributor Author

bogdanm commented Jul 20, 2016

You could probably do that, but I don't really see the advantages, especially since I believe the implementation effort is actually larger than the one presented in this PR.

@screamerbg
Copy link
Contributor

@bogdanm What implementation effort? We can simply stop validating the target argument and let the toolchain and config handle it.

@bogdanm
Copy link
Contributor Author

bogdanm commented Jul 20, 2016

The implementation effort that comes from re-implementing something that's already implemented in this PR, without real benefits from what I can tell.

@theotherjimmy
Copy link
Contributor

theotherjimmy commented Jul 25, 2016

Usage information is broken by this PR.
eg:

$ python tools/test.py -h 

optional arguments:
  -h, --help           show this help message and exit
  --source SOURCE_DIR  The source (input) directory (for sources other than
                       tests). Defaults to current directory

was:

$ python tools/test.py -h 
usage: test.py [-h] [-m MCU] [-t TOOLCHAIN] [--color] [-c] [-o OPTIONS]
               [-D MACROS] [-j JOBS] [--source SOURCE_DIR] [--build BUILD_DIR]
               [-l] [-p PATHS] [-f FORMAT] [--continue-on-build-fail]
               [-n NAMES] [--test-spec TEST_SPEC]
               [--build-report-junit BUILD_REPORT_JUNIT] [-v]

optional arguments:
  -h, --help            show this help message and exit
  -m MCU, --mcu MCU     build for the given MCU (ARCH_BLE, ARCH_BLE_BOOT,
                        ARCH_BLE_OTA, ARCH_GPRS, ARCH_LINK, ARCH_LINK_BOOT,
                        ARCH_LINK_OTA, ARCH_MAX, ARCH_PRO, ARM_BEETLE_SOC,
                        ARM_IOTSS_BEID, ARM_MPS2_M0, ARM_MPS2_M0P,
                        ARM_MPS2_M1, ARM_MPS2_M3, ARM_MPS2_M4, ARM_MPS2_M7,
                        B96B_F446VE, BLUEPILL_F103C8, DELTA_DFCM_NNN40,
                        DELTA_DFCM_NNN40_BOOT, DELTA_DFCM_NNN40_OTA,
                        DISCO_F051R8, DISCO_F100RB, DISCO_F303VC,
                        DISCO_F334C8, DISCO_F401VC, DISCO_F407VG,
                        DISCO_F429ZI, DISCO_F469NI, DISCO_F746NG,
                        DISCO_L053C8, DISCO_L476VG, EFM32GG_STK3700,
                        EFM32HG_STK3400, EFM32LG_STK3600, EFM32PG_STK3401,
                        EFM32WG_STK3800, EFM32ZG_STK3200, ELEKTOR_COCORICO,
                        ELMO_F411RE, HEXIWEAR, HRM1017, HRM1017_BOOT,
                        HRM1017_OTA, K20D50M, K22F, K64F, KL05Z, KL25Z, KL26Z,
                        KL27Z, KL43Z, KL46Z, LPC1114, LPC11C24, LPC11U24,
                        LPC11U24_301, LPC11U34_421, LPC11U35_401,
                        LPC11U35_501, LPC11U35_501_IBDAP, LPC11U35_Y5_MBUG,
                        LPC11U37H_401, LPC11U37_501, LPC11U68, LPC1347,
                        LPC1549, LPC1768, LPC2368, LPC2460, LPC4088,
                        LPC4088_DM, LPC4330_M0, LPC4330_M4, LPC4337, LPC810,
                        LPC812, LPC824, LPCCAPPUCCINO, MAX32600MBED,
                        MAXWSNENV, MICRONFCBOARD, MOTE_L152RC,
                        MTS_DRAGONFLY_F411RE, MTS_GAMBIT, MTS_MDOT_F405RG,
                        MTS_MDOT_F411RE, NRF51822, NRF51822_BOOT,
                        NRF51822_OTA, NRF51822_Y5_MBUG, NRF51_DK,
                        NRF51_DK_BOOT, NRF51_DK_OTA, NRF51_DONGLE,
                        NRF51_DONGLE_BOOT, NRF51_DONGLE_OTA, NRF51_MICROBIT,
                        NRF51_MICROBIT_B, NRF51_MICROBIT_BOOT,
                        NRF51_MICROBIT_B_BOOT, NRF51_MICROBIT_B_OTA,
                        NRF51_MICROBIT_OTA, NUCLEO_F030R8, NUCLEO_F031K6,
                        NUCLEO_F042K6, NUCLEO_F070RB, NUCLEO_F072RB,
                        NUCLEO_F091RC, NUCLEO_F103RB, NUCLEO_F207ZG,
                        NUCLEO_F302R8, NUCLEO_F303K8, NUCLEO_F303RE,
                        NUCLEO_F334R8, NUCLEO_F401RE, NUCLEO_F410RB,
                        NUCLEO_F411RE, NUCLEO_F429ZI, NUCLEO_F446RE,
                        NUCLEO_F446ZE, NUCLEO_F746ZG, NUCLEO_F767ZI,
                        NUCLEO_L011K4, NUCLEO_L031K6, NUCLEO_L053R8,
                        NUCLEO_L073RZ, NUCLEO_L152RE, NUCLEO_L432KC,
                        NUCLEO_L476RG, NZ32_SC151, OC_MBUINO, RBLAB_BLENANO,
                        RBLAB_BLENANO_BOOT, RBLAB_BLENANO_OTA, RBLAB_NRF51822,
                        RBLAB_NRF51822_BOOT, RBLAB_NRF51822_OTA, RZ_A1H,
                        SAMD21G18A, SAMD21J18A, SAMG55J19, SAML21J18A,
                        SAMR21G18A, SEEED_TINY_BLE, SEEED_TINY_BLE_BOOT,
                        SEEED_TINY_BLE_OTA, SSCI824, STM32F3XX, STM32F407,
                        TEENSY3_1, TY51822R3, TY51822R3_BOOT, TY51822R3_OTA,
                        UBLOX_C027, UBLOX_C029, VK_RZ_A1H, WALLBOT_BLE,
                        WALLBOT_BLE_BOOT, WALLBOT_BLE_OTA, WIZWIKI_W7500,
                        WIZWIKI_W7500ECO, WIZWIKI_W7500P, XADOW_M0,
                        XBED_LPC1768)
  -t TOOLCHAIN, --tool TOOLCHAIN
                        build using the given TOOLCHAIN (ARM, GCC_ARM, GCC_CR,
                        IAR, uARM)
  --color               print Warnings, and Errors in color
  -c, --clean           clean the build directory
  -o OPTIONS, --options OPTIONS
                        Add a build argument ("save-asm": save the asm
                        generated by the compiler, "debug-info": generate
                        debugging information, "analyze": run Goanna static
                        code analyzer")
  -D MACROS             Add a macro definition
  -j JOBS, --jobs JOBS  Number of concurrent jobs. Default: 0/auto (based on
                        host machine's number of CPUs)
  --source SOURCE_DIR   The source (input) directory (for sources other than
                        tests). Defaults to current directory.
  --build BUILD_DIR     The build (output) directory
  -l, --list            List (recursively) available tests in order and exit
  -p PATHS, --paths PATHS
                        Limit the tests to those within the specified comma
                        separated list of paths
  -f FORMAT, --format FORMAT
                        Change the format in which tests are listed. Choices
                        include: list, json. Default: list
  --continue-on-build-fail
                        Continue trying to build all tests if a build failure
                        occurs
  -n NAMES, --names NAMES
                        Limit the tests to a comma separated list of names
  --test-spec TEST_SPEC
                        Destination path for a test spec file that can be used
                        by the Greentea automated test tool
  --build-report-junit BUILD_REPORT_JUNIT
                        Destination path for a build report in the JUnit xml
                        format
  -v, --verbose         Verbose diagnostic output

'-h' was broken because it was showing help for the first (--source
only) argument parser. Fixed by creating the first argument parser
without `-h` and re-adding the `--source` argument to the second one to
get a full help text. Tested with `python tools make.py -h` and `python
tools/test.py -h`.
@bogdanm
Copy link
Contributor Author

bogdanm commented Jul 26, 2016

@theotherjimmy, thanks, I didn't notice that. Fix with the latest commit. Let me know if that looks OK to you; if so, I'll rebase against master and merge.

@jacobrosenthal
Copy link

Trying to use custom targets as documented, and I believe Im being bitten by this.

Jacobs-MacBook-Air-3:BLE_LED jacobrosenthal$ mbed compile -c -t GCC_ARM -m k64f_myapp
usage: make.py [-h] [-m MCU] [-t TOOLCHAIN] [--color] [-c] [-o OPTIONS]
               [-p PROGRAM] [-n PROGRAM] [-j JOBS] [-v] [--silent] [-D MACROS]
               [-S] [-f GENERAL_FILTER_REGEX] [--automated] [--host HOST_TEST]
               [--extra EXTRA] [--peripherals PERIPHERALS]
               [--dep DEPENDENCIES] [--source SOURCE_DIR]
               [--duration DURATION] [--build BUILD_DIR] [-N ARTIFACT_NAME]
               [-d DISK] [-s SERIAL] [-b BAUD] [-L] [--rtos] [--rpc] [--eth]
               [--usb_host] [--usb] [--dsp] [--fat] [--ublox] [--testlib]
               [-l LINKER_SCRIPT]
make.py: error: argument -m/--mcu: k64f_myapp is not a supported MCU. Supported MCUs are:
ARCH_BLE,              ARCH_BLE_BOOT,         ARCH_BLE_OTA, 
ARCH_GPRS,             ARCH_LINK,             ARCH_LINK_BOOT, 
ARCH_LINK_OTA,         ARCH_MAX,              ARCH_PRO, 
ARM_BEETLE_SOC,        ARM_IOTSS_BEID,        ARM_MPS2_M0, 
ARM_MPS2_M0P,          ARM_MPS2_M1,           ARM_MPS2_M3, 
ARM_MPS2_M4,           ARM_MPS2_M7,           B96B_F446VE, 
BLUEPILL_F103C8,       DELTA_DFCM_NNN40,      DELTA_DFCM_NNN40_BOOT, 
DELTA_DFCM_NNN40_OTA,  DISCO_F051R8,          DISCO_F100RB, 
DISCO_F303VC,          DISCO_F334C8,          DISCO_F401VC, 
DISCO_F407VG,          DISCO_F429ZI,          DISCO_F469NI, 
DISCO_F746NG,          DISCO_L053C8,          DISCO_L476VG, 
EFM32GG_STK3700,       EFM32HG_STK3400,       EFM32LG_STK3600, 
EFM32PG_STK3401,       EFM32WG_STK3800,       EFM32ZG_STK3200, 
ELEKTOR_COCORICO,      ELMO_F411RE,           HEXIWEAR, 
HRM1017,               HRM1017_BOOT,          HRM1017_OTA, 
K20D50M,               K22F,                  K64F, 
KL05Z,                 KL25Z,                 KL26Z, 
KL27Z,                 KL43Z,                 KL46Z, 
LPC1114,               LPC11C24,              LPC11U24, 
LPC11U24_301,          LPC11U34_421,          LPC11U35_401, 
LPC11U35_501,          LPC11U35_501_IBDAP,    LPC11U35_Y5_MBUG, 
LPC11U37H_401,         LPC11U37_501,          LPC11U68, 
LPC1347,               LPC1549,               LPC1768, 
LPC2368,               LPC2460,               LPC4088, 
LPC4088_DM,            LPC4330_M0,            LPC4330_M4, 
LPC4337,               LPC810,                LPC812, 
LPC824,                LPCCAPPUCCINO,         MAX32600MBED, 
MAXWSNENV,             MICRONFCBOARD,         MOTE_L152RC, 
MTS_DRAGONFLY_F411RE,  MTS_GAMBIT,            MTS_MDOT_F405RG, 
MTS_MDOT_F411RE,       NRF51822,              NRF51822_BOOT, 
NRF51822_OTA,          NRF51822_Y5_MBUG,      NRF51_DK, 
NRF51_DK_BOOT,         NRF51_DK_LEGACY,       NRF51_DK_OTA, 
NRF51_DONGLE,          NRF51_DONGLE_BOOT,     NRF51_DONGLE_OTA, 
NRF51_MICROBIT,        NRF51_MICROBIT_B,      NRF51_MICROBIT_BOOT, 
NRF51_MICROBIT_B_BOOT, NRF51_MICROBIT_B_OTA,  NRF51_MICROBIT_OTA, 
NRF52_DK,              NUCLEO_F030R8,         NUCLEO_F031K6, 
NUCLEO_F042K6,         NUCLEO_F070RB,         NUCLEO_F072RB, 
NUCLEO_F091RC,         NUCLEO_F103RB,         NUCLEO_F207ZG, 
NUCLEO_F302R8,         NUCLEO_F303K8,         NUCLEO_F303RE, 
NUCLEO_F334R8,         NUCLEO_F401RE,         NUCLEO_F410RB, 
NUCLEO_F411RE,         NUCLEO_F429ZI,         NUCLEO_F446RE, 
NUCLEO_F446ZE,         NUCLEO_F746ZG,         NUCLEO_F767ZI, 
NUCLEO_L011K4,         NUCLEO_L031K6,         NUCLEO_L053R8, 
NUCLEO_L073RZ,         NUCLEO_L152RE,         NUCLEO_L432KC, 
NUCLEO_L476RG,         NUMAKER_PFM_NUC472,    NZ32_SC151, 
OC_MBUINO,             RBLAB_BLENANO,         RBLAB_BLENANO_BOOT, 
RBLAB_BLENANO_OTA,     RBLAB_NRF51822,        RBLAB_NRF51822_BOOT, 
RBLAB_NRF51822_OTA,    RZ_A1H,                SAMD21G18A, 
SAMD21J18A,            SAMG55J19,             SAML21J18A, 
SAMR21G18A,            SEEED_TINY_BLE,        SEEED_TINY_BLE_BOOT, 
SEEED_TINY_BLE_OTA,    SSCI824,               STM32F3XX, 
STM32F407,             TEENSY3_1,             TY51822R3, 
TY51822R3_BOOT,        TY51822R3_OTA,         UBLOX_C027, 
UBLOX_C029,            VK_RZ_A1H,             WALLBOT_BLE, 
WALLBOT_BLE_BOOT,      WALLBOT_BLE_OTA,       WIZWIKI_W7500, 
WIZWIKI_W7500ECO,      WIZWIKI_W7500P,        XADOW_M0, 
XBED_LPC1768           
[mbed] ERROR: "python" returned error code 2.
[mbed] ERROR: Command "python -u /Users/jacobrosenthal/mbed-os-example-ble/BLE_LED/mbed-os/tools/make.py -t GCC_ARM -m k64f_myapp --source . --build ./.build/k64f_myapp/GCC_ARM -c" in "/Users/jacobrosenthal/mbed-os-example-ble/BLE_LED"
---

@bogdanm
Copy link
Contributor Author

bogdanm commented Aug 18, 2016

@theotherjimmy: please let me know if the current solution looks OK, so that I can rebase and merge it.

@theotherjimmy
Copy link
Contributor

I'm still opposed to making two argument parsers for each entry point, and maintaining two copies of the --source argument.

@sg-
Copy link
Contributor

sg- commented Sep 8, 2016

I don't think custom targets is something we should to support until the inheritance and structure of targets is better defined.

@bogdanm
Copy link
Contributor Author

bogdanm commented Sep 9, 2016

Agreed, they seem to be causing quit a bit of issues. Maybe we should deprecate them?

@sg-
Copy link
Contributor

sg- commented Sep 9, 2016

Maybe we should deprecate them?

Please do and we can come back to this later

@sg- sg- closed this Sep 9, 2016
@bogdanm bogdanm deleted the fix_custom_targets branch October 18, 2016 13:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants