Moved target definitions to JSON format

[bogdanm]

(long commit message ahead. Sorry about that, it can't be helped.)

This commit changs targets definition from Python to JSON format, as
part of the configuration mechanism implementation. There is a new file
under workspace_tools/ called "targets.json" which contains the target
definitions. "targets.py" remains, but becomes a wrapper on top of
"targets.json", with the same interface as before. This has the
advantage of not requiring code changes outside "targets.py".

Most of the JSON definitions of targets were automatically generated by a
script (available upon request since it doesn't make a lot of sense to
include it here), only those targets that had more than one parent in
the Python implementation were converted by hand. The target definitions
should be pretty self-explanatory. A number of things are different in
the JSON implementation (this is just a summary, check docs/mbed_targets.md
(also part of this PR) for a more complete description):

• "program_cycle_s" is now a value (as opposed to a function in the
  Python implementation), since it only returned a number in all the
  Python target implementations. The main definition that actually contains
  some code (in class "Target") remains in target.py
• array values in "macros" and "extra_labels" can be modified
  dynamically. Values can be added using "macros_add" and
  "extra_labels_add" or removed using "macros_remove" and
  "extra_labels_remove". This mechanism is available for all attributes
  with a list type, but it's currently enabled only for "macros" and
  "extra_labels" to keep things simple.
• "init_hooks"/"binary_hook" are now implemented in terms of a single
  JSON key valled "post_binary_hook". The corresponding code is also in
  "targets.py", under the various TargetCode classes (see for example
  LPC4088Code in targets.py).

Just like in the Python implementation, a target can inherit from zero,
one or more targets. The resolution order for the target's attributes
follows the one used by the Python code (I used
http://makina-corpus.com/blog/metier/2014/python-tutorial-understanding-python-mro-class-search-path as a reference for the implementation of resolution order).

This is obviously a very dangerous commit, since it affects all targets.
I tested compilation for a number of targets (K64F, LPC1768, NRF51822)
but there's definitely a lot more to be done in terms of testing.

I also tried to test in a different way: I wrote a script that imports the
old (Python) and the new (JSON) implementations and verifies that the
attributes in the old implementations exist and have the same values
in the new implementations (it also verifies that the attribute
resolution order is the same in the two implementations). If you're
interested, the script is here:

https://gist.github.com/bogdanm/c9d8cf34214109a4b9079befed6b3c0c

And the results of running the script are below (note that the script
outputs only the target names that were found to be problematic):

NRF51_MICROBIT_BOOT:
    Resolution order is different in old and new
        old: ['NRF51_MICROBIT_BOOT', 'MCU_NRF51_16K_BOOT_S110', 'MCU_NRF51_16K_BOOT_BASE', 'MCU_NRF51_16K_BASE', 'MCU_NRF51', 'Target', 'MCU_NRF51_S110']
        new: ['NRF51_MICROBIT_BOOT', 'MCU_NRF51_16K_BOOT_S110', 'MCU_NRF51_S110', 'MCU_NRF51_16K_BOOT_BASE', 'MCU_NRF51_16K_BASE', 'MCU_NRF51', 'Target']
    'extra_labels' has different values in old and new
        old: ['NORDIC', 'MCU_NRF51', 'MCU_NRF51822', 'MCU_NORDIC_16K', 'MCU_NRF51_16K', 'MCU_NRF51_16K_BOOT', 'MCU_NRF51_16K_S110', 'NRF51_MICROBIT']
        new: ['NORDIC', 'MCU_NRF51', 'MCU_NRF51822', 'MCU_NORDIC_16K', 'MCU_NRF51_16K', 'MCU_NRF51_16K_S110', 'MCU_NRF51_16K_BOOT', 'NRF51_MICROBIT']
    'macros' has different values in old and new
        old: ['NRF51', 'TARGET_NRF51822', 'TARGET_MCU_NORDIC_16K', 'TARGET_MCU_NRF51_16K', 'TARGET_MCU_NRF51_16K_BOOT', 'TARGET_OTA_ENABLED', 'TARGET_MCU_NRF51_16K_S110', 'TARGET_NRF51_MICROBIT', 'TARGET_NRF_LFCLK_RC']
        new: ['NRF51', 'TARGET_NRF51822', 'TARGET_MCU_NORDIC_16K', 'TARGET_MCU_NRF51_16K', 'TARGET_MCU_NRF51_16K_S110', 'TARGET_MCU_NRF51_16K_BOOT', 'TARGET_OTA_ENABLED', 'TARGET_NRF51_MICROBIT', 'TARGET_NRF_LFCLK_RC']
NRF51_MICROBIT:
    Resolution order is different in old and new
        old: ['NRF51_MICROBIT', 'MCU_NRF51_16K_S110', 'MCU_NRF51_16K_BASE', 'MCU_NRF51', 'Target', 'MCU_NRF51_S110']
        new: ['NRF51_MICROBIT', 'MCU_NRF51_16K_S110', 'MCU_NRF51_S110', 'MCU_NRF51_16K_BASE', 'MCU_NRF51', 'Target']
    'extra_labels' has different values in old and new
        old: ['NORDIC', 'MCU_NRF51', 'MCU_NRF51822', 'MCU_NORDIC_16K', 'MCU_NRF51_16K', 'MCU_NRF51_16K_S110']
        new: ['NORDIC', 'MCU_NRF51', 'MCU_NRF51822', 'MCU_NRF51_16K_S110', 'MCU_NORDIC_16K', 'MCU_NRF51_16K']
    'macros' has different values in old and new
        old: ['NRF51', 'TARGET_NRF51822', 'TARGET_MCU_NORDIC_16K', 'TARGET_MCU_NRF51_16K', 'TARGET_MCU_NRF51_16K_S110', 'TARGET_NRF_LFCLK_RC']
        new: ['NRF51', 'TARGET_NRF51822', 'TARGET_MCU_NRF51_16K_S110', 'TARGET_MCU_NORDIC_16K', 'TARGET_MCU_NRF51_16K', 'TARGET_NRF_LFCLK_RC']
NRF51_MICROBIT_OTA:
    Resolution order is different in old and new
        old: ['NRF51_MICROBIT_OTA', 'MCU_NRF51_16K_OTA_S110', 'MCU_NRF51_16K_OTA_BASE', 'MCU_NRF51_16K_BASE', 'MCU_NRF51', 'Target', 'MCU_NRF51_S110']
        new: ['NRF51_MICROBIT_OTA', 'MCU_NRF51_16K_OTA_S110', 'MCU_NRF51_S110', 'MCU_NRF51_16K_OTA_BASE', 'MCU_NRF51_16K_BASE', 'MCU_NRF51', 'Target']
    'extra_labels' has different values in old and new
        old: ['NORDIC', 'MCU_NRF51', 'MCU_NRF51822', 'MCU_NORDIC_16K', 'MCU_NRF51_16K', 'MCU_NRF51_16K_OTA', 'MCU_NRF51_16K_S110', 'NRF51_MICROBIT']
        new: ['NORDIC', 'MCU_NRF51', 'MCU_NRF51822', 'MCU_NORDIC_16K', 'MCU_NRF51_16K', 'MCU_NRF51_16K_S110', 'MCU_NRF51_16K_OTA', 'NRF51_MICROBIT']
    'macros' has different values in old and new
        old: ['NRF51', 'TARGET_NRF51822', 'TARGET_MCU_NORDIC_16K', 'TARGET_MCU_NRF51_16K', 'TARGET_MCU_NRF51_16K_OTA', 'TARGET_OTA_ENABLED', 'TARGET_MCU_NRF51_16K_S110', 'TARGET_NRF51_MICROBIT', 'TARGET_NRF_LFCLK_RC']
        new: ['NRF51', 'TARGET_NRF51822', 'TARGET_MCU_NORDIC_16K', 'TARGET_MCU_NRF51_16K', 'TARGET_MCU_NRF51_16K_S110', 'TARGET_MCU_NRF51_16K_OTA', 'TARGET_OTA_ENABLED', 'TARGET_NRF51_MICROBIT', 'TARGET_NRF_LFCLK_RC']
NOT OK: ['NRF51_MICROBIT', 'NRF51_MICROBIT_BOOT', 'NRF51_MICROBIT_OTA']

The reasons for the above output are subtle and related to the
extremely weird way in which we defined target data in the Python
implementation: we used both class attributes and instance attributes.
This can complicate resolution order quite a bit and those two levels
don't exist in JSON: there's only one attribute type (equivalent to
Python's instance attributes). To make that work, I had to change the
inheritance order of the above targets (that use multiple inheritance)
which in turn changed the order of some macros and extra_labels (and of
course the resolution order). No harm done: the values are the same,
only their ordering is different. I don't believe this causes any
problems for 'extra_labels' and 'macros'.

This method of testing has its limitations though; in particular, it
can't test the hooks. I'm opened to ideas about how to test this better,
but I think that we need to remember that this commit might break some
targets and keep an eye out for "weird errors" in the future.

[bogdanm]

@0xc0170 @screamerbg

[screamerbg]

@bogdanm I like the backward compatibility aspect with targets.py! Any chance to restore some of the target/silicon related comments and section separators? (not a blocker)

Going to check it with the online build system tomorrow.

[bogdanm]

Any chance to restore some of the target/silicon related comments and section separators? (not a blocker)

Sadly, that'd be quite difficult, since JSON doesn't support comments 😢 You could concievably add some form of comments to the target.json file and then pre-process it somehow, but that'd be quite weird.

[bogdanm]

Oh wow, the CI checks actually passed. I must confess that I'm pleasantly surprised by this :)

[sg-]

@mbed-bot: TEST

HOST_OSES=ALL
BUILD_TOOLCHAINS=ALL
TARGETS=ALL

[mbed-bot]

[Build 384]
FAILURE: Something went wrong when building and testing.

[bogdanm]

Does the test build need to be restarted?

[bridadan]

Test builds and test runs actually looks good, the failure looks to be caused by existing issues in the code base.

[bogdanm]

Thanks @bridadan, that's very good news!

[bogdanm]

So is this ready to merge then?

[0xc0170]

this used to be ARM/uARM only. Will possible values be other toolchains?

[0xc0170]

Looks good to me. Just had that question about default toolchain.

[bogdanm]

Good catch, thank you. Fixed. Anything else?

[0xc0170]

LGTM, shall we merge?

[screamerbg]

Go for it!

[0xc0170]

Lets merge this after the release this week (it's almost ready) and start using the new json target definitions.