Merge pull request #1893 from screamerbg/tools-improvements

mbed Tools features and improvements

Author: sg-

.gitignore

@@ -7,7 +7,7 @@ dist
 MANIFEST
 
 # Private settings
-private_settings.py
+mbed_settings.py
 
 # Default Build Directory
 .build/

.travis.yml

@@ -1,7 +1,7 @@
 python:
   - "2.7"
 
-script: "python workspace_tools/build_travis.py"
+script: "python tools/build_travis.py"
 before_install:
   - sudo add-apt-repository -y ppa:terry.guo/gcc-arm-embedded
   - sudo apt-get update -qq

MANIFEST.in

@@ -1,3 +1,3 @@
-graft workspace_tools
-recursive-exclude workspace_tools *.pyc
+graft tools
+recursive-exclude tools *.pyc
 include LICENSE

README.md

@@ -157,7 +157,7 @@ Develop
 ```
 > "venv/Scripts/activate"
 > pip install -r requirements.txt
-> cd workspace_tools
+> cd tools
 > ... do things ...
 > "venv/Scripts/deactivate"
 ```

docs/BUILDING.md

@@ -130,11 +130,11 @@ Checking out files: 100% (3994/3994), done.
 ```
 $ cd mbed
 $ ls
-LICENSE  MANIFEST.in  README.md  libraries  setup.py  travis  workspace_tools
+LICENSE  MANIFEST.in  README.md  libraries  setup.py  travis  tools
 ```
 Directory structure we are interested in:
 ```
-  mbed/workspace_tools/           - test suite scripts, build scripts etc.
+  mbed/tools/           - test suite scripts, build scripts etc.
   mbed/libraries/tests/           - mbed SDK tests,
   mbed/libraries/tests/mbed/      - tests for mbed SDK and peripherals tests,
   mbed/libraries/tests/net/echo/  - tests for Ethernet interface,
@@ -153,9 +153,9 @@ Workspace tools are set of Python scripts used off-line by Mbed SDK team to:
 Before we can run our first test we need to configure our test environment a little!
 Now we need to tell workspace tools where our compilers are.
 
-* Please to go ```mbed/workspace_tools/``` directory and create empty file called ```private_settings.py```.
+* Please to go ```mbed``` directory and create empty file called ```mbed_settings.py```.
 ```
-$ touch private_settings.py
+$ touch mbed_settings.py
 ```
 * Populate this file the Python code below: 
 ```python
@@ -203,13 +203,13 @@ GCC_CR_PATH = "C:/Work/toolchains/LPCXpresso_6.1.4_194/lpcxpresso/tools/bin"
 IAR_PATH = "C:/Work/toolchains/iar_6_5/arm"
 ```
 
-Note: Settings in ```private_settings.py``` will overwrite variables with default values in ```mbed/workspace_tools/settings.py``` file.
+Note: Settings in ```mbed_settings.py``` will overwrite variables with default values in ```mbed/default_settings.py``` file.
 
 ## Build Mbed SDK library from sources
-Let's build mbed SDK library off-line from sources using your compiler. We've already cloned mbed SDK sources, we've also installed compilers and added their paths to ```private_settings.py```.
+Let's build mbed SDK library off-line from sources using your compiler. We've already cloned mbed SDK sources, we've also installed compilers and added their paths to ```mbed_settings.py```.
 We now should be ready to use workspace tools script ```build.py``` to compile and build mbed SDK from sources.
 
-We are still using console. You should be already in ```mbed/workspace_tools/``` directory if not go to ```mbed/workspace_tools/``` and type below command:
+We are still using console. You should be already in ```mbed/tools/``` directory if not go to ```mbed/tools/``` and type below command:
 ```
 $ python build.py -m LPC1768 -t ARM
 ```
@@ -276,7 +276,7 @@ Build successes:
 
 ### build.py script
 
-Build script located in mbed/workspace_tools/ is our core script solution to drive compilation, linking and building process for:
+Build script located in mbed/tools/ is our core script solution to drive compilation, linking and building process for:
 
 * mbed SDK (with libs like Ethernet, RTOS, USB, USB host).
 * Tests which also can be linked with libraries like RTOS or Ethernet.
@@ -426,7 +426,7 @@ $ python build.py -t uARM -m NUCLEO_F334R8 --cppcheck
 ```
 
 # make.py script
-```make.py``` is a ```mbed/workspace_tools/``` script used to build tests (we call them sometimes 'programs') one by one manually. This script allows you to flash board, execute and test it. However, this script is deprecated and will not be described here. Instead please use ```singletest.py``` file to build mbed SDK, tests and run automation for test cases included in ```mbedmicro/mbed```.
+```make.py``` is a ```mbed/tools/``` script used to build tests (we call them sometimes 'programs') one by one manually. This script allows you to flash board, execute and test it. However, this script is deprecated and will not be described here. Instead please use ```singletest.py``` file to build mbed SDK, tests and run automation for test cases included in ```mbedmicro/mbed```.
 Note: ```make.py``` script depends on existing already built mbed SDK and library sources so you need to pre-build mbed SDK and other libraries (such as RTOS library) to link 'program' (test) with mbed SDK and RTOS library. To pre-build mbed SDK please use ```build.py``` script.
 
 Just for sake of example please see few ways to use ```make.py``` together with Freedom K64F board.

docs/COMMITTERS.md

@@ -198,7 +198,7 @@ $ astyle.exe --style=kr --indent=spaces=4 --indent-switches $(FULL_CURRENT_PATH)
 ```
 
 ## Python coding rules & coding guidelines
-Some of our tools in workspace_tools are written in ```Python 2.7```. In case of developing tools for python we prefer to keep similar code styles across all Python source code. Please note that not all rules must be enforced. For example we do not limit you to 80 characters per line, just be sure your code can fit to widescreen display.
+Some of our tools in tools are written in ```Python 2.7```. In case of developing tools for python we prefer to keep similar code styles across all Python source code. Please note that not all rules must be enforced. For example we do not limit you to 80 characters per line, just be sure your code can fit to widescreen display.
 
 Please stay compatible with ```Python 2.7``` but nothing stops you to write your code so in the future it will by Python 3 friendly.
 
@@ -211,7 +211,7 @@ Some general guidelines:
 * Please document your code, write comments and ```doc``` sections for each function or class you implement.
 
 ### Static Code Analizers for Python
-If you are old-school developer for sure you remember tools like lint. "lint was the name originally given to a particular program that flagged some suspicious and non-portable constructs (likely to be bugs) in C language source code." Now lint-like programs are used to check similar code issues for multiple languages, also for Python. Please do use them if you want to commit new code to workspace_tools and other mbed SDK Python tooling.
+If you are old-school developer for sure you remember tools like lint. "lint was the name originally given to a particular program that flagged some suspicious and non-portable constructs (likely to be bugs) in C language source code." Now lint-like programs are used to check similar code issues for multiple languages, also for Python. Please do use them if you want to commit new code to tools and other mbed SDK Python tooling.
 
 Below is the list Python lint tools you may want to use:
 
@@ -254,7 +254,7 @@ class HostRegistry:
 ```
 
 ## Testing
-Please refer to TESTING.md document for detais regarding mbed SDK test suite and build scripts included in ```mbed/workspace_tools/```.
+Please refer to TESTING.md document for detais regarding mbed SDK test suite and build scripts included in ```mbed/tools/```.
 
 ## Before pull request checklist
 * Your pull request description section contains:

docs/TESTING.md

@@ -6,15 +6,15 @@ Test suit allows users to run locally on their machines Mbed SDK’s tests inclu
 Each test is supervised by python script called “host test” which will at least Test suite is using build script API to compile and build test source together with required by test libraries like CMSIS, Mbed, Ethernet, USB etc.
 
 ## What is host test?
-Test suite supports test supervisor concept. This concept is realized by separate Python script called ```host test```. Host tests can be found in ```mbed/workspace_tools/host_tests/``` directory. Note: In newer mbed versions (mbed OS) host tests will be separate library.
+Test suite supports test supervisor concept. This concept is realized by separate Python script called ```host test```. Host tests can be found in ```mbed/tools/host_tests/``` directory. Note: In newer mbed versions (mbed OS) host tests will be separate library.
 
 Host test script is executed in parallel with test runner to monitor test execution. Basic host test just monitors device's default serial port for test results returned by test runner. Simple tests will print test result on serial port. In other cases host tests can for example judge by test results returned by test runner if test passed or failed. It all depends on test itself.
 
 In some cases host test can be TCP server echoing packets from test runner and judging packet loss. In other cases it can just check if values returned from accelerometer are actually valid (sane).
 
 ## Test suite core: singletest.py script
 
-```singletest.py``` script located in ```mbed/workspace_tools/``` is a test suite script which allows users to compile, build tests and test runners (also supports CppUTest unit test library). Script also is responsible for test execution on devices selected by configuration files.
+```singletest.py``` script located in ```mbed/tools/``` is a test suite script which allows users to compile, build tests and test runners (also supports CppUTest unit test library). Script also is responsible for test execution on devices selected by configuration files.
 
 ### Parameters of singletest.py
 
@@ -37,7 +37,7 @@ After connecting boards to our host machine (PC) we can check which serial ports
 * ```NUCLEO_F103RB``` serial port is on ```COM11``` and disk drive is ```I:```. 
 If you are working under Linux your port and disk could look like /dev/ttyACM5 and /media/usb5.
 
-This information is needed to create ```muts_all.json``` configuration file. You can create it in ```mbed/workspace_tools/``` directory:
+This information is needed to create ```muts_all.json``` configuration file. You can create it in ```mbed/tools/``` directory:
 ```
 $ touch muts_all.json
 ```
@@ -67,8 +67,8 @@ Its name will be passed to ```singletest.py``` script after ```-M``` (MUTs speci
 
 Note: We will leave field ```peripherals``` empty for the sake of this example. We will explain it later. All you need to do now is to properly fill fields ```mcu```, ```port``` and ```disk```. 
 
-Note: Please make sure files muts_all.json and test_spec.json are in workspace_tools/ directory. We will assume in this example they are.
-Where to find ```mcu``` names? You can use option ```-S``` of ```build.py``` script (in ```mbed/workspace_tools/``` directory) to check all supported off-line MCUs names.
+Note: Please make sure files muts_all.json and test_spec.json are in tools/ directory. We will assume in this example they are.
+Where to find ```mcu``` names? You can use option ```-S``` of ```build.py``` script (in ```mbed/tools/``` directory) to check all supported off-line MCUs names.
 
 Note: If you update mbed device firmware or even disconnect / reconnect mbed device you may find that serial port / disk configuration changed. You need to update configuration file accordingly or you will face connection problems and obviously tests will not run.
 
@@ -172,9 +172,9 @@ For our example purposes let's assume we only have Keil ARM compiler, so let's c
 ```
 #### Run your tests
 
-After you configure all your MUTs and compilers you are ready to run tests. Make sure your devices are connected and your configuration files reflect your current configuration (serial ports, devices). Go to workspace_tools directory in your mbed location.
+After you configure all your MUTs and compilers you are ready to run tests. Make sure your devices are connected and your configuration files reflect your current configuration (serial ports, devices). Go to tools directory in your mbed location.
 ```
-$ cd workspace_tools/
+$ cd tools/
 ```
 and execute test suite script.
 ```
@@ -244,7 +244,7 @@ In below example we would like to have all test binaries called ```firmware.bin`
 ```
 $ python singletest.py -i test_spec.json -M muts_all.json --firmware-name firmware
 ```
-* Where to find test list? Tests are defined in file ```tests.py``` in ```mbed/workspace_tools/``` directory. ```singletest.py``` uses test metadata in ```tests.py``` to resolve libraries dependencies and build tests for proper platforms and peripherals. Option ```-R``` can be used to get test names and direct path and test configuration.
+* Where to find test list? Tests are defined in file ```tests.py``` in ```mbed/tools/``` directory. ```singletest.py``` uses test metadata in ```tests.py``` to resolve libraries dependencies and build tests for proper platforms and peripherals. Option ```-R``` can be used to get test names and direct path and test configuration.
 ```
 $ python singletest.py -R
 +-------------+-----------+---------------------------------------+--------------+-------------------+----------+--------------------------------------------------------+
@@ -344,7 +344,7 @@ test_spec.json:
 ```
 Note: 
 * Please make sure device is connected before we will start running tests.
-* Please make sure files ```muts_all.json``` and ```test_spec.json``` are in ```mbed/workspace_tools/``` directory.
+* Please make sure files ```muts_all.json``` and ```test_spec.json``` are in ```mbed/tools/``` directory.
 Now you can call test suite and execute tests:
 ```
 $ python singletest.py -i test_spec.json -M muts_all.json
@@ -451,7 +451,7 @@ We want to create directory structure similar to one below:
 └───mbed
     ├───libraries
     ├───travis
-    └───workspace_tools
+    └───tools
 ```
 
 Please go to directory with your project. For example it could be c:\Projects\Project.
@@ -492,7 +492,7 @@ $ git clone https://github.com/mbedmicro/mbed.git
 $ hg clone https://mbed.org/users/rgrover1/code/cpputest/
 ```
 
-After above three steps you should have proper directory structure. All you need to do now is to configure your ```private_settings.py``` in ```mbed/workspace_tools/``` directory. Please refer to mbed SDK build script documentation for details.
+After above three steps you should have proper directory structure. All you need to do now is to configure your ```mbed_settings.py``` in ```mbed``` directory. Please refer to mbed SDK build script documentation for details.
 
 ## CppUTest with mbed port 
 To make sure you actualy have CppUTest library with mbed SDK port you can go to CppUTest ```armcc``` platform directory:
@@ -577,7 +577,7 @@ utest
 ```
 
 ## Define unit tests in mbed SDK test suite structure
-All tests defined in test suite are described in ```mbed/workspace_tools/tests.py``` file. This file stores data structure ```TESTS``` which is a list of simple structures describing each test. Below you can find example of ```TESTS``` structure which is configuring one of the unit tests.
+All tests defined in test suite are described in ```mbed/tools/tests.py``` file. This file stores data structure ```TESTS``` which is a list of simple structures describing each test. Below you can find example of ```TESTS``` structure which is configuring one of the unit tests.
 ```
 .
 .

docs/mbed_targets.md

@@ -1,6 +1,6 @@
 # Adding and configuring mbed targets
 
-mbed uses JSON as a description language for its build targets. The JSON description of mbed targets can be found in `workspace_tools/targets.json`. To better understand how a target is defined, we'll use this example (taken from `targets.json`):
+mbed uses JSON as a description language for its build targets. The JSON description of mbed targets can be found in `tools/targets.json`. To better understand how a target is defined, we'll use this example (taken from `targets.json`):
 
 ```
     "TEENSY3_1": {
@@ -173,4 +173,4 @@ This property is used to pass additional data to the project generator (used to
 ```
 
 The `target` property of `progen` specifies the target name that must be used for the exporter (if different than the mbed target name).
-For each exporter, a template for exporting can also be specified. In this example, the template used for generating a uVision project file is in a file called `uvision_microlib.uvproj.tmpl`. It is assumed that all the templates are located in `workspace_tools/export`.
+For each exporter, a template for exporting can also be specified. In this example, the template used for generating a uVision project file is in a file called `uvision_microlib.uvproj.tmpl`. It is assumed that all the templates are located in `tools/export`.

setup.py

@@ -16,17 +16,17 @@
 OWNER_NAMES = 'emilmont, bogdanm'
 OWNER_EMAILS = '[email protected], [email protected]'
 
-# If private_settings.py exists in workspace_tools, read it in a temporary file
+# If mbed_settings.py exists in tools, read it in a temporary file
 # so it can be restored later
-private_settings = join('workspace_tools', 'private_settings.py')
+mbed_settings = join('mbed_settings.py')
 backup = None
-if isfile(private_settings):
+if isfile(mbed_settings):
     backup = TemporaryFile()
-    with open(private_settings, "rb") as f:
+    with open(mbed_settings, "rb") as f:
         copyfileobj(f, backup)
 
-# Create the correct private_settings.py for the distribution
-with open(private_settings, "wt") as f:
+# Create the correct mbed_settings.py for the distribution
+with open(mbed_settings, "wt") as f:
     f.write("from mbed_settings import *\n")
 
 setup(name='mbed-tools',
@@ -42,8 +42,8 @@
       license=LICENSE,
       install_requires=["PrettyTable>=0.7.2", "PySerial>=2.7", "IntelHex>=1.3", "colorama>=0.3.3", "Jinja2>=2.7.3", "project-generator>=0.8.11,<0.9.0", "junit-xml", "requests", "pyYAML"])
 
-# Restore previous private_settings if needed
+# Restore previous mbed_settings if needed
 if backup:
     backup.seek(0)
-    with open(private_settings, "wb") as f:
+    with open(mbed_settings, "wb") as f:
         copyfileobj(backup, f)