Greentea tests: user-configurable timeouts

[michalpasztamobica]

Description

Now it is enough to add:
"macros": [
"MBED_GREENTEA_TEST_XXXSOCKET_TIMEOUT_S=20"
],
to mbed_app.json, where XXX is on of {DNS, TLS, UDP, TCP}.

Pull request type

[ ] Fix
[ ] Refactor
[ ] Target update
[ ] Functionality change
[ ] Docs update
[x] Test update
[ ] Breaking change

Reviewers

@SeppoTakalo
@AriParkkila
@pekkapesu

[0xc0170]

macros , not rather test config?

[michalpasztamobica]

@0xc0170 , could you elaborate on this idea or point me to some existing test config?

[michalpasztamobica]

@0xc0170 , do you mean adding something into the "config" section of the mbed_app.json? Like so:

{
    "config": {
        "tcp_timeout" : {
            "help" : "TCP timeout",
            "value" : 100
        },

? This would make sense, indeed. I didn't think about it...

[0xc0170]

I recall some tools had some problems with "MBED_GREENTEA_TEST_XXXSOCKET_TIMEOUT_S=20" but can't recall further details at the moment.

I am not fan of having macros used this way (setting default values and expect a user to overwrite them), rather use config.

[SeppoTakalo]

Well... test cases cannot expose mbed_lib.json so they cannot easily expose configuration values.

We do configure test cases, but that is done through mbed_app.json through well known configuration values. Like this: https://github.com/ARMmbed/mbed-os/blob/master/tools/test_configs/HeapBlockDeviceAndWifiInterface.json

I'm OK with both approaches, macros or known mbed_app.json values. I want default that work 90 % of the time, and is not required to specify anywhere. And the rest is 10 % which is Cellular testing and requires specific environment anyway.

[michalpasztamobica]

I assume the configs should be used for something that changes frequently (like communication baudrate, shield board type, etc.). Macros seem to be more of a possibility to make a temporary hack (like heap size increase).

I think macros are a better choice in our case. As Seppo suggested: most of the time we want to keep the same value of the timeout. In few special runs, at the moment only for cellular modules, we want to allow some larger timeouts and we want to avoid hacking the source code to do that.
I honestly don't fully support the idea of ports and server addresses for echo/discard tests being configured via the configs - they almost never change.

@0xc0170 , would you agree with the above?
@AriParkkila @pekkapesu , do you have any strong opinion here?

[ciarmcom]

@michalpasztamobica, thank you for your changes.
@AriParkkila @SeppoTakalo @pekkapesu @ARMmbed/mbed-os-test @ARMmbed/mbed-os-ipcore @ARMmbed/mbed-os-maintainers please review.

[VeijoPesonen]

I assume tests-network-interface doesn't require anything similar

[AriParkkila]

I assume tests-network-interface doesn't require anything similar

I guess it might due to joining to cellular network may take minutes...

[michalpasztamobica]

I assume tests-network-interface doesn't require anything similar

I guess it might due to joining to cellular network may take minutes...

Ok, I will add them in a minute.

[0xc0170]

Please fix astyle error (1 line)

[cmonr]

Neat!

[cmonr]

CI started

[mbed-ci]

Test run: FAILED

Summary: 1 of 10 test jobs failed
Build number : 1
Build artifacts

Failed test jobs:

• jenkins-ci/mbed-os-ci_exporter

[0xc0170]

CI was restart, all green

[cmonr]

Interesting. Unsure why this didn't want to be applied to the mbed-os-5.11 branch. Suspect that a config param was introduced into 5.12. Retargeting this PR for 5.12.