-
Notifications
You must be signed in to change notification settings - Fork 51
Compare logs for verification #30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
1742dff
Campare logs for verification
35bca8a
Moved compare.log files to tests dir
f43e411
Explained examples verification in tests/README.md
fd508b2
Fixed tests\README.md markup
d357fa2
Updated README.md with review suggestion and authcrypt.log with stric…
bf80693
Remove carriage return from example logs as that prevents comparison …
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,156 @@ | ||
| # Testing examples | ||
|
|
||
| Examples are tested using tool [mbedhtrun](https://github.com/ARMmbed/htrun) and templated print log. The templated logs represents expected behaviour of the example. | ||
|
|
||
| ## How to create templated log | ||
|
|
||
| The idea is to check that repeated execution of the example produces serial output that matches expected pattern. An example produces serial output when code contains ```printf``` statements. Serial output may change for legitimate reasons like use of random data or date and time stamps. That's why the log is converted to a template. This means that either the text/lines that differ in every execution are removed or they are converted into regular expressions. See the example below: | ||
|
|
||
| ``` | ||
|
|
||
| > Using Ethernet LWIP | ||
|
|
||
| > Client IP Address is 10.2.203.139 | ||
|
|
||
| > Connecting with developer.mbed.org | ||
|
|
||
| Starting the TLS handshake... Starting the TLS handshake... | ||
|
|
||
| > TLS connection to developer.mbed.org established | ||
|
|
||
| Server certificate: Server certificate: | ||
|
|
||
| > | ||
| cert. version : 3 | ||
| > | ||
| serial number : 11:21:B8:47:9B:21:6C:B1:C6:AF:BC:5D:0 | ||
| > | ||
| issuer name : C=BE, O=GlobalSign nv-sa, CN=GlobalSi | ||
| > | ||
| subject name : C=GB, ST=Cambridgeshire, L=Cambridge, | ||
| > | ||
| issued on : 2016-03-03 12:26:08 | ||
| > | ||
| expires on : 2017-04-05 10:31:02 | ||
| > | ||
| signed using : RSA with SHA-256 | ||
| > | ||
| RSA key size : 2048 bits | ||
| > | ||
| basic constraints : CA=false | ||
| > | ||
| subject alt name : *.mbed.com, mbed.org, *.mbed.org, mbe | ||
| > | ||
| key usage : Digital Signature, Key Encipherment | ||
| > | ||
| ext key usage : TLS Web Server Authentication, TLS We | ||
|
|
||
| Certificate verification passed Certificate verification passed | ||
|
|
||
|
|
||
|
|
||
|
|
||
| > HTTPS: Received 439 chars from server | ||
|
|
||
| > HTTPS: Received 200 OK status ... [OK] | ||
|
|
||
| HTTPS: Received 'Hello world!' status ... [OK] HTTPS: Received 'Hello world!' status ... [OK] | ||
|
|
||
| HTTPS: Received message: HTTPS: Received message: | ||
|
|
||
|
|
||
|
|
||
|
|
||
| > HTTP/1.1 200 OK | ||
|
|
||
| > Server: nginx/1.7.10 | ||
|
|
||
| > Date: Thu, 01 Dec 2016 13:56:32 GMT | ||
|
|
||
| > Content-Type: text/plain | ||
|
|
||
| > Content-Length: 14 | ||
|
|
||
| > Connection: keep-alive | ||
|
|
||
| > Last-Modified: Fri, 27 Jul 2012 13:30:34 GMT | ||
|
|
||
| > Accept-Ranges: bytes | ||
|
|
||
| > Cache-Control: max-age=36000 | ||
|
|
||
| > Expires: Thu, 01 Dec 2016 23:56:32 GMT | ||
|
|
||
| > X-Upstream-L3: 172.17.0.3:80 | ||
|
|
||
| > X-Upstream-L2: developer-sjc-indigo-2-nginx | ||
|
|
||
| > Strict-Transport-Security: max-age=31536000; includeSubdomain | ||
|
|
||
|
|
||
|
|
||
|
|
||
| Hello world! Hello world! | ||
|
|
||
| ``` | ||
|
|
||
| Please observe above that all the lines that have data that changes from execution to execution (on right) have been removed enabling htrun to compare these logs. htrun matches lines from the compare log (on left) one by one. It keeps on looking for a line until it matches. Once matched it moves on to match the next line. If it finds all lines from the compare log in the target serial output stream. Then it halts and the test passes. | ||
|
|
||
| Another example with regular examples is shown below: | ||
|
|
||
| ``` | ||
|
|
||
| SHA-256 :\s*\d+ Kb/s,\s*\d+ cycles/byte | SHA-256 : 1922 Kb/s, 61 cycl | ||
|
|
||
| SHA-512 :\s*\d+ Kb/s,\s*\d+ cycles/byte | SHA-512 : 614 Kb/s, 191 cycl | ||
|
|
||
| AES-CBC-128 :\s*\d+ Kb/s,\s*\d+ cycles/byte | AES-CBC-128 : 1401 Kb/s, 83 cycl | ||
|
|
||
| AES-CBC-192 :\s*\d+ Kb/s,\s*\d+ cycles/byte | AES-CBC-192 : 1231 Kb/s, 95 cycl | ||
|
|
||
| AES-CBC-256 :\s*\d+ Kb/s,\s*\d+ cycles/byte | AES-CBC-256 : 1097 Kb/s, 106 cycl | ||
|
|
||
| AES-GCM-128 :\s*\d+ Kb/s,\s*\d+ cycles/byte | AES-GCM-128 : 429 Kb/s, 273 cycl | ||
|
|
||
| AES-GCM-192 :\s*\d+ Kb/s,\s*\d+ cycles/byte | AES-GCM-192 : 412 Kb/s, 285 cycl | ||
|
|
||
| AES-GCM-256 :\s*\d+ Kb/s,\s*\d+ cycles/byte | AES-GCM-256 : 395 Kb/s, 297 cycl | ||
|
|
||
| AES-CCM-128 :\s*\d+ Kb/s,\s*\d+ cycles/byte | AES-CCM-128 : 604 Kb/s, 194 cycl | ||
|
|
||
| AES-CCM-192 :\s*\d+ Kb/s,\s*\d+ cycles/byte | AES-CCM-192 : 539 Kb/s, 217 cycl | ||
|
|
||
| AES-CCM-256 :\s*\d+ Kb/s,\s*\d+ cycles/byte | AES-CCM-256 : 487 Kb/s, 241 cycl | ||
|
|
||
| CTR_DRBG \(NOPR\) :\s*\d+ Kb/s,\s*\d+ cycles/byte | CTR_DRBG (NOPR) : 1145 Kb/s, 102 cycl | ||
|
|
||
| CTR_DRBG \(PR\) :\s*\d+ Kb/s,\s*\d+ cycles/byte | CTR_DRBG (PR) : 821 Kb/s, 142 cycl | ||
|
|
||
| HMAC_DRBG SHA-256 \(NOPR\) :\s*\d+ Kb/s,\s*\d+ cycles/byte | HMAC_DRBG SHA-256 (NOPR) : 219 Kb/s, 537 cycl | ||
|
|
||
| HMAC_DRBG SHA-256 \(PR\) :\s*\d+ Kb/s,\s*\d+ cycles/byte | HMAC_DRBG SHA-256 (PR) : 193 Kb/s, 612 cycl | ||
|
|
||
| RSA-2048 :\s*\d+ ms/ public | RSA-2048 : 30 ms/ public | ||
|
|
||
| RSA-2048 :\s*\d+ ms/private | RSA-2048 : 1054 ms/private | ||
|
|
||
| RSA-4096 :\s*\d+ ms/ public | RSA-4096 : 101 ms/ public | ||
|
|
||
| RSA-4096 :\s*\d+ ms/private | RSA-4096 : 5790 ms/private | ||
|
|
||
| ECDHE-secp384r1 :\s*\d+ ms/handshake | ECDHE-secp384r1 : 1023 ms/handshake | ||
|
|
||
| ECDHE-secp256r1 :\s*\d+ ms/handshake | ECDHE-secp256r1 : 678 ms/handshake | ||
|
|
||
| ECDHE-Curve25519 :\s*\d+ ms/handshake | ECDHE-Curve25519 : 580 ms/handshake | ||
|
|
||
| ECDH-secp384r1 :\s*\d+ ms/handshake | ECDH-secp384r1 : 503 ms/handshake | ||
|
|
||
| ECDH-secp256r1 :\s*\d+ ms/handshake | ECDH-secp256r1 : 336 ms/handshake | ||
|
|
||
| ECDH-Curve25519 :\s*\d+ ms/handshake | ECDH-Curve25519 : 300 ms/handshake | ||
|
|
||
| ``` | ||
|
|
||
| More details about ```htrun``` are [here](https://github.com/ARMmbed/htrun#testing-mbed-os-examples). | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| plaintext message: [0-9a-fA-F]+ | ||
| ciphertext: [0-9a-fA-F]+ | ||
| decrypted: [0-9a-fA-F]+ | ||
| DONE |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| SHA-256 :\s*\d+ Kb/s,\s*\d+ cycles/byte | ||
| SHA-512 :\s*\d+ Kb/s,\s*\d+ cycles/byte | ||
| AES-CBC-128 :\s*\d+ Kb/s,\s*\d+ cycles/byte | ||
| AES-CBC-192 :\s*\d+ Kb/s,\s*\d+ cycles/byte | ||
| AES-CBC-256 :\s*\d+ Kb/s,\s*\d+ cycles/byte | ||
| AES-GCM-128 :\s*\d+ Kb/s,\s*\d+ cycles/byte | ||
| AES-GCM-192 :\s*\d+ Kb/s,\s*\d+ cycles/byte | ||
| AES-GCM-256 :\s*\d+ Kb/s,\s*\d+ cycles/byte | ||
| AES-CCM-128 :\s*\d+ Kb/s,\s*\d+ cycles/byte | ||
| AES-CCM-192 :\s*\d+ Kb/s,\s*\d+ cycles/byte | ||
| AES-CCM-256 :\s*\d+ Kb/s,\s*\d+ cycles/byte | ||
| CTR_DRBG \(NOPR\) :\s*\d+ Kb/s,\s*\d+ cycles/byte | ||
| CTR_DRBG \(PR\) :\s*\d+ Kb/s,\s*\d+ cycles/byte | ||
| HMAC_DRBG SHA-256 \(NOPR\) :\s*\d+ Kb/s,\s*\d+ cycles/byte | ||
| HMAC_DRBG SHA-256 \(PR\) :\s*\d+ Kb/s,\s*\d+ cycles/byte | ||
| RSA-2048 :\s*\d+ ms/ public | ||
| RSA-2048 :\s*\d+ ms/private | ||
| RSA-4096 :\s*\d+ ms/ public | ||
| RSA-4096 :\s*\d+ ms/private | ||
| ECDHE-secp384r1 :\s*\d+ ms/handshake | ||
| ECDHE-secp256r1 :\s*\d+ ms/handshake | ||
| ECDHE-Curve25519 :\s*\d+ ms/handshake | ||
| ECDH-secp384r1 :\s*\d+ ms/handshake | ||
| ECDH-secp256r1 :\s*\d+ ms/handshake | ||
| ECDH-Curve25519 :\s*\d+ ms/handshake | ||
| DONE |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| Method 1: 315f5bdb76d078c43b8ac0064e4a0164612b1fce77c869345bfc94c75894edd3 | ||
| Method 2: 315f5bdb76d078c43b8ac0064e4a0164612b1fce77c869345bfc94c75894edd3 | ||
| Method 3: 315f5bdb76d078c43b8ac0064e4a0164612b1fce77c869345bfc94c75894edd3 | ||
| Method 4: 315f5bdb76d078c43b8ac0064e4a0164612b1fce77c869345bfc94c75894edd3 | ||
| DONE |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| Starting the TLS handshake... | ||
| Server certificate: | ||
| Certificate verification passed | ||
| HTTPS: Received 'Hello world!' status ... [OK] | ||
| HTTPS: Received message: | ||
| Hello world! |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the rationale of the
READMEis to provide some insight regarding the.logfiles. I think that having a short example is appropriate, however, I consider that the examples of htrun usage in this README are a bit too extensive and better suited to the htrun documentation itself. I suggest removing the tls-client example and simply leaving a fragment of the benchmark log comparison below (perhaps 4-5 lines). What are your thoughts @mazimkhan?Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would leave this example as it illustrates the one useful way of making a template log that is by removing random data.
We don't need to add anymore examples as these two cover the current approaches we take.