Initial port of TD Tutorial #1061
Conversation
Original doc in this google doc : https://docs.google.com/document/d/12MgoVez96LvC8xwVO1DfjuMrPUYE-hUldr2tVRhS8fI/edit?usp=sharing @ARMmbed/mbed-docs - please use the google doc as the guideline for what the tutorial should look like. I did my best to get the markdown right, but some things like embedded images / youtube videos I couldnt quite figure out.
|
Amanda, I gave Austin the wrong path for his file. This should live with the other TD stuff in the Pelion Platform area. https://os.mbed.com/docs/mbed-os/v5.12/mbed-os-pelion/integrating-with-arm-treasure-data.html |
Make initial edits to file.
Fix branding.
docs/tutorials/td-tutorial.md
Outdated
|
|
||
| ## HTTPS library | ||
|
|
||
| To use the HTTPS library, use this program: https://github.com/blackstoneengineering/mbed-os-example-treasuredata-rest. This program turns on Mbed OS device statistics by enabling the `MBED_ALL_STATS_ENABLED` macro and then send heap/CPU/stack/system information to Treasure Data. |
There was a problem hiding this comment.
Please move this example into an Arm repository.
There was a problem hiding this comment.
No one has committed to long term maintinance of this library, so for the moment I would like to keep it in my personal repo space.
@janjongboom - what do you think?
There was a problem hiding this comment.
Please move it to an Arm account. You can maintain it just as well from there.
docs/tutorials/td-tutorial.md
Outdated
|
|
||
| ``` | ||
|
|
||
| ### Configure |
There was a problem hiding this comment.
Are the two sections within this part of configuration, or should we change or remove this heading?
There was a problem hiding this comment.
I dont understand the question, mind chatting with me about this monday? I cant seem to get the context correct in my head
There was a problem hiding this comment.
Note to self: Remove h3 configure and replace with h3 setting Treasure Data databases and tables
docs/tutorials/td-tutorial.md
Outdated
|
|
||
| For mass deployments. we recommend you use fluentd or fluentbit to aggregate and forward the data into Treasure Data. Depending on where you host your fluentd instance, you will need to follow slightly different setup instructions. (localhost on your machine with self signed certificates or at a public IP address in the cloud with Certificate Authority (CA) signed certificates). This example uses MessagePack (a binary encoded JSON) to encode the data. | ||
|
|
||
| <INSERT YOUTUBE VIDEO FOR FLUENTD HERE: COMING SOON> |
There was a problem hiding this comment.
Query: When should we expect this?
There was a problem hiding this comment.
Here : https://youtu.be/elB22i4y1yU
There was a problem hiding this comment.
it has been completed.
There was a problem hiding this comment.
still waiting for y'all to review the videos and give approval before I upload them to the armmbed channel
Fix punctuation, and fix voice.
|
Please note I have not reviewed the videos yet. |
docs/tutorials/td-tutorial.md
Outdated
| There are two ways to send data securely from Mbed OS to Treasure Data: | ||
|
|
||
| - HTTPS library - Send data directly to the Treasure Data REST API. | ||
| - fluentd using fluent logger library - Send data to a hosted fluentd instance that aggregates and forwards the data on to your treasure data account. |
There was a problem hiding this comment.
ah ha this is how this works---I think Fluentd should be init capped
There was a problem hiding this comment.
I was told by the inventor that its always lowercase, see the website : https://www.fluentd.org/ .
I defer to your expertise.
docs/tutorials/td-tutorial.md
Outdated
|
|
||
| Both libraries are secured with Arm Mbed TLS in transit and are equally secure. We recommend the HTTPS library for development and the fluentd library for production. The tradeoff between the two is size of code on chip, size of data in transit and setup complexity: | ||
|
|
||
| - Code size on chip - The HTTPS library is ~50KB of ROM space on chip, this due to the HTTP stack. Both libraries use Mbed TLS to secure the connections, which is ~7KB per connection on your stack for both libraries. |
docs/tutorials/td-tutorial.md
Outdated
| Both libraries are secured with Arm Mbed TLS in transit and are equally secure. We recommend the HTTPS library for development and the fluentd library for production. The tradeoff between the two is size of code on chip, size of data in transit and setup complexity: | ||
|
|
||
| - Code size on chip - The HTTPS library is ~50KB of ROM space on chip, this due to the HTTP stack. Both libraries use Mbed TLS to secure the connections, which is ~7KB per connection on your stack for both libraries. | ||
| - Data size in transit - The HTTPS library sends data as a ASCII JSON string. The fluend library uses MessagePack (binary encoded json) across a TLS connection. This means that on average the fluentd library will use less bandwidth to send an equivalent message. When you pay per byte transmitted from both your power budget and data plan it matters. |
There was a problem hiding this comment.
Data size in transit - The HTTPS library sends data as an ASCII JSON string. The Fluend library uses MessagePack (binary encoded JSON) across a TLS connection. This means that on average the Fluentd library uses less bandwidth to send an equivalent message. When you pay per byte transmitted from both your power budget and data plan it matters.
docs/tutorials/td-tutorial.md
Outdated
|
|
||
| - Code size on chip - The HTTPS library is ~50KB of ROM space on chip, this due to the HTTP stack. Both libraries use Mbed TLS to secure the connections, which is ~7KB per connection on your stack for both libraries. | ||
| - Data size in transit - The HTTPS library sends data as a ASCII JSON string. The fluend library uses MessagePack (binary encoded json) across a TLS connection. This means that on average the fluentd library will use less bandwidth to send an equivalent message. When you pay per byte transmitted from both your power budget and data plan it matters. | ||
| - Maintenance - Initially, it may be simpler to setup the HTTPS library on a device and have it send data directly to treasure data, but what if you want to change what the device is doing or how its data is reported? If you are using the HTTPS library you will need to issue a firmware update to every device to change how it formats its data, but if you are using a fluend server you can simply modify the fluentd config file on the server to change how data is formatted/processed. |
There was a problem hiding this comment.
Maintenance - Initially, setting up the HTTPS library on a device and having it send data directly to Treasure Data is easier, but what if you want to change what the device is doing or how its data is reported? If you are using the HTTPS library you must issue a firmware update to every device to change how it formats its data. However, if you are using a Fluend server, you can modify the Fluentd config file on the server to change how data is formatted and processed.
docs/tutorials/td-tutorial.md
Outdated
|
|
||
| ## HTTPS library | ||
|
|
||
| To use the HTTPS library, use this program: https://github.com/blackstoneengineering/mbed-os-example-treasuredata-rest. This program turns on Mbed OS device statistics by enabling the `MBED_ALL_STATS_ENABLED` macro and then send heap/CPU/stack/system information to Treasure Data. |
There was a problem hiding this comment.
'and then sending heap....'
docs/tutorials/td-tutorial.md
Outdated
|
|
||
| ### Import code | ||
|
|
||
| You can compile the program through any of our three development tools: |
There was a problem hiding this comment.
You can compile the program using any of the following development tools:
docs/tutorials/td-tutorial.md
Outdated
|
|
||
| Next, you can compile and load your code onto your board. If you are unfamiliar with how to compile and load code, please look at the Mbed OS quick start tutorial. | ||
|
|
||
| Once you have compiled your code and loaded it onto your board, open a serial terminal, and connect it to the board. View the output: |
There was a problem hiding this comment.
?? not sure what this means?
There was a problem hiding this comment.
I changed it to "After you have compiled your code ...."
docs/tutorials/td-tutorial.md
Outdated
|
|
||
| ### Verify data in Treasure Data | ||
|
|
||
| Go to the [Database list in Treasure data](https://console.treasuredata.com/app/databases), and open the `test_database` you created earlier. You can see the data from the board in the database. There is a 3- to 5-minute delay from when the data is sent to the database until the visualization system lets you see it, so please be patient, and wait for it to arrive. Be sure to refresh the page. |
There was a problem hiding this comment.
again, not sure what this comment means
There was a problem hiding this comment.
I capitalized "Treasure Data" as requested by megmiranda.
docs/tutorials/td-tutorial.md
Outdated
| Now that you have data in Treasure data, it's time to analyze and use the data. | ||
|
|
||
| 1. Go to the [Queries tab] (https://console.treasuredata.com/app/queries/editor). | ||
| 2. Select the `test_database`, and run some queries. To learn more about how to run queries, please read the [Treasure Data documentation](https://support.treasuredata.com/hc/en-us/articles/360001457427-Presto-Query-Engine-Introduction). |
There was a problem hiding this comment.
How about this article as an intro to running queries in TD . https://support.treasuredata.com/hc/en-us/articles/360007995693
There was a problem hiding this comment.
Sure, add that too. I took the article that was reccomended to me by the TD folks
docs/tutorials/td-tutorial.md
Outdated
|
|
||
| #### Select certain fields, order by time | ||
|
|
||
| This query selects only certain columns from the table and then order them by the time field in ascending value, you can also replace `asc` with `desc` to get the order reversed. |
There was a problem hiding this comment.
....and orders them.... value. You can also ....
docs/tutorials/td-tutorial.md
Outdated
| To send data to fluentd over TLS (securely): | ||
|
|
||
| 1. Run `openssl s_client -connect localhost:24228 -showcerts`. | ||
| 1. Copy the certificate to `fluentd-sslcert.h`. If you are running the fluentd server on localhost, this certificate will change every time you restart fluentd. You eed to rerun this command and recompile your embedded code every time you restart fluentd. |
There was a problem hiding this comment.
make the change
There was a problem hiding this comment.
@BlackstoneEngineering , @megmiranda is our writer from Arm Treasure Data. I told her she could make the changes in comments, and I've already made these changes directly.
There was a problem hiding this comment.
Understood. Its hard for me to tell what is being referenced as such a wide swath is highlighted in the github comment tool. I'll leave it to y'all
Make changes requested in comments.
Fix typo.
docs/tutorials/td-tutorial.md
Outdated
|
|
||
| ## HTTPS library | ||
|
|
||
| To use the HTTPS library, use this program: https://github.com/blackstoneengineering/mbed-os-example-treasuredata-rest. This program turns on Mbed OS device statistics by enabling the `MBED_ALL_STATS_ENABLED` macro and then sends heap/CPU/stack/system information to Treasure Data. |
There was a problem hiding this comment.
Please move this example to an Arm repository.
docs/tutorials/td-tutorial.md
Outdated
|
|
||
| If you experience issues, ensure you have at least 10KB of space left on your stack. You can also change the `TD_DEBUG` macro to `true` to turn on the Treasure Data debug printfs. | ||
|
|
||
| ## fluentd |
There was a problem hiding this comment.
Query: Should this be capitalized?
There was a problem hiding this comment.
should what be capitalized? The stuff in should be just as shown as they are copied directly from the config file. so TD_DEBUG should be exactly as it is.
Standardize capitalization of Fluentd.
|
This PR is failing Travis because the code snippets don't compile. |
What code snippets? I have some quotes from the command line to show what success looks like, but i dont have any direct code snippets in here (i dont think) |
|
@kegilbert Do you have time to look into this? |
Fix heading to remove awkward double heading and make sense within the context of the document.
|
@iriark01 / @AnotherButler - please make sure to include the images (per the google doc link example at the top), otherwise all changes have been made and it should be good to go, lets get it published! |
Co-Authored-By: Kevin Gilbert <[email protected]>
Co-Authored-By: Kevin Gilbert <[email protected]>
Update video formatting for correct rendering.
|
|
||
| ### Setting Treasure Data databases and tables | ||
|
|
||
| The second field in the tag of your embedded code determines the database. For example, sending data to a tag called `td.mydatabase.mytable` logs the data to the database called `mydatabase` in the table `mytable`. You can modify the example configuration file to see this. |
There was a problem hiding this comment.
This is a little free of context unless you're already a TD user, I think
There was a problem hiding this comment.
This one I can't fix on my own, since I'm not a TD user. @megmiranda can you offer the readers more context?
|
@docs team, please consider all gramatical and minor changes pre-approved, feel free to re-write sentences as you see fit. |
Original doc in this google doc : https://docs.google.com/document/d/12MgoVez96LvC8xwVO1DfjuMrPUYE-hUldr2tVRhS8fI/edit?usp=sharing
@ARMmbed/mbed-docs - please use the google doc as the guideline for what the tutorial should look like. I did my best to get the markdown right, but some things like embedded images / youtube videos I couldnt quite figure out.