Fleet provisioning docs (#294)

* add detailed fleet provisioning instructions, make small formate changes, and add python file to parse cert

* make changes specific to c++

* fix arguemnt tags

* format command line instructions into a more readable form

* fix file path and add comment explaining where to run command

* explicitly use python3

* explicitly use python3

Author: jmklix

samples/README.md

@@ -40,7 +40,7 @@ The terminal prompts the user for input. Type something and press enter to publi
 Since the sample is subscribed to the same topic, it will also receive the message back from the server.
 Type `quit` and press enter to end the sample.
 
-Source: `samples/mqtt/basic_pub_sub`
+Source: `samples/mqtt/basic_pub_sub/main.cpp`
 
 Your Thing's
 [Policy](https://docs.aws.amazon.com/iot/latest/developerguide/iot-policies.html)
@@ -102,82 +102,7 @@ This sample is similar to the Basic Pub-Sub, but the connection setup is more ma
 This is a starting point for using custom
 [Configurable Endpoints](https://docs.aws.amazon.com/iot/latest/developerguide/iot-custom-endpoints-configurable.html).
 
-source: `samples/mqtt/raw_pub_sub`
-
-## Fleet provisioning
-
-This sample uses the AWS IoT
-[Fleet provisioning](https://docs.aws.amazon.com/iot/latest/developerguide/provision-wo-cert.html)
-to provision devices using either a CSR or KeysAndcertificate and subsequently calls RegisterThing.
-
-On startup, the script subscribes to topics based on the request type of either CSR or Keys topics,
-publishes the request to corresponding topic and calls RegisterThing.
-
-Source: `samples/identity/fleet_provisioning`
-cd ~/aws-iot-device-sdk-cpp-v2-build/samples/identity/fleet_provisioning
-
-Run the sample like this to provision using CreateKeysAndCertificate:
-
-``` sh
-./fleet-provisioning --endpoint <endpoint> --ca_file <path to root CA>
---cert <path to the certificate> --key <path to the private key>
---template_name <template name> --template_parameters <template parameters json>
-```
-
-Run the sample like this to provision using Csr:
-
-``` sh
-./fleet-provisioning --endpoint <endpoint> --ca_file <path to root CA>
---cert <path to the certificate> --key <path to the private key>
---template_name <template name> --template_parameters <template parameters json> --csr <path to the CSR in PEM format>
-```
-
-Your Thing's
-[Policy](https://docs.aws.amazon.com/iot/latest/developerguide/iot-policies.html)
-must provide privileges for this sample to connect, subscribe, publish,
-and receive.
-
-<details>
-<summary>(see sample policy)</summary>
-<pre>
-{
-  "Version": "2012-10-17",
-  "Statement": [
-    {
-      "Effect": "Allow",
-      "Action": [
-        "iot:Publish"
-      ],
-      "Resource": [
-        "arn:aws:iot:<b>region</b>:<b>account</b>:topic/$aws/certificates/create/json",
-        "arn:aws:iot:<b>region</b>:<b>account</b>:topic/$aws/certificates/create-from-csr/json",
-        "arn:aws:iot:<b>region</b>:<b>account</b>:topic/$aws/provisioning-templates/<b>templatename</b>/provision/json"
-      ]
-    },
-    {
-      "Effect": "Allow",
-      "Action": [
-        "iot:Receive",
-        "iot:Subscribe"
-      ],
-      "Resource": [
-        "arn:aws:iot:<b>region</b>:<b>account</b>:topic/$aws/certificates/create/json/accepted",
-        "arn:aws:iot:<b>region</b>:<b>account</b>:topic/$aws/certificates/create/json/rejected",
-        "arn:aws:iot:<b>region</b>:<b>account</b>:topic/$aws/certificates/create-from-csr/json/accepted",
-        "arn:aws:iot:<b>region</b>:<b>account</b>:topic/$aws/certificates/create-from-csr/json/rejected",
-        "arn:aws:iot:<b>region</b>:<b>account</b>:topic/$aws/provisioning-templates/<b>templatename</b>/provision/json/accepted",
-        "arn:aws:iot:<b>region</b>:<b>account</b>:topic/$aws/provisioning-templates/<b>templatename</b>/provision/json/rejected"
-      ]
-    },
-    {
-      "Effect": "Allow",
-      "Action": "iot:Connect",
-      "Resource": "arn:aws:iot:<b>region</b>:<b>account</b>:client/test-*"
-    }
-  ]
-}
-</pre>
-</details>
+source: `samples/mqtt/raw_pub_sub/main.cpp`
 
 ## Shadow
 
@@ -199,7 +124,7 @@ value. When the sample learns of a new desired value, that value is changed
 on the device and an update is sent to the server with the new "reported"
 value.
 
-Source: `samples/shadow/shadow_sync`
+Source: `samples/shadow/shadow_sync/main.cpp`
 
 Your Thing's
 [Policy](https://docs.aws.amazon.com/iot/latest/developerguide/iot-policies.html)
@@ -271,7 +196,7 @@ This sample requires you to create jobs for your device to execute. See
 
 On startup, the sample describes a job that is pending execution.
 
-Source: `samples/jobs/describe_job_execution`
+Source: `samples/jobs/describe_job_execution/main.cpp`
 
 Your Thing's
 [Policy](https://docs.aws.amazon.com/iot/latest/developerguide/iot-policies.html)
@@ -331,6 +256,193 @@ and receive.
 </pre>
 </details>
 
+## Fleet provisioning
+
+This sample uses the AWS IoT
+[Fleet provisioning](https://docs.aws.amazon.com/iot/latest/developerguide/provision-wo-cert.html)
+to provision devices using either a CSR or KeysAndcertificate and subsequently calls RegisterThing.
+
+On startup, the script subscribes to topics based on the request type of either CSR or Keys topics,
+publishes the request to corresponding topic and calls RegisterThing.
+
+Source: `samples/identity/fleet_provisioning/main.cpp`
+
+Run the sample like this to provision using CreateKeysAndCertificate:
+
+``` sh
+./fleet-provisioning --endpoint <endpoint> --ca_file <path to root CA>
+--cert <path to the certificate> --key <path to the private key>
+--template_name <template name> --template_parameters <template parameters json>
+```
+
+Run the sample like this to provision using Csr:
+
+``` sh
+./fleet-provisioning --endpoint <endpoint> --ca_file <path to root CA>
+--cert <path to the certificate> --key <path to the private key>
+--template_name <template name> --template_parameters <template parameters json> --csr <path to the CSR in PEM format>
+```
+
+Your Thing's
+[Policy](https://docs.aws.amazon.com/iot/latest/developerguide/iot-policies.html)
+must provide privileges for this sample to connect, subscribe, publish,
+and receive.
+
+<details>
+<summary>(see sample policy)</summary>
+<pre>
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": [
+        "iot:Publish"
+      ],
+      "Resource": [
+        "arn:aws:iot:<b>region</b>:<b>account</b>:topic/$aws/certificates/create/json",
+        "arn:aws:iot:<b>region</b>:<b>account</b>:topic/$aws/certificates/create-from-csr/json",
+        "arn:aws:iot:<b>region</b>:<b>account</b>:topic/$aws/provisioning-templates/<b>templatename</b>/provision/json"
+      ]
+    },
+    {
+      "Effect": "Allow",
+      "Action": [
+        "iot:Receive",
+        "iot:Subscribe"
+      ],
+      "Resource": [
+        "arn:aws:iot:<b>region</b>:<b>account</b>:topic/$aws/certificates/create/json/accepted",
+        "arn:aws:iot:<b>region</b>:<b>account</b>:topic/$aws/certificates/create/json/rejected",
+        "arn:aws:iot:<b>region</b>:<b>account</b>:topic/$aws/certificates/create-from-csr/json/accepted",
+        "arn:aws:iot:<b>region</b>:<b>account</b>:topic/$aws/certificates/create-from-csr/json/rejected",
+        "arn:aws:iot:<b>region</b>:<b>account</b>:topic/$aws/provisioning-templates/<b>templatename</b>/provision/json/accepted",
+        "arn:aws:iot:<b>region</b>:<b>account</b>:topic/$aws/provisioning-templates/<b>templatename</b>/provision/json/rejected"
+      ]
+    },
+    {
+      "Effect": "Allow",
+      "Action": "iot:Connect",
+      "Resource": "arn:aws:iot:<b>region</b>:<b>account</b>:client/test-*"
+    }
+  ]
+}
+</pre>
+</details>
+
+### Fleet Provisioning Detailed Instructions
+
+#### Aws Resource Setup
+
+Fleet provisioning requires some additional AWS resources be set up first. This section documents the steps you need to take to
+get the sample up and running. These steps assume you have the AWS CLI installed and the default user/credentials has
+sufficient permission to perform all of the listed operations. You will also need python3 to be able to run parse_cert_set_result.py. These steps are based on provisioning setup steps
+that can be found at [Embedded C SDK Setup](https://docs.aws.amazon.com/freertos/latest/lib-ref/c-sdk/provisioning/provisioning_tests.html#provisioning_system_tests_setup).
+
+First, create the IAM role that will be needed by the fleet provisioning template. Replace `RoleName` with a name of the role you want to create. 
+``` sh
+aws iam create-role \
+    --role-name [RoleName] \
+    --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Action":"sts:AssumeRole","Effect":"Allow","Principal":{"Service":"iot.amazonaws.com"}}]}'
+```
+Next, attach a policy to the role created in the first step. Replace `RoleName` with the name of the role you created previously.
+``` sh
+aws iam attach-role-policy \
+        --role-name [RoleName] \
+        --policy-arn arn:aws:iam::aws:policy/service-role/AWSIoTThingsRegistration
+```
+Finally, create the template resource which will be used for provisioning by the demo application. This needs to be done only 
+once.  To create a template, the following AWS CLI command may be used. Replace `TemplateName` with the name of the fleet 
+provisioning template you want to create. Replace `RoleName` with the name of the role you created previously. Replace 
+`TemplateJSON` with the template body as a JSON string (containing escape characters). Replace `account` with your AWS 
+account number. 
+``` sh
+aws iot create-provisioning-template \
+        --template-name [TemplateName] \
+        --provisioning-role-arn arn:aws:iam::[account]:role/[RoleName] \
+        --template-body "[TemplateJSON]" \
+        --enabled 
+```
+The rest of the instructions assume you have used the following for the template body:
+``` sh
+{\"Parameters\":{\"DeviceLocation\":{\"Type\":\"String\"},\"AWS::IoT::Certificate::Id\":{\"Type\":\"String\"},\"SerialNumber\":{\"Type\":\"String\"}},\"Mappings\":{\"LocationTable\":{\"Seattle\":{\"LocationUrl\":\"https://example.aws\"}}},\"Resources\":{\"thing\":{\"Type\":\"AWS::IoT::Thing\",\"Properties\":{\"ThingName\":{\"Fn::Join\":[\"\",[\"ThingPrefix_\",{\"Ref\":\"SerialNumber\"}]]},\"AttributePayload\":{\"version\":\"v1\",\"serialNumber\":\"serialNumber\"}},\"OverrideSettings\":{\"AttributePayload\":\"MERGE\",\"ThingTypeName\":\"REPLACE\",\"ThingGroups\":\"DO_NOTHING\"}},\"certificate\":{\"Type\":\"AWS::IoT::Certificate\",\"Properties\":{\"CertificateId\":{\"Ref\":\"AWS::IoT::Certificate::Id\"},\"Status\":\"Active\"},\"OverrideSettings\":{\"Status\":\"REPLACE\"}},\"policy\":{\"Type\":\"AWS::IoT::Policy\",\"Properties\":{\"PolicyDocument\":{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Action\":[\"iot:Connect\",\"iot:Subscribe\",\"iot:Publish\",\"iot:Receive\"],\"Resource\":\"*\"}]}}}},\"DeviceConfiguration\":{\"FallbackUrl\":\"https://www.example.com/test-site\",\"LocationUrl\":{\"Fn::FindInMap\":[\"LocationTable\",{\"Ref\":\"DeviceLocation\"},\"LocationUrl\"]}}}
+```
+If you use a different body, you may need to pass in different template parameters.
+
+#### Running the sample and provisioning using a certificate-key set from a provisioning claim
+
+To run the provisioning sample, you'll need a certificate and key set with sufficient permissions. Provisioning certificates are normally 
+created ahead of time and placed on your device, but for this sample, we will just create them on the fly. You can also
+use any certificate set you've already created if it has sufficient IoT permissions and in doing so, you can skip the step
+that calls `create-provisioning-claim`.
+ 
+We've included a script in the utils folder that creates certificate and key files from the response of calling
+`create-provisioning-claim`. These dynamically sourced certificates are only valid for five minutes. When running the command, 
+you'll need to substitute the name of the template you previously created, and on Windows, replace the paths with something appropriate.
+
+(Optional) Create a temporary provisioning claim certificate set. This command is executed in the debug folder(`aws-iot-device-sdk-cpp-v2-build\samples\identity\fleet_provisioning\Debug`):
+``` sh
+aws iot create-provisioning-claim \
+        --template-name [TemplateName] \
+        | python3 ../../../../../aws-iot-device-sdk-cpp-v2/utils/parse_cert_set_result.py \
+        --path /tmp \
+        --filename provision
+```
+
+The provisioning claim's cert and key set have been written to `/tmp/provision*`. Now you can use these temporary keys
+to perform the actual provisioning. If you are not using the temporary provisioning certificate, replace the paths for `--cert` 
+and `--key` appropriately:
+
+``` sh
+./fleet-provisioning \
+        --endpoint [your endpoint]-ats.iot.[region].amazonaws.com \
+        --ca_file [pathToRootCA] \
+        --cert /tmp/provision.cert.pem \
+        --key /tmp/provision.private.key \
+        --template_name [TemplateName] \
+        --template_parameters "{\"SerialNumber\":\"1\",\"DeviceLocation\":\"Seattle\"}"
+```
+
+Notice that we provided substitution values for the two parameters in the template body, `DeviceLocation` and `SerialNumber`.
+
+#### Run the sample using the certificate signing request workflow
+
+To run the sample with this workflow, you'll need to create a certificate signing request.
+
+First create a certificate-key pair:
+``` sh
+openssl genrsa -out /tmp/deviceCert.key 2048
+```
+
+Next create a certificate signing request from it:
+``` sh
+openssl req -new -key /tmp/deviceCert.key -out /tmp/deviceCert.csr
+```
+
+(Optional) As with the previous workflow, we'll create a temporary certificate set from a provisioning claim. This step can
+be skipped if you're using a certificate set capable of provisioning the device. This command is executed in the debug folder(`aws-iot-device-sdk-cpp-v2-build\samples\identity\fleet_provisioning\Debug`):
+
+``` sh
+aws iot create-provisioning-claim \
+        --template-name [TemplateName] \
+        | python3 ../../../../../aws-iot-device-sdk-cpp-v2/utils/parse_cert_set_result.py \
+        --path /tmp \
+        --filename provision
+```
+
+Finally, supply the certificate signing request while invoking the provisioning sample. As with the previous workflow, if
+using a permanent certificate set, replace the paths specified in the `--cert` and `--key` arguments:
+``` sh
+./fleet-provisioning \
+        --endpoint [your endpoint]-ats.iot.[region].amazonaws.com \
+        --ca_file [pathToRootCA] \
+        --cert /tmp/provision.cert.pem \
+        --key /tmp/provision.private.key \
+        --template_name [TemplateName] \
+        --template_parameters "{\"SerialNumber\":\"1\",\"DeviceLocation\":\"Seattle\"}" \
+        --csr /tmp/deviceCert.csr
+```
+
 ## Secure Tunneling
 
 This sample uses the AWS IoT [Secure Tunneling](https://docs.aws.amazon.com/iot/latest/developerguide/secure-tunneling.html) Service to receive a tunnel notification.

utils/parse_cert_set_result.py

@@ -0,0 +1,69 @@
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0.
+
+"""
+
+A simply utility script to parse out certificates and keys from some IoT operations.
+
+For example, you may be doing fleet provisioning and want to have a simple way of setting up the results from create-provisioning-claim
+into usable pem files that you can make IoT connections with.
+
+Example usage:
+
+aws iot create-provisioning-claim --template-name <TemplateName> | python3 parse_cert_set_result.py --path <PathToOutputtedCerts> --filename <Filename>
+
+"""
+
+import argparse
+import json
+import os
+import re
+import sys
+
+parser = argparse.ArgumentParser(description="Utility script to generate valid .cert.pem, .private.key, .public.key files from the JSON response of CreateProvisioningClaim, CreateCertificateFromCsr")
+parser.add_argument('--path', required=True, help="Path to extract the certificate set files to.  Created if does not exist")
+parser.add_argument('--filename', required=True, help="Filename (prefix) to use for the generated files")
+
+if __name__ == '__main__':
+    # Process input args
+    args = parser.parse_args()
+
+    path = args.path
+    filename = args.filename
+
+    if not os.path.exists(path):
+        os.makedirs(path)
+
+    body = json.load(sys.stdin)
+
+    raw_pem = body['certificatePem']
+    if raw_pem:
+        pem = re.sub("\\n", "\n", raw_pem)
+        pem_filename = os.path.join(path, filename + ".cert.pem")
+        with open(pem_filename, 'w') as file:
+            file.write(pem)
+
+    try:
+        raw_pub_key = body['keyPair']['PublicKey']
+        if raw_pub_key:
+            pub_key = re.sub("\\n", "\n", raw_pub_key)
+            pub_key_filename = os.path.join(path, filename + ".public.key")
+            with open(pub_key_filename, 'w') as file:
+                file.write(pub_key)
+
+        raw_private_key = body['keyPair']['PrivateKey']
+        if raw_private_key:
+            private_key = re.sub("\\n", "\n", raw_private_key)
+            private_key_filename = os.path.join(path, filename + ".private.key")
+            with open(private_key_filename, 'w') as file:
+                file.write(private_key)
+    except KeyError:
+        pass
+
+    print("Success!")
+
+
+
+
+
+