Update Mbed OS for PSA Crypto API 1.0b3 #11315

Patater · 2019-08-23T16:02:20Z

Description

Update Mbed OS for the PSA Crypto API 1.0 beta 3. This involves refactoring code that uses the PSA Crypto API, updating the PSA Crypto client/server implementation, and updating the versions of Mbed TLS and Mbed Crypto that come pre-integrated with Mbed OS.

Pull request type

[ ] Fix
[X] Refactor
[ ] Target update
[ ] Functionality change
[ ] Docs update
[ ] Test update
[ ] Breaking change

Release Notes

Background

Mbed Crypto is our implementation of the PSA Crypto APIs. We shipped an implementations of prerelease versions of the PSA Crypto API in Mbed OS 5.11, 5.12, and 5.13, with a warning in the documentation that these APIs were subject to change and that we did not intend to maintain backwards compatibility with them. The PSA Crypto API has continued to develop and change over the past few months and a number of breaking changes have been made since the pre-release version we based our Mbed Crypto implementation on.

What is being broken?

The PSA Crypto APIs in Mbed OS as shipped in Mbed OS 5.13 are breaking.

PSA Crypto API 1.0b3 API breaking changes, addressed in Mbed OS 5.14 (see #11315)

Use key attributes structures for key creation
Make generating or importing a key also allocate the key
Update key derivation functions to accept chunked inputs
Update key agreement API
Align PSA Crypto error codes with other PSA error codes
Rename functions for consistency with each other and the rest of PSA
Be consistent in use of stdint types

Expected PSA Crypto API 1.0 API breaking changes, to be addressed in Mbed OS 5.15

None

Why is it being broken?

Mbed Crypto needs to track the upstream PSA Crypto API as maintained by ATG in order to pass ATG's PSA Compliance Kit tests. The PSA Crypto API was not yet finalized in Mbed OS 5.13 and continues to evolve. We shipped an implementation in Mbed OS 5.13 to enable other teams working on Mbed OS to develop their services a top APIs that should be mostly similar to the final APIs, rather than save up all the integration pain for a later Mbed OS release when the APIs are finalized.

Analysis of impact on users

There should be no surprises to users based on our statements of PSA API instability. We've worked with Mbed TLS, Pelion Client, Storage (ITS), SPM, and Attestation teams to ensure the message of API instability was understood and to coordinate our changes to the API. There are potentially other users of the PSA Crypto API, and our documentation states the stability level of the API for these users.

Alternatives

We could continue to provide the version of the PSA Crypto API shipped with Mbed OS 5.13, but to save flash size and reduce the maintenance burden of maintaining an API we clearly communicated we'd be breaking in the next release, this was deemed not worth the cost.

Mitigation and migration path for users

Users must update to use the new version of the API.

Renaming of key pair names

Replace KEY_PAIR in names that would have used KEYPAIR in the previous API version. For example, PSA_KEY_TYPE_ECC_KEY_PAIR replaces PSA_KEY_TYPE_ECC_KEYPAIR.

Using persistent keys

Use psa_open_key() to open a persistent key. Previously, volatile keys could also be opened. With PSA Crypto API 1.0b3, keys are implicitly opened for you upon import, generation, or derivation.

psa_status_t psa_open_key(psa_key_id_t id,
                          psa_key_handle_t *handle);

Only persistent keys can be opened, so there is no need to pass the lifetime anymore.

It is no longer necessary to call psa_create_key() to make a key persistent. A key is persistent if it is created with a lifetime other than PSA_KEY_LIFETIME_VOLATILE. As part of key creation, use psa_set_key_id() to set both the key's persistent identifier and to set the lifetime to persistent and then call the key creation routine: like psa_generate_key() or psa_import_key()

Old	New
`psa_open_key()`	Only use for opening previously created persistent keys
`psa_create_key()`	`psa_set_key_id()` Keys with IDs are made persistent implicitly upon creation

Allocating keys

Key creation will implicitly allocate resources as necessary, so psa_allocate_key() has been removed from the API and is no longer needed.

Old	New
`psa_allocate_key()`	Not necessary. Delete calls to `psa_allocate_key()`.

Importing keys

Previously, you had create a policy structure and pass many function arguments to communicate the properties you wanted the imported key to have. Now, you describe them entirely within the attributes structure, passing only the attributes and data to psa_import_key().

Old	New
`psa_key_policy_init()`	`psa_key_attributes_init()`
`psa_key_policy_set_usage()`	`psa_set_key_usage_flags()`, `psa_set_key_algorithm()`
Pass key type to `psa_import_key()`	`psa_set_key_type()`
`psa_set_key_policy()`	Pass the attributes to psa_import_key()
`psa_import_key()`	`psa_import_key()`

Generating keys

Previously, you had create a policy structure and pass many function arguments to communicate the properties you wanted the imported key to have. Now, you describe them entirely within the attributes structure, passing only the attributes and data to psa_generate_key().

Old	New
`psa_key_policy_init()`	`psa_key_attributes_init()`
`psa_key_policy_set_usage()`	`psa_set_key_usage_flags()`, `psa_set_key_algorithm()`
Pass key type to `psa_import_key()`	`psa_set_key_type()`
`psa_set_key_policy()`	Pass the attributes to psa_import_key()
`psa_generate_key()`	`psa_generate_key()`

Reading key policy or information

What used to be two functions with many parameters each is now one function that returns the attributes in one structure, in the same format you'd use to create new keys.

Old	New
`psa_get_key_policy()`, `psa_get_key_information()`	`psa_get_key_attributes()`

Deriving keys

The previous "generator" class of functions has been renamed to "key_derivation". The psa_crypto_generator_t structure was previously used to derive keys. Use of the psa_key_derivation_operation_t structure replaces psa_crypto_generator_t for deriving keys.

Old	New
`psa_crypto_generator_t`	`psa_key_derivation_operation_t`
`psa_generator_abort()`	`psa_key_derivation_abort()`
`psa_get_generator_capacity()`	`psa_key_derivation_get_capacity()`
Function parameter	`psa_key_derivation_set_capacity()`
`psa_generator_read()`	`psa_key_derivation_output_bytes()`
Use of generator with `PSA_ALG_SELECT_RAW`	`psa_raw_key_agreement()`
`psa_key_derivation()`	Deriving keys now uses key derivation objects and consists of multiple parts. See the getting started guide for details.

Key agreement

Old	New
`psa_key_agreement()`	`psa_key_derivation_setup()`, `psa_key_derivation_key_agreement()`, `psa_key_derivation_output_key()`

Hashing

A few new functions have been added to help with hashing. Specifically, functions to perform one-shot computation or comparision of hashes.

Old	New
Many hash function calls	`psa_hash_compute()`
Many hash function calls	`psa_hash_compare()`

Computing or verifying a MAC

A few new functions have been added to help with working with MACs. Specifically, functions to perform one-shot computation or comparision of MACs.

Old	New
Many hash function calls	`psa_mac_compute()`
Many hash function calls	`psa_mac_verify()`

Symmetric cryptography

A few new functions have been added to help with working with symmetric ciphers. Specifically, functions to perform one-shot encryption or decryption. The types used by psa_cipher_generate_iv(), psa_cipher_set_iv(), and psa_cipher_update() have changed from unsigned char to uint8_t.

Old	New
Many hash function calls	`psa_cipher_encrypt()`
Many hash function calls	`psa_cipher_decrypt()`

Authenticated encryption

The PSA Crypto API 1.0b3 introduces multi-part authenticated encryption functions. The original one-shot AEAD functions still remain and aren't being replaced.

New functions for multipart AEAD

psa_aead_operation_init()
psa_aead_encrypt_setup()
psa_aead_decrypt_setup()
psa_aead_generate_nonce()
psa_aead_set_nonce()
psa_aead_set_lengths()
psa_aead_update_ad()
psa_aead_update()
psa_aead_finish()
psa_aead_verify()
psa_aead_abort()

Mbed Crypto entropy injection

Use of uint8_t replaces unsigned char in mbedtls_psa_inject_entropy(). The macro MBEDTLS_PSA_INJECT_ENTROPY replaces MBEDTLS_PSA_ENTROPY_INJECTION.

We'll use the new ARM-software/psa-arch-tests directly instead for PSA Crypto API 1.0b3. This commit removes the crypto compliance tests only for now, leaving attestation and storage tests.

Remove PSA_SUCCESS redefinitions in the PSA compliance test PAL.

Make crypto_struct_ipc.h (for use with PSA Crypto clients) match style with the file it is based on, crypto_struct.h (from Mbed Crypto). This helps to keep the file diff minimal so it's easy to see the meaningful (non-style) changes.

PSA_CRYPTO_INIT is defined as a service identifier. We don't need to redefine it as a secure function. There is only one function under this service identifier, so no secure function definitions are necessary to differentiate between.

Rename the reference count variable to something shorter and avoid using the mispelling "refence".

Order the IPC functions in the same order as the functions are listed in crypto.h for improved readability.

Make the service implementation use the same names as the API as declared in the crypto.h header. This improves readability and maintainability of the service.

Patater · 2019-08-23T16:17:43Z

Rebased to fix astyle issues.

Patater · 2019-08-23T16:24:13Z

Rebased to actually fix astyle issues.

ciarmcom · 2019-08-23T17:00:27Z

@Patater, thank you for your changes.
@maclobdell @ashok-rao @ARMmbed/mbed-os-crypto @ARMmbed/mbed-os-maintainers @ARMmbed/mbed-os-test @ARMmbed/mbed-os-tls please review.

0xc0170 · 2019-08-26T10:15:08Z

First CI run started

mbed-ci · 2019-08-26T11:06:12Z

Test run: FAILED

Summary: 4 of 4 test jobs failed
Build number : 1
Build artifacts

Failed test jobs:

jenkins-ci/mbed-os-ci_unittests
jenkins-ci/mbed-os-ci_build-ARM
jenkins-ci/mbed-os-ci_build-GCC_ARM
jenkins-ci/mbed-os-ci_build-IAR

0xc0170 · 2019-08-26T11:25:26Z

Build failures look related

Patater · 2019-08-27T08:02:08Z

Thanks for testing. I'll work on fixing.

Mbed Crypto also supplies include files. Ensure that our unit tests can find those headers.

dgreen-arm

Changes look good to me

mbed-ci · 2019-08-30T15:57:03Z

Test run: FAILED

Summary: 2 of 4 test jobs failed
Build number : 5
Build artifacts

Failed test jobs:

jenkins-ci/mbed-os-ci_build-GCC_ARM
jenkins-ci/mbed-os-ci_build-IAR

0xc0170 · 2019-08-30T15:57:18Z

CI aborted and restarted.

0xc0170 · 2019-08-30T18:21:19Z

Last test and we have the very last PR for 5.14.0-rc1 in 🎉

0xc0170 · 2019-08-30T18:26:32Z

Looks like tests are failing, related to this changeset. @ARMmbed/mbed-os-crypto Please review

mbed-ci · 2019-08-30T19:16:26Z

Test run: FAILED

Summary: 1 of 11 test jobs failed
Build number : 6
Build artifacts

Failed test jobs:

jenkins-ci/mbed-os-ci_greentea-test

Patater · 2019-09-02T08:41:37Z

CC @jenia81

adbridge · 2019-09-02T12:46:06Z

@Patater any status update on this ?

Patater · 2019-09-02T13:01:52Z

@Patater any status update on this ?

We've discovered that master doesn't build. ;)

Patater · 2019-09-02T13:34:07Z

@Patater any status update on this ?

We've discovered that master doesn't build. ;)

#11389

The PSA Crypto API has moved on from 1.0b2 to 1.0b3, bringing along with it some breaking changes. Update Mbed OS to use the 1.0b3 API.

The release script must be run from mbed-os root, otherwise the test partition binaries will not be created. Add a note in the tool's README to make this clear and hopefully help save someone some debugging time in the future.

If psa_attestation_inject_key() is called twice, exit with a non-fatal error status without attempting to create another attestation key. The key already exists and doesn't need to be added again.

Be robust when keys can't be opened for deletion by erasing storage and thereby all keys.

Rebuild the TF-M binaries for the LPC55S69 and MUSCA_A1 using the latest service updates. This allows the boards to use the PSA Crypto API 1.0b3.

Patater · 2019-09-02T16:18:07Z

Rebased to initialize attributes variable, which contains a pointer, domain_parameters. When psa_reset_key_attributes() calls free(attributes->domain_parameters) is called on a garbage initialized attributes, bad things like hard faults or worse can happen.

adbridge · 2019-09-02T16:22:19Z

CI started

mbed-ci · 2019-09-02T17:24:29Z

Test run: SUCCESS

Summary: 11 of 11 test jobs passed
Build number : 7
Build artifacts

0xc0170 · 2019-09-03T07:08:58Z

All good, merging now 👍

0xc0170 · 2019-09-03T07:20:04Z

The PSA Crypto APIs in Mbed OS as shipped in Mbed OS 5.13 are breaking.

Is this still true in the release notes after changes done in this PR?

Patater · 2019-09-03T13:46:54Z

The PSA Crypto APIs in Mbed OS as shipped in Mbed OS 5.13 are breaking.

Is this still true in the release notes after changes done in this PR?

Yes, the PSA Crypto APIs are different from what we shipped in Mbed OS 5.13.

Patater added 8 commits August 21, 2019 19:43

crypto: Remove old compliance tests

273c625

We'll use the new ARM-software/psa-arch-tests directly instead for PSA Crypto API 1.0b3. This commit removes the crypto compliance tests only for now, leaving attestation and storage tests.

psa: Eliminate some PSA_SUCCESS redefinition warnings

3e5fa80

Remove PSA_SUCCESS redefinitions in the PSA compliance test PAL.

attestation: Remove unused status variable

e31916f

crypto: Match styles of crypto struct headers

dda4c47

Make crypto_struct_ipc.h (for use with PSA Crypto clients) match style with the file it is based on, crypto_struct.h (from Mbed Crypto). This helps to keep the file diff minimal so it's easy to see the meaningful (non-style) changes.

crypto: Remove duplicate PSA_CRYPTO_INIT

9744767

PSA_CRYPTO_INIT is defined as a service identifier. We don't need to redefine it as a secure function. There is only one function under this service identifier, so no secure function definitions are necessary to differentiate between.

crypto: Fix "refence count" typo

dafbf27

Rename the reference count variable to something shorter and avoid using the mispelling "refence".

crypto: Make IPC declaration order follow crypto.h

b0c8350

Order the IPC functions in the same order as the functions are listed in crypto.h for improved readability.

crypto: Make service parameter names match API

0758d23

Make the service implementation use the same names as the API as declared in the crypto.h header. This improves readability and maintainability of the service.

Patater force-pushed the psa-crypto-api-1.0b3 branch from 0194997 to c9569c6 Compare August 23, 2019 16:17

Patater force-pushed the psa-crypto-api-1.0b3 branch from c9569c6 to a116293 Compare August 23, 2019 16:23

ciarmcom requested review from ashok-rao, maclobdell and a team August 23, 2019 17:00

ciarmcom added the needs: review label Aug 23, 2019

0xc0170 added do not merge release-version: 5.14.0-rc1 labels Aug 26, 2019

0xc0170 added needs: work and removed needs: review labels Aug 26, 2019

crypto: Add Mbed Crypto includes to unit tests

6dfd515

Mbed Crypto also supplies include files. Ensure that our unit tests can find those headers.

Patater force-pushed the psa-crypto-api-1.0b3 branch from a116293 to 69d7178 Compare August 28, 2019 15:59

dgreen-arm approved these changes Aug 30, 2019

View reviewed changes

0xc0170 added the BREAKING-CHANGE label Aug 30, 2019

0xc0170 requested a review from bulislaw August 30, 2019 15:14

Patater added 5 commits September 2, 2019 17:10

Use PSA Crypto API 1.0b3

3d1b836

The PSA Crypto API has moved on from 1.0b2 to 1.0b3, bringing along with it some breaking changes. Update Mbed OS to use the 1.0b3 API.

psa: Document a test partition creation gotcha

5c81d19

The release script must be run from mbed-os root, otherwise the test partition binaries will not be created. Add a note in the tool's README to make this clear and hopefully help save someone some debugging time in the future.

attestation: Don't create attestation key twice

43af68d

If psa_attestation_inject_key() is called twice, exit with a non-fatal error status without attempting to create another attestation key. The key already exists and doesn't need to be added again.

attestation: Erase storage before test

95d448b

Be robust when keys can't be opened for deletion by erasing storage and thereby all keys.

psa: Update LPC55S69 and MUSCA_A1 TF-M binaries

a848cd6

Rebuild the TF-M binaries for the LPC55S69 and MUSCA_A1 using the latest service updates. This allows the boards to use the PSA Crypto API 1.0b3.

Patater force-pushed the psa-crypto-api-1.0b3 branch from c80c32d to a848cd6 Compare September 2, 2019 16:16

0xc0170 added ready for merge BREAKING-CHANGE and removed BREAKING-CHANGE needs: CI labels Sep 3, 2019

0xc0170 merged commit e001216 into ARMmbed:master Sep 3, 2019

adbridge removed the ready for merge label Sep 3, 2019

Update Mbed OS for PSA Crypto API 1.0b3 #11315

Update Mbed OS for PSA Crypto API 1.0b3 #11315

Uh oh!

Conversation

Patater commented Aug 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Pull request type

Release Notes

Background

What is being broken?

PSA Crypto API 1.0b3 API breaking changes, addressed in Mbed OS 5.14 (see #11315)

Expected PSA Crypto API 1.0 API breaking changes, to be addressed in Mbed OS 5.15

Why is it being broken?

Analysis of impact on users

Alternatives

Mitigation and migration path for users

Renaming of key pair names

Using persistent keys

Allocating keys

Importing keys

Generating keys

Reading key policy or information

Deriving keys

Key agreement

Hashing

Computing or verifying a MAC

Symmetric cryptography

Authenticated encryption

New functions for multipart AEAD

Mbed Crypto entropy injection

Uh oh!

Patater commented Aug 23, 2019

Uh oh!

Patater commented Aug 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ciarmcom commented Aug 23, 2019

Uh oh!

0xc0170 commented Aug 26, 2019

Uh oh!

mbed-ci commented Aug 26, 2019

Test run: FAILED

Uh oh!

0xc0170 commented Aug 26, 2019

Uh oh!

Patater commented Aug 27, 2019

Uh oh!

dgreen-arm left a comment

Choose a reason for hiding this comment

Uh oh!

mbed-ci commented Aug 30, 2019

Test run: FAILED

Uh oh!

0xc0170 commented Aug 30, 2019

Uh oh!

0xc0170 commented Aug 30, 2019

Uh oh!

0xc0170 commented Aug 30, 2019

Uh oh!

mbed-ci commented Aug 30, 2019

Test run: FAILED

Uh oh!

Patater commented Sep 2, 2019

Uh oh!

adbridge commented Sep 2, 2019

Uh oh!

Patater commented Sep 2, 2019

Uh oh!

Patater commented Sep 2, 2019

Uh oh!

Patater commented Sep 2, 2019

Uh oh!

adbridge commented Sep 2, 2019

Uh oh!

mbed-ci commented Sep 2, 2019

Test run: SUCCESS

Uh oh!

0xc0170 commented Sep 3, 2019

Uh oh!

Patater commented Aug 23, 2019 •

edited

Loading

Patater commented Aug 23, 2019 •

edited

Loading