From 739e08a68d1d690d3c72d4da2f88ea900e564ce8 Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Mon, 14 Dec 2020 18:50:17 +0100 Subject: [PATCH 1/5] Keystore format stability test strategy Initial revision. Save-compare-load approach: the test case data contains attributes of the object under test and the expected file content. Create the object, save it, check that the file has the expected content, load the file and check that the new object has the expected attributes. Signed-off-by: Gilles Peskine --- .../testing/psa-storage-format-testing.md | 92 +++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 docs/architecture/testing/psa-storage-format-testing.md diff --git a/docs/architecture/testing/psa-storage-format-testing.md b/docs/architecture/testing/psa-storage-format-testing.md new file mode 100644 index 000000000..05653ffe7 --- /dev/null +++ b/docs/architecture/testing/psa-storage-format-testing.md @@ -0,0 +1,92 @@ +# Mbed TLS PSA keystore format stability testing strategy + +## Introduction + +The PSA crypto subsystem includes a persistent key store. It is possible to create a persistent key and read it back later. This must work even if Mbed TLS has been upgraded in the meantime (except for deliberate breaks in the backward compatibility of the storage). + +The goal of this document is to define a test strategy for the key store that not only validates that it's possible to load a key that was saved with the version of Mbed TLS under test, but also that it's possible to load a key that was saved with previous versions of Mbed TLS. + +No interoperability is sought. Downgrading is not required to work. + +## General approach + +### Limitations of a direct approach + +The goal of storage format stability testing is: as a user of Mbed TLS, I want to store a key under version V and read it back under version W, with W ≥ V. + +Doing the testing this way would be difficult because we'd need to have version V of Mbed TLS available when testing version W. + +An alternative, semi-direct approach consists of generating test data under version V, and reading it back under version W. Done naively, this would require keeping a large amount of test data (full test coverage multiplied by the number of versions that we want to preserve backward compatibility with). + +### Save-and-compare approach + +Importing and saving a key is deterministic. Therefore we can ensure the stability of the storage format by creating test cases under a version V of Mbed TLS, where the test case parameters include both the parameters to pass to key creation and the expected state of the storage after the key is created. The test case creates a key as indicated by the parameters, then compares the actual state of the storage with the expected state. In addition, the test case also loads the key and checks that it has the expected data and metadata. + +If the test passes with version V, this means that the test data is consistent with what the implementation does. When the test later runs under version W ≥ V, it creates and reads back a storage state which is known to be identical to the state that V would have produced. Thus, this approach validates that W can read storage states created by V. + +Use a similar approach for files other than keys where possible and relevant. + +### Keeping up with storage format evolution + +Test cases should normally not be removed from the code base: if something has worked before, it should keep working in future versions, so we should keep testing it. + +If the way certain keys are stored changes, and we don't deliberately decide to stop supporting old keys (which should only be done by retiring a version of the storage format), then we should keep the corresponding test cases in load-only mode: create a file with the expected content, load it and check the data that it contains. + +## Storage architecture overview + +The PSA subsystem provides storage on top of the PSA trusted storage interface. The state of the storage is a mapping from file identifer (a 64-bit number) to file content (a byte array). These files include: + +* [Key files](#key-storage) (files containing one key's metadata and, except for some secure element keys, key material). +* The [random generator injected seed or state file](#random-generator-state) (`PSA_CRYPTO_ITS_RANDOM_SEED_UID`). +* [Storage transaction file](#storage-transaction-resumption). +* [Driver state files](#driver-state-files). + +For a more detailed description, refer to the [Mbed Crypto storage specification](../mbed-crypto-storage-specification.md). + +In addition, Mbed TLS includes an implementation of the PSA trusted storage interface on top of C stdio. This document addresses the test strategy for [PSA ITS over file](#psa-its-over-file) in a separate section below. + +## Key storage + +### Keystore layout + +Objective: test that the key file name corresponds to the key identifier. + +Method: store keys having various identifiers and verify that a file with the expected name is created, and no other. + +### General key format + +Objective: test the format of the key file. + +Method: Write the test code based on the storage specification. Ensure that there are test cases covering all fields. + +### Enumeration of test cases for keys + +Objective: ensure that the coverage is sufficient to have assurance that all keys are stored correctly. This requires a sufficient selection of key types, sizes, policies, etc. + +In particular, the tests must validate that each `PSA_xxx` constant that is stored in a key is covered by at least once test case: + +* Usage flags: `PSA_KEY_USAGE_xxx`. +* Algorithms in policies: `PSA_ALG_xxx`. +* Key types: `PSA_KEY_TYPE_xxx`, `PSA_ECC_FAMILY_xxx`, `PSA_DH_FAMILY_xxx`. + +Method: Generate test cases automatically based on an enumeration of available constants and some knowledge of what attributes (sizes, algorithms, …) and content to use for keys of a certain type. Note that the generated test cases will be checked into the repository (generating test cases at runtime would not allow us to test the stability of the format, only that a given version is internally consistent). + +## Random generator state + +TODO + +## Driver state files + +Not yet implemented. + +TODO + +## Storage transaction resumption + +Only relevant for secure element support. Not yet fully implemented. + +TODO + +## PSA ITS over file + +TODO From cf62f10d3f521a5fde4203cdfe4283d7e8a3e1fb Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Tue, 5 Jan 2021 11:49:18 +0100 Subject: [PATCH 2/5] Clarify interoperability non-requirement Signed-off-by: Gilles Peskine --- docs/architecture/testing/psa-storage-format-testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/architecture/testing/psa-storage-format-testing.md b/docs/architecture/testing/psa-storage-format-testing.md index 05653ffe7..8fc98e7e3 100644 --- a/docs/architecture/testing/psa-storage-format-testing.md +++ b/docs/architecture/testing/psa-storage-format-testing.md @@ -6,7 +6,7 @@ The PSA crypto subsystem includes a persistent key store. It is possible to crea The goal of this document is to define a test strategy for the key store that not only validates that it's possible to load a key that was saved with the version of Mbed TLS under test, but also that it's possible to load a key that was saved with previous versions of Mbed TLS. -No interoperability is sought. Downgrading is not required to work. +Interoperability is not a goal: PSA crypto implementations are not intended to have compatible storage formats. Downgrading is not required to work. ## General approach From 528144f5232ed0c6614a1bde369df6b736e756da Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Mon, 18 Jan 2021 23:36:18 +0100 Subject: [PATCH 3/5] Clarify the methods of key storage testing Signed-off-by: Gilles Peskine --- .../architecture/testing/psa-storage-format-testing.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/architecture/testing/psa-storage-format-testing.md b/docs/architecture/testing/psa-storage-format-testing.md index 8fc98e7e3..770017414 100644 --- a/docs/architecture/testing/psa-storage-format-testing.md +++ b/docs/architecture/testing/psa-storage-format-testing.md @@ -45,19 +45,19 @@ For a more detailed description, refer to the [Mbed Crypto storage specification In addition, Mbed TLS includes an implementation of the PSA trusted storage interface on top of C stdio. This document addresses the test strategy for [PSA ITS over file](#psa-its-over-file) in a separate section below. -## Key storage +## Key storage testing ### Keystore layout Objective: test that the key file name corresponds to the key identifier. -Method: store keys having various identifiers and verify that a file with the expected name is created, and no other. +Method: Create a key with a given identifier (using `psa_import_key`) and verify that a file with the expected name is created, and no other. Repeat for different identifiers. ### General key format -Objective: test the format of the key file. +Objective: test the format of the key file: which field goes where and how big it is. -Method: Write the test code based on the storage specification. Ensure that there are test cases covering all fields. +Method: Create a key with certain metadata with `psa_import_key`. Read the file content and validate that it has the expected layout, deduced from the storage specification. Repeat with different metadata. Ensure that there are test cases covering all fields. ### Enumeration of test cases for keys @@ -69,7 +69,7 @@ In particular, the tests must validate that each `PSA_xxx` constant that is stor * Algorithms in policies: `PSA_ALG_xxx`. * Key types: `PSA_KEY_TYPE_xxx`, `PSA_ECC_FAMILY_xxx`, `PSA_DH_FAMILY_xxx`. -Method: Generate test cases automatically based on an enumeration of available constants and some knowledge of what attributes (sizes, algorithms, …) and content to use for keys of a certain type. Note that the generated test cases will be checked into the repository (generating test cases at runtime would not allow us to test the stability of the format, only that a given version is internally consistent). +Method: Each test case creates a key with `psa_import_key`, purges it from memory, then reads it back and exercises it. Generate test cases automatically based on an enumeration of available constants and some knowledge of what attributes (sizes, algorithms, …) and content to use for keys of a certain type. Note that the generated test cases will be checked into the repository (generating test cases at runtime would not allow us to test the stability of the format, only that a given version is internally consistent). ## Random generator state From 697ee190b51c48e0005079eb3392b3ec5cef63a0 Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Mon, 18 Jan 2021 23:38:21 +0100 Subject: [PATCH 4/5] Add a section about non-default lifetimes Alternative locations should be covered. We don't yet support alternative persistence levels. Signed-off-by: Gilles Peskine --- docs/architecture/testing/psa-storage-format-testing.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/architecture/testing/psa-storage-format-testing.md b/docs/architecture/testing/psa-storage-format-testing.md index 770017414..48c83ef59 100644 --- a/docs/architecture/testing/psa-storage-format-testing.md +++ b/docs/architecture/testing/psa-storage-format-testing.md @@ -71,6 +71,15 @@ In particular, the tests must validate that each `PSA_xxx` constant that is stor Method: Each test case creates a key with `psa_import_key`, purges it from memory, then reads it back and exercises it. Generate test cases automatically based on an enumeration of available constants and some knowledge of what attributes (sizes, algorithms, …) and content to use for keys of a certain type. Note that the generated test cases will be checked into the repository (generating test cases at runtime would not allow us to test the stability of the format, only that a given version is internally consistent). +### Testing with alternative lifetime values + +Objective: have test coverage for lifetimes other than the default persistent lifetime (`PSA_KEY_LIFETIME_PERSISTENT`). + +Method: + +* For alternative locations: have tests conditional on the presence of a driver for that location. +* For alternative persistence levels: TODO + ## Random generator state TODO From ff457506d3710af027f2634255a5753f6471ab54 Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Tue, 19 Jan 2021 12:51:10 +0100 Subject: [PATCH 5/5] Remind the reader of what is done about old formats Signed-off-by: Gilles Peskine --- docs/architecture/testing/psa-storage-format-testing.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/architecture/testing/psa-storage-format-testing.md b/docs/architecture/testing/psa-storage-format-testing.md index 48c83ef59..71bf96893 100644 --- a/docs/architecture/testing/psa-storage-format-testing.md +++ b/docs/architecture/testing/psa-storage-format-testing.md @@ -47,6 +47,8 @@ In addition, Mbed TLS includes an implementation of the PSA trusted storage inte ## Key storage testing +This section describes the desired test cases for keys created with the current storage format version. When the storage format changes, if backward compatibility is desired, old test data should be kept as described under [“Keeping up with storage format evolution”](#keeping-up-with-storage-format-evolution). + ### Keystore layout Objective: test that the key file name corresponds to the key identifier.