Commit graph

12864 commits

Author SHA1 Message Date
Gilles Peskine ab4b9b4165 New test strategy document: invasive testing
Evaluate possible approaches for invasive testing.

State some rules.

This commit was originally written for Mbed Crypto only.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2020-03-24 22:56:15 +01:00
Gilles Peskine 3b310c69a5 Add a note to some invasive tests
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2020-03-24 22:55:48 +01:00
Gilles Peskine 7a054d1def
Merge pull request #3094 from gilles-peskine-arm/merge-crypto-docs
Merge documentation from Mbed Crypto
2020-03-24 14:19:34 +01:00
Gilles Peskine 37b61814cb
Merge pull request #3085 from gilles-peskine-arm/merge-crypto-unremoved-20200304
Merge of Mbed Crypto

Merge Mbed Crypto back into Mbed TLS. See https://github.com/ARMmbed/mbed-crypto/issues/374 for more details.
2020-03-24 13:56:15 +01:00
Gilles Peskine 3ed2de9f54 Refresh generated files 2020-03-23 21:54:07 +01:00
Gilles Peskine f03bd81be0 Place MBEDTLS_SSL_VARIABLE_BUFFER_LENGTH in alphabetical order
The automatic merge of include/mbedtls/config.h had placed
MBEDTLS_SSL_VARIABLE_BUFFER_LENGTH before
MBEDTLS_SSL_TRUNCATED_HMAC_COMPAT.
2020-03-23 18:13:58 +01:00
Gilles Peskine 5e7d6fd240 Merge 'mbedtls/development' into merge-crypto-unremoved-20200304
Merge the latest state of the target branch (mbedtls/development) into the
pull request to merge mbed-crypto into mbedtls.

Conflicts:

* ChangeLog: add/add conflict. Resolve by using the usual section order.
2020-03-23 18:02:07 +01:00
Gilles Peskine 5ec51f923a Future-proof make clean under Windows
In Windows cmd, `del foo` succeeds if `foo` doesn't exist, but
`del *.ext` fails if `*.ext` doesn't match any files. Therefore we use
the idiom `if exist *.ext del *.ext` to delete files matching
wildcards.

We've accidentally used `if exist $(some_list) del $(some_list)`, and
that's wrong, because it's only syntactically correct if
`$(some_list)` contains exactly one element. As long as `$(some_list)`
contains actual file names and not wildcards, just use `del $(some_list)`.
2020-03-23 17:55:32 +01:00
Gilles Peskine a647f1296e Delete third-party objects in cleanup 2020-03-23 17:55:32 +01:00
Gilles Peskine 049fefff90 Add dependencies on !MBEDTLS_SHA512_NO_SHA384 to x509/ssl unit tests
Add dependencies on !MBEDTLS_SHA512_NO_SHA384 to X.509 and SSL unit
tests that use SHA-384 (identified based on having a description that
contains "SHA384" or "SHA-384").
2020-03-23 17:55:32 +01:00
Gilles Peskine 6eb45f99f0 Add missing dependency on MBEDTLS_CIPHER_MODE_CBC 2020-03-23 17:55:32 +01:00
Gilles Peskine 19e841e19f Fix warning about a potentially initialized variable
The variable drop_len is only used under conditions where it has been
initialized, but this is somewhat fragile and not obvious for a
compiler to see.
2020-03-23 17:55:32 +01:00
Gilles Peskine eccd888717 Rename identifiers containing double-underscore
Rename identifiers containing double-underscore (`__`) to avoid `__`.
The reason to avoid double-underscore is that all identifiers
containing double-underscore are reserved in C++. Rename all such
identifiers that appear in any public header, including ssl_internal.h
which is in principle private but in practice is installed with the
public headers.

This commit makes check-names.sh pass.

```
perl -i -pe 's/\bMBEDTLS_SSL__ECP_RESTARTABLE\b/MBEDTLS_SSL_ECP_RESTARTABLE_ENABLED/g; s/\bMBEDTLS_KEY_EXCHANGE_(_\w+)_(_\w+)\b/MBEDTLS_KEY_EXCHANGE${1}${2}/g' include/mbedtls/*.h library/*.c programs/*/*.c scripts/data_files/rename-1.3-2.0.txt tests/suites/*.function
```
2020-03-23 17:55:32 +01:00
Gilles Peskine 2f084fe02c Add changelog entries about the crypto submodule
Add a changelog entry for an already-released version to indicate when
the crypto submodule became mandatory.

Add a changelog entry for the removal of the crypto submodule.
2020-03-23 17:55:32 +01:00
Gilles Peskine 7415f2fd12 Add comment to help syntax highlighting in editors
Restore this change which was accidentally reverted during the merge
of Mbed Crypto.
2020-03-23 17:55:32 +01:00
Gilles Peskine 0a968f7c7f Fix the shared library build with CMake
Fix an error in the earlier merge.
2020-03-23 17:55:32 +01:00
Gilles Peskine 55d5353b3a Remove lingering build code referencing the crypto submodule
Remove code guarded by `USE_CRYPTO_SUBMODULE`. It's dead now that
crypto can no longer be a submodule.

In `library/Makefile`:

* Replace `$(CRYPTO_INCLUDE)` with the single include directory
  `-I../include`.
* Remove references to `$(OBJS_CRYPTO)` when it's in addition to the
  local objects (`*.o`) since `$(OBJS_CRYPTO)` is now a subset of the
  local objects.
* Merge modules that were duplicated between the mbedtls and the
  mbed-crypto repositories back into the single list for `OBJS_CRYPTO`.
2020-03-23 17:55:32 +01:00
Gilles Peskine f8c2965d11 Remove the crypto submodule
All crypto files are now in the main tree.
2020-03-23 17:55:31 +01:00
Gilles Peskine aae57bffd9 Remove remaining references to the crypto subdirectory 2020-03-23 17:55:31 +01:00
Gilles Peskine 18a9d5b425 Use an indirect reference to detect the 3rdparty directory 2020-03-23 17:55:31 +01:00
Gilles Peskine 37940d923e Restore linking against all libraries in the cmake_subproject test 2020-03-23 17:55:31 +01:00
Gilles Peskine 4973196e5a Restore fuzz to the cmake build 2020-03-23 17:55:31 +01:00
Gilles Peskine d2e77b51a1 Restore the library SO version in Makefile 2020-03-23 17:55:31 +01:00
Gilles Peskine 4e616f893b Remove config-default.h
This file was the default mbedtls configuration in the mbedtls-psa
fork. It is no longer relevant.
2020-03-23 17:55:31 +01:00
Gilles Peskine b22d0cfa99 Re-generate automatically generated files
```
scripts/generate_features.pl
scripts/generate_errors.pl
scripts/generate_query_config.pl
scripts/generate_visualc_files.pl
```
2020-03-23 17:55:31 +01:00
Gilles Peskine d3d02900f2 Restore X509 and SSL functionality
Look for changes that remove X509/SSL functionality.
```
git diff 'HEAD^{/^Merge}~1' HEAD --diff-filter=M -- . ':!library/error.c' ':!library/version_features.c' ':!programs/test/query_config.c' ':!visualc' ':!*.pdf' ':!*.der' | grep -E "^-.*MBEDTLS_(ERR_)?(PKCS11|X509|NET|SSL)_"
```

All of these removals are in `config.h` or `check_config.h`. Selectively revert the differences in these two files.

```
git diff 'HEAD^{/^Merge}~1' 'HEAD^{/^Merge}' include/mbedtls/config.h include/mbedtls/check_config.h | git apply -p1 -R
```

* `include/mbedtls/check_config.h`:
    * ARIA for GCM: don't remove it.
    * `MBEDTLS_PSA_CRYPTO_SE_C`: don't remove it.
    * `MBEDTLS_SHA512_NO_SHA384`: don't remove it.
    * `MBEDTLS_SSL_DTLS_CONNECTION_ID`: restore it.
* `include/mbedtls/config.h`:
    * warning about `MBEDTLS_ECDSA_SIGN_ALT`: don't remove it.
    * `MBEDTLS_CTR_DRBG_USE_128_BIT_KEY` full paragraph: don't remove it.
    * `MBEDTLS_PSA_CRYPTO_KEY_FILE_ID_ENCODES_OWNER`: don't remove it.
    * `MBEDTLS_SHA512_SMALLER`…: don't remove it.
    * `MBEDTLS_SSL_RECORD_CHECKING`: restore it.
    * `MBEDTLS_SSL_CONTEXT_SERIALIZATION`: restore it.
    * `MBEDTLS_USE_PSA_CRYPTO` note: don't restore the tls version.
    * `MBEDTLS_USE_PSA_CRYPTO` warning: restore the tls version.
    * `MBEDTLS_CMAC_C`: restore it to being disabled by default. It's a minor API change in Mbed TLS because it changes the layout of `mbedtls_cipher_context_t`.
    * `MBEDTLS_CTR_DRBG_C`: don't restore the older version of the description from tls.
    * `MBEDTLS_GCM_C`: don't restore the older description from tls.
    * `MBEDTLS_PSA_CRYPTO_C`: don't restore `crypto/`.
    * `MBEDTLS_PSA_CRYPTO_SE_C`: don't remove it.
    * `MBEDTLS_PSA_CRYPTO_STORAGE_C`: don't restore `crypto/`. Don't disable it by default.
    * `MBEDTLS_PSA_ITS_FILE_C`: don't restore, like for ``MBEDTLS_PSA_CRYPTO_STORAGE_C``.
    * `MBEDTLS_CTR_DRBG_USE_128_BIT_KEY` single line: don't restore it since there is now a full paragraph in the proper section above.
    * `MBEDTLS_SSL_CID_IN_LEN_MAX`…: restore it.
    * `MBEDTLS_TLS_DEFAULT_ALLOW_SHA1_IN_CERTIFICATES`: restore the version without a space to format the line like the others.
2020-03-23 17:55:31 +01:00
Gilles Peskine a57a80e213 Restore full tls coverage to remaining scripts
Look for any other invocation of test scripts that was removed: look for a change that removes a line (`^-.*…`) containing one of the names of a test script (without its path because it may be used with a relative path). Look for `ssl-opt.sh` and `compat.sh`, as well any file in `tests/scripts` that only exists in tls.
```
git diff 'HEAD^{/^Merge}~1' HEAD --diff-filter=M -- . ':!library/error.c' ':!library/version_features.c' ':!programs/test/query_config.c' ':!visualc' ':!*.pdf' ':!*.der' | grep -E "^-.*($(comm -23 <(git ls-tree -r --name-only 'HEAD^{/^Merge}~1') <(git ls-tree -r --name-only $(git merge-base upstream-crypto/development 'HEAD^{/^Merge}^2')) | sed -n 's!^tests/scripts/!!p' | sed 's/\./\\./g' | tr '\n' '|')ssl-opt\.sh|compat\.sh)" | grep -v '^---'
```
This only turns up changes in `basic-in-docker.sh`.
2020-03-23 17:55:31 +01:00
Gilles Peskine 636c26ad75 Restore full tls coverage to all.sh
The merge of mbed-crypto removed some tls coverage. Restore it. Also
remove references to the `crypto` subdirectory brought by the mbedtls
side of the merge. In more detail:

* `tests/scripts/all.sh`:
    * `fuzz` in comments (×2): restore it.
    * `CTEST_OUTPUT_ON_FAILURE=1`: don't remove it.
    * `cd crypto` for `make clean`: don't restore it.
    * `cleanup`: do restore `programs/fuzz/Makefile`. Don't go into `crypto`. Keep only one copy of the calls to `rm` in `cmake_subproject`.
    * Comment legacy options: don't remove it.
    * `crypto/Makefile` and `pre_check_seedfile`: don't restore either. See below regarding the lack of need for `pre_check_seedfile`.
    * blank line in `pre_print_configuration`: restore it.
    * blank line before `#### Build and test`: restore it.
    * SSL tests in `component_test_full_cmake_gcc_asan` and zlib components: restore it.
    * `component_test_no_pem_no_fs` (×2): the merge placed two copies in different locations. Reconcile them: unset PSA storage like in crypto, and call `ssl-opt.sh` like in tls. Put the merged version at the tls location.
    * `component_test_everest`: do add it at the tls location.
    * `component_test_small_mbedtls_ssl_dtls_max_buffering`: restore the tls value.
    * `component_test_new_ecdh_context`…: move `component_test_new_ecdh_context` before `component_test_everest` and add a calls to `compat.sh` and `ssl-opt.sh` like in `component_test_everest`. Remove the redundant crypto-only `component_test_everest`. Don't remove `component_test_psa_collect_statuses`.
    * `component_test_full_cmake_clang`: don't remove `clang` in the `msg` call. Don't remove the call to `test_psa_constant_names.py`.
    * `component_test_full_make_gcc_o0`: remove it. It's subsumed by `component_test_gcc_opt`.
    * `component_build_deprecated`: don't remove anything.
    * `component_test_memory_buffer_allocator`: restore `ssl-opt.sh`.
    * `component_test_when_no_ciphersuites_have_mac`: restore it.
    * `component_test_platform_calloc_macro`: don't restore `unset MBEDTLS_MEMORY_BUFFER_ALLOC_C` which is now redundant. Don't restore explicit flags instead of `$ASAN_CFLAGS`.
    * `component_test_aes_fewer_tables`…: don't remove it.
    * `component_test_m32_o1`: restore SSL testing.
    * `component_test_m32_everest`: restore SSL testing.
    * `component_test_min_mpi_window_size`…: don't remove it.
    * `component_test_valgrind`: do restore the tls version of the comment.
    * `run_component`: don't remove the seedfile creation. This is better than `pre_check_seedfile` (see below).
    * `pre_check_seedfile`: don't restore it. `pre_check_seedfile` (from tls) creates a seedfile once and for all. This is not good enough if a component fails in such a way as to leave a broken seedfile, or if a component leaves a seedfile with a size that's wrong for the next component to run. Instead (from crypto), `run_component` creates a sufficiently large seedfile before each component.
2020-03-23 17:55:31 +01:00
Gilles Peskine b99bd39b4e Merge mbed-crypto into mbedtls: the merge commit
Merge `unremove-non-crypto` into `mbedtls/development`. The branch
`unremove-non-crypto` was obtained by starting from `mbed-crypto/development`,
then reverting many commits that removed X.509 and TLS functionality when Mbed
Crypto forked from Mbed TLS (the “unremoval”), then make a few tweaks to
facilitate the merge.

The unremoval step restored old versions of some tls files. If a file doesn't
exist in mbed-crypto, check out the mbedtls version, regardless of what
happened during the unremoval of tls files in the crypto tree. Also
unconditionally take the mbedtls version of a few files where the
modifications are completely project-specific and are not relevant in
mbed-crypto:

* `.github/issue_template.md`: completely different. We may want to reconcile
  them independently as a follow-up.
* `.travis.yml`: would only be reverted to an earlier tls version.
* `README.md`: completely different. We may want to reconcile them
  independently as a follow-up.
* `doxygen/input/doc_mainpage.h`: the changes in crypto were minimal and not
  relevant except as a stopgap as mbed-crypto did not have its own product
  versioning in the Doxygen documentation.
* `tests/.jenkins/Jenkinsfile`: completely different.
* `tests/data_files/Makefile`: there were no changes in mbed-crypto,
  but the unremoval step restored an old version.

Shell script for everything to do after the merge apart from the conflict
resolution:
```
tls_files=($(comm -23 <(git ls-tree -r --name-only HEAD) <(git ls-tree -r --name-only $(git merge-base upstream-crypto/development MERGE_HEAD))))
tls_files+=($tls_files .github/issue_template.md .travis.yml README.md doxygen/input/doc_mainpage.h tests/.jenkins/Jenkinsfile tests/data_files/Makefile)
git checkout --theirs HEAD -- $tls_files
git add -- $tls_files
```

Resolve the remaining conflicts:

* `library/CMakeLists.txt`:
    * Keep the TLS definition of `src_crypto`
    * `USE_SHARED_MBEDTLS_LIBRARY`: keep all three libraries, with both
      `include` and `crypto/include` in `target_include_directories`, all with
      version `2.21.0`.
* `programs/Makefile`:
    * Reconcile the APPS lists (add/add from a differently-formatted common
      ancestor): insert the `psa/*` from crypto into the tls list.
    * Keep the `fuzz` target defined only in tls version.
    * Keep the recipe (only in tls version) cleaning `ssl_pthread_server`
      stuff for the `clean` target.
* `scripts/config.py`:
    * `include_in_full`: add/add conflict. Keep both.
* `tests/scripts/all.sh`:
    * `component_test_no_use_psa_crypto_full_cmake_asan`: partially old
      version in crypto. Take the tls version.
    * `component_test_malloc_0_null` and more: take
      `component_test_malloc_0_null` from crypto (with `config.py` rather than
      `config.pl`, and with `$ASAN_FLAGS` rather than an explicit list), but
      add the call to `ssl-opt.sh` from tls. Take the other components from
      crypto.

With this commit, building and running the unit tests with both `make ` and
`cmake` work in the default configuration on Linux. Other platforms, build
systems and configurations are likely not to work, and there is some
regression in test coverage.

There is some loss of functionality because the unremoval step restored older
versions of tls content. This commit contains the latest tls version of
tls-only files, but some changes from the tls side in files that existed on
both sides have regressed. Most problematic changes are hunks that remove some
tls-specific feature and contain either a C preprocessor symbol identifying a
tls-specific module or option, or the name of a tls-specific file. Hunks
that remove a tls-specific preprocessor symbol can be identified with the
regular expression `^-.*MBEDTLS_(ERR_)?(PKCS11|X509|NET|SSL)_`.

Subsequent commits will revert a few parts of the patch from this merge commit
in order to restore the tls functionality that it removes, ensure that the
test coverage includes what was covered in either branch, and fix test
failures.
2020-03-23 17:54:46 +01:00
Manuel Pégourié-Gonnard 1fce45da6a
Merge pull request #3103 from AndrzejKurek/ssl-read-improve-ret-code-doc
ssl - improve documentation on mbedtls_ssl_read and PEER_CLOSE_NOTIFY
2020-03-23 11:36:31 +01:00
Manuel Pégourié-Gonnard 6bbfcc7c8e
Merge pull request #3105 from AndrzejKurek/ssl-srv-initialize-flags
ssl_srv.c: initialize flags on each iteration of the loop
2020-03-23 11:36:17 +01:00
Gilles Peskine 43aa905d1e DHM functions are not part of x509
In the old days, key parsing was part of x509, but these days it's
part of crypto.
2020-03-19 14:23:45 +01:00
Gilles Peskine 3b46cd3f15 Invoke config.py instead of config.pl in reverted content
perl -i -pe 's/\bconfig\.pl/config.py/g' $(git grep -l -Fw config.pl
-- . '#!tests/scripts/test_config_script.py')
2020-03-19 14:23:45 +01:00
Gilles Peskine 7e771c767f Link test programs that only use platform functions with mbedcrypto
Even if other higher-level libraries were added, these programs would
only link with the crypto library, which is the one that contains
platform functions.
2020-03-19 14:23:45 +01:00
Gilles Peskine f66346eaf8 Revert "Remove Diffie-Hellman examples"
This reverts commit bea98b4581.

Conflicts:
* programs/Makefile:
  * APPS: the layout of the definition has changed. re-add dh_client
    and dh_server appropriately.

Run scripts/generate_visualc_files.pl to account for the added programs.
2020-03-19 14:23:45 +01:00
Gilles Peskine 1bc9c135b3 Revert "selftest: Remove X.509 selftest"
This reverts commit 47a3635fc7.
2020-03-19 14:23:45 +01:00
Gilles Peskine 169087482f Revert "pkey: Remove dependency on X.509"
This reverts commit bf564c77fa.
2020-03-19 14:23:45 +01:00
Gilles Peskine 3a67150bf2 Revert "md: Remove dependency on X.509"
This reverts commit ebbc5f7940.
2020-03-19 14:23:45 +01:00
Gilles Peskine 1174db5bac Revert "ecp: Remove dependency on TLS and X.509"
This reverts commit de0a41b716.
2020-03-19 14:23:45 +01:00
Gilles Peskine d40f0070ec Revert "dhm: Remove dependency on TLS"
This reverts commit ed16ca7b63.
2020-03-19 14:23:45 +01:00
Gilles Peskine 69e8f7ffe3 Revert "all.sh: Remove dependency on TLS, NET, and X.509"
This reverts commit 9b90f2e294.

Conflicts:
* tests/scripts/all.sh: do the same changes, dancing around the new
  outcome file feature and components added in the same places.
  Make sure that the components that are getting added back are at the
  same locations as where they are now in mbedtls.
2020-03-19 14:23:45 +01:00
Gilles Peskine 96ddb0ab4d Revert "asn1: Remove dependency on X.509"
This reverts commit d8087713ae.
2020-03-19 14:23:45 +01:00
Gilles Peskine 9e277f4408 Revert "cpp_dummy_build: Remove X.509 dependency"
This reverts commit 4c1fdb5129.
2020-03-19 14:23:45 +01:00
Gilles Peskine 920b77524d Revert "recursion.pl: Don't depend on X.509"
This reverts commit e23737c618.
2020-03-19 14:23:45 +01:00
Gilles Peskine 7dc97048d6 Revert "Remove tests that depend on TLS or X.509"
This reverts commit 9afb2e9921.

Conflicts:
* include/CMakeLists.txt
  * "Make config.h available" comment: there has been a change
    adjacent to where it was removed. Just re-add what was removed.
* tests/CMakeLists.txt:
  * compat.sh: there has been a change immediately before where it was
    removed. Just re-add what was removed.
2020-03-19 14:17:54 +01:00
Gilles Peskine 1a9c624fce Revert "tests: Use parent module includes when used as a submodule"
This reverts commit 120d571e8e.

Conflicts:
* tests/CMakeLists.txt:
  * target_include_directories: the instruction whose addition is to
    be reverted has changed. Remove what is there now.
2020-03-19 14:13:59 +01:00
Gilles Peskine 5748757615 Revert "tests: Exclude version suite when used as a submodule"
This reverts commit 1264c2a86f.
2020-03-19 14:13:22 +01:00
Gilles Peskine b85b20dfdc Revert "tests: Add a crypto prefix to submodule tests"
This reverts commit b478bb6ddb.

Conflicts:
* tests/CMakeLists.txt: revert the introduction of exe_name, but keep
  the addition of ${CMAKE_SOURCE_DIR}/crypto/library/ to
  target_include_directories.
2020-03-19 14:13:13 +01:00
Gilles Peskine 6bbe783908 Revert "Remove pkcs11-helper option"
This reverts commit d832f187f7.

Conflicts:
* CMakeLists.txt:
  * USE_PKCS11_HELPER_LIBRARY: there has been a change immediately before
    where it was removed. Just re-add what was removed.
* tests/CMakeLists.txt:
  * USE_PKCS11_HELPER_LIBRARY: there has been a change immediately before
    where it was removed. Just re-add what was removed.
2020-03-19 14:07:55 +01:00
Gilles Peskine 5bb8bec1de Revert "Remove zlib"
This reverts commit d874a1fd14.

Conflicts:
* CMakeLists.txt:
  * ENABLE_ZLIB_SUPPORT: there has been a change immediately after
    where it was removed. Just re-add what was removed.
* tests/CMakeLists.txt:
  * ENABLE_ZLIB_SUPPORT: there has been a change immediately after
    where it was removed. Just re-add what was removed.
2020-03-19 14:07:55 +01:00