Instead of working with Markdown format, keep the classic Mbed TLS
ChangeLog format, with the classic category names. Keep the classic
file name as well. This way there's no risk of breaking third-party
scripts that may copy or even parse the changelog file.
Accordingly, expect ChangeLog/*.txt files instead of ChangeLog/*.md.
This commit completely rewrites the parsing and output code.
This commit systematically appends to the existing top version. A
subsequent commit will restore the capability of creating a new
version.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Document that git is needed.
Be clearer about the entry sort key being an entry sort key, not just
a merge order. Be clearer about what "merge order" means.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Changelog entry files were listed in reverse alphabetical order of the
file name, by happenstance. Now, changelog entry files are listed in
the order in which the changes were merged. More precisely: look for
the git commit where the entry file was created, and look where this
commit was merged into the current branch. List older merges first.
List never-merged commits in date order after all the merged ones.
List never-committed files in file timestamp order after all the
committed ones.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Conflicts:
* .github/issue_template.md, .github/pull_request_template.md:
The changes in Mbed Crypto are not relevant to Mbed TLS. Keep the
mbedtls versions.
Merge the latest state of the target branch (mbedtls/development) into the
pull request to merge mbed-crypto into mbedtls.
Conflicts:
* ChangeLog: add/add conflict. Resolve by using the usual section order.
Rename identifiers containing double-underscore (`__`) to avoid `__`.
The reason to avoid double-underscore is that all identifiers
containing double-underscore are reserved in C++. Rename all such
identifiers that appear in any public header, including ssl_internal.h
which is in principle private but in practice is installed with the
public headers.
This commit makes check-names.sh pass.
```
perl -i -pe 's/\bMBEDTLS_SSL__ECP_RESTARTABLE\b/MBEDTLS_SSL_ECP_RESTARTABLE_ENABLED/g; s/\bMBEDTLS_KEY_EXCHANGE_(_\w+)_(_\w+)\b/MBEDTLS_KEY_EXCHANGE${1}${2}/g' include/mbedtls/*.h library/*.c programs/*/*.c scripts/data_files/rename-1.3-2.0.txt tests/suites/*.function
```
Merge `unremove-non-crypto` into `mbedtls/development`. The branch
`unremove-non-crypto` was obtained by starting from `mbed-crypto/development`,
then reverting many commits that removed X.509 and TLS functionality when Mbed
Crypto forked from Mbed TLS (the “unremoval”), then make a few tweaks to
facilitate the merge.
The unremoval step restored old versions of some tls files. If a file doesn't
exist in mbed-crypto, check out the mbedtls version, regardless of what
happened during the unremoval of tls files in the crypto tree. Also
unconditionally take the mbedtls version of a few files where the
modifications are completely project-specific and are not relevant in
mbed-crypto:
* `.github/issue_template.md`: completely different. We may want to reconcile
them independently as a follow-up.
* `.travis.yml`: would only be reverted to an earlier tls version.
* `README.md`: completely different. We may want to reconcile them
independently as a follow-up.
* `doxygen/input/doc_mainpage.h`: the changes in crypto were minimal and not
relevant except as a stopgap as mbed-crypto did not have its own product
versioning in the Doxygen documentation.
* `tests/.jenkins/Jenkinsfile`: completely different.
* `tests/data_files/Makefile`: there were no changes in mbed-crypto,
but the unremoval step restored an old version.
Shell script for everything to do after the merge apart from the conflict
resolution:
```
tls_files=($(comm -23 <(git ls-tree -r --name-only HEAD) <(git ls-tree -r --name-only $(git merge-base upstream-crypto/development MERGE_HEAD))))
tls_files+=($tls_files .github/issue_template.md .travis.yml README.md doxygen/input/doc_mainpage.h tests/.jenkins/Jenkinsfile tests/data_files/Makefile)
git checkout --theirs HEAD -- $tls_files
git add -- $tls_files
```
Resolve the remaining conflicts:
* `library/CMakeLists.txt`:
* Keep the TLS definition of `src_crypto`
* `USE_SHARED_MBEDTLS_LIBRARY`: keep all three libraries, with both
`include` and `crypto/include` in `target_include_directories`, all with
version `2.21.0`.
* `programs/Makefile`:
* Reconcile the APPS lists (add/add from a differently-formatted common
ancestor): insert the `psa/*` from crypto into the tls list.
* Keep the `fuzz` target defined only in tls version.
* Keep the recipe (only in tls version) cleaning `ssl_pthread_server`
stuff for the `clean` target.
* `scripts/config.py`:
* `include_in_full`: add/add conflict. Keep both.
* `tests/scripts/all.sh`:
* `component_test_no_use_psa_crypto_full_cmake_asan`: partially old
version in crypto. Take the tls version.
* `component_test_malloc_0_null` and more: take
`component_test_malloc_0_null` from crypto (with `config.py` rather than
`config.pl`, and with `$ASAN_FLAGS` rather than an explicit list), but
add the call to `ssl-opt.sh` from tls. Take the other components from
crypto.
With this commit, building and running the unit tests with both `make ` and
`cmake` work in the default configuration on Linux. Other platforms, build
systems and configurations are likely not to work, and there is some
regression in test coverage.
There is some loss of functionality because the unremoval step restored older
versions of tls content. This commit contains the latest tls version of
tls-only files, but some changes from the tls side in files that existed on
both sides have regressed. Most problematic changes are hunks that remove some
tls-specific feature and contain either a C preprocessor symbol identifying a
tls-specific module or option, or the name of a tls-specific file. Hunks
that remove a tls-specific preprocessor symbol can be identified with the
regular expression `^-.*MBEDTLS_(ERR_)?(PKCS11|X509|NET|SSL)_`.
Subsequent commits will revert a few parts of the patch from this merge commit
in order to restore the tls functionality that it removes, ensure that the
test coverage includes what was covered in either branch, and fix test
failures.
This reverts commit 9afb2e9921.
Conflicts:
* include/CMakeLists.txt
* "Make config.h available" comment: there has been a change
adjacent to where it was removed. Just re-add what was removed.
* tests/CMakeLists.txt:
* compat.sh: there has been a change immediately before where it was
removed. Just re-add what was removed.
This reverts commit 8298d70bee.
Conflicts:
* library/Makefile: removal of SOEXT_X509 and SOEXT_TLS vs change of
value of SOEXT_CRYPTO. Keep all, with the new value of SOEXT_CRYPTO.
This reverts commit 1c66e48670.
Conflicts:
* include/mbedtls/check_config.h:
* MBEDTLS_SSL_PROTO_SSL3: there has been an addition (of
MBEDTLS_SHA512_NO_SHA384) at the place where it was removed. Re-add it
after (alphabetical order).
* MBEDTLS_ENABLE_WEAK_CIPHERSUITES: there has been an addition (of
MBEDTLS_CTR_DRBG_USE_128_BIT_KEY) at the place where it was removed.
Re-add it after (alphabetical order).
* MBEDTLS_SSL_ALL_ALERT_MESSAGES: there has been an addition (of
MBEDTLS_SHA512_SMALLER) at the place where it was removed. Re-add it
after (alphabetical order).
* include/mbedtls/config.h:
* MBEDTLS_ENABLE_WEAK_CIPHERSUITES: there has been an addition (of
MBEDTLS_CTR_DRBG_USE_128_BIT_KEY) at the place where it was removed.
Re-add it after (alphabetical order).
* MBEDTLS_SSL_ALL_ALERT_MESSAGES: there has been an addition (of
MBEDTLS_SHA512_SMALLER) at the place where it was removed. Re-add it
after (alphabetical order).
* library/version_features.c: re-generate by running
scripts/generate_features.pl.
* programs/test/query_config.c: re-generate by running
scripts/generate_query_config.pl.
* scripts/config.pl: this file has been replaced by config.py. Port
the reversed changes to config.py:
* Revert removing three symbols from the list of symbols to
exclude from full.
* Revert removing one symbol (MBEDTLS_NET_C) from the list of symbols
to exclude from baremetal.
* scripts/footprint.sh:
* Re-add the line to unset MBEDTLS_NET_C, but with config.py instead of
config.pl.
* tests/scripts/all.sh:
* component_test_no_platform: re-add the line to unset MBEDTLS_NET_C, but
with config.py instead of config.pl.
* component_build_arm_none_eabi_gcc,
component_build_arm_none_eabi_gcc_no_udbl_division,
component_build_arm_none_eabi_gcc_no_64bit_multiplication,
component_build_armcc: these components now use the baremetal
configuration, so they do not need to turn off MBEDTLS_NET_C explicitly.
This reverts commit bb1f701212.
* include/mbedtls/check_config.h:
* MBEDTLS_X509_RSASSA_PSS_SUPPORT: there has been an addition (of
MBEDTLS_SHA512_NO_SHA384) at the place where it was removed.
Re-add it before MBEDTLS_SHA512_NO_SHA384 to keep it grouped
with MBEDTLS_RSA_C.
Conflicts:
* scripts/config.pl: this file has been replaced by config.py. Port
the reversed changes to config.py:
* Revert removing three symbols from the list of symbols to
exclude from full.
This reverts commit 356acc82ad.
Conflicts:
* scripts/generate_errors.pl: a line adjacent to a changed line has
independently changed in the meantime. Just revert the change done
in the commit that's being reverted.
Pylint when installed as a distro package can be installed as pylint3, whilst as
a PEP egg, it can be installed as pylint.
This commit changes the scripts to first use pylint if installed, and optionally
look for pylint3 if not installed. This is to allow a preference for the PEP
version over the distro version, assuming the PEP one is more likely to be
the correct one.
Signed-off-by: Simon Butcher <simon.butcher@arm.com>
`pylint3 --version` will output to stderr the status of the config file it's
using. This can be "No config file found" or "Using config file" or nothing.
This means the pylint version may or may not be on the first line.
Therefore this commit changes the filters on the pylint3 version output to first
strip out the config line, and then to select only the pylint line.
Signed-off-by: Simon Butcher <simon.butcher@arm.com>
Add the versions of Python, Perl, and Pylint to the version dump provided by
the output_env.sh script.
Signed-off-by: Simon Butcher <simon.butcher@arm.com>
A number of clean-up improvements following review.
* removal of redundant `` quotes
* removal of non-portable echo "\n", in favour of additional echo commands
* change to use of uname to detemine if the platform is Linux or not
* revised formatting of output
* change to dpkg-query from dpkg to find installed libasan variants
Co-Authored-By: Gilles Peskine <gilles.peskine@arm.com>
Signed-off-by: Simon Butcher <simon.butcher@arm.com>
This commit adds additional information to the output_env.sh script of:
* Linux distribution version (if available)
* GDB version (if available)
It also makes some information clearer:
* the type of OpenSSL/GNUTLS version (legacy/default/next)
* and whether certain versions are not installed, or not configured
And it simplifies the error messages for absent tools.
Signed-off-by: Simon Butcher <simon.butcher@arm.com>
generate_visualc_files.pl has a list of directories that it pulls
headers from, so it knows what directories to put on the include path.
Make it inject the include path into the output files, rather than
hard-coding the include paths in template files.
A similar change (but with different code) was made in Mbed TLS in
commit b78cf2b261
"Adjust visual studio file generation to always use the crypto submodule".
No semantic change: this commit does not change the generated files.
There's no need to keep the directory lists separated until the last
minute.
No semantic change. The generated files change slightly because there
was one directory list where slashes were not changed to backslashes
like in the other five. This does not affect their semantics.
Other third-party components can now be added by just adding lines to
the definitions of @thirdparty_header_dirs and
@thirdparty_source_dirs.
No semantic change. The output does not change at all.
As the SSL programs, like ssl_client2 and ssl_server2, are dependent on
SSL and therefore about to be removed, the only consumer of query_config
is the query_compile_time_config test. As such, it makes sense to move
query_config to be next to what uses it.
Files deleted by us: keep them deleted.
```
git rm $(git status -s | sed -n 's/^DU //p')
```
Individual files with conflicts:
* `README.md`: keep the crypto version.
* `doxygen/input/doc_mainpage.h`: keep the crypto version (with an obsolete Mbed Crypto version number).
* `include/mbedtls/error.h`:
* `ERROR`: similar additions made through parallel commits, with only whitespace differences. Align with the tls version.
* `library/CMakeLists.txt`: keep the crypto version.
* `library/Makefile`: keep the crypto version.
* `scripts/generate_errors.pl`: keep the crypto version (the relevant changes were made through parallel commits).
* `tests/scripts/check-test-cases.py`:
* `Results`: keep the crypto version, which has both the new argument to the constructor (added in crypto only) and the class docstring (added through parallel commits).
* `tests/suites/helpers.function`:
* `ARRAY_LENGTH`, `ASSERT_ALLOC`: additions in the same location. Keep both, in indifferent order.
* `tests/suites/target_test.function`:
* `receive_uint32`: keep the crypto version which has an additional bug fix. The tls changes made in tls are irrelevant after this bug fix.
* `visualc/VS2010/mbedTLS.vcxproj`: run `scripts/generate_visualc_files.pl`.
Review of non-conflicting changes:
* `all.sh`: 1 change.
* zlib test components: don't add them.
* `include/CMakeLists.txt`: 1 change.
* `target_include_directories`: doesn't work as is (different target name). Don't take the change.
* All other non-conflicting changes: take them.
All key types now have an encoding on 32 bits where the bottom 16 bits
are zero. Change to using 16 bits only.
Keep 32 bits for key types in storage, but move the significant
half-word from the top to the bottom.
Likewise, change EC curve and DH group families from 32 bits out of
which the top 8 and bottom 16 bits are zero, to 8 bits only.
Reorder psa_core_key_attributes_t to avoid padding.
