Document that git is needed.
Be clearer about the entry sort key being an entry sort key, not just
a merge order. Be clearer about what "merge order" means.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Changelog entry files were listed in reverse alphabetical order of the
file name, by happenstance. Now, changelog entry files are listed in
the order in which the changes were merged. More precisely: look for
the git commit where the entry file was created, and look where this
commit was merged into the current branch. List older merges first.
List never-merged commits in date order after all the merged ones.
List never-committed files in file timestamp order after all the
committed ones.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Conflicts:
* .github/issue_template.md, .github/pull_request_template.md:
The changes in Mbed Crypto are not relevant to Mbed TLS. Keep the
mbedtls versions.
Rather than sometimes returning an integer, sometimes a boolean and
sometimes implicitly returning None, always return 0 for success and 1
for failure.
No behavior change for the program as a whole, since the None/True/False
values were implicitly converted to the desired numerical value.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Don't use a function argument as a for loop variable. It worked (mostly) but
Pylint frowns on it (redefined-argument-from-local) and I think Pylint has a
point.
If the configuration file is not found, raise an exception mentioning the
search path rather than just its last element.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Merge the latest state of the target branch (mbedtls/development) into the
pull request to merge mbed-crypto into mbedtls.
Conflicts:
* ChangeLog: add/add conflict. Resolve by using the usual section order.
Rename identifiers containing double-underscore (`__`) to avoid `__`.
The reason to avoid double-underscore is that all identifiers
containing double-underscore are reserved in C++. Rename all such
identifiers that appear in any public header, including ssl_internal.h
which is in principle private but in practice is installed with the
public headers.
This commit makes check-names.sh pass.
```
perl -i -pe 's/\bMBEDTLS_SSL__ECP_RESTARTABLE\b/MBEDTLS_SSL_ECP_RESTARTABLE_ENABLED/g; s/\bMBEDTLS_KEY_EXCHANGE_(_\w+)_(_\w+)\b/MBEDTLS_KEY_EXCHANGE${1}${2}/g' include/mbedtls/*.h library/*.c programs/*/*.c scripts/data_files/rename-1.3-2.0.txt tests/suites/*.function
```
Merge `unremove-non-crypto` into `mbedtls/development`. The branch
`unremove-non-crypto` was obtained by starting from `mbed-crypto/development`,
then reverting many commits that removed X.509 and TLS functionality when Mbed
Crypto forked from Mbed TLS (the “unremoval”), then make a few tweaks to
facilitate the merge.
The unremoval step restored old versions of some tls files. If a file doesn't
exist in mbed-crypto, check out the mbedtls version, regardless of what
happened during the unremoval of tls files in the crypto tree. Also
unconditionally take the mbedtls version of a few files where the
modifications are completely project-specific and are not relevant in
mbed-crypto:
* `.github/issue_template.md`: completely different. We may want to reconcile
them independently as a follow-up.
* `.travis.yml`: would only be reverted to an earlier tls version.
* `README.md`: completely different. We may want to reconcile them
independently as a follow-up.
* `doxygen/input/doc_mainpage.h`: the changes in crypto were minimal and not
relevant except as a stopgap as mbed-crypto did not have its own product
versioning in the Doxygen documentation.
* `tests/.jenkins/Jenkinsfile`: completely different.
* `tests/data_files/Makefile`: there were no changes in mbed-crypto,
but the unremoval step restored an old version.
Shell script for everything to do after the merge apart from the conflict
resolution:
```
tls_files=($(comm -23 <(git ls-tree -r --name-only HEAD) <(git ls-tree -r --name-only $(git merge-base upstream-crypto/development MERGE_HEAD))))
tls_files+=($tls_files .github/issue_template.md .travis.yml README.md doxygen/input/doc_mainpage.h tests/.jenkins/Jenkinsfile tests/data_files/Makefile)
git checkout --theirs HEAD -- $tls_files
git add -- $tls_files
```
Resolve the remaining conflicts:
* `library/CMakeLists.txt`:
* Keep the TLS definition of `src_crypto`
* `USE_SHARED_MBEDTLS_LIBRARY`: keep all three libraries, with both
`include` and `crypto/include` in `target_include_directories`, all with
version `2.21.0`.
* `programs/Makefile`:
* Reconcile the APPS lists (add/add from a differently-formatted common
ancestor): insert the `psa/*` from crypto into the tls list.
* Keep the `fuzz` target defined only in tls version.
* Keep the recipe (only in tls version) cleaning `ssl_pthread_server`
stuff for the `clean` target.
* `scripts/config.py`:
* `include_in_full`: add/add conflict. Keep both.
* `tests/scripts/all.sh`:
* `component_test_no_use_psa_crypto_full_cmake_asan`: partially old
version in crypto. Take the tls version.
* `component_test_malloc_0_null` and more: take
`component_test_malloc_0_null` from crypto (with `config.py` rather than
`config.pl`, and with `$ASAN_FLAGS` rather than an explicit list), but
add the call to `ssl-opt.sh` from tls. Take the other components from
crypto.
With this commit, building and running the unit tests with both `make ` and
`cmake` work in the default configuration on Linux. Other platforms, build
systems and configurations are likely not to work, and there is some
regression in test coverage.
There is some loss of functionality because the unremoval step restored older
versions of tls content. This commit contains the latest tls version of
tls-only files, but some changes from the tls side in files that existed on
both sides have regressed. Most problematic changes are hunks that remove some
tls-specific feature and contain either a C preprocessor symbol identifying a
tls-specific module or option, or the name of a tls-specific file. Hunks
that remove a tls-specific preprocessor symbol can be identified with the
regular expression `^-.*MBEDTLS_(ERR_)?(PKCS11|X509|NET|SSL)_`.
Subsequent commits will revert a few parts of the patch from this merge commit
in order to restore the tls functionality that it removes, ensure that the
test coverage includes what was covered in either branch, and fix test
failures.
This reverts commit 9afb2e9921.
Conflicts:
* include/CMakeLists.txt
* "Make config.h available" comment: there has been a change
adjacent to where it was removed. Just re-add what was removed.
* tests/CMakeLists.txt:
* compat.sh: there has been a change immediately before where it was
removed. Just re-add what was removed.
This reverts commit 8298d70bee.
Conflicts:
* library/Makefile: removal of SOEXT_X509 and SOEXT_TLS vs change of
value of SOEXT_CRYPTO. Keep all, with the new value of SOEXT_CRYPTO.
This reverts commit 1c66e48670.
Conflicts:
* include/mbedtls/check_config.h:
* MBEDTLS_SSL_PROTO_SSL3: there has been an addition (of
MBEDTLS_SHA512_NO_SHA384) at the place where it was removed. Re-add it
after (alphabetical order).
* MBEDTLS_ENABLE_WEAK_CIPHERSUITES: there has been an addition (of
MBEDTLS_CTR_DRBG_USE_128_BIT_KEY) at the place where it was removed.
Re-add it after (alphabetical order).
* MBEDTLS_SSL_ALL_ALERT_MESSAGES: there has been an addition (of
MBEDTLS_SHA512_SMALLER) at the place where it was removed. Re-add it
after (alphabetical order).
* include/mbedtls/config.h:
* MBEDTLS_ENABLE_WEAK_CIPHERSUITES: there has been an addition (of
MBEDTLS_CTR_DRBG_USE_128_BIT_KEY) at the place where it was removed.
Re-add it after (alphabetical order).
* MBEDTLS_SSL_ALL_ALERT_MESSAGES: there has been an addition (of
MBEDTLS_SHA512_SMALLER) at the place where it was removed. Re-add it
after (alphabetical order).
* library/version_features.c: re-generate by running
scripts/generate_features.pl.
* programs/test/query_config.c: re-generate by running
scripts/generate_query_config.pl.
* scripts/config.pl: this file has been replaced by config.py. Port
the reversed changes to config.py:
* Revert removing three symbols from the list of symbols to
exclude from full.
* Revert removing one symbol (MBEDTLS_NET_C) from the list of symbols
to exclude from baremetal.
* scripts/footprint.sh:
* Re-add the line to unset MBEDTLS_NET_C, but with config.py instead of
config.pl.
* tests/scripts/all.sh:
* component_test_no_platform: re-add the line to unset MBEDTLS_NET_C, but
with config.py instead of config.pl.
* component_build_arm_none_eabi_gcc,
component_build_arm_none_eabi_gcc_no_udbl_division,
component_build_arm_none_eabi_gcc_no_64bit_multiplication,
component_build_armcc: these components now use the baremetal
configuration, so they do not need to turn off MBEDTLS_NET_C explicitly.
This reverts commit bb1f701212.
* include/mbedtls/check_config.h:
* MBEDTLS_X509_RSASSA_PSS_SUPPORT: there has been an addition (of
MBEDTLS_SHA512_NO_SHA384) at the place where it was removed.
Re-add it before MBEDTLS_SHA512_NO_SHA384 to keep it grouped
with MBEDTLS_RSA_C.
Conflicts:
* scripts/config.pl: this file has been replaced by config.py. Port
the reversed changes to config.py:
* Revert removing three symbols from the list of symbols to
exclude from full.
This reverts commit 356acc82ad.
Conflicts:
* scripts/generate_errors.pl: a line adjacent to a changed line has
independently changed in the meantime. Just revert the change done
in the commit that's being reverted.
Pylint when installed as a distro package can be installed as pylint3, whilst as
a PEP egg, it can be installed as pylint.
This commit changes the scripts to first use pylint if installed, and optionally
look for pylint3 if not installed. This is to allow a preference for the PEP
version over the distro version, assuming the PEP one is more likely to be
the correct one.
Signed-off-by: Simon Butcher <simon.butcher@arm.com>
`pylint3 --version` will output to stderr the status of the config file it's
using. This can be "No config file found" or "Using config file" or nothing.
This means the pylint version may or may not be on the first line.
Therefore this commit changes the filters on the pylint3 version output to first
strip out the config line, and then to select only the pylint line.
Signed-off-by: Simon Butcher <simon.butcher@arm.com>
Add the versions of Python, Perl, and Pylint to the version dump provided by
the output_env.sh script.
Signed-off-by: Simon Butcher <simon.butcher@arm.com>
A number of clean-up improvements following review.
* removal of redundant `` quotes
* removal of non-portable echo "\n", in favour of additional echo commands
* change to use of uname to detemine if the platform is Linux or not
* revised formatting of output
* change to dpkg-query from dpkg to find installed libasan variants
Co-Authored-By: Gilles Peskine <gilles.peskine@arm.com>
Signed-off-by: Simon Butcher <simon.butcher@arm.com>
This commit adds additional information to the output_env.sh script of:
* Linux distribution version (if available)
* GDB version (if available)
It also makes some information clearer:
* the type of OpenSSL/GNUTLS version (legacy/default/next)
* and whether certain versions are not installed, or not configured
And it simplifies the error messages for absent tools.
Signed-off-by: Simon Butcher <simon.butcher@arm.com>
generate_visualc_files.pl has a list of directories that it pulls
headers from, so it knows what directories to put on the include path.
Make it inject the include path into the output files, rather than
hard-coding the include paths in template files.
A similar change (but with different code) was made in Mbed TLS in
commit b78cf2b261
"Adjust visual studio file generation to always use the crypto submodule".
No semantic change: this commit does not change the generated files.
There's no need to keep the directory lists separated until the last
minute.
No semantic change. The generated files change slightly because there
was one directory list where slashes were not changed to backslashes
like in the other five. This does not affect their semantics.
Other third-party components can now be added by just adding lines to
the definitions of @thirdparty_header_dirs and
@thirdparty_source_dirs.
No semantic change. The output does not change at all.
As the SSL programs, like ssl_client2 and ssl_server2, are dependent on
SSL and therefore about to be removed, the only consumer of query_config
is the query_compile_time_config test. As such, it makes sense to move
query_config to be next to what uses it.
Files deleted by us: keep them deleted.
```
git rm $(git status -s | sed -n 's/^DU //p')
```
Individual files with conflicts:
* `README.md`: keep the crypto version.
* `doxygen/input/doc_mainpage.h`: keep the crypto version (with an obsolete Mbed Crypto version number).
* `include/mbedtls/error.h`:
* `ERROR`: similar additions made through parallel commits, with only whitespace differences. Align with the tls version.
* `library/CMakeLists.txt`: keep the crypto version.
* `library/Makefile`: keep the crypto version.
* `scripts/generate_errors.pl`: keep the crypto version (the relevant changes were made through parallel commits).
* `tests/scripts/check-test-cases.py`:
* `Results`: keep the crypto version, which has both the new argument to the constructor (added in crypto only) and the class docstring (added through parallel commits).
* `tests/suites/helpers.function`:
* `ARRAY_LENGTH`, `ASSERT_ALLOC`: additions in the same location. Keep both, in indifferent order.
* `tests/suites/target_test.function`:
* `receive_uint32`: keep the crypto version which has an additional bug fix. The tls changes made in tls are irrelevant after this bug fix.
* `visualc/VS2010/mbedTLS.vcxproj`: run `scripts/generate_visualc_files.pl`.
Review of non-conflicting changes:
* `all.sh`: 1 change.
* zlib test components: don't add them.
* `include/CMakeLists.txt`: 1 change.
* `target_include_directories`: doesn't work as is (different target name). Don't take the change.
* All other non-conflicting changes: take them.
All key types now have an encoding on 32 bits where the bottom 16 bits
are zero. Change to using 16 bits only.
Keep 32 bits for key types in storage, but move the significant
half-word from the top to the bottom.
Likewise, change EC curve and DH group families from 32 bits out of
which the top 8 and bottom 16 bits are zero, to 8 bits only.
Reorder psa_core_key_attributes_t to avoid padding.
Check that no line from any of the input files was lost.
This is not perfect for several reasons.
It doesn't check that the content goes to the desired location.
It doesn't check that sections are created as necessary.
It doesn't support whitespace normalization that the parsing code does.
But it's a good start.
Now that the Error module has error codes as well and is processed by
the generate_errors script like any other module, we don't need to
include the header manually.
One of the error codes was already reserved, this commit just makes it
explicit. The other one is a new error code for initializing return
values in the library: `MBEDTLS_ERR_ERROR_CORRUPTION_DETECTED` should
not be returned by the library. If it is returned, then it is surely a
bug in the library or somebody is tampering with the device.
Filter out non-ASCII characters in automatically processed headers.
Do this in a way that minimizes the code change: keep manipulating
strings, but strip off non-ASCII characters when reading lines, which
should only remove characters in comments that we don't parse anyway.
One of the error codes was already reserved, this commit just makes it
explicit. The other one is a new error code for initializing return
values in the library: `MBEDTLS_ERR_ERROR_CORRUPTION_DETECTED` should
not be returned by the library. If it is returned, then it is surely a
bug in the library or somebody is tampering with the device.
This is a variant toggle, not an extra feature, so it should be tested
separately.
We test most of the effect of MBEDTLS_ENTROPY_FORCE_SHA256 (namely,
using SHA-256 in the entropy module) when we test the library with the
SHA512 module disabled (which we do at least via depends-hashes.pl).
This commit removes testing of the MBEDTLS_ENTROPY_FORCE_SHA256 option
itself, which should be added separately.
Enabling MBEDTLS_MEMORY_BUFFER_ALLOC_C module together with
MBEDTLS_PLATFORM_MEMORY causes the library to use its own malloc
replacement. This makes memory management analyzers such as ASan
largely ineffective. We now test MBEDTLS_MEMORY_BUFFER_ALLOC_C
separately. Disable it in the "full" config.
This mirrors a change that was made in Mbed TLS on config.pl and had
not been ported to Mbed Crypto yet.
With this commit, config.py is aligned in Mbed Crypto and Mbed TLS.
Normally a valueless symbol remains valueless and a symbol with a
value keeps having one. But just in case a symbol does get changed
from valueless to having a value, make sure there's a space between
the symbol and the value. And if a symbol gets changed from having a
value to valueless, strip trailing whitespace.
Add corresponding tests.
Also fix the case of a valueless symbol added with the set method,
which would have resulted in attempting to use None as a string. This
only happened with the Python API, not with the command line API.
By default, this script looks for include/mbedtls/config.h relative to
the current directory. This allows running config.py from outside the
build tree.
To support out-of-tree builds where config.h and config.py are in the
source tree and the current directory is in the build tree, also try
DIRECTORY_CONTAINING_SCRIPT/../include/mbedtls/config.h, and the
equivalent with symbolic links traversed.
git grep -Fl /config.pl | xargs sed -i -e 's!/config\.pl!/config.py!g'
Also:
* Change one comment in include/mbedtls/check_config.h.
* Change PERL to PYTHON in CMakeLists.txt.
They're easier to maintain that way. The old lists were partly
alphabetized, partly based on config.h order, and partly in the order
in which symbols had been added to config.pl.
Also fix 'realfull' to only affect the appropriate sections.
Tested to produce the same results as config.pl on the default
configuration. This commit deliberately contains a direct copy the
lists of symbol names from config.pl.
This is meant to be a drop-in replacement for config.pl which can
additionally be used as a library in a Python script.
So far this script supports the commands 'get', 'set' and 'realfull'
but not the other built-in configurations.
assemble_changelog.py reads changelog entries from ChangeLog.d/*.md
and merges them into ChangeLog.md.
The changelog entries are merged into the first version in
ChangeLog.md. The script does not yet support creating a new version
in ChangeLog.md.
The changelog entries are merged in alphabetical order of the file
names. Future versions of this script are likely to adopt a different
order that reflects the git history of the entries.
Normally a valueless symbol remains valueless and a symbol with a
value keeps having one. But just in case a symbol does get changed
from valueless to having a value, make sure there's a space between
the symbol and the value. And if a symbol gets changed from having a
value to valueless, strip trailing whitespace.
Add corresponding tests.
Also fix the case of a valueless symbol added with the set method,
which would have resulted in attempting to use None as a string. This
only happened with the Python API, not with the command line API.
By default, this script looks for include/mbedtls/config.h relative to
the current directory. This allows running config.py from outside the
build tree.
To support out-of-tree builds where config.h and config.py are in the
source tree and the current directory is in the build tree, also try
DIRECTORY_CONTAINING_SCRIPT/../include/mbedtls/config.h, and the
equivalent with symbolic links traversed.
git grep -Fl /config.pl | xargs sed -i -e 's!/config\.pl!/config.py!g'
Also:
* Change one comment in include/mbedtls/check_config.h.
* Change PERL to PYTHON in CMakeLists.txt.
They're easier to maintain that way. The old lists were partly
alphabetized, partly based on config.h order, and partly in the order
in which symbols had been added to config.pl.
