mbedtls/tests
Manuel Pégourié-Gonnard b6929891d6 Adjust timeout of tests with "no resend" assertions
There are currently 4 tests in ssl-opt.sh with either -C "resend" or -S
"resend", that is, asserting that no retransmission will occur. They sometimes
fail on loaded CI machines as one side doesn't send a message fast enough,
causing the other side to retransmit, causing the test to fail.

(For the "reconnect" test there was an other issue causing random failures,
fixed in a previous commit, but even after that fix the test would still
sometimes randomly fail, even if much more rarely.)

While it's a hard problem to fix in a general and perfect way, in practice the
probability of failures can be drastically reduced by making the timeout
values much larger.

For some tests, where retransmissions are actually expected, this would have
the negative effect of increasing the average running time of the test, as
each side would wait for longer before it starts retransmission, so we have a
trade-off between average running time and probability of spurious failures.

But for tests where retransmission is not expected, there is no such trade-off
as the expected running time of the test (assuming the code is correct most of
the time) is not impacted by the timeout value. So the only negative effect of
increasing the timeout value is on the worst-case running time on the test,
which is much less important, as test should only fail quite rarely.

This commit addresses the easy case of tests that don't expect retransmission
by increasing the value of their timeout range to 10s-20s. This value
corresponds to the value used for tests that assert `-S "autoreduction"` which
are in the same case and where the current value seems acceptable so far.

It also represents an increase, compared to the values before this commit, of
a factor 20 for the "reconnect" tests which were frequently observed to fail
in the CI, and of a factor 10 for the first two "DTLS proxy" tests, which were
observed to fail much less frequently, so hopefully the new values are enough
to reduce the probability of spurious failures to an acceptable level.

Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
2020-03-27 10:50:05 +01:00
..
.jenkins Add a Jenkinsfile for PR job testing 2018-10-19 16:41:54 +01:00
configs Add a test component with malloc(0) returning NULL 2020-02-11 19:26:28 +01:00
data_files Merge mbed-crypto into mbedtls: the merge commit 2020-03-23 17:54:46 +01:00
docker/bionic Improve compatibility with firewalled networks 2019-06-29 17:45:34 -05:00
git-scripts Merge development commit 8e76332 into development-psa 2019-01-31 08:20:20 -05:00
scripts Merge 'mbedtls/development' into merge-crypto-unremoved-20200304 2020-03-23 18:02:07 +01:00
suites Merge remote-tracking branch 'upstream-crypto/development' into HEAD 2020-03-25 15:47:43 +01:00
.gitignore PSA return status coverage script 2019-09-06 19:28:47 +02:00
CMakeLists.txt Merge mbed-crypto into mbedtls: the merge commit 2020-03-23 17:54:46 +01:00
compat-in-docker.sh Clean up file prologue comments 2019-06-29 17:45:34 -05:00
compat.sh Revert "Remove tests that depend on TLS or X.509" 2020-03-19 14:17:54 +01:00
Descriptions.txt Revert "Remove tests that depend on TLS or X.509" 2020-03-19 14:17:54 +01:00
make-in-docker.sh Clean up file prologue comments 2019-06-29 17:45:34 -05:00
Makefile Revert "tests: Use parent module includes when used as a submodule" 2020-03-19 14:13:59 +01:00
psa_crypto_helpers.h PSA return status coverage script 2019-09-06 19:28:47 +02:00
psa_helpers.h Move the one non-crypto-specific PSA helper macro to a new header 2019-06-20 12:54:43 +02:00
ssl-opt-in-docker.sh Clean up file prologue comments 2019-06-29 17:45:34 -05:00
ssl-opt.sh Adjust timeout of tests with "no resend" assertions 2020-03-27 10:50:05 +01:00