8081927c33 scripts: add key for fanquake to trusted keys list (fanquake)
Pull request description:
Adding my key to the [trusted keys list](https://github.com/bitcoin/bitcoin/blob/master/contrib/verify-commits/trusted-keys) to join the maintainer group. I'll gain merge access and will continue with all triage/repo management work. I'll be focusing primarily on build system development with some guidance from theuni.
Some maintainer related discussion from the Core Dev meetup in Amsterdam is available [here](http://diyhpl.us/wiki/transcripts/bitcoin-core-dev-tech/2019-06-06-maintainers/).
ACKs for commit 808192:
MarcoFalke:
ACK 8081927c33
laanwj:
ACK, this matches the key I have 8081927c33:
meshcollider:
ACK, this matches the key I have from the CoreDev New York 2018 keysigning party 8081927c33
IlyasRidhuan:
ACK. 8081927c33 matches the key I have.
Tree-SHA512: 63c390f5ede316263def2f9a897a1046d4ae58e4ea758f379164c6c5afce4928e2c9463fbcac004742838fd4ac1e48679e7a2a0f9095fd134a900f9064584056
00fade007c contrib: use newer config.guess & config.sub in install_db4.sh (fanquake)
Pull request description:
The `config.guess` and `config.sub` packaged with `db-4.8.30.NC.tar.gz` are fairly old (2009) and fail to identify some system types i.e `ppc64le`. Replace them with more modern versions before configuring `db4`.
Fixes#16064.
ACKs for commit 00fade:
jamesob:
tACK 00fade007c
dongcarl:
utACK 00fade007c
laanwj:
Tested ACK 00fade007c
Tree-SHA512: 54f9f2b0fcf802e6a284214035fcb3833577aa5881beb293921b6036b71d0bcd0872f06d14ddc76a4f8c0eee7ba71461dcc99dc907e7ddb5ae403fc02b245538
0f22a0cf2f Fix gitian-build.py --verify option (Hennadii Stepanov)
4c56a798c0 Set/unset USE_LXC, USE_VBOX, USE_DOCKER explicitly (Hennadii Stepanov)
cbbd98863b Fix Docker related issues for gitian-build.py (Hennadii Stepanov)
Pull request description:
1. The Docker does not depend on `apt-cacher-ng` package. Ref: #14002.
2. Do not try to install the Docker if `docker.service` is detected on the system (e.g., the Docker was installed manually). Fix https://github.com/bitcoin/bitcoin/pull/13623#issuecomment-405684241 by **Sjors**.
3. Prevent the setting of more than one environment variable for the `gitian-builder` (an alternative to #13999). E.g., USE_LXC being set shadows USE_DOCKER; for details see [`gitian-builder/libexec/make-clean-vm`](93a62c7d7d/libexec/make-clean-vm (L7)):
```sh
VMSW=KVM
if [ -n "$USE_LXC" ]; then
VMSW=LXC
elif [ -n "$USE_VBOX" ]; then
VMSW=VBOX
elif [ -n "$USE_DOCKER" ]; then
VMSW=DOCKER
fi
```
4. The [`gitian-builder/bin/gverify`](https://github.com/devrandom/gitian-builder/blob/master/bin/gverify) script returns the exit code 1 if a signature verification ends with 'BAD SIGNATURE' or 'MISMATCH' by design. This PR allows to see the verification results for all signatures without a premature fail of the `gitian-build.py` script. Ref: #14014.
ACKs for commit 0f22a0:
Tree-SHA512: 55f8a5cffa20d0c745f51a687f3199cea015fa616e56a0aee4c25b5ca0985036c61e8cf1922515338d8c6a85f873674ebe7a9a56a5069d65a187e383150f1a83
The gitian-builder/bin/gverify script returns the exit code 1 if a
signature verification ends with 'BAD SIGNATURE' or 'MISMATCH' by
design. This commit allows to see the verification results for all
signatures without a premature fail of the gitian-build.py script.
This prevents the setting of more than one environment variable for the
gitian-builder (e.g., USE_LXC being set shadows USE_DOCKER; for details
see gitian-builder/libexec/make-clean-vm).
The Docker does not depend on apt-cacher-ng package.
Do not try to install the Docker if docker.service is detected on the
system (e.g., the Docker was installed manually).
Also small style corrections were applied.
5c04814b2d Move non-linux source tarball to bitcoin-binaries (Hennadii Stepanov)
Pull request description:
Currently, if a user makes a non-linux (`--os=w`, `--os=m` or `--os=wm`) gitian building with the `gitian-build.py` script, source tarballs are not moved to the `bitcoin-binaries/${VERSION}` directory.
This PR fixes this bug.
~~In addition, the `src` subdirectory in the `gitian-builder/build/out` directory is no longer used as unnecessary.~~
ACKs for commit 5c0481:
fanquake:
utACK 5c04814
ken2812221:
utACK 5c04814b2d
Tree-SHA512: 8648b6cbf502c012b12642783870e37aea385bd5f4cba5cb577fee924c09685e9a117676be502e4d4783c7a8ab31a2bd495970eec42a42d78e86ac5d39323091
fa193dc8e6 doc: Remove win32 from the release process (MarcoFalke)
faf666f814 Remove Windows 32 bit build (MarcoFalke)
Pull request description:
The Windows 32 bit build has been removed from https://bitcoincore.org/en/download/, so unless there are complaints, we don't need to build it even
ACKs for commit fa193d:
fanquake:
utACK fa193dc8e6
Tree-SHA512: d6f2976a2e0c407698f720b00ac23ec4056626de4eff8621f4c5581120af0460afd1bdef72329cc0e7d92afca48d94ae5fce6777cb36bfabb60b8034ff08fd88
beda0dae95 Upgrade gitian image before signing (Hennadii Stepanov)
Pull request description:
The package upgrade in the Ubuntu repositories in the period between the building and the signing causes (particularly, using LXC) an error:
```
Creating package manifest
Could not download some packages, please run gbuild --upgrade
```
For example, the [`busybox-initramfs`](https://packages.ubuntu.com/bionic/busybox-initramfs) package was [upgraded](http://changelogs.ubuntu.com/changelogs/pool/main/b/busybox/busybox_1.27.2-2ubuntu3.2/changelog) from `1:1.27.2-2ubuntu3.1` to `1:1.27.2-2ubuntu3.2` on 2019-03-06.
This PR forces gitian image upgrade for the `--sign` command.
Ref:
[devrandom/gitian-builder/target-bin/grab-packages.sh](04ab7c1218/target-bin/grab-packages.sh)
```
#!/bin/sh
# Get an installed package manifest
set -e
cd /var/cache/apt/archives
# make sure all packages with installed versions are downloaded
# (except for held packages, which may not be available for download)
dpkg-query -W -f '${Status}\t${Package}=${Version}\n' | grep -v ^hold | cut -f2- | xargs -n 50 apt-get install -q --reinstall -y -d > /tmp/download.log
grep "cannot be downloaded" /tmp/download.log && { echo Could not download some packages, please run gbuild --upgrade 1>&2 ; exit 1 ; }
sha256sum *.deb | sort --key 2
```
ACKs for commit beda0d:
laanwj:
utACK beda0dae95
fanquake:
utACK beda0da
Tree-SHA512: e2e3b3e3719e098d266ceec39bd69b950344a4eb2f43ae6ad3e696add70f743b363cc83676e339f7caa207d6478029869a8af01fe1f6d5690d2857003f7d8ce8
8b8d8eeae9 Remove travis_wait from lint script (Graham Krizek)
Pull request description:
Using the `travis_wait` command in conjunction with `set -o errexit` causes problems. The `travis_wait` command will correctly log the command's output if successful, but if the command fails the process exits before the `travis_wait` command can dump the logs. This will hide important debugging information like error messages and stack traces. We ran into this in #15196 and it was very hard to debug because output was being suppressed.
`travis_wait` was being used because the `contrib/verify-commits/verify-commits.py` script can sometimes run for a long time without producing any output. If a script runs for 10 minutes without logging anything, the CI run times out. The `travis_wait` command will extend this timeout by logging a message for you, while sending stderr and stdout to a file.
This PR removes the `travis_wait` command from our CI system and adds additional logging to the `verify-commits.py` script so it doesn't make Travis timeout.
ACKs for commit 8b8d8e:
MarcoFalke:
utACK 8b8d8eeae9
Tree-SHA512: 175a8dd3f4d4e03ab272ddba94fa8bb06875c9027c3f3f81577feda4bc8918b5f0e003a19027f04f8cf2d0b56c68633716a6ab23f95b910121a8d1132428767d
b8705a0915 Set 'distro' explicitly (Hennadii Stepanov)
Pull request description:
The [gitian-builder](https://github.com/devrandom/gitian-builder) implicitly uses `ubuntu` as a default distro.
[bin/gbuild#L237](81edd2fc8e/bin/gbuild (L237)):
```ruby
distro = build_desc["distro"] || "ubuntu"
```
This PR sets a gitian building distro explicitly in description files.
Tree-SHA512: d2a692047f3466a5c637433610854d1100fe68a78fb03b4a81c70911fa14b0228d9cf25bcf115930aab9cc8c5063dacaf79bcd151f1c3f6fee6419389aefcb8b
ff7f31e07d [doc] productivity: more advanced git range-diff (Sjors Provoost)
3a21905a4e [doc] devtools: mention clang-format dependency (Sjors Provoost)
bf12093191 [doc] productivity: fix broken link (Sjors Provoost)
Pull request description:
Fixes a broken link to `devtools/README.md`, points out the `clang-format` dependency and adds a `git range-diff` incantation that works even with rebases and squashes.
Tree-SHA512: 36e46282f1e28d1bf3f48ada995fbac548f61b7747091eb032b60919cf76c7bdad0fa8aecb0c47adbdaa9ef986d3ec7752b0bb94c63191401856e2ddeec48f3e
32da92bdf6 gitian: Improve error handling (Wladimir J. van der Laan)
Pull request description:
Improve error handling in gitian builds:
- Set fail-on-error and pipefail flag, this causes a command to fail when either of the pipe stages fails, not only when the last of the stages fails, so this improves error detection.
- Also use `xargs` instead of `find -exec`, because `find` will not propagate errors in the executed command, but `xargs` will.
This will avoid some issues like #15541 where non-determinism is silently introduced due to errors caused by environment conditions (such as lack of disk space in that case).
Tree-SHA512: d5d3f22ce2d04a75e5c25e935744327c3adc704c2d303133f2918113573a564dff3d3243d5569a2b93ee7eb0e97f8e1b1ba81767e966af9015ea711a14091035
fa2797808e test: Remove python3.4 workaround in feature_dbcrash (MarcoFalke)
dddd1d05d3 .python-version: Specify full version 3.5.6 (MarcoFalke)
faa7cdf764 scripted-diff: Update copyright in ./test (MarcoFalke)
fa0e65b772 scripted-diff: test: Remove brackets after assert (MarcoFalke)
fab5a1e0f4 build: Require python 3.5 (MarcoFalke)
fa6bf21f5e scripted-diff: test: Use py3.5 bytes::hex() method (MarcoFalke)
Pull request description:
Python 3.4 is EOL after March 2019, so switch to 3.5. See https://devguide.python.org/#status-of-python-branches
This pull does the following in a bunch of commits:
* scripted diff to use the `bytes::hex()` method in place of previous wrappers (`b2x`, `bytes_to_hex_str`, `hexlify`, ...)
* Update the build system (gitian and travis) to remove python2.7 and replace it with python3.5
* Another scripted-diff to remove brackets after `assert`. This is unrelated to the python3.5 switch, but a stylistic commit, so probably not worth to split up. The motivation behind it is to avoid asserting on data structures (such as tuples of length one), which never fails:
```py
>>> assert(False,) # with brackets
>>> assert False, # without brackets
SyntaxError: invalid syntax
>>> assert False # proper assertion
AssertionError
```
* And then a final scripted diff to update the copyright headers in the `test` subfolder, since I touched most of the files anyway and it wouldn't make sense to split this commit out into a separate pull.
For reference (contributed by luke-jr):
Ubuntu LTS (bionic): 3.6.5
Debian stable (stretch): 3.5.3
RHEL 8 (expected before v0.19): 3.6.x
Gentoo stable: 3.6.5
Arch: 3.7.1
Tree-SHA512: 643c28cd2d5b9543ce4bf8ad2a8b282bc79b37dc5b25c9c8358e6ce201e2a67a546463e5f3430b16652eb2489d7c3ed4b0772cd2e2bf790fe68a5e3cc8a25029
f7efd87c8f Change brace formatting (Russell Yanofsky)
a1df1b48a8 Remove use of IsInitialBlockDownload in wallet code (Russell Yanofsky)
1106a6fde4 Remove use of uiInterface.LoadWallet in wallet code (Russell Yanofsky)
318f41fb2c circular-dependencies: Avoid treating some .h/.cpp files as a unit (Russell Yanofsky)
d02b34c8a8 Remove use of AcceptToMemoryPool in wallet code (Russell Yanofsky)
e2c8ba9f6e Remove uses of InitMessage/Warning/Error in wallet code (Russell Yanofsky)
c5e59a96a8 Remove uses of GetAdjustedTime in wallet code (Russell Yanofsky)
6d6bcc77c0 Remove use of g_connman / PushInventory in wallet code (Russell Yanofsky)
00dfb2a440 Remove uses of g_connman in wallet code (Russell Yanofsky)
cc3836e8f9 Remove uses of fPruneMode in wallet code (Russell Yanofsky)
cc02c796d3 Remove uses of fee globals in wallet code (Russell Yanofsky)
1fb0a4a04e Remove use of CalculateMemPoolAncestors in wallet code (Russell Yanofsky)
cd32160af0 Remove use of GetTransactionAncestry in wallet code (Russell Yanofsky)
291276f7f4 Remove use of GetCountWithDescendants in wallet code (Russell Yanofsky)
bdc6628683 Remove use of IsRBFOptIn in wallet code (Russell Yanofsky)
80f52a2267 Remove uses of CheckFinalTx in wallet code (Russell Yanofsky)
Pull request description:
This change removes wallet calls to node functions that access global chain and mempool state.
This is the next step in the larger #10973 refactoring change, which removes all other accesses to node global variables from wallet code. Doing this is useful to provide a better defined interface between the wallet and node, and necessary to allow wallet and node code to run in separate processes in #10102.
Tree-SHA512: 40dbaf1f59fb22b32e70b054b30ba5638d638aa3240fa30e0f721d53c721cd6138a7ab4d423a24d7d2fda0b956e68d44c733abc2c9259c3d6c9fd6d4be89aa23
43206239a8 tests: Add script checking for deterministic line coverage (practicalswift)
Pull request description:
Add script checking for deterministic line coverage in unit tests.
Context: #14343 ("coverage reports non-deterministic")
When the coverage is deterministic this script can be invoked from Travis to guard against regressions, but left inactive for now.
Output in case of determinism:
```
$ contrib/test_deterministic_coverage.sh 2
[2019-01-30 20:08:46] Measuring coverage, run #1 of 2
[2019-01-30 20:10:45] Measuring coverage, run #2 of 2
Coverage test passed: Deterministic coverage across 2 runs.
```
Output in case of non-determinism:
```
$ contrib/test_deterministic_coverage.sh 2
[2019-01-30 20:08:46] Measuring coverage, run #1 of 2
[2019-01-30 20:10:45] Measuring coverage, run #2 of 2
The line coverage is non-deterministic between runs.
The test suite must be deterministic in the sense that the set of lines executed at least
once must be identical between runs. This is a neccessary condition for meaningful coverage
measuring.
--- gcovr.run-1.txt 2019-01-30 23:14:07.419418694 +0100
+++ gcovr.run-2.txt 2019-01-30 23:15:57.998811282 +0100
@@ -471,7 +471,7 @@
test/crypto_tests.cpp 270 270 100%
test/cuckoocache_tests.cpp 142 142 100%
test/dbwrapper_tests.cpp 148 148 100%
-test/denialofservice_tests.cpp 225 225 100%
+test/denialofservice_tests.cpp 225 224 99% 363
test/descriptor_tests.cpp 116 116 100%
test/fs_tests.cpp 24 3 12% 14,16-17,19-20,23,25-26,29,31-32,35-36,39,41-42,45-46,49,51-52
test/getarg_tests.cpp 111 111 100%
@@ -585,5 +585,5 @@
zmq/zmqpublishnotifier.h 5 0 0% 12,31,37,43,49
zmq/zmqrpc.cpp 21 0 0% 16,18,20,22,33-35,38-45,49,52,56,60,62-63
------------------------------------------------------------------------------
-TOTAL 61561 27606 44%
+TOTAL 61561 27605 44%
------------------------------------------------------------------------------
```
In this case line 363 of `test/denialofservice_tests.cpp` was executed only in the second run. Non-determinism detected!
Tree-SHA512: 03f45590e70a87146f89aa7838beeff0925d7fd303697ff03e0e69f8a5861694be5f0dd10cb0020e3e3d40c9cf662f71dfcd838f6affb31bd5212314e0a4e3a9
This avoids a bogus circular dependency error in the next commit:
interfaces/chain -> interfaces/wallet -> wallet/wallet -> interfaces/chain
Which is incorrect, because interfaces/chain.cpp depends only on the
interfaces/wallet.h file, not the interfaces/wallet.cpp file, and it is
wrong to treat these as a unit. Inside the interfaces directory, .h files
contain abstract class definitions and .cpp files contain implementations of
those classes, so you don't need to link against .cpp files if you're only
using the abstract class definition in the .h file.
An alternative fix might be to rename all the cpp files in the interfaces
directory like: chain.cpp->chain_impl.cpp, node.cpp->node_impl.cpp. But just
getting the linter to treat these files as independent dependencies seemed
like it would allow keeping code organization straightforward and avoiding
the need to rename things.
a786c3b306 Slight tweak to the verify-commits script directions (Douglas Roark)
Pull request description:
Clarify that GnuPG may be used on both Linux and macOS to obtain the keys required to verify the commits.
Tree-SHA512: cec556370f03e00bbd6f585d26b360ca236cf55cb5c0996f6d950d8a98f77c92cc02f1719c8f9b9dc9eac6900eb341a13b50a012752832f39095b7e84046f2cd
