fa324a8b15 doc: Rewrite some validation doc as lock annotations (MarcoFalke)
Pull request description:
#13402 added some lock annotations in comments. This pull removes them and adds clang-readable locking annotations instead.
Tree-SHA512: 2d392efa8ac4978830a9df08b2009e69d6f1ac031f62be2275ae8d7c7e483331c7f8d458d865443af907a7af27a592421c6cca6b2df3f2877e0f369b9198f383
66b2cf1ccf Use immintrin.h everywhere for intrinsics (Pieter Wuille)
4c935e2eee Add SHA256 implementation using using Intel SHA intrinsics (Pieter Wuille)
268400d318 [Refactor] CPU feature detection logic for SHA256 (Pieter Wuille)
Pull request description:
Based on #13191.
This adds SHA256 implementations that use Intel's SHA Extension instructions (using intrinsics). This needs GCC 4.9 or Clang 3.4.
In addition to #13191, two extra implementations are provided:
* (a) A variable-length SHA256 implementation using SHA extensions.
* (b) A 2-way 64-byte input double-SHA256 implementation using SHA extensions.
Benchmarks for 9001-element Merkle tree root computation on an AMD Ryzen 1800X system:
* Using generic C++ code (pre-#10821): 6.1ms
* Using SSE4 (master, #10821): 4.6ms
* Using 4-way SSE4 specialized for 64-byte inputs (#13191): 2.8ms
* Using 8-way AVX2 specialized for 64-byte inputs (#13191): 2.1ms
* Using 2-way SHA-NI specialized for 64-byte inputs (this PR): 0.56ms
Benchmarks for 32-byte SHA256 on the same system:
* Using SSE4 (master, #10821): 190ns
* Using SHA-NI (this PR): 53ns
Benchmarks for 1000000-byte SHA256 on the same system:
* Using SSE4 (master, #10821): 2.5ms
* Using SHA-NI (this PR): 0.51ms
Tree-SHA512: 2b319e33b22579f815d91f9daf7994a5e1e799c4f73c13e15070dd54ba71f3f6438ccf77ae9cbd1ce76f972d9cbeb5f0edfea3d86f101bbc1055db70e42743b7
d280617bf5 [qa] Add a test for merkle proof malleation (Suhas Daftuar)
ed82f17000 have verifytxoutproof check the number of txns in proof structure (Gregory Sanders)
Pull request description:
Recent publication of a weakness in Bitcoin's merkle tree construction demonstrates many SPV applications vulnerable to an expensive to pull off yet still plausible attack: https://bitslog.wordpress.com/2018/06/09/leaf-node-weakness-in-bitcoin-merkle-tree-design/
This change would at least allow `verifytxoutproof` to properly validate that the proof matches a known block, with known number of transactions any time after the full block is processed. This should neuter the attack entirely.
The negative is that a header-only processed block/future syncing mode would cause this to fail until the node has imported the data required.
related: #13451
`importprunedfunds` needs this check as well. Can expand it to cover this if people like the idea.
Tree-SHA512: 0682ec2b622a38b29f3f635323e0a8b6fc071e8a6fd134c954579926ee7b516e642966bafa667016744ce49c16e19b24dbc8801f982a36ad0a6a4aff6d93f82b
57889e688d bitcoin-tx: Stricter check for valid integers (Daniel Kraft)
Pull request description:
Just calling `atoi` to convert strings to integers does not check for valid integers very thoroughly; in particular, it just ignores everything starting from the first non-numeral character. Even a string like "foo" is fine and silently returns 0.
This meant that `bitcoin-tx` would not fail if such a string was passed in various places where an integer is expected (like the `locktime` or an input/output index); this means that it would, for instance, silently accept a typo and interpret it in an unexpected way.
In this change, we use `ParseInt64` for parsing strings to integers, which actually verifies that the full string is valid as number. New tests in the `bitcoin-util-test` cover the new error paths.
This fixes#13599.
Tree-SHA512: 146a0af275e9f57784e5d0582d3defbac35551b54b6b7232f8a0b20db04aa611125e52aa4512ef2f8ed2cafc2a12fe586f9d10ed66d641cff090288f279b1988
161e8d40a4 RPC: Add new getzmqnotifications method. (Daniel Kraft)
caac39b0ac Make ZMQ notification interface instance global. (Daniel Kraft)
Pull request description:
This adds a new RPC method `getzmqnotifications`, which returns information about all active ZMQ notification endpoints. This is useful for software that layers on top of bitcoind, so it can verify that ZeroMQ is enabled and also figure out where it should listen.
See #13526.
Tree-SHA512: edce722925741c84ddbf7b3a879fc9db1907e5269d0d97138fe724035d93ee541c2118c24fa92f4197403f380d0e25c2fda5ca6c62d526792ea749cf527a99a0
5e362c0cf0 Fix command line help for -printtoconsole and -debuglogfile (Samuel B. Atwood)
Pull request description:
This is a rebased version of #13589 with the changes to the 0.16.x release notes removed.
> #13004 changed the default behavior for printtoconsole but this has not been reflected in the command line help.
> This fixes the description of -printtoconsole to reflect this change, and also provides the user with missing information on how to explicitly disable logging to debug.log.
> At present I have made the latter update to two separate places (-printtoconsole and -debuglogfile) because a user looking for information on how to disable logging is probably going to look in the "Debugging/Testing Options" section. Moving -debuglogfile from the "General" options category to the "Debugging/Testing" section could potentially remove the need for this redundancy but may be out of the scope of this PR.
Tree-SHA512: 7461d59a1864039d5a9dfcce765a1169df882f51a4ca50a6066416c0803821cd821be07be534e0bd57f0a22c0b45adb881a93abbe91962bc37d2d228f35ee712
4b6ab02122 Remove unused argument to ProcessGetBlockData(...) (practicalswift)
c469ecf22e net: Remove unused interrupt from SendMessages (fanquake)
Pull request description:
Discussed very briefly with cfields.
Includes 65b4400 from #13554 as it's a similar refactor.
Tree-SHA512: 45cd64208a5c8164242db74e6687e9344ea592bab5e7f9ba8e1bb449057fc908ec9d8b8523748a68426e4a4304e3388a138cd834698b39837b2149b72beefdc9
63c16ed507 Use __cpuid_count for gnu C to avoid gitian build fail. (Chun Kuan Lee)
Pull request description:
Fixes#13538
Tree-SHA512: 161ae4db022288ae8631a166eaea2d08cf2c90bcd27218a094a754276de30b92ca9cfb5a79aa899c5a9d0534c5d7261037e7e915e1b92bc7067ab1539dc2b51e
#13004 changed the default behavior for printtoconsole but this has not been reflected in the command line help.
This fixes the description of -printtoconsole to reflect this change, and also provides the user with missing information on how to explicitly disable logging to debug.log.
At present I have made the latter update to two separate places (-printtoconsole and -debuglogfile) because a user looking for information on how to disable logging is probably going to look in the "Debugging/Testing Options" section. Moving -debuglogfile from the "General" options category to the "Debugging/Testing" section could potentially remove the need for this redundancy but may be out of the scope of this PR.
ea65182f03 [wallet] loadwallet shouldn't create new wallets. (John Newbery)
Pull request description:
A bug in the initial implementation of loadwallet meant that if the
arguement was a directory that didn't contain a wallet.dat file, a new
wallet would be created in that directory. Fix that so that if a
directory is passed in, it must contain a wallet.dat file.
Bug reported by promag (João Barbosa).
Tree-SHA512: 0a59fa8a33fde51a88544ad288b00e4995284fe16424f643076aaba42b8244fff362145217650ee53d518dfab7efbed4237632c34cdd3dcbbecaa9ecaab5fd7b
Just calling atoi to convert strings to integers does not check for
valid integers very thoroughly; in particular, it just ignores
everything starting from the first non-numeral character. Even a string
like "foo" is fine and silently returns 0.
This meant that bitcoin-tx would not fail if such a string was passed in
various places where an integer is expected (like the locktime or an
input/output index); this means that it would, for instance, silently
accept a typo and interpret it in an unexpected way.
In this change, we use ParseInt64 for parsing strings to integers,
which actually verifies that the full string is valid as number.
New tests in the bitcoin-util-test cover the new error paths.
1fc605a8ae fix bench/prevector.cpp (Akio Nakamura)
Pull request description:
This patch intends to fix some incorrect action of bench/prevector.cpp.
1. PrevectorClear()
2nd call of ```clear()``` should to operate t1 instead of t0.
This patch changes t0 to t1.
2. PREVECTOR_TEST()
PREVECTOR_TEST macro should to call both
```PrevectorXX<nontrivial_t>(state)``` and ```PrevectorXX<trivial_t>(state)```
by specific ```"name"``` which given by parameter instead of calling
```PrevectorResize<>()``` regardless of ```"name"```.
This patch changes ```"PrevectorResize<"``` of this macro to
```"Prevector ## name<"```.
Tree-SHA512: d0498c6d627d7e96fc8ccfb329ca0be2641535b1ce1923d9b1fc720825f9bf4d7281dc8d5ae929038e37b3e625189af9807cb62e6d20933d73832a6dff4b5596
98b181323 [build] Tune wildcards for LIBSECP256K1 target (Karl-Johan Alm)
Pull request description:
Automake would think the target was out of date every time because e.g. '.deps' was updated.
Note: I am assuming that secp256k1 depends on `*.h`, `*.c`, ~~and `libsecp256k1-config.h`~~ (it's `.h` so already included), aside from pre-existing `include/*`. If there are other files that would require a rebuild of the `LIBSECP256K1` target, they should probably be added.
It would be neat if you could exclude specific files, rather than split it up like this, but it doesn't seem possible (https://www.gnu.org/software/make/manual/html_node/Wildcard-Function.html#Wildcard-Function)
Should probably note this:
```Bash
$ V=1 make check VERBOSE=1
Making check in src
make[1]: Entering directory '/home/user/workspace/bitcoin/src'
make[2]: Entering directory '/home/user/workspace/bitcoin/src'
make -C secp256k1 libsecp256k1.la
make[3]: Entering directory '/home/user/workspace/bitcoin/src/secp256k1'
make[3]: 'libsecp256k1.la' is up to date.
make[3]: Leaving directory '/home/user/workspace/bitcoin/src/secp256k1'
make check-TESTS check-local
make[3]: Entering directory '/home/user/workspace/bitcoin/src'
make[4]: Entering directory '/home/user/workspace/bitcoin/src'
make -C secp256k1 libsecp256k1.la
make[5]: Entering directory '/home/user/workspace/bitcoin/src/secp256k1'
make[5]: 'libsecp256k1.la' is up to date.
make[5]: Leaving directory '/home/user/workspace/bitcoin/src/secp256k1'
PASS: test/test_bitcoin.exe
```
Tree-SHA512: 62b133c76e882788dae0c14208a9f5acdbd731c2e7a248f9e01f488b8ec13f9d637d7ad0d63e18d324bb4e088f1836a936649b0fb97bee679eaadedbeed5c981
2f1a30c63 Fix MAX_STANDARD_TX_WEIGHT check (Johnson Lau)
Pull request description:
As suggested by the constant name and its comment in policy.h, a transaction with a weight of exactly MAX_STANDARD_TX_WEIGHT should be allowed. Users could be confused.
Tree-SHA512: af417de1c6a2e6796ebbb39aa0caad8764302ded155cb1bbfbe457e4567c199cc53256189832b17d4aeec369e190b3edd4c6116d5f0b8cf0ede6dfb4ed83bdd3
2dcd7b4ec logging: avoid nStart may be used uninitialized in AppInitMain warning (mruddy)
Pull request description:
Was getting the following compiler warning:
```
init.cpp: In function ‘bool AppInitMain()’:
init.cpp:1616:60: warning: ‘nStart’ may be used uninitialized in this function [-Wmaybe-uninitialized]
LogPrintf(" block index %15dms\n", GetTimeMillis() - nStart);
```
It's ok without this PR, but this PR renames `nStart` to `load_block_index_start_time`, makes it `const`, and also reduces the scope of the variable.
The logging line is moved such that the the time spent will be logged even if a shutdown is requested while the index is being loaded.
Having the log message output even when a shutdown is requested may be how this was intended to work before anyways. That could explain the leading space, as such a log message now looks like:
```
2018-06-30T11:34:05Z [0%]...[16%]...[33%]...[50%]... block index 25750ms
2018-06-30T11:34:17Z Shutdown requested. Exiting.
```
Tree-SHA512: 967048afbc31f2ce8f80ae7d33fee0bdcbe94550cf2b5b662087e2a7cff14a8bf43d909b30f930660c184ec6c3c7e1302a84e3e54fc1723f7412827f4bf2c518
b81560029 Remove CombineSignatures and replace tests (Andrew Chow)
ed94c8b55 Replace CombineSignatures with ProduceSignature (Andrew Chow)
0422beb9b Make SignatureData able to store signatures and scripts (Andrew Chow)
b6edb4f5e Inline Sign1 and SignN (Andrew Chow)
Pull request description:
Currently CombineSignatures is used to create the final scriptSig or an input. However ProduceSignature is capable of doing this itself. Using both CombineSignatures and ProduceSignature results in code duplication which is unnecessary.
To move the scriptSig construction to ProduceSignatures, the SignatureData class contains two maps to hold pubkeys mapped to signatures, and script ids mapped to scripts. DataFromTransaction is extended to be able to extract signatures, their public keys, and scripts from existing ScriptSigs.
The SignaureData are then passed down to SignStep which can use the aforementioned maps to get the signatures, pubkeys, and scripts that it needs, falling back to the actual SigningProvider and SignatureCreator if the data are not available in the SignatureData.
Additionally, Sign1 and SignN have been removed and their functionality inlined into SignStep since Sign1 is really just a wrapper around CreateSig.
Since ProduceSignature can produce the final scriptSig or scriptWitness by using SignatureData which has extracted data from the transaction, CombineSignatures is unnecessary as ProduceSignature is able to replicate all of CombineSignatures' functionality.
This also furthers BIP 174 support and begins moving towards a BIP 174 style backend.
The tests have also been updated to use the new combining methodology.
Tree-SHA512: 78cd58a4ebe37f79229bd5eee2958a0bb45cd7f36d0e993eee13ff685b3665dd76ef2dfd5f47d34678995bb587f5594100ee5f6c09b1c69ee96d3684d470d01e
49d1f4cdd Detect if char equals int8_t (Chun Kuan Lee)
Pull request description:
Probably fixes#13576. I'm not able to test this. @stacepellegrino, can you test this?
Tree-SHA512: b750e00e11e6b6f6341fec668ec2254cc101c8ebdd4878f320d6cb3b07cf326761146e4ceff0b6405b7e503ff64c093a8274bd524a097e2c49382dc296972c4f
1. PrevectorClear()
2nd call of clear() should to operate t1 instead of t0.
This patch changes t0 to t1.
2. PREVECTOR_TEST()
PREVECTOR_TEST macro should to call both
PrevectorXX<nontrivial_t>(state) and PrevectorXX<trivial_t>(state)
by specific "name" which given by parameter instead of calling
PrevectorResize<>() regardless of "name".
This patch changes "PrevectorResize<" of this macro to
"Prevector ## name<".
This adds a new RPC method "getzmqnotifications", which returns
information about all active ZMQ notification endpoints. This is useful
for software that layers on top of bitcoind, so it can verify that
ZeroMQ is enabled and also figure out where it should listen.
See https://github.com/bitcoin/bitcoin/issues/13526.
b330c3001 Docs: Improve doc of options addnode, connect, seednode (wodry)
Pull request description:
Just clarify that options `addnode`, `connect` and `seednode` can be specified multiple times.
Tree-SHA512: ed149cabe7fc1d40f2fb6ad8b643656e0ec49cfae1834c157c89170eac1241efa3c5683d97266ff921f5229f28d732c9f7ee030e7902d9a79db1e0c8716fa3db
1fabd59e7 Break circular dependency: init -> * -> init by extracting shutdown.h (Ben Woosley)
e62fdfeea Drop unused init.h includes (Ben Woosley)
Pull request description:
Most includers just wanted to react to pending shutdown.
This isolates access to `fRequestShutdown` and limits access to the shutdown api functions, including the new `CancelShutdown` for setting it to `false`.
Tree-SHA512: df42f75dfbba163576710e9a67cf1228531fd99d70a2f187bfba0bcc476d6749cf88180a97e66a81bb5b6c3c7f0917de7402d26039ba7b644cb7509b02f7e267
07c493f2d scripted-diff: Replace NET_TOR with NET_ONION (wodry)
Pull request description:
This is a follow-up to #13532, where @promag already asked if this renaming would make sense.
If network shall be named _Onion_ instead of _Tor_ (like in the option `onlynet`), renaming the network enum NET_TOR to NET_ONION maybe would make sense and be stringent.
Change was produced with the following script:
```
#!/bin/bash
for file in $(grep --exclude-dir='.git' --files-with-matches --binary-files=without-match --recursive NET_TOR bitcoin/)
do
sed --in-place --expression='s/NET_TOR/NET_ONION/g' $file
done
```
_Tor_ is used at many other places in the code, though.
Tree-SHA512: 4ffdeca8115031465eb64e1c76694fb77b5900c4ea465d3c13d9b6b75a1eb04c45913f83cdc8bdbef28936aeec4655f1d4905b3b98407da3263632a2128a8d23
bb582a59c Add P2WSH destination helper and use it instead of manual hashing (Pieter Wuille)
eaba1c111 Add additional unit tests for invalid IsMine combinations (Pieter Wuille)
e6b9730c4 Do not expose invalidity from IsMine (Pieter Wuille)
Pull request description:
This improves the handling of INVALID in IsMine:
* Extra INVALID conditions were added to `IsMine` (following https://github.com/bitcoin/bitcoin/pull/13142/files#r185349057), but these were untested. Add unit tests for them.
* In https://github.com/bitcoin/bitcoin/pull/13142#issuecomment-386396975 it was suggested to merge `isInvalid` into the return status. This PR takes a different approach, and removes the `isInvalid` entirely. It was only ever used inside tests, as normal users of IsMine don't care about the reason for non-mine-ness, only whether it is or not. As the unit tests are extensive enough, it seems sufficient to have a black box text (with tests for both compressed and uncompressed keys).
Some addition code simplification is done as well.
Tree-SHA512: 3267f8846f3fa4e994f57504b155b0e1bbdf13808c4c04dab7c6886c2c0b88716169cee9c5b350513297e0ca2a00812e3401acf30ac9cde5d892f9fb59ad7fef
Instead of using CombineSignatures to create the final scriptSig or
scriptWitness of an input, use ProduceSignature itself.
To allow for ProduceSignature to place signatures, pubkeys, and scripts
that it does not know about, we pass down the SignatureData to SignStep
which pulls out the information that it needs from the SignatureData.
In addition to having the scriptSig and scriptWitness, have SignatureData
also be able to store just the signatures (pubkeys mapped to sigs) and
scripts (script ids mapped to scripts).
Also have DataFromTransaction be able to extract signatures and scripts
from the scriptSig and scriptWitness of an input to put them in SignatureData.
Adds a new SignatureChecker which takes a SignatureData and puts pubkeys
and signatures into it when it successfully verifies a signature.
Adds a new field in SignatureData which stores whether the SignatureData
was complete. This allows us to also update the scriptSig and
scriptWitness to the final one when updating a SignatureData with another
one.
-BEGIN VERIFY SCRIPT-
sed --in-place'' --expression='s/NET_TOR/NET_ONION/g' $(git grep -I --files-with-matches 'NET_TOR')
-END VERIFY SCRIPT-
The --in-place'' hack is required for sed on macOS to edit files in-place without passing a backup extension.
f618ebc4e4 validation: count blocks correctly for check level < 3 (Karl-Johan Alm)
Pull request description:
As noted in https://github.com/bitcoin/bitcoin/pull/13428#issuecomment-396129295 there is a bug where if check level < 3, the resulting count for blocks is wrong, because `pindexState` is never updated.
Post-commit `./bitcoin-cli verifychain 1 3`:
```
2018-06-11T07:12:28Z Verifying last 3 blocks at level 1
2018-06-11T07:12:28Z [0%]...[33%]...[66%]...[99%]...[DONE].
2018-06-11T07:12:28Z No coin database inconsistencies in last 3 blocks (0 transactions)
```
Pre-commit `./bitcoin-cli verifychain 1 3`:
```
2018-06-11T07:13:34Z Verifying last 3 blocks at level 1
2018-06-11T07:13:34Z [0%]...[33%]...[66%]...[99%]...[DONE].
2018-06-11T07:13:34Z No coin database inconsistencies in last 0 blocks (0 transactions)
```
Tree-SHA512: 3d82ed26665162c9615fb0e6e91a46ed4d229a5e6797c6c420e6b0bf1be6e5e02401c6e9a93b7a5aec503a2650d8c20d1b45fe300a922379e4cef8ee26e18d96
4132ad3bf Show symbol for inbound/outbound in peer table (wodry)
Pull request description:
Fixes#13483
The address in the network peer table is prefixed with an up-arrow symbolizing an outbound connection, or an down-array symbolizing an inbound connection. See screenshot.
The user has an easy visual confirmation about the connection direction state. I really like it :)
Impact to columns sorting is grouping by inbound/outbound first, which in my opinion is an advantage, too.
Tree-SHA512: d355f679d34c3006743c06750be5f36a083c1a8376da8f5f35045fcd9df964153409946fdde5007734f23bd692c91355962dc42df31122cdcf88e4affce8bc0e
This moves the used instance of CZMQNotificationInterface from a static
variable in init.cpp to a globally-accessible one declared in
zmq/zmqnotificationinterface.h. The variable is also renamed to
g_zmq_notification_interface, to be consistent with other globals.
We need this to implement a new RPC method "getzmqnotifications" (see
https://github.com/bitcoin/bitcoin/issues/13526) in a follow up.
c2e4fc84ec bench: Simplify CoinSelection (João Barbosa)
Pull request description:
Closes#13549.
As pointed by @MarcoFalke:
- `SelectCoinsMinConf` should always succeed as there are enough coins in the wallet.
- Removed creating the coins in the wallet.
Tree-SHA512: 965c363bcaf0ca7a1dec35b5cf4866abcf190c53eb7012dc4aeb4d29830f13a7465644bfb5a47f6ea3eaa86e4d4a57fe41e7b2593bf5094b76a551c4c71625bb
A bug in the initial implementation of loadwallet meant that if the
arguement was a directory that didn't contain a wallet.dat file, a new
wallet would be created in that directory. Fix that so that if a
directory is passed in, it must contain a wallet.dat file.
Bug reported by promag (João Barbosa).
9fdf05d70c tests: Fix lock-order-inversion (potential deadlock) in DoS_tests. Reported by TSAN. (practicalswift)
Pull request description:
Fix lock-order-inversion (potential deadlock) in `DoS_tests`. Reported by Clang's TSAN.
Makes `src/test/test_bitcoin` pass also when compiled with TreadSanitizer (`./configure --with-sanitizers=thread` with `clang`).
Tree-SHA512: 41403bb7b6e26bdf1b830b5699e27c637d522bae1799d2a19ed4b68b21b2555438b42170d8b1189613beb32a69b76a65175d29a83f5f4e493896c3d0d94ae26d
9f8c54b1b5 Log warning message when deprecated network name 'tor' is used (e.g. option onlynet=tor) (wodry)
Pull request description:
As @laanwj mentioned [here](https://github.com/bitcoin/bitcoin/pull/13418#discussion_r197645385), using option `onlynet=tor` is deprecated.
I think it would be good to give the user a depcreaction warning feedback, so users can switch to `onlynet=onion` so there is a perspective for removing the deprecated `tor` in the future to decrease confusion.
Currently, users maybe just wonder that they can use a undocumented option, or they are not aware that they use a deprecated option.
Alternatively for the log warning message, I think at least this deprecetaion should be documented in the source code in a comment for readers of the source code.
Tree-SHA512: f4889793cdd62a0a13353e13994ed50ca7d367fa9da9897ce909f86cf0b0ce6151b3c484c8e514b8ac332949c6bbc71001e06e918248a1089f73756bd4840602
