c7efb652f3 blockfilter: Update BIP 158 test vectors. (Jim Posen)
19308c9e21 rpc: Add getblockfilter RPC method. (Jim Posen)
ff35105096 init: Add CLI option to enable block filter index. (Jim Posen)
accc8b8b18 index: Access functions for global block filter indexes. (Jim Posen)
2bc90e4e7b test: Unit test for block filter index reorg handling. (Jim Posen)
6bcf0998c0 test: Unit tests for block index filter. (Jim Posen)
b5e8200db7 index: Implement lookup methods on block filter index. (Jim Posen)
75a76e3619 index: Implement block filter index with write operations. (Jim Posen)
2ad2338ef9 serialize: Serialization support for big-endian 32-bit ints. (Jim Posen)
ba6ff9a6f7 blockfilter: Functions to translate filter types to/from names. (Jim Posen)
62b7a4f094 index: Ensure block locator is not stale after chain reorg. (Jim Posen)
4368384f1d index: Allow atomic commits of index state to be extended. (Jim Posen)
Pull request description:
This introduces a new BlockFilterIndex class, which is required for BIP 157 support.
The index is uses the asynchronous BaseIndex infrastructure driven by the ValidationInterface callbacks. Filters are stored sequentially in flat files and the disk location of each filter is indexed in LevelDB along with the filter hash and header. The index is designed to ensure persistence of filters reorganized out of the main chain to simplify the BIP 157 net implementation.
Stats (block height = 565500):
- Syncing the index from scratch takes 45m
- Total index size is 3.8 GiB
ACKs for commit c7efb6:
MarcoFalke:
utACK c7efb652f3
ryanofsky:
Slightly tested ACK c7efb652f3 (I just rebuilt the index with the updated PR and tested the RPC). Changes since last review: rebase, fixed compile errors in internal commits, new comments, updated error messages, tweaked cache size logic, renamed commit method, renamed constants and globals, fixed whitespace, extra BlockFilterIndex::Init error check.
Tree-SHA512: f8ed7a9b6f76df45933aa5eba92b27b3af83f6df2ccb3728a5c89eec80f654344dc14f055f6f63eb9b3a7649dd8af6553fe14969889e7e2fd2f8461574d18f28
Adds the following util units and adds them to libbitcoin_util:
- `util/url.cpp` takes `urlDecode` from `httpserver.cpp`
- `util/error.cpp` takes `TransactionErrorString` from
`node/transaction.cpp` and `AmountHighWarn` and `AmountErrMsg` from
`ui_interface.cpp`
- `util/fees.cpp` takes `StringForFeeReason` and `FeeModeFromString` from `policy/fees.cpp`
- `util/rbf.cpp` takes `SignalsOptInRBF` from `policy/rbf.cpp`
- 'util/validation.cpp` takes `FormatStateMessage` and `strMessageMagic` from 'validation.cpp`
This moves the following policy settings functions and globals to a new
src/policy/settings unit in lib_server:
- `incrementalRelayFee`
- `dustRelayFee`
- `nBytesPerSigOp`
- `fIsBareMultisigStd`
These settings are only required by the node and should not be accessed
by other libraries.
519b0bc5dc Make last disconnected block BLOCK_FAILED_VALID, even when aborted (Pieter Wuille)
8d220417cd Optimization: don't add txn back to mempool after 10 invalidates (Pieter Wuille)
9ce9c37004 Prevent callback overruns in InvalidateBlock and RewindBlockIndex (Pieter Wuille)
9bb32eb571 Release cs_main during InvalidateBlock iterations (Pieter Wuille)
9b1ff5c742 Call InvalidateBlock without cs_main held (Pieter Wuille)
241b2c74ac Make RewindBlockIndex interruptible (Pieter Wuille)
880ce7d46b Call RewindBlockIndex without cs_main held (Pieter Wuille)
436f7d735f Release cs_main during RewindBlockIndex operation (Pieter Wuille)
1d342875c2 Merge the disconnection and erasing loops in RewindBlockIndex (Pieter Wuille)
32b2696ab4 Move erasure of non-active blocks to a separate loop in RewindBlockIndex (Pieter Wuille)
9d6dcc52c6 Abstract EraseBlockData out of RewindBlockIndex (Pieter Wuille)
Pull request description:
This PR makes a number of improvements to the InvalidateBlock (`invalidateblock` RPC) and RewindBlockIndex functions, primarily around breaking up their long-term cs_main holding. In addition:
* They're made safely interruptible (`bitcoind` can be shutdown, and no progress in either will be lost, though if incomplete, `invalidateblock` won't continue after restart and will need to be called again)
* The validation queue is prevented from overflowing (meaning `invalidateblock` on a very old block will not drive bitcoind OOM) (see #14289).
* `invalidateblock` won't bother to move transactions back into the mempool after 10 blocks (optimization).
This is not an optimal solution, as we're relying on the scheduler call sites to make sure the scheduler doesn't overflow. Ideally, the scheduler would guarantee this directly, but that needs a few further changes (moving the signal emissions out of cs_main) to prevent deadlocks.
I have manually tested the `invalidateblock` changes (including interrupting, and running with -checkblockindex and -checkmempool), but haven't tried the rewinding (which is probably becoming increasingly unnecessary, as very few pre-0.13.1 nodes remain that would care to upgrade).
Tree-SHA512: 692e42758bd3d3efc2eb701984a8cb5db25fbeee32e7575df0183a00d0c2c30fdf72ce64c7625c32ad8c8bdc56313da72a7471658faeb0d39eefe39c4b8b8474
6f6514a080 Correct units for "-dbcache" and "-prune" (Hennadii Stepanov)
Pull request description:
Actually, all `dbcache`-related values in the code are measured in MiB (not in megabytes, MB) or in bytes (e.g., `nTotalCache`).
See: https://github.com/bitcoin/bitcoin/blob/master/src/txdb.hba8c8b2227/src/init.cpp (L1405-L1424)
Also, "-prune" is fixed:
1. The GUI values in GB are translated to the node values in MiB correctly.
2. The maximum of the "prune" `QSpinBox` is not limited by default value of 99 (GB).
Fix: #15106
Tree-SHA512: 151ec43b31b1074db8b345fedb1dcc10bde225899a5296bfc183f57e1553d13ac27db8db100226646769ad03c9fcab29d88763065a471757c6c41ac51108459d
04da9f4834 [RPC] Update getrawtransaction interface (Amiti Uttarwar)
Pull request description:
- stop checking unspent UTXOs for a transaction when txindex is not enabled, as per conversation here: https://github.com/bitcoin/bitcoin/issues/3220#issuecomment-377458383
- code contributed by sipa
Tree-SHA512: aa07353bccc14b81b7803992a25d076d6bc06d15ec7c1b85828dc10aea7e0498d9b49f71783e352ab8a14b0bb2010cfb7835de3dfd1bc6f2323f460449348e66
All dbcache-related values in the code are measured in MiB (not in
megabytes, MB) or in bytes.
The GUI "-prune" values in GB are translated to the node values in MiB
correctly. The maximum of the "-prune" QSpinBox is not limited by the
default value of 99 (GB).
Also, this improves log readability.
bf2e01097 uint256: Remove unnecessary crypto/common.h use (Karl-Johan Alm)
Pull request description:
This is an alternative to #13242 which keeps the `ReadLE64` part, but moves the `crypto/common.h` dependency into `crypto/common.h` as a function outside of `uint256`.
**Reason:** this change will remove dependencies for `uint256` to `crypto/common.h`, `compat/endian.h`, and `compat/byteswap.h`.
This PR removes the need to update tests to be endian-aware/-independent, but keeps the (arguably dubious) `ReadLE64` part (which was only introduced to fix the tests, not for any functionality).
Tree-SHA512: 78b35123cdb185b3b3ec59aba5ca8a5db72624d147f2d6a5484ffa5ce626a72f782a01dc6893fc8f5619b03e2eae7b5a03b0df5d43460f3bda428e719e188aec
fa511e8dad Pass tx pool reference into CheckSequenceLocks (MarcoFalke)
Pull request description:
`CheckSequenceLocks` is called from ATMP and the member function `CTxMemPool::removeForReorg` without passing in the tx pool object that is used in those function's scope and instead using the global `::mempool` instance.
This fix should be refactoring only, since currently there is only one (global) tx pool in normal operation. Though, it fixes hard to track down issues in future settings where more than one mempool exists at a time. (E.g. for tests, rpc or p2p tx relay purposes)
Tree-SHA512: f0804588c7d29bb6ff05ec14f22a16422b89ab31ae714f38cd07f811d7dc7907bfd14e799c4c1c3121144ff22711019bbe9212b39e2fd4531936a4119950fa49
Call sync.h primitives "locks" and "mutexes" instead of "blocks" and "waitable
critical sections" to match current coding conventions and c++11 standard
names.
This PR does not rename the "CCriticalSection" class (though this could be done
as a followup) because it is used everywhere and would swamp the other changes
in this PR. Plain mutexes should mostly be preferred instead of recursive
mutexes in new code anyway.
-BEGIN VERIFY SCRIPT-
set -x
set -e
ren() { git grep -l $1 | xargs sed -i s/$1/$2/; }
ren CCriticalBlock UniqueLock
ren CWaitableCriticalSection Mutex
ren CConditionVariable std::condition_variable
ren cs_GenesisWait g_genesis_wait_mutex
ren condvar_GenesisWait g_genesis_wait_cv
perl -0777 -pi -e 's/.*typedef.*condition_variable.*\n\n?//g' src/sync.h
-END VERIFY SCRIPT-
3fc20632a3 qt: Set BLOCK_CHAIN_SIZE = 220 (DrahtBot)
2b6a2f4a28 Regenerate manpages (DrahtBot)
eb7daf4d60 Update copyright headers to 2018 (DrahtBot)
Pull request description:
Some trivial maintenance to avoid having to do it again after the 0.17 branch off.
(The scripts to do this are in `./contrib/`)
Tree-SHA512: 16b2af45e0351b1c691c5311d48025dc6828079e98c2aa2e600dc5910ee8aa01858ca6c356538150dc46fe14c8819ed8ec8e4ec9a0f682b9950dd41bc50518fa
d45b344ffd Bucket for inbound when scheduling invs to hide tx time (Gleb)
Pull request description:
It has been brought up to my attention that current random delays mechanism (originally intended to obfuscate transaction metadata) allows to easily estimate the time a transaction was received by a node.
It may be done by connecting multiple observer nodes to the same node. Each of those nodes will generate its own schedule of delays. Combined metadata regarding those events from different sources allows an observer to estimate transaction time.
After this patch a spy won't gain additional information by just creating multiple connections to a target.
Tree-SHA512: c71dae5ff350b614cb40a8e201fd0562d3e03e3e72a5099718cd451f0d84c66d5e52bbaf0d5b4b75137514c8efdedcc6ef4df90142b360153f04ad0721545ab1
25bc9615b7 Document validationinterace callback blocking deadlock potential. (Matt Corallo)
Pull request description:
From the branches-I've-had-lying-around-and-forgot-to-PR department...
This is a comment-only PR, but the comments point out an API quirk that isn't exactly trivial. None of our use-cases right now hit this, but if we were to call SyncWithValidationInterfaceQueue (eg to limit queue depth) in ATMP, I'm pretty sure we'd hit a deadlock there.
Tree-SHA512: 889dd8fc9eb15d1f2aa5ca467e783bc8f07bc543b166b032741795b0db7a0df11a2846d3cb7c69bafa8d1acf970021001b742f52be06725a932813230c5b4a7b
In `ProcessGetBlockData`, send the block data directly from disk if
type MSG_WITNESS_BLOCK is requested. This is a valid shortcut as the
on-disk format matches the network format.
This is expected to increase performance because a deserialization and
subsequent serialization roundtrip is avoided.
4a6c0e3dcf Modernize best block mutex/cv/hash variable naming (Pieter Wuille)
45dd135039 Fix csBestBlock/cvBlockChange waiting in rpc/mining (Pieter Wuille)
Pull request description:
This is an alternative to #11694.
It reintroduces a uint256 variable with the best block hash, protected by csBestBlock, and only updated while holding it.
Also rename the involved variable to modern guidelines, as there are very few uses.
Tree-SHA512: 826a86c7d3cee7fe49f99f4398ae99e81cb0563197eaeba77306a3ca6072b67cdb932bc35720fc0f99c2a57b218efa029d0b8bdfb240591a629b2e90efa3199d
a5bca13 Bugfix: Include <memory> for std::unique_ptr (Luke Dashjr)
Pull request description:
Not sure why all these includes were missing, but it's breaking builds for some users:
https://bugs.gentoo.org/show_bug.cgi?id=652142
(Added to all files with a reference to `std::unique_ptr`)
Tree-SHA512: 8a2c67513ca07b9bb52c34e8a20b15e56f8af2530310d9ee9b0a69694dd05e02e7a3683f14101a2685d457672b56addec591a0bb83900a0eb8e2a43d43200509
b55555d rpc: Add testmempoolaccept (MarcoFalke)
Pull request description:
To check if a single raw transaction makes it into the current transaction pool, one had to call `sendrawtransaction`. However, on success, this adds the transaction to the mempool with no easy way to undo.
The call `testmempoolaccept` is introduced to provide a way to solely check the result without changing the mempool state.
Tree-SHA512: 5afd9311190135cee8fc1f229c7d39bf893f1028f29e28d34f70df820198ff97b4bf86b41cbbd6e6c36a5c30073cefa92d541c74a4939c7a2a6fa283dfd41b63
adc2586 doc: Refer to witness reserved value as spec. in the BIP (MarcoFalke)
Pull request description:
BIP141 refers to the coinbase's input's witness that consists of a single 32-byte array as "witness reserved value".
This updates the code to follow the BIP
Tree-SHA512: 49c9463519bd11b9ff322eeecd638f7627aa8efdfb869f8549f9a160ff34281e1b5a0b9d83545a692de6f5ff795055292c423403b0f3ce7597e3f32273cf1deb
a192636 -blocksdir: keep blockindex leveldb database in datadir (Jonas Schnelli)
f38e4fd QA: Add -blocksdir test (Jonas Schnelli)
386a6b6 Allow to optional specify the directory for the blocks storage (Jonas Schnelli)
Pull request description:
Since the actual block files taking up more and more space, it may be desirable to have them stored in a different location then the data directory (use case: SSD for chainstate, etc., HD for blocks).
This PR adds a `-blocksdir` option that allows one to keep the blockfiles and the blockindex external from the data directory (instead of creating symlinks).
I fist had an option to keep the blockindex within the datadir, but seems to make no sense since accessing the index will (always) lead to access (r/w) the block files.
Tree-SHA512: f8b9e1a681679eac25076dc30e45e6e12d4b2d9ac4be907cbea928a75af081dbcb0f1dd3e97169ab975f73d0bd15824c00c2a34638f3b284b39017171fce2409
