Technically cs_sendProcessing is entirely useless now because it
is only ever taken on the one MessageHandler thread, but because
there may be multiple of those in the future, it is left in place
cs_vSend is used for two purposes - to lock the datastructures used
to queue messages to place on the wire and to only call
SendMessages once at a time per-node. I believe SendMessages used
to access some of the vSendMsg stuff, but it doesn't anymore, so
these locks do not need to be on the same mutex, and also make
deadlocking much more likely.
e60360e net: remove cs_vRecvMsg (Cory Fields)
991955e net: add a flag to indicate when a node's send buffer is full (Cory Fields)
c6e8a9b net: add a flag to indicate when a node's process queue is full (Cory Fields)
4d712e3 net: add a new message queue for the message processor (Cory Fields)
c5a8b1b net: rework the way that the messagehandler sleeps (Cory Fields)
c72cc88 net: remove useless comments (Cory Fields)
ef7b5ec net: Add a simple function for waking the message handler (Cory Fields)
f5c36d1 net: record bytes written before notifying the message processor (Cory Fields)
60befa3 net: handle message accounting in ReceiveMsgBytes (Cory Fields)
56212e2 net: set message deserialization version when it's actually time to deserialize (Cory Fields)
0e973d9 net: remove redundant max sendbuffer size check (Cory Fields)
6042587 net: wait until the node is destroyed to delete its recv buffer (Cory Fields)
f6315e0 net: only disconnect if fDisconnect has been set (Cory Fields)
5b4a8ac net: make GetReceiveFloodSize public (Cory Fields)
e5bcd9c net: make vRecvMsg a list so that we can use splice() (Cory Fields)
53ad9a1 net: fix typo causing the wrong receive buffer size (Cory Fields)
This disentangles the script validation skipping from checkpoints.
A new option is introduced "assumevalid" which specifies a block whos
ancestors we assume all have valid scriptsigs and so we do not check
them when they are also burried under the best header by two weeks
worth of work.
Unlike checkpoints this has no influence on consensus unless you set
it to a block with an invalid history. Because of this it can be
easily be updated without risk of influencing the network consensus.
This results in a massive IBD speedup.
This approach was independently recommended by Peter Todd and Luke-Jr
since POW based signature skipping (see PR#9180) does not have the
verifiable properties of a specific hash and may create bad incentives.
The downside is that, like checkpoints, the defaults bitrot and older
releases will sync slower. On the plus side users can provide their
own value here, and if they set it to something crazy all that will
happen is more time will be spend validating signatures.
Checkblocks and checklevel are also moved to the hidden debug options:
Especially now that checkblocks has a low default there is little need
to change these settings, and users frequently misunderstand them as
influencing security or IBD speed. By hiding them we offset the
space added by this new option.
vRecvMsg is now only touched by the socket handler thread.
The accounting vars (nRecvBytes/nLastRecv/mapRecvBytesPerMsgCmd) are also
only used by the socket handler thread, with the exception of queries from
rpc/gui. These accesses are not threadsafe, but they never were. This needs to
be addressed separately.
Also, update comment describing data flow
Similar to the recv flag, but this one indicates whether or not the net's send
buffer is full.
The socket handler checks the send queue when a new message is added and pauses
if necessary, and possibly unpauses after each message is drained from its buffer.
Messages are dumped very quickly from the socket handler to the processor, so
it's the depth of the processing queue that's interesting.
The socket handler checks the process queue's size during the brief message
hand-off and pauses if necessary, and the processor possibly unpauses each time
a message is popped off of its queue.
In order to sleep accurately, the message handler needs to know if _any_ node
has more processing that it should do before the entire thread sleeps.
Rather than returning a value that represents whether ProcessMessages
encountered a message that should trigger a disconnnect, interpret the return
value as whether or not that node has more work to do.
Also, use a global fProcessWake value that can be set by other threads,
which takes precedence (for one cycle) over the messagehandler's decision.
Note that the previous behavior was to only process one message per loop
(except in the case of a bad checksum or invalid header). That was changed in
PR #3180.
The only change here in that regard is that the current node now falls to the
back of the processing queue for the bad checksum/invalid header cases.
In spite of the name FindLatestBefore used std::lower_bound to try
to find the earliest block with a nTime greater or equal to the
the requested value. But lower_bound uses bisection and requires
the input to be ordered with respect to the comparison operation.
Block times are not well ordered.
I don't know what lower_bound is permitted to do when the data
is not sufficiently ordered, but it's probably not good.
(I could construct an implementation which would infinite loop...)
To resolve the issue this commit introduces a maximum-so-far to the
block indexes and searches that.
For clarity the function is renamed to reflect what it actually does.
An issue that remains is that there is no grace period in importmulti:
If a address is created at time T and a send is immediately broadcast
and included by a miner with a slow clock there may not yet have been
any block with at least time T.
The normal rescan has a grace period of 7200 seconds, but importmulti
does not.
0c50909 testcases: explicitly specify transaction version 1 (John Newbery)
b7e144b Add test cases to test new bitcoin-tx functionality (jnewbery)
61a1534 Add all transaction output types to bitcoin-tx. (jnewbery)
1814b08 add p2sh and segwit options to bitcoin-tx outscript command (Stanislas Marion)
There is still a call to ActivateBestChain with cs_main if a peer
requests the block prior to it being validated, but this one is
more specifically-gated, so should be less of an issue.
AcceptToMemoryPool has several classes of return false statements.
- return state.Invalid or state.DoS directly itself
- return false and set fMissingInputs (state is valid)
- return false and state is set by failed CheckTransaction
- return false and state is set by failed CheckInputs.
This commit patches the last case where the state variable was reused for additional calls to CheckInputs to identify witness stripping as cause of validation failure. After this commit, it should be the case that if !fMissingInputs, state is always Invalid if AcceptToMemoryPool returns false.
4e7e2e1 Update RPC argument names (John Newbery)
481f289 rpc: Named argument support for bitcoin-cli (Wladimir J. van der Laan)
9adb4e1 rpc: Argument name consistency (Wladimir J. van der Laan)
8d713f7 rpc: Named arguments for rawtransaction calls (Wladimir J. van der Laan)
37a166f rpc: Named arguments for wallet calls (Wladimir J. van der Laan)
78b684f rpc: Named arguments for mining calls (Wladimir J. van der Laan)
b8ebc59 rpc: Named arguments for net calls (Wladimir J. van der Laan)
2ca9dcd test: Add test for RPC named arguments (Wladimir J. van der Laan)
fba1a61 rpc: Named arguments for misc calls (Wladimir J. van der Laan)
286ec08 rpc: Add 'echo' call for testing (Wladimir J. van der Laan)
495eb44 rpc: Named arguments for blockchain calls (Wladimir J. van der Laan)
6f1c76a rpc: Support named arguments (Wladimir J. van der Laan)
5865d41 authproxy: Add support for RPC named arguments (Wladimir J. van der Laan)
5113474 wallet: Use CDataStream.data() (Wladimir J. van der Laan)
e2300ff bench: Use CDataStream.data() (Wladimir J. van der Laan)
adff950 dbwrapper: Use new .data() method of CDataStream (Wladimir J. van der Laan)
a2141e4 streams: Remove special cases for ancient MSVC (Wladimir J. van der Laan)
af4c44c streams: Add data() method to CDataStream (Wladimir J. van der Laan)
20449ef Don't overpay fee if we have selected new coins that result in a smaller transaction. (Alex Morcos)
42f5ce4 Try to reduce change output to make needed fee in CreateTransaction (Alex Morcos)
032ba3f RPC help documentation for addnode peerinfo. (Gregory Maxwell)
90f13e1 Add release notes for addnode changes. (Gregory Maxwell)
50bd12c Break addnode out from the outbound connection limits. (Gregory Maxwell)
On repeated calls to SelectCoins we try to meet the fee necessary for the last transaction, the new fee required might be smaller, so increase our change by the difference if we can.
Once we've picked coins and dummy-signed the transaction to calculate fee, if we don't have sufficient fee, then try to meet the fee by reducing change before resorting to picking new coins.
Previously addnodes were in competition with outbound connections
for access to the eight outbound slots.
One result of this is that frequently a node with several addnode
configured peers would end up connected to none of them, because
while the addnode loop was in its two minute sleep the automatic
connection logic would fill any free slots with random peers.
This is particularly unwelcome to users trying to maintain links
to specific nodes for fast block relay or purposes.
Another result is that a group of nine or more nodes which are
have addnode configured towards each other can become partitioned
from the public network.
This commit introduces a new limit of eight connections just for
addnode peers which is not subject to any of the other connection
limitations (including maxconnections).
The choice of eight is sufficient so that under no condition would
a user find themselves connected to fewer addnoded peers than
previously. It is also low enough that users who are confused
about the significance of more connections and have gotten too
copy-and-paste happy will not consume more than twice the slot
usage of a typical user.
Any additional load on the network resulting from this will likely
be offset by a reduction in users applying even more wasteful
workaround for the prior behavior.
The retry delays are reduced to avoid nodes sitting around without
their added peers up, but are still sufficient to prevent overly
aggressive repeated connections. The reduced delays also make
the system much more responsive to the addnode RPC.
Ban-disconnects are also exempted for peers added via addnode since
the outbound addnode logic ignores bans. Previously it would ban
an addnode then immediately reconnect to it.
A minor change was also made to CSemaphoreGrant so that it is
possible to re-acquire via an object whos grant was moved.
Performing signing in the inner loop has terrible performance
when many passes through are needed to complete the selection.
Signing before the algorithm is complete also gets in the way
of correctly setting the fee (e.g. preventing over-payment when
the fee required goes down on the final selection.)
Use of the dummy might overpay on the signatures by a couple bytes
in uncommon cases where the signatures' DER encoding is smaller
than the dummy: Who cares?
In order to do this, we must call ActivateBestChain prior to
responding getdata requests for blocks which we announced using
compact blocks.
For getheaders responses we dont need code changes, but do note
that we must reset the bestHeaderSent so that the SendMessages call
re-announces the header in question.
While we could do something smarter for getblocks, calling
ActivateBestChain is simple and more obviously correct, instead of
doing something more similar to getheaders.
See-also the BIP clarifications at
https://github.com/bitcoin/bips/pull/486
The meaning is clear from the context, and we're inconsistent here.
Also save typing when using named arguments.
- `bitcoinaddress` -> `address`
- `bitcoinprivkey` -> `privkey`
- `bitcoinpubkey` -> `pubkey`
The [JSON-RPC specification](http://www.jsonrpc.org/specification) allows passing parameters as an Array, for by-position
arguments, or an Object, for by-name arguments.
This implements by-name arguments, but preserves full backwards compatibility. API using by-name arguments are
easier to extend, and easier to use (no need to guess which argument goes where).
Named are mapped to positions by a per-call structure, provided through the RPC command table.
Missing arguments will be replaced by null, except if at the end, then the argument is left out completely.
Currently calls fail (though not crash) on intermediate nulls, but this should be improved on a per-call basis later.
05a55a6 Added EVENT_CFLAGS to test makefile to explicitly include libevent headers. (Karl-Johan Alm)
280a559 Added some simple tests for the RAII-style events. (Karl-Johan Alm)
7f7f102 Switched bitcoin-cli.cpp to use RAII unique pointers with deleters. (Karl-Johan Alm)
e5534d2 Added std::unique_ptr<> wrappers with deleters for libevent modules. (Karl-Johan Alm)
Make a more conservative notion of whether the node is caught up to the rest of the network and only count transactions as fee estimation data points if the node is caught up.
All decisions about whether the transactions are valid data points are made at the time the transaction arrives. Updating on blocks all the time will now cause stale fee estimates to decay quickly when we restart a node.
Fee estimation can just check its own mapMemPoolTxs to determine the same information. Note that now fee estimation for block processing must happen before those transactions are removed, but this shoudl be a speedup.
This was an oversight, where blocks and mempool tracking were ignored during IBD, but transactions that arrived during IBD but were included in blocks after IBD were not ignored.
Once priority estimation was removed, not all transactions in the mempool are tracked in the fee estimation mempool tracking. So there is no error if a transaction is not found for removal.
We were marking coins FRESH before being sure they were not overwriting dirty undo data. This condition was never reached in existing code because undo data was always flushed before UpdateCoins was called with new transactions, but could have been exposed in an otherwise safe refactor.
Clarify in the comments the assumptions made in ModifyNewCoins.
Add ability to undo transactions to UpdateCoins unit test.
Thanks to Russ Yanofsky for suggestion on how to make logic clearer and fixing up the ccoins_modify_new test cases.
Surprisingly this hasn't been causing me any issues while testing, probably
because it requires lots of large blocks to be flying around.
Send/Recv corks need tests!
f8d43b8 Avoid rollingMinimumFeeRate never being able to decay below half (Alex Morcos)
eab8e1b fix a bug if the min fee is 0 for FeeFilterRounder (Alex Morcos)
- Drop the interruption point directly after the pnode allocation. This would
be leaky if hit.
- Rearrange thread creation so that the socket handler comes first
5394b39 Wallet: Split main logic from InitLoadWallet into CreateWalletFromFile (Luke Dashjr)
fb0c934 Wallet: Let the interval-flushing thread figure out the filename (Luke Dashjr)
8562792 GUI/RPCConsole: Include importmulti in history sensitive-command filter (Luke Dashjr)
ff77faf Qt/RPCConsole: Use RPCParseCommandLine to perform command filtering (Luke Dashjr)
a79598d Qt/Test: Make sure filtering sensitive data works correctly in nested commands (Luke Dashjr)
629cd42 Qt/RPCConsole: Teach RPCParseCommandLine how to filter out arguments to sensitive commands (Luke Dashjr)
e2d9213 Qt/RPCConsole: Make it possible to parse a command without executing it (Luke Dashjr)
1755c04 Qt/RPCConsole: Truncate filtered commands to just the command name, rather than skip it entirely in history (Luke Dashjr)
d80a006 Qt/RPCConsole: Add signmessagewithprivkey to list of commands filtered from history (Luke Dashjr)
afde12f Qt/RPCConsole: Refactor command_may_contain_sensitive_data function out of RPCConsole::on_lineEdit_returnPressed (Luke Dashjr)
de8980d Bugfix: Do not add sensitive information to history for real (Luke Dashjr)
9044908 Qt/RPCConsole: Don't store commands with potentially sensitive information in the history (Jonas Schnelli)
fc95daa Qt/RPCConsole: Save current command entry when browsing history (Jonas Schnelli)
This commit add testcases to test the following functions in bitcoin-tx:
- add a pay to non-standard script output
- add a P2SH output
- add a P2WSH output
- add a P2WSH wrapped in a P2SH output
- add a pay to pub key output
- add a P2WPKH output
- add a P2WPKH wrapped in a P2SH output
- add a bare multisig output
- add a multisig in P2SH output
- add a multisig in a P2WSH output
- add a multisig in a P2WSH wrapped in as P2SH output
This commit enhances bitcoin-tx so all remaining standard TXO types can be created:
- Pay to Pub Key
- Multi-sig
- bare multi-sig
- multi-sig in Pay To Script Hash
- multi-sig in Pay to Witness Script Hash
- multi-sig in Pay to Witness Script Hash, wrapped in P2SH
- Pay to Witness Pub Key Hash
- Pay to Witness Pub Key Hash, wrapped in P2SH
- Pay to Witness Script Hash
- Pay to Witness Script Hash, wrapped in P2SH
If the mempool is not completely full, treat the difference between
the maximum size and the actual usage as available for the coin cache.
This also changes the early flush trigger from (usage > 0.9 * space)
to (usage > 0.9 * space && usage > space - 100MB). This means we're not
permanently leaving 10% of the space unused when the space is large.
cee1612 reduce number of lookups in TransactionWithinChainLimit (Gregory Sanders)
af9bedb Test for fix of txn chaining in wallet (Gregory Sanders)
5882c09 CreateTransaction: Don't return success with too-many-ancestor txn (Gregory Sanders)
0b2294a SelectCoinsMinConf: Prefer coins with fewer ancestors (Gregory Sanders)
749be01 Move GetWarnings() into its own file. (Gregory Maxwell)
e3ba0ef Eliminate data races for strMiscWarning and fLargeWork*Found. (Gregory Maxwell)
c63198f Make QT runawayException call GetWarnings instead of directly access strMiscWarning. (Gregory Maxwell)
813ede9 [qa] Update compactblocks test for multi-peer reconstruction (Suhas Daftuar)
7017298 Allow compactblock reconstruction when block is in flight (Suhas Daftuar)
Defers to pre-defined version if found (e.g. protobuf). For protobuf case, the definitions are identical and thus include order should not affect results.
c5c92c4 Update python tests for default tx version=2 (BtcDrak)
dab207e Preserve tx version=1 for certain tests (BtcDrak)
c5d746a tiny test fix for mempool_tests (Alex Morcos)
1f0ca1a Bump default transaction version to 2 (BtcDrak)
67dac4e Add unit tests for the CuckooCache (Jeremy Rubin)
c9e69fb Add CuckooCache implementation and replace the sigcache map_type with it (Jeremy Rubin)
SQUASHME: Change cuckoocache to only work for powers of two, to avoid mod operator
SQUASHME: Update Documentation and simplify logarithm logic
SQUASHME: OSX Build Errors
SQUASHME: minor Feedback from sipa + bluematt
SQUASHME: DOCONLY: Clarify a few comments.
8225239 Merge #433: Make the libcrypto detection fail the newer API.
12de863 Make the libcrypto detection fail the newer API.
2928420 Merge #427: Remove Schnorr from travis as well
8eecc4a Remove Schnorr from travis as well
a8abae7 Merge #310: Add exhaustive test for group functions on a low-order subgroup
b4ceedf Add exhaustive test for verification
83836a9 Add exhaustive tests for group arithmetic, signing, and ecmult on a small group
20b8877 Add exhaustive test for group functions on a low-order subgroup
80773a6 Merge #425: Remove Schnorr experiment
e06e878 Remove Schnorr experiment
04c8ef3 Merge #407: Modify parameter order of internal functions to match API parameter order
6e06696 Merge #411: Remove guarantees about memcmp-ability
40c8d7e Merge #421: Update scalar_4x64_impl.h
a922365 Merge #422: Restructure nonce clearing
3769783 Restructure nonce clearing
0f9e69d Restructure nonce clearing
9d67afa Update scalar_4x64_impl.h
7d15cd7 Merge #413: fix auto-enabled static precompuatation
00c5d2e fix auto-enabled static precompuatation
91219a1 Remove guarantees about memcmp-ability
353c1bf Fix secp256k1_ge_set_table_gej_var parameter order
541b783 Fix secp256k1_ge_set_all_gej_var parameter order
7d893f4 Fix secp256k1_fe_inv_all_var parameter order
git-subtree-dir: src/secp256k1
git-subtree-split: 8225239f490f79842a5a3b82ad6cc8aa11d5208e
1a6eacb [QA] add fundrawtransaction test on a locked wallet with empty keypool (Jonas Schnelli)
c24a4f5 [Wallet] Bugfix: FRT: don't terminate when keypool is empty (Jonas Schnelli)
f36349e qt: Remove on_toggleNetworkActiveButton_clicked from RPCConsole (Wladimir J. van der Laan)
297cc20 qt: layoutAboutToChange signal is called layoutAboutToBeChanged (Wladimir J. van der Laan)
This resolves an issue where a wallet transaction which failed to
relay previously because it couldn't make it into the mempool
will not try again until restart, even though mempool conditions
may have changed.
Abandoned and known-conflicted transactions are skipped.
Some concern was expressed that there may be users with many
unknown conflicts would waste a lot of CPU time trying to
add them to their memory pools over and over again. But I am
doubtful these users exist in any number, if they do exist
they have worse problems, and they can mitigate any performance
issue this might have by abandoning the transactions in question.
