76faa3c Rename the remaining main.{h,cpp} to validation.{h,cpp} (Matt Corallo)
e736772 Move network-msg-processing code out of main to its own file (Matt Corallo)
87c35f5 Remove orphan state wipe from UnloadBlockIndex. (Matt Corallo)
Sorry for the churn on this, but the current message (introduced in #9073)
isn't acceptable:
$ src/bitcoin-cli getinfo
rpc: couldn't connect to server
(make sure server is running and you are connecting to the correct RPC port: -1 unknown)
Putting the error code after the words "RPC port" made me wonder whether
there was a port configuration issue.
This changes it to:
$ src/bitcoin-cli getinfo
error: couldn't connect to server: unknown (code -1)
(make sure server is running and you are connecting to the correct RPC port)
As orphan state is now "network state", like in
d6ea737be1,
UnloadBlockIndex is only used during init if we end up reindexing
to clear our block state so that we can start over. However, at
that time no connections have been brought up as CConnman hasn't
been started yet, so all of the network processing state logic is
empty when its called.
2c8c57e Document cs_main status when calling into PNB or PNBH (Matt Corallo)
58a215c Use ProcessNewBlockHeaders in CMPCTBLOCK processing (Matt Corallo)
a8b936d Use exposed ProcessNewBlockHeaders from ProcessMessages (Matt Corallo)
63fd101 Split ::HEADERS processing into two separate cs_main locks (Matt Corallo)
4a6b1f3 Expose AcceptBlockHeader through main.h (Matt Corallo)
This was misnamed, resulting in a warning message and missing
functionality. I'm not sure what the change in behavior will be here,
this needs testing.
Also remove connection to non-existing slot "test".
This was used for testing if the signal arrived. It is no
longer necessary.
Fixes:
2016-12-01 10:04:06 GUI: QObject::connect: No such signal PeerTableModel::layoutAboutToChange() in qt/rpcconsole.cpp:518
2016-12-01 10:04:06 GUI: QObject::connect: (receiver name: 'RPCConsole')
2016-12-01 10:04:06 GUI: QObject::connect: No such slot RPCConsole::test() in qt/rpcconsole.cpp:781
2016-12-01 10:04:06 GUI: QObject::connect: (receiver name: 'RPCConsole')
deec83f init: Get rid of fServer flag (Wladimir J. van der Laan)
16ca0bf init: Try to aquire datadir lock before and after daemonization (Wladimir J. van der Laan)
0cc8b6b init: Split up AppInit2 into multiple phases (Wladimir J. van der Laan)
Before daemonization, just probe the data directory lock and print an
early error message if possible.
After daemonization get the data directory lock again and hold on to it until exit
This creates a slight window for a race condition to happen, however this condition is harmless: it
will at most make us exit without printing a message to console.
$ src/bitcoind -testnet -daemon
Bitcoin server starting
$ src/bitcoind -testnet -daemon
Error: Cannot obtain a lock on data directory /home/orion/.bitcoin/testnet3. Bitcoin Core is probably already running.
When generating a new service key, explicitly request a RSA1024 one.
The bitcoin P2P protocol has no support for the longer hidden service names
that will come with ed25519 keys, until it does, we depend on the old
hidden service type so make this explicit.
See #9214.
Output instances of "BloomFilter" changed to "Bloom filter", in accordance with Wikipedia standard notation:
https://en.wikipedia.org/wiki/Bloom_filter
also to sync with the majority of cases in the self-same file
c7be56d net: push only raw data into CConnman (Cory Fields)
2ec935d net: add CVectorWriter and CNetMsgMaker (Cory Fields)
b7695c2 net: No need to check individually for disconnection anymore (Cory Fields)
fedea8a net: don't send any messages before handshake or after requested disconnect (Cory Fields)
d74e352 net: Set feelers to disconnect at the end of the version message (Cory Fields)
This fixes one of the last major layer violations in the networking stack.
The network side is no longer in charge of message serialization, so it is now
decoupled from Bitcoin structures. Only the header is serialized and attached
to the payload.
CVectorWriter is useful for overwriting or appending an existing byte vector.
CNetMsgMaker is a shortcut for creating messages on-the-fly which are suitable
for pushing to CConnman.
Bring up the modal progress overlay when the user clicks the spinner
icon in the task bar.
I think this is the intuitive thing to do when that icon is clicked.
ed998ea qt: Avoid OpenSSL certstore-related memory leak (Wladimir J. van der Laan)
5204598 qt: Avoid shutdownwindow-related memory leak (Wladimir J. van der Laan)
e4f126a qt: Avoid splash-screen related memory leak (Wladimir J. van der Laan)
693384e qt: Prevent thread/memory leak on exiting RPCConsole (Wladimir J. van der Laan)
47db075 qt: Plug many memory leaks (Wladimir J. van der Laan)
Also, send reject messages earlier in SendMessages(), so that disconnections are
processed earlier.
These changes combined should ensure that no message is ever sent after
fDisconnect is set.
This way we're not relying on messages going out after fDisconnect has been
set.
This should not cause any real behavioral changes, though feelers should
arguably disconnect earlier in the process. That can be addressed in a later
functional change.
d2b88f9 Move orphan-conflict removal from main logic into a callback (Matt Corallo)
97e2802 Erase orphans per-transaction instead of per-block (Matt Corallo)
ec4525c Move orphan processing to ActivateBestChain (Matt Corallo)
Make splash screen queue its own deletion when it receives the finished
command, instead of relying on WA_DeleteOnClose which doesn't work under
these circumstances.
Make ownership of the QThread object clear, so that the RPCConsole
can wait for the executor thread to quit before shutdown is called. This
increases overall thread safety, and prevents some objects from leaking
on exit.
None of these are very serious, and are leaks in objects that are
created at most one time.
In most cases this means properly using the QObject parent hierarchy,
except for BanTablePriv/PeerTablePriv which are not QObject,
so use a std::unique_ptr instead.
f004e67 Minor change to comment above new NODE_WITNESS service flag to keep it consitent with existing comment structure. Helps with readability. (Greg Walker)
Make sure that the count is a zero modulo the new mask before
scaling, otherwise the next time until a measure triggers
will take only 1/2 as long as accounted for. This caused
the 'min time' to be potentially off by as much as 100%.
When a BIP152 HB-mode peer is in the least preferred position and
disconnects, they will not be by ForNode on the next loop. They
will continue to sit in that position and prevent deactivating
HB mode for peers that are still connected.
There is no reason for them to stay in the list if already gone,
so drop the first element unconditionally if there are too many.
Fixes issue #9163.
037159c Remove block-request logic from INV message processing (Matt Corallo)
3451203 [qa] Respond to getheaders and do not assume a getdata on inv (Matt Corallo)
d768f15 [qa] Make comptool push blocks instead of relying on inv-fetch (mrbandrews)
- Do sorting for date, amount and confirmations column as longlong, not
unsigned longlong.
- Use `UserRole` to store our own data. This makes it treated as
ancillary data prevents it from being displayed.
- Get rid of `getMappedColumn` `strPad` - these are no longer necessary.
- Get rid of hidden `_INT64` columns.
- Start enumeration from 0 (otherwise values are undefined).
In 0.13 orphan transactions began being treated as implicit
INVs for their parents. But the resulting getdata were
not getting the witness flag.
This fixes issue #9182 reported by chjj and roasbeef on IRC.
Make a copy of the boost time-point to wait for, otherwise the head of
the queue may be deleted by another thread while this one is waiting,
while the boost function still has a reference to it.
Although this problem is in non-test code, this is not an actual problem
outside of the tests because we use the thread scheduler with only one
service thread, so there will never be threads fighting at the head of
the queue.
The old boost fallback escapes this problem because it passes a scalar
value to wait_until instead of a const object reference.
Found by running the tests in LLVM-4.0-master asan.
This further decouples "main" and "net" processing logic by moving
orphan processing out of the chain-connecting cs_main lock and
into its own cs_main lock, beside all of the other chain callbacks.
Once further decoupling of net and main processing logic occurs,
orphan handing should move to its own lock, out of cs_main.
Note that this will introduce a race if there are any cases where
we assume the orphan map to be consistent with the current chain
tip, however I am confident there is no such case (ATMP will fail
without DoS score in all such cases).
ae22357 Replace CValidationState param in ProcessNewBlock with BlockChecked (Matt Corallo)
7c98ce5 Remove pfrom parameter from ProcessNewBlock (Matt Corallo)
e2e069d Revert "RPC: Give more details when "generate" fails" (Matt Corallo)
dac53b5 Modify getblocktxn handler not to drop requests for old blocks (Russell Yanofsky)
55bfddc [qa] Fix stale data bug in test_compactblocks_not_at_tip (Russell Yanofsky)
47e9659 [qa] Fix bug in compactblocks v2 merge (Russell Yanofsky)
e2b3fb3 Optimize vInOutPoints insertion a bit (Matt Corallo)
eecffe5 Remove redundant duplicate-input check from CheckTransaction (Matt Corallo)
b2e178a Add deserialize + CheckBlock benchmarks, and a full block hex (Matt Corallo)
d59a518 Use fixed preallocation instead of costly GetSerializeSize (Pieter Wuille)
25a211a Add optimized CSizeComputer serializers (Pieter Wuille)
a2929a2 Make CSerAction's ForRead() constexpr (Pieter Wuille)
a603925 Avoid -Wshadow errors (Pieter Wuille)
5284721 Get rid of nType and nVersion (Pieter Wuille)
657e05a Make GetSerializeSize a wrapper on top of CSizeComputer (Pieter Wuille)
fad9b66 Make nType and nVersion private and sometimes const (Pieter Wuille)
c2c5d42 Make streams' read and write return void (Pieter Wuille)
50e8a9c Remove unused ReadVersion and WriteVersion (Pieter Wuille)
It is unexpected behavior for `ToString` to raise an exception. It
is expected to do a best-effort attempt at formatting but never fail.
Catch the exception and simply print unknown inv types as hexadecimal.
Fixes#9110.
4441018 Every main()/exit() should return/use one of EXIT_ codes instead of magic numbers (UdjinM6)
bd0de13 Fix exit codes: - `--help`, `--version` etc should exit with `0` i.e. no error ("not enough args" case should still trigger an error) - error reading config file should exit with `1` (UdjinM6)
d4833ff Bump the protocol version to distinguish new banning behavior. (Suhas Daftuar)
88c3549 Fix compact block handling to not ban if block is invalid (Suhas Daftuar)
c93beac [qa] Test that invalid compactblocks don't result in ban (Suhas Daftuar)
Dbwrapper used GetSerializeSize() to compute the size of the buffer
to preallocate. For some cases (specifically: CCoins) this requires
a costly compression call. Avoid this by just using fixed size
preallocations instead.
To get the advantages of faster GetSerializeSize() implementations
back that were removed in "Make GetSerializeSize a wrapper on top of
CSizeComputer", reintroduce them in the few places in the form of a
specialized Serialize() implementation. This actually gets us in a
better state than before, as these even get used when they're invoked
indirectly in the serialization of another object.
The CSerAction's ForRead() method does not depend on any runtime
data, so guarantee that requests to it can be optimized out by
making it constexpr.
Suggested by Cory Fields.
Remove the nType and nVersion as parameters to all serialization methods
and functions. There is only one place where it's read and has an impact
(in CAddress), and even there it does not impact any of the recursively
invoked serializers.
Instead, the few places that need nType or nVersion are changed to read
it directly from the stream object, through GetType() and GetVersion()
methods which are added to all stream classes.
Given that in default GetSerializeSize implementations created by
ADD_SERIALIZE_METHODS we're already using CSizeComputer(), get rid
of the specialized GetSerializeSize methods everywhere, and just use
CSizeComputer. This removes a lot of code which isn't actually used
anywhere.
For CCompactSize and CVarInt this actually removes a more efficient
size computing algorithm, which is brought back in a later commit.
The current getblocktxn implementation drops and ignores requests for old
blocks, which causes occasional sync_block timeouts during the
p2p-compactblocks.py test as reported in
https://github.com/bitcoin/bitcoin/issues/8842.
The p2p-compactblocks.py test setup creates many new blocks in a short
period of time, which can lead to getblocktxn requests for blocks below the
hardcoded depth limit of 10 blocks. This commit changes the getblocktxn
handler not to ignore these requests, so the peer nodes in the test setup
will reliably be able to sync.
The protocol change is documented in BIP-152 update "Allow block responses
to getblocktxn requests" at https://github.com/bitcoin/bips/pull/469.
The protocol change is not expected to affect nodes running outside the test
environment, because there shouldn't normally be lots of new blocks being
rapidly added that need to be synced.
The stream implementations had two cascading layers (the upper one
with operator<< and operator>>, and a lower one with read and write).
The lower layer's functions are never cascaded (nor should they, as
they should only be used from the higher layer), so make them return
void instead.
9027680 net: handle version push in InitializeNode (Cory Fields)
7588b85 net: construct CNodeStates in place (Cory Fields)
440f1d3 net: remove now-unused ssSend and Fuzz (Cory Fields)
5c2169c drop the optimistic write counter hack (Cory Fields)
ea33268 net: switch all callers to connman for pushing messages (Cory Fields)
3e32cd0 connman is in charge of pushing messages (Cory Fields)
b98c14c serialization: teach serializers variadics (Cory Fields)
Three categories of modifications:
1)
1 instance of 'The Bitcoin Core developers \n',
1 instance of 'the Bitcoin Core developers\n',
3 instances of 'Bitcoin Core Developers\n', and
12 instances of 'The Bitcoin developers\n'
are made uniform with the 443 instances of 'The Bitcoin Core developers\n'
2)
3 instances of 'BitPay, Inc\.\n' are made uniform with the other 6
instances of 'BitPay Inc\.\n'
3)
4 instances where there was no '(c)' between the 'Copyright' and the year
where it deviates from the style of the local directory.
- `--help`, `--version` etc should exit with `0` i.e. no error ("not enough args" case should still trigger an error)
- error reading config file should exit with `1`
Slightly refactor AppInitRPC/AppInitRawTx to return standard exit codes (EXIT_FAILURE/EXIT_SUCCESS) or CONTINUE_EXECUTION (-1)
The changes here are dense and subtle, but hopefully all is more explicit
than before.
- CConnman is now in charge of sending data rather than the nodes themselves.
This is necessary because many decisions need to be made with all nodes in
mind, and a model that requires the nodes calling up to their manager quickly
turns to spaghetti.
- The per-node-serializer (ssSend) has been replaced with a (quasi-)const
send-version. Since the send version for serialization can only change once
per connection, we now explicitly tag messages with INIT_PROTO_VERSION if
they are sent before the handshake. With this done, there's no need to lock
for access to nSendVersion.
Also, a new stream is used for each message, so there's no need to lock
during the serialization process.
- This takes care of accounting for optimistic sends, so the
nOptimisticBytesWritten hack can be removed.
- -dropmessagestest and -fuzzmessagestest have not been preserved, as I suspect
they haven't been used in years.
This allows future software that would relay compact blocks before
full validation to announce only to peers that will not ban if the
block turns out to be invalid.
f5b960b Move nTimeBestReceived updating into net processing code (Matt Corallo)
d8670fb Move all calls to CheckBlockIndex out of net-processing logic (Matt Corallo)
d6ea737 Remove network state wipe from UnloadBlockIndex. (Matt Corallo)
fc0c24f Move MarkBlockAsReceived out of ProcessNewMessage (Matt Corallo)
65f35eb Move FlushStateToDisk call out of ProcessMessages::TX into ATMP (Matt Corallo)
59ac5c5 net: Use deterministic randomness for CNode's nonce, and make it const (Cory Fields)
aff6584 net: constify a few CNode vars to indicate that they're threadsafe (Cory Fields)
Check for unreasonable alloc size in LockedPool rather than lancing through new
Arenas until we improbably find one worthy of the quixotic request or the system
can support no more Arenas.
- Use the python standard logging library
- Run all tests and report all failing test-cases (rather than stop after one test case fails)
- If output is different from expected output, log a contextual diff.
Refer to the right file in the top-level README.md.
Having only one file with test documentation saves some confusion about
where things are documented.
444c673 bench: Add benchmark for lockedpool allocation/deallocation (Wladimir J. van der Laan)
6567999 rpc: Add `getmemoryinfo` call (Wladimir J. van der Laan)
4536148 support: Add LockedPool (Wladimir J. van der Laan)
f4d1fc2 wallet: Get rid of LockObject and UnlockObject calls in key.h (Wladimir J. van der Laan)
999e4c9 wallet: Change CCrypter to use vectors with secure allocator (Wladimir J. van der Laan)
GetTotalBlocksEstimate is no longer used and it was the only thing
the checkpoint tests were testing.
Since checkpoints are on their way out it makes more sense to remove
the test file than to cook up a new pointless test.
This introduces a 'minimum chain work' chainparam which is intended
to be the known amount of work in the chain for the network at the
time of software release. If you don't have this much work, you're
not yet caught up.
This is used instead of the count of blocks test from checkpoints.
This criteria is trivial to keep updated as there is no element of
subjectivity, trust, or position dependence to it. It is also a more
reliable metric of sync status than a block count.
Fixes newly initialized bloom filters being
constructed with isEmpty(false), which still
works but loses the possible speedup when
checking for key membership in an empty filter.
This will result in many more calls to CheckBlockIndex when
connecting a list of headers (eg in ::HEADERS messages processing)
but its only enabled in debug mode, and that should mostly just be
during IBD, so it should be OK.
UnloadBlockIndex is only used during init if we end up reindexing
to clear our block state so that we can start over. However, at
that time no connections have been brought up as CConnman hasn't
been started yet, so all of the network processing state logic is
empty when its called.
Additionally, the initialization of the recentRejects set is moved
to InitPeerLogic.
This splits the output comparison for `bitcoin-tx` into two steps:
- First, check for data mismatch, parsing the data as json or hex
depending on the extension of the output file
- Then, check if the literal string matches
For either of these cases give a different error.
This prevents wild goose chases when e.g. a trailing space doesn't match
exactly, and makes sure that both test output and examples are valid
data of the purported format.
Recent discussion (in IRC meetings, and e.g. #8989) has shown a
preference for the default confirm target for smartfees to be 6 instead
of 2, to avoid overpaying fees for questionable gain.
6 is also a compromise between the GUI's pre-#8989 value of 25 and the
bitcoind `-txconfirmtarget` default of 2. These were unified in #8989,
but this has made the (overly expensive) default of 2 as GUI default.
```
getmemoryinfo
Returns an object containing information about memory usage.
Result:
{
"locked": { (json object) Information about locked memory manager
"used": xxxxx, (numeric) Number of bytes used
"free": xxxxx, (numeric) Number of bytes available in current arenas
"total": xxxxxxx, (numeric) Total number of bytes managed
"locked": xxxxxx, (numeric) Amount of bytes that succeeded locking. If this number is smaller than total, locking pages failed at some point and key data could be swapped to disk.
}
}
Examples:
> bitcoin-cli getmemoryinfo
> curl --user myusername --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "getmemoryinfo", "params": [] }' -H 'content-type: text/plain;' http://127.0.0.1:8332/
```
Add a pool for locked memory chunks, replacing LockedPageManager.
This is something I've been wanting to do for a long time. The current
approach of locking objects where they happen to be on the stack or heap
in-place causes a lot of mlock/munlock system call overhead, slowing
down any handling of keys.
Also locked memory is a limited resource on many operating systems (and
using a lot of it bogs down the system), so the previous approach of
locking every page that may contain any key information (but also other
information) is wasteful.
This change is needed to prevent sync_blocks timeouts in the mempool_reorg
test after the sync_blocks update in the upcoming commit
"[qa] Change sync_blocks to pick smarter maxheight".
This change was initially suggested by Suhas Daftuar <sdaftuar@chaincode.com>
in https://github.com/bitcoin/bitcoin/pull/8680#r78209060
0334430 Add some missing includes (Pieter Wuille)
4100499 Return shared_ptr<CTransaction> from mempool removes (Pieter Wuille)
51f2783 Make removed and conflicted arguments optional to remove (Pieter Wuille)
f48211b Bypass removeRecursive in removeForReorg (Pieter Wuille)
nMaxInbound might very well be 0 or -1, if the user prefers to keep
a small number of maxconnections.
Note: nMaxInbound of -1 means that the user set maxconnections
to 8 or less, but we still want to keep an additional slot for
the feeler connection.
Add getNetworkActive()/setNetworkActive() method to client model.
Send network active status through NotifyNetworkActiveChanged.
Indicate in tool tip of gui status bar network indicator whether network activity is disabled.
Indicate in debug window whether network activity is disabled and add button to allow user to toggle network activity state.
Added the function SetNetworkActive() which when called with argument set to false disconnects all nodes and sets the flag fNetworkActive to false. As long as this flag is false no new connections are attempted and no incoming connections are accepted. Network activity is reenabled by calling the function with argument true.
Note that this is not a major issue as, in order for the missing
lock to cause issues, you have to receive a GETBLOCKTXN message
while reindexing, adding a block header via RPC, etc, which results
in either a table rehash or an insert into the bucket which you are
currently looking at.
3154d6e [Qt] use NotifyHeaderTip's height and date for the progress update (Jonas Schnelli)
0a261b6 Use pindexBestHeader instead of setBlockIndexCandidates for NotifyHeaderTip() (Jonas Schnelli)
Makes it an error to use flags that have not been defined
on the libconsensus API.
There has been some confusion as to what pass to libconsensus, and
(combined with mention in the release notes) this should clear it up.
Using undocumented flags is a risk because their meaning,
and what combinations are allowed, changes from release to release.
E.g. it is no longer possible to pass (CLEANSTACK | P2SH) without
running into an assertion after the segwit changes.
There were discrepancies between usage of "block chain" and "blockchain", I've changed them to the latter. The reason for this was that Wikipedia when describing this data structure writes "A blockchain — *originally block chain*", so it seemed the more appropriate term.
