When responding to a getblocks message, only return inv's as
long as we HAVE_DATA for blocks in the chain, and only for blocks
that we aren't likely to delete in the near future.
Make it possible to opt-out of the centralized alert system by providing
an option `-noalerts` or `-alerts=0`. The default remains unchanged.
This is a gentler form of #6260, in which I went a bit overboard by
removing the alert system completely.
I intend to add this to the GUI options in another pull after this.
The partition checking code was using chainActive timestamps
to detect partitioning; with headers-first syncing, it should use
(and with this pull request, does use) pIndexBestHeader timestamps.
Fixes issue #6251
This prevents an edge case where a block downloaded and pruned
in-between successive calls to FindNextBlocksToDownload could
cause the block to be unnecessarily re-requested.
In some corner cases, it may be possible for recent blocks to end up in
the same block file as much older blocks. Previously, the pruning code
would stop looking for files to remove upon first encountering a file
containing a block that cannot be pruned, now it will keep looking for
candidate files until the target is met and all other criteria are
satisfied.
This can result in a noncontiguous set of block files (by number) on
disk, which is fine except for during some reindex corner cases, so
make reindex preparation smarter such that we keep the data we can
actually use and throw away the rest. This allows pruning to work
correctly while downloading any blocks needed during the reindex.
Previously due to an off-by-one error the wallet ignored
nLockTime-by-height transactions that would be valid in the next block
even though they are accepted into the mempool. The transactions
wouldn't show up until confirmed, nor would they be included in the
unconfirmed balance. Similar to the mempool behavior fix in 665bdd3b,
the wallet code was calling IsFinalTx() directly without taking into
account the fact that doing so tells you if the transaction could have
been mined in the *current* block, rather than the next block.
To fix this we strip IsFinalTx() of non-consensus-critical
functionality, removing the default arguments, and add CheckFinalTx() to
check if a transaction will be final in the next block.
Previously this was cleared only after UnlinkPrunedFiles, but it should really be cleared after FindFilesToPrune, regardless of whether there are any files to be pruned.
86a5f4b Relocate calls to CheckDiskSpace (Alex Morcos)
67708ac Write block index more frequently than cache flushes (Pieter Wuille)
b3ed423 Cache tweak and logging improvements (Pieter Wuille)
fc684ad Use accurate memory for flushing decisions (Pieter Wuille)
046392d Keep track of memory usage in CCoinsViewCache (Pieter Wuille)
540629c Add memusage.h (Pieter Wuille)
Create a monitoring task that counts how many blocks have been found in the last four hours.
If very few or too many have been found, an alert is triggered.
"Very few" and "too many" are set based on a false positive rate of once every fifty years of constant running with constant hashing power, which works out to getting 5 or fewer or 48 or more blocks in four hours (instead of the average of 24).
Only one alert per day is triggered, so if you get disconnected from the network (or are being Sybil'ed) -alertnotify will be triggered after 3.5 hours but you won't get another -alertnotify for 24 hours.
Tested with a new unit test and by running on the main network with -debug=partitioncheck
Run test/test_bitcoin --log_level=message to see the alert messages:
WARNING: check your network connection, 3 blocks received in the last 4 hours (24 expected)
WARNING: abnormally high number of blocks generated, 60 blocks received in the last 4 hours (24 expected)
The -debug=partitioncheck debug.log messages look like:
ThreadPartitionCheck : Found 22 blocks in the last 4 hours
ThreadPartitionCheck : likelihood: 0.0777702
Instead of only checking height to decide whether to disable script checks,
actually check whether a block is an ancestor of a checkpoint, up to which
headers have been validated. This means that we don't have to prevent
accepting a side branch anymore - it will be safe, just less fast to
do.
We still need to prevent being fed a multitude of low-difficulty headers
filling up our memory. The mechanism for that is unchanged for now: once
a checkpoint is reached with headers, no headers chain branching off before
that point are allowed anymore.
This class groups transactions that have been confirmed in blocks into buckets, based on either their fee or their priority. Then for each bucket, the class calculates what percentage of the transactions were confirmed within various numbers of blocks. It does this by keeping an exponentially decaying moving history for each bucket and confirm block count of the percentage of transactions in that bucket that were confirmed within that number of blocks.
-Eliminate txs which didn't have all inputs available at entry from fee/pri calcs
-Add dynamic breakpoints and tracking of confirmation delays in mempool transactions
-Remove old CMinerPolicyEstimator and CBlockAverage code
-New smartfees.py
-Pass a flag to the estimation code, using IsInitialBlockDownload as a proxy for when we are still catching up and we shouldn't be counting how many blocks it takes for transactions to be included.
-Add a policyestimator unit test
We don't want to erase orphans that still have missing inputs, they should still be tracked as orphans. Also, the transaction thats being accepted can't be an orphan otherwise it would have previously been accepted, so doesn't need to be added to the erase queue.
a8cdaf5 checkpoints: move the checkpoints enable boolean into main (Cory Fields)
11982d3 checkpoints: Decouple checkpoints from Params (Cory Fields)
6996823 checkpoints: make checkpoints a member of CChainParams (Cory Fields)
9f13a10 checkpoints: store mapCheckpoints in CCheckpointData rather than a pointer (Cory Fields)
Use a probabilistic bloom filter to keep track of which addresses
we think we have given our peers, instead of a list.
This uses much less memory, at the cost of sometimes failing to
relay an address to a peer-- worst case if the bloom filter happens
to be as full as it gets, 1-in-1,000.
Measured memory usage of a full mruset setAddrKnown: 650Kbytes
Constant memory usage of CRollingBloomFilter addrKnown: 37Kbytes.
This will also help heap fragmentation, because the 37K of storage
is allocated when a CNode is created (when a connection to a peer
is established) and then there is no per-item-remembered memory
allocation.
I plan on testing by restarting a full node with an empty peers.dat,
running a while with -debug=addrman and -debug=net, and making sure
that the 'addr' message traffic out is reasonable.
(suggestions for better tests welcome)
This adds a -prune=N option to bitcoind, which if set to N>0 will enable block
file pruning. When pruning is enabled, block and undo files will be deleted to
try to keep total space used by those files to below the prune target (N, in
MB) specified by the user, subject to some constraints:
- The last 288 blocks on the main chain are always kept (MIN_BLOCKS_TO_KEEP),
- N must be at least 550MB (chosen as a value for the target that could
reasonably be met, with some assumptions about block sizes, orphan rates,
etc; see comment in main.h),
- No blocks are pruned until chainActive is at least 100,000 blocks long (on
mainnet; defined separately for mainnet, testnet, and regtest in chainparams
as nPruneAfterHeight).
This unsets NODE_NETWORK if pruning is enabled.
Also included is an RPC test for pruning (pruning.py).
Thanks to @rdponticelli for earlier work on this feature; this is based in
part off that work.
Compare the block download timeout to what the timeout would be if calculated
based on current time and current value of nQueuedValidatedHeaders, but
ignoring other in-flight blocks from the same peer. If the calculation based on
present conditions is shorter, then set that to be the time after which we
disconnect the peer for not delivering this block.
Some tests in CheckBlockIndex require chainActive.Tip(), but when reindexing, chainActive has not been set on the first call to CheckBlockIndex.
reindex.py starts a node, mines 3 blocks, stops, and reindexes with CheckBlockIndex enabled.
SendMessages will now call getheaders on both inbound and outbound peers,
once the headers chain is close to synced. It will also try downloading
blocks from inbound peers once we're out of initial block download (so
inbound peers will participate in parallel block fetching for the last day
or two of blocks being downloaded).
This adds more tests to CheckBlockIndex:
- HAVE_DATA is true iff nTx > 0
- BLOCK_VALID_TRANSACTIONS is true iff nTx > 0
- BLOCK_VALID_TRANSACTIONS is true for a block and all parents iff
nChainTx > 0
This adds a -checkblockindex (defaulting to true for regtest), which occasionally
does a full consistency check for mapBlockIndex, setBlockIndexCandidates, chainActive, and
mapBlocksUnlinked.
Adds a regression test for the wallet's ResendWalletTransactions function, which uses a new, hidden RPC command "resendwallettransactions."
I refactored main's Broadcast signal so it is passed the best-block time, which let me remove a global variable shared between main.cpp and the wallet (nTimeBestReceived).
I also manually tested the "rebroadcast unconfirmed every half hour or so" functionality by:
1. Running bitcoind -connect=0.0.0.0:8333
2. Creating a couple of send-to-self transactions
3. Connect to a peer using -addnode
4. Waited a while, monitoring debug.log, until I see:
```2015-03-23 18:48:10 ResendWalletTransactions: rebroadcast 2 unconfirmed transactions```
One last change: don't bother putting ResendWalletTransactions messages in debug.log unless unconfirmed transactions were actually rebroadcast.
When re-indexing, there are a few cases where garbage data may be skipped in
the block files. In these cases, the indices are correctly written to the index
db, however the pointer to the next position for writing in the current block
file is calculated by adding the sizes of the valid blocks found.
As a result, when the re-index is finished, the index db is correct for all
existing blocks, but the next block will be written to an incorrect offset,
likely overwriting existing blocks.
Rather than using the sum of all valid blocks to determine the next write
position, use the end of the last block written to the file. Don't assume that
the current block is the last one in the file, since they may be read
out-of-order.
Normally bitcoin core does not display any network originated strings without
sanitizing or hex encoding. This wasn't done for strcommand in many places.
This could be used to play havoc with a terminal displaying the logs,
especially with printtoconsole in use.
Thanks to Evil-Knievel for reporting this issue.
The only time when a client sends a "getaddr" message is when he
esatblishes an Outbound connection (see ProcessMessage() in
src/main.cpp). Another bitcoin client is expected to receive a
"getaddr" message only on Inbound connection. Ignoring "gettaddr"
requests on Outbound connections can resolve potential privacy issues
(and as was said such request normally do not happen anyway).
Instead, create a separate function that applies the undo operation of a
CTxInUndo object onto a CCoinsViewCache. This method is used from
DisconnectBlock.
Note that this will also require translation changes in Transifex for the key
"A fee higher than %1 is considered an insanely high fee." which is now
"A fee higher than %1 is considered an absurdly high fee."
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
This harmonizes the block fetch timeout with the existing ping timeout
and eliminates a guaranteed eventual failure from congestion collapse
for a network operating right at its limit.
It's unlikely that we wouldn't suffer other failures if we were really
anywhere near the network's limit, and a complete avoidance of congestion
collapse risk requires (I think) an exponential back-off. So this isn't
a major concern, but I think it's also useful for reducing the complexity
of understanding out timeouts.
This will disconnect peers that do not transfer a block in 10 minutes, plus
5 minutes for every previously queued block with validated headers
(accomodating downstream bandwidth down to a few kilobytes per second - below
that the node would have trouble staying synchronized anyway).
856e862 namespace: drop most boost namespaces and a few header cleanups (Cory Fields)
9b1ab86 namespace: drop boost::assign altogether here (Cory Fields)
a324199 namespace: remove boost namespace pollution (Cory Fields)
If uint256() constructor takes a string, uint256(0) will become
dangerous when uint256 does not take integers anymore (it will go
through std::string(const char*) making a NULL string, and the explicit
keyword is no help).
Previous behavior with IsFinalTx() being an IsStandard() rule was rather
confusing and interferred with testing of protocols that depended on
nLockTime.
With the splashscreen being able to be closed it is possible to
shutdown during the lengthy verifyDB method. (Takes about a minute
on my machine). This change allows us to shutdown much sooner.
Github-Pull: #5557
1b178a7 Bugfix: ConnectBlock: In case the genesis block gets in with fJustCheck, behave correctly (Luke Dashjr)
228d238 Make CCoinsViewCache's copy constructor private (Luke Dashjr)
Don't allow immediate inv driven block downloads if
a peer already has MAX_BLOCKS_IN_TRANSIT_PER_PEER
active downloads. Prevents bogus inv spam from
blowing up block transfer tracking data structures.
This still leaves transactions in mempool that are potentially
invalid if the maturity period has been reorged out of, but at
least they're not missing inputs entirely.
9ec75c5 Add a locking mechanism to IsInitialBlockDownload to ensure it never goes from false to true. (Ruben Dario Ponticelli)
a2d0fc6 Fix IsInitialBlockDownload which was broken by headers first. (Ruben Dario Ponticelli)
There are 3 pieces of data that are maintained on disk. The actual block
and undo data, the block index (which can refer to positions on disk),
and the chainstate (which refers to the best block hash).
Earlier, there was no guarantee that blocks were written to disk before
block index entries referring to them were written. This commit introduces
dirty flags for block index data, and delays writing entries until the actual
block data is flushed.
With this stricter ordering in writes, it is now safe to not always flush
after every block, so there is no need for the IsInitialBlockDownload()
check there - instead we just write whenever enough time has passed or
the cache size grows too large. Also updating the wallet's best known block
is delayed until this is done, otherwise the wallet may end up referring to an
unknown block.
In addition, only do a write inside the block processing loop if necessary
(because of cache size exceeded). Otherwise, move the writing to a point
after processing is done, after relaying.
Like in a real world situation, a safe mode test should also be visible in the
UI. A test of safe mode is furthermore mostly relevant for developers, so it
should not be overwritten by a warning about a pre-release test build.
Previously, AcceptBlockHeader did not check the header (in particular
PoW). This made the client accept invalid-PoW-headers from peers in
headers-first sync.
Previously transactions were only tested again the
STANDARD_SCRIPT_VERIFY_FLAGS prior to mempool acceptance, so any bugs in
those flags that allowed actually-invalid transactions to pass would
result in allowing invalid transactions into the mempool. Fortunately
there is a second check in CreateNewBlock() that would prevent those
transactions from being mined, resulting in an invalid block, however
this could still be exploited as a DoS attack.
This is a simplified re-do of closed pull #3088.
This patch eliminates the privacy and reliability problematic use
of centralized web services for discovering the node's addresses
for advertisement.
The Bitcoin protocol already allows your peers to tell you what
IP they think you have, but this data isn't trustworthy since
they could lie. So the challenge is using it without creating a
DOS vector.
To accomplish this we adopt an approach similar to the one used
by P2Pool: If we're announcing and don't have a better address
discovered (e.g. via UPNP) or configured we just announce to
each peer the address that peer told us. Since peers could
already replace, forge, or drop our address messages this cannot
create a new vulnerability... but if even one of our peers is
giving us a good address we'll eventually make a useful
advertisement.
We also may randomly use the peer-provided address for the
daily rebroadcast even if we otherwise have a seemingly routable
address, just in case we've been misconfigured (e.g. by UPNP).
To avoid privacy problems, we only do these things if discovery
is enabled.
50b43fd Be a bit more verbose during -loadblock if we already have blocks (Matt Corallo)
8375e22 Fix -loadblock after shutdown during IBD (Matt Corallo)
4ead850 Fix for crash during block download (Matt Corallo)
1bea2bb Rename ProcessBlock to ProcessNewBlock to indicate change of behaviour, and document it (Luke Dashjr)
d29a291 Rename RPC_TRANSACTION_* errors to RPC_VERIFY_* and use RPC_VERIFY_ERROR for submitblock (Luke Dashjr)
f877aaa Bugfix: submitblock: Use a temporary CValidationState to determine accurately the outcome of ProcessBlock, now that it no longer does the full block validity check (Luke Dashjr)
24e8896 Add CValidationInterface::BlockChecked notification (Luke Dashjr)
a873823 CAutoFile: Explicit Get() and remove unused methods (Wladimir J. van der Laan)
fef24ca Add IsNull() to class CAutoFile and remove operator ! (Ruben Dario Ponticeli)
