Previous refactorings broke the ability to rebuild the chainstate by deleting the chainstate
directory, resulting in an incorrect "Incorrect or no genesis block found" error message. Fix
that.
Also, improve the performance of ActivateBestBlockStep by using the skiplist to only discover
a few potential blocks to connect at a time, instead of all blocks forever - as we likely bail
out after connecting a single one anyway.
Instead of skipping to the last reindexed block in each file (which could
jump over processed out-of-order blocks), just skip each already processed
block individually.
Remember out-of-order block headers along with disk positions. This is
likely the simplest and least-impact way to make -reindex work with
headers first.
Based on top of #4468.
Many changes:
* Do not use 'getblocks', but 'getheaders', and use it to build a headers tree.
* Blocks are fetched in parallel from all available outbound peers, using a
limited moving window. When one peer stalls the movement of the window, it is
disconnected.
* No more orphan blocks. At all. We only ever request a block for which we have
verified the headers, and store it to disk immediately. This means that a
disk-fill attack would require PoW.
* Require protocol version 31800 for every peer (released in december 2010).
* No more syncnode (we sync from everyone we can, though limited to 1 during
initial *headers* sync).
* Introduce some extra named constants, comments and asserts.
This adds a -regetest-only undocumented (for regression testing only)
command-line option -blockversion=N to set block.nVersion.
Adds to the "has the rest of the network upgraded to a
block.nVersion we don't understand" code so it calls
-alertnotify when 51 of the last 100 blocks are up-version.
But it only alerts once, not with every subsequent new, upversion
block.
And adds a forknotify.py regression test to make sure it works.
Tested using forknotify.py:
Before adding CAlert::Notify, get:
Assertion failed: -alertnotify did not warn of up-version blocks
Before adding code to only alert once:
Assertion failed: -alertnotify excessive warning of up-version blocks
After final code in this pull:
Tests successful
7c70438 Get rid of the dummy CCoinsViewCache constructor arg (Pieter Wuille)
ed27e53 Add coins_tests with a large randomized CCoinViewCache test. (Pieter Wuille)
058b08c Do not keep fully spent but unwritten CCoins entries cached. (Pieter Wuille)
c9d1a81 Get rid of CCoinsView's SetCoins and SetBestBlock. (Pieter Wuille)
f28aec0 Use ModifyCoins instead of mutable GetCoins. (Pieter Wuille)
e790c37 Replace SCRIPT_VERIFY_NOCACHE by flag directly to checker (Pieter Wuille)
5c1e798 Make signature cache optional (Pieter Wuille)
c7829ea Abstract out SignatureChecker (Pieter Wuille)
938bcce CAutoFile: make file private (Philip Kaufmann)
0c35486 CBufferedFile: add explicit close function (Philip Kaufmann)
c9fb27d CBufferedFile: convert into a non-refcounted RAII wrapper (Philip Kaufmann)
There is only one message passed to AbortNode() that makes sense to
translate to the user specifically: Disk space is low. For the others
show a generic message and refer to debug.log for details.
Reduces the number of confusing jargon translation messages.
- it now takes over the passed file descriptor and closes it in the
destructor
- this fixes a leak in LoadExternalBlockFile(), where an exception could
cause the file to not getting closed
- disallow copies (like recently added for CAutoFile)
- make nType and nVersion private
4705902 Avoid introducing a virtual into CChainParams (Wladimir J. van der Laan)
5e2e7fc Suggested corrections on comments, variable names. Also new test case testing the PoW skip in UNITTEST. (SergioDemianLerner)
a25fd6b Switch testing framework from MAIN to new UNITTEST network (SergioDemianLerner)
f74fc9b Print input index when signature validation fails, to aid debugging. (Mark Friedenbach)
217a5c9 When transaction outputs exceed inputs, show the offending amounts so as to aid debugging. (Mark Friedenbach)
Move the txid duplicates check into BuildMerkleTree, where it can be done
much more efficiently (without needing to build a full txid set to detect
duplicates).
The previous version (using the std::set<uint256> to detect duplicates) was
also slightly too weak. A block mined with actual duplicate transactions
(which is invalid, due to the inputs of the duplicated transactions being
seen as double spends) would trigger the duplicates logic, resulting in the
block not being stored on disk, and rerequested. This change fixes that by
only triggering in the case of duplicated transactions that can actually
result in an identical merkle root.
Instead of storing CCoins entries directly in CCoinsMap, store a CCoinsCacheEntry
which additionally keeps track of whether a particular entry is:
* dirty: potentially different from its parent view.
* fresh: the parent view is known to not have a non-pruned version.
This allows us to skip non-dirty cache entries when pushing batches of changes up,
and to remove CCoins entries about transactions that are fully spent before the
parent cache learns about them.
All direct modifications are now done through ModifyCoins, and BatchWrite is
used for pushing batches of queued modifications up, so we don't need the
low-level SetCoins and SetBestBlock anymore in the top-level CCoinsView class.
Replace the mutable non-copying GetCoins method with a ModifyCoins, which
returns an encapsulated iterator, so we can keep track of concurrent
modifications (as iterators can be invalidated by those) and run cleanup
code after a modification is finished.
This also removes the overloading of the 'GetCoins' name.
There is no reason to store thousands of orphan transactions;
normally an orphan's parents will either be broadcast or
mined reasonably quickly.
This pull drops the maximum number of orphans from 10,000 down
to 100, and adds a command-line option (-maxorphantx) that is
just like -maxorphanblocks to override the default.
Prevent denial-of-service attacks by banning
peers that send us invalid orphan transactions
and only storing orphan transactions given to
us by a peer while the peer is connected.
2c2cc5d Remove some unnecessary c_strs() in logging and the GUI (Philip Kaufmann)
f7d0a86 netbase: Use .data() instead of .c_str() on binary string (Wladimir J. van der Laan)
The efficient version of CCoinsViewCache::GetCoins only works for known-to-exist
cache entries, requiring a separate HaveCoins call beforehand. This is
inefficient as both perform a hashtable lookup.
Replace the non-mutable GetCoins with AccessCoins, which returns a potentially-NULL
pointer. This also decreases the overloading of GetCoins.
Also replace some copying (inefficient) GetCoins calls with equivalent AccessCoins,
decreasing the copying.
Bypassing the main coins cache allows more thorough checking with the same
memory budget.
This has no effect on performance because everything ends up in the child
cache created by VerifyDB itself.
It has bugged me ever since #4675, which effectively reduced the
number of checked blocks to reduce peak memory usage.
- Pass the coinsview to use as argument to VerifyDB
- This also avoids that the first `pcoinsTip->Flush()` after VerifyDB
writes a large slew of unchanged coin records back to the database.
ad49c25 Split up util.cpp/h (Wladimir J. van der Laan)
f841aa2 Move `COIN` and `CENT` to core.h (Wladimir J. van der Laan)
6e5fd00 Move `*Version()` functions to version.h/cpp (Wladimir J. van der Laan)
b4aa769 Move `S_I*` constants and `MSG_NOSIGNAL` to compat.h (Wladimir J. van der Laan)
af8297c Move functions in wallet.h to implementation file (Wladimir J. van der Laan)
651480c move functions in main and net to implementation files (Wladimir J. van der Laan)
610a8c0 Move SetThreadPriority implementation to util.cpp instead of the header (Wladimir J. van der Laan)
f780e65 Remove unused function `ByteReverse` from util.h (Wladimir J. van der Laan)
121d6ad Remove unused `alignup` function from util.h (Wladimir J. van der Laan)
d1e26d4 Move CMedianFilter to timedata.cpp (Wladimir J. van der Laan)
Split up util.cpp/h into:
- string utilities (hex, base32, base64): no internal dependencies, no dependency on boost (apart from foreach)
- money utilities (parsesmoney, formatmoney)
- time utilities (gettime*, sleep, format date):
- and the rest (logging, argument parsing, config file parsing)
The latter is basically the environment and OS handling,
and is stripped of all utility functions, so we may want to
rename it to something else than util.cpp/h for clarity (Matt suggested
osinterface).
Breaks dependency of sha256.cpp on all the things pulled in by util.
Due to growing coinsviewcaches, the memory usage with checklevel=3
(and standard settings for dbcache) could be up to 500MiB on a
64-bit system. This is about twice the peak during reindexing,
unnecessarily extending bitcoind's memory envelope.
This commit reduces the maximum total size of the caches used during
verification to just nCoinCacheSize, which should be the limit.
Remove the 'state' and 'exceptmask' from serialize.h's stream implementations,
as well as related methods.
As exceptmask always included 'failbit', and setstate was always called with
bits = failbit, all it did was immediately raise an exception. Get rid of
those variables, and replace the setstate with direct exception throwing
(which also removes some dead code).
As a result, good() is never reached after a failure (there are only 2
calls, one of which is in tests), and can just be replaced by !eof().
fail(), clear(n) and exceptions() are just never called. Delete them.
The only other method of logging remote addresses is via
-logips=1 -debug=net
which increases the logged activity by 100x or more.
Github-Pull: #4608
Amended-By: Wladimir J. van der Laan <laanwj@gmail.com>
Port over https://github.com/chronokings/huntercoin/pull/19 from
Huntercoin: This implements a new RPC command "getchaintips" that can be
used to find all currently active chain heads. This is similar to the
-printblocktree startup option, but it can be used without restarting
just via the RPC interface on a running daemon.
* Replace -benchmark (and the related fBenchmark) with a regular debug option, -debug=bench.
* Increase coverage and granularity of individual block processing steps.
* Add cummulative times.
First and foremost, this defaults to OFF.
This option lets a node consider such transactions non-standard,
meaning they will not be relayed or mined by default, but other miners
are free to mine these as usual.
4eedf4f make RandAddSeed() use OPENSSL_cleanse() (Philip Kaufmann)
6354935 move rand functions from util to new random.h/.cpp (Philip Kaufmann)
001a53d add GetRandBytes() as wrapper for RAND_bytes() (Philip Kaufmann)
This adds a -whitelist option to specify subnet ranges from which peers
that connect are whitelisted. In addition, there is a -whitebind option
which works like -bind, except peers connecting to it are also
whitelisted (allowing a separate listen port for trusted connections).
Being whitelisted has two effects (for now):
* They are immune to DoS disconnection/banning.
* Transactions they broadcast (which are valid) are always relayed,
even if they were already in the mempool. This means that a node
can function as a gateway for a local network, and that rebroadcasts
from the local network will work as expected.
Whitelisting replaces the magic exemption localhost had for DoS
disconnection (local addresses are still never banned, though), which
implied hidden service connects (from a localhost Tor node) were
incorrectly immune to DoS disconnection as well. This old
behaviour is removed for that reason, but can be restored using
-whitelist=127.0.0.1 or -whitelist=::1 can be specified. -whitebind
is safer to use in case non-trusted localhost connections are expected
(like hidden services).
- add a small wrapper in util around RAND_bytes() and replace with
GetRandBytes() in the code to log errors from calling RAND_bytes()
- remove OpenSSL header rand.h where no longer needed
75f51f2a introduced asynchronous processing for blocks, where reject messages
and DoS scoring could be applied outside of ProcessBlock, because block
validation may happen later.
However, some types of errors are still detected immediately (in particular,
CheckBlock violations), which need acting after ProcessBlock returns.
The wallet now uses the mempool fee estimator with a new
command-line option: -txconfirmtarget (default: 1) instead
of using hard-coded fees or priorities.
A new bitcoind that hasn't seen enough transactions to estimate
will fall back to the old hard-coded minimum priority or
transaction fee.
-paytxfee option overrides -txconfirmtarget.
Relaying and mining code isn't changed.
For Qt, the coin control dialog now uses priority estimates to
label transaction priority (instead of hard-coded constants);
unspent outputs were consistently labeled with a much higher
priority than is justified by the free transactions actually
being accepted into blocks.
I did not implement any GUI for setting -txconfirmtarget; I would
suggest getting rid of the "Pay transaction fee" GUI and replace
it with either "target number of confirmations" or maybe
a "faster confirmation <--> lower fee" slider or select box.
The original comment forgets to account for the script push which will
need an OP_PUSHDATA2 + 2-bytes for the 513 script bytes.
props davecgh
fixes#4224
Allows network wallets and other clients to see transactions that respend
a prevout already spent in an unconfirmed transaction in this node's mempool.
Knowledge of an attempted double-spend is of interest to recipients of the
first spend. In some cases, it will allow these recipients to withhold
goods or services upon being alerted of a double-spend that deprives them
of payment.
As before, respends are not added to the mempool.
Anti-Denial-of-Service-Attack provisions:
- Use a bloom filter to relay only one respend per mempool prevout
- Rate-limit respend relays to a default of 100 thousand bytes/minute
- Define tx2.IsEquivalentTo(tx1): equality when scriptSigs are not considered
- Do not relay these equivalent transactions
Remove an unused variable declaration in txmempool.cpp.
Relax the AreInputsStandard() tests for P2SH transactions --
allow any Script in a P2SH transaction to be relayed/mined,
as long as it has 15 or fewer signature operations.
Rationale: https://gist.github.com/gavinandresen/88be40c141bc67acb247
I don't have an easy way to test this, but the code changes are
straightforward and I've updated the AreInputsStandard unit tests.
... instead of after 30 minutes of no sending, for latency measurement
and keep-alive. Also, disconnect if no reply arrives within 20 minutes,
instead of 90 of inactivity (for peers supporting the 'pong' message).
f0a83fc Use Params().NetworkID() instead of TestNet() from the payment protocol (jtimon)
2871889 net.h was using std namespace through chainparams.h included in protocol.h (jtimon)
c8c52de Replace virtual methods with static attributes, chainparams.h depends on protocol.h instead of the other way around (jtimon)
a3d946e Get rid of TestNet() (jtimon)
6fc0fa6 Add RPCisTestNet chain parameter (jtimon)
cfeb823 Add RequireStandard chain parameter (jtimon)
21913a9 Add AllowMinDifficultyBlocks chain parameter (jtimon)
d754f34 Move majority constants to chainparams (jtimon)
8d26721 Get rid of RegTest() (jtimon)
cb9bd83 Add DefaultCheckMemPool chain parameter (jtimon)
2595b9a Add DefaultMinerThreads chain parameter (jtimon)
bfa9a1a Add MineBlocksOnDemand chain parameter (jtimon)
1712adb Add MiningRequiresPeers chain parameter (jtimon)
New RPC methods: return an estimate of the fee (or priority) a
transaction needs to be likely to confirm in a given number of
blocks.
Mike Hearn created the first version of this method for estimating fees.
It works as follows:
For transactions that took 1 to N (I picked N=25) blocks to confirm,
keep N buckets with at most 100 entries in each recording the
fees-per-kilobyte paid by those transactions.
(separate buckets are kept for transactions that confirmed because
they are high-priority)
The buckets are filled as blocks are found, and are saved/restored
in a new fee_estiamtes.dat file in the data directory.
A few variations on Mike's initial scheme:
To estimate the fee needed for a transaction to confirm in X buckets,
all of the samples in all of the buckets are used and a median of
all of the data is used to make the estimate. For example, imagine
25 buckets each containing the full 100 entries. Those 2,500 samples
are sorted, and the estimate of the fee needed to confirm in the very
next block is the 50'th-highest-fee-entry in that sorted list; the
estimate of the fee needed to confirm in the next two blocks is the
150'th-highest-fee-entry, etc.
That algorithm has the nice property that estimates of how much fee
you need to pay to get confirmed in block N will always be greater
than or equal to the estimate for block N+1. It would clearly be wrong
to say "pay 11 uBTC and you'll get confirmed in 3 blocks, but pay
12 uBTC and it will take LONGER".
A single block will not contribute more than 10 entries to any one
bucket, so a single miner and a large block cannot overwhelm
the estimates.
Use CFeeRate instead of an int64_t for quantities that are
fee-per-size.
Helps prevent unit-conversion mismatches between the wallet,
relaying, and mining code.
f40dbee remove CPubKey::VerifyCompact( ) which is never used (Kamil Domanski)
28b6c1d remove GetMedianTime( ) which is never used (Kamil Domanski)
5bd4adc remove LookupHostNumeric( ) which is never used (Kamil Domanski)
595f691 remove LogException( ) which is never used (Kamil Domanski)
f4057cb remove CTransaction::IsNewerThan which is never used (Kamil Domanski)
0e31e56 remove CWallet::AddReserveKey which is never used (Kamil Domanski)
8c93bf4 LoadBlockIndexDB(): Require block db reindex if any blk*.dat files are missing. (Ashley Holman)
7a0e84d ProcessGetData(): abort if a block file is missing from disk (Ashley Holman)
397668e Deduplicate uint* comparison operator logic (Pieter Wuille)
df9eb5e Move {Get,Set}Compact from bignum to uint256 (Pieter Wuille)
a703150 Add multiplication and division to uint160/uint256 (Pieter Wuille)
4d480c8 Exception instead of assigning 0 in case of wrong vector length (Pieter Wuille)
eb2cbd7 Deduplicate shared code between uint160 and uint256 (Pieter Wuille)
787ee0c Check redeemScript size does not exceed 520 byte limit (Peter Todd)
4d79098 Increase IsStandard() scriptSig length (Peter Todd)
f80cffa Do not trigger a DoS ban if SCRIPT_VERIFY_NULLDUMMY fails (Peter Todd)
6380180 Add rejection of non-null CHECKMULTISIG dummy values (Peter Todd)
29c1749 Let tx (in)valid tests use any SCRIPT_VERIFY flag (Peter Todd)
68f7d1d Create (MANDATORY|STANDARD)_SCRIPT_VERIFY_FLAGS constants (Peter Todd)
Removes the limits on number of pubkeys for P2SH CHECKMULTISIG outputs.
Previously with the 500 byte scriptSig limit there were odd restrictions
where even a 1-of-12 P2SH could be spent in a standard transaction(1),
yet multisig scriptPubKey's requiring more signatures quickly ran out of
scriptSig space.
From a "stuff-data-in-the-blockchain" point of view not much has changed
as with the prior commit now only allowing the dummy value to be null
the newly allowed scriptSig space can only be used for signatures. In
any case, just using more outputs is trivial and doesn't cost much.
1) See 779b519480d8c5346de6e635119c7ee772e97ec872240c45e558f582a37b4b73
Mined by BTC Guild.
Size specifiers are no longer needed now that we use typesafe tinyformat
for string formatting, instead of the system's sprintf.
No functional changes.
This continues the work in #3735.
Generally useless information. Only updates on connect time, not after
that. Peers can easily lie and the median filter is not effective in
preventing that.
In the past it was used for progress display in the GUI but
`CheckPoints::guessVerificationProgress` provides a better way that is now used.
It was too easy to mislead it. Peers do lie about it in practice, see issue #4065.
From the RPC, `getpeerinfo` gives the peer raw values, which are more
useful.
- In wallet and GUI code LOCK cs_main as well as cs_wallet when
necessary
- In main.cpp SendMessages move the TRY_LOCK(cs_main) up, to encompass the call
to IsInitialBlockDownload.
- Make ActivateBestChain, AddToBlockIndex, IsInitialBlockDownload,
InitBlockIndex acquire the cs_main lock
Fixes#3997
All functions that use ChainActive but do not aquire the cs_main
lock themselves, need to be called with the cs_main lock held.
This commit adds assertions to all externally callable functions
that use chainActive or chainMostWork.
This will flag usages when built with -DDEBUG_LOCKORDER.
PrintBlockTree output was broken starting from e010af70.
Everything appears on one line.
PrintWallet() added the newline after a block, but this functionality
was removed and no newline was added.
Seemingly, no one noticed. Add a newline after the block information
to fix this.
This resolves a case in which a mismatch could be used to bloat up the
mempool by sending transactions that pay enough fee to relay, but not
to be mined, with the default policies.
Amend to d5f1e72. It turns out that BerkelyDB was including inttypes.h
indirectly, so we cannot fix this with just macros.
Trivial commit: apply the following script to all .cpp and .h files:
# Middle
sed -i 's/"PRIx64"/x/g' "$1"
sed -i 's/"PRIu64"/u/g' "$1"
sed -i 's/"PRId64"/d/g' "$1"
# Initial
sed -i 's/PRIx64"/"x/g' "$1"
sed -i 's/PRIu64"/"u/g' "$1"
sed -i 's/PRId64"/"d/g' "$1"
# Trailing
sed -i 's/"PRIx64/x"/g' "$1"
sed -i 's/"PRIu64/u"/g' "$1"
sed -i 's/"PRId64/d"/g' "$1"
After this commit, `git grep` for PRI.64 should turn up nothing except
the defines in util.h.
As the tinyformat-based formatting system (introduced in b77dfdc) is
type-safe, no special format characters are needed to specify sizes.
Tinyformat can support (ignore) the C99 prefixes such as "ll" but
chokes on MSVC's inttypes.h defines prefixes such as "I64X". So don't
include inttypes.h and define our own for compatibility.
(an alternative would be to sweep the entire codebase using sed -i to
get rid of the size specifiers but this has less diff impact)
5770254 Copyright header updates s/2013/2014 on files whose last git commit was done in 2014. contrib/devtools/fix-copyright-headers.py script to be able to perform this maintenance task with ease during the rest of the year, every year. Modifications to contrib/devtools/README.md to document what fix-copyright-headers.py does. (gubatron)
Extend CMerkleTx::GetDepthInMainChain with the concept of
a "conflicted" transaction-- a transaction generated by the wallet
that is not in the main chain or in the mempool, and, therefore,
will likely never be confirmed.
GetDepthInMainChain() now returns -1 for conflicted transactions
(0 for unconfirmed-but-in-the-mempool, and >1 for confirmed).
This makes getbalance, getbalance '*', and listunspent all agree when there are
mutated transactions in the wallet.
Before:
listunspent: one 49BTC output
getbalance: 96 BTC (change counted twice)
getbalance '*': 46 BTC (spends counted twice)
After: all agree, 49 BTC available to spend.
contrib/devtools/fix-copyright-headers.py script to be able to perform this maintenance task with ease during the rest of the year, every year. Modifications to contrib/devtools/README.md to document what fix-copyright-headers.py does.
Keep track of which block is being requested (and to be requested) from
each peer, and limit the number of blocks in-flight per peer. In addition,
detect stalled downloads, and disconnect if they persist for too long.
This means blocks are never requested twice, and should eliminate duplicate
downloads during synchronization.
In case the total number of orphan blocks in memory exceeds a limit
(currently set to 750), a random orphan block (which is not
depended on by another orphan block) is dropped. This means it will
need to be downloaded again, but it won't consume memory until then.
c117d9e Support for error messages and a few more rejection reasons (Luke Dashjr)
14e7ffc Use standard BIP 22 rejection reasons where applicable (Luke Dashjr)
This changes the block processing logic from "try to atomically switch
to a new block" to a continuous "(dis)connect a block, aiming for the
assumed best chain".
This means the smallest atomic operations on the chainstate become
individual block connections or disconnections, instead of entire
reorganizations. It may mean that we try to reorganize to one block,
fail, and rereorganize again to the old block. This is slower, but
doesn't require unbounded RAM.
It also means that a ConnectBlock which fails may be no longer called
from the ProcessBlock which knows which node sent it. To deal with that,
a mapBlockSource is kept, and invalid blocks cause asynchronous "reject"
messages and banning (if necessary).
Previously CreateNewBlock() didn't take into account the fact that
IsFinalTx() without any arguments tests if the transaction is considered
final in the *current* block, when both those functions really needed to
know if the transaction would be final in the *next* block.
Additionally the UI had a similar misunderstanding.
Also adds some basic tests to check that CreateNewBlock() is in fact
mining nLockTime-using transactions correctly.
Thanks to Wladimir J. van der Laan for rebase.
After the tinyformat switch sprintf() family functions support passing
actual std::string objects.
Remove unnecessary c_str calls (236 of them) in logging and formatting.
012ca1c LoadWallet: acquire cs_wallet mutex before clearing setKeyPool (Wladimir J. van der Laan)
9569168 Document cs_wallet lock and add AssertLockHeld (Wladimir J. van der Laan)
19a5676 Use mutex pointer instead of name for AssertLockHeld (Wladimir J. van der Laan)
