Amend to d5f1e72. It turns out that BerkelyDB was including inttypes.h
indirectly, so we cannot fix this with just macros.
Trivial commit: apply the following script to all .cpp and .h files:
# Middle
sed -i 's/"PRIx64"/x/g' "$1"
sed -i 's/"PRIu64"/u/g' "$1"
sed -i 's/"PRId64"/d/g' "$1"
# Initial
sed -i 's/PRIx64"/"x/g' "$1"
sed -i 's/PRIu64"/"u/g' "$1"
sed -i 's/PRId64"/"d/g' "$1"
# Trailing
sed -i 's/"PRIx64/x"/g' "$1"
sed -i 's/"PRIu64/u"/g' "$1"
sed -i 's/"PRId64/d"/g' "$1"
After this commit, `git grep` for PRI.64 should turn up nothing except
the defines in util.h.
As the tinyformat-based formatting system (introduced in b77dfdc) is
type-safe, no special format characters are needed to specify sizes.
Tinyformat can support (ignore) the C99 prefixes such as "ll" but
chokes on MSVC's inttypes.h defines prefixes such as "I64X". So don't
include inttypes.h and define our own for compatibility.
(an alternative would be to sweep the entire codebase using sed -i to
get rid of the size specifiers but this has less diff impact)
5770254 Copyright header updates s/2013/2014 on files whose last git commit was done in 2014. contrib/devtools/fix-copyright-headers.py script to be able to perform this maintenance task with ease during the rest of the year, every year. Modifications to contrib/devtools/README.md to document what fix-copyright-headers.py does. (gubatron)
Extend CMerkleTx::GetDepthInMainChain with the concept of
a "conflicted" transaction-- a transaction generated by the wallet
that is not in the main chain or in the mempool, and, therefore,
will likely never be confirmed.
GetDepthInMainChain() now returns -1 for conflicted transactions
(0 for unconfirmed-but-in-the-mempool, and >1 for confirmed).
This makes getbalance, getbalance '*', and listunspent all agree when there are
mutated transactions in the wallet.
Before:
listunspent: one 49BTC output
getbalance: 96 BTC (change counted twice)
getbalance '*': 46 BTC (spends counted twice)
After: all agree, 49 BTC available to spend.
contrib/devtools/fix-copyright-headers.py script to be able to perform this maintenance task with ease during the rest of the year, every year. Modifications to contrib/devtools/README.md to document what fix-copyright-headers.py does.
Keep track of which block is being requested (and to be requested) from
each peer, and limit the number of blocks in-flight per peer. In addition,
detect stalled downloads, and disconnect if they persist for too long.
This means blocks are never requested twice, and should eliminate duplicate
downloads during synchronization.
In case the total number of orphan blocks in memory exceeds a limit
(currently set to 750), a random orphan block (which is not
depended on by another orphan block) is dropped. This means it will
need to be downloaded again, but it won't consume memory until then.
c117d9e Support for error messages and a few more rejection reasons (Luke Dashjr)
14e7ffc Use standard BIP 22 rejection reasons where applicable (Luke Dashjr)
This changes the block processing logic from "try to atomically switch
to a new block" to a continuous "(dis)connect a block, aiming for the
assumed best chain".
This means the smallest atomic operations on the chainstate become
individual block connections or disconnections, instead of entire
reorganizations. It may mean that we try to reorganize to one block,
fail, and rereorganize again to the old block. This is slower, but
doesn't require unbounded RAM.
It also means that a ConnectBlock which fails may be no longer called
from the ProcessBlock which knows which node sent it. To deal with that,
a mapBlockSource is kept, and invalid blocks cause asynchronous "reject"
messages and banning (if necessary).
Previously CreateNewBlock() didn't take into account the fact that
IsFinalTx() without any arguments tests if the transaction is considered
final in the *current* block, when both those functions really needed to
know if the transaction would be final in the *next* block.
Additionally the UI had a similar misunderstanding.
Also adds some basic tests to check that CreateNewBlock() is in fact
mining nLockTime-using transactions correctly.
Thanks to Wladimir J. van der Laan for rebase.
After the tinyformat switch sprintf() family functions support passing
actual std::string objects.
Remove unnecessary c_str calls (236 of them) in logging and formatting.
012ca1c LoadWallet: acquire cs_wallet mutex before clearing setKeyPool (Wladimir J. van der Laan)
9569168 Document cs_wallet lock and add AssertLockHeld (Wladimir J. van der Laan)
19a5676 Use mutex pointer instead of name for AssertLockHeld (Wladimir J. van der Laan)
There were quite a few places where assert() was used with side effects,
making operation with NDEBUG non-functional. This commit fixes all the
cases I know about, but also adds an #error on NDEBUG because the code
is untested without assertions and may still have vulnerabilities if
used without assert.
This dead code can be resurrected from git history if
transaction replacement is ever implemented. Keeping
dead code in the source is a bad idea, because it implies
it was tested and worked at some point, which is not true.
The last fee drop was by 5x (from 50k satoshis to 10k satoshis)
in the 0.8.2 release which was about 6 months ago.
The current fee is (assuming a $500 exchange rate) about 5 dollar
cents. The new fee after this patch is 0.5 cents.
Miners who prefer the higher fees are obviously still able to
use the command line flags to override this setting. Miners who
choose to create smaller blocks will select the highest-fee paying
transactions anyway.
This would hopefully be the last manual adjustment ever required
before floating fees become normal.
Use misc methods of avoiding unnecesary header includes.
Replace int typedefs with int##_t from stdint.h.
Replace PRI64[xdu] with PRI[xdu]64 from inttypes.h.
Normalize QT_VERSION ifs where possible.
Resolve some indirect dependencies as direct ones.
Remove extern declarations from .cpp files.
caca6aa Make some globals in main non-public. (Pieter Wuille)
85eb2ce Do not use the redundant BestInvalidWork record in the block database. (Pieter Wuille)
As block index entries have a flag for marking invalid blocks, the
'best invalid work' information can be derived from there. In addition,
remove the global from main.h
- re-work -debug help message text
- make -debug log every debugging information again (even all categories)
- remove unneeded fDebug checks in front of LogPrint()/qDebug(), as that
check is done in LogPrintf() when category is != NULL (true for all
LogPrint() calls
- remove fDebug ONLY in code which is NOT performance-critical
- harmonize addrman category name
- deprecate -debugnet usage, should be used via -debug=net and remove the
corresponding global
Instead of explicitly testing for the presence of any output, and
dealing with this case specially, just interpret it as an empty
CCoins.
The case previously caught using the HaveCoins check, is now handled
by the generic outs != outsBlock test.
This required some code movement (what was CWalletTx::AcceptToMemoryPool
doing in main?), and adding a few explicit includes that used to be
implicit through init.h.
INIT_PROTO_VERSION is the initial version, after a succesful version/verack it is increased to a negotiated version.
MIN_PEER_PROTO_VERSION could be a different value to disconnect from peers older than a specified version.
Changes the response to the 'mempool' command so that if
the memory pool has more than MAX_INV_SZ transactions (50,000)
it will respond with multiple 'inv' messages.
SendMessages() tries to acquire a cs_main lock now, but this isn't nessecary
for much of its functionality. Move those parts out of the locked section,
so they can always be performed, and we hold cs_main for a shorter time.
This removes a few unused CBlockLocator methods, and moves the
construction and fork-finding logic to CChain (which can do these
more efficiently, as it has a height-indexable chain available).
It also makes CBlockLocator independent from the validation code.
971bb3e Added ping time measurement. New RPC "ping" command to request ping. Implemented "pong" message handler. New "pingtime" field in getpeerinfo, to provide results to user. New "pingwait" field, to show pings still in flight, to better see newly lagging peers. (Josh Lehan)
Changes the maximum size of a free transaction that will be created
from 10,000 bytes to 1,000 bytes.
The idea behind this change is to make the free transaction area
available to a greater number of people; with the default 27K-per-block,
just three very-large very-high-priority transactions could fill the space.
Remove the (relay/mempool) rule that all outputs of free transactions
must be greater than 0.01 XBT. Dust spam is now taken care of by making
dusty outputs non-standard.
New RPC "ping" command to request ping.
Implemented "pong" message handler.
New "pingtime" field in getpeerinfo, to provide results to user.
New "pingwait" field, to show pings still in flight, to better see newly lagging peers.
- rename URL into URI in paymentserver where correct
- add some missing Qt-coding-stuff in paymentserver
- change QSpinBox to QLineEdit as base for BitcoinAmountField in .ui files
(as this is the result when converting the BAF back into base)
- remove some c_str() and replace with QString::fromStdString()
- remove several new-lines
- remove unneeded spaces
- indentation fixes
This also makes negative transaction versions non-standard.
This avoids an issue triggered in block 256818 where transactions with
negative version numbers were incorrectly serialized into the UTXO set.
On restart nodes detect the inconsistency and refuse to start so long as
a block with these transactions is inside the self-consistency check
window, logging "coin database inconsistencies found". The software
recommends reindexing, but reindexing does not correct the problem.
This should be fixed by changing the chainstate serialization, but
working around it seems harmless for now because the version is not
used by any network rule currently.
A patch free workaround is to start with -checklevel=2 which skips
the consistency checks, but the IsStandard change is important for
miners in order to protect unpatched nodes.
There have been several incidents where mainnet experimentation with
raw transactions resulted in insane fees. This is hard to prevent
in the raw transaction api because the inputs may not be known.
Since sending doesn't work if the inputs aren't known, we can catch
it there.
This rejects fees > than 10000 * nMinRelayTxFee or 1 BTC with the
defaults and can be overridden with a bool at the rpc.
We're not seeing large reorgs that would justify waiting a large
amount past the rule required maturity, and the extra three
hours is just a nuisance. Take one more block to at least give
the 100th block time to propagate.
This reduces a peer's ability to attack network resources by
using a full bloom filter, but without reducing the usability
of bloom filters. It sets a default match everything filter
for peers and it generalizes a prior optimization to
cover more cases.
The length of vectors, maps, sets, etc are serialized using
Write/ReadCompactSize -- which, unfortunately, do not use a
unique encoding.
So deserializing and then re-serializing a transaction (for example)
can give you different bits than you started with. That doesn't
cause any problems that we are aware of, but it is exactly the type
of subtle mismatch that can lead to exploits.
With this pull, reading a non-canonical CompactSize throws an
exception, which means nodes will ignore 'tx' or 'block' or
other messages that are not properly encoded.
Please check my logic... but this change is safe with respect to
causing a network split. Old clients that receive
non-canonically-encoded transactions or blocks deserialize
them into CTransaction/CBlock structures in memory, and then
re-serialize them before relaying them to peers.
And please check my logic with respect to causing a blockchain
split: there are no CompactSize fields in the block header, so
the block hash is always canonical. The merkle root in the block
header is computed on a vector<CTransaction>, so
any non-canonical encoding of the transactions in 'tx' or 'block'
messages is erased as they are read into memory by old clients,
and does not affect the block hash. And, as noted above, old
clients re-serialize (with canonical encoding) 'tx' and 'block'
messages before relaying to peers.
Orphan transactions were stored as a CDataStream pointer;
this changes the mapOrphanTransactions data structures to
store orphans as a CTransaction.
This also fixes CVE-2013-4627 by always re-serializing
transactions before relaying them.
The new class is accessed via the Params() method and holds
most things that vary between main, test and regtest networks.
The regtest mode has two purposes, one is to run the
bitcoind/bitcoinj comparison tool which compares two separate
implementations of the Bitcoin protocol looking for divergence.
The other is that when run, you get a local node which can mine
a single block instantly, which is highly convenient for testing
apps during development as there's no need to wait 10 minutes for
a block on the testnet.
This (nearly) doesn't change fee rules at all:
* To make it into the fee transaction area, the dPriority comparison
changed from < to <=
* We now just ignore transactions > MAX_BLOCK_SIZE/4 instead of
doing some calculations to require increasingly large fees as
size increases.
Removed AreInputsStandard from CTransaction, made it a regular function in main.
Moved CTransaction::GetOutputFor to CCoinsViewCache.
Moved GetLegacySigOpCount and GetP2SHSigOpCount out of CTransaction into regular functions in main.
Moved GetValueIn and HaveInputs from CTransaction into CCoinsViewCache.
Moved AllowFree, ClientCheckInputs, CheckInputs, UpdateCoins, and CheckTransaction out of CTransaction and into main.
Moved IsStandard and IsFinal out of CTransaction and put them in main as IsStandardTx and IsFinalTx. Moved GetValueOut out of CTransaction into main. Moved CTxIn, CTxOut, and CTransaction into core.
Added minimum fee parameter to CTxOut::IsDust() temporarily until CTransaction is moved to core.h so that CTxOut needn't know about CTransaction.
- explicitly set the default of all GetBoolArg() calls
- rework getarg_test.cpp and util_tests.cpp to cover this change
- some indentation fixes
- move macdockiconhandler.h include in bitcoin.cpp to the "our headers"
section
Write bestblock records in wallets:
* Every 20160 blocks synced, no matter what (before: none during IBD)
* Every 144 blocks after IBD (before: for every block, slow)
* When creating a new wallet
* At shutdown
This should result in far fewer spurious rescans.
Remove the pnext pointer in CBlockIndex, and replace it with a
vBlockIndexByHeight vector (no effect on memory usage). pnext can
now be replaced by vBlockIndexByHeight[nHeight+1], but
FindBlockByHeight becomes constant-time.
This also means the entire mapBlockIndex structure and the block
index entries in it become purely blocktree-related data, and
independent from the currently active chain, potentially allowing
them to be protected by separate mutexes in the future.
Every block index entry currently requires a separately-allocated
CBigNum. By replacing them with uint256, it's just 32 bytes extra
in CBlockIndex itself.
This should save us a few megabytes in RAM, and less allocation
overhead.
This introduces the concept of the 'sync node', which is the one we
asked for missing blocks. In case the sync node goes away, a new one
will be selected.
For now, the heuristic is very simple, but it can easily be extended
later to add better policies.
As these were not updated when 'backporting' the 225430 checkpoint
into head.
Additionally, also report verification progress in debug.log, and
tweak the sigcheck-verification-speed-factor a bit.
Two reasons for this change:
1. Need to always use boost::thread's sleep, even on Windows, so the
sleeps can be interrupted (prior code used Windows' built-in Sleep).
2. I always forgot what units the old Sleep took.
Create a boost::thread_group object at the qt/bitcoind main-loop level
that will hold pointers to all the main-loop threads.
This will replace the vnThreadsRunning[] array.
For testing, ported the BitcoinMiner threads to use its
own boost::thread_group.
There exists a per-message-processed send buffer overflow protection,
where processing is halted when the send buffer is larger than the
allowed maximum.
This protection does not apply to individual items, however, and
getdata has the potential for causing large amounts of data to be
sent. In case several hundreds of blocks are requested in one getdata,
the send buffer can easily grow 50 megabytes above the send buffer
limit.
This commit breaks up the processing of getdata requests, remembering
them inside a CNode when too many are requested at once.
* Change CNode::vRecvMsg to be a deque instead of a vector (less copying)
* Make sure to acquire cs_vRecvMsg in CNode::CloseSocketDisconnect (as it
may be called without that lock).
Replaces CNode::vRecv buffer with a vector of CNetMessage's. This simplifies
ProcessMessages() and eliminates several redundant data copies.
Overview:
* socket thread now parses incoming message datastream into
header/data components, as encapsulated by CNetMessage
* socket thread adds each CNetMessage to a vector inside CNode
* message thread (ProcessMessages) iterates through CNode's CNetMessage vector
Message parsing is made more strict:
* Socket is disconnected, if message larger than MAX_SIZE
or if CMessageHeader deserialization fails (latter is impossible?).
Previously, code would simply eat garbage data all day long.
* Socket is disconnected, if we fail to find pchMessageStart.
We do not search through garbage, to find pchMessageStart. Each
message must begin precisely after the last message ends.
ProcessMessages() always processes a complete message, and is more efficient:
* buffer is always precisely sized, using CDataStream::resize(),
rather than progressively sized in 64k chunks. More efficient
for large messages like "block".
* whole-buffer memory copy eliminated (vRecv -> vMsg)
* other buffer-shifting memory copies eliminated (vRecv.insert, vRecv.erase)
- remove an unneeded MODAL flag, as MSG_ERROR sets MODAL
- re-order an if-clause in main to have bool checks before a function call
- fix some log messages that used wrong function names
- make a log message use a correct ellipsis
- remove some unneded spaces, brackets and line-breaks
- fix style for adding files in the Qt project
Extremely large transactions with lots of inputs can cost the network
almost as much to process as they cost the sender in fees.
We would never create transactions larger than 100K big; this change
makes transactions larger than 100K non-standard, so they are not
relayed/mined by default. This is most important for miners that might
create blocks larger than 250K big, who could be vulnerable to a
make-your-blocks-so-expensive-to-verify-they-get-orphaned attack.
At least one service that accepted zero-confirmation transactions
was vulnerable because an attacker could send a transaction
with a lock time far in the future, and then have plenty of time in
which to get a double-spend mined (perhaps from a miner who wasn't
on the network when the first transaction was broadcast).
That is a variation on the "Finney attack". We still don't
recommend anybody accept 0-confirmation transactions as final
payment for anything. This change keeps non-final transactions
from appearing in the wallet, and, assuming most of the network
accepts this change, will prevent them from being relayed until
they are final.
* Pass txid's to CCoinsView functions by reference instead of by value
* Add a method to swap CCoins, and use it in some places to avoid a
allocating copy + destruct.
* Optimize CCoinsViewCache::FetchCoins to do only a single search
through the backing map.
This actually simplifies some SPV code, as they can keep track of
a filtered block and its txn before accepting both in one step.
The previous argument was that SPV nodes should handle the txn the
same as any other free txn and then mark them as connected to a
block when they get the filtered block itself. However, it now
appears that SPV nodes will need to put in more effort to verify
loose txn than they would to verify txn in blocks, thus making it
more approriate to send the txn after the filtered block.
By specifying -txindex when initializing the database, a txid-to-diskpos
index is maintained in the blktree database. This database is used to
help answering getrawtransaction() RPC queries, when enabled.
Changing the -txindex value requires a -reindex; the client will abort
at startup if the database and the specified -txindex mismatch.
Note that the default value for fRelayTxes is false, meaning we
now no longer relay tx inv messages before receiving the remote
peer's version message.
Fixes issue #2178 : attacker could penny-flood with invalid-signature
transactions to deduce which addresses belonged to your node.
I'm committing this early for code review; I still need to write up
a test plan.
Executive summary of fix: check all transactions received from the network
for penny-flood rate-limiting before adding to the memory pool. But do NOT
ratelimit transactions added to the memory pool:
- because of blockchain reorgs
- stored in the wallet and added at startup
- sent from the GUI or one of the send* RPC commands (CWallet::CommitTransaction)
The limit-free-transactions code really should be a method on CNode, with
counters per-peer. But that is a bigger change for another day.
Client (SPV) mode never got implemented entirely, and whatever part was already
working, is likely not been tested (or even executed at all) for the past two
years. This removes it entirely.
If we want an SPV implementation, I think we should first get the block chain
data structures to be encapsulated in a class implementing a standard interface,
and then writing an alternate implementation with SPV semantics.
Since block validation happens in parallel, multiple threads may be
accessing the signature cache simultaneously. To prevent contention:
* Turn the signature cache lock into a shared mutex
* Make reading from the cache only acquire a shared lock
* Let block validations not store their results in the cache
* During block verification (when parallelism is requested), script
check actions are stored instead of being executed immediately.
* After every processed transactions, its signature actions are
pushed to a CScriptCheckQueue, which maintains a queue and some
synchronization mechanism.
* Two or more threads (if enabled) start processing elements from
this queue,
* When the block connection code is finished processing transactions,
it joins the worker pool until the queue is empty.
As cs_main is held the entire time, and all verification must be
finished before the block continues processing, this does not reach
the best possible performance. It is a less drastic change than
some more advanced mechanisms (like doing verification out-of-band
entirely, and rolling back blocks when a failure is detected).
The -par=N flag controls the number of threads (1-16). 0 means auto,
and is the default.
- some users reported it as weird, that the estimated block count could be
lower than our own nodes block number (which is indeed true and not good)
- this pull adds a new default behaviour, which displays our own block
number as estimated block number, if own >= est. block count
- the pull raises space for nodes block counts in cPeerBlockCounts to 8 to
be more accurate
- also removes a reduntant setNumBlocks() call in RPCConsole and moves
initialisation of numBlocksAtStartup in ClientModel, where it belongs
-checklevel gets a new meaning:
0: verify blocks can be read from disk (like before)
1: verify (contextless) block validity (like before)
2: verify undo files can be read and have good checksums
3: verify coin database is consistent with the last few blocks
(close to level 6 before)
4: verify all validity rules of the last few blocks
Level 3 is the new default, as it's reasonably fast. As level 3 and
4 are implemented using an in-memory rollback of the database, they
are limited to as many blocks as possible without exceeding the
limits set by -dbcache. The default of -dbcache=25 allows for some
150-200 blocks to be rolled back.
In case an error is found, the application quits with a message
instructing the user to restart with -reindex. Better instructions,
and automatic recovery (when possible) or automatic reindexing are
left as future work.
When the coin database is out of date with the block database, the
best block in it is automatically switched to. This reconnection
process can take time, so allow it to be interrupted.
This also stops block connection as soon as shutdown is requested,
leading to a faster shutdown.
This problem is like earth (mostly harmless). After/during a
-reindex, it means the statistics about the last block file
reported in debug.log are always of blk00000.dat instead of the
last file. Apart from that, it means a few more database entries
need to be read when finding a file to append to the first time.
- even if we are allowed to fail pre-allocating, it's better to check
for sufficient space before calling AllocateFileRange() and if we
are out of disk space return with error()
- the above change allows us to remove the CheckDiskSpace() check
in CBlock::AcceptBlock()
In case a reorganisation fails, the internal state could become
inconsistent (memory only). Previously, a cache per block connect
or disconnect action was used, so blocks could not be applied in
a partial way. Extend this to a cache for the entire reorganisation,
making it atomic entirely. This also simplifies the code a bit.
- fix ThreadSafeMessageBox always displays error icon
- allow to specify MSG_ERROR / MSG_WARNING or MSG_INFORMATION without a
custom caption / title
- allow to specify CClientUIInterface::ICON_ERROR / ICON_WARNING and
ICON_INFORMATION (which is default) as message box icon
- remove CClientUIInterface::OK from ThreadSafeMessageBox-calls, as
the OK button will be set as default, if none is specified
- prepend "Bitcoin - " to used captions
- rename BitcoinGUI::error() -> BitcoinGUI::message() and add function
documentation
- change all style parameters and enum flags to unsigned
- update code to use that new API
- update Client- and WalletModel to use new BitcoinGUI::message() and
rename the classes error() method into message()
- include the possibility to supply the wanted icon for messages from
Client- and WalletModel via "style" parameter
When a transaction A is in the memory pool, while a transaction B
(which shares an input with A) gets accepted into a block, A was
kept forever in the memory pool.
This commit adds a CTxMemPool::removeConflicts method, which
removes transactions that conflict with a given transaction, and
all their children.
This results in less transactions in the memory pool, and faster
construction of new blocks.
These flags select features to be enabled/disabled during script
evaluation/checking, instead of several booleans passed along.
Currently these flags are defined:
* SCRIPT_VERIFY_P2SH: enable BIP16-style subscript evaluation
* SCRIPT_VERIFY_STRICTENC: enforce strict adherence to pubkey/sig encoding standards.
- remove an unwanted ";" at the end of the ~CCoinsView() destructor
- in FindBlockPos() and FindUndoPos() only call fclose(), is file is open
- fix an error string in the CBlockUndo class
Flushes the blktree/ and coins/ databases, and reindexes the
block chain files, as if their contents was loaded via -loadblock.
Based on earlier work by Jeff Garzik.
Implements #1948
- Add macro `CLIENT_VERSION_IS_RELEASE` to clientversion.h
- When running a prerelease (the above macro is `false`):
- In UI, show an orange warning bar at the top. This will be used for other
warnings (and alerts) as well, instead of the status bar.
- For `bitcoind`, show the warning in the "errors" field in `getinfo`
response.
CreateNewBlock was reading pindexBest at the start before taking the lock
so it was possible to have the the block content not match the prevheader
and this can also trigger a newly added assert in ConnectBlock.
I noticed this during a code review after twobitcoins reported that ab91bf39
(BIP30 for all blocks) could cause a null dereference on a modified node
that mined during the IBD, or on testnet when it reached heights 91842 and
91880 due to CreateNewBlock calling ConnectBlock with pindex->phashBlock NULL.
- remove uiInterface.InitMessage() calls from ThreadImport(), as Qt
doesn't like them getting called out of it's main thread and because the
thread will continue to run after the GUI was loaded
Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files,
implemented by either LevelDB or BDB.
Based on code from earlier commits by Mike Hearn in his leveldb
branch.
To prevent excessive copying of CCoins in and out of the CCoinsView
implementations, introduce a GetCoins() function in CCoinsViewCache
with returns a direct reference. The block validation and connection
logic is updated to require caching CCoinsViews, and exploits the
GetCoins() function heavily.
Use CBlock's vMerkleTree to cache transaction hashes, and pass them
along as argument in more function calls. During initial block download,
this results in every transaction's hash to be only computed once.
During the initial block download (or -loadblock), delay connection
of new blocks a bit, and perform them in a single action. This reduces
the load on the database engine, as subsequent blocks often update an
earlier block's transaction already.
This switches bitcoin's transaction/block verification logic to use a
"coin database", which contains all unredeemed transaction output scripts,
amounts and heights.
The name ultraprune comes from the fact that instead of a full transaction
index, we only (need to) keep an index with unspent outputs. For now, the
blocks themselves are kept as usual, although they are only necessary for
serving, rescanning and reorganizing.
The basic datastructures are CCoins (representing the coins of a single
transaction), and CCoinsView (representing a state of the coins database).
There are several implementations for CCoinsView. A dummy, one backed by
the coins database (coins.dat), one backed by the memory pool, and one
that adds a cache on top of it. FetchInputs, ConnectInputs, ConnectBlock,
DisconnectBlock, ... now operate on a generic CCoinsView.
The block switching logic now builds a single cached CCoinsView with
changes to be committed to the database before any changes are made.
This means no uncommitted changes are ever read from the database, and
should ease the transition to another database layer which does not
support transactions (but does support atomic writes), like LevelDB.
For the getrawtransaction() RPC call, access to a txid-to-disk index
would be preferable. As this index is not necessary or even useful
for any other part of the implementation, it is not provided. Instead,
getrawtransaction() uses the coin database to find the block height,
and then scans that block to find the requested transaction. This is
slow, but should suffice for debug purposes.
Introduce a AllocateFileRange() function in util, which wipes or
at least allocates a given range of a file. It can be overriden
by more efficient OS-dependent versions if necessary.
Block and undo files are now allocated in chunks of 16 and 1 MiB,
respectively.
Change the block storage layer again, this time with multiple files
per block, but tracked by txindex.dat database entries. The file
format is exactly the same as the earlier blk00001.dat, but with
smaller files (128 MiB for now).
The database entries track how many bytes each block file already
uses, how many blocks are in it, which range of heights is present
and which range of dates.
Special serializer/deserializer for amount values. It is optimized for
values which have few non-zero digits in decimal representation. Most
amounts currently in the txout set take only 1 or 2 bytes to
represent.
These command are a leftover from send-to-IP transactions, which have been
removed a long time ago.
Also removes CNode::mapRequests and CNode::PushRequests, as these were
only used for the mentioned commands.
Matt pointed out some time ago that there existed a minor DOS
attack where a node in its initial block download could be wedged
by an overwrite attack in a fork created between checkpoints before
a time where BIP30 was enforced. Now that the BIP30 timestamp
is irreversibly past the check can be more aggressive and apply to
all blocks except the two historic violations.
Hard-code a special nId=max int alert, to be broadcast if the
alert key is ever compromised. It applies to all versions, never
expires, cancels all previous alerts, and has a fixed message:
URGENT: Alert key compromised, upgrade required
Variations are not allowed (ignored), so an attacker with
the private key cannot broadcast empty-message nId=max alerts.
This fixes two alert system vulnerabilities found by
Sergio Lerner; you could send peers unlimited numbers
of invalid alert message to try to either fill up their
debug.log with messages and/or keep their CPU busy
checking signatures.
Fixed by disconnecting/banning peers if they send 10 or more
bad (invalid/expired/cancelled) alerts.
If 950 of the last 1,000 blocks are nVersion=2, reject nVersion=1
(or zero, but no bitcoin release has created block.nVersion=0) blocks
-- 75 of last 100 on testnet3.
This rule is being put in place now so that we don't have to go
through another "express support" process to get what we really
want, which is for every single new block to include the block height
in the coinbase.
"Version 2" blocks are blocks that have nVersion=2 and
have the block height as the first item in their coinbase.
Block-height-in-the-coinbase is strictly enforced when
version=2 blocks are a supermajority in the block chain
(750 of the last 1,000 blocks on main net, 51 of 100 for
testnet). This does not affect old clients/miners at all,
which will continue producing nVersion=1 blocks, and
which will continue to be valid.
- If the height is in the first half, start at the genesis block and go up, rather than at the top
- Cache the last lookup and use it as a reference point if it's close to the next request, to make linear lookups always fast
- ensure warnings always start with "Warning:" and that the first
character after ":" is written uppercase
- ensure the first sentence in warnings ends with an "!"
- remove unneeded spaces from Warning-strings
- add missing Warning-string translation
- remove a "\n" and replace with untranslatable "<br><br>"
The new bytes are based on "11" to appeal to Gavin's 11 fetish.
This breaks existing testnet3 nodes as the blockchain files
are also versioned. To upgrade a node delete everything
except wallet.dat from your .bitcoin/testnet3 folder.
Modify CreateNewBlock so that instead of processing all transactions
in priority order, process the first 27K of transactions in
priority order and then process the rest in fee-per-kilobyte
order.
This is the first, minimal step towards better a better fee-handling
system for both miners and end-users; this patch should be easy
to backport to the old versions of Bitcoin, and accomplishes the
most important goal-- allow users to "buy their way in" to blocks
using transaction fees.
* Fix wrong thread name for wallet *relocking* thread
- Was named the unlocking thread
* Use consistent naming
Signed-off-by: Giel van Schijndel <me@mortis.eu>
NOTE: These thread names are visible in gdb when using 'info threads'.
Additionally both 'top' and 'ps' show these names *unless* told to
display the command-line instead of task name.
Signed-off-by: Giel van Schijndel <me@mortis.eu>
Adds CBlock::CURRENT_VERSION and CTransaction::CURRENT_VERSION
constants, and makes non-CURRENT_VERSION transactions nonstandard.
This will help make future upgrades smoother.
Prior to this change, each TX typically generated 3+ debug messages,
askfor tx 8644cc97480ba1537214 0
sending getdata: tx 8644cc97480ba1537214
askfor tx 8644cc97480ba1537214 1339640761000000
askfor tx 8644cc97480ba1537214 1339640881000000
CTxMemPool::accept() : accepted 8644cc9748 (poolsz 6857)
After this change, there is only one message for each valid TX received
CTxMemPool::accept() : accepted 22a73c5d8c (poolsz 42)
and two messages for each orphan tx received
ERROR: FetchInputs() : 673dc195aa mempool Tx prev not found 1e439346fc
stored orphan tx 673dc195aa (mapsz 19)
The -debugnet option, or its superset -debug, will restore the full debug
output.
- Signals now go directly from the core to WalletModel/ClientModel.
- WalletModel subscribes to signals on CWallet: Prepares for multi-wallet support, by no longer assuming an implicit global wallet.
- Gets rid of noui.cpp, the few lines that were left are merged into init.cpp
- Rename wxXXX message flags to MF_XXX, to make them UI indifferent.
- ThreadSafeMessageBox no longer returns the value `4` which was never used, converted to void.
Gets rid of `MainFrameRepaint` in favor of specific update functions that tell the UI exactly what changed.
This improves the efficiency of various handlers. Also fixes problems with mined transactions not showing up until restart.
The following notifications were added:
- `NotifyBlocksChanged`: Block chain changed
- `NotifyKeyStoreStatusChanged`: Wallet status (encrypted, locked) changed.
- `NotifyAddressBookChanged`: Address book entry changed.
- `NotifyTransactionChanged`: Wallet transaction added, removed or updated.
- `NotifyNumConnectionsChanged`: Number of connections changed.
- `NotifyAlertChanged`: New, updated or cancelled alert. As this finally makes it possible for the UI to know when a new alert arrived, it can be shown as OS notification.
These notifications could also be useful for RPC clients. However, currently, they are ignored in bitcoind (in noui.cpp).
Also brings back polling with timer for numBlocks in ClientModel. This value updates so frequently during initial download that the number of signals clogs the UI thread and causes heavy CPU usage. And after initial block download, the value changes so rarely that a delay of half a second until the UI updates is unnoticable.
If Reorganize() fails, then its caller, CBlock::SetBestChain(),
will call TxnAbort().
Redundant TxnAbort() calls are harmless. The second will return an
error return value, with no other side effects. TxnAbort() return
values are generally never checked. The impact is nil.
Loop over all inputs doing inexpensive validity checks first,
and then loop over them a second time doing expensive signature
checks. This helps prevent possible CPU exhaustion attacks
where an attacker tries to make a victim waste time checking
signatures for invalid transactions.
Remove orphan transactions from memory once
all of their parent transactions are received
and they're still not valid.
Thanks to Sergio Demian Lerner for suggesting this fix.
Old log message:
storing orphan tx df2244f6bc
New log message:
storing orphan tx df2244f6bc (mapsz 51)
Also, trim a few trailing whitespace in main.cpp.
Immediately issue a "getblocks", instead of a "getdata" (which will
trigger the relevant "inv" to be sent anyway), and only do so when
the previous set of invs led us into a known and attached part of
the block tree.
This prevents an undefined operation in main.cpp, when shifting the hash value
left by 32 bits.
Shifting a signed int left into the sign bit is undefined in C++11.
Introduce a boolean variable for each "network" (ipv4, ipv6, tor, i2p),
and track whether we are likely to able to connect to it. Addresses in
"addr" messages outside of our network get limited relaying and are not
stored in addrman.
This will make bitcoin relay valid routable IPv6 addresses, and when
USE_IPV6 is enabled, listen on IPv6 interfaces and attempt connections
to IPv6 addresses.
FetchInputs already logs failures internally. This commit makes the logging
more consistent with other FetchInputs callsites also.
Prior to this commit, two log lines were logged for one condition:
ERROR: FetchInputs() : de15fde415 mempool Tx prev not found a2c75da227
ERROR: CTxMemPool::accept() : FetchInputs failed de15fde415
After this commit, only one line is logged:
ERROR: FetchInputs() : e0507ab2c7 mempool Tx prev not found 9a620262cd
Previously, a single TX would trigger two log lines in quick succession,
addUnchecked(): size 152
CTxMemPool::accept() : accepted c4cfdd48b7
After this change, only one log line is used:
CTxMemPool::accept() : accepted 98885e65db (poolsz 26)
Pull request #948 introduced a fix for nodes stuck on a long side branch
of the main chain. The fix was non-functional however, as the additional
getdata request was created in a first step of processing, but dropped
in a second step as it was considered redundant. This commits fixes it
by sending the request directly.
Works for wallet transactions, memory-pool transaction and block chain
transactions.
Available for all:
* txid
* version
* locktime
* size
* coinbase/inputs/outputs
* confirmations
Available only for wallet transactions:
* amount
* fee
* details
* blockindex
Available for wallet transactions and block chain transactions:
* blockhash
* time
This commit removes the dependency of serialize.h on PROTOCOL_VERSION,
and makes this parameter required instead of implicit. This is much saner,
as it makes the places where changing a version number can have an
influence obvious.
Conflict:
* cs_main in ProcessMessages() (before calling ProcessMessages)
* cs_vSend in CNode::BeginMessage
versus:
* cs_vSend in ThreadMessageHandler2 (before calling SendMessages)
* cs_main in SendMessages
Even though cs_vSend is a try_lock, if it succeeds simultaneously with
the locking of cs_main in ProcessMessages(), it could cause a deadlock.
Open database once per "tx" message, rather than multiple times,
in the case of orphan transaction presence.
As a side effect, a now-unused CTransaction::AcceptToMemoryPool()
variant is removed.
Reference miner exists for testnet-in-a-box type situations, and as a
reference. We don't care enough about highly optimized internal
mining to keep workarounds like this.
Add a pong message that is sent in reply to a ping. It echoes back a nonce
field that is now added to the ping message. Send a nonce of zero in ping
messages.
Original author: Mike Hearn @ Google
Modified Mike's change to introduce a mild form of protocol documentation in
version.h.
Where possible, use boost::filesystem::path instead of std::string or
char* for filenames. This avoids a lot of manual string tinkering, in
favor of path::operator/.
GetDataDir is also reworked significantly, it now only keeps two cached
directory names (the network-specific data dir, and the root data dir),
which are decided through a parameter instead of pre-initialized global
variables.
Finally, remove the "upgrade from 0.1.5" case where a debug.log in the
current directory has to be removed.
All client version information is moved to version.cpp, which optionally
(-DHAVE_BUILD_INFO) includes build.h. build.h is automatically generated
on supporting platforms via contrib/genbuild.sh, using git describe.
The git export-subst attribute is used to put the commit id statically
in version.cpp inside generated archives, and this value is used if no
build.h is present.
The gitian descriptors are modified to use git archive instead of a
copy, to create the src/ directory in the output. This way,
src/src/version.cpp will contain the static commit id. To prevent
gitian builds from getting the "-dirty" marker in their git-describe
generated identifiers, no touching of files or running sed on the
makefile is performed anymore. This does not seem to influence
determinism.
- rename wxMessageBox, remove redundant arguments to noui/qtui calls
- also, add flag to force blocking, modal dialog box for disk space warning etc
- clarify function naming
- no more special MessageBox needed from AppInit2, as window object is created before calling AppInit2
In cases of very large reorganisations (hundreds of blocks), a situation
may appear where an 'inv' is sent as response to a 'getblocks', but the
last block mentioned in the inv is already known to the receiver node.
However, the supplying node uses a request for this last block as a
trigger to send the rest of the inv blocks. If it never comes, the block
chain download is stuck.
This commit makes the receiver node always request the last inv'ed block,
even if it is already known, to prevent this problem.
Sometimes a new block arrives in a new chain that was already the
best valid one, but wasn't marked that way. This happens for example
when network rules change to recover after a fork.
In this case, it is not necessary to do the entire reorganisation
inside a single db commit. These can become huge, and exceed the
objects/lockers limits in bdb. This patch limits the blocks the
actual reorganisation is applied to, and adds the next blocks
afterwards in separate db transactions.
Introduce the following network rule:
* a block is not valid if it contains a transaction whose hash
already exists in the block chain, unless all that transaction's
outputs were already spent before said block.
Warning: this is effectively a network rule change, with potential
risk for forking the block chain. Leaving this unfixed carries the
same risk however, for attackers that can cause a reorganisation
in part of the network.
Thanks to Russell O'Connor and Ben Reeves.
Doing so would allow an attack on old nodes, which would relay a
standard transaction spending a BIP16 output in an invalid way,
until reaching a new node, which will disconnect their peer.
Reported by makomk on IRC.
Design goals:
* Only keep a limited number of addresses around, so that addr.dat does not grow without bound.
* Keep the address tables in-memory, and occasionally write the table to addr.dat.
* Make sure no (localized) attacker can fill the entire table with his nodes/addresses.
See comments in addrman.h for more detailed information.
This also avoids flushing setAddrKnown until 24 hours has passed,
and avoids contacting the external IP services when not listening.
Advertising non-listening nodes is just addr message spam.
It doesn't help the network, in fact it hurts the network,
and it also hurts user's privacy.
Advertising far out of sync nodes doesn't help the network—
they can't even forward (most) transactions and wastes nodes
outbound slots.
Allow mining of min-difficulty blocks if 20 minutes have gone by without mining a regular-difficulty block.
Normal rules apply every 2016 blocks, though, so there may be a very-slow-to-confirm block at the difficulty-adjustment blocks.
This also removes an un-needed sigops-per-byte check when accepting transactions to the memory pool (un-needed assuming only standard transactions are being accepted). And it only counts P2SH sigops after the switchover date.
This introduces CNetAddr and CService, respectively wrapping an
(IPv6) IP address and an IP+port combination. This functionality used
to be part of CAddress, which also contains network flags and
connection attempt information. These extra fields are however not
always necessary.
These classes, along with logic for creating connections and doing
name lookups, are moved to netbase.{h,cpp}, which does not depend on
headers.h.
Furthermore, CNetAddr is mostly IPv6-ready, though IPv6
functionality is not yet enabled for the application itself.
so it takes a flag for how to interpret OP_EVAL.
Also increased IsStandard size of scriptSigs to 500 bytes, so
a 3-of-3 multisig transaction IsStandard.
OP_EVAL is a new opcode that evaluates an item on the stack as a script.
It enables a new type of bitcoin address that needs an arbitrarily
complex script to redeem.
During the rushed transition from 0.01 BTC to 0.0005 BTC fees, we took the
approach of dropping the relay and block-inclusion fee to 0.0005 BTC
immediately, and only delayed adjusting the sending fee for the next release.
Afterward, the relay fee was lowered to 0.0001 BTC to avoid having the same
problem in the future. However, the block inclusion code was left setting
fForRelay to true! This fixes that, so the lower 0.0001 BTC allowance is (as
intended) only permitted for real relaying.
getmemorypool [data]
If [data] is not specified, returns data needed to construct a block to work on:
"version" : block version
"previousblockhash" : hash of current highest block
"transactions" : contents of non-coinbase transactions that should be included in the next block
"coinbasevalue" : maximum allowable input to coinbase transaction, including the generation award and transaction fees
"time" : timestamp appropriate for next block
"bits" : compressed target of next block
If [data] is specified, tries to solve the block and returns true if it was successful.
Collapsed multiple wallet mutexes to a single cs_wallet, to avoid deadlocks with wallet methods that acquired locks in different order.
Also change master RPC call handler to acquire cs_main and cs_wallet locks before executing RPC calls; requiring each RPC call to acquire the right set of locks in the right order was too error-prone.
Explicitly make these global variables less-global to reduce the maximum
scope of this global state.
In my experience global variables tend to be a major source of bugs. As
such the less accessible they are the less likely they are to be the
source of a bug.
Signed-off-by: Giel van Schijndel <me@mortis.eu>
Regarding https://bitcointalk.org/index.php?topic=28022.0
main.cpp has: "char pchMessageStart[4] = { 0xf9, 0xbe, 0xb4, 0xd9 };"
Per discussion on the thread linked, leaving the signedness of
pchMessageStart is unsafe for values > 0x80. This patch specifies
'unsigned char' in main.cpp and net.h.
Signed-off-by: Jeff Garzik <jgarzik@pobox.com>
Previously, mapAlreadyAskedFor was read from, but never added to.
The original intent was to use mapAlreadyAskedFor to keep track
of the time an item was requested and "Each retry is 2 minutes
after the last".
This implements that intent.
In the assert()s take advantage of the fact that string constants
("string") are effectively of type 'const char []', which when used in
an expression yield a non-NULL pointer.
An assertion that should always fail can thus be formulated as:
assert(!"fail);
An assertion where a text message should be added to the expression can
be written as such:
assert("message" && expression);
Signed-off-by: Giel van Schijndel <me@mortis.eu>
This commit adds support for ckeys, or enCrypted private keys, to the wallet.
All keys are stored in memory in their encrypted form and thus the passphrase
is required from the user to spend coins, or to create new addresses.
Keys are encrypted with AES-256-CBC using OpenSSL's EVP library. The key is
calculated via EVP_BytesToKey using SHA512 with (by default) 25000 rounds and
a random salt.
By default, the user's wallet remains unencrypted until they call the RPC
command encryptwallet <passphrase> or, from the GUI menu, Options->
Encrypt Wallet.
When the user is attempting to call RPC functions which require the password
to unlock the wallet, an error will be returned unless they call
walletpassphrase <passphrase> <time to keep key in memory> first.
A keypoolrefill command has been added which tops up the users keypool
(requiring the passphrase via walletpassphrase first).
keypoolsize has been added to the output of getinfo to show the user the
number of keys left before they need to specify their passphrase (and call
keypoolrefill).
Note that walletpassphrase will automatically fill keypool in a separate
thread which it spawns when the passphrase is set. This could cause some
delays in other threads waiting for locks on the wallet passphrase, including
one which could cause the passphrase to be stored longer than expected,
however it will not allow the passphrase to be used longer than expected as
ThreadCleanWalletPassphrase will attempt to get a lock on the key as soon
as the specified lock time has arrived.
When the keypool runs out (and wallet is locked) GetOrReuseKeyFromPool
returns vchDefaultKey, meaning miners may start to generate many blocks to
vchDefaultKey instead of a new key each time.
A walletpassphrasechange <oldpassphrase> <newpassphrase> has been added to
allow the user to change their password via RPC.
Whenever keying material (unencrypted private keys, the user's passphrase,
the wallet's AES key) is stored unencrypted in memory, any reasonable attempt
is made to mlock/VirtualLock that memory before storing the keying material.
This is not true in several (commented) cases where mlock/VirtualLocking the
memory is not possible.
Although encryption of private keys in memory can be very useful on desktop
systems (as some small amount of protection against stupid viruses), on an
RPC server, the password is entered fairly insecurely. Thus, the only main
advantage encryption has for RPC servers is for RPC servers that do not spend
coins, except in rare cases, eg. a webserver of a merchant which only receives
payment except for cases of manual intervention.
Thanks to jgarzik for the original patch and sipa, gmaxwell and many others
for all their input.
Conflicts:
src/wallet.cpp
Introduce SendBufferSize() and ReceiveBufferSize(), and limit
the blocks sent as response to the "getblocks" message to
half of the active send buffer size.
* A new class CKeyStore manages private keys, and script.cpp depends on access to CKeyStore.
* A new class CWallet extends CKeyStore, and contains all former wallet-specific globals; CWallet depends on script.cpp, not the other way around.
* Wallet-specific functions in CTransaction/CTxIn/CTxOut (GetDebit, GetCredit, GetChange, IsMine, IsFromMe), are moved to CWallet, taking their former 'this' argument as an explicit parameter
* CWalletTx objects know which CWallet they belong to, for convenience, so they have their own direct (and caching) GetDebit/... functions.
* Some code was moved from CWalletDB to CWallet, such as handling of reserve keys.
* Main.cpp keeps a set of all 'registered' wallets, which should be informed about updates to the block chain, and does not have any notion about any 'main' wallet. Function in main.cpp that require a wallet (such as GenerateCoins), take an explicit CWallet* argument.
* The actual CWallet instance used by the application is defined in init.cpp as "CWallet* pwalletMain". rpc.cpp and ui.cpp use this variable.
* Functions in main.cpp and db.cpp that are not used by other modules are marked static.
* The code for handling the 'submitorder' message is removed, as it not really compatible with the idea that a node is independent from the wallet(s) connected to it, and obsolete anyway.
This introduces two new source files, keystore.cpp and wallet.cpp with
corresponding headers. Code is moved from main and db, in a preparation
for a follow-up commit which introduces the classes CWallet and CKeyStore.
Transactions created with the new minimal fee policy would not be
relayed by the network. Therefore, we separate the minimal fee that
is necessary to relay and to create, leaving the creation one at
the old amount, for now.
With the separation of CENT and MIN_TX_FEE, it is now reasonable
to create change outputs between 0.01 and 0.0005, as these are
spendable according to the policy, even though they require a fee
to be paid.
Also, when enough fee was already present, everything can go into
a change output, without further increasing the fee.
When rescanning, if the scanned transaction is already in the wallet, it
is skipped. However, if someone sends a transaction, does not wait for
confirmation, switches wallets, waits for a block that contains his original
transaction, and switches wallets again, a rescan will leave his wallet
transaction (which has no merkle branch, so no confirmations) untouched.
there is no internal modification of any file in this commit
files are moved into directories according to established standards in
sourcecode distribution; these directories contain:
src - Files that are used in constructing the executable binaries,
but are not installed.
doc - Files in HTML and text format that document usage, quirks of
the implementation, and contributor checklists.
locale - Files that contain human language translation of strings
used in the program
contrib - Files contributed from distributions or other third party
implementing scripts and auxiliary programs
