Instead of building a full copy of a CTransaction being signed, and
then modifying bits and pieces until its fits the form necessary
for computing the signature hash, use a wrapper serializer that
only serializes the necessary bits on-the-fly.
This makes it easier to see which data is actually being hash,
reduces load on the heap, and also marginally improves performances
(around 3-4us/sigcheck here). The performance improvements are much
larger for large transactions, though.
The old implementation of SignatureHash is moved to a unit tests,
to test whether the old and new algorithm result in the same value
for randomly-constructed transactions.
This change moves test data into the binaries rather than reading them from
the disk at runtime.
Advantages:
- Tests become distributable
- Cross-compile friendly. Build on one machine and execute in an arbitrary
location on another.
- Easier testing for backports. Users can verify that tests pass without having
to track down corresponding test data.
- More trustworthy test results and easier quality assurance as tests make
fewer assumptions about their environment.
- Tests could theoretically run at client/daemon startup and exit on failure.
Disadvantages:
- Required 'hexdump' build-dependency. This is a standard bsd tool that should
be usable everywhere. It is likely already installed on all build-machines.
- Tests can no longer be fudged after build by altering test-data.
Seems it was forgotten about when IsPushOnly() and the unittests were
written. A particular oddity is that OP_RESERVED doesn't count towards
the >201 opcode limit unlike every other named opcode.
To fix a minor malleability found by Sergio Lerner (reported here:
https://bitcointalk.org/index.php?topic=8392.msg1245898#msg1245898)
The problem is that if (R,S) is a valid ECDSA signature for a given
message and public key, (R,-S) is also valid. Modulo N (the order
of the secp256k1 curve), this means that both (R,S) and (R,N-S) are
valid. Given that N is odd, S and N-S have a different lowest bit.
We solve the problem by forcing signatures to have an even S value,
excluding one of the alternatives.
This commit just changes the signing code to always produce even S
values, and adds a verification mode to check it. This code is not
enabled anywhere yet. Existing tests in key_tests.cpp verify that
the produced signatures are still valid.
The length of vectors, maps, sets, etc are serialized using
Write/ReadCompactSize -- which, unfortunately, do not use a
unique encoding.
So deserializing and then re-serializing a transaction (for example)
can give you different bits than you started with. That doesn't
cause any problems that we are aware of, but it is exactly the type
of subtle mismatch that can lead to exploits.
With this pull, reading a non-canonical CompactSize throws an
exception, which means nodes will ignore 'tx' or 'block' or
other messages that are not properly encoded.
Please check my logic... but this change is safe with respect to
causing a network split. Old clients that receive
non-canonically-encoded transactions or blocks deserialize
them into CTransaction/CBlock structures in memory, and then
re-serialize them before relaying them to peers.
And please check my logic with respect to causing a blockchain
split: there are no CompactSize fields in the block header, so
the block hash is always canonical. The merkle root in the block
header is computed on a vector<CTransaction>, so
any non-canonical encoding of the transactions in 'tx' or 'block'
messages is erased as they are read into memory by old clients,
and does not affect the block hash. And, as noted above, old
clients re-serialize (with canonical encoding) 'tx' and 'block'
messages before relaying to peers.
Fixes issue#2838; this is a tweaked version of pull#2845 that
should not leak the length of the password and is more generic,
in case we run into other situations where we need
timing-attack-resistant comparisons.
Orphan transactions were stored as a CDataStream pointer;
this changes the mapOrphanTransactions data structures to
store orphans as a CTransaction.
This also fixes CVE-2013-4627 by always re-serializing
transactions before relaying them.
The new class is accessed via the Params() method and holds
most things that vary between main, test and regtest networks.
The regtest mode has two purposes, one is to run the
bitcoind/bitcoinj comparison tool which compares two separate
implementations of the Bitcoin protocol looking for divergence.
The other is that when run, you get a local node which can mine
a single block instantly, which is highly convenient for testing
apps during development as there's no need to wait 10 minutes for
a block on the testnet.
Removed AreInputsStandard from CTransaction, made it a regular function in main.
Moved CTransaction::GetOutputFor to CCoinsViewCache.
Moved GetLegacySigOpCount and GetP2SHSigOpCount out of CTransaction into regular functions in main.
Moved GetValueIn and HaveInputs from CTransaction into CCoinsViewCache.
Moved AllowFree, ClientCheckInputs, CheckInputs, UpdateCoins, and CheckTransaction out of CTransaction and into main.
Moved IsStandard and IsFinal out of CTransaction and put them in main as IsStandardTx and IsFinalTx. Moved GetValueOut out of CTransaction into main. Moved CTxIn, CTxOut, and CTransaction into core.
Added minimum fee parameter to CTxOut::IsDust() temporarily until CTransaction is moved to core.h so that CTxOut needn't know about CTransaction.
- explicitly set the default of all GetBoolArg() calls
- rework getarg_test.cpp and util_tests.cpp to cover this change
- some indentation fixes
- move macdockiconhandler.h include in bitcoin.cpp to the "our headers"
section
This fixes test_bitcoin failures on openbsd reported by dhill on IRC.
On some systems rand() is a simple LCG over 2^31 and so it produces
an even-odd sequence. ApproximateBestSubset was only using the least
significant bit and so every run of the iterative solver would be the
same for some inputs, resulting in some pretty dumb decisions.
Using something other than the least significant bit would paper over
the issue but who knows what other way a system's rand() might get us
here. Instead we use an internal RNG with a period of something like
2^60 which is well behaved. This also makes it possible to make the
selection deterministic for the tests, if we wanted to implement that.
* During block verification (when parallelism is requested), script
check actions are stored instead of being executed immediately.
* After every processed transactions, its signature actions are
pushed to a CScriptCheckQueue, which maintains a queue and some
synchronization mechanism.
* Two or more threads (if enabled) start processing elements from
this queue,
* When the block connection code is finished processing transactions,
it joins the worker pool until the queue is empty.
As cs_main is held the entire time, and all verification must be
finished before the block continues processing, this does not reach
the best possible performance. It is a less drastic change than
some more advanced mechanisms (like doing verification out-of-band
entirely, and rolling back blocks when a failure is detected).
The -par=N flag controls the number of threads (1-16). 0 means auto,
and is the default.
These flags select features to be enabled/disabled during script
evaluation/checking, instead of several booleans passed along.
Currently these flags are defined:
* SCRIPT_VERIFY_P2SH: enable BIP16-style subscript evaluation
* SCRIPT_VERIFY_STRICTENC: enforce strict adherence to pubkey/sig encoding standards.
Flushes the blktree/ and coins/ databases, and reindexes the
block chain files, as if their contents was loaded via -loadblock.
Based on earlier work by Jeff Garzik.
signrawtransaction was unable to sign pay-to-script-hash inputs
when given the list of private keys to use. With this commit
you can provide the p2sh redemption script in the list of
inputs.
To prevent excessive copying of CCoins in and out of the CCoinsView
implementations, introduce a GetCoins() function in CCoinsViewCache
with returns a direct reference. The block validation and connection
logic is updated to require caching CCoinsViews, and exploits the
GetCoins() function heavily.
This switches bitcoin's transaction/block verification logic to use a
"coin database", which contains all unredeemed transaction output scripts,
amounts and heights.
The name ultraprune comes from the fact that instead of a full transaction
index, we only (need to) keep an index with unspent outputs. For now, the
blocks themselves are kept as usual, although they are only necessary for
serving, rescanning and reorganizing.
The basic datastructures are CCoins (representing the coins of a single
transaction), and CCoinsView (representing a state of the coins database).
There are several implementations for CCoinsView. A dummy, one backed by
the coins database (coins.dat), one backed by the memory pool, and one
that adds a cache on top of it. FetchInputs, ConnectInputs, ConnectBlock,
DisconnectBlock, ... now operate on a generic CCoinsView.
The block switching logic now builds a single cached CCoinsView with
changes to be committed to the database before any changes are made.
This means no uncommitted changes are ever read from the database, and
should ease the transition to another database layer which does not
support transactions (but does support atomic writes), like LevelDB.
For the getrawtransaction() RPC call, access to a txid-to-disk index
would be preferable. As this index is not necessary or even useful
for any other part of the implementation, it is not provided. Instead,
getrawtransaction() uses the coin database to find the block height,
and then scans that block to find the requested transaction. This is
slow, but should suffice for debug purposes.
Special serializer/deserializer for amount values. It is optimized for
values which have few non-zero digits in decimal representation. Most
amounts currently in the txout set take only 1 or 2 bytes to
represent.
Variable-length integers: bytes are a MSB base-128 encoding of the number.
The high bit in each byte signifies whether another digit follows. To make
the encoding is one-to-one, one is subtracted from all but the last digit.
Thus, the byte sequence a[] with length len, where all but the last byte
has bit 128 set, encodes the number:
(a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1))
Properties:
* Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes)
* Every integer has exactly one encoding
* Encoding does not depend on size of original integer type
For backward compatibility, new accounting data is stored after a \0 in the comment string.
This way, old versions and third-party software should load and store them, but all actual use (listtransactions, for example) ignores it.
Memory locks do not stack, that is, pages which have been locked several times by calls to mlock()
will be unlocked by a single call to munlock(). This can result in keying material ending up in swap when
those functions are used naively. In this commit a class "LockedPageManager" is added
that simulates stacking memory locks by keeping a counter per page.
This does two things:
1) Now does not output to debug.log if -printtodebugger flag is passed
2) Unit tests set -printtodebugger so only test results are output to stdout
Note that -printtodebugger only actually prints to the debugger on Windows.
Implement listunspent / getrawtransaction / createrawtransaction /
signrawtransaction, to support creation and
signing-on-multiple-device multisignature transactions.
This commit adds support for .onion addresses (mapped into the IPv6
by using OnionCat's range and encoding), and the ability to connect
to them via a SOCKS5 proxy.
AvailableCoins() makes a vector of available outputs which is then passed to SelectCoinsMinConf(). This allows unit tests to test the coin selection algorithm without having the whole blockchain available.
Newlines in JSON strings are against the JSON spec,
so remove them from the script*.json unit tests to
make python's jsonrpc happy (json::spirit didn't care).
This introduces internal types:
* CKeyID: reference (hash160) of a key
* CScriptID: reference (hash160) of a script
* CTxDestination: a boost::variant of the former two
CBitcoinAddress is retrofitted to be a Base58 encoding of a
CTxDestination. This allows all internal code to only use the
internal types, and only have RPC and GUI depend on the base58 code.
Furthermore, the header dependencies are a lot saner now. base58.h is
at the top (right below rpc and gui) instead of at the bottom. For the
rest: wallet -> script -> keystore -> key. Only keystore still requires
a forward declaration of CScript. Solving that would require splitting
script into two layers.
test/DoS_tests.cpp: In member function ‘void DoS_tests::DoS_mapOrphans::test_method()’:
test/DoS_tests.cpp:200:41: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
test/DoS_tests.cpp:208:41: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
test/DoS_tests.cpp: In member function ‘void DoS_tests::DoS_checkSig::test_method()’:
test/DoS_tests.cpp:260:37: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
test/DoS_tests.cpp:267:37: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
test/DoS_tests.cpp:280:41: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
test/DoS_tests.cpp:307:37: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
Create a maximum-10MB signature verification result cache.
This should almost double the number of transactions that
can be processed on a given CPU, because before this change
ECDSA signatures were verified when transactions were added
to the memory pool and then again when they appeared in
a block.
One of the test cases currently aborts when using gcc's flag -ftrapv, due to
negating an INT64_MIN int64 variable, which is an undefined operation.
This will be fixed in a subsequent commit.
