Commit graph

3378 commits

Author SHA1 Message Date
Dave Collins
74fe2a4dfd
mining: Introduce a block template generator.
This introduces a new type named BlkTmplGenerator which encapsulates the
various state needed to generate block templates.

This is useful since it means code that needs to generate block
templates can simply accept the generator rather than needing access to
all of the additional state which in turn will ultimately make it easier
to split the mining code into its own package.
2016-10-26 15:17:21 -05:00
Dave Collins
2274d36333 bmgr: Remove block manager chain state.
This removes the block manager chain state in favor of using the
blockchain.BlockChain instance now that it is safe for concurrency.
2016-10-26 14:23:14 -05:00
Dave Collins
2cfc6478ce
mining/mempool: Move priority code to mining pkg.
This moves the priority-related code from the mempool package to the
mining package and also exports a new constant named UnminedHeight which
takes the place of the old unexported mempoolHeight.

Even though the mempool makes use of the priority code to make decisions
about what it will accept, priority really has to do with mining since
it influences which transactions will end up into a block.  This change
also has the side effect of being a step towards enabling separation of
the mining code into its own package which, as previously mentioned,
needs access to the priority calculation code as well.

Finally, the mempoolHeight variable was poorly named since what it
really represents is a transaction that has not been mined into a block
yet.  Renaming the variable to more accurately reflect its purpose makes
it clear that it belongs in the mining package which also needs the
definition now as well since the priority calculation code relies on it.
This will also benefit an outstanding PR which needs access to the same
value.
2016-10-26 12:01:49 -05:00
Jonathan Gillham
c2fb0cb18e mining: Stop transactions losing their dependants.
This fixes a bug where a transaction would lose reference to other
transactions dependant on it when being considered for inclusion in a
new block template. The issue only occurs when the transaction being
considered triggers a change of priority queue ordering to ordering by
fee. It results in none of the dependant transactions being considered
for inclusion in the new block template.
2016-10-26 10:57:48 -05:00
David Hill
18bb90b4ad peer: Advertise _our_ max protocol version, 2016-10-26 08:17:46 -04:00
Dave Collins
54b454ed3d mining: Fix duplicate transactions in prio heap 2016-10-25 16:43:40 -05:00
Dave Collins
e306158e25
mempool: Implement orphan expiration.
This implements orphan expiration in the mempool such that any orphans
that have not had their ancestors materialize within 15 minutes of their
initial arrival time will be evicted which in turn will remove any other
orphans that attempted to redeem it.

In order to perform the evictions with reasonable efficiency, an
opportunistic scan interval of 5 minutes is used.  That is to say that
there is not a hard deadline on the scan interval and instead it runs
when a new orphan is added to the pool if enough time has passed.

The following is an example of running this code against the main
network for around 30 minutes:

23:05:34 2016-10-24 [DBG] TXMP: Expired 3 orphans (remaining: 254)
23:10:38 2016-10-24 [DBG] TXMP: Expired 112 orphans (remaining: 231)
23:15:43 2016-10-24 [DBG] TXMP: Expired 95 orphans (remaining: 206)
23:20:44 2016-10-24 [DBG] TXMP: Expired 90 orphans (remaining: 191)
23:25:51 2016-10-24 [DBG] TXMP: Expired 71 orphans (remaining: 191)
23:30:55 2016-10-24 [DBG] TXMP: Expired 70 orphans (remaining: 105)
23:36:19 2016-10-24 [DBG] TXMP: Expired 55 orphans (remaining: 107)

As can be seen from the above, without orphan expiration on this data
set, the orphan pool would have grown an additional 496 entries.
2016-10-25 15:37:29 -05:00
Dave Collins
c0def9d613
integration: Initial implementation.
This introduces a new package to house tests which use the rpctest
package to programmatically drive end-to-end integration testing and
move the existing RPC integration tests into the new package.
2016-10-25 14:36:23 -05:00
Dave Collins
70db324663
mempool: Stricter orphan evaluation and eviction.
This modifies the way orphan removal and processing is done to more
aggressively remove orphans that can no longer be valid due to other
transactions being added or removed from the primary transaction pool.

The net effect of these changes is that orphan pool will typically be
much smaller which greatly improves its effectiveness.  Previously, it
would typically quickly reach the max allowed worst-case usage and
effectively stay there forever.

The following is a summary of the changes:
- Modify the map that tracks which orphans redeem a given transaction to
  instead track by the specific outpoints that are redeemed
- Modify the various orphan removal and processing functions to accept
  the full transaction rather than just its hash
- Introduce a new flag on removeOrphans which specifies whether or not
  to remove the transactions that redeem the orphan being removed as
  well which is necessary since only some paths require it
- Add a new function named removeOrphanDoubleSpends that is invoked
  whenever a transaction is added to the main pool and thus the outputs
  they spent become concrete spends
- Introduce a new flag on maybeAcceptTransaction which specifies whether
  or not duplicate orphans should be rejected since only some paths
  require it
- Modify processOrphans as follows:
  - Make use of the modified map
  - Use newly available flags and logic work more strictly work with tx
    chains
  - Recursively remove any orphans that also redeem any outputs redeemed
    by the accepted transactions
- Several new tests to ensure proper functionality
  - Removing an orphan that doesn't exist is removed both when there is
    another orphan that redeems it and when there is not
  - Removing orphans works properly with orphan chains per the new
    remove redeemers flag
  - Removal of multi-input orphans that double spend an output when a
    concrete redeemer enters the transaction pool
2016-10-25 10:44:18 -05:00
Dave Collins
60355258a7
mempool: Refactor pool membership test logic.
This introduces a new pool membership test function to the mempool
testing infrastructure and refactors the tests to make use of it.

It is useful since it is common logic that is not only needed in the
existing tests, but will be needed by most mempool-related tests.
2016-10-25 10:42:51 -05:00
Tibor Bősze
6b8a24918e rpcserver: Improve JSON-RPC compatibility
Avoid compatibility issues with software that relies on the behavior of
bitcoind's JSON-RPC implementation.

The JSON-RPC 1.0 spec defines that notifications must have their "id"
set to null and states that notifications do not have a response.

A JSON-RPC 2.0 notification is a request with "json-rpc":"2.0", and
without an "id" member. The specification states that notifications
must not be responded to. JSON-RPC 2.0 permits the null value as a
valid request id, therefore such requests are not notifications.

Bitcoin Core serves requests with "id":null or even an absent "id", and
responds to such requests with "id":null in the response.

Btcd does not respond to any request without and "id" or with "id":null,
regardless the indicated JSON-RPC protocol version.

In order to avoid compatibility issues with software relying on
Core's behavior, this commit implements "quirks mode" as follows:
 - quirks mode can be enabled via configuration (disabled by default)
 - If no JSON-RPC version is indicated in the request, accept and
respond to request with "id":null
 - If no JSON-RPC version is indicated in the request, accept and
respond to requests without an "id" member
 - In both cases above, use "id":null in the response
 - Do not respond to request without an "id" or with "id":null when
JSON-RPC version is indicated in the request (process as notification)
2016-10-24 13:24:18 -05:00
Tibor Bősze
9799f0e547 rpcserver: Improve JSON-RPC compatibility
In order to avoid compatibility issues with software relying on 
Core's behavior, terminate HTTP POST JSON-RPC responses with a newline.
2016-10-24 13:24:18 -05:00
Dave Collins
0e71867dfe
mempool: Optimize orphan map limiting.
This optimizes the way in which the mempool oprhan map is limited in the
same way the server block manager maps were previously optimized.

Previously the code would read a cryptographically random value large
enough to construct a hash, find the first entry larger than that value,
and evict it.

That approach is quite inefficient and could easily become a
bottleneck when processing transactions due to the need to read from a
source such as /dev/urandom and all of the subsequent hash comparisons.

Luckily, strong cryptographic randomness is not needed here. The primary
intent of limiting the maps is to control memory usage with a secondary
concern of making it difficult for adversaries to force eviction of
specific entries.

Consequently, this changes the code to make use of the pseudorandom
iteration order of Go's maps along with the preimage resistance of the
hashing function to provide the desired functionality.  It has
previously been discussed that the specific pseudorandom iteration order
is not guaranteed by the Go spec even though in practice that is how it
is implemented.  This is not a concern however because even if the
specific compiler doesn't implement that, the preimage resistance of the
hashing function alone is enough.

The following is a before and after comparison of the function for both
speed and memory allocations:

benchmark                    old ns/op     new ns/op     delta
----------------------------------------------------------------
BenchmarkLimitNumOrphans     3727          243           -93.48%

benchmark                    old allocs    new allocs    delta
-----------------------------------------------------------------
BenchmarkLimitNumOrphans     4             0             -100.00%
2016-10-23 21:14:09 -05:00
David Hill
e6062595db travis: go1.7.3 2016-10-23 21:53:29 -04:00
Dave Collins
25de9ce5d9
mempool: Add docs.go and flesh out README.md. 2016-10-23 20:47:12 -05:00
Dave Collins
e90b0c967f
docs: Add mempool entry to developer tools section. 2016-10-23 20:47:05 -05:00
Dave Collins
26e22790cd
mempool: Rename RelayNonStd config option.
This renames the mempool.Config.RelayNonStd option to AcceptNonStd which
more accurately describes its behavior since the mempool was refactored
into a separate package.

The reasoning for this change is that the mempool is not responsible for
relaying transactions (nor should it be).  Its job is to maintain a pool
of unmined transactions that are validated according to consensus and
policy configuration options which are then used to provide a source of
transactions that need to be mined.

Instead, it is the server that is responsible for relaying transactions.
While it is true that the current server code currently only relays txns
that were accepted to the mempool, this does not necessarily have to
be the case.  It would be entirely possible (and perhaps even a good
idea as something do in the future), to separate the relay policy from
the mempool acceptance policy (and thus indirectly the mining policy).
2016-10-23 20:41:54 -05:00
David Hill
f161d6b69e chaincfg: Introduce new type DNSSeed
DNSSeed defines a DNS Seed with a hostname and whether it supports
filtering by service flag bits.
2016-10-23 15:59:15 -05:00
David Hill
0d508e6522 peer: BestLocalAddress config option is no longer used. 2016-10-23 16:38:45 -04:00
David Hill
a09d052f96 peer: Stop setting AddrMe in the version message.
Older nodes previously added the IP and port information to the address
manager which proved to be unreliable as an inbound connection from a
peer didn't necessarily mean the peer itself accepted inbound
connections.

This also fixes a bug where the peer package was incorrectly sending
the peer's services as its own.
2016-10-23 16:37:44 -04:00
David Hill
bca9877796 Do not add inbound peers to the address manager. 2016-10-23 16:37:44 -04:00
Dave Collins
b8df516b4b
docs: Add fullblocktests entry and make consistent. 2016-10-23 13:34:14 -05:00
Dave Collins
9634a8cb0d
fullblocktests: Add missing doc.go file. 2016-10-23 13:31:26 -05:00
Javed Khan
bff2ba70fd connmgr: Refactor connection management into pkg
This commit introduces package connmgr which contains connection
management related functionality.

The following is an overview of the features the package provides:

- Maintain fixed number of outbound connections
- Optional connect-only mode
- Retry persistent connections with increasing back-off
- Source peers from DNS seeds
- Use Tor to resolve DNS
- Dynamic ban scores
- Test coverage

In addition, btcd has been refactored to make use of the new package by
extending the connection manager to work with the server to source and
maintain peer connections. The following is a broad overview of the
changes to integrate the package:

- Simplify peer state by removing pending, retry peers
- Refactor to remove retries which are now handled by connmgr
- Use callback to add addresses sourced from the  DNS seed

Finally the following connection-related things have been improved as a
part of this refactor:

- Fixes 100% cpu usage when network is down (#129)
- Fixes issues with max peers (#577)
- Simplify outbound peer connections management
2016-10-22 01:11:57 -05:00
Marco Peereboom
69fca4d9b1 Reconcile differences between btcd/dcrd.
Fixes #793
2016-10-21 16:37:30 -05:00
David Hill
4494f0f852 txscript: Remove OP_SMALLDATA 2016-10-21 15:18:42 -04:00
Javed Khan
f6cd49ac51 peer: rename peer.Connect to AssociateConnection 2016-10-21 14:13:18 -05:00
Dave Collins
0731f2ddc9 txscript: Cleanup and improve NullDataScript tests.
This modifies the recently-added NullDataScript function in several
ways in an effort to make them more consistent with the tests in the
rest of the code base and improve/correct the logic:

- Use the hexToBytes and mustParseShortForm functions
- Consistently format the test errors
- Replace the valid bool flag with an expected error and test against it
- Ensure the returned script type is the expected type in all cases
2016-10-21 10:26:09 -05:00
DanielKrawisz
b77654f8d4 txscript: Add null data script creator
This adds a new function named NullDataScript to the txscript package that returns a provably-pruneable  OP_RETURN script with the provided data.  The function will return an error if the provided data is larger than the maximum allowed length for a nulldata script to be be considered standard.
2016-10-21 09:37:48 -05:00
Josh Rickmar
d0a9c03844 Concurrently handle websocket client JSON-RPC requests. 2016-10-20 19:55:58 -04:00
Dave Collins
da04285e0d
rpctest: Choose flags based on provided params.
This modifies the rpctest framework to start btcd with the appropriate
network flags depending on the provided parameters.

Previously, it always started btcd with --simnet even if other
parameters, such as those for the regression test network, were
provided.
2016-10-20 12:19:58 -05:00
Dave Collins
49cbaf23dd
blockchain: Support small coinbase block size
This modifies the ExtractCoinbaseHeight function to recognize small
canonically serialized block heights in coinbase scripts of blocks
higher than version 2.

This allows regression test chains in which blocks encode the serialized
height in the coinbase starting from block 1.
2016-10-20 12:09:09 -05:00
Dave Collins
59a3fc2f66 txscript: Consolidate tests into txscript package.
Putting the test code in the same package makes it easier for forks
since they don't have to change the import paths as much and it also
gets rid of the need for internal_test.go to bridge.

Also, do some light cleanup on a few tests while here.
2016-10-20 09:28:33 -05:00
Dave Collins
b60e3547d2 txscript: Correct nulldata standardness check.
This corrects the isNullData standard transaction type test to work
properly with canonically-encoded data pushes.  In particular, single
byte data pushes that are small integers (0-16) are converted to the
equivalent numeric opcodes when canonically encoded and the code failed
to detect them properly.

It also adds several tests to ensure that both canonical and
non-canonical nulldata scripts are recognized properly and modifies the
test failure print to include the script that failed.

This does not affect consensus since it is just a standardness check.
2016-10-20 01:44:58 -05:00
David Hill
07e1e308f1 rpc: Add localaddr and relaytxes to getpeerinfo 2016-10-19 19:58:50 -04:00
David Hill
3b5bb9fd43 btcjson: Add preciousblock 2016-10-19 14:08:32 -05:00
David Hill
403aaf5cf3 rpcserver: avoid nested decodescript p2sh addrs 2016-10-19 13:59:50 -05:00
Olaoluwa Osuntokun
e7caccc866
mempool: transaction finality checks now use median-time-past
This coincides with the mempool only, policy change which enforces
transaction finality according to the median-time-past rather than
blockheader timestamps. The behavior is pre-cursor to full blown BIP
113 consensus deployment, and subsequent activation.

As a result, the TimeSource field in the mempoolConfig is no longer
needed so it has been removed. Additionally, checkTransactionStandard has been
modified to instead take a time.Time as the mempool is no longer explicitly
dependant on a Chain instance.
2016-10-19 11:13:34 -07:00
Olaoluwa Osuntokun
a82f67b538
mempool: add closure to compute median time past to config
This commit adds an additional closure function to the mempool’s config
which computes the median time past from the point of view of the best
node in the chain. The mempool test harness has also been updated to allow
setting a mock median time past for testing purposes.

In addition to increasing the testability of the mempool, this commit
should also speed up transaction and block validation for BIP 113 as
the MTP no longer needs to be re-calculated each time from scratch.
2016-10-19 11:13:25 -07:00
David Hill
a6bf1d9850 txscript: Implement CheckSequenceVerify (BIP0112) 2016-10-19 12:06:44 -04:00
Dave Collins
fdfa07b0be
btcec: Consolidate tests into the btcec package.
Putting the test code in the same package makes it easier for forks
since they don't have to change the import paths as much and it also
gets rid of the need for internal_test.go to bridge.

Also, remove the exception from the lint checks about returning the
unexported type since it is no longer required.
2016-10-19 00:55:23 -05:00
David Hill
b1621332cc Optimize by removing defers
defer's are nice for readability but they do add overhead.  This
gets rid of defer's where it is just as easy as not to use one.
2016-10-18 17:56:51 -04:00
Jimmy Song
294b5d46da btcec: Add regression tests for field.go.
This adds new tests to the TestNormalize, TestMul, TestAdd2 functions
which trigger an issue with modular reduction that was fixed in the
prevous commit to prevent regressions.
2016-10-18 16:21:45 -05:00
Dave Collins
a52eb04aaa
btcec: Ensure reduction when > P in all cases.
As noted in issue #706, the existing code had an issue where the
normalized result was > P when both the first and second words of the
field representation being normalized were BOTH greater than or equal to
the first and second words of P.  Although this condition is rare in
practice, it needs to be handled properly.

This resolves the issue by comparing the low words in the final
reduction step against the normalized low order prime bits to ensure the
final subtraction occurs correctly any time they're > P.  This approach
retains the constant time property as well.
2016-10-18 16:21:36 -05:00
David Hill
d009185a56 peer: Implement feefilter p2p message (bip0133) 2016-10-17 15:45:56 -04:00
David Hill
9935fe5dba wire: Bump minor due to feefilter addition 2016-10-17 13:45:12 -04:00
David Hill
ca4e9b82d6 wire: implement feefilter message (bip0133)
feefilter is used to request the receiving peer does not announce any
transactions below the specified minimum fee rate.
2016-10-17 13:33:16 -04:00
Dave Collins
f21410e47c
blockchain: Add block validation infrastructure.
This adds a full-blown testing infrastructure in order to test consensus
validation rules.  It is built around the idea of dynamically generating
full blocks that target specific rules linked together to form a block
chain.  In order to properly test the rules, each test instance starts
with a valid block that is then modified in the specific way needed to
test a specific rule.

Blocks which exercise following rules have been added for this initial
version.  These tests were largely ported from the original Java-based
'official' block acceptance tests as well as some additional tests
available in the Core python port.  It is expected that further tests
can be added over time as consensus rules change.

* Enough valid blocks to have a stable base of mature coinbases to spend
  for futher tests
* Basic forking and chain reorganization
* Double spends on forks
* Too much proof-of-work coinbase (extending main chain, in block that
  forces a reorg, and in a valid fork)
* Max and too many signature operations via various combinations of
  OP_CHECKSIG, OP_MULTISIG, OP_CHECKSIGVERIFY, and OP_MULTISIGVERIFY
* Too many and max signature operations with offending sigop after
  invalid data push
* Max and too many signature operations via pay-to-script-hash redeem
  scripts
* Attempt to spend tx created on a different fork
* Attempt to spend immature coinbase (on main chain and fork)
* Max size block and block that exceeds the max size
* Children of rejected blocks are either orphans or rejected
* Coinbase script too small and too large
* Max length coinbase script
* Attempt to spend tx in blocks that failed to connect
* Valid non-coinbase tx in place of coinbase
* Block with no transactions
* Invalid proof-of-work
* Block with a timestamp too far in the future
* Invalid merkle root
* Invalid proof-of-work limit (bits header field)
* Negative proof-of-work limit (bits header field)
* Two coinbase transactions
* Duplicate transactions
* Spend from transaction that does not exist
* Timestamp exactly at and one second after the median time
* Blocks with same hash via merkle root tricks
* Spend from transaction index that is out of range
* Transaction that spends more that its inputs provide
* Transaction with same hash as an existing tx that has not been
  fully spent (BIP0030)
* Non-final coinbase and non-coinbase txns
* Max size block with canonical encoding which exceeds max size with
  non-canonical encoding
* Spend from transaction earlier in same block
* Spend from transaction later in same block
* Double spend transaction from earlier in same block
* Coinbase that pays more than subsidy + fees
* Coinbase that includes subsidy + fees
* Invalid opcode in dead execution path
* Reorganization of txns with OP_RETURN outputs
* Spend of an OP_RETURN output
* Transaction with multiple OP_RETURN outputs
* Large max-sized block reorganization test (disabled by default since
  it takes a long time and a lot of memory to run)

Finally, the README.md files in the main and docs directories have been
updated to reflect the use of the new testing framework.
2016-10-17 12:16:53 -05:00
Dave Collins
1cba5c8fc0
blockchain: Remove exported CalcPastTimeMedian func.
This removes the exported CalcPastTimeMedian function from the
blockchain package as it is no longer needed since the information is
now available via the BestState snapshot.

Also, update the only known caller of this, which is the chain state in
block manager, to use the snapshot instead.  In reality, now that
everything the block manager chain state provides is available via the
blockchain BestState snapshot, the entire thing can be removed, however
that will be done in a separate to commit to keep the changes targeted.
2016-10-17 10:41:25 -05:00
Dave Collins
e88f2d7bf4
mempool: Add test for max orphan entry eviction.
This adds a test to the mempool for ensuring that orphans are evicted
when exceeding the max orphan policy setting as expected.
2016-10-17 10:33:09 -05:00