Commit graph

3374 commits

Author SHA1 Message Date
Dave Collins
20910511e9
blockchain: Refactor to use new chain view.
- Remove inMainChain from block nodes since that can now be efficiently
  determined by using the chain view
- Track the best chain via a chain view instead of a single block node
  - Use the tip of the best chain view everywhere bestNode was used
  - Update chain view tip instead of updating best node
- Change reorg logic to use more efficient chain view fork finding logic
- Change block locator code over to use more efficient chain view logic
  - Remove now unused block-index-based block locator code
  - Move BlockLocator definition to chain.go
  - Move BlockLocatorFromHash and LatestBlockLocator to chain.go
    - Update both to use more efficient chain view logic
- Rework IsCheckpointCandidate to use block index and chain view
- Optimize MainChainHasBlock to use chain view instead of hitting db
  - Move to chain.go since it no longer involves database I/O
  - Removed error return since it can no longer fail
- Optimize BlockHeightByHash to use chain view instead of hitting db
  - Move to chain.go since it no longer involves database I/O
  - Removed error return since it can no longer fail
- Optimize BlockHashByHeight to use chain view instead of hitting db
  - Move to chain.go since it no longer involves database I/O
  - Removed error return since it can no longer fail
- Optimize HeightRange to use chain view instead of hitting db
  - Move to chain.go since it no longer involves database I/O
- Optimize BlockByHeight to use chain view for main chain check
- Optimize BlockByHash to use chain view for main chain check
2017-08-23 23:43:37 -05:00
Jim Posen
08955805d5 blockmanager: Remove serverPeer from blockmanager completely.
The purpose is to remove the dependency of blockmanager on serverPeer,
which is defined in the main package. Instead, we split out some of
the fields from serverPeer into a separate struct called peerSyncState
in blockmanager.go. While they are in the same package now, this
change makes it easier to move blockManager into its own package along
with peerSyncState. The blockManager tracks a map of Peer pointers to
the peer state and keeps it updated as peers connect and disconnect.
2017-08-23 10:02:12 -07:00
Jim Posen
088ccfd828 blockmanager: Remove dependency on cfg in main package. 2017-08-23 11:45:25 -05:00
Janus Troelsen
45ea940039 docs: Correct rpcclient link. 2017-08-21 15:56:57 -05:00
Dave Collins
2d84a74d28
blockchain: Correct chainview locator comment. 2017-08-21 14:16:31 -05:00
Dave Collins
991a72e34e
blockchain: Remove unused verify disable code.
This removes the DisableVerify function and related state since nothing
uses it anymore since the command line option was removed.  It is a
remnant of initial development.
2017-08-21 04:10:23 -05:00
Dave Collins
1a947c46b2
blockchain: Faster chain view block locator.
This exposes the ability to more efficiently create a block locator from
a chain view for a given block node by using their ability to do O(1)
lookups.

It also adds tests to ensure the behavior is correct.
2017-08-19 23:58:04 -05:00
Dave Collins
93d8dfc760
blockchain: Optimize block locator generation.
This significantly optimizes and simplifies the generation of block
locators by making use of the fact that all block nodes are now in
memory and therefore it is no longer necessary to consult the database
for the hashes or worry about issues related to dynamic loading of nodes.

Also, it slightly modifies the algorithm so that the doubling doesn't
start for one additional iteration in order to mirror other prominent
clients on the network.  Due to the way block locators are used, this
does not change any semantics in terms of requesting and locating
blocks.

Finally, the semantics of BlockLocatorFromHash have been changed to
return a locator for the current tip in the case the hash is unknown.
This is far preferable since only including the passed block hash, when
it isn't known, could end up leading to causing a redownload of the
entire chain under certain circumstances.
2017-08-19 23:19:58 -05:00
Dave Collins
e02fbcf5a1
blockchain: Consolidate tests into package.
Putting the test code in the same package makes it easier for forks
since they don't have to change the import paths as much and it also
gets rid of the need for internal_test.go to bridge.

While here, remove the reorganization test since it is much better
handled by the full block tests and is no longer needed and do some
light cleanup on a few other tests.

The full block tests had to remain in the separate test package since it
is a circular dependency otherwise.  This did require duplicating some
of the chain setup code, but given the other benefits this is
acceptable.
2017-08-19 23:05:17 -05:00
Dave Collins
f4fe6c373e
blockchain: Convert seq lock tests to synthetic.
This introduces the concept of a synthetic block chain that can be used
in the tests to avoid needing setup a full blown chain instance with a
database and generate valid blocks and converts the sequence lock tests
in TestCalcSequenceLock to use it.

Not only does this speed up the test execution time, but it allows the
dependency on rpctest to be removed which will allow the sequence locks
tests to be consolidated into the main package without creating a
circular dependency.
2017-08-19 22:29:22 -05:00
Dave Collins
2a4be16b6c
blockchain: Improve and correct chainview set tip.
This modifies the function to set the tip in the new chainview code to
bulk copy existing nodes when it needs to expand the cap rather than
simply creating a new empty slice and allowing the walk code below it to
repopulate it.  This is a nice optimization since, in practice, most of
the time expanding the cap is only required when the active chain is
being extended after having run for a while which means the end result
is that it will be able to bulk copy all the nodes and just add the most
recent one versus having to walk them all and add them back.

Also, while here expand the tests for setting the tip to ensure the
nodes contained in the resulting view are correct after forcing the
resizes and correct a bug they exposed where changing between a
longer-shorter-longer chain where the longer chain is the same chain
could result in not populating the view correctly.

Finally, update the fake nodes generated by the tests to use a
nonce generated by a deterministic prng in order to ensure the hashes of
all fake nodes are unique, but reproducible.
2017-08-19 22:17:30 -05:00
Anatoli Babenia
2804f4cffe Work on review comments
https://github.com/btcsuite/btcd/pull/1007#discussion_r133563489
2017-08-19 21:10:58 -05:00
Anatoli Babenia
55e0d5c298 Show info when JSON-RPC is not available 2017-08-19 21:10:58 -05:00
Dave Collins
2a753ae9c7
btcec: Regenerate and update precomputed data.
This regenerates the precomputed secp256k1 byte points used to optimize
scalar multiplication.  This should have been done as part of the
normalization correction.
2017-08-18 12:53:28 -05:00
Dave Collins
1d77a611e9
blockchain: Implement new chain view.
This implements a new type in the blockchain package that takes
advantage of the fact that all block nodes are now in memory to provide
a flat view of a specific chain of blocks (a specific branch of the
overall block tree) from a given tip all the way back to the genesis
block along with several convenience functions such as efficiently
comparing two views, quickly finding the fork point (if any) between two
views, and O(1) lookup of the node at a specific height.

The view is not currently used, but the intent is that the code will be
refactored to make use of these views to simplify and optimize several
areas such as best chain selection and reorg logic and finding successor
nodes.  They will also greatly simplify the process of disconnecting the
download logic from the connection logic.

A comprehensive suite of tests is provided to ensure the chain views
behave correctly.
2017-08-17 17:04:39 -05:00
Dave Collins
7e5e6b4610
blockchain: Cleanup a few sequence lock tests.
This cleans up the test in TestCalcSequenceLock in the following ways:
- Use calculated values instead of magic value so it is easier to update
  the tests as needed
- Make tests match the comments
- Change comments to be more consistent and fix some grammar errors
- Set mempool flag for unconfirmed tx tests since they are intended to
  mimic transactions in the mempool
2017-08-16 15:54:03 -05:00
Dave Collins
614b799198
rpcclient: Merge btcrpcclient repo. 2017-08-15 20:09:19 -05:00
Dave Collins
074b2374b8
Import btcrpcclient repo into rpcclient directory.
This commit contains the entire btcrpcclient repository along with
several changes needed to move all of the files into the rpcclient
directory in order to prepare it for merging.  This does NOT update btcd
or any of the other packages to use the new location as that will be
done separately.

- All import paths in the old btcrpcclient files have been changed to
  the new location
- All references to btcrpcclient as the package name have been changed to
  rpcclient
2017-08-15 19:51:58 -05:00
Dave Collins
72f2a3fe49
blockchain: Optimize checkpoint handling.
This modifies the code that determines the most recently known
checkpoint to take advantage of recent changes which make the entire
block index available in memory by only storing a reference to the
specific node in the index that represents the latest known checkpoint.

Previously, the entire block was stored and new checkpoints required
loading it from the database.
2017-08-15 17:06:16 -05:00
Dave Collins
349450379f
blockchain: Remove threshold state db cache.
This completely removes the threshold state database caching code since
it can very quickly be calculated at startup now that the entire block
index is loaded first.
2017-08-15 16:55:40 -05:00
Dave Collins
296fa0a5a0
blockchain: Convert to full block index in mem.
This reworks the block index code such that it loads all of the headers
in the main chain at startup and constructs the full block index
accordingly.

Since the full index from the current best tip all the way back to the
genesis block is now guaranteed to be in memory, this also removes all
code related to dynamically loading the nodes and updates some of the
logic to take advantage of the fact traversing the block index can
longer potentially fail.  There are also more optimizations and
simplifications that can be made in the future as a result of this.

Due to removing all of the extra overhead of tracking the dynamic state,
and ensuring the block node structs are aligned to eliminate extra
padding, the end result of a fully populated block index now takes quite
a bit less memory than the previous dynamically loaded version.

The main downside is that it now takes a while to start whereas it was
nearly instant before, however, it is much better to provide more
efficient runtime operation since that is its ultimate purpose and the
benefits far outweigh this downside.

Some benefits are:

- Since every block node is in memory, the recent code which
  reconstructs headers from block nodes means that all headers can
  always be served from memory which is important since the majority of
  the network has moved to header-based semantics
- Several of the error paths can be removed since they are no longer
  necessary
- It is no longer expensive to calculate CSV sequence locks or median
  times of blocks way in the past
- It will be possible to create much more efficient iteration and
  simplified views of the overall index
- The entire threshold state database cache can be removed since it is
  cheap to construct it from the full block index as needed

An overview of the logic changes are as follows:

- Move AncestorNode from blockIndex to blockNode and greatly simplify
  since it no longer has to deal with the possibility of dynamically
  loading nodes and related failures
- Rename RelativeNode to RelativeAncestor, move to blockNode, and
  redefine in terms of AncestorNode
- Move CalcPastMedianTime from blockIndex to blockNode and remove no
  longer necessary test for nil
- Change calcSequenceLock to use Ancestor instead of RelativeAncestor
  since it reads more clearly
2017-08-15 15:42:34 -05:00
Jim Posen
28606122c3 main: Reduce shared state between server and blockManager.
Instead of having both server and blockManager be aware of the
txProcessed and blockProcessed channels, now the server passed them as
method arguments to blockProcessor.
2017-08-15 15:41:59 -05:00
Jim Posen
1b50c7300f peer: Test that witnessEnabled is set properly on peer connection. 2017-08-15 12:08:49 -07:00
Jim Posen
095bba1a25 peer: Move IsWitnessEnabled() from serverPeer to Peer.
serverPeer is a problematic struct because it is local to the main
package.
2017-08-15 11:34:17 -07:00
Dave Collins
3b5c2baa2e
blockchain: Consistency pass on subscribe code.
This takes care of a few minor nits on the recently merged subscribe
code.  In particular:

- Avoid extra unnecessary allocation on notifications slice
- Avoid defer overhead on notification mutex in sendNotifications
- Make test function comment start with the name of the function per
  Effective Go guidelines
- Use constant for number of subscribers in test
- Don't exceed column 80 in test print
2017-08-15 11:12:56 -05:00
Dave Collins
c5c46376ba
rpcserver: Refactor listener logic to server.
This refactors the RPC server to accept and take ownership of already
configured listeners and refactors the logic to setup those listeners to
the server.  This mirrors the logic used by the connection manager and
is desirable since it is another step closer to being able to split the
RPC server code out into a separate package and will make it much easier
to internally test since it allows creating mock listeners.
2017-08-15 01:07:38 -05:00
Dave Collins
ff700325ac
rpcserver: Don't use activeNetParams.
This modifies all of the RPC code to use the chain parameters that are
associated with the RPC server instead of the global activeNetParams and
thus moves one step closer to being able to split the RPC server out
into a separate package.
2017-08-15 00:48:41 -05:00
Dave Collins
a7a1029445
rpcserver: Decouple from server.
This decouples the RPC server from the internal btcd server to move
closer to being able to split it out into a separate package.

In order to accomplish this, it introduces an rpcserverConfig type and
several new interfaces, named rpcserverPeer, rpcserverConnManager, and
rpcserverBlockManager, which are necessary to break the direct
dependencies on the main server and block manager instances.

It also adds concrete implementations of the new interfaces and uses
them to configure the RPC server.

Ultimately, the RPC server should ideally be decoupled even more such
that all of the types in the configuration struct use interfaces instead
of the concrete types.  Doing this would make the RPC server much easier
to internally test since it would allow creating lightweight stubs for
the various pieces.
2017-08-14 23:01:07 -05:00
Jim Posen
ca9baf77ec blockchain: Fix race condition in notifications_test 2017-08-14 22:48:32 -05:00
Jim Posen
b71d6c3010 Create blockManagerConfig struct passed to newBlockManager.
The config struct accepts an instance of server as an implementation
of the new PeerNotifier wrapping interface.
2017-08-14 20:19:02 -07:00
Jim Posen
49949d4c96 Remove references from blockManager to rpcServer.
Instead of having the block manager notify the RPC server about
accepted, connected, and disconnected blocks, the RPC server will
directly listen for notifications from the blockchain.
2017-08-14 20:14:42 -07:00
Jim Posen
22de1f6d08 Create blockmanager with reference to txMemPool.
The objective is to remove the reference from blockManager to
server. Since the blockManager is responsible for keeping the mempool
in sync, it should have a direct reference to it.
2017-08-14 20:14:42 -07:00
Jim Posen
19b42c3c9b Allow multiple subscribers to blockchain notifications.
The BlockChain struct emits notifications for various events, but
it is only possible to register one listener. This changes the
interface and implementations to allow multiple listeners.
2017-08-14 22:03:24 -05:00
Jim Posen
bc36cf51c6 peer: Don't send unsupported reject to old peers. 2017-08-14 16:58:57 -05:00
Dave Collins
05bdf3b741
mempool: Remove potential negative locktime check.
This removes the standardness check to reject transactions with a lock
time greater than a maxint32 because the old bitcoind nodes which it was
designed to protect against are no longer valid for other reasons and
thus there are no longer any of them on the network to worry about.
2017-08-14 16:49:08 -05:00
Jim Posen
f4d376d7af peer: Improve address sampling in PushAddrMsg.
This changes fixes shuffling algorithm in PushAddrMsg to select a
uniformly random subset of the given addresses using Fisher-Yates.
2017-08-14 15:44:08 -05:00
Dave Collins
fd081f5ae4
rpcserver: Cleanup getheaders handler.
This makes the code for getheaders more consistent with the rest of the
code in terms of making use of existing error functions and using the
same RPC error codes as other handlers.

While here, it also performs the fetching of headers directly instead of
using a function from server which makes it more tightly coupled.
2017-08-14 11:57:06 -05:00
Dave Collins
19eada0b4b
blockchain: Combine ErrDoubleSpend & ErrMissingTx.
This replaces the ErrDoubleSpend and ErrMissingTx error codes with a
single error code named ErrMissingTxOut and updates the relevant errors
and expected test results accordingly.

Once upon a time, the code relied on a transaction index, so it was able
to definitively differentiate between a transaction output that
legitimately did not exist and one that had already been spent.

However, since the code now uses a pruned utxoset, it is no longer
possible to reliably differentiate since once all outputs of a
transaction are spent, it is removed from the utxoset completely.
Consequently, a missing transaction could be either because the
transaction never existed or because it is fully spent.
2017-08-14 11:40:39 -05:00
Jim Posen
e736ae125d Read RPC username/pass from correct config file for btcctl defaults.
If no existing btcctl.conf file exists, btcctl creates a default one
using the RPC username and password from the btcd.conf. If the
--wallet flag is passed, however, it should read from btcwallet.conf
instead.

https://github.com/btcsuite/btcd/issues/875.
2017-08-14 04:07:17 -05:00
Olaoluwa Osuntokun
01f26a142b build: update glide.lock to target latest btcutil commit 2017-08-13 23:17:40 -05:00
Olaoluwa Osuntokun
1405670930 integration/rpctest: update API usage due to segwit 2017-08-13 23:17:40 -05:00
Olaoluwa Osuntokun
fad0d1d56c blockchain: only populate hashcache if tx is segwitty 2017-08-13 23:17:40 -05:00
Olaoluwa Osuntokun
f5dec67086 mining: update GBT generation to include witness commitment
This commit updates the block template generation logic to only include
witness transactions once the soft-fork has activated and to also
include the OP_RETURN witness commitment (with additional block weight
accounting).
2017-08-13 23:17:40 -05:00
Olaoluwa Osuntokun
728c0a4398 blockchain: guard enforcement of segwit rules by a BIP0009 state check
This commit updates the new segwit validation logic within block
validation to be guarded by an initial check to the version bits state
before conditionally enforcing the logic based off of the state.
2017-08-13 23:17:40 -05:00
Olaoluwa Osuntokun
b1893ed228 chaincfg+integration: add BIP0009 deployment parameters for segwit
This commit adds set of BIP0009 (Version Bits) deployment parameters
for all networks detailing the activation parameters for the segwit
soft-fork.

Additionally, the BIP0009 integration test has been updated to test for
the proper transitioning of version bits state for the segwit soft
fork. Finally, the `getblockchaininfo` test has also been updated to
properly display the state of segwit.
2017-08-13 23:17:40 -05:00
Olaoluwa Osuntokun
1244c45b88 mining+config: modify GBT mining to limit by weight, add witness commitment
This commit modifies the existing block selection logic to limit
preferentially by weight instead of serialized block size, and also to
adhere to the new sig-op cost limits which are weighted according to
the witness discount.
2017-08-13 23:17:40 -05:00
Olaoluwa Osuntokun
8b130ec4ea btcjson: update RPC calls to return segwit related data 2017-08-13 23:17:40 -05:00
Olaoluwa Osuntokun
26ff8ddce4 mempool: modify mempool sanity checks to be segwit aware 2017-08-13 23:17:40 -05:00
Olaoluwa Osuntokun
137aabd631 blockchain/indexers: add addrindex support for p2wkh and p2wsh 2017-08-13 23:17:40 -05:00
Olaoluwa Osuntokun
d0768abcc4 BIP0141+blockchain: implement segwit block validation rules
This commit implements the new block validation rules as defined by
BIP0141. The new rules include the constraints that if a block has
transactions with witness data, then there MUST be a commitment within
the conies transaction to the root of a new merkle tree which commits
to the wtxid of all transactions. Additionally, rather than limiting
the size of a block by size in bytes, blocks are now limited by their
total weight unit. Similarly, a newly define “sig op cost” is now used
to limit the signature validation cost of transactions found within
blocks.
2017-08-13 23:17:40 -05:00