Commit graph

3001 commits

Author SHA1 Message Date
Jim Posen
e1ef2f899b blockchain: Track block validation status in block index.
Each node in the block index records some flags about its validation
state. This is just stored in memory for now, but can save effort if
attempting to reconnect a block that failed validation or was
disconnected.
2017-10-23 04:33:15 -05:00
Jim Posen
11d7cae82b rpctest: Compile current version of btcd and run that.
Previously, rpctest would start a btcd node using the btcd executable
in the environment PATH. This caused difficult-to-find issues where
the code would be tested against an older version of btcd, or another
fork entirely. Now it compiles btcd the first time it is needed and
uses that fresh version when launching nodes.
2017-10-20 14:29:45 -07:00
Steven Roose
fb43a179cb btcec: Add case to signature serialization test
It adds the case where the S value of the signature is bigger than the
half of the order of the curve.
2017-10-13 03:37:29 -05:00
Steven Roose
79445fbd97 btcec: Prevent static initialization of S256
This is achieved by introducing a new variable `halfOrder` on the
KoblitzCurve struct that is half the order.
2017-10-13 03:37:29 -05:00
Jim Posen
64d60f2ef2 blockchain: Remove BFDryRun behaviour flag.
This was only used to test block proposals, which has been changed to
instead use CheckConnectBlockTemplate. The flag complicated the
implementation of some chain processing routines and would be
difficult to implement with headers-first syncing.
2017-10-12 06:07:46 -05:00
Jim Posen
04444c1d0e blockchain: Use CheckConnectBlockTemplate for RPC block proposals.
This renames CheckConnectBlock to CheckConnectBlockTemplate and
modifies it to be easily consumable by the getblocktemplate RPC
handler. Performs full block validation now instead of partial
validation.
2017-10-12 06:07:46 -05:00
Josh Rickmar
4803a8291c txscript: Add API to parse atomic swap contracts. 2017-09-20 12:44:35 -05:00
Craig Sturdy
91e5ba1a80 btcjson: Implement addwitnessaddress functionality. 2017-09-10 23:10:46 -05:00
Alex Bosworth
63d1550d42 txscript: Trivial typo fixes. 2017-09-08 12:56:38 -05:00
Dave Collins
8c883d1fca
blockchain/indexers: Allow interrupts.
This propagates the interrupt channel through to blockchain and the
indexers so that it is possible to interrupt long-running operations
such as catching up indexes.
2017-09-05 11:02:46 -05:00
Jim Posen
30d4caeac6 server: Simplify logic to bind listeners.
The helper function parseListeners has been changed to return a
slice of net.Addrs with Network() returning tcp4 or tcp6 instead of
returning two slices of IPv4 and IPv6 addresses to simplify calling
code. Also improves how local addresses are added to the address
manager when listening on wildcard addresses.

Also splits some newServer logic into new method initListeners.
2017-09-04 11:55:37 -07:00
David Hill
a2085c68f8 config: Add --whitelist support. 2017-08-31 09:59:43 -04:00
Dave Collins
42b969a827
blockchain: Remove unused min memory nodes field.
This removes the minMemoryNodes field from the BlockChain struct since
it is no longer used anywhere.
2017-08-30 17:08:34 -05:00
Dave Collins
02a06a2cd8
blockchain: Update some comments to match reality. 2017-08-30 17:07:46 -05:00
Jim Posen
0a9fb53548 blockchain: Add node to block index in maybeAcceptBlock.
This has the same effect but makes it clearer that all blocks written to
the database end up in the block index.
2017-08-30 11:29:52 -07:00
Dave Collins
0b50802d52
btcjson/rpcserver: Encode witness stack as slice.
The modifies the encoding of witness stacks in JSON responses to use a
slice of strings instead of a single space-separated string for
compatibility with Core.
2017-08-30 03:05:35 -05:00
Dave Collins
9bd7bcfff6
travis: Update to go 1.8 and 1.9.
Also, make the gosimple linter happy while here.
2017-08-30 01:03:26 -05:00
Alok Menghrajani
3cb87afa2f rpcclient: fix links in README.md 2017-08-28 01:50:25 -07:00
Andy Weidenbaum
a1d1ea70dd rm extra word segwit, s/segwit segwit/segwit 2017-08-26 10:56:46 -07:00
Dave Collins
feee952cd3
readme: Remove mailing lists and include GPG key.
This updates the README to remove the mailing lists which no longer
exist and to link the GPG public key used to sign the release tags
directly from the repository instead of the no longer active website.
2017-08-25 14:08:55 -05:00
Jim Posen
46fd4ec358 netsync: Change name of blockManager to syncManager. 2017-08-25 13:41:22 -05:00
Jim Posen
f2fc24d0fc netsync: Use package-local logger. 2017-08-25 13:41:22 -05:00
Jim Posen
1cf7e233e5 netsync: Initialize netsync package.
Create doc.go, interface.go, and README for new package.
2017-08-25 13:41:22 -05:00
Jim Posen
3135a40371 MOVEONLY: Move blockmanager and blocklogger to netsync directory. 2017-08-25 13:41:22 -05:00
Dave Collins
9b0884286f
Update deps to pull in additional logging changes.
This update adds additional callsite logging options via btclog and
fixes an error with the rotator package that caused it to stop running
when creating any log messages larger than 4096 bytes.

While here, switch to the new Write method of the Rotator object as
this is more efficient than using the Reader interface with a pipe.

Changes from @jrick.
2017-08-24 17:29:08 -05:00
Dave Collins
34b1373a68
blockchain: Refactor inv discovery for chain view.
This refactors the code that locates blocks (inventory discovery) out of
server and into blockchain where it can make use of the new much more
efficient chain view and more easily be tested.  As an aside, it really
belongs in blockchain anyways since it's purely dealing with the block
index and best chain.

Since the majority of the network has moved to header-based semantics,
this also provides an additional optimization to allow headers to be
located directly versus needing to first discover the hashes and then
fetch the headers.

The new functions are named LocateBlocks and LocateHeaders.  The former
returns a slice of located hashes and the latter returns a slice of
located headers.

Finally, it also updates the RPC server getheaders call and related
plumbing to use the new LocateHeaders function.

A comprehensive suite of tests is provided to ensure both functions
behave correctly for both correct and incorrect block locators.
2017-08-24 14:15:34 -05:00
Dave Collins
6b802379ec
txscript: Shallow tx copy for signature hash calc.
This modifies calcSignatureHash to use a shallow copy of the transaction
versus a deep copy since the actual scripts themselves are not modified
and therefore don't need to be copied.

This is being done because profiling the most overall allocated space
shows that the deep copy performed in calcSignatureHash accounts for
nearly 20% of all allocations on a synced running instance.  Also,
copying all of the additional data makes it more time consuming as well.

With this change, that figure drops from ~20% to ~5% of all allocations.

The following benchmark shows the relative speedups and allocation
reduction as a result of the optimization on my system.  In particular,
the changes result in approximately a 15% speedup and a whopping 99.89%
reduction in allocations when using a large transaction with thousands
of inputs which was the worst case scenario.

benchmark                        old allocs    new allocs    delta
--------------------------------------------------------------------
BenchmarkCalcSignatureHash       11151         12            -99.89%

benchmark                        old ns/op     new ns/op     delta
--------------------------------------------------------------------
BenchmarkCalcSignatureHash       3599845       3056359       -15.10%
2017-08-24 12:48:11 -05:00
Dave Collins
20910511e9
blockchain: Refactor to use new chain view.
- Remove inMainChain from block nodes since that can now be efficiently
  determined by using the chain view
- Track the best chain via a chain view instead of a single block node
  - Use the tip of the best chain view everywhere bestNode was used
  - Update chain view tip instead of updating best node
- Change reorg logic to use more efficient chain view fork finding logic
- Change block locator code over to use more efficient chain view logic
  - Remove now unused block-index-based block locator code
  - Move BlockLocator definition to chain.go
  - Move BlockLocatorFromHash and LatestBlockLocator to chain.go
    - Update both to use more efficient chain view logic
- Rework IsCheckpointCandidate to use block index and chain view
- Optimize MainChainHasBlock to use chain view instead of hitting db
  - Move to chain.go since it no longer involves database I/O
  - Removed error return since it can no longer fail
- Optimize BlockHeightByHash to use chain view instead of hitting db
  - Move to chain.go since it no longer involves database I/O
  - Removed error return since it can no longer fail
- Optimize BlockHashByHeight to use chain view instead of hitting db
  - Move to chain.go since it no longer involves database I/O
  - Removed error return since it can no longer fail
- Optimize HeightRange to use chain view instead of hitting db
  - Move to chain.go since it no longer involves database I/O
- Optimize BlockByHeight to use chain view for main chain check
- Optimize BlockByHash to use chain view for main chain check
2017-08-23 23:43:37 -05:00
Jim Posen
08955805d5 blockmanager: Remove serverPeer from blockmanager completely.
The purpose is to remove the dependency of blockmanager on serverPeer,
which is defined in the main package. Instead, we split out some of
the fields from serverPeer into a separate struct called peerSyncState
in blockmanager.go. While they are in the same package now, this
change makes it easier to move blockManager into its own package along
with peerSyncState. The blockManager tracks a map of Peer pointers to
the peer state and keeps it updated as peers connect and disconnect.
2017-08-23 10:02:12 -07:00
Jim Posen
088ccfd828 blockmanager: Remove dependency on cfg in main package. 2017-08-23 11:45:25 -05:00
Janus Troelsen
45ea940039 docs: Correct rpcclient link. 2017-08-21 15:56:57 -05:00
Dave Collins
2d84a74d28
blockchain: Correct chainview locator comment. 2017-08-21 14:16:31 -05:00
Dave Collins
991a72e34e
blockchain: Remove unused verify disable code.
This removes the DisableVerify function and related state since nothing
uses it anymore since the command line option was removed.  It is a
remnant of initial development.
2017-08-21 04:10:23 -05:00
Dave Collins
1a947c46b2
blockchain: Faster chain view block locator.
This exposes the ability to more efficiently create a block locator from
a chain view for a given block node by using their ability to do O(1)
lookups.

It also adds tests to ensure the behavior is correct.
2017-08-19 23:58:04 -05:00
Dave Collins
93d8dfc760
blockchain: Optimize block locator generation.
This significantly optimizes and simplifies the generation of block
locators by making use of the fact that all block nodes are now in
memory and therefore it is no longer necessary to consult the database
for the hashes or worry about issues related to dynamic loading of nodes.

Also, it slightly modifies the algorithm so that the doubling doesn't
start for one additional iteration in order to mirror other prominent
clients on the network.  Due to the way block locators are used, this
does not change any semantics in terms of requesting and locating
blocks.

Finally, the semantics of BlockLocatorFromHash have been changed to
return a locator for the current tip in the case the hash is unknown.
This is far preferable since only including the passed block hash, when
it isn't known, could end up leading to causing a redownload of the
entire chain under certain circumstances.
2017-08-19 23:19:58 -05:00
Dave Collins
e02fbcf5a1
blockchain: Consolidate tests into package.
Putting the test code in the same package makes it easier for forks
since they don't have to change the import paths as much and it also
gets rid of the need for internal_test.go to bridge.

While here, remove the reorganization test since it is much better
handled by the full block tests and is no longer needed and do some
light cleanup on a few other tests.

The full block tests had to remain in the separate test package since it
is a circular dependency otherwise.  This did require duplicating some
of the chain setup code, but given the other benefits this is
acceptable.
2017-08-19 23:05:17 -05:00
Dave Collins
f4fe6c373e
blockchain: Convert seq lock tests to synthetic.
This introduces the concept of a synthetic block chain that can be used
in the tests to avoid needing setup a full blown chain instance with a
database and generate valid blocks and converts the sequence lock tests
in TestCalcSequenceLock to use it.

Not only does this speed up the test execution time, but it allows the
dependency on rpctest to be removed which will allow the sequence locks
tests to be consolidated into the main package without creating a
circular dependency.
2017-08-19 22:29:22 -05:00
Dave Collins
2a4be16b6c
blockchain: Improve and correct chainview set tip.
This modifies the function to set the tip in the new chainview code to
bulk copy existing nodes when it needs to expand the cap rather than
simply creating a new empty slice and allowing the walk code below it to
repopulate it.  This is a nice optimization since, in practice, most of
the time expanding the cap is only required when the active chain is
being extended after having run for a while which means the end result
is that it will be able to bulk copy all the nodes and just add the most
recent one versus having to walk them all and add them back.

Also, while here expand the tests for setting the tip to ensure the
nodes contained in the resulting view are correct after forcing the
resizes and correct a bug they exposed where changing between a
longer-shorter-longer chain where the longer chain is the same chain
could result in not populating the view correctly.

Finally, update the fake nodes generated by the tests to use a
nonce generated by a deterministic prng in order to ensure the hashes of
all fake nodes are unique, but reproducible.
2017-08-19 22:17:30 -05:00
Anatoli Babenia
2804f4cffe Work on review comments
https://github.com/btcsuite/btcd/pull/1007#discussion_r133563489
2017-08-19 21:10:58 -05:00
Anatoli Babenia
55e0d5c298 Show info when JSON-RPC is not available 2017-08-19 21:10:58 -05:00
Dave Collins
2a753ae9c7
btcec: Regenerate and update precomputed data.
This regenerates the precomputed secp256k1 byte points used to optimize
scalar multiplication.  This should have been done as part of the
normalization correction.
2017-08-18 12:53:28 -05:00
Dave Collins
1d77a611e9
blockchain: Implement new chain view.
This implements a new type in the blockchain package that takes
advantage of the fact that all block nodes are now in memory to provide
a flat view of a specific chain of blocks (a specific branch of the
overall block tree) from a given tip all the way back to the genesis
block along with several convenience functions such as efficiently
comparing two views, quickly finding the fork point (if any) between two
views, and O(1) lookup of the node at a specific height.

The view is not currently used, but the intent is that the code will be
refactored to make use of these views to simplify and optimize several
areas such as best chain selection and reorg logic and finding successor
nodes.  They will also greatly simplify the process of disconnecting the
download logic from the connection logic.

A comprehensive suite of tests is provided to ensure the chain views
behave correctly.
2017-08-17 17:04:39 -05:00
Dave Collins
7e5e6b4610
blockchain: Cleanup a few sequence lock tests.
This cleans up the test in TestCalcSequenceLock in the following ways:
- Use calculated values instead of magic value so it is easier to update
  the tests as needed
- Make tests match the comments
- Change comments to be more consistent and fix some grammar errors
- Set mempool flag for unconfirmed tx tests since they are intended to
  mimic transactions in the mempool
2017-08-16 15:54:03 -05:00
Dave Collins
614b799198
rpcclient: Merge btcrpcclient repo. 2017-08-15 20:09:19 -05:00
Dave Collins
074b2374b8
Import btcrpcclient repo into rpcclient directory.
This commit contains the entire btcrpcclient repository along with
several changes needed to move all of the files into the rpcclient
directory in order to prepare it for merging.  This does NOT update btcd
or any of the other packages to use the new location as that will be
done separately.

- All import paths in the old btcrpcclient files have been changed to
  the new location
- All references to btcrpcclient as the package name have been changed to
  rpcclient
2017-08-15 19:51:58 -05:00
Dave Collins
72f2a3fe49
blockchain: Optimize checkpoint handling.
This modifies the code that determines the most recently known
checkpoint to take advantage of recent changes which make the entire
block index available in memory by only storing a reference to the
specific node in the index that represents the latest known checkpoint.

Previously, the entire block was stored and new checkpoints required
loading it from the database.
2017-08-15 17:06:16 -05:00
Dave Collins
349450379f
blockchain: Remove threshold state db cache.
This completely removes the threshold state database caching code since
it can very quickly be calculated at startup now that the entire block
index is loaded first.
2017-08-15 16:55:40 -05:00
Dave Collins
296fa0a5a0
blockchain: Convert to full block index in mem.
This reworks the block index code such that it loads all of the headers
in the main chain at startup and constructs the full block index
accordingly.

Since the full index from the current best tip all the way back to the
genesis block is now guaranteed to be in memory, this also removes all
code related to dynamically loading the nodes and updates some of the
logic to take advantage of the fact traversing the block index can
longer potentially fail.  There are also more optimizations and
simplifications that can be made in the future as a result of this.

Due to removing all of the extra overhead of tracking the dynamic state,
and ensuring the block node structs are aligned to eliminate extra
padding, the end result of a fully populated block index now takes quite
a bit less memory than the previous dynamically loaded version.

The main downside is that it now takes a while to start whereas it was
nearly instant before, however, it is much better to provide more
efficient runtime operation since that is its ultimate purpose and the
benefits far outweigh this downside.

Some benefits are:

- Since every block node is in memory, the recent code which
  reconstructs headers from block nodes means that all headers can
  always be served from memory which is important since the majority of
  the network has moved to header-based semantics
- Several of the error paths can be removed since they are no longer
  necessary
- It is no longer expensive to calculate CSV sequence locks or median
  times of blocks way in the past
- It will be possible to create much more efficient iteration and
  simplified views of the overall index
- The entire threshold state database cache can be removed since it is
  cheap to construct it from the full block index as needed

An overview of the logic changes are as follows:

- Move AncestorNode from blockIndex to blockNode and greatly simplify
  since it no longer has to deal with the possibility of dynamically
  loading nodes and related failures
- Rename RelativeNode to RelativeAncestor, move to blockNode, and
  redefine in terms of AncestorNode
- Move CalcPastMedianTime from blockIndex to blockNode and remove no
  longer necessary test for nil
- Change calcSequenceLock to use Ancestor instead of RelativeAncestor
  since it reads more clearly
2017-08-15 15:42:34 -05:00
Jim Posen
28606122c3 main: Reduce shared state between server and blockManager.
Instead of having both server and blockManager be aware of the
txProcessed and blockProcessed channels, now the server passed them as
method arguments to blockProcessor.
2017-08-15 15:41:59 -05:00
Jim Posen
1b50c7300f peer: Test that witnessEnabled is set properly on peer connection. 2017-08-15 12:08:49 -07:00