Commit graph

3488 commits

Author SHA1 Message Date
Dave Collins
02a06a2cd8
blockchain: Update some comments to match reality. 2017-08-30 17:07:46 -05:00
Jim Posen
0a9fb53548 blockchain: Add node to block index in maybeAcceptBlock.
This has the same effect but makes it clearer that all blocks written to
the database end up in the block index.
2017-08-30 11:29:52 -07:00
Dave Collins
0b50802d52
btcjson/rpcserver: Encode witness stack as slice.
The modifies the encoding of witness stacks in JSON responses to use a
slice of strings instead of a single space-separated string for
compatibility with Core.
2017-08-30 03:05:35 -05:00
Dave Collins
9bd7bcfff6
travis: Update to go 1.8 and 1.9.
Also, make the gosimple linter happy while here.
2017-08-30 01:03:26 -05:00
Alok Menghrajani
3cb87afa2f rpcclient: fix links in README.md 2017-08-28 01:50:25 -07:00
Andy Weidenbaum
a1d1ea70dd rm extra word segwit, s/segwit segwit/segwit 2017-08-26 10:56:46 -07:00
Dave Collins
feee952cd3
readme: Remove mailing lists and include GPG key.
This updates the README to remove the mailing lists which no longer
exist and to link the GPG public key used to sign the release tags
directly from the repository instead of the no longer active website.
2017-08-25 14:08:55 -05:00
Jim Posen
46fd4ec358 netsync: Change name of blockManager to syncManager. 2017-08-25 13:41:22 -05:00
Jim Posen
f2fc24d0fc netsync: Use package-local logger. 2017-08-25 13:41:22 -05:00
Jim Posen
1cf7e233e5 netsync: Initialize netsync package.
Create doc.go, interface.go, and README for new package.
2017-08-25 13:41:22 -05:00
Jim Posen
3135a40371 MOVEONLY: Move blockmanager and blocklogger to netsync directory. 2017-08-25 13:41:22 -05:00
Dave Collins
9b0884286f
Update deps to pull in additional logging changes.
This update adds additional callsite logging options via btclog and
fixes an error with the rotator package that caused it to stop running
when creating any log messages larger than 4096 bytes.

While here, switch to the new Write method of the Rotator object as
this is more efficient than using the Reader interface with a pipe.

Changes from @jrick.
2017-08-24 17:29:08 -05:00
Dave Collins
34b1373a68
blockchain: Refactor inv discovery for chain view.
This refactors the code that locates blocks (inventory discovery) out of
server and into blockchain where it can make use of the new much more
efficient chain view and more easily be tested.  As an aside, it really
belongs in blockchain anyways since it's purely dealing with the block
index and best chain.

Since the majority of the network has moved to header-based semantics,
this also provides an additional optimization to allow headers to be
located directly versus needing to first discover the hashes and then
fetch the headers.

The new functions are named LocateBlocks and LocateHeaders.  The former
returns a slice of located hashes and the latter returns a slice of
located headers.

Finally, it also updates the RPC server getheaders call and related
plumbing to use the new LocateHeaders function.

A comprehensive suite of tests is provided to ensure both functions
behave correctly for both correct and incorrect block locators.
2017-08-24 14:15:34 -05:00
Dave Collins
6b802379ec
txscript: Shallow tx copy for signature hash calc.
This modifies calcSignatureHash to use a shallow copy of the transaction
versus a deep copy since the actual scripts themselves are not modified
and therefore don't need to be copied.

This is being done because profiling the most overall allocated space
shows that the deep copy performed in calcSignatureHash accounts for
nearly 20% of all allocations on a synced running instance.  Also,
copying all of the additional data makes it more time consuming as well.

With this change, that figure drops from ~20% to ~5% of all allocations.

The following benchmark shows the relative speedups and allocation
reduction as a result of the optimization on my system.  In particular,
the changes result in approximately a 15% speedup and a whopping 99.89%
reduction in allocations when using a large transaction with thousands
of inputs which was the worst case scenario.

benchmark                        old allocs    new allocs    delta
--------------------------------------------------------------------
BenchmarkCalcSignatureHash       11151         12            -99.89%

benchmark                        old ns/op     new ns/op     delta
--------------------------------------------------------------------
BenchmarkCalcSignatureHash       3599845       3056359       -15.10%
2017-08-24 12:48:11 -05:00
Dave Collins
20910511e9
blockchain: Refactor to use new chain view.
- Remove inMainChain from block nodes since that can now be efficiently
  determined by using the chain view
- Track the best chain via a chain view instead of a single block node
  - Use the tip of the best chain view everywhere bestNode was used
  - Update chain view tip instead of updating best node
- Change reorg logic to use more efficient chain view fork finding logic
- Change block locator code over to use more efficient chain view logic
  - Remove now unused block-index-based block locator code
  - Move BlockLocator definition to chain.go
  - Move BlockLocatorFromHash and LatestBlockLocator to chain.go
    - Update both to use more efficient chain view logic
- Rework IsCheckpointCandidate to use block index and chain view
- Optimize MainChainHasBlock to use chain view instead of hitting db
  - Move to chain.go since it no longer involves database I/O
  - Removed error return since it can no longer fail
- Optimize BlockHeightByHash to use chain view instead of hitting db
  - Move to chain.go since it no longer involves database I/O
  - Removed error return since it can no longer fail
- Optimize BlockHashByHeight to use chain view instead of hitting db
  - Move to chain.go since it no longer involves database I/O
  - Removed error return since it can no longer fail
- Optimize HeightRange to use chain view instead of hitting db
  - Move to chain.go since it no longer involves database I/O
- Optimize BlockByHeight to use chain view for main chain check
- Optimize BlockByHash to use chain view for main chain check
2017-08-23 23:43:37 -05:00
Jim Posen
08955805d5 blockmanager: Remove serverPeer from blockmanager completely.
The purpose is to remove the dependency of blockmanager on serverPeer,
which is defined in the main package. Instead, we split out some of
the fields from serverPeer into a separate struct called peerSyncState
in blockmanager.go. While they are in the same package now, this
change makes it easier to move blockManager into its own package along
with peerSyncState. The blockManager tracks a map of Peer pointers to
the peer state and keeps it updated as peers connect and disconnect.
2017-08-23 10:02:12 -07:00
Jim Posen
088ccfd828 blockmanager: Remove dependency on cfg in main package. 2017-08-23 11:45:25 -05:00
Janus Troelsen
45ea940039 docs: Correct rpcclient link. 2017-08-21 15:56:57 -05:00
Dave Collins
2d84a74d28
blockchain: Correct chainview locator comment. 2017-08-21 14:16:31 -05:00
Dave Collins
991a72e34e
blockchain: Remove unused verify disable code.
This removes the DisableVerify function and related state since nothing
uses it anymore since the command line option was removed.  It is a
remnant of initial development.
2017-08-21 04:10:23 -05:00
Dave Collins
1a947c46b2
blockchain: Faster chain view block locator.
This exposes the ability to more efficiently create a block locator from
a chain view for a given block node by using their ability to do O(1)
lookups.

It also adds tests to ensure the behavior is correct.
2017-08-19 23:58:04 -05:00
Dave Collins
93d8dfc760
blockchain: Optimize block locator generation.
This significantly optimizes and simplifies the generation of block
locators by making use of the fact that all block nodes are now in
memory and therefore it is no longer necessary to consult the database
for the hashes or worry about issues related to dynamic loading of nodes.

Also, it slightly modifies the algorithm so that the doubling doesn't
start for one additional iteration in order to mirror other prominent
clients on the network.  Due to the way block locators are used, this
does not change any semantics in terms of requesting and locating
blocks.

Finally, the semantics of BlockLocatorFromHash have been changed to
return a locator for the current tip in the case the hash is unknown.
This is far preferable since only including the passed block hash, when
it isn't known, could end up leading to causing a redownload of the
entire chain under certain circumstances.
2017-08-19 23:19:58 -05:00
Dave Collins
e02fbcf5a1
blockchain: Consolidate tests into package.
Putting the test code in the same package makes it easier for forks
since they don't have to change the import paths as much and it also
gets rid of the need for internal_test.go to bridge.

While here, remove the reorganization test since it is much better
handled by the full block tests and is no longer needed and do some
light cleanup on a few other tests.

The full block tests had to remain in the separate test package since it
is a circular dependency otherwise.  This did require duplicating some
of the chain setup code, but given the other benefits this is
acceptable.
2017-08-19 23:05:17 -05:00
Dave Collins
f4fe6c373e
blockchain: Convert seq lock tests to synthetic.
This introduces the concept of a synthetic block chain that can be used
in the tests to avoid needing setup a full blown chain instance with a
database and generate valid blocks and converts the sequence lock tests
in TestCalcSequenceLock to use it.

Not only does this speed up the test execution time, but it allows the
dependency on rpctest to be removed which will allow the sequence locks
tests to be consolidated into the main package without creating a
circular dependency.
2017-08-19 22:29:22 -05:00
Dave Collins
2a4be16b6c
blockchain: Improve and correct chainview set tip.
This modifies the function to set the tip in the new chainview code to
bulk copy existing nodes when it needs to expand the cap rather than
simply creating a new empty slice and allowing the walk code below it to
repopulate it.  This is a nice optimization since, in practice, most of
the time expanding the cap is only required when the active chain is
being extended after having run for a while which means the end result
is that it will be able to bulk copy all the nodes and just add the most
recent one versus having to walk them all and add them back.

Also, while here expand the tests for setting the tip to ensure the
nodes contained in the resulting view are correct after forcing the
resizes and correct a bug they exposed where changing between a
longer-shorter-longer chain where the longer chain is the same chain
could result in not populating the view correctly.

Finally, update the fake nodes generated by the tests to use a
nonce generated by a deterministic prng in order to ensure the hashes of
all fake nodes are unique, but reproducible.
2017-08-19 22:17:30 -05:00
Anatoli Babenia
2804f4cffe Work on review comments
https://github.com/btcsuite/btcd/pull/1007#discussion_r133563489
2017-08-19 21:10:58 -05:00
Anatoli Babenia
55e0d5c298 Show info when JSON-RPC is not available 2017-08-19 21:10:58 -05:00
Dave Collins
2a753ae9c7
btcec: Regenerate and update precomputed data.
This regenerates the precomputed secp256k1 byte points used to optimize
scalar multiplication.  This should have been done as part of the
normalization correction.
2017-08-18 12:53:28 -05:00
Dave Collins
1d77a611e9
blockchain: Implement new chain view.
This implements a new type in the blockchain package that takes
advantage of the fact that all block nodes are now in memory to provide
a flat view of a specific chain of blocks (a specific branch of the
overall block tree) from a given tip all the way back to the genesis
block along with several convenience functions such as efficiently
comparing two views, quickly finding the fork point (if any) between two
views, and O(1) lookup of the node at a specific height.

The view is not currently used, but the intent is that the code will be
refactored to make use of these views to simplify and optimize several
areas such as best chain selection and reorg logic and finding successor
nodes.  They will also greatly simplify the process of disconnecting the
download logic from the connection logic.

A comprehensive suite of tests is provided to ensure the chain views
behave correctly.
2017-08-17 17:04:39 -05:00
Dave Collins
7e5e6b4610
blockchain: Cleanup a few sequence lock tests.
This cleans up the test in TestCalcSequenceLock in the following ways:
- Use calculated values instead of magic value so it is easier to update
  the tests as needed
- Make tests match the comments
- Change comments to be more consistent and fix some grammar errors
- Set mempool flag for unconfirmed tx tests since they are intended to
  mimic transactions in the mempool
2017-08-16 15:54:03 -05:00
Dave Collins
614b799198
rpcclient: Merge btcrpcclient repo. 2017-08-15 20:09:19 -05:00
Dave Collins
074b2374b8
Import btcrpcclient repo into rpcclient directory.
This commit contains the entire btcrpcclient repository along with
several changes needed to move all of the files into the rpcclient
directory in order to prepare it for merging.  This does NOT update btcd
or any of the other packages to use the new location as that will be
done separately.

- All import paths in the old btcrpcclient files have been changed to
  the new location
- All references to btcrpcclient as the package name have been changed to
  rpcclient
2017-08-15 19:51:58 -05:00
Dave Collins
72f2a3fe49
blockchain: Optimize checkpoint handling.
This modifies the code that determines the most recently known
checkpoint to take advantage of recent changes which make the entire
block index available in memory by only storing a reference to the
specific node in the index that represents the latest known checkpoint.

Previously, the entire block was stored and new checkpoints required
loading it from the database.
2017-08-15 17:06:16 -05:00
Dave Collins
349450379f
blockchain: Remove threshold state db cache.
This completely removes the threshold state database caching code since
it can very quickly be calculated at startup now that the entire block
index is loaded first.
2017-08-15 16:55:40 -05:00
Dave Collins
296fa0a5a0
blockchain: Convert to full block index in mem.
This reworks the block index code such that it loads all of the headers
in the main chain at startup and constructs the full block index
accordingly.

Since the full index from the current best tip all the way back to the
genesis block is now guaranteed to be in memory, this also removes all
code related to dynamically loading the nodes and updates some of the
logic to take advantage of the fact traversing the block index can
longer potentially fail.  There are also more optimizations and
simplifications that can be made in the future as a result of this.

Due to removing all of the extra overhead of tracking the dynamic state,
and ensuring the block node structs are aligned to eliminate extra
padding, the end result of a fully populated block index now takes quite
a bit less memory than the previous dynamically loaded version.

The main downside is that it now takes a while to start whereas it was
nearly instant before, however, it is much better to provide more
efficient runtime operation since that is its ultimate purpose and the
benefits far outweigh this downside.

Some benefits are:

- Since every block node is in memory, the recent code which
  reconstructs headers from block nodes means that all headers can
  always be served from memory which is important since the majority of
  the network has moved to header-based semantics
- Several of the error paths can be removed since they are no longer
  necessary
- It is no longer expensive to calculate CSV sequence locks or median
  times of blocks way in the past
- It will be possible to create much more efficient iteration and
  simplified views of the overall index
- The entire threshold state database cache can be removed since it is
  cheap to construct it from the full block index as needed

An overview of the logic changes are as follows:

- Move AncestorNode from blockIndex to blockNode and greatly simplify
  since it no longer has to deal with the possibility of dynamically
  loading nodes and related failures
- Rename RelativeNode to RelativeAncestor, move to blockNode, and
  redefine in terms of AncestorNode
- Move CalcPastMedianTime from blockIndex to blockNode and remove no
  longer necessary test for nil
- Change calcSequenceLock to use Ancestor instead of RelativeAncestor
  since it reads more clearly
2017-08-15 15:42:34 -05:00
Jim Posen
28606122c3 main: Reduce shared state between server and blockManager.
Instead of having both server and blockManager be aware of the
txProcessed and blockProcessed channels, now the server passed them as
method arguments to blockProcessor.
2017-08-15 15:41:59 -05:00
Jim Posen
1b50c7300f peer: Test that witnessEnabled is set properly on peer connection. 2017-08-15 12:08:49 -07:00
Jim Posen
095bba1a25 peer: Move IsWitnessEnabled() from serverPeer to Peer.
serverPeer is a problematic struct because it is local to the main
package.
2017-08-15 11:34:17 -07:00
Dave Collins
3b5c2baa2e
blockchain: Consistency pass on subscribe code.
This takes care of a few minor nits on the recently merged subscribe
code.  In particular:

- Avoid extra unnecessary allocation on notifications slice
- Avoid defer overhead on notification mutex in sendNotifications
- Make test function comment start with the name of the function per
  Effective Go guidelines
- Use constant for number of subscribers in test
- Don't exceed column 80 in test print
2017-08-15 11:12:56 -05:00
Dave Collins
c5c46376ba
rpcserver: Refactor listener logic to server.
This refactors the RPC server to accept and take ownership of already
configured listeners and refactors the logic to setup those listeners to
the server.  This mirrors the logic used by the connection manager and
is desirable since it is another step closer to being able to split the
RPC server code out into a separate package and will make it much easier
to internally test since it allows creating mock listeners.
2017-08-15 01:07:38 -05:00
Dave Collins
ff700325ac
rpcserver: Don't use activeNetParams.
This modifies all of the RPC code to use the chain parameters that are
associated with the RPC server instead of the global activeNetParams and
thus moves one step closer to being able to split the RPC server out
into a separate package.
2017-08-15 00:48:41 -05:00
Dave Collins
a7a1029445
rpcserver: Decouple from server.
This decouples the RPC server from the internal btcd server to move
closer to being able to split it out into a separate package.

In order to accomplish this, it introduces an rpcserverConfig type and
several new interfaces, named rpcserverPeer, rpcserverConnManager, and
rpcserverBlockManager, which are necessary to break the direct
dependencies on the main server and block manager instances.

It also adds concrete implementations of the new interfaces and uses
them to configure the RPC server.

Ultimately, the RPC server should ideally be decoupled even more such
that all of the types in the configuration struct use interfaces instead
of the concrete types.  Doing this would make the RPC server much easier
to internally test since it would allow creating lightweight stubs for
the various pieces.
2017-08-14 23:01:07 -05:00
Jim Posen
ca9baf77ec blockchain: Fix race condition in notifications_test 2017-08-14 22:48:32 -05:00
Jim Posen
b71d6c3010 Create blockManagerConfig struct passed to newBlockManager.
The config struct accepts an instance of server as an implementation
of the new PeerNotifier wrapping interface.
2017-08-14 20:19:02 -07:00
Jim Posen
49949d4c96 Remove references from blockManager to rpcServer.
Instead of having the block manager notify the RPC server about
accepted, connected, and disconnected blocks, the RPC server will
directly listen for notifications from the blockchain.
2017-08-14 20:14:42 -07:00
Jim Posen
22de1f6d08 Create blockmanager with reference to txMemPool.
The objective is to remove the reference from blockManager to
server. Since the blockManager is responsible for keeping the mempool
in sync, it should have a direct reference to it.
2017-08-14 20:14:42 -07:00
Jim Posen
19b42c3c9b Allow multiple subscribers to blockchain notifications.
The BlockChain struct emits notifications for various events, but
it is only possible to register one listener. This changes the
interface and implementations to allow multiple listeners.
2017-08-14 22:03:24 -05:00
Jim Posen
bc36cf51c6 peer: Don't send unsupported reject to old peers. 2017-08-14 16:58:57 -05:00
Dave Collins
05bdf3b741
mempool: Remove potential negative locktime check.
This removes the standardness check to reject transactions with a lock
time greater than a maxint32 because the old bitcoind nodes which it was
designed to protect against are no longer valid for other reasons and
thus there are no longer any of them on the network to worry about.
2017-08-14 16:49:08 -05:00
Jim Posen
f4d376d7af peer: Improve address sampling in PushAddrMsg.
This changes fixes shuffling algorithm in PushAddrMsg to select a
uniformly random subset of the given addresses using Fisher-Yates.
2017-08-14 15:44:08 -05:00