Commit graph

282 commits

Author SHA1 Message Date
Dave Collins
8c883d1fca
blockchain/indexers: Allow interrupts.
This propagates the interrupt channel through to blockchain and the
indexers so that it is possible to interrupt long-running operations
such as catching up indexes.
2017-09-05 11:02:46 -05:00
Jim Posen
30d4caeac6 server: Simplify logic to bind listeners.
The helper function parseListeners has been changed to return a
slice of net.Addrs with Network() returning tcp4 or tcp6 instead of
returning two slices of IPv4 and IPv6 addresses to simplify calling
code. Also improves how local addresses are added to the address
manager when listening on wildcard addresses.

Also splits some newServer logic into new method initListeners.
2017-09-04 11:55:37 -07:00
David Hill
a2085c68f8 config: Add --whitelist support. 2017-08-31 09:59:43 -04:00
Dave Collins
9bd7bcfff6
travis: Update to go 1.8 and 1.9.
Also, make the gosimple linter happy while here.
2017-08-30 01:03:26 -05:00
Andy Weidenbaum
a1d1ea70dd rm extra word segwit, s/segwit segwit/segwit 2017-08-26 10:56:46 -07:00
Jim Posen
46fd4ec358 netsync: Change name of blockManager to syncManager. 2017-08-25 13:41:22 -05:00
Dave Collins
34b1373a68
blockchain: Refactor inv discovery for chain view.
This refactors the code that locates blocks (inventory discovery) out of
server and into blockchain where it can make use of the new much more
efficient chain view and more easily be tested.  As an aside, it really
belongs in blockchain anyways since it's purely dealing with the block
index and best chain.

Since the majority of the network has moved to header-based semantics,
this also provides an additional optimization to allow headers to be
located directly versus needing to first discover the hashes and then
fetch the headers.

The new functions are named LocateBlocks and LocateHeaders.  The former
returns a slice of located hashes and the latter returns a slice of
located headers.

Finally, it also updates the RPC server getheaders call and related
plumbing to use the new LocateHeaders function.

A comprehensive suite of tests is provided to ensure both functions
behave correctly for both correct and incorrect block locators.
2017-08-24 14:15:34 -05:00
Jim Posen
08955805d5 blockmanager: Remove serverPeer from blockmanager completely.
The purpose is to remove the dependency of blockmanager on serverPeer,
which is defined in the main package. Instead, we split out some of
the fields from serverPeer into a separate struct called peerSyncState
in blockmanager.go. While they are in the same package now, this
change makes it easier to move blockManager into its own package along
with peerSyncState. The blockManager tracks a map of Peer pointers to
the peer state and keeps it updated as peers connect and disconnect.
2017-08-23 10:02:12 -07:00
Jim Posen
088ccfd828 blockmanager: Remove dependency on cfg in main package. 2017-08-23 11:45:25 -05:00
Jim Posen
28606122c3 main: Reduce shared state between server and blockManager.
Instead of having both server and blockManager be aware of the
txProcessed and blockProcessed channels, now the server passed them as
method arguments to blockProcessor.
2017-08-15 15:41:59 -05:00
Jim Posen
095bba1a25 peer: Move IsWitnessEnabled() from serverPeer to Peer.
serverPeer is a problematic struct because it is local to the main
package.
2017-08-15 11:34:17 -07:00
Dave Collins
c5c46376ba
rpcserver: Refactor listener logic to server.
This refactors the RPC server to accept and take ownership of already
configured listeners and refactors the logic to setup those listeners to
the server.  This mirrors the logic used by the connection manager and
is desirable since it is another step closer to being able to split the
RPC server code out into a separate package and will make it much easier
to internally test since it allows creating mock listeners.
2017-08-15 01:07:38 -05:00
Dave Collins
a7a1029445
rpcserver: Decouple from server.
This decouples the RPC server from the internal btcd server to move
closer to being able to split it out into a separate package.

In order to accomplish this, it introduces an rpcserverConfig type and
several new interfaces, named rpcserverPeer, rpcserverConnManager, and
rpcserverBlockManager, which are necessary to break the direct
dependencies on the main server and block manager instances.

It also adds concrete implementations of the new interfaces and uses
them to configure the RPC server.

Ultimately, the RPC server should ideally be decoupled even more such
that all of the types in the configuration struct use interfaces instead
of the concrete types.  Doing this would make the RPC server much easier
to internally test since it would allow creating lightweight stubs for
the various pieces.
2017-08-14 23:01:07 -05:00
Jim Posen
b71d6c3010 Create blockManagerConfig struct passed to newBlockManager.
The config struct accepts an instance of server as an implementation
of the new PeerNotifier wrapping interface.
2017-08-14 20:19:02 -07:00
Jim Posen
49949d4c96 Remove references from blockManager to rpcServer.
Instead of having the block manager notify the RPC server about
accepted, connected, and disconnected blocks, the RPC server will
directly listen for notifications from the blockchain.
2017-08-14 20:14:42 -07:00
Jim Posen
22de1f6d08 Create blockmanager with reference to txMemPool.
The objective is to remove the reference from blockManager to
server. Since the blockManager is responsible for keeping the mempool
in sync, it should have a direct reference to it.
2017-08-14 20:14:42 -07:00
Olaoluwa Osuntokun
1244c45b88 mining+config: modify GBT mining to limit by weight, add witness commitment
This commit modifies the existing block selection logic to limit
preferentially by weight instead of serialized block size, and also to
adhere to the new sig-op cost limits which are weighted according to
the witness discount.
2017-08-13 23:17:40 -05:00
Olaoluwa Osuntokun
26ff8ddce4 mempool: modify mempool sanity checks to be segwit aware 2017-08-13 23:17:40 -05:00
Olaoluwa Osuntokun
0db14c740b BIP0144: properly fetch witness data from witness-enabled peers
This commit modifies the logic within the block manager and service to
preferentially fetch transactions and blocks which include witness data
from fully upgraded peers.

Once the initial version handshake has completed, the server now tracks
which of the connected peers are witness enabled (they advertise
SFNodeWitness). From then on, if a peer is witness enabled, then btcd
will always request full witness data when fetching
transactions/blocks.
2017-08-13 23:17:40 -05:00
Ricardo Velhote
948d80b198
New RPC command to display the uptime of the server
Version 0.15.0 of Bitcoin Core will include a new RPC command that will
allow us to obtain the amount of time (in seconds) that the server has
been running.
2017-07-03 00:07:27 +01:00
Steven Roose
bf43e56f2f Fix warnings from ineffassign
I left one at the end of fullblocktest, since I suspected the unused
variable assignments there were set for the possibility of extending the
tests.
2017-06-07 17:59:33 -05:00
Steven Roose
3d0dfed40b Fix a ton of typos accumulated over time 2017-05-30 16:59:51 +02:00
Steven Roose
53f55a4634 config: Add user agent comments flag --uacomment
Just like Core's -uacomment, this flag allows to specify user agent
comments like defined in BIP 14.
2017-05-17 13:22:26 +02:00
Dave Collins
d06c0bb181
blockchain: Use hash values in structs.
This modifies the blockNode and BestState structs in the blockchain
package to store hashes directly instead of pointers to them and updates
callers to deal with the API change in the exported BestState struct.

In general, the preferred approach for hashes moving forward is to store
hash values in complex data structures, particularly those that will be
used for cache entries, and accept pointers to hashes in arguments to
functions.

Some of the reasoning behind making this change is:

- It is generally preferred to avoid storing pointers to data in cache
  objects since doing so can easily lead to storing interior pointers
  into other structs that then can't be GC'd
- Keeping the hash values directly in the block node provides better
  cache locality
2017-02-03 11:36:33 -06:00
Dave Collins
59169540c3
blockchain: Reconstruct headers from block nodes.
This modifies the block node structure to include a couple of extra
fields needed to be able to reconstruct the block header from a node,
and exposes a new function from chain to fetch the block headers which
takes advantage of the new functionality to reconstruct the headers from
memory when possible.  Finally, it updates both the p2p and RPC servers
to make use of the new function.

This is useful since many of the block header fields need to be kept in
order to form the block index anyways and storing the extra fields means
the database does not have to be consulted when headers are requested if
the associated node is still in memory.

The following timings show representative performance gains as measured
from one system:

new: Time to fetch 100000 headers:   59ms
old: Time to fetch 100000 headers: 4783ms
2017-02-01 12:27:56 -06:00
David Hill
ecd348b2a7 btcd: disable mempool requests if bloom filtering is disabled 2017-01-27 20:51:25 -06:00
David Hill
db5b9aef91 btcd: fix rebroadcasting of local txs. 2017-01-27 10:28:48 -05:00
David Hill
9bedd7720c btcd: only allow one getaddr request per peer 2017-01-25 12:05:55 -05:00
Javed Khan
8caa921ac3 server: Fix connections to onion nodes.
A DNS lookup was being attempted on onion addresses causing
connections to fail. This has been fixed by introducing type
onionAddr (which implements a net.Addr interface) and passing
it to btcdDial.

Also, the following onion related fixes have been made:

* getaddednodeinfo - updated to handle onion addrs.
* TorLookupIP - fixed err being shadowed.
* newServer - rename tcpAddr to netAddr
* addrStringToNetAddr - skip if host is already an IP addr.
* addrStringToNetAddr - err if tor is disabled
* getaddednodeinfo - check if host is already an IP addr.
2017-01-18 21:40:41 +05:30
Alex
7c44b6472f Port getheaders JSON-RPC command from dcrd 2017-01-10 19:11:18 -07:00
David de Kloet
4021ae2f6e server.go: Optimize newAddressFunc
Change the order of conditions to avoid calling fmt.Sprintf
unnecessarily.
2016-12-08 22:55:43 +01:00
Steven Roose
95e6de00b8 btcd: fix error in mempool response inv counting 2016-11-25 10:30:24 +01:00
dskloet
e1b2ceca80 addrmgr: Remove unused param from GetAddress()
addrmgr.GetAddress() had a parameter `class string` originally intended
to support looking up addresses according to some type of filter such as
IPv4, IPv6, and only those which support specific wire.ServiceFlags
(full nodes, nodes that support bloom filters, nodes that support
segwit, etc). But currently the parameter is unused and also has an
inappropriate type `string`.

If it would ever be used, it's easy to add back and should then get an
appropriate type such as something that allows bitflags to be set so
that the caller could request combinations such as peers that support
IPv6, are full nodes, and support bloom filters.
2016-11-21 08:56:46 -06:00
David Hill
807d344fe9 Unassign some TODO's 2016-11-15 17:47:33 -06:00
Dave Collins
0d4546c965
server: Return error on address lookup fails.
This corrects an issue introduced by commit
e8f63bc295 where a failure to lookup a
hostname could lead to a panic in certain circumstances.  An error is
now returned in that case as expected.
2016-11-11 20:54:25 -06:00
Olaoluwa Osuntokun
e8f63bc295
connmgr: switch to using net.Addr interface throughout for addresses
This commit modifies the `ConnManager` to use the `net.Add` interface
through the package instead of a plain string to represent and
manipulate addresses. This change makes the package much more general as
users of the package can possibly utilize custom implementations of the
`net.Addr` interface to establish connections.

More precisely, the `ConnReq` struct has been modified to use a net.Addr
instance explicitly, and the `DialFunc` type has also been modified to
take a `net.Addr` directly. This latter change gives functions that
adhere to the `DialFunc` type more flexibility as to exactly how the
connection is established.

Additionally, the `connmgr.Config.GetNewAddress` configuration option
now directly returns a `net.Addr. This change allows the `connmgr` to be
decoupled from all DNS queries which allows callers to preferentially
select more secure methods like performing DNS lookups over a Tor proxy.
2016-11-10 11:22:36 -08:00
Dave Collins
df33d4340e
server: Ensure callbacks use the server peer.
This modifies the signatures of all serverPeer callbacks that are
provided as peer.Listeners to use _ for the first parameter name which
ensures the passed peer can't be used within the function and updates
all references to the server peer.

This helps ensure any overridden methods that might be defined on a
serverPeer will be invoked where directly calling methods on the passed
peer would not.

Also, while here, add a comment to the OnFeeFilter function.
2016-11-04 13:38:13 -05:00
Dave Collins
d98430d8ca
connmgr: Implement inbound connection handling.
This modifies the connection manager to provide support for accepting
inbound connections on a caller-provided set of listeners and notify the
caller via a callback.

This is only the minimum work necessary to get inbound support into the
connection manager.  The intention for future commits is to move more
connection-related logic such as limiting the maximum number of overall
connections and banned peer tracking into the connection manager.
2016-11-04 13:14:44 -05:00
Dave Collins
ea9bf748bb
connmgr: Remove type defs for callbacks.
This removes the type definitions for the callback functions in favor of
declaring them directly in the Config struct.  This is more consistent
with the rest of the code base and is preferred since it means callers
reviewing the documentation don't have to follow another level of
indirection to figure out the signature.
2016-11-04 13:14:39 -05:00
Javed Khan
a041b4349b server: Use Disconnect in handleDonePeerMsg 2016-11-04 00:36:13 -05:00
Javed Khan
b320129e9b server: Check max peers before connecting 2016-11-04 00:36:13 -05:00
Javed Khan
aca9fc040c connmgr: Rename max outbound to target outbound 2016-11-04 00:36:13 -05:00
David Hill
2510baac35 btcd: support feefilter requests.
This only adds support for handling remote peer requests.
2016-11-03 14:47:30 -04:00
Dave Collins
915fa6639b
multi: Simplify code per gosimple linter.
This simplifies the code based on the recommendations of the gosimple
lint tool.
2016-11-03 13:00:35 -05:00
Javed Khan
d1c39edee8 server: Cap max outbound in connmgr cfg at max peers 2016-11-02 15:10:14 -05:00
Dave Collins
6d5714e1b7
server/mempool: Evict orphans on peer disconnect.
This removes any remaining orphan transactions that were sent by a peer
when it disconnects since it is extremely unlikely that the missing
parents will ever materialize from elsewhere.
2016-10-28 15:27:57 -05:00
David Hill
2615fa0849 mempool: Return type TxDesc instead of type btcutil.Tx
This will provide callers more information on the accepted transaction.
2016-10-28 14:52:31 -04:00
Dave Collins
1a69eb0617
cpuminer: Refactor code to its own package.
This does the minimum work necessary to refactor the CPU miner code into
its own package.  The idea is that separating this code into its own
package will improve its testability and ultimately be useful to other
parts of the codebase such as the various tests which currently
effectively have their own stripped-down versions of this code.

The API will certainly need some additional cleanup and changes to make
it more usable outside of the specific circumstances it was originally
designed to support (namely the generate RPC), however it is better to
do that in future commits in order to keep the changeset as small as
possible during this refactor.

Overview of the major changes:

- Create the new package
- Move cpuminer.go -> cpuminer/cpuminer.go
- Update mining logging to use the new cpuminer package logger
- Rename cpuminerConfig to Config (so it's now cpuminer.Config)
- Rename newCPUMiner to New (so it's now cpuminer.New)
- Update all references to the cpuminer to use the package
- Add a skeleton README.md
2016-10-28 11:06:11 -05:00
Dave Collins
214d975adf
server: Notify connmgr if server peer assoc fails.
This corrects a few issues introduced with the connection manager where
the server was not notifying the connection manager when a connection
request is available again.

The cases resolved are:
- Unable to initialize a server peer instance in response to the connection
- Failure to associate the connection with the server peer instance
- Disconnection of a non-persistent outbound peer

It also changes the log message to a debug in the former case because
it's not something that should be shown to the user as an error given
it's not due to anything the user has misconfigured nor is it even
unexpected if an invalid address is provided.
2016-10-28 10:49:00 -05:00
Dave Collins
14b51fc5f8
multi: Correct misspellings detected by misspell. 2016-10-28 09:43:38 -05:00
Dave Collins
f3c442deb2
server: Remove unused wakeup channel.
This channel is no longer used since the outgoing connection logic is
handled by the connmgr package now.
2016-10-27 20:04:58 -05:00
Dave Collins
61ca40e0e9
mining: Refactor template code into mining package.
This does the minimum work necessary to refactor the block template
generation code into the mining package.  The idea is that separating
this code into the mining package will greatly improve its testability,
allow independent benchmarking and profiling, and open up some
interesting opportunities for future development related to mining.

There are some areas related to policy and other configuration that
could be further refactored, however it is better to do that in future
commits in order to keep the changeset as small as possible during this
refactor.

Overview of the major changes:

- Move mining.go -> mining/mining.go
- Move mining_test.go -> mining/mining_test.go
- Add logger to mining package
- Update the MINR subsystem to use the new mining package logger
- Export CoinbaseFlags from the mining package
- BlkTmplGenerator is now mining.BlkTmplGenerator
- Update all references to the mining code to use the package
2016-10-27 11:48:48 -05:00
Dave Collins
660467259e
mining: Break dependency on block manager instance.
This modifies the block template generate for the mining code such that
it takes chain instance and params instead of requiring a fully
initialized blockManager instance.

Also, in preparation for being able to more easily separate the code, it
exposes and makes use of two new functions:
- BestSnapshot which returns the state snapshot from the underlying
  chain instance
- TxSource which returns the underlying transaction source

This is a step towards being able to separate the mining code into its
own package.  No functional change.
2016-10-27 11:48:32 -05:00
Dave Collins
671901486c
cpuminer: Introduce cpuminerConfig.
This introduces a cpuminerConfig type which contains the necessary
information to break the direct dependency on the main server instance.

This change is a step towards being able to separate the cpu miner into
its own subpackage.  No functional change.
2016-10-27 09:37:30 -05:00
Olaoluwa Osuntokun
e3eeb4a34a
mempool: add policy config option for transaction version
This commit adds a new option to the mempool’s policy configuration
which determines which transaction versions should be accepted as
standard.

The default version set by the policy within the server is 2; this
allows accepting transactions which have version 2 enabled in order to
utilize the new sequence locks feature.
2016-10-26 21:49:14 -07:00
Olaoluwa Osuntokun
7eb0ab5f8d
mempool: add function to config for computing sequence locks 2016-10-26 21:49:04 -07:00
Dave Collins
74fe2a4dfd
mining: Introduce a block template generator.
This introduces a new type named BlkTmplGenerator which encapsulates the
various state needed to generate block templates.

This is useful since it means code that needs to generate block
templates can simply accept the generator rather than needing access to
all of the additional state which in turn will ultimately make it easier
to split the mining code into its own package.
2016-10-26 15:17:21 -05:00
Dave Collins
26e22790cd
mempool: Rename RelayNonStd config option.
This renames the mempool.Config.RelayNonStd option to AcceptNonStd which
more accurately describes its behavior since the mempool was refactored
into a separate package.

The reasoning for this change is that the mempool is not responsible for
relaying transactions (nor should it be).  Its job is to maintain a pool
of unmined transactions that are validated according to consensus and
policy configuration options which are then used to provide a source of
transactions that need to be mined.

Instead, it is the server that is responsible for relaying transactions.
While it is true that the current server code currently only relays txns
that were accepted to the mempool, this does not necessarily have to
be the case.  It would be entirely possible (and perhaps even a good
idea as something do in the future), to separate the relay policy from
the mempool acceptance policy (and thus indirectly the mining policy).
2016-10-23 20:41:54 -05:00
David Hill
f161d6b69e chaincfg: Introduce new type DNSSeed
DNSSeed defines a DNS Seed with a hostname and whether it supports
filtering by service flag bits.
2016-10-23 15:59:15 -05:00
David Hill
0d508e6522 peer: BestLocalAddress config option is no longer used. 2016-10-23 16:38:45 -04:00
David Hill
bca9877796 Do not add inbound peers to the address manager. 2016-10-23 16:37:44 -04:00
Javed Khan
bff2ba70fd connmgr: Refactor connection management into pkg
This commit introduces package connmgr which contains connection
management related functionality.

The following is an overview of the features the package provides:

- Maintain fixed number of outbound connections
- Optional connect-only mode
- Retry persistent connections with increasing back-off
- Source peers from DNS seeds
- Use Tor to resolve DNS
- Dynamic ban scores
- Test coverage

In addition, btcd has been refactored to make use of the new package by
extending the connection manager to work with the server to source and
maintain peer connections. The following is a broad overview of the
changes to integrate the package:

- Simplify peer state by removing pending, retry peers
- Refactor to remove retries which are now handled by connmgr
- Use callback to add addresses sourced from the  DNS seed

Finally the following connection-related things have been improved as a
part of this refactor:

- Fixes 100% cpu usage when network is down (#129)
- Fixes issues with max peers (#577)
- Simplify outbound peer connections management
2016-10-22 01:11:57 -05:00
Javed Khan
f6cd49ac51 peer: rename peer.Connect to AssociateConnection 2016-10-21 14:13:18 -05:00
Olaoluwa Osuntokun
a82f67b538
mempool: add closure to compute median time past to config
This commit adds an additional closure function to the mempool’s config
which computes the median time past from the point of view of the best
node in the chain. The mempool test harness has also been updated to allow
setting a mock median time past for testing purposes.

In addition to increasing the testability of the mempool, this commit
should also speed up transaction and block validation for BIP 113 as
the MTP no longer needs to be re-calculated each time from scratch.
2016-10-19 11:13:25 -07:00
David Hill
b1621332cc Optimize by removing defers
defer's are nice for readability but they do add overhead.  This
gets rid of defer's where it is just as easy as not to use one.
2016-10-18 17:56:51 -04:00
Dave Collins
8965d88893
peer: Strictly enforce bloom filter service bit.
This makes the enforcement of the bloom filter service bit much more
strict.  In particular, it does the following:

- Moves the enforcement of the bloom filter service bit out of the peer
  package and into the server so the server can ban as necessary
- Disconnect peers that send filter commands when the server is
  configured to disable them regardless of the protocol version
- Bans peers that are a high enough protocol version that they are
  supposed to observe the service bit is disabled, but ignore it and
  send filter commands regardless.

As an added bonus, this fixes the old logic which had a bug in that it
was examining the *remote* peer's supported services in order to choose
whether or not to disconnect instead of the *local* server's supported
services.
2016-10-16 02:19:28 -05:00
Olaoluwa Osuntokun
815ded348e
config: introduce new flags to accept/reject non-std transactions
This commit adds two new cli flags: one for accepting non-std
transactions, and the other for rejecting non-std transactions.

The two flag are rejected when using concurrently. Config parsing is
set up such that, the desired policy expressed via the config always
overrides the policy set by default for a particular chain.

The doc.go files and the sample-btcd.conf file have been updated to document
the new flags exposing further policy control.
2016-08-24 15:43:26 -07:00
Olaoluwa Osuntokun
dc5486a579
mempool: add non-standard tx relaying to policy config 2016-08-24 15:43:22 -07:00
Dave Collins
641182b2ad
mempool: Break dependency on chain instance.
This modifies the config for the new mempool package such that it takes
a callback function to obtain the best chain height instead of requiring
a fully initialized blockchain.BlockChain instance.

This will make it much easier to test the mempool since the tests will
be able to provide their own height function to test various
functionality without having create and manipulate full blocks and chain
instances.
2016-08-23 12:29:45 -05:00
jadeblaquiere
47ced81d44 don't spin forever looking for peers (#724) 2016-08-20 22:11:12 -05:00
Dave Collins
7fac099bee mempool: Refactor mempool code to its own package. (#737)
This does the minimum work necessary to refactor the mempool code into
its own package.  The idea is that separating this code into its own
package will greatly improve its testability, allow independent
benchmarking and profiling, and open up some interesting opportunities
for future development related to the memory pool.

There are likely some areas related to policy that could be further
refactored, however it is better to do that in future commits in order
to keep the changeset as small as possible during this refactor.

Overview of the major changes:

- Create the new package
- Move several files into the new package:
  - mempool.go -> mempool/mempool.go
  - mempoolerror.go -> mempool/error.go
  - policy.go -> mempool/policy.go
  - policy_test.go -> mempool/policy_test.go
- Update mempool logging to use the new mempool package logger
- Rename mempoolPolicy to Policy (so it's now mempool.Policy)
- Rename mempoolConfig to Config (so it's now mempool.Config)
- Rename mempoolTxDesc to TxDesc (so it's now mempool.TxDesc)
- Rename txMemPool to TxPool (so it's now mempool.TxPool)
- Move defaultBlockPrioritySize to the new package and export it
- Export DefaultMinRelayTxFee from the mempool package
- Export the CalcPriority function from the mempool package
- Introduce a new RawMempoolVerbose function on the TxPool and update
  the RPC server to use it
- Update all references to the mempool to use the package.
- Add a skeleton README.md
2016-08-19 11:08:37 -05:00
Dave Collins
87b3756c8c server: Remove superfluous check in OnMemPool. (#736)
This reduces the mempool lock contention by removing an unnecessary
check when responding to a "mempool" request.

In particular, the code first gets a list of all transactions from the
mempool and then iterates them in order to construct the inventory
vectors and apply bloom filtering if it is enabled.  Since it is
possible that the transaction was removed from the mempool by another
thread while that list is being iterated, the code was checking if each
transaction was still in the mempool.  This is a pointless check because
the transaction might still be removed at any point after the check
anyways.  For example, it might be removed after the mempool response
has been sent to the remote peer or even while the loop is still
iterating.
2016-08-19 11:04:16 -05:00
Dave Collins
044a11c9fc btcd: Simplify shutdown signal handling logic. (#733)
This rewrites the shutdown logic to simplify the shutdown signalling.
All cleanup is now run from deferred functions in the main function and
channels are used to signal shutdown either from OS signals or from
other subsystems such as the RPC server and windows service controller.

The RPC server has been modified to use a new channel for signalling
shutdown that is exposed via the RequestedProcessShutdown function
instead of directly calling Stop on the server as it previously did.

Finally, it adds a few checks for early termination during the main
start sequence so the process can be stopped without starting all the
subsystems if desired.

This is a backport of the equivalent logic from Decred with a few slight
modifications.  Credits go to @jrick.
2016-08-11 13:39:23 -05:00
Dave Collins
bd4e64d1d4 chainhash: Abstract hash logic to new package. (#729)
This is mostly a backport of some of the same modifications made in
Decred along with a few additional things cleaned up.  In particular,
this updates the code to make use of the new chainhash package.

Also, since this required API changes anyways and the hash algorithm is
no longer tied specifically to SHA, all other functions throughout the
code base which had "Sha" in their name have been changed to Hash so
they are not incorrectly implying the hash algorithm.

The following is an overview of the changes:

- Remove the wire.ShaHash type
- Update all references to wire.ShaHash to the new chainhash.Hash type
- Rename the following functions and update all references:
  - wire.BlockHeader.BlockSha -> BlockHash
  - wire.MsgBlock.BlockSha -> BlockHash
  - wire.MsgBlock.TxShas -> TxHashes
  - wire.MsgTx.TxSha -> TxHash
  - blockchain.ShaHashToBig -> HashToBig
  - peer.ShaFunc -> peer.HashFunc
- Rename all variables that included sha in their name to include hash
  instead
- Update for function name changes in other dependent packages such as
  btcutil
- Update copyright dates on all modified files
- Update glide.lock file to use the required version of btcutil
2016-08-08 14:04:33 -05:00
Jonathan Gillham
1ffc3dc18d peer: Fix logging of connected peer. 2016-06-24 13:39:50 +01:00
Jonathan Gillham
777ccdade3 peer: Remove error return from Connect. 2016-06-24 13:12:01 +01:00
David Hill
1b23410214 btcd: sendheaders server support (#671)
This adds support for serving headers instead of inventory messages in
accordance with BIP0130.  btcd itself does not yet make use of the
feature when receiving data.
2016-04-26 13:24:03 -05:00
David Hill
a1bb291b28 mempool: Have ProcessTransaction return accepted transactions. (#547)
It is not the responsibility of mempool to relay transactions, so
return a slice of transactions accepted to the mempool due to the
passed transaction to the caller.
2016-04-14 12:58:09 -05:00
Tadge Dryja
432ad76952 fix memory allignment for 32-bit architectures (#668)
having 3 int32s above the uint64s in the struct
will cause misalignment for some 32-bit architectures.
see https://golang.org/pkg/sync/atomic/#pkg-note-BUG
This aligns bytesReceived and bytesSent.
2016-04-13 22:51:02 -05:00
Dave Collins
b580cdb7d3 database: Replace with new version.
This commit removes the old database package, moves the new package into
its place, and updates all imports accordingly.
2016-04-12 14:55:15 -05:00
Dave Collins
7c174620f7 indexers: Implement optional tx/address indexes.
This introduces a new indexing infrastructure for supporting optional
indexes using the new database and blockchain infrastructure along with
two concrete indexer implementations which provide both a
transaction-by-hash and a transaction-by-address index.

The new infrastructure is mostly separated into a package named indexers
which is housed under the blockchain package.  In order to support this,
a new interface named IndexManager has been introduced in the blockchain
package which provides methods to be notified when the chain has been
initialized and when blocks are connected and disconnected from the main
chain.  A concrete implementation of an index manager is provided by the
new indexers package.

The new indexers package also provides a new interface named Indexer
which allows the index manager to manage concrete index implementations
which conform to the interface.

The following is high level overview of the main index infrastructure
changes:

- Define a new IndexManager interface in the blockchain package and
  modify the package to make use of the interface when specified
- Create a new indexers package
  - Provides an Index interface which allows concrete indexes to plugin
    to an index manager
  - Provides a concrete IndexManager implementation
    - Handles the lifecycle of all indexes it manages
    - Tracks the index tips
    - Handles catching up disabled indexes that have been reenabled
    - Handles reorgs while the index was disabled
    - Invokes the appropriate methods for all managed indexes to allow
      them to index and deindex the blocks and transactions
  - Implement a transaction-by-hash index
    - Makes use of internal block IDs to save a significant amount of
      space and indexing costs over the old transaction index format
  - Implement a transaction-by-address index
    - Makes use of a leveling scheme in order to provide a good tradeoff
      between space required and indexing costs
- Supports enabling and disabling indexes at will
- Support the ability to drop indexes if they are no longer desired

The following is an overview of the btcd changes:

- Add a new index logging subsystem
- Add new options --txindex and --addrindex in order to enable the
  optional indexes
  - NOTE: The transaction index will automatically be enabled when the
    address index is enabled because it depends on it
- Add new options --droptxindex and --dropaddrindex to allow the indexes
  to be removed
  - NOTE: The address index will also be removed when the transaction
    index is dropped because it depends on it
- Update getrawtransactions RPC to make use of the transaction index
- Reimplement the searchrawtransaction RPC that makes use of the address
  index
- Update sample-btcd.conf to include sample usage for the new optional
  index flags
2016-04-11 17:16:42 -05:00
Dave Collins
491acd4ca6 blockchain: Rework to use new db interface.
This commit is the first stage of several that are planned to convert
the blockchain package into a concurrent safe package that will
ultimately allow support for multi-peer download and concurrent chain
processing.  The goal is to update btcd proper after each step so it can
take advantage of the enhancements as they are developed.

In addition to the aforementioned benefit, this staged approach has been
chosen since it is absolutely critical to maintain consensus.
Separating the changes into several stages makes it easier for reviewers
to logically follow what is happening and therefore helps prevent
consensus bugs.  Naturally there are significant automated tests to help
prevent consensus issues as well.

The main focus of this stage is to convert the blockchain package to use
the new database interface and implement the chain-related functionality
which it no longer handles.  It also aims to improve efficiency in
various areas by making use of the new database and chain capabilities.

The following is an overview of the chain changes:

- Update to use the new database interface
- Add chain-related functionality that the old database used to handle
  - Main chain structure and state
  - Transaction spend tracking
- Implement a new pruned unspent transaction output (utxo) set
  - Provides efficient direct access to the unspent transaction outputs
  - Uses a domain specific compression algorithm that understands the
    standard transaction scripts in order to significantly compress them
  - Removes reliance on the transaction index and paves the way toward
    eventually enabling block pruning
- Modify the New function to accept a Config struct instead of
  inidividual parameters
- Replace the old TxStore type with a new UtxoViewpoint type that makes
  use of the new pruned utxo set
- Convert code to treat the new UtxoViewpoint as a rolling view that is
  used between connects and disconnects to improve efficiency
- Make best chain state always set when the chain instance is created
  - Remove now unnecessary logic for dealing with unset best state
- Make all exported functions concurrent safe
  - Currently using a single chain state lock as it provides a straight
    forward and easy to review path forward however this can be improved
    with more fine grained locking
- Optimize various cases where full blocks were being loaded when only
  the header is needed to help reduce the I/O load
- Add the ability for callers to get a snapshot of the current best
  chain stats in a concurrent safe fashion
  - Does not block callers while new blocks are being processed
- Make error messages that reference transaction outputs consistently
  use <transaction hash>:<output index>
- Introduce a new AssertError type an convert internal consistency
  checks to use it
- Update tests and examples to reflect the changes
- Add a full suite of tests to ensure correct functionality of the new
  code

The following is an overview of the btcd changes:

- Update to use the new database and chain interfaces
- Temporarily remove all code related to the transaction index
- Temporarily remove all code related to the address index
- Convert all code that uses transaction stores to use the new utxo
  view
- Rework several calls that required the block manager for safe
  concurrency to use the chain package directly now that it is
  concurrent safe
- Change all calls to obtain the best hash to use the new best state
  snapshot capability from the chain package
- Remove workaround for limits on fetching height ranges since the new
  database interface no longer imposes them
- Correct the gettxout RPC handler to return the best chain hash as
  opposed the hash the txout was found in
- Optimize various RPC handlers:
  - Change several of the RPC handlers to use the new chain snapshot
    capability to avoid needlessly loading data
  - Update several handlers to use new functionality to avoid accessing
    the block manager so they are able to return the data without
    blocking when the server is busy processing blocks
  - Update non-verbose getblock to avoid deserialization and
    serialization overhead
  - Update getblockheader to request the block height directly from
    chain and only load the header
  - Update getdifficulty to use the new cached data from chain
  - Update getmininginfo to use the new cached data from chain
  - Update non-verbose getrawtransaction to avoid deserialization and
    serialization overhead
  - Update gettxout to use the new utxo store versus loading
    full transactions using the transaction index

The following is an overview of the utility changes:
- Update addblock to use the new database and chain interfaces
- Update findcheckpoint to use the new database and chain interfaces
- Remove the dropafter utility which is no longer supported

NOTE: The transaction index and address index will be reimplemented in
another commit.
2016-04-11 16:47:27 -05:00
David Hill
123ff368f4 mempool: Create and use mempoolPolicy. (#571)
mempoolPolicy contains the values that configure the mempool policy.
This decouples the values from the internals of btcd to move closer
to a mempool package.
2016-04-11 16:37:52 -05:00
David Hill
d1e493f4ee config: New option --blocksonly (#553)
The --blocksonly configuration option disables accepting transactions
from remote peers.  It will still accept, relay, and rebroadcast
valid transactions sent via RPC or websockets.
2016-04-07 18:16:46 -05:00
David Hill
7b31349023 Cleanup and optimize handleBroadcastMsg 2016-04-07 14:50:42 -04:00
David Hill
8a58f8cf3a peer: Implement sendheaders support (BIP0130).
This modifies the peer package to add support for the sendheaders
protocol message introduced by BIP0030.

NOTE: This does not add support to btcd itself. That requires the server
and sync code to make use of the new functionality exposed by these
changes.  As a result, btcd will still be using protocol version 70011.
2016-04-06 16:56:48 -05:00
David Hill
c1861bc8fa peer: declare QueueMessage()'s doneChan as send only.
This ensures the channel passed to QueueMessage is writable and that
QueueMessage will not read from the channel (write-only).

This change is merely a safety change.  If a user of the API passes
a read-only channel to QueueMessage, it will now be caught at compile
time instead of panicking during runtime.

Also update internal functions.
2016-04-06 13:50:27 -05:00
Jonathan Gillham
391d5e4a01 server: Stop main loop from blocking when RPC server is not running.
When the RPC server is not running a buffered transaction notification

channel fills and eventually blocks.  This commit ensures that the

channel continues to be drained irrespective of the RPC server status.
2016-04-06 13:20:01 -05:00
Jonathan Gillham
5c59b685e6 server: Appropriately name inbound peers map in peerState. 2016-02-27 15:52:40 +00:00
Dave Collins
eb882f39f8 multi: Fix several misspellings in the comments.
This commit corrects several typos in the comments found by misspell.
2016-02-25 11:17:12 -06:00
Tibor Bősze
c75fea9c94 Implement banning based on dynamic ban scores
Dynamic ban scores consist of a persistent and a decaying component. The
persistent score can be used to create simple additive banning policies
simlar to those found in other bitcoin node implementations. The
decaying score enables the creation of evasive logic which handles
misbehaving peers (especially application layer DoS attacks) gracefully
by disconnecting and banning peers attempting various kinds of flooding.
Dynamic ban scores allow these two approaches to be used in tandem.

This pull request includes the following:

 - Dynamic ban score type & functions, with tests for core functionality
 - Ban score of connected peers can be queried via rpc (getpeerinfo)
 - Example policy with decaying score increments on mempool and getdata
 - Logging of misbehavior once half of the ban threshold is reached
 - Banning logic can be disabled via configuration (enabled by default)
 - User defined ban threshold can be set via configuration
2016-02-16 10:10:29 +01:00
Dave Collins
d127ad4083 server: Make consistent use of svr peer stringer.
This updates a couple of logging statements to use the serverPeer
instance instead of the embedded peer.Peer so they are consistent with
all of the other log statements.
2016-02-10 22:29:30 -06:00
Jonathan Gillham
73d353247c peer: Consolidate Connect, Disconnect, Start, Shutdown public methods.
This commit does not change functionality. It makes the creation of inbound and outbound peers more homogeneous. As a result the Start method of peer was removed as it was found not to be necessary. This is the first of several pull requests/commits designed to make the peer public API and internals less complex.
2016-02-06 11:11:15 +00:00
David Hill
383ed041ec Use atomic operations instead of mutexes. 2016-02-04 15:20:04 -05:00
Mawuli Adzoe
1944637333 Bump copyright date to reflect fixes since the beginning of this year. 2016-01-06 15:29:58 -07:00
Javed Khan
7996eb1f9d peer: drain chans before exiting peerHandler
Also disconnect the failed peer to allow the peerDoneHandler goroutine
to exit, instead of hanging around. Fixes #583.
2016-01-06 20:59:43 +05:30
Dario Nieuwenhuis
d0cdd53720 server: Fix persistent peers not being removed properly
When a persistent peer is disconnected (for example due to a
network timeout), a connection retry is issued. The logic for
doing so failed to remove the peer from the peerState, causing
dead peer connections to fill the peerState. Since connections
in the peerState are counted towards the maxPeers limit, this
would cause btcd to eventually stop retrying connection.

This commit fixes the issue by properly removing the peer from
the peerState.
2015-12-28 01:23:10 +01:00
Dave Collins
2f6aeacfab server: Correct mempool/CPU miner initialize order.
The CPU miner relies on the mempool, so the mempool has to be created
before calling the function to create the CPU miner.  When PR #568
introduced the mempool config struct, it moved the mempool creation
after the miner creation, which leads to the CPU miner crashing due to
trying to access a nil mempool.

This move the CPU miner creation after the mempool creation
appropriately.
2015-12-08 02:01:53 -06:00
David Hill
2a7f41cddb peer: Add DisableRelayTx to config.
DisableRelayTx sets the DisableRelayTx value in the version
message which informs the remote peer on whether to send
inv messages for transactions.
2015-12-03 10:29:06 -05:00
Dave Collins
ce981f45c2 mining: Create skeleton package.
This creates a skeleton mining package that simply contains a few of the
definitions used by the mining and mempool code.

This is a step towards decoupling the mining code from the internals of
btcd and ultimately will house all of the code related to creating block
templates and CPU mining.

The main reason a skeleton package is being created before the full
blown package is ready is to avoid blocking mempool separation which
relies on these type definitions.
2015-11-30 12:23:50 -06:00