This commit is the first stage of several that are planned to convert
the blockchain package into a concurrent safe package that will
ultimately allow support for multi-peer download and concurrent chain
processing. The goal is to update btcd proper after each step so it can
take advantage of the enhancements as they are developed.
In addition to the aforementioned benefit, this staged approach has been
chosen since it is absolutely critical to maintain consensus.
Separating the changes into several stages makes it easier for reviewers
to logically follow what is happening and therefore helps prevent
consensus bugs. Naturally there are significant automated tests to help
prevent consensus issues as well.
The main focus of this stage is to convert the blockchain package to use
the new database interface and implement the chain-related functionality
which it no longer handles. It also aims to improve efficiency in
various areas by making use of the new database and chain capabilities.
The following is an overview of the chain changes:
- Update to use the new database interface
- Add chain-related functionality that the old database used to handle
- Main chain structure and state
- Transaction spend tracking
- Implement a new pruned unspent transaction output (utxo) set
- Provides efficient direct access to the unspent transaction outputs
- Uses a domain specific compression algorithm that understands the
standard transaction scripts in order to significantly compress them
- Removes reliance on the transaction index and paves the way toward
eventually enabling block pruning
- Modify the New function to accept a Config struct instead of
inidividual parameters
- Replace the old TxStore type with a new UtxoViewpoint type that makes
use of the new pruned utxo set
- Convert code to treat the new UtxoViewpoint as a rolling view that is
used between connects and disconnects to improve efficiency
- Make best chain state always set when the chain instance is created
- Remove now unnecessary logic for dealing with unset best state
- Make all exported functions concurrent safe
- Currently using a single chain state lock as it provides a straight
forward and easy to review path forward however this can be improved
with more fine grained locking
- Optimize various cases where full blocks were being loaded when only
the header is needed to help reduce the I/O load
- Add the ability for callers to get a snapshot of the current best
chain stats in a concurrent safe fashion
- Does not block callers while new blocks are being processed
- Make error messages that reference transaction outputs consistently
use <transaction hash>:<output index>
- Introduce a new AssertError type an convert internal consistency
checks to use it
- Update tests and examples to reflect the changes
- Add a full suite of tests to ensure correct functionality of the new
code
The following is an overview of the btcd changes:
- Update to use the new database and chain interfaces
- Temporarily remove all code related to the transaction index
- Temporarily remove all code related to the address index
- Convert all code that uses transaction stores to use the new utxo
view
- Rework several calls that required the block manager for safe
concurrency to use the chain package directly now that it is
concurrent safe
- Change all calls to obtain the best hash to use the new best state
snapshot capability from the chain package
- Remove workaround for limits on fetching height ranges since the new
database interface no longer imposes them
- Correct the gettxout RPC handler to return the best chain hash as
opposed the hash the txout was found in
- Optimize various RPC handlers:
- Change several of the RPC handlers to use the new chain snapshot
capability to avoid needlessly loading data
- Update several handlers to use new functionality to avoid accessing
the block manager so they are able to return the data without
blocking when the server is busy processing blocks
- Update non-verbose getblock to avoid deserialization and
serialization overhead
- Update getblockheader to request the block height directly from
chain and only load the header
- Update getdifficulty to use the new cached data from chain
- Update getmininginfo to use the new cached data from chain
- Update non-verbose getrawtransaction to avoid deserialization and
serialization overhead
- Update gettxout to use the new utxo store versus loading
full transactions using the transaction index
The following is an overview of the utility changes:
- Update addblock to use the new database and chain interfaces
- Update findcheckpoint to use the new database and chain interfaces
- Remove the dropafter utility which is no longer supported
NOTE: The transaction index and address index will be reimplemented in
another commit.
mempoolPolicy contains the values that configure the mempool policy.
This decouples the values from the internals of btcd to move closer
to a mempool package.
The --blocksonly configuration option disables accepting transactions
from remote peers. It will still accept, relay, and rebroadcast
valid transactions sent via RPC or websockets.
Dynamic ban scores consist of a persistent and a decaying component. The
persistent score can be used to create simple additive banning policies
simlar to those found in other bitcoin node implementations. The
decaying score enables the creation of evasive logic which handles
misbehaving peers (especially application layer DoS attacks) gracefully
by disconnecting and banning peers attempting various kinds of flooding.
Dynamic ban scores allow these two approaches to be used in tandem.
This pull request includes the following:
- Dynamic ban score type & functions, with tests for core functionality
- Ban score of connected peers can be queried via rpc (getpeerinfo)
- Example policy with decaying score increments on mempool and getdata
- Logging of misbehavior once half of the ban threshold is reached
- Banning logic can be disabled via configuration (enabled by default)
- User defined ban threshold can be set via configuration
This commit introduces package peer which contains peer related features
refactored from peer.go.
The following is an overview of the features the package provides:
- Provides a basic concurrent safe bitcoin peer for handling bitcoin
communications via the peer-to-peer protocol
- Full duplex reading and writing of bitcoin protocol messages
- Automatic handling of the initial handshake process including protocol
version negotiation
- Automatic periodic keep-alive pinging and pong responses
- Asynchronous message queueing of outbound messages with optional
channel for notification when the message is actually sent
- Inventory message batching and send trickling with known inventory
detection and avoidance
- Ability to wait for shutdown/disconnect
- Flexible peer configuration
- Caller is responsible for creating outgoing connections and listening
for incoming connections so they have flexibility to establish
connections as they see fit (proxies, etc.)
- User agent name and version
- Bitcoin network
- Service support signalling (full nodes, bloom filters, etc.)
- Maximum supported protocol version
- Ability to register callbacks for handling bitcoin protocol messages
- Proper handling of bloom filter related commands when the caller does
not specify the related flag to signal support
- Disconnects the peer when the protocol version is high enough
- Does not invoke the related callbacks for older protocol versions
- Snapshottable peer statistics such as the total number of bytes read
and written, the remote address, user agent, and negotiated protocol
version
- Helper functions for pushing addresses, getblocks, getheaders, and
reject messages
- These could all be sent manually via the standard message output
function, but the helpers provide additional nice functionality such
as duplicate filtering and address randomization
- Full documentation with example usage
- Test coverage
In addition to the addition of the new package, btcd has been refactored
to make use of the new package by extending the basic peer it provides to
work with the blockmanager and server to act as a full node. The
following is a broad overview of the changes to integrate the package:
- The server is responsible for all connection management including
persistent peers and banning
- Callbacks for all messages that are required to implement a full node
are registered
- Logic necessary to serve data and behave as a full node is now in the
callback registered with the peer
Finally, the following peer-related things have been improved as a part
of this refactor:
- Don't log or send reject message due to peer disconnects
- Remove trace logs that aren't particularly helpful
- Finish an old TODO to switch the queue WaitGroup over to a channel
- Improve various comments and fix some code consistency cases
- Improve a few logging bits
- Implement a most-recently-used nonce tracking for detecting self
connections and generate a unique nonce for each peer
Introduce an ECDSA signature verification into btcd in order to
mitigate a certain DoS attack and as a performance optimization.
The benefits of SigCache are two fold. Firstly, usage of SigCache
mitigates a DoS attack wherein an attacker causes a victim's client to
hang due to worst-case behavior triggered while processing attacker
crafted invalid transactions. A detailed description of the mitigated
DoS attack can be found here: https://bitslog.wordpress.com/2013/01/23/fixed-bitcoin-vulnerability-explanation-why-the-signature-cache-is-a-dos-protection/
Secondly, usage of the SigCache introduces a signature verification
optimization which speeds up the validation of transactions within a
block, if they've already been seen and verified within the mempool.
The server itself manages the sigCache instance. The blockManager and
txMempool respectively now receive pointers to the created sigCache
instance. All read (sig triplet existence) operations on the sigCache
will not block unless a separate goroutine is adding an entry (writing)
to the sigCache. GetBlockTemplate generation now also utilizes the
sigCache in order to avoid unnecessarily double checking signatures
when generating a template after previously accepting a txn to the
mempool. Consequently, the CPU miner now also employs the same
optimization.
The maximum number of entries for the sigCache has been introduced as a
config parameter in order to allow users to configure the amount of
memory consumed by this new additional caching.
SFNodeBloom is a new service flag that a node is required to use to
indicate that it supports bloom filtering. This includes a protocol
version bump to 70011 and a wire version bump to 0.3.0.
btcd:
The SFNodeBloom flag is set by default. A new configuration option
--nopeerbloomfilters has been added to to disable bloom filtering.
Any node advertising a version greater than or equal to 70011 that
attempts to use bloom filtering will be disconnected if bloom
filtering is disabled.
This mimics Bitcoin Core commit afb0ccaf9c9e4e8fac7db3564c4e19c9218c6b03
The configured onion proxy was not being used due to checking if the
passed string had suffix '.onion', which never matched because the
port number is part of the string.
Tor stream isolation randomizes proxy user credentials resulting in
Tor creating a new circuit for each connection. This makes it more
difficult to correlate connections.
Idea from Wladimir J. van der Laan via Bitcoin Core.
The option -maxorphantx allows the user to specify the number of
orphan transactions to keep in memory.
Also, lower the default max orphan count from 10000 to 1000.
The limited user is specified with the --rpclimituser and
--rpclimitpass options (or the equivalent in the config file).
The config struct and loadConfig() are updated to take the
new options into account. The limited user can have neither
the same username nor the same password as the admin user.
The package-level rpcLimit map in rpcserver.go specifies
the RPC commands accessible by limited users. This map
includes both HTTP/S and websocket commands.
The checkAuth function gets a new return parameter to
signify whether the user is authorized to change server
state. The result is passed to the jsonRPCRead function and
to the WebsocketHandler function in rpcwebsocket.go.
The wsClient struct is updated with an "isAdmin" field
signifying that the client is authorized to change server
state, written by WebsocketHandler and handleMessage.
The handleMessage function also checks the field to
allow or disallow an RPC call.
The following documentation files are updated:
- doc.go
- sample-btcd.conf
- docs/README.md
- docs/json_rpc_api.md
- docs/configure_rpc_server_listen_interfaces.md
By default, have the mempool reject free and low-fee transactions that
have insufficient priority to be mined in the next block.
Addtionally, add a new configuration option, -norelaypriority, to
disable the check.
* Address index is built up concurrently with the `--addrindex` flag.
* Entire index can be deleted with `--dropaddrindex`.
* New RPC call: `searchrawtransaction`
* Returns all transacitons related to a particular address
* Includes mempool transactions
* Requires `--addrindex` to be activated and fully caught up.
* New `blockLogger` struct has been added to factor our common logging
code
* Wiki and docs updated with new features.
This commit introduces a new flag, --notls, which can be used to disable
TLS for the RPC server. However, the flag can only be used when the RPC
server is bound to localhost interfaces. This is intended to prevent the
situation where someone decides they want to expose the RPC server to the
web for remote management/access, but forgot they have TLS disabled.
This commit improves a couple of issues surrounding the creation of the
btcd home directory.
First, the code was previously attempting to log any errors that occurred
while creating the directory using the logging system which is not
initialized at that point. Thus, nothing was displayed to the user.
Second, if any component of btcd home directory path already exists, but
is not a directory, such as in the case of symlinks, the error returned
from the os.MkDirAll call indicates the directory can't be created. While
this is true, it's not always the most helpful error to display to the
user. So, this commit adds logic to detect when the failure case is due
to an existing symlink and displays a nicer error message suggesting the
user check if the destination of the link is mounted.
Fixes#193.
This commit resolves a minor issue where an error in the config file would
prevent the help from being shown until the config file error was
resolved.
This results in the following behavior:
- With a malformed header:
$ ./btcd
Error parsing config file: ~/btcd/btcd.conf:14: malformed section header
Use btcd -h to show usage
- With an invalid option:
$ ./btcd
Error parsing config file: unknown option: bogus
Use btcd -h to show usage
- Invoking help with an error in the config file:
$ ./btcd -h
Usage:
...
- Invoking with an invalid command line option:
$ ./btcd --bogus=bogus
unknown flag `bogus'
Use btcd -h to show usage
ok @jrick
Rather than showing the usage when an error is encounted during options
parsing, show a message that describes how to invoke help instead. This
is useful because the help is long enough now that the error is often
overlooked since it scrolls out of view.
ok @jrick
This commits removes a number of golint warnings. There is a class of
warnings which I can't fix due to unsufficient knowledge of the domain
at this point. These are listed here:
addrmanager.go:907:1: comment on exported method AddrManager.Attempt
should be of the form "Attempt ..."
addrmanager.go:1048:1: exported function RFC1918 should have comment or
be unexported
addrmanager.go:1058:1: exported function RFC3849 should have comment or
be unexported
addrmanager.go:1065:1: exported function RFC3927 should have comment or
be unexported
addrmanager.go:1073:1: exported function RFC3964 should have comment or
be unexported
addrmanager.go:1081:1: exported function RFC4193 should have comment or
be unexported
addrmanager.go:1089:1: exported function RFC4380 should have comment or
be unexported
addrmanager.go:1097:1: exported function RFC4843 should have comment or
be unexported
addrmanager.go:1105:1: exported function RFC4862 should have comment or
be unexported
addrmanager.go:1113:1: exported function RFC6052 should have comment or
be unexported
addrmanager.go:1121:1: exported function RFC6145 should have comment or
be unexported
addrmanager.go:1128:1: exported function Tor should have comment or be
unexported
addrmanager.go:1143:1: exported function Local should have comment or be
unexported
addrmanager.go:1228:2: exported const InterfacePrio should have comment
(or a comment on this block) or be unexported
discovery.go:26:2: exported var ErrTorInvalidAddressResponse should have
comment or be unexported
limits/limits_unix.go:19:1: exported function SetLimits should have
comment or be unexported
limits/limits_windows.go:7:1: exported function SetLimits should have
comment or be unexported
util/dropafter/dropafter.go:22:6: exported type ShaHash should have
comment or be unexported
util/dropafter/dropafter.go:38:2: exported const ArgSha should have
comment (or a comment on this block) or be unexported
util/dropafter/dropafter.go:128:5: exported var ErrBadShaPrefix should
have comment or be unexported
util/dropafter/dropafter.go:129:5: exported var ErrBadShaLen should have
comment or be unexported
util/dropafter/dropafter.go:130:5: exported var ErrBadShaChar should
have comment or be unexported
util/showblock/showblock.go:24:6: exported type ShaHash should have
comment or be unexported
util/showblock/showblock.go:46:2: exported const ArgSha should have
comment (or a comment on this block) or be unexported
util/showblock/showblock.go:163:1: exported function DumpBlock should
have comment or be unexported
util/showblock/showblock.go:211:5: exported var ErrBadShaPrefix should
have comment or be unexported
util/showblock/showblock.go:212:5: exported var ErrBadShaLen should have
comment or be unexported
util/showblock/showblock.go:213:5: exported var ErrBadShaChar should
have comment or be unexported
This commit implements a built-in concurrent CPU miner that can be enabled
with the combination of the --generate and --miningaddr options. The
--blockminsize, --blockmaxsize, and --blockprioritysize configuration
options wich already existed prior to this commit control the block
template generation and hence affect blocks mined via the new CPU miner.
The following is a quick overview of the changes and design:
- Starting btcd with --generate and no addresses specified via
--miningaddr will give an error and exit immediately
- Makes use of multiple worker goroutines which independently create block
templates, solve them, and submit the solved blocks
- The default number of worker threads are based on the number of
processor cores in the system and can be dynamically changed at
run-time
- There is a separate speed monitor goroutine used to collate periodic
updates from the workers to calculate overall hashing speed
- The current mining state, number of workers, and hashes per second can
be queried
- Updated sample-btcd.conf file has been updated to include the coin
generation (mining) settings
- Updated doc.go for the new command line options
In addition the old --getworkkey option is now deprecated in favor of the
new --miningaddr option. This was changed for a few reasons:
- There is no reason to have a separate list of keys for getwork and CPU
mining
- getwork is deprecated and will be going away in the future so that means
the --getworkkey flag will also be going away
- Having the work 'key' in the option can be confused with wanting a
private key while --miningaddr make it a little more clear it is an
address that is required
Closes#137.
Reviewed by @jrick.
This commit, along with recent commits to btcnet and btcwire, expose a new
network that is intended to provide a private network useful for
simulation testing. To that end, it has the special property that it has
no DNS seeds and will actively ignore all addr and getaddr messages. It
will also not try to connect to any nodes other than those specified via
--connect. This allows the network to remain private to the specific
nodes involved in the testing and not simply become another public
testnet.
The network difficulty is also set extremely low like the regression test
network so blocks can be created extremely quickly without requiring a lot
of hashing power.
This change modifies the params struct to embed a *btcnet.Params,
removing the old parameter fields that are handled by the btcnet
package.
Hardcoded network checks have also been removed in favor of modifying
behavior based on the current active net's parameters.
Not all library packages, notable btcutil and btcchain, have been
updated to use btcnet yet, but with this change, each package can be
updated one at a time since the active net's btcnet.Params are
available at each callsite.
ok @davecgh
This commit adds a new configuration option, --rpcmaxclients, to limit the
number of max standard RPC clients that are served concurrently. Note
that this value does not apply to websocket connections. A future commit
will add support for limiting those separately.
Closes#68.
This commit adds a new option, --logdir, which works in the same fashion
as the --datadir option. Consequently, the logging directory is name
"namespaced" by the network as well. This resolves the issue where two
btcd instances running (one for mainnet and one for testnet) would
overwrite each other's log files by default.
It also provides the user with a method to change the logging location to
non-default locations if they prefer. For example, it enables multiple
btcd instances on the same network to specify unique logging directories
(even though running multiple btcd instances on the same network is not
the most sane configuration).
Closes#95.
This commit adds the btcdb memdb backend as a supported database type.
Note that users will NOT want to run in this mode because, being memory
only, it obviously does not persist the database when shutdown.
It is being added for testing purposes to help prevent constant abuse to
developer's hard drive when churning the block database multiple times a
day.
The warning about a missing config file should only be shown after all
other configuration has succeeded so it's not shown when there are invalid
options specified. Also add a comment about it where its intended
placement is for the future.
This implements --onion (and --onionuser/--onionpass) that enable a
different proxy to be used to connect to .onion addresses. If no main
proxy is supplied then no proxy will be used for non-onion addresses.
Additionally we add --noonion that blocks connection attempts to .onion
addresses entirely (and avoids using tor for proxy dns lookups).
the --tor option has been supersceded and thus removed.
Closes#47
This code borrows and fixes up a chunk of code to handle upnp from
Taipei-Torrent (https://github.com/jackpal/Taipei-Torrent), under
current versions of go none of the xml parsing was working correctly.
This fixes that and also refactors the SOAP code to be a little nicer by
stripping off the soap containers. It is still rather rough but seems to
correctly redirect ports and advertise the correct address.
Upnp is not run by default. --upnp will enable it, but it will still not
run if we are not listening or if --externalip is in use.
Closes#51
This commit modifies btcd to run cleanly as a Windows service. btcd is
intended to be a long running process that stays synchronized with the
bitcoin block chain and provides chain services to multiple users. It
follows that a service is the best option on Windows for this
functionality.
A few key points are:
- Supports graceful shutdown via the service stop/shutdown commands
- Integrates cleanly with the Windows event log
- Adds a new /s flag that can be used to install/remove/start/stop the
service
One outstanding issue is that the application data directory is currently
user specific which means, by default, if you start btcd as a user, the
same data won't be used as when it's running as a service. This needs to
be resovled. The most likely approach will be to put all data into the
common appdata directory Windows provides, but it will require some
additional work to deal with permissions properly as user processes can't
write there by default.
Closes#42.
This commit modifies the logging to also log all output to a rolling log
file in the btcd home directory under the logs folder. It uses a maximum
size of 10MB per log file and a max rotation size of 3. This means the
log files will not exceed 30 megabytes.
The code was previously only changing the logging level if it wasn't the
default which is accurate for setting the log level once at startup time,
but it needs to set it unconditionally to allow dynamic updates.
Also, make every subsystem within btcd use its own logger instance so each
subsystem can have its own level specified independent of the others.
This is work towards #48.
- Remove periods from --rpccert/--rpckey since none of the other options use
them
- Make it a little more clear that the --listen and --rpclisten options can be
used multiple times by changing the summary to start with "Add an ..."
which also matches the --addpeer style
All rpc sockets now listen using TLS by default, and this can not be
turned off. The keys (defauling to the datadirectory) may be provided by
--rpccert and --rpckey. If the keys do not exist we will generate a new
self-signed keypair with some sane defaults (hostname and all current
interface addresses).
Additionally add tls capability to btcctl so that it can still be used.
The certificate to use for verify can be provided on the commandline or
verification can be turned off (this leaves you susceptible to MITM
attacks)
Initial code from dhill (rpc tls support) and jrick (key generation),
cleanup, debugging and polishing from me.
The regression test mode is special and therefore most likely will not
want to use the same settings that are in the configuration file. The -C
option can still be used to specify a config file in regression test mode
if desired.
This commit makes use of the new default-mask go-flags option in
conjunction with delaying the usage display until after the config file is
parsed. This has a couple of nice properties such as showing the actual
values that will be used as loaded from the specific config file instead
of the defaults specified in btcd itself, and also allows any config file
parsing errors to be shown prior to displaying the usage.
This allows the provision of address/port pairs to be listened on instead
of just providing the port. e.g.:
btcd --listen 1.2.3.4:4321 --listen 127.0.0.01 --listen [::1]:5432
When --proxy and --connect are used, we disable listening *unless* any --listen
arguments have been provided, when we will listen on those addresses as
requested.
Initial code by davec, integration by myself.
Closes#33
allow listens to fail, but warn. error if all failed
fmt
This commit makes use of the new btcutil.AppDataDir function which chooses
appropriate data directories for each supported operating system. It also
adds code to the upgrade path to properly migrate existing data from the
old to new locations.
This is part of work toward issue #30.
This commit provides a new --cpuprofile flag that can be used to specify a
file path to write CPU profile data into. The resulting profile can then be
consumed by the 'go tool pprof' command.
This commit provides a new flag, --nocheckpoints, to disable built-in
checkpoints.
Checkpoints are used for a number of things such a ensuring
the block chain being downloaded matches various known good blocks,
allowing quicker verification on old blocks since scripts don't have to be
executed, and preventing forks from old blocks, etc.
This commit adds environment variable expansion and path cleaning to the
data directory. This allows the user to specify data paths in the config
file such as datadir=~/.btcd/data and datadir=$SOMEVAR/btcd. It also
adds usage instructions and an example to the sample btcd.conf file.
This commit modifies the way the data paths are handled. Since there will
ultimately be more data associated with each network than just the block
database, the data path has been modified to be "namespaced" based on the
network. This allows all data associated with a specific network to
simply use the data path without having to worry about conflicts with data
from other networks.
In addition, this commit renames the block database to "blocks" plus a
suffix which denotes the database type. This prevents issues that would
otherwise arise if the user decides to use a different database type and
a file/folder with the same name already eixsts but is of the old database
type. For most users this won't matter, but it does provide nice
properties for testing and development as well since it makes it easy to
go back and forth between database types.
This commit also includes code to upgrade the old database paths to the
new ones so the change is seamless for the user.
Finally, bump the version to 0.2.0.
This change paves the way for saving more than just the block database to
the filesystem (such as address manager data, index data, etc) where the
name "dbdir" no longer makes sense.
When running in regression test mode, it is unlikely the user wants to
connect to permanent peers they have configured in their config file.
This commit modifies the code to ignore the config file entry when in
regression test mode. The user can still provide -a (or --addpeer) on the
command line to override this if they really want to connect out to a
specific peer during regression test mode.
Although not required if the proxy set is indeed Tor, setting this option
does the following:
- Sends DNS queries over the Tor network (during dns seed lookup). This
stops your IP from being leaked via DNS.
- Does not disable the listening port. This allows the hidden services
feature of Tor to be used.
