Also, update TravisCI goclean script to remove the special casing which
ignored 'Id' from the lint output since that exception is no longer
needed. It was previously required due to the old version of btcjson,
but that is no longer in the repo.
Introduce an ECDSA signature verification into btcd in order to
mitigate a certain DoS attack and as a performance optimization.
The benefits of SigCache are two fold. Firstly, usage of SigCache
mitigates a DoS attack wherein an attacker causes a victim's client to
hang due to worst-case behavior triggered while processing attacker
crafted invalid transactions. A detailed description of the mitigated
DoS attack can be found here: https://bitslog.wordpress.com/2013/01/23/fixed-bitcoin-vulnerability-explanation-why-the-signature-cache-is-a-dos-protection/
Secondly, usage of the SigCache introduces a signature verification
optimization which speeds up the validation of transactions within a
block, if they've already been seen and verified within the mempool.
The server itself manages the sigCache instance. The blockManager and
txMempool respectively now receive pointers to the created sigCache
instance. All read (sig triplet existence) operations on the sigCache
will not block unless a separate goroutine is adding an entry (writing)
to the sigCache. GetBlockTemplate generation now also utilizes the
sigCache in order to avoid unnecessarily double checking signatures
when generating a template after previously accepting a txn to the
mempool. Consequently, the CPU miner now also employs the same
optimization.
The maximum number of entries for the sigCache has been introduced as a
config parameter in order to allow users to configure the amount of
memory consumed by this new additional caching.
SFNodeBloom is a new service flag that a node is required to use to
indicate that it supports bloom filtering. This includes a protocol
version bump to 70011 and a wire version bump to 0.3.0.
btcd:
The SFNodeBloom flag is set by default. A new configuration option
--nopeerbloomfilters has been added to to disable bloom filtering.
Any node advertising a version greater than or equal to 70011 that
attempts to use bloom filtering will be disconnected if bloom
filtering is disabled.
This mimics Bitcoin Core commit afb0ccaf9c9e4e8fac7db3564c4e19c9218c6b03
The comment says "only allow recent nodes (10mins) after we failed 30 times",
but the server actually did the opposite and allowed only recent nodes before
30 failed connection attempts. This corrects the server's behavior.
This commit correctly replaces persistent peers that are being retried in
the list of persistent peers so it will continue to be retried as
intended.
Also, limit the maximum retry interval for persistent peers to 5 minutes.
Fixes#463.
* The cases for the 'addnode' command were previously
stacked on top the new cases for the 'node' command.
The intended behavior was to create a fall through and
handle both commands. However, trying to use this
syntax with a type switch caused the first case to be
ignored.
* addnode' specific functions and structs in the server
have been removed. Instead, the 'add' and 'del' subcommands
are now proxied to the matching 'node' cmd functions.
* Gives node operators full control of peer connectivity
* RPC adds ability to disconnect all matching non-persistent peers,
remove persistent peers, and connect to peers making them either
temporary or persistent.
This change was suggested as Countermeasure 2 in
Eclipse Attacks on Bitcoin's Peer-to-Peer Network, Ethan
Heilman, Alison Kendler, Aviv Zohar, Sharon Goldberg. ePrint Archive
Report 2015/263. March 2015.
This mimics Bitcoin Core commit c6a63ceeb4956933588995bcf01dc3095aaeb1fc
This commit contains three classes of optimizations:
- Reducing the number of unnecessary hash copies
- Improve the performance of the DoubleSha256 function
- A couple of minor optimizations of the ShaHash functions
The first class is a result of the Bytes function on a ShaHash making a
copy of the bytes before returning them. It really should have been named
CloneBytes, but that would break the API now.
To address this, a comment has been added to the function which explicitly
calls out the copy behavior. In addition, all call sites of .Bytes on a
ShaHash in the code base have been updated to simply slice the array when
a copy is not needed. This saves a significant amount of data copying.
The second optimization modifies the DoubleSha256 function to directly use
fastsha256.Sum256 instead of the hasher interface. This reduces the
number of allocations needed. A benchmark for the function has been added
as well.
old: BenchmarkDoubleSha256 500000 3691 ns/op 192 B/op 3 allocs/op
new: BenchmarkDoubleSha256 500000 3081 ns/op 32 B/op 1 allocs/op
The final optimizations are for the ShaHash IsEqual and SetBytes functions
which have been modified to make use of the fact the type is an array and
remove an unneeded subslice.
In order to avoid prior situations of stalled syncs due to
outdated peer height data, we now update block heights up peers in
real-time as we learn of their announced
blocks.
Updates happen when:
* A peer sends us an orphan block. We update based on
the height embedded in the scriptSig for the coinbase tx
* When a peer sends us an inv for a block we already know
of
* When peers announce new blocks. Subsequent
announcements that lost the announcement race are
recognized and peer heights are updated accordingly
Additionally, the `getpeerinfo` command has been modified
to include both the starting height, and current height of
connected peers.
Docs have been updated with `getpeerinfo` extension.
This commit converts the RPC server over to use the new features available
in the latest version of btcjson and improve a few things along the way.
This following summarizes the changes:
- All btcjson imports have been updated to the latest package version
- The help has been significantly improved
- Invoking help with no command specified now provides an alphabetized
list of all supported commands along with one-line usage
- The help for each command is automatically generated and provides much
more explicit information such as the type of each parameter, whether
or not it's optional or required, etc
- The websocket-specific commands are now provided when accessing the
help when connected via websockets
- Help has been added for all websocket-specific commands and is only
accessible when connected via websockets
- The error returns and handling of both the standard and websocket
handlers has been made consistent
- All RPC errors have been converted to the new RPCError type
- Various variables have been renamed for consistency
- Several RPC errors have been improved
- The commands that are marked as unimplemented have been moved into the
separate map where they belong
- Several comments have been improved
- An unnecessary check has been removed from the createrawtransaction
handler
- The command parsing has been restructured a bit to pave the way for
JSON-RPC 2.0 batching support
* Address index is built up concurrently with the `--addrindex` flag.
* Entire index can be deleted with `--dropaddrindex`.
* New RPC call: `searchrawtransaction`
* Returns all transacitons related to a particular address
* Includes mempool transactions
* Requires `--addrindex` to be activated and fully caught up.
* New `blockLogger` struct has been added to factor our common logging
code
* Wiki and docs updated with new features.
* When an inv is to be sent to the server for relaying, the sender
already has access to the underlying data. So
instead of requiring the relay to look up the data by
hash, the data is now coupled in the request message.
This commit uses the new MedianTimeSource API in btcchain to create a
median time source which is stored in the server and is fed time samples
from all remote nodes that are connected. It also modifies all call sites
which now require the the time source to pass it in.
This ensures we backoff when reconnecting to peers for which we don't
understand the replies, just like we do for peers we fail to connect to.
Closes#103
This commit implements full support for filtering based on the filterload,
filteradd, filterclear, and merkleblock messages introduced by BIP0037.
This allows btcd to work seamlessly with SPV wallets such as BitcoinJ.
Original code by @dajohi. Cleanup, bug fixes, and polish by @davecgh.
These changes are a joint effort between myself and @dajohi.
- Separate IP address range/network code into its own file
- Group all of the RFC range declarations together
- Introduces a new unexported function to simplify the range declarations
- Add comments for all exported functions
- Use consistent variable casing in refactored code
- Add initial doc.go package overview
- Bump serialize interval to 10 minutes
- Correct GroupKey to perform as intended
- Make AddLocalAddress return error instead of just a debug message
- Add tests for AddLocalAddress
- Add tests for GroupKey
- Add tests for GetBestLocalAddress
- Use time.Time to improve readability
- Make address manager code golint clean
- Misc cleanup
- Add test coverage reporting
This commit does just enough to move the address manager into its own
package. Since it was not originally written as a package, it will
require a bit of refactoring and cleanup to turn it into a robust
package with a friendly API.
By putting each DNS seed in its own go routine, btcd can start connecting
to nodes as they are found instead of waiting for all seeds to respond. This
significantly speeds up startup time.
Additionally, logging was added to show how many addresses were fetched from
each seed.
This commits removes a number of golint warnings. There is a class of
warnings which I can't fix due to unsufficient knowledge of the domain
at this point. These are listed here:
addrmanager.go:907:1: comment on exported method AddrManager.Attempt
should be of the form "Attempt ..."
addrmanager.go:1048:1: exported function RFC1918 should have comment or
be unexported
addrmanager.go:1058:1: exported function RFC3849 should have comment or
be unexported
addrmanager.go:1065:1: exported function RFC3927 should have comment or
be unexported
addrmanager.go:1073:1: exported function RFC3964 should have comment or
be unexported
addrmanager.go:1081:1: exported function RFC4193 should have comment or
be unexported
addrmanager.go:1089:1: exported function RFC4380 should have comment or
be unexported
addrmanager.go:1097:1: exported function RFC4843 should have comment or
be unexported
addrmanager.go:1105:1: exported function RFC4862 should have comment or
be unexported
addrmanager.go:1113:1: exported function RFC6052 should have comment or
be unexported
addrmanager.go:1121:1: exported function RFC6145 should have comment or
be unexported
addrmanager.go:1128:1: exported function Tor should have comment or be
unexported
addrmanager.go:1143:1: exported function Local should have comment or be
unexported
addrmanager.go:1228:2: exported const InterfacePrio should have comment
(or a comment on this block) or be unexported
discovery.go:26:2: exported var ErrTorInvalidAddressResponse should have
comment or be unexported
limits/limits_unix.go:19:1: exported function SetLimits should have
comment or be unexported
limits/limits_windows.go:7:1: exported function SetLimits should have
comment or be unexported
util/dropafter/dropafter.go:22:6: exported type ShaHash should have
comment or be unexported
util/dropafter/dropafter.go:38:2: exported const ArgSha should have
comment (or a comment on this block) or be unexported
util/dropafter/dropafter.go:128:5: exported var ErrBadShaPrefix should
have comment or be unexported
util/dropafter/dropafter.go:129:5: exported var ErrBadShaLen should have
comment or be unexported
util/dropafter/dropafter.go:130:5: exported var ErrBadShaChar should
have comment or be unexported
util/showblock/showblock.go:24:6: exported type ShaHash should have
comment or be unexported
util/showblock/showblock.go:46:2: exported const ArgSha should have
comment (or a comment on this block) or be unexported
util/showblock/showblock.go:163:1: exported function DumpBlock should
have comment or be unexported
util/showblock/showblock.go:211:5: exported var ErrBadShaPrefix should
have comment or be unexported
util/showblock/showblock.go:212:5: exported var ErrBadShaLen should have
comment or be unexported
util/showblock/showblock.go:213:5: exported var ErrBadShaChar should
have comment or be unexported
This commit implements a built-in concurrent CPU miner that can be enabled
with the combination of the --generate and --miningaddr options. The
--blockminsize, --blockmaxsize, and --blockprioritysize configuration
options wich already existed prior to this commit control the block
template generation and hence affect blocks mined via the new CPU miner.
The following is a quick overview of the changes and design:
- Starting btcd with --generate and no addresses specified via
--miningaddr will give an error and exit immediately
- Makes use of multiple worker goroutines which independently create block
templates, solve them, and submit the solved blocks
- The default number of worker threads are based on the number of
processor cores in the system and can be dynamically changed at
run-time
- There is a separate speed monitor goroutine used to collate periodic
updates from the workers to calculate overall hashing speed
- The current mining state, number of workers, and hashes per second can
be queried
- Updated sample-btcd.conf file has been updated to include the coin
generation (mining) settings
- Updated doc.go for the new command line options
In addition the old --getworkkey option is now deprecated in favor of the
new --miningaddr option. This was changed for a few reasons:
- There is no reason to have a separate list of keys for getwork and CPU
mining
- getwork is deprecated and will be going away in the future so that means
the --getworkkey flag will also be going away
- Having the work 'key' in the option can be confused with wanting a
private key while --miningaddr make it a little more clear it is an
address that is required
Closes#137.
Reviewed by @jrick.
This commit resolves an issue where it was possible the block manager
could hang on shutdown due to inventory rebroadcasting. In particular, it
adds checks to prevent modification of the of rebroadcast inventory during
shutdown and adds a drain on the channel to ensure any outstanding
messages are discarded.
Found by @dajohi who also provided some of the code.
This commit, along with recent commits to btcnet and btcwire, expose a new
network that is intended to provide a private network useful for
simulation testing. To that end, it has the special property that it has
no DNS seeds and will actively ignore all addr and getaddr messages. It
will also not try to connect to any nodes other than those specified via
--connect. This allows the network to remain private to the specific
nodes involved in the testing and not simply become another public
testnet.
The network difficulty is also set extremely low like the regression test
network so blocks can be created extremely quickly without requiring a lot
of hashing power.
This change modifies the params struct to embed a *btcnet.Params,
removing the old parameter fields that are handled by the btcnet
package.
Hardcoded network checks have also been removed in favor of modifying
behavior based on the current active net's parameters.
Not all library packages, notable btcutil and btcchain, have been
updated to use btcnet yet, but with this change, each package can be
updated one at a time since the active net's btcnet.Params are
available at each callsite.
ok @davecgh
In practise the races caused by not protecting these quite simply didn't
matter, they couldn't actually cause any damage whatsoever. However, I
am sick of hearing about these essentially false positivies whenever
someone runs the race detector (yes, i know that race detector has no
false positives but this was effectively harmess).
verified to shut the detector up by dhill.
This commit cleans up and moves a couple of comments in the recent pull
request which implements a rebroadcast handler (#114) in order to avoid
discussing internal state in the exported function comment.
How a function actually accomplishes the stated functionality is not
something that a caller is concerned with. The details about the internal
state are better handled with comments inside the function body.
This commit implements a rebroadcast handler which deals with
rebroadcasting inventory at a random time interval between 0 and 30
minutes. It then uses the new rebroadcast logic to ensure transactions
which were submitted via the sendrawtransaction RPC are rebroadcast until
they make it into a block.
Closes#99.
Rather than using a type specifically in btcd for the getpeerinfo, this
commit, along with a recent commit to btcjson, changes the code over to
use the type from btcjson. This is more consistent with other RPC results
and provides a few extra benefits such as the ability for btcjson to
automatically unmarshal the results into a concrete type with proper field
types as opposed to a generic interface.
Recent commits to the reference implementation have changed the syncnode
field to be present in the getpeerinfo RPC even when it is false. This
commit changes btcd to match.
This commit changes the server byte counters over to use a mutex instead
of the atomic package. The atomic.AddUint64 function requires the struct
fields to be 64-bit aligned on 32-bit platforms. The byte counts are
fields in the server struct and are not 64-bit aligned. While it would be
possible to arrange the fields to be aligned through various means, it
would make the code too fragile for my tastes. I prefer code that doesn't
depend on platform specific alignment.
Fixes#96.
Previously the getnettotals was just looping through all of the currently
connected peers to sum the byte counts and returning that. However, the
intention of the getnettotals RPC is to get all bytes since the server was
started, so this logic was not correct.
This commit modifies the code to keep an atomic counter on the server for
bytes read/written and has each peer update the server counters as well as
the per-peer counters.
This commit adds byte counters to each peer using the new btcwire
ReadMessageN and WriteMessageN functions to obtain the number of bytes
read and written, respectively. It also returns those byte counters via
the PeerInfo struct which is used to populate the RPC getpeerinfo reply.
Closes#83.
Rather than having the iterator functions a separate entities that access
the state to iterate, just expose the iterators as receivers on the state
itself. This is more consistent with the style used throughout the code
and the other receivers on the state such as Count, OutboundCount, etc.
This commit adds full support for the getaddednodeinfo RPC command
including DNS lookups which abide by proxy/onion/tor rules when the DNS
flag is specified. Note that it returns an array of strings when the DNS
flag is not set which is different than the current version of bitcoind
which is bugged and scheduled to be fixed per issue 3581 on the bitcoind
issue tracker.
This implements --onion (and --onionuser/--onionpass) that enable a
different proxy to be used to connect to .onion addresses. If no main
proxy is supplied then no proxy will be used for non-onion addresses.
Additionally we add --noonion that blocks connection attempts to .onion
addresses entirely (and avoids using tor for proxy dns lookups).
the --tor option has been supersceded and thus removed.
Closes#47
The fields of the PeerInfo should not have been marked omit as the only
ones that should be omitted to for compatibility are the SyncNode and
BanScore fields.
In order to match the Satohsi client, the return is supposed to be an
8-digit string representation of the services instead of the actual
services numeric value.
since we don't wait for peers, this largely just waits for the server procs
themselves to die. Unless the entire server is wedged (which is what kill -9 is
for) this should always shut down fairly swiftly.
This should mean we sync addrmanager and disestablish upnp correctly on
interrupt.
Discussed with davec.
This code borrows and fixes up a chunk of code to handle upnp from
Taipei-Torrent (https://github.com/jackpal/Taipei-Torrent), under
current versions of go none of the xml parsing was working correctly.
This fixes that and also refactors the SOAP code to be a little nicer by
stripping off the soap containers. It is still rather rough but seems to
correctly redirect ports and advertise the correct address.
Upnp is not run by default. --upnp will enable it, but it will still not
run if we are not listening or if --externalip is in use.
Closes#51
This implements only the bare bones of external ip address selection
using very similar algorithms and selection methods to bitcoind. Every
address we bind to, and if we bind to the wildcard, every listening
address is recorded, and one for the appropriate address type of the
peer is selected.
Support for fetching addresses via upnp, external services, or via the
command line are not yet implemented.
Closes#35
Also, make every subsystem within btcd use its own logger instance so each
subsystem can have its own level specified independent of the others.
This is work towards #48.
This allows the provision of address/port pairs to be listened on instead
of just providing the port. e.g.:
btcd --listen 1.2.3.4:4321 --listen 127.0.0.01 --listen [::1]:5432
When --proxy and --connect are used, we disable listening *unless* any --listen
arguments have been provided, when we will listen on those addresses as
requested.
Initial code by davec, integration by myself.
Closes#33
allow listens to fail, but warn. error if all failed
fmt
persistentpeers and outbound(nonpersistent) peers get their own lists,
so iterating over them can be much simpler (and quicker when you have
125 peer of which 8 are outbound).
We have a channel for queries and commands in server, where we pass in
args and the channel to reply from, let rpcserver use these interfaces
to provide the requistie information.
So far not all of the informaation is 100% correct, the syncpeer
information needs to be fetched from blockmanager, the subversion isn't
recorded and the number of bytes sent and recieved needs to be obtained
from btcwire. The rest should be correct.
If we don't hear from a peer for 5 minutes, we disconnect them. To keep
traffic flowing we send a ping every 2 minutes if we have not send any
other message that should get a reply.
This commit is a first pass at improving the logging. It changes a number
of things to improve the readability of the output. The biggest addition
is message summaries for each message type when using the debug logging
level.
There is sitll more to do here such as allowing the level of each
subsystem to be independently specified, syslog support, and allowing the
logging level to be changed run-time.
Also, the loops which only remove a single element and break or return
don't need the extra logic for iteration since they don't continue
iteration after removal.
It is not safe to remove an item from a container/list while iterating the
list without first saving the next pointer since removing the item nils
the internal list element's next pointer.
This commit is a rather large one which implements transaction pool and
relay according to the protocol rules of the reference implementation.
It makes use of btcchain to ensure the transactions are valid for the
block chain and includes several stricter checks which determine if they
are "standard" or not before admitting them into the pool and relaying
them.
There are still a few TODOs around the more strict rules which determine
which transactions are willing to be mined, but the core checks which
are imperative (everything except the all of the "standard" checks really)
to operate as a good citizen on the bitcoin network are in place.
Rather than having all of the various places that print peer figure out
the direction and form the string, centralize it by implementing the
Stringer interface on the peer.
Only log errors for most cases if the peer is persisent (and thus requested).
Only log by default after version exchange, and after losing a peer that had
completed version exchange. Make most other messages debug.
We would occasionally hang or a while during server shudown, this is due
to an outbound peer waiting on a connection or a sleep. However, we
don't actually require to wait for the peers to finish at all. So just
let them finish.
Secondly, make peer.disconnnect and server.shutdown atomic varaibles so
that checking them from multiple goroutines isn't race, and clean up
their usage.
- Remove leftover debug log prints
- Increment waitgroup outside of goroutine
- Various comment and log message consistency
- Combine peer setup and newPeer -> newInboundPeer
- Save and load peers.json to/from cfg.DataDir
- Only claim addrmgr needs more addresses when it has less than 1000
- Add warning if unkown peer on orphan block.
Use it to add multiple peer support. We try and keep 8 outbound peers
active at all times.
This address manager is not as complete as the one in bitcoind yet, but
additional functionality is being worked on.
We currently handle (in a similar manner to bitcoind):
- biasing between new and already tried addresses based on number of connected
peers.
- rejection of non-default ports until desparate
- address selection probabilities based on last successful connection and number
of failures.
- routability checks based on known unroutable subnets.
- only connecting to each network `group' once at any one time.
We currently lack support for:
- tor ``addresses'' (an .onion address encoded in 64 bytes of ip address)
- full state save and restore (we just save a json with the list of known
addresses in it)
- multiple buckets for new and tried addresses selected by a hash of address and
source. The current algorithm functions the same as bitcoind would with only
one bucket for new and tried (making the address cache rather smaller than it
otherwise would be).
This commit changes the code so that all calls to .Add on waitgroups
happen before the associated goroutines are launched. Doing this after
the goroutine could technically cause a race where the goroutine started
and finished before the main goroutine has a chance to increment the
counter. In our particular case none of the goroutines exit quickly
enough for this to be an issue, but nevertheless the correct way should be
used.
This commit adds support for relaying blocks between peers. It keeps
track of inventory that has either already been advertised to remote peers
or advertised by remote peers using a size-limited most recently used
cache. This helps avoid relaying inventory the peer already knows as
much as possible while not allowing rogue peers to eat up arbitrary
amounts of memory with bogus inventory.
The commit reworks the server statup and shutdown sequence to ensure the
server can always shutdown cleanly. The peer code depends on being able
to send messages to the address and block managers, so they need to have
their lifecycle tied to the peer handler to prevent issues with
asynchronous shutdown order.
