This commit implements stall detection logic at the peer level to detect
and disconnect peers that are either not following the protocol in
regards to expected response messages or have otherwise stalled. This
is accomplished by setting deadlines for each message type which expects
a response and periodically checking them while properly taking into
account processing time.
There are an increasing number of nodes on the network which claim to be
full nodes, but don't actually properly implement the entire p2p
protocol even though they implement it enough to cause properly
implemented nodes to make data requests to which they never respond.
Since btcd currently only syncs new blocks via single sync peer and,
prior to this commit only had very basic stall detection, this could
lead to a situation where the block download became stalled indefinitely
due to one of these misbehaving peers. This commit fixes that issue
since the stalled peer will now be detected and disconnected which leads
to a new sync peer being selected.
This logic will also fit nicely with the future multi-peer sync model
which is on the roadmap and for which infrastructure work is underway.
Fixes#486 and fixes#229.
This commit modifies the ping logic in the peer to ping on an interval
regardless of what other messages are being sent versus the previous
method of delaying the ping each time a message that is expected to
receive data is sent.
This helps improve the ping statistics and simplifies its logic.
This fleshes out the doc.go documentation which is shown on godoc, the
README.md shown on github, and improves a couple of comments for the
fields in the Config struct.
This commit introduces package peer which contains peer related features
refactored from peer.go.
The following is an overview of the features the package provides:
- Provides a basic concurrent safe bitcoin peer for handling bitcoin
communications via the peer-to-peer protocol
- Full duplex reading and writing of bitcoin protocol messages
- Automatic handling of the initial handshake process including protocol
version negotiation
- Automatic periodic keep-alive pinging and pong responses
- Asynchronous message queueing of outbound messages with optional
channel for notification when the message is actually sent
- Inventory message batching and send trickling with known inventory
detection and avoidance
- Ability to wait for shutdown/disconnect
- Flexible peer configuration
- Caller is responsible for creating outgoing connections and listening
for incoming connections so they have flexibility to establish
connections as they see fit (proxies, etc.)
- User agent name and version
- Bitcoin network
- Service support signalling (full nodes, bloom filters, etc.)
- Maximum supported protocol version
- Ability to register callbacks for handling bitcoin protocol messages
- Proper handling of bloom filter related commands when the caller does
not specify the related flag to signal support
- Disconnects the peer when the protocol version is high enough
- Does not invoke the related callbacks for older protocol versions
- Snapshottable peer statistics such as the total number of bytes read
and written, the remote address, user agent, and negotiated protocol
version
- Helper functions for pushing addresses, getblocks, getheaders, and
reject messages
- These could all be sent manually via the standard message output
function, but the helpers provide additional nice functionality such
as duplicate filtering and address randomization
- Full documentation with example usage
- Test coverage
In addition to the addition of the new package, btcd has been refactored
to make use of the new package by extending the basic peer it provides to
work with the blockmanager and server to act as a full node. The
following is a broad overview of the changes to integrate the package:
- The server is responsible for all connection management including
persistent peers and banning
- Callbacks for all messages that are required to implement a full node
are registered
- Logic necessary to serve data and behave as a full node is now in the
callback registered with the peer
Finally, the following peer-related things have been improved as a part
of this refactor:
- Don't log or send reject message due to peer disconnects
- Remove trace logs that aren't particularly helpful
- Finish an old TODO to switch the queue WaitGroup over to a channel
- Improve various comments and fix some code consistency cases
- Improve a few logging bits
- Implement a most-recently-used nonce tracking for detecting self
connections and generate a unique nonce for each peer
isMultiSig was not verifying the number of pubkeys specified matched
the number of pubkeys provided. This caused certain non-standard
scripts to be considered multisig scripts.
However, the script still would have failed during execution.
NOTE: This only affects whether or not the script is considered
standard and does NOT affect consensus.
Also, add a test for this check.
Also, update TravisCI goclean script to remove the special casing which
ignored 'Id' from the lint output since that exception is no longer
needed. It was previously required due to the old version of btcjson,
but that is no longer in the repo.
This commit exports the ReadVarString and WriteVarString functions so
they are available for callers to use.
A variable length string is encoded as a variable length integer
containing the length of the string followed by the bytes that represent
the string itself.
See https://github.com/bitcoin/bips/blob/master/bip-0065.mediawiki for
more information.
This commit mimics Bitcoin Core commit bc60b2b4b401f0adff5b8b9678903ff8feb5867b
and includes additional tests from Bitcoin Core commit
cb54d17355864fa08826d6511a0d7692b21ef2c9
We've already been generating lowS sigs for quite a while. This removes
the malleability vector.
This mimics Bitcoin Core commit 49dd5c629df0a08cf3b1ea8085c03312d1a81696
Introduce an ECDSA signature verification into btcd in order to
mitigate a certain DoS attack and as a performance optimization.
The benefits of SigCache are two fold. Firstly, usage of SigCache
mitigates a DoS attack wherein an attacker causes a victim's client to
hang due to worst-case behavior triggered while processing attacker
crafted invalid transactions. A detailed description of the mitigated
DoS attack can be found here: https://bitslog.wordpress.com/2013/01/23/fixed-bitcoin-vulnerability-explanation-why-the-signature-cache-is-a-dos-protection/
Secondly, usage of the SigCache introduces a signature verification
optimization which speeds up the validation of transactions within a
block, if they've already been seen and verified within the mempool.
The server itself manages the sigCache instance. The blockManager and
txMempool respectively now receive pointers to the created sigCache
instance. All read (sig triplet existence) operations on the sigCache
will not block unless a separate goroutine is adding an entry (writing)
to the sigCache. GetBlockTemplate generation now also utilizes the
sigCache in order to avoid unnecessarily double checking signatures
when generating a template after previously accepting a txn to the
mempool. Consequently, the CPU miner now also employs the same
optimization.
The maximum number of entries for the sigCache has been introduced as a
config parameter in order to allow users to configure the amount of
memory consumed by this new additional caching.
While current existing numeric opcodes are limited to 4 bytes, new
opcodes may need different limits.
This mimics Bitcoin Core commit 99088d60d8a7747c6d1a7fd5d8cd388be1b3e138
This commit optimizes the createVinList function which is used to
generate the JSON list of transaction inputs. It also makes it more
consistent with the createVinListPrevOut function.
In particular, it entails the following changes:
- Only do a single coinbase check and return right away instead of
checking multiple times inside the loop over the inputs
- Use a pointer for populating the details of each entry to avoid
multiple unnecessary array lookups and bounds checks
- Group all fields that populate the entry for better readability
SFNodeBloom is a new service flag that a node is required to use to
indicate that it supports bloom filtering. This includes a protocol
version bump to 70011 and a wire version bump to 0.3.0.
btcd:
The SFNodeBloom flag is set by default. A new configuration option
--nopeerbloomfilters has been added to to disable bloom filtering.
Any node advertising a version greater than or equal to 70011 that
attempts to use bloom filtering will be disconnected if bloom
filtering is disabled.
This mimics Bitcoin Core commit afb0ccaf9c9e4e8fac7db3564c4e19c9218c6b03
This commit modifies the DisasmString function to use a bytes buffer for
constructing the disassembled string instead of naive string
concatenation. This makes a huge difference when disassembling scripts
with large numbers of opcodes.
The Bitcoin wire protocol includes several fields with their lengths
encoded according to a variable length integer encoding scheme that does
not enforce a unique encoding for all numbers.
This can lead to a situation where deserializing and re-serializing the
same data can result in different bytes. There are no currently known
issues due to this, but it is safer to reject such subtle differences as
they could potentially lead to exploits.
Consequently, this commit modifies the varint decoding function to error
when the value is not canonically encoded which effectively means that
all messages with varints that are not canonically encoded will now be
rejected. This will not cause issues with old client versions in
regards to blocks and transactions since the data is deserialized into
memory and then reserialized before being relayed thereby effectively
erasing any non-canonical encodings.
Also, new tests have been added to ensure non-canonical encodings are
properly rejected and exercise the new code, and the default user agent
version for wire has been bumped to version 0.2.1 to differentiate the
new behavior.
The equivalent logic was implemented in Bitcoin Core by PR 2884.
This commit improves the most-recently used inventory map human readable
string to only show the inventory vectors. and adds tests for the entire
structure to bring its coverage to 100%.
In addition, removes the type assertion check in the Add function since
the internal inventory list is only managed by the type itself and the
tests would now catch any mistakes in the type of entries in the list.
This commit converts all block height references to int32 instead of
int64. The current target block production rate is 10 mins per block
which means it will take roughly 40,800 years to reach the maximum
height an int32 affords. Even if the target rate were lowered to one
block per minute, it would still take roughly another 4,080 years to
reach the maximum.
In the mean time, there is no reason to use a larger type which results
in higher memory and disk space usage. However, for now, in order to
avoid having to reserialize a bunch of database information, the heights
are still serialized to the database as 8-byte uint64s.
This is being mainly being done in preparation for further upcoming
infrastructure changes which will use the smaller and more efficient
4-byte serialization in the database as well.
IsUnspendable takes a public key script and returns whether it is
spendable.
Additionally, hook this into the mempool isDust function, since
unspendable outputs can't be spent.
This mimics Bitcoin Core commit 0aad1f13b2430165062bf9436036c1222a8724da
The comment says "only allow recent nodes (10mins) after we failed 30 times",
but the server actually did the opposite and allowed only recent nodes before
30 failed connection attempts. This corrects the server's behavior.
This commit modifies the createTxRawResult code path along with callers
to work with block headers as opposed to btcutil.Blocks. This in turn
allows the code in handleGetRawTransaction and
handleSearchRawTransactions to perform a much cheaper block header load
as opposed to a full block load.
While here, also very slightly optimize the createVinList function to
avoid creating a util.Tx wrapper and to take advantage of the
btcutil.Amount type added after the function was originally written
This commit updates the merkle block handling to for the latest changes
to the btcutil API and optimizes it along the way.
Previously, the code was inefficiently reloading the transactions for
the matched hashes from the database instead of simply pulling them from
the full block that was used to create the merkle block.
At the current time, there is no difference between the wire encoding
at protocol version 0 and the stable long-term storage format. These
methods are simply for consistency with the other types.
This commit updates the wire tests for transactions which force
serialization and deserialization errors to force an error in the the
transaction lock time path.
This brings the wire test coverage back up to 100%.
This commit correctly replaces persistent peers that are being retried in
the list of persistent peers so it will continue to be retried as
intended.
Also, limit the maximum retry interval for persistent peers to 5 minutes.
Fixes#463.
