This introduces the concept of a new interface named TxSource which aims
to generically provide a concurrent safe source of transactions to be
considered for inclusion in a new block. This is a step towards
decoupling the mining code from the internals of btcd. Ultimately the
intent is to create a separate mining package.
The new TxSource interface relies on a new struct named miningTxDesc,
which describes each entry in the transaction source. Once this code is
refactored into a separate mining package, the mining prefix can simply
be dropped leaving the type exported as mining.TxDesc.
To go along with this, the existing TxDesc type in the mempool has been
renamed to mempoolTxDesc and changed to embed the new miningTxDesc type.
This allows the mempool to efficiently implement the MiningTxDescs
method needed to satisfy the TxSource interface.
This approach effectively separates the direct reliance of the mining
code on the mempool and its data structures. Even though the memory
pool will still be the default concrete implementation of the interface,
making it an interface offers much more flexibility in terms of testing
and even provides the potential to allow more than one source (perhaps
multiple independent relay networks, for example).
Finally, the memory pool and all of the mining code has been updated to
implement and use the new interface.
This does three things:
- Splits the priority calculation logic from the TxDesc type
- Modifies the calcPriority function to perform the value age
calculation instead of accepting it as a parameter
- Changes the starting priority to be calculated when the transaction is
added to the pool
The first is useful as it is a step towards decoupling the mining code
from the internals of the memory pool. Also, the concept of priority is
related to mining policy, so it makes more sense to have the
calculations separate than being defined on the memory pool tx
descriptor.
The second change has been made because everywhere that uses the
calcPriority function first has to calculate the value age anyways and
by making it part of the function it can be avoided altogether in
certain circumstances thereby provided a bit of optimization.
The third change ensure the starting priority is safe for reentrancy
which will be important once the mempool is split into a separate
package.
This fixes an issue introduced during the peer refactor where persistent
peers that failed the initial connection are not retried as intended.
It also improves the retry logic as follows:
- Make the retry handler goroutine simply use a for loop instead of
launching a new goroutine for each backoff attempt. Even though
goroutines are fairly cheap to create, it is much more efficient to
simply loop
- Change the retry handler to accept a flag if it is the initial attempt
- Rather than dividing the const interval by 2 everywhere and passing
the retry duration in, just half the constant and set the initial
duration to it in the retry handler
Finally, include the address of the peer in the error message when a new
outbound peer can't be created.
This introduces the concept of a mining policy struct which is used to
control block template generation instead of directly accessing the
config struct. This is a step toward decoupling the mining code from
the internals of btcd. Ultimately the intent is to create a separate
mining package.
Now that the memory pool minimum fee calculation code is also
calculating a more precise value instead of rounding up to the nearest
kilobyte boundary, the comment in NewBlockTemplate regarding this
behavior is no longer accurate. Thus, this removes the comment.
Also, while here, change the calculation to use an int64 instead of
float since it matches the precision of the new calculation code used by
the memory pool and can also avoid the need for the slower floating
point math.
When the peer code was refactored, the lists of peers were converted to
maps however the code which runs when a peer disconnects still iterates
them like a slice. This is no longer necessary since they are maps
which means the peer can simply be looked up by its ID.
Also, the old code was comparing the map entry and the peer being
removed by their pointers which could lead to potentially not properly
finding the peer. This issue is also resolved by this change since it
looks up the peer by its ID.
This defines the peer stat fields directly in the Peer struct instead of
defining them in a separate struct and embedding them. It is slightly
more efficient this way and there is really no reason to have them
defined separately anyways since they are not passed around or otherwise
used independently.
Don't log "Established connection" message on new when
DisableConnectOnNew is set and no connection was actually established.
Do log that same message when Connect() successfully connects.
Fixes a rare hang during peer tests due to the same connection being
used for different outbound peers. Also fixed passing a chan to
QueueMessage(..) which was not being waited on, so was not being used.
500 tests with various transactions and scripts, verifying that
calcSignatureHash generates the expected hash in each case.
This requires changing SigHashType to uint32; that won't affect the
standard use-cases, but will make calcSignatureHash behave more like the
Core counterpart for non-standard SigHashType settings, like those in
some of these tests.
This is more efficient and prevents duplicate entries which can
lead to no-longer-orphans being attempted be added to the mempool
multiple times.
Bug found by me, debugged with @davecgh, and patch from @davecgh
This refactors the checkTransactionStandard function, along with related
constants, from the mempool to the policy.go file since it is strictly
related to policy.
In addition, it adds tests for the function which cover all code paths.
ProcessTransaction could have accepted a new transaction into mempool
but could have returned a reject message if a no-longer-orphan
transaction failed to be accepted. This would also skip any
additional no-longer-orphans, keeping them in the orphan pool.
Instead of returning an error incorrectly, log the error and skip
the no-longer-orphan transaction. This allows the rest of the
no-longer-orphans to be processed as well.
This modifies the IP parsing code to work with IPv6 zone ids. This is
needed since the net.ParseIP function does not allow zone ids even
though net.Listen does.
rpcserver:
If the locktime is given, the transaction inputs will be set to a
non-max value, activating the locktime. The locktime for the
new transaction will be set to the given value.
This mimics Bitcoin Core commit 212bcca92089f406d9313dbe6d0e1d25143d61ff
This commit modifies the peer logging so that it will not log an error
when it's due to the remote peer disconnecting or the peer being
forcibly disconnected or shutdown.
First, it removes the documentation section from all the README.md files
and instead puts a web-based godoc badge and link at the top with the
other badges. This is being done since the local godoc tool no longer
ships with Go by default, so the instructions no longer work without
first installing godoc. Due to this, pretty much everyone uses the
web-based godoc these days anyways. Anyone who has manually installed
godoc won't need instructions.
Second, it makes sure the ISC license badge is at the top with the other
badges and removes the textual reference in the overview section.
Finally, it's modifies the Installation section to Installation and
Updating and adds a '-u' to the 'go get' command since it works for both
and thus is simpler.
This commit implements stall detection logic at the peer level to detect
and disconnect peers that are either not following the protocol in
regards to expected response messages or have otherwise stalled. This
is accomplished by setting deadlines for each message type which expects
a response and periodically checking them while properly taking into
account processing time.
There are an increasing number of nodes on the network which claim to be
full nodes, but don't actually properly implement the entire p2p
protocol even though they implement it enough to cause properly
implemented nodes to make data requests to which they never respond.
Since btcd currently only syncs new blocks via single sync peer and,
prior to this commit only had very basic stall detection, this could
lead to a situation where the block download became stalled indefinitely
due to one of these misbehaving peers. This commit fixes that issue
since the stalled peer will now be detected and disconnected which leads
to a new sync peer being selected.
This logic will also fit nicely with the future multi-peer sync model
which is on the roadmap and for which infrastructure work is underway.
Fixes#486 and fixes#229.
This commit modifies the ping logic in the peer to ping on an interval
regardless of what other messages are being sent versus the previous
method of delaying the ping each time a message that is expected to
receive data is sent.
This helps improve the ping statistics and simplifies its logic.
This fleshes out the doc.go documentation which is shown on godoc, the
README.md shown on github, and improves a couple of comments for the
fields in the Config struct.
This commit introduces package peer which contains peer related features
refactored from peer.go.
The following is an overview of the features the package provides:
- Provides a basic concurrent safe bitcoin peer for handling bitcoin
communications via the peer-to-peer protocol
- Full duplex reading and writing of bitcoin protocol messages
- Automatic handling of the initial handshake process including protocol
version negotiation
- Automatic periodic keep-alive pinging and pong responses
- Asynchronous message queueing of outbound messages with optional
channel for notification when the message is actually sent
- Inventory message batching and send trickling with known inventory
detection and avoidance
- Ability to wait for shutdown/disconnect
- Flexible peer configuration
- Caller is responsible for creating outgoing connections and listening
for incoming connections so they have flexibility to establish
connections as they see fit (proxies, etc.)
- User agent name and version
- Bitcoin network
- Service support signalling (full nodes, bloom filters, etc.)
- Maximum supported protocol version
- Ability to register callbacks for handling bitcoin protocol messages
- Proper handling of bloom filter related commands when the caller does
not specify the related flag to signal support
- Disconnects the peer when the protocol version is high enough
- Does not invoke the related callbacks for older protocol versions
- Snapshottable peer statistics such as the total number of bytes read
and written, the remote address, user agent, and negotiated protocol
version
- Helper functions for pushing addresses, getblocks, getheaders, and
reject messages
- These could all be sent manually via the standard message output
function, but the helpers provide additional nice functionality such
as duplicate filtering and address randomization
- Full documentation with example usage
- Test coverage
In addition to the addition of the new package, btcd has been refactored
to make use of the new package by extending the basic peer it provides to
work with the blockmanager and server to act as a full node. The
following is a broad overview of the changes to integrate the package:
- The server is responsible for all connection management including
persistent peers and banning
- Callbacks for all messages that are required to implement a full node
are registered
- Logic necessary to serve data and behave as a full node is now in the
callback registered with the peer
Finally, the following peer-related things have been improved as a part
of this refactor:
- Don't log or send reject message due to peer disconnects
- Remove trace logs that aren't particularly helpful
- Finish an old TODO to switch the queue WaitGroup over to a channel
- Improve various comments and fix some code consistency cases
- Improve a few logging bits
- Implement a most-recently-used nonce tracking for detecting self
connections and generate a unique nonce for each peer
isMultiSig was not verifying the number of pubkeys specified matched
the number of pubkeys provided. This caused certain non-standard
scripts to be considered multisig scripts.
However, the script still would have failed during execution.
NOTE: This only affects whether or not the script is considered
standard and does NOT affect consensus.
Also, add a test for this check.
Also, update TravisCI goclean script to remove the special casing which
ignored 'Id' from the lint output since that exception is no longer
needed. It was previously required due to the old version of btcjson,
but that is no longer in the repo.
This commit exports the ReadVarString and WriteVarString functions so
they are available for callers to use.
A variable length string is encoded as a variable length integer
containing the length of the string followed by the bytes that represent
the string itself.
See https://github.com/bitcoin/bips/blob/master/bip-0065.mediawiki for
more information.
This commit mimics Bitcoin Core commit bc60b2b4b401f0adff5b8b9678903ff8feb5867b
and includes additional tests from Bitcoin Core commit
cb54d17355864fa08826d6511a0d7692b21ef2c9
We've already been generating lowS sigs for quite a while. This removes
the malleability vector.
This mimics Bitcoin Core commit 49dd5c629df0a08cf3b1ea8085c03312d1a81696
Introduce an ECDSA signature verification into btcd in order to
mitigate a certain DoS attack and as a performance optimization.
The benefits of SigCache are two fold. Firstly, usage of SigCache
mitigates a DoS attack wherein an attacker causes a victim's client to
hang due to worst-case behavior triggered while processing attacker
crafted invalid transactions. A detailed description of the mitigated
DoS attack can be found here: https://bitslog.wordpress.com/2013/01/23/fixed-bitcoin-vulnerability-explanation-why-the-signature-cache-is-a-dos-protection/
Secondly, usage of the SigCache introduces a signature verification
optimization which speeds up the validation of transactions within a
block, if they've already been seen and verified within the mempool.
The server itself manages the sigCache instance. The blockManager and
txMempool respectively now receive pointers to the created sigCache
instance. All read (sig triplet existence) operations on the sigCache
will not block unless a separate goroutine is adding an entry (writing)
to the sigCache. GetBlockTemplate generation now also utilizes the
sigCache in order to avoid unnecessarily double checking signatures
when generating a template after previously accepting a txn to the
mempool. Consequently, the CPU miner now also employs the same
optimization.
The maximum number of entries for the sigCache has been introduced as a
config parameter in order to allow users to configure the amount of
memory consumed by this new additional caching.
