This optimizes the way in which the maps are limited by the block
manager.
Previously the code would read a cryptographically random value large
enough to construct a hash, find the first entry larger than that value,
and evict it.
That approach is quite inefficient and could easily become a bottleneck
when processing transactions due to the need to read from a source such
as /dev/urandom and all of the subsequent hash comparisons.
Luckily, strong cryptographic randomness is not needed here. The primary
intent of limiting the maps is to control memory usage with a secondary
concern of making it difficult for adversaries to force eviction of
specific entries.
Consequently, this changes the code to make use of the pseudorandom
iteration order of Go's maps along with the preimage resistance of the
hashing function to provide the desired functionality. It has
previously been discussed that the specific pseudorandom iteration order
is not guaranteed by the Go spec even though in practice that is how it
is implemented. This is not a concern however because even if the
specific compiler doesn't implement that, the preimage resistance of the
hashing function alone is enough.
Thanks to @Roasbeef for pointing out the efficiency concerns and the
fact that strong cryptographic randomness is not necessary.
This prevents the node from repeatedly requesting and rejecting the
same transaction as different peers inv the same transaction.
Idea from Bitcoin Core commit 0847d9cb5fcd2fdd5a21bde699944d966cf5add9
Also, limit the number of both requested blocks and transactions.
This commit introduces package peer which contains peer related features
refactored from peer.go.
The following is an overview of the features the package provides:
- Provides a basic concurrent safe bitcoin peer for handling bitcoin
communications via the peer-to-peer protocol
- Full duplex reading and writing of bitcoin protocol messages
- Automatic handling of the initial handshake process including protocol
version negotiation
- Automatic periodic keep-alive pinging and pong responses
- Asynchronous message queueing of outbound messages with optional
channel for notification when the message is actually sent
- Inventory message batching and send trickling with known inventory
detection and avoidance
- Ability to wait for shutdown/disconnect
- Flexible peer configuration
- Caller is responsible for creating outgoing connections and listening
for incoming connections so they have flexibility to establish
connections as they see fit (proxies, etc.)
- User agent name and version
- Bitcoin network
- Service support signalling (full nodes, bloom filters, etc.)
- Maximum supported protocol version
- Ability to register callbacks for handling bitcoin protocol messages
- Proper handling of bloom filter related commands when the caller does
not specify the related flag to signal support
- Disconnects the peer when the protocol version is high enough
- Does not invoke the related callbacks for older protocol versions
- Snapshottable peer statistics such as the total number of bytes read
and written, the remote address, user agent, and negotiated protocol
version
- Helper functions for pushing addresses, getblocks, getheaders, and
reject messages
- These could all be sent manually via the standard message output
function, but the helpers provide additional nice functionality such
as duplicate filtering and address randomization
- Full documentation with example usage
- Test coverage
In addition to the addition of the new package, btcd has been refactored
to make use of the new package by extending the basic peer it provides to
work with the blockmanager and server to act as a full node. The
following is a broad overview of the changes to integrate the package:
- The server is responsible for all connection management including
persistent peers and banning
- Callbacks for all messages that are required to implement a full node
are registered
- Logic necessary to serve data and behave as a full node is now in the
callback registered with the peer
Finally, the following peer-related things have been improved as a part
of this refactor:
- Don't log or send reject message due to peer disconnects
- Remove trace logs that aren't particularly helpful
- Finish an old TODO to switch the queue WaitGroup over to a channel
- Improve various comments and fix some code consistency cases
- Improve a few logging bits
- Implement a most-recently-used nonce tracking for detecting self
connections and generate a unique nonce for each peer
Introduce an ECDSA signature verification into btcd in order to
mitigate a certain DoS attack and as a performance optimization.
The benefits of SigCache are two fold. Firstly, usage of SigCache
mitigates a DoS attack wherein an attacker causes a victim's client to
hang due to worst-case behavior triggered while processing attacker
crafted invalid transactions. A detailed description of the mitigated
DoS attack can be found here: https://bitslog.wordpress.com/2013/01/23/fixed-bitcoin-vulnerability-explanation-why-the-signature-cache-is-a-dos-protection/
Secondly, usage of the SigCache introduces a signature verification
optimization which speeds up the validation of transactions within a
block, if they've already been seen and verified within the mempool.
The server itself manages the sigCache instance. The blockManager and
txMempool respectively now receive pointers to the created sigCache
instance. All read (sig triplet existence) operations on the sigCache
will not block unless a separate goroutine is adding an entry (writing)
to the sigCache. GetBlockTemplate generation now also utilizes the
sigCache in order to avoid unnecessarily double checking signatures
when generating a template after previously accepting a txn to the
mempool. Consequently, the CPU miner now also employs the same
optimization.
The maximum number of entries for the sigCache has been introduced as a
config parameter in order to allow users to configure the amount of
memory consumed by this new additional caching.
This commit converts all block height references to int32 instead of
int64. The current target block production rate is 10 mins per block
which means it will take roughly 40,800 years to reach the maximum
height an int32 affords. Even if the target rate were lowered to one
block per minute, it would still take roughly another 4,080 years to
reach the maximum.
In the mean time, there is no reason to use a larger type which results
in higher memory and disk space usage. However, for now, in order to
avoid having to reserialize a bunch of database information, the heights
are still serialized to the database as 8-byte uint64s.
This is being mainly being done in preparation for further upcoming
infrastructure changes which will use the smaller and more efficient
4-byte serialization in the database as well.
This commit corrects an issue where the reject message sent in response
to rejected transactions had the Cmd field set to "block" instead of
"tx".
Fixes#436.
The option -maxorphantx allows the user to specify the number of
orphan transactions to keep in memory.
Also, lower the default max orphan count from 10000 to 1000.
This commit removes the error returns from the BlockHeader.BlockSha,
MsgBlock.BlockSha, and MsgTx.TxSha functions since they can never fail and
end up causing a lot of unneeded error checking throughout the code base.
It also updates all call sites for the change.
Because FetchTransactionStore in GetBlockTemplate occasionally accesses
the internal blockchain memory structure while it is being read or modified,
a race can occur. To prevent this, FetchTransactionStore is instead
routed through the internal channel for blockchain requests.
In order to avoid prior situations of stalled syncs due to
outdated peer height data, we now update block heights up peers in
real-time as we learn of their announced
blocks.
Updates happen when:
* A peer sends us an orphan block. We update based on
the height embedded in the scriptSig for the coinbase tx
* When a peer sends us an inv for a block we already know
of
* When peers announce new blocks. Subsequent
announcements that lost the announcement race are
recognized and peer heights are updated accordingly
Additionally, the `getpeerinfo` command has been modified
to include both the starting height, and current height of
connected peers.
Docs have been updated with `getpeerinfo` extension.
For every transaction in a newly accepted block, process the orphan
pool moving now no longer orphan transactions to the mempool.
Previously, no longer orphan transactions would remain in the orphan
pool.
This modifies the recently added code which rejects free/low-fee
transactions with insufficient priority to ignore resurrected transactions
from disconnected blocks. It also exempts resurrected transactions from
the free/low-fee rate limiting.
This changes the behavior of the rescan RPC to automatically set the
client up for transaction notifications for transactions paying to any
rescanned address and spending outputs in the final rescan UTXO set
after a rescanned is performed through the best block in the chain.
* Address index is built up concurrently with the `--addrindex` flag.
* Entire index can be deleted with `--dropaddrindex`.
* New RPC call: `searchrawtransaction`
* Returns all transacitons related to a particular address
* Includes mempool transactions
* Requires `--addrindex` to be activated and fully caught up.
* New `blockLogger` struct has been added to factor our common logging
code
* Wiki and docs updated with new features.
* When an inv is to be sent to the server for relaying, the sender
already has access to the underlying data. So
instead of requiring the relay to look up the data by
hash, the data is now coupled in the request message.
Only two operations are performed with this data structure: adding to
the back and removing from the front. Because middle inserts and
deletions are never needed, a linked list results in overall worse
performance due to an extra allocation for each element's node, worse
cache locality, and the runtime cost of boxing/unboxing each item
during accesses.
On top of the performance gains, a slice is more type safe as it is a
true generic data structure making it is impossible to insert or
access an element with the wrong type.
The mempool's MaybeAcceptTransaction methods have also been modified
to return a slice of transaction hashes referenced by the transaction
inputs which are unknown (totally spent or never seen). While this is
currently used to include the first hash in a ProcessTransaction error
message if inserting orphans is not allowed, it may also be used in
the future to request orphan transactions from peers.
This commit uses the new MedianTimeSource API in btcchain to create a
median time source which is stored in the server and is fed time samples
from all remote nodes that are connected. It also modifies all call sites
which now require the the time source to pass it in.
This change removes transactions from a newly connected block
from the orphan pool if they exist. Additionally, any orphan
transactions that are no longer orphan transactions are moved
to the mempool and inv'd to the currently connected peers.
This commit implements reject handling as defined by BIP0061 and bumps the
maximum supported protocol version to 70002 accordingly.
As a part of supporting this a new error type named RuleError has been
introduced which encapsulates and underlying error which could be one of
the existing TxRuleError or btcchain.RuleError types.
This allows a single high level type assertion to be used to determine if
the block or transaction was rejected due to a rule error or due to an
unexpected error. Meanwhile, an appropriate reject error can be created
from the error by pulling the underlying error out and using it.
Also, a check for minimum protocol version of 209 has been added.
Closes#133.
This commit implements the long polling portion of the getblocktemplate
RPC as defined by BIP0022. Per the specification, each block template is
returned with a longpollid which can be used in a subsequent
getblocktemplate request to keep the connection open until the server
determines the block template associated with the longpollid should be
replaced with a new one.
This is work towards #124.
This commit implements the non-optional and template tweaking support for
the getblocktemplate RPC as defined by BIP0022. This implementation does
not yet include long polling support.
This is work towards #124.
BitcoinJ, and possibly other wallets, don't follow the spec of sending an
inventory message and allowing the remote peer to decide whether or not
they want to request the transaction via a getdata message. Unfortuantely
the reference implementation permits unrequested data, so it has allowed
wallets that don't follow the spec to proliferate.
While this is not ideal, this commit removes the functionality which
disconnects peers for sending unsolicited transactions to provide
interoperability.
