This change modifies the RPC server's notifiation manager from a
struct with requests, protected by a mutux, to two goroutines. The
first maintains a queue of all notifications and control requests
(registering/unregistering notifications), while the second reads from
the queue and processes notifications and requests one at a time.
Previously, to prevent slowing down block and mempool processing, each
notification would be handled by spawning a new goroutine. This lead
to cases where notifications would end up being sent to clients in a
different order than they were created. Adding a queue keeps the
order of notifications originating from the same goroutine, while also
not slowing down processing while waiting for notifications to be
processed and sent.
ok @davecgh
This mirrors a recent change in the reference implementation. Since the
vast majority of the network still uses it and transactions which aren't
considered standard won't be relayed, the rules for standard transactions
need to be kept consistent.
Ideally this should be handled via floating fees, but we're rather limited
until we have miner support and a larger network share so the probability
of a transaction being relayed and confirmed is high.
Closes#100.
ok @owainga
This change improves the mechanism by which btcd notifies a websocket
client of transaction relating to watched address and unspent outputs
in the following ways:
1. The old processedtx notification has been replaced with the new
recvtx notification. This notification, rather than parsing out
details used by wallet clients, sends the serialized transaction
(as hexadecimal) and any block details (if mined) if any transaction
output sends to one of the websocket client's watched addresses.
2. The old txspent notification has been replaced with the new
redeemingtx notification. This notification, rather than parsing
out details used by wallet clients, sends the serialized transaction
(as hexadecimal) and any block details (if mined) if any transaction
input spends a watched output.
3. When processing notifications for transaction outputs, if any output
spends to a client's watched address, a corresponding spent request
is automatically registered.
4. Transaction notifications originating from mempool now include both
transaction inputs and outputs, rather than only processing
5. When processing notifications for transaction inputs, a client's
output spent request is only removed if the transaction being
processed has also been mined into a block. In combination with the
4th change, this results in two redeemingtx notifications for
transactions which first appear in mempool and are subsequently mined
into a block.
This commit makes use of the new btcscript.HasCanonicalPushes to enforce
canonical data pushes for transactions that are considered standard.
A canonical data push is one where the fewest number of bytes possible to
encode the size of the data being pushed is used. This includes using the
small integer opcodes for single byte data that can be represented
directly.
This commit increases the block priority size to 50000 and the max
standard tx size to 100k. This matches relatively recent changes in the
reference implementation. The max block size was also increased to
750000, but since btcd does not currently create blocks, there is no
constant for it. That constant will likely be added as a part of the
getwork implementation since it requires block creation.
Closes#71.
This commit refactors the entire websocket client code to resolve several
issues with the previous implementation. Note that this commit does not
change the public API for websockets. It only consists of internal
improvements.
The following is the major issues which have been addressed:
- A slow websocket client could impede notifications to all clients
- Long-running operations such as rescans would block all other requests
until it had completed
- The above two points taken together could lead to apparant hangs since
the client doing the rescan would eventually run out of channel buffer
and block the entire group of clients until the rescan completed
- Disconnecting a websocket during certain operations could lead to a hang
- Stopping the rpc server with operations under way could lead to a hang
- There were no limits to the number of websocket clients that could
connect
The following is a summary of the major changes:
- The websocket code has been split into two entities: a
connection/notification manager and a websocket client
- The new connection/notification manager acts as the entry point from
the rest of the subsystems to feed data which potentially needs to
notify clients
- Each websocket client now has its own instance of the new websocket
client type which controls its own lifecycle
- The data flow has been completely redesigned to closely resemble the
peer data flow
- Each websocket now has its own long-lived goroutines for input, output,
and queuing of notifications
- Notifications use the new notification queue goroutine along with
queueing to ensure they dont't block on stalled or slow peers
- There is a new infrastructure for asynchronously executing long-running
commands such as a rescan while still allowing the faster operations to
continue to be serviced by the same client
- Since long-running operations now run asynchronously, they have been
limited to one at a time
- Added a limit of 10 websocket clients. This is hard coded for now, but
will be made configurable in the future
Taken together these changes make the code far easier to reason about and
update as well solve the aforementioned issues.
Further optimizations to improve performance are possible in regards to
the way the connection/notification manager works, however this commit
already contains a ton of changes, so they are being left for another
time.
Changed mempool.MaybeAcceptTransaction to accept an additional parameter
to differentiate betwee new transactions and those added from
disconnected blocks.
Added new fields to requestContexts to indicate which clients want to
receive all new transaction notifications.
Added NotifyForNewTx to rpcServer to deliver approriate transaction
notification.
The fee field of the getrawmempool RPC JSON response should be in Bitcoins
instead of Satoshi. This commit corrects that issue.
Also, add a couple of comments and fix a comment typo while here.
Since there is already a variable for the current block height in addition
to the next block height, use the existing curHeight variable instead
doing nextBlockHeight-1 in mempool add.
Also, make every subsystem within btcd use its own logger instance so each
subsystem can have its own level specified independent of the others.
This is work towards #48.
- Lock the mempool when removing transactions during a notification as
intended
- When generating the inventory vectors to serve on a mempool request,
recheck the memory pool for each hash since it's possible another thread
could have removed an entry after the initial query for available
hashes
- When a block is connected, remove any transactions which are now double
spends as a result of the newly connected transactions
When a transaction is being checked for acceptance into the transation
memory pool, it makes use of a chain function to ensure the invariant rules
for what transactions are allowed into a block are not violated. That
function returns a btcchain.RuleError if rules are violated. However,
since this code path is tailored to free-standing transactions, the error
needs to be converted to a TxRuleError so the caller can properly detect
the transaction as a rejected transaction instead of treating it like an
real error.
This commit modifies the transaction memory pool handling so that it does
not relay resurrected transactions. The other peers on the network will
also be reorganizing to the same block, so they already know about them.
This commit adds a few more checks to restrict what transactions are
allowed into the transaction memory pool and therefore are candidates
to be mined and relayed.
In particular, the following changes were made to what is considered
standard:
- nulldata scripts are now supported and considered standard
- multi-signature transaction are now checked to ensure they only have a
max of 3 pubkeys and the number of signatures doesn't exceed the number
of pubkeys
- the number of inputs to a signature script must now match the expected
number of inputs for the script type (includes support for additional
pay-to-script-hash inputs)
- the number of inputs pushed onto the stack by a redeeming sig script
must match the number of inputs consumed by the referenced pk script
- there can now only be a max of one nulldata output per transaction
This commit changes the various cases that were serializing transactions
into a buffer and taking the length to use the new faster SerializeSize
API. It also completes a TODO since the serialized size of a transaction
output is now available.
This commit updates btcd to work with the new btcchain APIs which now
accept btcutil.Tx instead of raw btcwire.MsgTx. It also modifies the
transaction memory pool to store btcutil.Tx.
This is part of the ongoing transaction hash optimization effort noted in
conformal/btcd#25.
The ValidateTransactionScripts was recently changed to accept script flags
which pass through to the script engine in order to control its validation
behavior. This commit modifies the transaction memory pool script
validation code for this change and additionally adds the new flag to
perform canonical signtaure checking.
It was previously possible for the unprotected iteration of the mempool
orphans to lead to undefined results. This commit remedies that by
reworking the locking code a bit. It also embeds the mutex directly into
the mempool struct rather than having a separate field for it so the
syntax is a slightly cleaner.
This commit adds code to properly respond to getdata requests for
transactions by fetching them from the transaction pool. Previously, we
advertised newly available transactions, but the code to respond with the
actual transaction was not written yet.
Also, fix a couple of comments and make the pushTxMsg and pushBlockMsg
functions consistent.
This commit is a first pass at improving the logging. It changes a number
of things to improve the readability of the output. The biggest addition
is message summaries for each message type when using the debug logging
level.
There is sitll more to do here such as allowing the level of each
subsystem to be independently specified, syslog support, and allowing the
logging level to be changed run-time.
The block manager handles inventory messges to know which inventory should
be requested based on what is already known and what is already in flight.
So, this commit adds logic to ask the transaction memory pool if the
transaction is already known before requesting it and tracks pending
requests into an in-flight transaction map owned by the block manager.
It also moves the transaction processing into the block manager so the
in-flight map can be properly cleaned.
Also, the loops which only remove a single element and break or return
don't need the extra logic for iteration since they don't continue
iteration after removal.
It is not safe to remove an item from a container/list while iterating the
list without first saving the next pointer since removing the item nils
the internal list element's next pointer.
This commit is a rather large one which implements transaction pool and
relay according to the protocol rules of the reference implementation.
It makes use of btcchain to ensure the transactions are valid for the
block chain and includes several stricter checks which determine if they
are "standard" or not before admitting them into the pool and relaying
them.
There are still a few TODOs around the more strict rules which determine
which transactions are willing to be mined, but the core checks which
are imperative (everything except the all of the "standard" checks really)
to operate as a good citizen on the bitcoin network are in place.
