Copying the RIPEMD160 after SHA256 hash result into a new stack array
to be used as a map lookup key can be quite expensive, and this should
be avoided if possible on intensive tasks such as rescans. This
change takes advantage of the new Hash160 methods of the
AddressPubKeyHash and AddressScriptHash types to use the address's
underlying hash array directly, rather than creating a copy from the
ScriptAddress result.
Unfortunately, for AddressPubKey, ScriptAddress may return either a
byte slice of len 33 or 65 depending on whether the pubkey is
compressed or not, so no such straightforward optimization is
possible.
As a result of this change, I have seen rescans perform roughly 3.5x
faster than before.
The websocket extension command to register for notifications when a new
transaction has been accepted to the memory pool and the resulting
notifications have been renamed. This commit catches up to the change.
This change periodically (about every 10 seconds) notifies the
connected websocket client of the height of the last processed block
as part of the rescan. This enables clients to listen for the
notification and keep track of how much progress a rescan has made
even without any results being found. If the websocket connection is
lost during the rescan, on reconnect, clients may safely start over at
the last notified height.
This change adds select statements to each channel write that may
occur from a non-RPCS goroutine to unblock once the server has begun
shutting down. This prevents issues in clean shutdown where non-RPCS
goroutines may block indefinitely on message sends to the RPC server.
This change adds rescan fast paths for all current implementations of
the btcutil.Address interface. Rather than encoding a payment address
string from a txout script, the minimum number of bytes (which depends
on the address type) are copied into a stack array and used as a
lookup key to a map exclusive for that address type. This performs
better than the previous implementation due to avoiding the base58
encoding and generating less garbage (so each GC cycle completes
faster).
If the address type switch falls into the default (unhandled) case,
the encoded payment address is used as a fallback. This keeps the
intended rescan behavior even when new implementations of
btcutil.Address are added without a rescan fast path.
Benchmarks indicate that when a fast path is followed, for 20 byte
RIPEMD-160 hashes (such as P2PKH and P2SH), skipping the payment
address encoding (and thus bypassing the base58 encoding code) can
result in map lookups up to 60x faster.
This change modifies the RPC server's notifiation manager from a
struct with requests, protected by a mutux, to two goroutines. The
first maintains a queue of all notifications and control requests
(registering/unregistering notifications), while the second reads from
the queue and processes notifications and requests one at a time.
Previously, to prevent slowing down block and mempool processing, each
notification would be handled by spawning a new goroutine. This lead
to cases where notifications would end up being sent to clients in a
different order than they were created. Adding a queue keeps the
order of notifications originating from the same goroutine, while also
not slowing down processing while waiting for notifications to be
processed and sent.
ok @davecgh
This commit changes the websocket client code to use a mutex for
disconnect since it's theoretically possible a non-blocking select on the
quit channel could fall through from two different goroutines thus causing
a second call to close.
ok @jrick.
This change improves the mechanism by which btcd notifies a websocket
client of transaction relating to watched address and unspent outputs
in the following ways:
1. The old processedtx notification has been replaced with the new
recvtx notification. This notification, rather than parsing out
details used by wallet clients, sends the serialized transaction
(as hexadecimal) and any block details (if mined) if any transaction
output sends to one of the websocket client's watched addresses.
2. The old txspent notification has been replaced with the new
redeemingtx notification. This notification, rather than parsing
out details used by wallet clients, sends the serialized transaction
(as hexadecimal) and any block details (if mined) if any transaction
input spends a watched output.
3. When processing notifications for transaction outputs, if any output
spends to a client's watched address, a corresponding spent request
is automatically registered.
4. Transaction notifications originating from mempool now include both
transaction inputs and outputs, rather than only processing
5. When processing notifications for transaction inputs, a client's
output spent request is only removed if the transaction being
processed has also been mined into a block. In combination with the
4th change, this results in two redeemingtx notifications for
transactions which first appear in mempool and are subsequently mined
into a block.
Previously the websocket notifications for addresses were limited to
pay-to-pubkey-hash only. This commit removes that restriction so
all btcutil.Address types are supported. This includes pay-to-pubkey,
pay-to-pubkey-hash, and pay-to-script-hash.
When a spent notification and address notification is removed, the
tracking entry in the client which is used to track what to remove on
shutdown needs to be removed as well.
This commit refactors the entire websocket client code to resolve several
issues with the previous implementation. Note that this commit does not
change the public API for websockets. It only consists of internal
improvements.
The following is the major issues which have been addressed:
- A slow websocket client could impede notifications to all clients
- Long-running operations such as rescans would block all other requests
until it had completed
- The above two points taken together could lead to apparant hangs since
the client doing the rescan would eventually run out of channel buffer
and block the entire group of clients until the rescan completed
- Disconnecting a websocket during certain operations could lead to a hang
- Stopping the rpc server with operations under way could lead to a hang
- There were no limits to the number of websocket clients that could
connect
The following is a summary of the major changes:
- The websocket code has been split into two entities: a
connection/notification manager and a websocket client
- The new connection/notification manager acts as the entry point from
the rest of the subsystems to feed data which potentially needs to
notify clients
- Each websocket client now has its own instance of the new websocket
client type which controls its own lifecycle
- The data flow has been completely redesigned to closely resemble the
peer data flow
- Each websocket now has its own long-lived goroutines for input, output,
and queuing of notifications
- Notifications use the new notification queue goroutine along with
queueing to ensure they dont't block on stalled or slow peers
- There is a new infrastructure for asynchronously executing long-running
commands such as a rescan while still allowing the faster operations to
continue to be serviced by the same client
- Since long-running operations now run asynchronously, they have been
limited to one at a time
- Added a limit of 10 websocket clients. This is hard coded for now, but
will be made configurable in the future
Taken together these changes make the code far easier to reason about and
update as well solve the aforementioned issues.
Further optimizations to improve performance are possible in regards to
the way the connection/notification manager works, however this commit
already contains a ton of changes, so they are being left for another
time.
The wsContext was being locked twice when NewBlockNotifyCheckTxIn is
called. Fixed by changing handlers to assume lock is acquired and
renamed methods to not be exported.
Since the websocket handlers run in their own separate goroutines, it's
possible that they execute after the websocket connection has been
closed and cleaned up. This commit add the necessary checks to ensure
stale data isn't added to notification lists and that requests to closed
connections are ignored.
This closes#92.
Access to connections map and associated notification maps in rpcServer
need to be protected with s.ws.Lock to prevent race with add/remove new
clients.
This closes#88.
Changed mempool.MaybeAcceptTransaction to accept an additional parameter
to differentiate betwee new transactions and those added from
disconnected blocks.
Added new fields to requestContexts to indicate which clients want to
receive all new transaction notifications.
Added NotifyForNewTx to rpcServer to deliver approriate transaction
notification.
The previous websocket code required HTTP auth headers to be sent in order
to use the websocket. While this makes sense for most applications, some
use cases such as javascript-based websockets from browsers do no have the
ability to send headers.
This commit modifies the authentication logic to allow an alternative
authentication mechanism when HTTP auth headers can't be sent. In
particular, it introduces a new JSON-RPC command named authenticate which
accepts the username and passphrase for the RPC server. The command is
only required if the connetion has not already been authenticated via HTTP
auth headers and it must be the first command to be received. Sending any
other command will immediately disconnect the websocket.
ok from @owainga and @jrick.
This closes#77.
This fixes two issues: first, the sendrawtransaction handler had an
extra character in the key in the websocket handler map, preventing
the handler from never running. Second, a nil pointer dereference was
removed from the handler.
This change fixes the minedtx notifications for btcwallet, since the
websocket-handler now runs instead of falling back to the legacy RPC
handler.
The websocket package assumes binary blobs if []byte is used. Since we're
using JSON-RPC for all the websocket communications, it should be text
based.
This commit changes the websocket Send/Receive and associated channels to
strings accordingly.
This removes the last notification that was being sent unsolicited.
Since it is no longer needed, the code to duplicate notifications to
all clients has been removed.
The RPC server was performing some of the shutdown logic in the wrong
order, that is, logging the the server has shut down, waiting for all
server goroutines to finish, and then closing a channel to notify
server goroutines to stop. These three items have been reversed to
fix a hang where goroutines currently being waited on had not shut
down because they did not receive the notification.
While here, the server waitgroup was incremented for a goroutine that
was running without it, another select statement was added to stop a
duplicate close (which never occured last commit when I added the
select statements), and the "stopping rescan" logging was moved to
debug to make the ^C shutdown logging nicer.
This changes the protocol between btcd and btcwallet to follow
JSON-RPC specifications sending notifications as requests with an
empty ID.
The notification request context handling has been greatly cleaned up
now that IDs no longer need to be saved when sending notifications.
This commit changes all code which deals with extracting addresses from
scripts to use the btcscript API ExtractPkScriptAddrs which in turn makes
use of the new btcutil.Address interface.
This provides much cleaner code for dealing with arbitrary script
destinations which is extensible without having to churn the APIs if new
destination types are added.
Since the command to handler mappings are the most often modified and
referenced code in rpcserver.go and rpcwebsocket.go, move them near the
top of their respective files.
This commit cleans up the standard RPC command hanlding a bit by removing
the websocket specific notification channel from the handlers. This was
previously required because the sendrawtransaction, when called from a
websocket enabled connection, needs to add a notification for when the
transaction is mined.
This commit modifies that to instead implement a websocket extended
version of sendrawtransaction which invokes the standard handler and adds
the notification. In addition, the main send was modified to first look
if the command has a websocket specific handler first, and then falls back
to standard commands, rather than the previous approach of first checking
for a standard command and falling through to websocket commands. This
essentially allows websockets connections to extend commands with the same
name with additional functionality such as what was done in this commit.
The rpcserver.go file is starting to get a bit unwieldy. This commit
moves the separable websocket specific bits into a separate file named
rpcwebsocket.go.
