This commit refactors the entire websocket client code to resolve several
issues with the previous implementation. Note that this commit does not
change the public API for websockets. It only consists of internal
improvements.
The following is the major issues which have been addressed:
- A slow websocket client could impede notifications to all clients
- Long-running operations such as rescans would block all other requests
until it had completed
- The above two points taken together could lead to apparant hangs since
the client doing the rescan would eventually run out of channel buffer
and block the entire group of clients until the rescan completed
- Disconnecting a websocket during certain operations could lead to a hang
- Stopping the rpc server with operations under way could lead to a hang
- There were no limits to the number of websocket clients that could
connect
The following is a summary of the major changes:
- The websocket code has been split into two entities: a
connection/notification manager and a websocket client
- The new connection/notification manager acts as the entry point from
the rest of the subsystems to feed data which potentially needs to
notify clients
- Each websocket client now has its own instance of the new websocket
client type which controls its own lifecycle
- The data flow has been completely redesigned to closely resemble the
peer data flow
- Each websocket now has its own long-lived goroutines for input, output,
and queuing of notifications
- Notifications use the new notification queue goroutine along with
queueing to ensure they dont't block on stalled or slow peers
- There is a new infrastructure for asynchronously executing long-running
commands such as a rescan while still allowing the faster operations to
continue to be serviced by the same client
- Since long-running operations now run asynchronously, they have been
limited to one at a time
- Added a limit of 10 websocket clients. This is hard coded for now, but
will be made configurable in the future
Taken together these changes make the code far easier to reason about and
update as well solve the aforementioned issues.
Further optimizations to improve performance are possible in regards to
the way the connection/notification manager works, however this commit
already contains a ton of changes, so they are being left for another
time.
This commit adds a new configuration option, --rpcmaxclients, to limit the
number of max standard RPC clients that are served concurrently. Note
that this value does not apply to websocket connections. A future commit
will add support for limiting those separately.
Closes#68.
Rather than using a type specifically in btcd for the getrawmempool, this
commit, along with a recent commit to btcjson, changes the code over to
use the type from btcjson. This is more consistent with other RPC results
and provides a few extra benefits such as the ability for btcjson to
automatically unmarshal the results into a concrete type with proper field
types as opposed to a generic interface.
Previously the getnettotals was just looping through all of the currently
connected peers to sum the byte counts and returning that. However, the
intention of the getnettotals RPC is to get all bytes since the server was
started, so this logic was not correct.
This commit modifies the code to keep an atomic counter on the server for
bytes read/written and has each peer update the server counters as well as
the per-peer counters.
This commit moves the connection endpoint for websockets to /ws instead of
/wallet. First, the former is more standard, and second the latter
presumes how the websocket is to be used.
Closes#80.
The getinfo RPC method requires access to information only available in
the wallet. Therefore, it has been moved to the list of methods which
return an error information the caller to send the request to the wallet
instead.
The getnewaddress RPC method deals with wallet-related functionality and
therefore has been moved to the list of methods which return an error
information the caller to send the request to the wallet instead.
Both of these RPC methods require access to information ony available in
the wallet. Therefore they have been moved to the list of methods which
return an error information the caller to send the request to the wallet
instead.
This commit adds full support for the getaddednodeinfo RPC command
including DNS lookups which abide by proxy/onion/tor rules when the DNS
flag is specified. Note that it returns an array of strings when the DNS
flag is not set which is different than the current version of bitcoind
which is bugged and scheduled to be fixed per issue 3581 on the bitcoind
issue tracker.
This commit improves how the legacy RPC server responds to authentication
failures so things like web browsers can react better. The following
changes have been made:
First, authentication failures were only printing the 401 error response
in the body instead of setting the http status code. This means the
response had a 200 OK header with a body of 401 Unauthorized. Therefore
the client would think everything was ok, but see the response as
malformed JSON.
Second, the spec for 401 Unauthorized responses state they must include a
WWW-Authenticate header to instruct the client how to authenticate.
Without this, browsers won't prompt the user for credentials.
The previous websocket code required HTTP auth headers to be sent in order
to use the websocket. While this makes sense for most applications, some
use cases such as javascript-based websockets from browsers do no have the
ability to send headers.
This commit modifies the authentication logic to allow an alternative
authentication mechanism when HTTP auth headers can't be sent. In
particular, it introduces a new JSON-RPC command named authenticate which
accepts the username and passphrase for the RPC server. The command is
only required if the connetion has not already been authenticated via HTTP
auth headers and it must be the first command to be received. Sending any
other command will immediately disconnect the websocket.
ok from @owainga and @jrick.
This closes#77.
The websocket.Server used the by websocket.Handler type automatically adds
a handshake function which prevents connections when the Origin header is
not set. Not all clients send this information and we already require
authentication headers as the auth mechanism anyways.
This removes the last notification that was being sent unsolicited.
Since it is no longer needed, the code to duplicate notifications to
all clients has been removed.
The RPC server was performing some of the shutdown logic in the wrong
order, that is, logging the the server has shut down, waiting for all
server goroutines to finish, and then closing a channel to notify
server goroutines to stop. These three items have been reversed to
fix a hang where goroutines currently being waited on had not shut
down because they did not receive the notification.
While here, the server waitgroup was incremented for a goroutine that
was running without it, another select statement was added to stop a
duplicate close (which never occured last commit when I added the
select statements), and the "stopping rescan" logging was moved to
debug to make the ^C shutdown logging nicer.
This changes the protocol between btcd and btcwallet to follow
JSON-RPC specifications sending notifications as requests with an
empty ID.
The notification request context handling has been greatly cleaned up
now that IDs no longer need to be saved when sending notifications.
This commit changes all code which deals with extracting addresses from
scripts to use the btcscript API ExtractPkScriptAddrs which in turn makes
use of the new btcutil.Address interface.
This provides much cleaner code for dealing with arbitrary script
destinations which is extensible without having to churn the APIs if new
destination types are added.
The name handlers for a package level is a bit too generic and could
easily cause a name collision. Even though the compiler would catch it,
use something a bit more descriptive.
Since the command to handler mappings are the most often modified and
referenced code in rpcserver.go and rpcwebsocket.go, move them near the
top of their respective files.
This commit cleans up the standard RPC command hanlding a bit by removing
the websocket specific notification channel from the handlers. This was
previously required because the sendrawtransaction, when called from a
websocket enabled connection, needs to add a notification for when the
transaction is mined.
This commit modifies that to instead implement a websocket extended
version of sendrawtransaction which invokes the standard handler and adds
the notification. In addition, the main send was modified to first look
if the command has a websocket specific handler first, and then falls back
to standard commands, rather than the previous approach of first checking
for a standard command and falling through to websocket commands. This
essentially allows websockets connections to extend commands with the same
name with additional functionality such as what was done in this commit.
The rpcserver.go file is starting to get a bit unwieldy. This commit
moves the separable websocket specific bits into a separate file named
rpcwebsocket.go.
Added error checking for script disassembley
Changed vout to handle errors in processing the way bitcoind does: the
type displayed is "nonstandard" when the calculated type is nonstandard
or nulltype and also when there is an error getting the address.
Still doesn't properly support multisig addresses, but now it should
return "nonstandard" since since address lookup fails for those cases.
Since the decoderawtransaction result makes use of the same vin and vout
lists, this commit also factors the logic for those out into separate
functions.
The ScriptSig field of the Vin type for TxRawResult is now a pointer in
btcjson so it can be properly omitted. This commit updates the code to
create the new ScriptSig object as needed.
The getrawtransaction RPC call should return a hex-encoded string of the
transaction when verbose is false instead of a TxRawResult object with the
Hex field set to be compatible with the Sathoshi client. This commit,
along with a recent commit to btcjson corrects this.
Also, while here, do a bit of cleanup, finish a TODO to check for an
invalid hash, and optimize the handling of non-verbose slightly.
The getblock RPC call should return a hex-encoded string of the block when
verbose is false instead of a BlockResult object with a Hex field set to
be compatible with the Sathoshi client. This commit, along with a recent
commit to btcjson corrects this.
Also, while here, move code which only applies to verbose mode after the
call which handles the non-verbose logic. This saves a few cycles since
the non-verbose logic doesn't need the extra information.
The fee field of the getrawmempool RPC JSON response should be in Bitcoins
instead of Satoshi. This commit corrects that issue.
Also, add a couple of comments and fix a comment typo while here.
This implements only the bare bones of external ip address selection
using very similar algorithms and selection methods to bitcoind. Every
address we bind to, and if we bind to the wildcard, every listening
address is recorded, and one for the appropriate address type of the
peer is selected.
Support for fetching addresses via upnp, external services, or via the
command line are not yet implemented.
Closes#35
Updated handleGetRawTransaction to populate all the fields required to
match bitcoind. It still doesn't handle MULTISIG addresses correctly.
Changed handleGetBlock to implement new optional verbose (default true)
flag and also added a verboseTx flag to return TxRawDefault instead of
Txid. When verbose=false, GetBlock returns hex-encoded wire bytes for
the block.
The vout field (as part of the getrawtransaction JSON reply) should be
set to the input's previous outpoint's index, not the current input
index.
Found by flam and reported on IRC. Thanks!
This switches a break to a continue if a txout does not include a
pay-to-pubkey-hash script type. btcwallet only supports
pay-to-pubkey-hash at the moment, and this fixes an issue where a tx
may have an different type of output, as well as pay-to-pubkey-hash,
which may be ignored by the wallet notification code.
Found by dhill.
I previously fixed the duplicate send (before seeing GH issue #54),
but forgot that btcwallet expects a nil reply when rescan has
finished. This adds the final reply back, but replies with nil.
Fixes#54.
This updates the replies for rescan and tx notifications with
additional information that is needed for wallet to properly support
the listtransactions command.
While here, drastically improve rescan performance by not looking up
every sha in rescan's block range.
Also, make every subsystem within btcd use its own logger instance so each
subsystem can have its own level specified independent of the others.
This is work towards #48.
All rpc sockets now listen using TLS by default, and this can not be
turned off. The keys (defauling to the datadirectory) may be provided by
--rpccert and --rpckey. If the keys do not exist we will generate a new
self-signed keypair with some sane defaults (hostname and all current
interface addresses).
Additionally add tls capability to btcctl so that it can still be used.
The certificate to use for verify can be provided on the commandline or
verification can be turned off (this leaves you susceptible to MITM
attacks)
Initial code from dhill (rpc tls support) and jrick (key generation),
cleanup, debugging and polishing from me.
This allows the provision of address/port pairs to be listened on instead
of just providing the port. e.g.:
btcd --listen 1.2.3.4:4321 --listen 127.0.0.01 --listen [::1]:5432
When --proxy and --connect are used, we disable listening *unless* any --listen
arguments have been provided, when we will listen on those addresses as
requested.
Initial code by davec, integration by myself.
Closes#33
allow listens to fail, but warn. error if all failed
fmt
So far we only do level 0 and level 1 checks (precense and basic
sanity). The checks done at higher levels in bitcoind are closely
coupled with their database layout.
arguably Closes#13
Previously, RemoveMinedTxRequest was being run from a caller which
held a reader lock for the websocket request contexts. When
RemoveMinedTxRequest tried to grab a writer lock, it would block.
This change creates a new function, removeMinedTxRequest, that does
not grab any locks, and the caller (NotifyBlockConnected) grabs a
writer lock instead of a reader lock.
Previously, on a blockconnected notification, the websocket context
reader lock was not always being given up properly. This change
defers the unlock so it will always happen.
This fixes an issue where wallet will stop responding (due to not
being able to complete its handshake) on reconnect.
This change reworks where the command parsing occurs to be done before
handlers are checked. Before, the websocket extension handler called
the standard handler with the same message, and if it was unhandled,
would unmarshal it a second time for checking extension handlers.
Redo the datastructures we search so that we only do one lookup per txin and
txout instead of doing a loop per wallet connection.
Don't send spent data on tx notifications, this can be worked out in wallet and
it is expensiveish to calculate. However we DO check upon getting a notification
request if the output is already spent, and in which case we send an immediate
notification to force a rescan.
MinedTxNotfications are handled separately to the connected block messages
largely to enable this to scale rather better.
Tested by jrick (who found one bug i had introduced, thanks!)
Additionally (accidentally squashed in):
Add handlers for all known commands.
We have handlers for all wallet-requiring commands that will return a suitable
error.
Unimplemented commands temporarily return an error stating so.
