Since a chain verification can take a long time depending on the
parameters, this commit adds a debug print to the RPC server at the info
level for how many blocks are being verified and at what level.
The logic was also slightly modified so the number of blocks being checked
can easily be calculated and shown.
This commit implements a rebroadcast handler which deals with
rebroadcasting inventory at a random time interval between 0 and 30
minutes. It then uses the new rebroadcast logic to ensure transactions
which were submitted via the sendrawtransaction RPC are rebroadcast until
they make it into a block.
Closes#99.
Rather than using the deprecated TxShas function on a btcutil.Block,
convert handleGetBlock to use the newer preferred method of ranging over
the Transactions to obtain the cached hash of each transaction.
This is a little more efficient since it can avoid creating and caching an
extra slice to keep the hashes in addition to having the hash cached with
each transaction.
Rather than returning an error when creating the RPC server an it can't
listen on any of the specified interfaces, only error when it can't listen
on all of the specified interfaces.
This change modifies the RPC server's notifiation manager from a
struct with requests, protected by a mutux, to two goroutines. The
first maintains a queue of all notifications and control requests
(registering/unregistering notifications), while the second reads from
the queue and processes notifications and requests one at a time.
Previously, to prevent slowing down block and mempool processing, each
notification would be handled by spawning a new goroutine. This lead
to cases where notifications would end up being sent to clients in a
different order than they were created. Adding a queue keeps the
order of notifications originating from the same goroutine, while also
not slowing down processing while waiting for notifications to be
processed and sent.
ok @davecgh
This changes the implementation of the sendrawtransaction RPC handler
to match bitcoind behavior by always returning a rejection error for
any error processing or accepting the tx by the mempool. Previously,
if the tx was rejected for a rule error rather than an actual failure,
a client would still receive the tx sha as a result with no error.
This commit refactors the entire websocket client code to resolve several
issues with the previous implementation. Note that this commit does not
change the public API for websockets. It only consists of internal
improvements.
The following is the major issues which have been addressed:
- A slow websocket client could impede notifications to all clients
- Long-running operations such as rescans would block all other requests
until it had completed
- The above two points taken together could lead to apparant hangs since
the client doing the rescan would eventually run out of channel buffer
and block the entire group of clients until the rescan completed
- Disconnecting a websocket during certain operations could lead to a hang
- Stopping the rpc server with operations under way could lead to a hang
- There were no limits to the number of websocket clients that could
connect
The following is a summary of the major changes:
- The websocket code has been split into two entities: a
connection/notification manager and a websocket client
- The new connection/notification manager acts as the entry point from
the rest of the subsystems to feed data which potentially needs to
notify clients
- Each websocket client now has its own instance of the new websocket
client type which controls its own lifecycle
- The data flow has been completely redesigned to closely resemble the
peer data flow
- Each websocket now has its own long-lived goroutines for input, output,
and queuing of notifications
- Notifications use the new notification queue goroutine along with
queueing to ensure they dont't block on stalled or slow peers
- There is a new infrastructure for asynchronously executing long-running
commands such as a rescan while still allowing the faster operations to
continue to be serviced by the same client
- Since long-running operations now run asynchronously, they have been
limited to one at a time
- Added a limit of 10 websocket clients. This is hard coded for now, but
will be made configurable in the future
Taken together these changes make the code far easier to reason about and
update as well solve the aforementioned issues.
Further optimizations to improve performance are possible in regards to
the way the connection/notification manager works, however this commit
already contains a ton of changes, so they are being left for another
time.
This commit adds a new configuration option, --rpcmaxclients, to limit the
number of max standard RPC clients that are served concurrently. Note
that this value does not apply to websocket connections. A future commit
will add support for limiting those separately.
Closes#68.
Rather than using a type specifically in btcd for the getrawmempool, this
commit, along with a recent commit to btcjson, changes the code over to
use the type from btcjson. This is more consistent with other RPC results
and provides a few extra benefits such as the ability for btcjson to
automatically unmarshal the results into a concrete type with proper field
types as opposed to a generic interface.
Previously the getnettotals was just looping through all of the currently
connected peers to sum the byte counts and returning that. However, the
intention of the getnettotals RPC is to get all bytes since the server was
started, so this logic was not correct.
This commit modifies the code to keep an atomic counter on the server for
bytes read/written and has each peer update the server counters as well as
the per-peer counters.
This commit moves the connection endpoint for websockets to /ws instead of
/wallet. First, the former is more standard, and second the latter
presumes how the websocket is to be used.
Closes#80.
The getinfo RPC method requires access to information only available in
the wallet. Therefore, it has been moved to the list of methods which
return an error information the caller to send the request to the wallet
instead.
The getnewaddress RPC method deals with wallet-related functionality and
therefore has been moved to the list of methods which return an error
information the caller to send the request to the wallet instead.
Both of these RPC methods require access to information ony available in
the wallet. Therefore they have been moved to the list of methods which
return an error information the caller to send the request to the wallet
instead.
This commit adds full support for the getaddednodeinfo RPC command
including DNS lookups which abide by proxy/onion/tor rules when the DNS
flag is specified. Note that it returns an array of strings when the DNS
flag is not set which is different than the current version of bitcoind
which is bugged and scheduled to be fixed per issue 3581 on the bitcoind
issue tracker.
This commit improves how the legacy RPC server responds to authentication
failures so things like web browsers can react better. The following
changes have been made:
First, authentication failures were only printing the 401 error response
in the body instead of setting the http status code. This means the
response had a 200 OK header with a body of 401 Unauthorized. Therefore
the client would think everything was ok, but see the response as
malformed JSON.
Second, the spec for 401 Unauthorized responses state they must include a
WWW-Authenticate header to instruct the client how to authenticate.
Without this, browsers won't prompt the user for credentials.
The previous websocket code required HTTP auth headers to be sent in order
to use the websocket. While this makes sense for most applications, some
use cases such as javascript-based websockets from browsers do no have the
ability to send headers.
This commit modifies the authentication logic to allow an alternative
authentication mechanism when HTTP auth headers can't be sent. In
particular, it introduces a new JSON-RPC command named authenticate which
accepts the username and passphrase for the RPC server. The command is
only required if the connetion has not already been authenticated via HTTP
auth headers and it must be the first command to be received. Sending any
other command will immediately disconnect the websocket.
ok from @owainga and @jrick.
This closes#77.
The websocket.Server used the by websocket.Handler type automatically adds
a handshake function which prevents connections when the Origin header is
not set. Not all clients send this information and we already require
authentication headers as the auth mechanism anyways.
This removes the last notification that was being sent unsolicited.
Since it is no longer needed, the code to duplicate notifications to
all clients has been removed.
The RPC server was performing some of the shutdown logic in the wrong
order, that is, logging the the server has shut down, waiting for all
server goroutines to finish, and then closing a channel to notify
server goroutines to stop. These three items have been reversed to
fix a hang where goroutines currently being waited on had not shut
down because they did not receive the notification.
While here, the server waitgroup was incremented for a goroutine that
was running without it, another select statement was added to stop a
duplicate close (which never occured last commit when I added the
select statements), and the "stopping rescan" logging was moved to
debug to make the ^C shutdown logging nicer.
This changes the protocol between btcd and btcwallet to follow
JSON-RPC specifications sending notifications as requests with an
empty ID.
The notification request context handling has been greatly cleaned up
now that IDs no longer need to be saved when sending notifications.
This commit changes all code which deals with extracting addresses from
scripts to use the btcscript API ExtractPkScriptAddrs which in turn makes
use of the new btcutil.Address interface.
This provides much cleaner code for dealing with arbitrary script
destinations which is extensible without having to churn the APIs if new
destination types are added.
The name handlers for a package level is a bit too generic and could
easily cause a name collision. Even though the compiler would catch it,
use something a bit more descriptive.
Since the command to handler mappings are the most often modified and
referenced code in rpcserver.go and rpcwebsocket.go, move them near the
top of their respective files.
This commit cleans up the standard RPC command hanlding a bit by removing
the websocket specific notification channel from the handlers. This was
previously required because the sendrawtransaction, when called from a
websocket enabled connection, needs to add a notification for when the
transaction is mined.
This commit modifies that to instead implement a websocket extended
version of sendrawtransaction which invokes the standard handler and adds
the notification. In addition, the main send was modified to first look
if the command has a websocket specific handler first, and then falls back
to standard commands, rather than the previous approach of first checking
for a standard command and falling through to websocket commands. This
essentially allows websockets connections to extend commands with the same
name with additional functionality such as what was done in this commit.
The rpcserver.go file is starting to get a bit unwieldy. This commit
moves the separable websocket specific bits into a separate file named
rpcwebsocket.go.
Added error checking for script disassembley
Changed vout to handle errors in processing the way bitcoind does: the
type displayed is "nonstandard" when the calculated type is nonstandard
or nulltype and also when there is an error getting the address.
Still doesn't properly support multisig addresses, but now it should
return "nonstandard" since since address lookup fails for those cases.
Since the decoderawtransaction result makes use of the same vin and vout
lists, this commit also factors the logic for those out into separate
functions.
The ScriptSig field of the Vin type for TxRawResult is now a pointer in
btcjson so it can be properly omitted. This commit updates the code to
create the new ScriptSig object as needed.
The getrawtransaction RPC call should return a hex-encoded string of the
transaction when verbose is false instead of a TxRawResult object with the
Hex field set to be compatible with the Sathoshi client. This commit,
along with a recent commit to btcjson corrects this.
Also, while here, do a bit of cleanup, finish a TODO to check for an
invalid hash, and optimize the handling of non-verbose slightly.
The getblock RPC call should return a hex-encoded string of the block when
verbose is false instead of a BlockResult object with a Hex field set to
be compatible with the Sathoshi client. This commit, along with a recent
commit to btcjson corrects this.
Also, while here, move code which only applies to verbose mode after the
call which handles the non-verbose logic. This saves a few cycles since
the non-verbose logic doesn't need the extra information.
The fee field of the getrawmempool RPC JSON response should be in Bitcoins
instead of Satoshi. This commit corrects that issue.
Also, add a couple of comments and fix a comment typo while here.
This implements only the bare bones of external ip address selection
using very similar algorithms and selection methods to bitcoind. Every
address we bind to, and if we bind to the wildcard, every listening
address is recorded, and one for the appropriate address type of the
peer is selected.
Support for fetching addresses via upnp, external services, or via the
command line are not yet implemented.
Closes#35
Updated handleGetRawTransaction to populate all the fields required to
match bitcoind. It still doesn't handle MULTISIG addresses correctly.
Changed handleGetBlock to implement new optional verbose (default true)
flag and also added a verboseTx flag to return TxRawDefault instead of
Txid. When verbose=false, GetBlock returns hex-encoded wire bytes for
the block.
The vout field (as part of the getrawtransaction JSON reply) should be
set to the input's previous outpoint's index, not the current input
index.
Found by flam and reported on IRC. Thanks!
This switches a break to a continue if a txout does not include a
pay-to-pubkey-hash script type. btcwallet only supports
pay-to-pubkey-hash at the moment, and this fixes an issue where a tx
may have an different type of output, as well as pay-to-pubkey-hash,
which may be ignored by the wallet notification code.
Found by dhill.
I previously fixed the duplicate send (before seeing GH issue #54),
but forgot that btcwallet expects a nil reply when rescan has
finished. This adds the final reply back, but replies with nil.
Fixes#54.
This updates the replies for rescan and tx notifications with
additional information that is needed for wallet to properly support
the listtransactions command.
While here, drastically improve rescan performance by not looking up
every sha in rescan's block range.
Also, make every subsystem within btcd use its own logger instance so each
subsystem can have its own level specified independent of the others.
This is work towards #48.
All rpc sockets now listen using TLS by default, and this can not be
turned off. The keys (defauling to the datadirectory) may be provided by
--rpccert and --rpckey. If the keys do not exist we will generate a new
self-signed keypair with some sane defaults (hostname and all current
interface addresses).
Additionally add tls capability to btcctl so that it can still be used.
The certificate to use for verify can be provided on the commandline or
verification can be turned off (this leaves you susceptible to MITM
attacks)
Initial code from dhill (rpc tls support) and jrick (key generation),
cleanup, debugging and polishing from me.
This allows the provision of address/port pairs to be listened on instead
of just providing the port. e.g.:
btcd --listen 1.2.3.4:4321 --listen 127.0.0.01 --listen [::1]:5432
When --proxy and --connect are used, we disable listening *unless* any --listen
arguments have been provided, when we will listen on those addresses as
requested.
Initial code by davec, integration by myself.
Closes#33
allow listens to fail, but warn. error if all failed
fmt
So far we only do level 0 and level 1 checks (precense and basic
sanity). The checks done at higher levels in bitcoind are closely
coupled with their database layout.
arguably Closes#13
Previously, RemoveMinedTxRequest was being run from a caller which
held a reader lock for the websocket request contexts. When
RemoveMinedTxRequest tried to grab a writer lock, it would block.
This change creates a new function, removeMinedTxRequest, that does
not grab any locks, and the caller (NotifyBlockConnected) grabs a
writer lock instead of a reader lock.
Previously, on a blockconnected notification, the websocket context
reader lock was not always being given up properly. This change
defers the unlock so it will always happen.
This fixes an issue where wallet will stop responding (due to not
being able to complete its handshake) on reconnect.
This change reworks where the command parsing occurs to be done before
handlers are checked. Before, the websocket extension handler called
the standard handler with the same message, and if it was unhandled,
would unmarshal it a second time for checking extension handlers.
Redo the datastructures we search so that we only do one lookup per txin and
txout instead of doing a loop per wallet connection.
Don't send spent data on tx notifications, this can be worked out in wallet and
it is expensiveish to calculate. However we DO check upon getting a notification
request if the output is already spent, and in which case we send an immediate
notification to force a rescan.
MinedTxNotfications are handled separately to the connected block messages
largely to enable this to scale rather better.
Tested by jrick (who found one bug i had introduced, thanks!)
Additionally (accidentally squashed in):
Add handlers for all known commands.
We have handlers for all wallet-requiring commands that will return a suitable
error.
Unimplemented commands temporarily return an error stating so.
This change allows map lookups using address hashes (which are
returned as []byte) instead of either copying the hash into an array,
or doing a bytes.Equal().
A stupid range over a map until the right key is found was also just
changed to a single map lookup.
This adds to the initial rescan implementation, but switches it to
rescan based on a group of addresses, rather than just one. Due to
how expensive database lookups are during a rescan, wallets should
take advantage of this to rescan once for all needed addresses for all
accounts.
This commit changes the various cases that were serializing transactions
into a buffer and taking the length to use the new faster SerializeSize
API. It also completes a TODO since the serialized size of a transaction
output is now available.
Rather than showing all errors from ProcessTransaction as an error, check
if the error is a TxRuleError meaning the transaction was rejected as
opposed to something actually going wrong and log it accordingly.
Looking up transactions from the database is an expensive operation.
This commit modifies the NotifyNewTxListener code to simply iterate the
transactions in the block instead of looking them up from the db.
Currently the wallet code needs a spent flag which ultimately shouldn't be
required. For now, the spent data is simply created on the fly which is
still significantly faster than doing database transaction lookups.
Closes#24.
This change unbreaks the case where an unknown command is sent to the
RPC server. Instead of replying back with a nil JSON id, if the
initial unmarshal was successful (and thus, the message was valid
JSON-RPC), the unmarshaled id will be used in the error reply.
We have a channel for queries and commands in server, where we pass in
args and the channel to reply from, let rpcserver use these interfaces
to provide the requistie information.
So far not all of the informaation is 100% correct, the syncpeer
information needs to be fetched from blockmanager, the subversion isn't
recorded and the number of bytes sent and recieved needs to be obtained
from btcwire. The rest should be correct.
This commit updates btcd to work with the new btcchain APIs which now
accept btcutil.Tx instead of raw btcwire.MsgTx. It also modifies the
transaction memory pool to store btcutil.Tx.
This is part of the ongoing transaction hash optimization effort noted in
conformal/btcd#25.
This change allows btcwallet to keep a pool of transactions that have
not yet been mined into a block, notifying wallet when transactions
are mined, as well as introducing a new way to send the
btcd:blockconnected notification with wallet-specific information as
part of the same notification. When a transaction is sent using the
RPC call 'sendrawtransaction', a notification request will be
automatically registered with the connected wallet (if using
websockets) to notify the wallet when the transaction first appears in
a block.
To perform this notification, and to avoid requiring wallets from
waiting for seperate mined tx notifications (and resend after a
timeout) or from sending an additional tx mined request for every tx
in the pool after each new block, the blockconnected notification is
now created seperately for each wallet. If the notified wallet has
sent a transaction, an additional JSON field "minedtxs" will include
an array of transaction IDs that the wallet has created and which are
included in the new block.
This new unique blockconnected notification can also be used for
additional notifications that may happen each new block in the future,
and to cut down on existing notification handlers in btcwallet, such
as for transactions to a watched address.
Rather than relying on the http package's DefaultServeMux for the RPC
server, create a unique mux specifically for the RPC server. This ensures
things, such as the http profiling handlers, do not commingle.
Results from FetchTxByShaList must each be checked for a nil Err and a
non-nil Tx. Fix this issue in two places where these conditions were
not being checked.
The latest websockets code added a quit channel to the RPC server, but did
not initialize it. This commit corrects that so shutdown works properly
again.
This refactors the wallet notification code to reverse the order of
how notification contexts are stored. Before, watched addresses and
outpoints were used as keys, with a special reply channel as the
value. This channel was read from and replies were marshalled and
sent to the main wallet notification chan, but the goroutine handling
this marshalling never exited because the reply channel was never
closed (and couldn't have been, because there was no way to tell it
was handling notifications for any particular wallet).
Notification contexts are now primarily mapped by wallet notification
channels, and code to send the notifications send directly to the
wallet channel, with the previous goroutine reading the reply chan
properly closing.
The RPC code is also refactored with this change as well, to separate
it more from websocket code. Websocket JSON extensions are no longer
available to RPC clients.
While here, unbreak RPC. Previously, replies were never sent back.
This broke when I merged in my websocket code, as sends for the reply
channel in jsonRead blocked before a reader for the channel was
opened. A 3 liner could have fixed this, but doing a proper fix
(changing jsonRead so it did not use the reply channel as it is
unneeded for the standard RPC API) is preferred.
This change adds additional http listeners for websocket connections
on "/wallet". Websockets are used to provide asynchronous messaging
between wallet daemons (i.e. btcwallet) and btcd as they allow an easy
way for btcd to provide instant notifications (instead of a wallet
polling for updates) and multiple replies to a single request.
Standard RPC commands sent over a websocket connection are handled
just like RPC, returning the same results, the only difference being
that the connection is async. In cases where the standard RPC
commands fall short of wallet daemons requests, and to request
notifications for addresses and events, extension JSON methods are
used.
Multiple wallets can be connected to the same btcd, and replies to
websocket requests and notifications are properly routed back to the
original requesting wallet.
Due to the nature of turning a synchronous protocol asynchronous, it
is highly recommended to use the JSON id field as a type of sequence
number, so replies from btcd can be routed back to the proper handler
in a wallet daemon.
This commit is a first pass at improving the logging. It changes a number
of things to improve the readability of the output. The biggest addition
is message summaries for each message type when using the debug logging
level.
There is sitll more to do here such as allowing the level of each
subsystem to be independently specified, syslog support, and allowing the
logging level to be changed run-time.
This commit changes the code so that all calls to .Add on waitgroups
happen before the associated goroutines are launched. Doing this after
the goroutine could technically cause a race where the goroutine started
and finished before the main goroutine has a chance to increment the
counter. In our particular case none of the goroutines exit quickly
enough for this to be an issue, but nevertheless the correct way should be
used.
