This is a rather monolithic commit that moves the old RPC server to
its own package (rpc/legacyrpc), introduces a new RPC server using
gRPC (rpc/rpcserver), and provides the ability to defer wallet loading
until request at a later time by an RPC (--noinitialload).
The legacy RPC server remains the default for now while the new gRPC
server is not enabled by default. Enabling the new server requires
setting a listen address (--experimenalrpclisten). This experimental
flag is used to effectively feature gate the server until it is ready
to use as a default. Both RPC servers can be run at the same time,
but require binding to different listen addresses.
In theory, with the legacy RPC server now living in its own package it
should become much easier to unit test the handlers. This will be
useful for any future changes to the package, as compatibility with
Core's wallet is still desired.
Type safety has also been improved in the legacy RPC server. Multiple
handler types are now used for methods that do and do not require the
RPC client as a dependency. This can statically help prevent nil
pointer dereferences, and was very useful for catching bugs during
refactoring.
To synchronize the wallet loading process between the main package
(the default) and through the gRPC WalletLoader service (with the
--noinitialload option), as well as increasing the loose coupling of
packages, a new wallet.Loader type has been added. All creating and
loading of existing wallets is done through a single Loader instance,
and callbacks can be attached to the instance to run after the wallet
has been opened. This is how the legacy RPC server is associated with
a loaded wallet, even after the wallet is loaded by a gRPC method in a
completely unrelated package.
Documentation for the new RPC server has been added to the
rpc/documentation directory. The documentation includes a
specification for the new RPC API, addresses how to make changes to
the server implementation, and provides short example clients in
several different languages.
Some of the new RPC methods are not implementated exactly as described
by the specification. These are considered bugs with the
implementation, not the spec. Known bugs are commented as such.
This change moves the chain and network parameter definitions, along
with the default client and server ports, to a package for reuse by
other utilities (most notably, tools in the cmd dir). Along with it,
functions commonly used for config parsing and validation are moved to
an internal package since they will also be useful for distributed
tools.
This a refactor of the btcwallet main package to create a new wallet
package.
The main feature of this package is the integration of all the other
wallet components (waddrmgr, txstore, and chain) and the Wallet type is
'runnable', so it will be continuously updating itself against changes
notified by the remote btcd instance.
It also includes several methods which provide access to information
necessary to run a wallet RPC server.
The createtemp flag is used to create a temporary simnet wallet for
use with btcsim or other testing suites.
The flag only works if the datadir and network to use it on (simnet)
are specified.
After starting btcwallet with the flag, the wallet immediately opens
and is functional.
This commit converts the wallet to use the new secure hierarchical
deterministic wallet address manager package as well as the walletdb
package.
The following is an overview of modified functionality:
- The wallet must now be created before starting the executable
- A new flag --create has been added to create the new wallet using wizard
style question and answer prompts
- Starting the process without an existing wallet will instruct now
display a message to run it with --create
- Providing the --create flag with an existing wallet will simply show an
error and return
In addition the snacl package has been modified to return the memory after
performing scrypt operations to the OS.
Previously a runtime.GC was being invoked which forced it to release the
memory as far as the garbage collector is concerned, but the memory was
not released back to the OS immediatley. This modification allows the
memory to be released immedately since it won't be needed again until the
next wallet unlock.
This commit introduces a new flag, --noclienttls, which can be used to disable
TLS for the RPC client. However, the flag can only be used when the RPC
client is connecting to localhost interfaces. This is intended to prevent
accidentally leaking sensitive data when switching between local and
remote servers.
This commit introduces a new flag, --noservertls, which can be used to disable
TLS for the RPC server. However, the flag can only be used when the RPC
server is bound to localhost interfaces. This is intended to prevent the
situation where someone decides they want to expose the RPC server to the
web for remote management/access, but forgot they have TLS disabled.
This commit is the result of several big changes being made to the
wallet. In particular, the "handshake" (initial sync to the chain
server) was quite racy and required proper synchronization. To make
fixing this race easier, several other changes were made to the
internal wallet data structures and much of the RPC server ended up
being rewritten.
First, all account support has been removed. The previous Account
struct has been replaced with a Wallet structure, which includes a
keystore for saving keys, and a txstore for storing relevant
transactions. This decision has been made since it is the opinion of
myself and other developers that bitcoind accounts are fundamentally
broken (as accounts implemented by bitcoind support both arbitrary
address groupings as well as moving balances between accounts -- these
are fundamentally incompatible features), and since a BIP0032 keystore
is soon planned to be implemented (at which point, "accounts" can
return as HD extended keys). With the keystore handling the grouping
of related keys, there is no reason have many different Account
structs, and the AccountManager has been removed as well. All RPC
handlers that take an account option will only work with "" (the
default account) or "*" if the RPC allows specifying all accounts.
Second, much of the RPC server has been cleaned up. The global
variables for the RPC server and chain server client have been moved
to part of the rpcServer struct, and the handlers for each RPC method
that are looked up change depending on which components have been set.
Passthrough requests are also no longer handled specially, but when
the chain server is set, a handler to perform the passthrough will be
returned if the method is not otherwise a wallet RPC. The
notification system for websocket clients has also been rewritten so
wallet components can send notifications through channels, rather than
requiring direct access to the RPC server itself, or worse still,
sending directly to a websocket client's send channel. In the future,
this will enable proper registration of notifications, rather than
unsolicited broadcasts to every connected websocket client (see
issue #84).
Finally, and the main reason why much of this cleanup was necessary,
the races during intial sync with the chain server have been fixed.
Previously, when the 'Handshake' was run, a rescan would occur which
would perform modifications to Account data structures as
notifications were received. Synchronization was provided with a
single binary semaphore which serialized all access to wallet and
account data. However, the Handshake itself was not able to run with
this lock (or else notifications would block), and many data races
would occur as both notifications were being handled. If GOMAXPROCS
was ever increased beyond 1, btcwallet would always immediately crash
due to invalid addresses caused by the data races on startup. To fix
this, the single lock for all wallet access has been replaced with
mutexes for both the keystore and txstore. Handling of btcd
notifications and client requests may now occur simultaneously.
GOMAXPROCS has also been set to the number of logical CPUs at the
beginning of main, since with the data races fixed, there's no reason
to prevent the extra parallelism gained by increasing it.
Closes#78.
Closes#101.
Closes#110.
The info log level (default) will produce output about confirmed and
unconfirmed transactions being inserted into the store, as well as
unconfirmed transactions which have been mined into blocks. By
enabling the debug log level (-d TXST=debug), additional information
about transaction inputs and outputs is logged. This includes the
total amount of previously-unspent outputs which have been marked
spent by the inserted transaction, and the output indexes and amounts
for each spendable output. Additionally, the debug log level will log
whenever transactions are removed due to being a double spend of
another inserted transaction.
18555 is the network port, and wallet cannot listen on this port.
Instead, follow the pattern laid out by mainnet, testnet3, and regtest
by listening on 18554 (one less than the network port).
If the data directory is modified on the command line or from the
config file, all paths relative to the directory, if unmodified, must
be changed to reference it.
This is an intial pass at converting the btcwallet and deps codebases
to pass a network by their parameters, rather than by a magic number
to identify the network. The parameters in params.go have been
updated to embed a *btcnet.Params, and all previous uses of cfg.Net()
have been replaced with activeNet.{Params,Net} (where activeNet is
the global var for the active network).
Although dependancy packages have not yet been updated from using
btcwire.BitcoinNet to btcnet.Params, the parameters are now accessible
at all callsites, and individual packages can be updated to use btcnet
without requiring updates in each external btc* package at once.
While here, the exported API for btcwallet internal library packages
(txstore and wallet) have been updated to pass full network parameters
rather than the btcwire definition of a network.
This change adds the new btcdusername and btcdpassword options which,
if set, are used instead of the username and password when
authenticating to a btcd RPC server. If these new options are unset,
the btcd user and password settings are shared with the client auth
settings.
The connect option is already used by btcd to force a connection to
other full node peers. Wallet does not talk directly with these
peers, so the connect option is being renamed to something unique for
an RPC client connection.
Boolean options cannot be unset from a default true value on the
command line, so invert the allowfree option, renaming it
disallowfree, so attaching fees may always be forced by specifying
disallowfree = true in the configuration file, or --disallowfree on
the command line.
It may be desirable to never allow free transactions, even if the
calculated priority is high enough that a fee would not be required,
so this change adds a global configuration option to remove this check
and always attach a fee.
If ~/.btcwallet/btcd.cert does not exist and the CA file has not been
explicitly set using the config file or command line flags, it's
possible that the cert can be found in ~/.btcd. If connecting to a
localhost btcd and the previous statements are true, the default CA
file config option is updated for the certificate in them btcd
homedir.
If ~/.btcwallet/btcd.cert does exist and the CA file has not been set,
it is used without checking for a cert in the btcd homedir.
This change enables the --mainnet flag to connect to a mainnet btcd
instance and open and create mainnet wallets. Attempting to connect
to a testnet btcd, or opening a testnet wallet when running in mainnet
mode will result in an error printed to the logging output.
Testnet (version 3) remains the default network for now. btcwallet
will continue to receive fixes and new features, but at the moment we
believe that early adopters familar with using btcwallet on testnet
can safely use it for mainnet now as well.
Reminder: when dealing with mainnet coins, it is always a good idea to
keep backups of your wallet. btcwallet uses a deterministic wallet,
and any later addresses for an account can be recreated with just the
account's original wallet file (this does *not* include imported
addresses). Might we recommend Cyphertite for a backup solution?
This change copies the listening behavior of btcd by replacing the
--serverport option with --listen. By default, btcwallet will only
listen for localhost connections, but with this change it will be
possible to add listeners for remote connections.
This was added due to finding a bug with updateConfigWithActiveParams.
After consulting the btcd source code, the bug was fixed by replacing
the function (as it was no longer needed) when the new listening code
was introduced.
While here, mask out the password flag from being shown in the help
message.
This changes the locations that account files (wallet.bin, utxo.bin,
and tx.bin) are searched for when opening or disk syncing accounts.
Previously, files were saved in the following layout:
~/.btcwallet/
- btcwallet/
- wallet.bin
- tx.bin
- utxo.bin
- btcwallet-AccountA/
- wallet.bin
- tx.bin
- utxo.bin
This format had two issues. First, each account would require its own
directory, causing a scalability issue on unix (and perhaps other)
platforms. Second, there was no distinction between testnet and
mainnet wallets, and if mainnet support was enabled, btcwallet would
attempt to open accounts with testnet wallets.
Instead, the following file structure is now used:
~/.btcwallet/
- testnet/
- wallet.bin
- tx.bin
- utxo.bin
- AccountA-wallet.bin
- AccountA-tx.bin
- AccountA-utxo.bin
This solves both of the previously-mentioned issues by requiring only
two subdirectories (one each for the testnet and mainnet bitcoin
networks), and by separating the locations to open and save testnet
and mainnet account files.
At startup, a check for the old account file structure is performed.
If found, files are moved to the new locations, and the old account
directories are removed. Account files are moved to the testnet
directory, as only testnet support is currently enabled.
The version has been bumped to 0.1.1 to reflect this change.
Fixes#16.
This changes the default CA filename from 'cert.pem' to 'btcd.cert' to
reflect the fact that this cert is used for securely connecting to
btcd. With the introduction of autogenerated btcwallet certs (saved
to rpc.cert and rpc.key) this change was made to better differentiate
the two cert files.
To upgrade across this change, simply move the btcd cert. On unix,
use the command:
$ mv ~/.btcwallet/cert.pem ~/.btcwallet/btcd.cert
This adds an additional config option, -cafile, to specify the root
certificates checked when verifying a btcd TLC connection. btcd will
now automatically generate certs in
~/.btcd/data/{main,test}net/rpc.cert, and this file should be copied
to ~/.btcwallet/cert.pem.
The -btcdport option is also gone now, and replaced with -connect (or
-c), to specify both the hostname/ip and port of the server running
btcd.
This change copies the behavior of btcd for using seelog for logging,
including making timestamps human readable, and setting the default
logging level to info.
Fixes#8.
This copies the functionality of btcd for choosing the ports for the
HTTP client (for btcd) and server (for frontends). On testnet, the
following ports are used as default:
- btcd: 18334
- frontends: 18332
When running with the (currently disabled) --mainnet flag, btcwallet
will choose the following ports by default:
- btcd: 8334
- frontends: 8332
Both ports can be overridden no matter the chosen network using the -b
and -p flags.
