This commit is the result of several big changes being made to the
wallet. In particular, the "handshake" (initial sync to the chain
server) was quite racy and required proper synchronization. To make
fixing this race easier, several other changes were made to the
internal wallet data structures and much of the RPC server ended up
being rewritten.
First, all account support has been removed. The previous Account
struct has been replaced with a Wallet structure, which includes a
keystore for saving keys, and a txstore for storing relevant
transactions. This decision has been made since it is the opinion of
myself and other developers that bitcoind accounts are fundamentally
broken (as accounts implemented by bitcoind support both arbitrary
address groupings as well as moving balances between accounts -- these
are fundamentally incompatible features), and since a BIP0032 keystore
is soon planned to be implemented (at which point, "accounts" can
return as HD extended keys). With the keystore handling the grouping
of related keys, there is no reason have many different Account
structs, and the AccountManager has been removed as well. All RPC
handlers that take an account option will only work with "" (the
default account) or "*" if the RPC allows specifying all accounts.
Second, much of the RPC server has been cleaned up. The global
variables for the RPC server and chain server client have been moved
to part of the rpcServer struct, and the handlers for each RPC method
that are looked up change depending on which components have been set.
Passthrough requests are also no longer handled specially, but when
the chain server is set, a handler to perform the passthrough will be
returned if the method is not otherwise a wallet RPC. The
notification system for websocket clients has also been rewritten so
wallet components can send notifications through channels, rather than
requiring direct access to the RPC server itself, or worse still,
sending directly to a websocket client's send channel. In the future,
this will enable proper registration of notifications, rather than
unsolicited broadcasts to every connected websocket client (see
issue #84).
Finally, and the main reason why much of this cleanup was necessary,
the races during intial sync with the chain server have been fixed.
Previously, when the 'Handshake' was run, a rescan would occur which
would perform modifications to Account data structures as
notifications were received. Synchronization was provided with a
single binary semaphore which serialized all access to wallet and
account data. However, the Handshake itself was not able to run with
this lock (or else notifications would block), and many data races
would occur as both notifications were being handled. If GOMAXPROCS
was ever increased beyond 1, btcwallet would always immediately crash
due to invalid addresses caused by the data races on startup. To fix
this, the single lock for all wallet access has been replaced with
mutexes for both the keystore and txstore. Handling of btcd
notifications and client requests may now occur simultaneously.
GOMAXPROCS has also been set to the number of logical CPUs at the
beginning of main, since with the data races fixed, there's no reason
to prevent the extra parallelism gained by increasing it.
Closes#78.
Closes#101.
Closes#110.
This package is used solely for the storage of private and public
keys, and the addresses they represent. Since "wallet" is an
overloaded term and a working wallet requires transaction history as
well, rename this package and its data structures to more clearly
reflect what it is for.
This change is the result of using the errcheck tool
(https://github.com/kisielk/errcheck) to find all unchecked errors,
both unassigned and those assigned to the blank identifier.
Every returned error is now handled in some manner. These include:
- Logging errors that would otherwise be missed
- Returning errors to the caller for further processing
- Checking error values to determine what to do next
- Panicking for truely exceptional "impossible" errors
On the subject of panics, they are a sharp tool and should be used
sparingly. That being said, I have added them to check errors that
were previously explicitly ignored, because they were expected to
always return without failure. This could be due to fake error paths
(i.e. writing to a bytes.Buffer panics for OOM and should never return
an error) or previous logic asserts that an error case is impossible.
Rather than leaving these unhandled and letting code fail later,
either with incorrect results or a nil pointer dereference, it now
produces a stack trace at the error emit site, which I find far more
useful when debugging.
While here, a bunch of dead code was removed, including code to move
pre-0.1.1 uxto and transaction history account files to the new
directory (as they would be unreadable anyways) and a big chunk of
commented out rpcclient code.
This is an intial pass at converting the btcwallet and deps codebases
to pass a network by their parameters, rather than by a magic number
to identify the network. The parameters in params.go have been
updated to embed a *btcnet.Params, and all previous uses of cfg.Net()
have been replaced with activeNet.{Params,Net} (where activeNet is
the global var for the active network).
Although dependancy packages have not yet been updated from using
btcwire.BitcoinNet to btcnet.Params, the parameters are now accessible
at all callsites, and individual packages can be updated to use btcnet
without requiring updates in each external btc* package at once.
While here, the exported API for btcwallet internal library packages
(txstore and wallet) have been updated to pass full network parameters
rather than the btcwire definition of a network.
Recent btcd versions only allow one rescan to run at any given time
per websocket client. To better handle this, a new set of goroutines
are started by the account manager which batch and serialize rescan
jobs.
If no rescans are currently running, a new rescan starts. If a rescan
is already being processed, the request is queued and runs after the
current rescan finishes. For any additional incoming requests before
the current rescan finishes, the requests are merged with the
currently-waiting request so both can be handled with a single rescan.
This change also prepares for rescan progress notifications from btcd,
but are still unhandled until the necessary details for
partially-synced addresses are added to the wallet file format.
This change replaces the old transaction store file format and
implementation. The most important change is how the full backing
transactions for any received or sent transaction are now saved,
rather than simply saving parsed-out details of the tx (tx shas, block
height/hash, pkScripts, etc.).
To support the change, notifications for received transaction outputs
and txs spending watched outpoints have been updated to use the new
redeemingtx and recvtx notifications as these contain the full tx,
which is deserializead and inserted into the store.
The old transaction store serialization code is completely removed, as
updating to the new format automatically cannot be done. Old wallets
first running past this change will error reading the file and start a
full rescan to rebuild the data. Unlike previous rescan code,
transactions spending outpoint managed by wallet are also included.
This results in recovering not just received history, but history for
sent transactions as well.
The disk syncer now maintains its own countdown timer, creating a new
timer only when necessary (when there is no timer running, and
something is scheduled to be written). When the timer expires, the
select loop begins selecting on a grab of the account manager's binary
semaphore, and if read, performs the sync and nils the select channel
to prevent a future grab until a new timer has expired.
Tested with a race-enabled build on Windows. No lockups or races
related to the disk syncing experienced with constant client requests
and incoming btcd notifications, and scheduled writes run as expected
once the countdown timer expires, locking out all server request and
notifiation handling.
This change removes the three separate mutexes which used to lock an
account's wallet, tx store, and utxo store. Accounts no longer
contain any locking mechanism and rely on go's other synchronization
constructs (goroutines and channels) for correct access.
All accounts are now managed as a collection through the new
AccountManager, rather than the old AccountStore. AccountManager runs
as its own goroutine to provide access to accounts.
RPC requests are now queued for handling, being denied if the queue
buffer is exhausted. Notifications are also queued (instead of being
sent from their own goroutine after being received, in which order is
undefined), however, notifications are never dropped and will
potentially grow a queue of infinite size if unhandled.
Fixes several hangs cased by incorrect locking, by removing the
locking. Instead, a single goroutine manages all file writes.
The old account 'dirty' boolean flags have been removed. Instead,
anytime an account structure is modified, the portion that was
modified (wallet, tx store, or utxo store) must be scheduled to be
written.
This commit fixes two issues in the writeDirtyToDisk function:
First, closing the temporary files is now done using a defer, so they
are always closed.
Second, the various account mutexs are no longer unlocked using a
defer, preventing more than one from being held at once and causing a
deadlock caused by incorrect locking order.
This change allows for the use of watching-only wallets. Unlike
normal, "hot" wallets, watching-only wallets do not contain any
private keys, and can be used in situations where you want to keep one
wallet online to create new receiving addresses and watch for received
transactions, while keeping the hot wallet offline (possibly on an
air-gapped computer).
Two (websocket) extension RPC calls have been added:
First, exportwatchingwallet, which will export the current hot wallet
to a watching-only wallet, saving either to disk or returning the
base64-encoded wallet files to the caller.
Second, recoveraddresses, which is used to recover the next n
addresses from the address chain. This is used to "sync" a watching
wallet with the hot wallet, or vice versa.
This code is based off leveldb (https://github.com/syndtr/goleveldb),
and the leveldb copyright notice (a 2-clause BSD license) has been
included where used.
This changes the locations that account files (wallet.bin, utxo.bin,
and tx.bin) are searched for when opening or disk syncing accounts.
Previously, files were saved in the following layout:
~/.btcwallet/
- btcwallet/
- wallet.bin
- tx.bin
- utxo.bin
- btcwallet-AccountA/
- wallet.bin
- tx.bin
- utxo.bin
This format had two issues. First, each account would require its own
directory, causing a scalability issue on unix (and perhaps other)
platforms. Second, there was no distinction between testnet and
mainnet wallets, and if mainnet support was enabled, btcwallet would
attempt to open accounts with testnet wallets.
Instead, the following file structure is now used:
~/.btcwallet/
- testnet/
- wallet.bin
- tx.bin
- utxo.bin
- AccountA-wallet.bin
- AccountA-tx.bin
- AccountA-utxo.bin
This solves both of the previously-mentioned issues by requiring only
two subdirectories (one each for the testnet and mainnet bitcoin
networks), and by separating the locations to open and save testnet
and mainnet account files.
At startup, a check for the old account file structure is performed.
If found, files are moved to the new locations, and the old account
directories are removed. Account files are moved to the testnet
directory, as only testnet support is currently enabled.
The version has been bumped to 0.1.1 to reflect this change.
Fixes#16.
This change better organizes account handling by creating a new
AccountStore type and accountstore global variable, with receiver
funcs for all operations that require all accounts. More Account
funcs are also added to clean up account handling in the RPC code.
Intial work on this done by dhill.
This change modifies the order in which transaction to watched
addresses are processed and when frontend notifications occur. Due to
btcd notifying all transactions before sending the blockconnected
notification, the UTXO and transaction stores can be modified without
sending any frontend notifications, and then a single frontend
notification is sent when the blockconnected notification arrives.
The order in which each file is synced to disk was also changed to
write out the UTXO and transaction stores before writing the wallet.
This is to prevent a race where wallet closes after writing the dirty
wallet, but before the dirty UTXO store is written. In this
situation, newly added UTXOs will be missed and not found again on the
next wallet open during the rescan. Writing the wallet (which holds
the synced-to-block information) last prevents this.
An issue where the unconfirmed change UTXO created from a new
transaction never being properly notified to frontends is fixed now as
well.
When a wallet is opened, a rescan request will be sent to btcd with
all active addresses from the wallet, to rescan from the last synced
block (now saved to the wallet file) and the current best block.
As multi-account support is further explored, rescan requests should
be batched together to send a single request for all addresses from
all wallets.
This change introduces several changes to the wallet, tx, and utxo
files. Wallet files are still compatible, however, a rescan will try
to start at the genesis block since no correct "last synced to" or
"created at block X" was saved. The tx and utxo files, however, are
not compatible and should be deleted (or an error will occur on read).
If any errors occur opening the utxo file, a rescan will start
beginning at the creation block saved in the wallet.
While fixing this code, the dirty flag was also cleared so that
unneeded syncs wouldn't be needed later. The dirty flag set and sync
was also added for the 'getnewaddress' handler, as it was previously
missing.
This runs a syncer once every minute to write any dirty wallet data
structures out to disk. As currently implemented, dirty wallets will
be lost if not written before btcwallet closes or crashes.
Deterministic wallet help migitate this issue (as private keys can be
created again as long as a previous wallet file was written) but this
can still be a nuisance as a longer rescan will be required to catch
up to chain.
