The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
The websocket extension command to register for notifications when an
address receives funds has been renamed. This commit catches up to the
change.
ok @jrick
This commit modifies all code paths which work with transaction result
objects to use the concrete ListTransactionsResult provided by the btcjson
package. This provides nicer marshalling and unmarshalling as well as
access to properly typed fields.
- Instead of returning a special constructed type whenever queries for an
address. Return the internal object with an immutable external
interface.
- Make the private key gettable from PubKeyAddress to prevent having to look up
multiple times to get information from the same structure
- Enforce addresses always have public keys.
- Move the MarkAddresForAccount and LookupAccountByAddress functionality
into account maanger.
- Move the wallet opeing logic into account manager (the only place that calls
it) and unexport.
- Move accountHandler to using a single channel for commands. Many of
the commands have ordering restraints (add account, list all accounts,
remove account, access account, mark account for address) which are very
much undefined with the multi-channel model.
- Rework all callers of LookupAccountByAddress to get the account structure
directly.
This change fixes the reply for listunspent to return a JSON object in
the same format as done by the reference implementation. Previously,
listunspent would return an array of the same objects as returned for
listtransactions.
Recent btcd versions only allow one rescan to run at any given time
per websocket client. To better handle this, a new set of goroutines
are started by the account manager which batch and serialize rescan
jobs.
If no rescans are currently running, a new rescan starts. If a rescan
is already being processed, the request is queued and runs after the
current rescan finishes. For any additional incoming requests before
the current rescan finishes, the requests are merged with the
currently-waiting request so both can be handled with a single rescan.
This change also prepares for rescan progress notifications from btcd,
but are still unhandled until the necessary details for
partially-synced addresses are added to the wallet file format.
Calling the Bytes method for a big.Int does not pad the result to
required size for EncodePrivateKey. This change adds the leading
padding, preventing seemingly-random "malformed private key" errors
from being returned to users of dumpprivkey.
The private key import codepath (called when handling the
importprivkey RPC method) was not triggering rescans for the imported
address. This change begins a new rescan for each import and adds
additional logic to the wallet file to keep track of unsynced imported
addresses. After a rescan on an imported address completes, the
address is marked as in sync with the rest of wallet and future
handshake rescans will start from the last seen block, rather than the
import height of the unsynced address.
While here, improve the logging for not just import rescans, but
rescanning on btcd connect (part of the handshake) as well.
Fixes#74.
Shortly we will add new types of address, so make AddressInfo an
interface, with concrete types providing address-specific information.
Adapt existing code to this new status quo.
This change replaces the old transaction store file format and
implementation. The most important change is how the full backing
transactions for any received or sent transaction are now saved,
rather than simply saving parsed-out details of the tx (tx shas, block
height/hash, pkScripts, etc.).
To support the change, notifications for received transaction outputs
and txs spending watched outpoints have been updated to use the new
redeemingtx and recvtx notifications as these contain the full tx,
which is deserializead and inserted into the store.
The old transaction store serialization code is completely removed, as
updating to the new format automatically cannot be done. Old wallets
first running past this change will error reading the file and start a
full rescan to rebuild the data. Unlike previous rescan code,
transactions spending outpoint managed by wallet are also included.
This results in recovering not just received history, but history for
sent transactions as well.
This change removes the three separate mutexes which used to lock an
account's wallet, tx store, and utxo store. Accounts no longer
contain any locking mechanism and rely on go's other synchronization
constructs (goroutines and channels) for correct access.
All accounts are now managed as a collection through the new
AccountManager, rather than the old AccountStore. AccountManager runs
as its own goroutine to provide access to accounts.
RPC requests are now queued for handling, being denied if the queue
buffer is exhausted. Notifications are also queued (instead of being
sent from their own goroutine after being received, in which order is
undefined), however, notifications are never dropped and will
potentially grow a queue of infinite size if unhandled.
Fixes several hangs cased by incorrect locking, by removing the
locking. Instead, a single goroutine manages all file writes.
The old account 'dirty' boolean flags have been removed. Instead,
anytime an account structure is modified, the portion that was
modified (wallet, tx store, or utxo store) must be scheduled to be
written.
Now that it has been decided that all account wallets will share the
same passphrase, the walletlock and walletpassphrase RPC handlers now
go through the accountstore to lock or unlock all account wallets,
rather than only changing the default account.
There were several places where various account files (wallet, tx, or
utxo stores) were being marked as dirty, and then not being either
immediately synced to disk or marked as a dirty account so they would
be scheduled to be synced to disk. This change adds Account functions
to mark as dirty and add the account to the map of scheduled accounts
so they won't be missed by the disk syncer goroutine.
This change allows for the use of watching-only wallets. Unlike
normal, "hot" wallets, watching-only wallets do not contain any
private keys, and can be used in situations where you want to keep one
wallet online to create new receiving addresses and watch for received
transactions, while keeping the hot wallet offline (possibly on an
air-gapped computer).
Two (websocket) extension RPC calls have been added:
First, exportwatchingwallet, which will export the current hot wallet
to a watching-only wallet, saving either to disk or returning the
base64-encoded wallet files to the caller.
Second, recoveraddresses, which is used to recover the next n
addresses from the address chain. This is used to "sync" a watching
wallet with the hot wallet, or vice versa.
This change introduces a new function to export a wallet in memory to
a watching wallet. Watching wallets allow to watch for balance
changes and transactions to wallet addresses while only storing the
public parts of a wallet (no private keys). New addresses created by
the watching wallet will use pubkey address chaining and will allow to
receive funds to an indefinite number of new addresses, and create the
private keys for said addresses from the non-watching wallet later.
The actual exporting of a watching wallet to a file (triggered by an
RPC request) is not yet implemented.
While here, fix an issue found by new test code for the chained
address code which incorrectly set the starting index of addresses in
the chain needing private keys to be created.
When disk syncing a wallet file, if the wallet is flagged dirty, the
disk syncer must grab the wallet writer lock to set dirty=false. The
disk syncing code was being called in the end of
(*Account).RescanActiveAddresses with the reader lock held (unlocked
using a defer), which prevented the writer lock from being aquired.
This change removes the defered unlock to release the reader lock
before syncing to disk.
This change greatly cleans up the RPC connection between btcwallet and
btcd. Proper (JSON-RPC spec-following) notifications are now expected
rather than Responses with a non-empty IDs.
A new RPCConn interface type has also been introduced with a
BtcdRPCConn concrete type for btcd RPC connections. Non-btcd-specific
code handles the RPCConn, while the btcd details have been abstracted
away to a handful of functions. This will make it easier to write
tests by creating a new fake RPC connection with hardcoded expected
replies.
