As calls os.Exit do not run deferred functions (such as log flushing),
the real main function should simply run a main helper function that,
rather than exiting the program, runs all defers and returns a
possibly non-nil error. The real main function can then check the
error and close the program with an error exit status when a fatal
error occured.
The info log level (default) will produce output about confirmed and
unconfirmed transactions being inserted into the store, as well as
unconfirmed transactions which have been mined into blocks. By
enabling the debug log level (-d TXST=debug), additional information
about transaction inputs and outputs is logged. This includes the
total amount of previously-unspent outputs which have been marked
spent by the inserted transaction, and the output indexes and amounts
for each spendable output. Additionally, the debug log level will log
whenever transactions are removed due to being a double spend of
another inserted transaction.
This commit is the result of inspecting the results of both cpu and
memory profiling, to improve areas where wallet can be more efficient
on transaction inserts.
One problem that's very evident by profiling is how much waiting there
is for file (txstore, wallet) writes. This commit does not attempt to
fix this yet, but focuses on the easier-to-fix memory allocation
issues which can slow down the rest of wallet due to excessive garbage
collection scanning.
While here, fix a race where a closure run as a goroutine was closing
over a range iterator.
This was only necessary for a very old version of the transaction
store. The current implementation stores both sent (debit) and
received (credit) records for individual transactions.
This change rewrites much of the error handling for the RPC server
components to match a more idiomatic Go error handling style as well as
fix several issues regarding error equality checks.
Closes#94.
This change is the result of using the errcheck tool
(https://github.com/kisielk/errcheck) to find all unchecked errors,
both unassigned and those assigned to the blank identifier.
Every returned error is now handled in some manner. These include:
- Logging errors that would otherwise be missed
- Returning errors to the caller for further processing
- Checking error values to determine what to do next
- Panicking for truely exceptional "impossible" errors
On the subject of panics, they are a sharp tool and should be used
sparingly. That being said, I have added them to check errors that
were previously explicitly ignored, because they were expected to
always return without failure. This could be due to fake error paths
(i.e. writing to a bytes.Buffer panics for OOM and should never return
an error) or previous logic asserts that an error case is impossible.
Rather than leaving these unhandled and letting code fail later,
either with incorrect results or a nil pointer dereference, it now
produces a stack trace at the error emit site, which I find far more
useful when debugging.
While here, a bunch of dead code was removed, including code to move
pre-0.1.1 uxto and transaction history account files to the new
directory (as they would be unreadable anyways) and a big chunk of
commented out rpcclient code.
- Move the MarkAddresForAccount and LookupAccountByAddress functionality
into account maanger.
- Move the wallet opeing logic into account manager (the only place that calls
it) and unexport.
- Move accountHandler to using a single channel for commands. Many of
the commands have ordering restraints (add account, list all accounts,
remove account, access account, mark account for address) which are very
much undefined with the multi-channel model.
- Rework all callers of LookupAccountByAddress to get the account structure
directly.
Recent btcd versions only allow one rescan to run at any given time
per websocket client. To better handle this, a new set of goroutines
are started by the account manager which batch and serialize rescan
jobs.
If no rescans are currently running, a new rescan starts. If a rescan
is already being processed, the request is queued and runs after the
current rescan finishes. For any additional incoming requests before
the current rescan finishes, the requests are merged with the
currently-waiting request so both can be handled with a single rescan.
This change also prepares for rescan progress notifications from btcd,
but are still unhandled until the necessary details for
partially-synced addresses are added to the wallet file format.
Notifications ariving from btcd were being reordered (each handled by
its own goroutine, rather then being always sent in the order they
originated). This was breaking the new transaction store by inserting
transaction records in an 'impossible' manner, that is, inserting txs
without block info after the store already held records of the same tx
with block info, without first performing a rollback.
This is handled by the transaction store insert methods by checking
for identical transactions (double spends with the same tx sha), but
where the block heights mismatch and the new record does not have a
block set. The error is returned all the way up to the goroutine
running each rpc request/notification handler, and if hit, the btcd
connection is closed and all accounts are reopened from disk. This is
not optimal, but it allows us to use the connect logic to correctly
catch us up to the best chain with the last good state of all accounts
while only rescanning a few blocks.
Fixes#72.
This change replaces the old transaction store file format and
implementation. The most important change is how the full backing
transactions for any received or sent transaction are now saved,
rather than simply saving parsed-out details of the tx (tx shas, block
height/hash, pkScripts, etc.).
To support the change, notifications for received transaction outputs
and txs spending watched outpoints have been updated to use the new
redeemingtx and recvtx notifications as these contain the full tx,
which is deserializead and inserted into the store.
The old transaction store serialization code is completely removed, as
updating to the new format automatically cannot be done. Old wallets
first running past this change will error reading the file and start a
full rescan to rebuild the data. Unlike previous rescan code,
transactions spending outpoint managed by wallet are also included.
This results in recovering not just received history, but history for
sent transactions as well.
This change removes the three separate mutexes which used to lock an
account's wallet, tx store, and utxo store. Accounts no longer
contain any locking mechanism and rely on go's other synchronization
constructs (goroutines and channels) for correct access.
All accounts are now managed as a collection through the new
AccountManager, rather than the old AccountStore. AccountManager runs
as its own goroutine to provide access to accounts.
RPC requests are now queued for handling, being denied if the queue
buffer is exhausted. Notifications are also queued (instead of being
sent from their own goroutine after being received, in which order is
undefined), however, notifications are never dropped and will
potentially grow a queue of infinite size if unhandled.
Fixes several hangs cased by incorrect locking, by removing the
locking. Instead, a single goroutine manages all file writes.
The old account 'dirty' boolean flags have been removed. Instead,
anytime an account structure is modified, the portion that was
modified (wallet, tx store, or utxo store) must be scheduled to be
written.
This change checks for all wallet files in the account directory, and
then attempts to open each of the available accounts for those
wallets. Before, only the default account would ever be opened.
This change greatly cleans up the RPC connection between btcwallet and
btcd. Proper (JSON-RPC spec-following) notifications are now expected
rather than Responses with a non-empty IDs.
A new RPCConn interface type has also been introduced with a
BtcdRPCConn concrete type for btcd RPC connections. Non-btcd-specific
code handles the RPCConn, while the btcd details have been abstracted
away to a handful of functions. This will make it easier to write
tests by creating a new fake RPC connection with hardcoded expected
replies.
This change copies the listening behavior of btcd by replacing the
--serverport option with --listen. By default, btcwallet will only
listen for localhost connections, but with this change it will be
possible to add listeners for remote connections.
This was added due to finding a bug with updateConfigWithActiveParams.
After consulting the btcd source code, the bug was fixed by replacing
the function (as it was no longer needed) when the new listening code
was introduced.
While here, mask out the password flag from being shown in the help
message.
This changes the locations that account files (wallet.bin, utxo.bin,
and tx.bin) are searched for when opening or disk syncing accounts.
Previously, files were saved in the following layout:
~/.btcwallet/
- btcwallet/
- wallet.bin
- tx.bin
- utxo.bin
- btcwallet-AccountA/
- wallet.bin
- tx.bin
- utxo.bin
This format had two issues. First, each account would require its own
directory, causing a scalability issue on unix (and perhaps other)
platforms. Second, there was no distinction between testnet and
mainnet wallets, and if mainnet support was enabled, btcwallet would
attempt to open accounts with testnet wallets.
Instead, the following file structure is now used:
~/.btcwallet/
- testnet/
- wallet.bin
- tx.bin
- utxo.bin
- AccountA-wallet.bin
- AccountA-tx.bin
- AccountA-utxo.bin
This solves both of the previously-mentioned issues by requiring only
two subdirectories (one each for the testnet and mainnet bitcoin
networks), and by separating the locations to open and save testnet
and mainnet account files.
At startup, a check for the old account file structure is performed.
If found, files are moved to the new locations, and the old account
directories are removed. Account files are moved to the testnet
directory, as only testnet support is currently enabled.
The version has been bumped to 0.1.1 to reflect this change.
Fixes#16.
This change adds an additional check when creating a new wallet or
extending the keypool. All public and private keypairs are parsed
from their serialized forms, and an ecdsa signature is created and
verified using the keypairs. If the verifiction fails at any point,
the wallet creation or keypool extension is aborted to prevent any
errors where an address is returned to a user, but any funds send to
that address are unspendable due to a mismatched keypair.
This change better organizes account handling by creating a new
AccountStore type and accountstore global variable, with receiver
funcs for all operations that require all accounts. More Account
funcs are also added to clean up account handling in the RPC code.
Intial work on this done by dhill.
This change adds support for the listtransactions RPC command. To
properly reply to this command, additonal information about received
transactions was added, and is now saved in an account's tx.bin file.
Additionally, when sending a transaction, a *tx.SendTx is now saved to
the Tx store, and is included in listtransactions replies under the
"send" category.
WARNING: All account's tx.bin and utxo.bin files should be removed
before running with this change, or else the files may not be read
correctly. Removing tx.bin is not an issue as it was not being used
before, and was being saved with incorrect data. Removing utxo.bin is
not an issue as it will just trigger a rescan on next start. File
format versions are now included in both files, so automatic updates
from previous file formats will be possible with future changes.
Fixes#12.
This adds an additional config option, -cafile, to specify the root
certificates checked when verifying a btcd TLC connection. btcd will
now automatically generate certs in
~/.btcd/data/{main,test}net/rpc.cert, and this file should be copied
to ~/.btcwallet/cert.pem.
The -btcdport option is also gone now, and replaced with -connect (or
-c), to specify both the hostname/ip and port of the server running
btcd.
With the exception of the createencryptedwallet extension (which is
required to make a wallet), all websocket-specific handlers are now
only available from a websocket connection, and standard RPC requests
are handled with a normal HTTP request and reply.
As an added bonus, listening on IPv6 now works.
This change modifies the order in which transaction to watched
addresses are processed and when frontend notifications occur. Due to
btcd notifying all transactions before sending the blockconnected
notification, the UTXO and transaction stores can be modified without
sending any frontend notifications, and then a single frontend
notification is sent when the blockconnected notification arrives.
The order in which each file is synced to disk was also changed to
write out the UTXO and transaction stores before writing the wallet.
This is to prevent a race where wallet closes after writing the dirty
wallet, but before the dirty UTXO store is written. In this
situation, newly added UTXOs will be missed and not found again on the
next wallet open during the rescan. Writing the wallet (which holds
the synced-to-block information) last prevents this.
An issue where the unconfirmed change UTXO created from a new
transaction never being properly notified to frontends is fixed now as
well.
This change moves the handlers to a map (instead of falling through a
switch statement), and updates each handler to use a btcjson.Cmd
instead of passing parameters in a btcjson.Message manually.
Plenty of comments were also added, which should also make the code
much more understandable.
This removes the enforced check for the spent field for tx-to-me
notifications, as this is no longer sent, and should be calculated by
wallet (not done yet). Additionally, the full CreatedTx information
is saved with the unmined tx map, so when a tx is mined, information
about which inputs and ouputs it creates that are relevant to the
wallet can be used.
When a wallet is opened, a rescan request will be sent to btcd with
all active addresses from the wallet, to rescan from the last synced
block (now saved to the wallet file) and the current best block.
As multi-account support is further explored, rescan requests should
be batched together to send a single request for all addresses from
all wallets.
This change introduces several changes to the wallet, tx, and utxo
files. Wallet files are still compatible, however, a rescan will try
to start at the genesis block since no correct "last synced to" or
"created at block X" was saved. The tx and utxo files, however, are
not compatible and should be deleted (or an error will occur on read).
If any errors occur opening the utxo file, a rescan will start
beginning at the creation block saved in the wallet.
This change copies the behavior of btcd for using seelog for logging,
including making timestamps human readable, and setting the default
logging level to info.
Fixes#8.
This change removes a lot of unnecessary and complicated locking (if
serializing requests is needed in the future, a goroutine will be used
instead) and also shifts the heavy lifting from frontends to btcwallet
itself to handle any notifications when they can be properly handled.
Although it's still legal to, frontends no longer need to explicitly
request account balances as these are calculated and sent as an async
notification on frontend connect, and these notifications will only
occur if btcd is currently connected. Likewise, when btcd connects,
all frontends are immediately notified of all notifications that
require btcd information, such as the current block height for
calculating account balances.
CalculateBalance now works correctly: if confirmations is 0, all UTXOs
will be used for the balance. Otherwise, unconfirmed UTXOs will be
exclused. 1 confirmation will allow the UTXO height and current block
height to be equal. Even though the difference is zero, the
transaction including the UTXO has been mined into one block.
This change also remove extraneous account balance notifications for
connected and disconnected blocks.
This change fixes many issues with the tracking of unspent transaction
outputs. First, notifications for when UTXOs arse spent are now
requested from btcd, and when spent, will be removed from the
UtxoStore.
Second, when transactions are created, the unconfirmed (not yet in a
block) Utxo (with block height -1 and zeroed block hash) is added to
the wallet's UtxoStore. Notifications for when this UTXO is spent are
also requested from btcd. After the tx appears in a block, because
the UTXO has a pkScript to be spent by another owned wallet address, a
notification with the UTXO will be sent to btcwallet. We already
store the unconfirmed UTXO, so at this point the actual block height
and hash are filled in.
Finally, when calculating the balance, if confirmations is zero,
unconfirmed UTXOs (block height -1) will be included in the balance.
Otherwise, they are ignored.
The notification ID for new transactions to a watched address comment
is unneeded, as OpenWallet does not attempt to track wallets against a
connected btcd instance. Added an additional comment to the function
noting this.
A log.Debugf was also added so tracked addresses are shown in the
debug output.
