In this commit, we add a sanity check for the wallet's birthday block
before syncing as a result of the migration that populated it for
existing wallets. This is done as the second part to the migration to
ensure we do not miss any relevant events throughout rescans.
The sanity check performs two main checks: whether the birthday block
timestamp reflects a time before the birthday timestamp and whether the
delta between these two timestamps is a reasonable amount. The birthday
block is then found as the first candidate that satisfies both of these
conditions.
ImportPrivateKey
In this commit, we ensure that when an external private key is imported
into the wallet, that we do not overwrite our existing birthday with the
one provided. If this were to happen and we forced a wallet rescan using
the birthday as our starting point, then we'd miss detecting relevant
on-chain events that occurred between them.
In this commit, we modify the wallet to use the new migration logic
provided by the recently introduced migration package. Additionally,
we'll also perform all of our upgrades within the same database
transaction to guarantee fault-tolerance of the wallet.
In this commit, we relax the initial sync detection logic a bit. We do
this as right now, if a user creates an address during the sync point,
if they restart, then we'll fall back to performing a rescan from that
height as we'll detect that we aren't performing the initial sync, so
won't pick up the birthday timestamp.
To fix this, we now declare that if we have no UTXO's, then we're still
performing the initial sync. This solves this issue as when the user
restarts, we'll continue to wait for the backend to sync, and pick up
the proper birthday height before we attempt to scan forward for the
rescan. However, the one tradeoff is that we'll now always start the
rescan from the birthday height until the wallet has gained it's first
UTXO. I don't think this is too bad, as after all, the point of a wallet
is to manage utxos.
In this commit, we refactor the logic outside of PublishTransaction into
another unexported method. This will pave the road for unifying the
logic between SendOutputs and PublishTransaction.
In this commit, we simplify the logic when broadcasting transactions to
the greater network. Rather than special casing when running with a
Neutrino backend, we'll always add the transaction to the store as
relevant when attempting to broadcast it. This will properly insert it
into the store and update unconfirmed balances. In the event that the
transaction failed to broadcast, it can be removed from the store with
no side-effects, essentially acting as if the transaction was never
added to the store in the first place.
In this commit, we modify the SendOutputs method to also notify new
outgoing transctions for neutriino. For the full node backends, they'll
get this notification when the transactino hits the mempool. However,
for neutrino it will only be notified once the transaction has been
confirmed. This commit ensures that we'll notify on send as well.
In this commit, we avoid notifying clients of transactions that we've
received chain.RelevantTx notifications for, but are not found within
the wallet. This can happen as now we'll prevent adding an unconfirmed
transaction to the wallet that already exists as confirmed. Due to this,
UniqueTxDetails will be unable to find the transaction and return nil,
casuing a panic for potential callers.
This PR moves any address notifications outside of the
db transaction that creates them. This is known to have
resulted in deadlocks, since chainClient.NotifyReceived
could block the db transaction from committing.
Doing so also prevents the situation where we send
notifications about the new addresses, but the db txn
fails to commit and the addresses are in fact never
created.
This commit adds rescanWithTarget, in order to facilitate
rescans beginning a certain height. This is done as a
precursor to fixing a bug in the initial sync, that would
cause us to miss relevant txns if they are confirmed before
starting the initial rescan.
In this commit, we alter the behavior for handling chain notifications
within the wallet. The previous code would assume that the channel would
close, but due to now using a ConcurrentQueue to handle notifications,
this assumption no longer stands. Now, we'll stop handling notifications
either once the wallet has or stopped or once the notifications channel
has been closed.
In this commit, ensure that upon restart, if any of the full-node based
backends we support reject the transaction, then we'll properly remove
the now invalid transaction from the tx store. Before this commit, we
could miss a few errors from bitcoind. To remedy this, we explicitly
catch those errors, but then also attempt to precisely catch the set of
generic json RPC errors that can be returned.
In this commit, we fix a bug introduced in an earlier commit. Before
this commit, we would *always* remove an unmined transaction if it
failed to be accepted by the network upon restart. Instead, we should
only remove transaction that are actually due to us trying to spend an
output that’s already spent, or an orphan transaction.
In this commit, we extend the PublishTransction method to be a more
general semi reliable transaction broadcast mechanism. We do this by
removing the special casing for neutrino. With this change, we’ll
_always_ write any transactions to be broadcast to disk. A side effect
of this, is that if the transaction doesn’t *directly* involve any
outputs we control, then it’ll linger around until a restart, when we
try to rebroadcast, and observe that it has bene rejected.
This commit makes use of the recently added EstimateVirtualSize
method to estimated the size of a transaction when calculating
fees. This makes fee estimation more accurate when we are spending
segwit outputs, as before we wouldn't account for the witness
descount, resulting in overshooting fee estimates.
This commit adds a new method EstimateVirtualSize that calculates
the worst case estimate vsize for a transaction with a given set
of inputs and outputs. This method is aware of P2PKH, P2WPKH and
P2SH-P2WPKH inputs, and caulculates the transaction vsize with
the witness data included.
In this commit, we do away with the internal relayFee all together.
Instead, we’ll pass in the fee rate when we’re crafting any
transactions. This allows the caller to manually dictate their desired
fee rate.
This commit makes sure the wallet db is closed if the call to
open the wallet fails, as subsequent calls to OpenExistingWallet
would fail to open the already open database.
During the time of initial block hash catch-up, it is possible to
request an address be generated. This commit updates the active
addresses by calling `w.activeData` after the catch-up is complete.
This changes the database access APIs and each of the "manager"
packages (waddrmgr/wstakemgr) so that transactions are opened (only)
by the wallet package and the namespace buckets that each manager
expects to operate on are passed in as parameters.
This helps improve the atomicity situation as it means that many
calls to these APIs can be grouped together into a single
database transaction.
This change does not attempt to completely fix the "half-processed"
block problem. Mined transactions are still added to the wallet
database under their own database transaction as this is how they are
notified by the consensus JSON-RPC server (as loose transactions,
without the rest of the block that contains them). It will make
updating to a fixed notification model significantly easier, as the
same "manager" APIs can still be used, but grouped into a single
atomic transaction.
Remove the addresses field from TransactionDetails.Output. It is
assumed that the caller is able to deserialize the transaction and
encode the output scripts to addresses, so this is unnecessary server
overhead and conflicts with the current API philosophy of not
duplicating data already included in another field.
Since there is no additional data included for outputs not controlled
by the wallet, remove the `mine` specifier from the Output message and
replace it with an output index. Only include messages for controlled
outputs, rather than creating messages for both controlled and
uncontrolled outputs. Rename the repeated field from `outputs` to
`credits` to be consistent with the `debits` field.
Bump major API version as this is a breaking change.
Closes#408.
This commit enabled the wallet to properly spend nested and normal
p2wkh outputs under its control.
For regular p2wkh outputs, spending simply involves presenting the
original pub key, and signature as the witness data.
For nested p2wkh outputs, in addition to the above, the version zero
witness p2wkh witness program is placed in the sigScript in order to
allow clients who are aware of BIP 16 to validate the witness spend.
When spending a segwit output, the wallet also needs the input value of
the previous output script. Therefore when selecting outputs the input
value is now returned. Additionally when validating newly signed
outputs the input value as also passed into `txscript.Engine`
Previously, this would not increment the spendable balance for matured
coinbase outputs and would only increment the immature balance if the
output was still immature.
This updates both btcsuite and external dependencies to their latest
versions. In particular, gRPC was updated to version 1.0.3 and bolt
to 1.3.0.
The walletrpc package needed to be regenerated for the gRPC update.
While here, update the Travis-CI script so this can be tested there.
Since the coinbase maturity is now allowed to be defined per chain and
the old blockchain.CoinbaseMaturity constant has been removed, this
updates the code accordingly.
Also, update glide.lock to use the required version of btcd.
This updates all code to make use of the new chainhash package since the
old wire.ShaHash type and related functions have been removed in favor
of the abstracted package.
Also, while here, rename all variables that included sha in their name
to include hash instead.
Finally, update glide.lock to use the required version of btcd, btcutil,
and btcrpcclient.
Remove the addresses field from TransactionDetails.Output. It is
assumed that the caller is able to deserialize the transaction and
encode the output scripts to addresses, so this is unnecessary server
overhead and conflicts with the current API philosophy of not
duplicating data already included in another field.
Since there is no additional data included for outputs not controlled
by the wallet, remove the `mine` specifier from the Output message and
replace it with an output index. Only include messages for controlled
outputs, rather than creating messages for both controlled and
uncontrolled outputs. Rename the repeated field from `outputs` to
`credits` to be consistent with the `debits` field.
Bump major API version as this is a breaking change.
Closes#408.
Due to the way dust is calculated, if the transaction relay fee is
zero, then a zero output amount is not considered dust. As the
transaction authoring code used this dust check to determine whether a
change output can be included or not, it could create unnecessary
change outputs which return no value back to the wallet. Prevent this
by including an explicit check for zero values.
This commit corrects various things found by the static checkers
(comments, unkeyed fields, return after some if/else).
Add generated files and legacy files to the whitelist to be ignored.
Catch .travis.yml up with btcd so goclean can be run.
This changes the wallet.Open function signature to remove the database
namespace parameters. This is done so that the wallet package itself
is responsible for the location and opening of these namespaces from
the database, rather than requiring the caller to open these ahead of
time.
A new wallet.Create function has also been added. This function
initializes a new wallet in an empty database, using the same
namespaces as wallet.Open will eventually use. This relieves the
caller from needing to manage wallet database namespaces explicitly.
Fixes#397.
These notifications were added to support real time updates for
btcgui. As the btcgui project is no longer being developed, there are
no more consumers of this API, and it makes sense to remove them given
their various issues (the largest being that notifiations are sent
unsubscribed to clients that may never be interrested in them).
A new notification server has already been added to the wallet package
to handle notifications in a RPC-server agnostic way. This server is
the means by which the wallet notifies changes for gRPC clients. If
per-client registered notifications are to be re-added for the
JSON-RPC server, they should be integrated with the new notification
server rather than using this legacy code.
This corrects and simplifies the shutdown logic for interrupts, the
walletrpc.WalletLoaderService/CloseWallet RPC, and the legacy stop RPC
by both stopping all wallet processes and closing the wallet database.
It appears that this behavior broke as part of the wallet package
refactor, causing occasional nil pointer panics and memory faults when
closing the wallet database with active transactions.
Fixes#282.
Fixes#283.
This began as a change to improve the fee calculation code and evolved
into a much larger refactor which improves the readability and
modularity of all of the transaction creation code.
Transaction fee calculations have been switched from full increments
of the relay fee to a proportion based on the transaction size. This
means that for a relay fee of 1e3 satoshis/kB, a 500 byte transaction
is only required to pay a 5e2 satoshi fee and a 1500 byte transaction
only need pay a 1.5e3 fee. The previous code would end up estimating
these fees to be 1e3 and 2e3 respectively.
Because the previous code would add more fee than needed in almost
every case, the transaction size estimations were optimistic
(best/smallest case) and signing was done in a loop where the fee was
incremented by the relay fee again each time the actual size of the
signed transaction rendered the fee too low. This has switched to
using worst case transaction size estimates rather than best case, and
signing is only performed once.
Transaction input signature creation has switched from using
txscript.SignatureScript to txscript.SignTxOutput. The new API is
able to redeem outputs other than just P2PKH, so the previous
restrictions about P2SH outputs being unspendable (except through the
signrawtransaction RPC) no longer hold.
Several new public packages have been added:
wallet/txauthor - transaction authoring and signing
wallet/txfees - fee estimations and change output inclusion
wallet/txrules - simple consensus and mempool policy rule checks
Along with some internal packages:
wallet/internal/txsizes - transaction size estimation
internal/helpers - context free convenience functions
The txsizes package is internal as the estimations it provides are
specific for the algorithms used by these new packages.
Previously, when creating a change address during the process of
creating a new transaction an error case would be hit in the waddrmgr
triggered by attempting to derive a new internal address from under a
waddrmgr.ImportedAddrAccount. To remedy this error, we now use the
default account for change when spending outputs from an imported
key. This approach allows funds under the control of imported
private keys to be protected under the wallet's seed as soon as
they've been partially spent.
Previously, if a nil seed was passed into loader.CreateNewWallet, a
random seed was never generated. This would cause an error within the
waddrmgr due to the seed being of invalid (0) length.
This is a rather monolithic commit that moves the old RPC server to
its own package (rpc/legacyrpc), introduces a new RPC server using
gRPC (rpc/rpcserver), and provides the ability to defer wallet loading
until request at a later time by an RPC (--noinitialload).
The legacy RPC server remains the default for now while the new gRPC
server is not enabled by default. Enabling the new server requires
setting a listen address (--experimenalrpclisten). This experimental
flag is used to effectively feature gate the server until it is ready
to use as a default. Both RPC servers can be run at the same time,
but require binding to different listen addresses.
In theory, with the legacy RPC server now living in its own package it
should become much easier to unit test the handlers. This will be
useful for any future changes to the package, as compatibility with
Core's wallet is still desired.
Type safety has also been improved in the legacy RPC server. Multiple
handler types are now used for methods that do and do not require the
RPC client as a dependency. This can statically help prevent nil
pointer dereferences, and was very useful for catching bugs during
refactoring.
To synchronize the wallet loading process between the main package
(the default) and through the gRPC WalletLoader service (with the
--noinitialload option), as well as increasing the loose coupling of
packages, a new wallet.Loader type has been added. All creating and
loading of existing wallets is done through a single Loader instance,
and callbacks can be attached to the instance to run after the wallet
has been opened. This is how the legacy RPC server is associated with
a loaded wallet, even after the wallet is loaded by a gRPC method in a
completely unrelated package.
Documentation for the new RPC server has been added to the
rpc/documentation directory. The documentation includes a
specification for the new RPC API, addresses how to make changes to
the server implementation, and provides short example clients in
several different languages.
Some of the new RPC methods are not implementated exactly as described
by the specification. These are considered bugs with the
implementation, not the spec. Known bugs are commented as such.
This change introduces additional network activity with the btcd
process to ensure that the network connection is not silently dropped.
Previously, if the connection was lost (e.g. wallet runs on a laptop
and connects to remote btcd, and the laptop is suspended/resumed) the
lost connection would not be detectable since all normal RPC activity
(excluding requests from btcwallet to btcd made by the user) is in the
direction of btcd to wallet in the form of websocket notifications.
sync.Locker cannot be safely used to switch a sync.Mutex to a noop
locker since other goroutines that attempt to lock the mutex will race
on the changing interface. Instead, just statically dispatch
sync.Mutex methods.
Rather than the main package being responsible for opening the address
and transaction managers, the namespaces of these components are
passed as parameters to the wallet.Open function.
Additionally, the address manager Options struct has been split into
two: ScryptOptions which holds the scrypt parameters needed during
passphrase key derivation, and OpenCallbacks which is only passed to
the Open function to allow the caller to provide additional details
during upgrades.
These changes are being done in preparation for a notification server
in the wallet package, with callbacks passed to the Open and Create
functions in waddrmgr and wtxmgr. Before this could happen, the
wallet package had to be responsible for actually opening the managers
from their namespaces.
If a long reorganize occurs farther back than the last saved recent
block hash (currently max 20 are saved) a full rescan is triggered
since there is no guarantee the previous blocks weren't also removed
in the reorg. In this case, the address manager was set unsynced, but
transaction history was not rolled back as well. This commit corrects
this by unconfirming all transactions but those in the genesis block.
To increase compatibility with Bitcoin Core Wallet, additional fields
were added to and other fields made optional for the listtransactions
and gettransaction results structs. For both, fee was changed to be
optional (including the zero value is allowed).
Rather than disallowing the default account to be renamed as was
proposed in #245 (and implemented in #246), the default account name
is no longer considered a reserved name by the address manager.
Instead, it is simply the initial name used for the first initial
account.
A database upgrade removes any additional aliases for the default
account in the database. This prevents a lookup for some name which
is not an account name from mapping to the default account
unexpectedly (potentially preventing incorrect account usage from the
RPC server due to bad iteraction with default parameters).
All unset account names in a JSON-RPC request are expected to be set
nil by btcjson. This behavior depends on btcsuite/btcd#399.
Additionally, the manager no longer considers the wildcard * to be a
reserved account name. Due to poor API decisions, the RPC server
overloads the meaning of account fields to optionally allow referring
to all accounts at a time, or a single account. This is not a address
manager responsibility, though, as a future cleaner API should not use
multiple differet meanings for the same field across multiple
requests. Therefore, don't burden down future APIs with this quirk
and prevent incorrect wildcard usage from the RPC server.
Closes#245.
This a refactor of the btcwallet main package to create a new wallet
package.
The main feature of this package is the integration of all the other
wallet components (waddrmgr, txstore, and chain) and the Wallet type is
'runnable', so it will be continuously updating itself against changes
notified by the remote btcd instance.
It also includes several methods which provide access to information
necessary to run a wallet RPC server.
This package is used solely for the storage of private and public
keys, and the addresses they represent. Since "wallet" is an
overloaded term and a working wallet requires transaction history as
well, rename this package and its data structures to more clearly
reflect what it is for.
This change fixes the asynchronous deferred locking that used to be
performed after some timeout after a call to walletpassphrase by
managing the locked state of each account in a new account manager
goroutine. The timeouts for new unlock requests replace any running
timeouts for older requests, rather than allowing previous timeouts to
expire before the most recent one.
Fixes#105.
If the transaction store cannot be opened and read (i.e. the version
is too old to be deserialized), the wallet is marked unsynced and
rewritten, and a new empty transaction store is written over the
previous.
This change immediately writes a new empty transaction store out to
disk if the old one could not be read. Since old transaction store
versions are not read in at start, and were previously not written out
until new transaction history was received, it was possible that a
full rescan started and finished without ever marking a synced tx
history for the next wallet start.
This is an intial pass at converting the btcwallet and deps codebases
to pass a network by their parameters, rather than by a magic number
to identify the network. The parameters in params.go have been
updated to embed a *btcnet.Params, and all previous uses of cfg.Net()
have been replaced with activeNet.{Params,Net} (where activeNet is
the global var for the active network).
Although dependancy packages have not yet been updated from using
btcwire.BitcoinNet to btcnet.Params, the parameters are now accessible
at all callsites, and individual packages can be updated to use btcnet
without requiring updates in each external btc* package at once.
While here, the exported API for btcwallet internal library packages
(txstore and wallet) have been updated to pass full network parameters
rather than the btcwire definition of a network.
Calling Bytes() on a big.Int strips any leading padding zeros. This
change fixes the test to always pad the byte slice for a private key
to a length of 32.
- Instead of returning a special constructed type whenever queries for an
address. Return the internal object with an immutable external
interface.
- Make the private key gettable from PubKeyAddress to prevent having to look up
multiple times to get information from the same structure
- Enforce addresses always have public keys.
This change reappropriates the unused `last block` field from Armory's
wallet format to hold the block chain height for a partially synced
address, that is, an address that has been partially synced to
somewhere between its first seen block and the most recently seen
block. The wallet's SyncHeight method has been updated to return
partial heights as well.
The actual marking of partially unsynced address from a rescan
progress update is not implemented yet.
Now using w.IsLocked() for all instances of above.
Also changed one other place where the logic had to be reversed
in nextChainedAddress (len(w.secret) == 32 was the condition).
The private key import codepath (called when handling the
importprivkey RPC method) was not triggering rescans for the imported
address. This change begins a new rescan for each import and adds
additional logic to the wallet file to keep track of unsynced imported
addresses. After a rescan on an imported address completes, the
address is marked as in sync with the rest of wallet and future
handshake rescans will start from the last seen block, rather than the
import height of the unsynced address.
While here, improve the logging for not just import rescans, but
rescanning on btcd connect (part of the handshake) as well.
Fixes#74.
Move the stuff that scripts can't possibly support out of the interface
and move about two type assertions so that everything still works. They
key-using interfaces can be made into a KeyedAddress itnerface if we add
any more.
Add a walletAddress interface to handle the differences betweent he
different types. Stop using btcutil.AddressPubKeyHash everywhere and just use
the standard address.
Shortly we will add new types of address, so make AddressInfo an
interface, with concrete types providing address-specific information.
Adapt existing code to this new status quo.
The flag marking chained addresses as needing private keys be
generated on the next wallet unlock was not being correctly unset
after creating and encrypting the private key. After
serializing/deserializing the wallet, on next unlock, recreating
missing private keys would begin too early in the chain and fail due
to trying to encrypt an already encrypted address.
This change correctly unsets the flag and bumps the version so a
special case can be created for ignoring duplicate encryption attempts
when reading an old wallet file. Tests have also been added to the
chained pubkey test to test for this error case.
Previous to this commit, all change addresses were indistinguishable
from manually requested addresses. This adds a new function to return
the new address, setting a new change flag to true, and return the
change status with the AddressInfo.
This is needed as part of resolving #41 (getrawchangeaddress).
The exported watching wallet would include all imported addresses'
chain indexes added to the chain index map. Imported addresses are
special as they do not belong to the address chain, and this would
cause issues when serializing the wallet.
This adds a new function, ExtendActiveAddresses, to extend the number
of active addresses, generated by the address chain, by any arbitrary
positive number. This will be used for recovering wallet addresses
from a wallet backup, as well as getting the next as many addresses as
required to keep a watching wallet in sync with an original wallet
with the private keys, or sync an original wallet with new addresses
generated by the watching wallet.
This change introduces a new function to export a wallet in memory to
a watching wallet. Watching wallets allow to watch for balance
changes and transactions to wallet addresses while only storing the
public parts of a wallet (no private keys). New addresses created by
the watching wallet will use pubkey address chaining and will allow to
receive funds to an indefinite number of new addresses, and create the
private keys for said addresses from the non-watching wallet later.
The actual exporting of a watching wallet to a file (triggered by an
RPC request) is not yet implemented.
While here, fix an issue found by new test code for the chained
address code which incorrectly set the starting index of addresses in
the chain needing private keys to be created.
This change adds a check for a valid (32-byte length) secret before
attempting to encrypt or decrypt any addresses. If the check fails, a
meaningful error (ErrWalletLocked) is returned to the caller, rather
than an error out of the aes package.
This function was not implemented (returning zero values) and all
versioning is handled internally by the wallet's ReadFrom and WriteTo
functions. If the read wallet does not match the latest file version,
it is read in the old format, but always written back with the new
format.
The wallet package was using a sync.Mutex around the saved decryption
key (kept in memory for an unlocked wallet). As the wallet package is
designed to use no internal locking, and correct synchronization is
provided by the importers of the package, this mutex has been removed.
This change uses the ChainedPubKey function to create addresses from
the previous address's public key and the wallet chaincode, without
the need for the private key, if the keypool has been depleted and the
wallet is locked. This is done since the next chained private key is
unsolvable without a locked wallet.
If a wallet contains any of these chained addresses with missing
private keys, the private keys are created during the next wallet
unlock, using ChainedPrivKey.
This change adds a function to generate the next public key of the
address chain from the previous public key and chaincode, without
needing the previous address's private key. This will be used to
allow generating new addresses with an unlocked wallet, where the
private keys are created on the next unlock.
Tests have been added to verify that the chained private and private
keys match each other (using both chaining functions) as well as an
expected value in the test case. ECDSA signature creation and
verifiction is also performed using the generated keypairs to verify
the next keypair is valid.
btcwallet originally correctly copied the behavior of armory when
chaining new private keys off the previous private key, but this
regressed in commit e8265eca41 with a
switch from double sha256 to a single sha256.
This change reverts the behavior (using the clearly-named function
btcwire.DoubleSha256) to match armory's and old btcwallet's address
chaining algorithm.
If you have flipped the switch to use btcwallet on mainnet, any newly
generated addresses created for the keypool after Jan 6, 2014 (the
date the regression was committed) should be backed up with
dumpprivkey (using btcctl) and a new wallet backup should be made of
your current wallet file (~/.btcwallet/mainnet/wallet.bin on unix) to
avoid the possibility of losing any real money.
