lbcwallet/wallet/rescan.go

283 lines
7.5 KiB
Go
Raw Normal View History

// Copyright (c) 2013-2017 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
package wallet
import (
"github.com/roasbeef/btcd/wire"
"github.com/roasbeef/btcutil"
"github.com/roasbeef/btcwallet/chain"
"github.com/roasbeef/btcwallet/waddrmgr"
"github.com/roasbeef/btcwallet/walletdb"
"github.com/roasbeef/btcwallet/wtxmgr"
)
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
// RescanProgressMsg reports the current progress made by a rescan for a
// set of wallet addresses.
type RescanProgressMsg struct {
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
Addresses []btcutil.Address
Notification *chain.RescanProgress
}
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
// RescanFinishedMsg reports the addresses that were rescanned when a
// rescanfinished message was received rescanning a batch of addresses.
type RescanFinishedMsg struct {
Addresses []btcutil.Address
Notification *chain.RescanFinished
}
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
// RescanJob is a job to be processed by the RescanManager. The job includes
// a set of wallet addresses, a starting height to begin the rescan, and
// outpoints spendable by the addresses thought to be unspent. After the
// rescan completes, the error result of the rescan RPC is sent on the Err
// channel.
type RescanJob struct {
InitialSync bool
Addrs []btcutil.Address
OutPoints []*wire.OutPoint
BlockStamp waddrmgr.BlockStamp
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
err chan error
}
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
// rescanBatch is a collection of one or more RescanJobs that were merged
// together before a rescan is performed.
type rescanBatch struct {
initialSync bool
addrs []btcutil.Address
outpoints []*wire.OutPoint
bs waddrmgr.BlockStamp
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
errChans []chan error
}
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
// SubmitRescan submits a RescanJob to the RescanManager. A channel is
// returned with the final error of the rescan. The channel is buffered
// and does not need to be read to prevent a deadlock.
func (w *Wallet) SubmitRescan(job *RescanJob) <-chan error {
errChan := make(chan error, 1)
job.err = errChan
w.rescanAddJob <- job
return errChan
}
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
// batch creates the rescanBatch for a single rescan job.
func (job *RescanJob) batch() *rescanBatch {
return &rescanBatch{
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
initialSync: job.InitialSync,
addrs: job.Addrs,
outpoints: job.OutPoints,
bs: job.BlockStamp,
errChans: []chan error{job.err},
}
}
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
// merge merges the work from k into j, setting the starting height to
// the minimum of the two jobs. This method does not check for
// duplicate addresses or outpoints.
func (b *rescanBatch) merge(job *RescanJob) {
if job.InitialSync {
b.initialSync = true
}
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
b.addrs = append(b.addrs, job.Addrs...)
b.outpoints = append(b.outpoints, job.OutPoints...)
if job.BlockStamp.Height < b.bs.Height {
b.bs = job.BlockStamp
}
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
b.errChans = append(b.errChans, job.err)
}
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
// done iterates through all error channels, duplicating sending the error
// to inform callers that the rescan finished (or could not complete due
// to an error).
func (b *rescanBatch) done(err error) {
for _, c := range b.errChans {
c <- err
}
}
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
// rescanBatchHandler handles incoming rescan request, serializing rescan
// submissions, and possibly batching many waiting requests together so they
// can be handled by a single rescan after the current one completes.
func (w *Wallet) rescanBatchHandler() {
var curBatch, nextBatch *rescanBatch
quit := w.quitChan()
out:
for {
select {
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
case job := <-w.rescanAddJob:
if curBatch == nil {
// Set current batch as this job and send
// request.
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
curBatch = job.batch()
w.rescanBatch <- curBatch
} else {
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
// Create next batch if it doesn't exist, or
// merge the job.
if nextBatch == nil {
nextBatch = job.batch()
} else {
nextBatch.merge(job)
}
}
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
case n := <-w.rescanNotifications:
switch n := n.(type) {
case *chain.RescanProgress:
w.rescanProgress <- &RescanProgressMsg{
Addresses: curBatch.addrs,
Notification: n,
}
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
case *chain.RescanFinished:
if curBatch == nil {
log.Warnf("Received rescan finished " +
"notification but no rescan " +
"currently running")
continue
}
w.rescanFinished <- &RescanFinishedMsg{
Addresses: curBatch.addrs,
Notification: n,
}
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
curBatch, nextBatch = nextBatch, nil
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
if curBatch != nil {
w.rescanBatch <- curBatch
}
default:
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
// Unexpected message
panic(n)
}
case <-quit:
break out
}
}
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
w.wg.Done()
}
// rescanProgressHandler handles notifications for partially and fully completed
// rescans by marking each rescanned address as partially or fully synced.
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
func (w *Wallet) rescanProgressHandler() {
quit := w.quitChan()
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
out:
for {
// These can't be processed out of order since both chans are
// unbuffured and are sent from same context (the batch
// handler).
select {
case msg := <-w.rescanProgress:
n := msg.Notification
log.Infof("Rescanned through block %v (height %d)",
n.Hash, n.Height)
bs := waddrmgr.BlockStamp{
Hash: *n.Hash,
Height: n.Height,
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
}
err := walletdb.Update(w.db, func(tx walletdb.ReadWriteTx) error {
ns := tx.ReadWriteBucket(waddrmgrNamespaceKey)
return w.Manager.SetSyncedTo(ns, &bs)
})
if err != nil {
log.Errorf("Failed to update address manager "+
"sync state for hash %v (height %d): %v",
n.Hash, n.Height, err)
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
}
case msg := <-w.rescanFinished:
n := msg.Notification
addrs := msg.Addresses
noun := pickNoun(len(addrs), "address", "addresses")
log.Infof("Finished rescan for %d %s (synced to block "+
"%s, height %d)", len(addrs), noun, n.Hash,
n.Height)
bs := waddrmgr.BlockStamp{
Height: n.Height,
Hash: *n.Hash,
}
err := walletdb.Update(w.db, func(tx walletdb.ReadWriteTx) error {
ns := tx.ReadWriteBucket(waddrmgrNamespaceKey)
return w.Manager.SetSyncedTo(ns, &bs)
})
if err != nil {
log.Errorf("Failed to update address manager "+
"sync state for hash %v (height %d): %v",
n.Hash, n.Height, err)
continue
}
w.SetChainSynced(true)
go w.resendUnminedTxs()
case <-quit:
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
break out
}
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
}
w.wg.Done()
}
// rescanRPCHandler reads batch jobs sent by rescanBatchHandler and sends the
// RPC requests to perform a rescan. New jobs are not read until a rescan
// finishes.
func (w *Wallet) rescanRPCHandler() {
Modernize the RPC server. This is a rather monolithic commit that moves the old RPC server to its own package (rpc/legacyrpc), introduces a new RPC server using gRPC (rpc/rpcserver), and provides the ability to defer wallet loading until request at a later time by an RPC (--noinitialload). The legacy RPC server remains the default for now while the new gRPC server is not enabled by default. Enabling the new server requires setting a listen address (--experimenalrpclisten). This experimental flag is used to effectively feature gate the server until it is ready to use as a default. Both RPC servers can be run at the same time, but require binding to different listen addresses. In theory, with the legacy RPC server now living in its own package it should become much easier to unit test the handlers. This will be useful for any future changes to the package, as compatibility with Core's wallet is still desired. Type safety has also been improved in the legacy RPC server. Multiple handler types are now used for methods that do and do not require the RPC client as a dependency. This can statically help prevent nil pointer dereferences, and was very useful for catching bugs during refactoring. To synchronize the wallet loading process between the main package (the default) and through the gRPC WalletLoader service (with the --noinitialload option), as well as increasing the loose coupling of packages, a new wallet.Loader type has been added. All creating and loading of existing wallets is done through a single Loader instance, and callbacks can be attached to the instance to run after the wallet has been opened. This is how the legacy RPC server is associated with a loaded wallet, even after the wallet is loaded by a gRPC method in a completely unrelated package. Documentation for the new RPC server has been added to the rpc/documentation directory. The documentation includes a specification for the new RPC API, addresses how to make changes to the server implementation, and provides short example clients in several different languages. Some of the new RPC methods are not implementated exactly as described by the specification. These are considered bugs with the implementation, not the spec. Known bugs are commented as such.
2015-06-01 21:57:50 +02:00
chainClient, err := w.requireChainClient()
if err != nil {
log.Errorf("rescanRPCHandler called without an RPC client")
w.wg.Done()
return
}
quit := w.quitChan()
out:
for {
select {
case batch := <-w.rescanBatch:
// Log the newly-started rescan.
numAddrs := len(batch.addrs)
noun := pickNoun(numAddrs, "address", "addresses")
log.Infof("Started rescan from block %v (height %d) for %d %s",
batch.bs.Hash, batch.bs.Height, numAddrs, noun)
Modernize the RPC server. This is a rather monolithic commit that moves the old RPC server to its own package (rpc/legacyrpc), introduces a new RPC server using gRPC (rpc/rpcserver), and provides the ability to defer wallet loading until request at a later time by an RPC (--noinitialload). The legacy RPC server remains the default for now while the new gRPC server is not enabled by default. Enabling the new server requires setting a listen address (--experimenalrpclisten). This experimental flag is used to effectively feature gate the server until it is ready to use as a default. Both RPC servers can be run at the same time, but require binding to different listen addresses. In theory, with the legacy RPC server now living in its own package it should become much easier to unit test the handlers. This will be useful for any future changes to the package, as compatibility with Core's wallet is still desired. Type safety has also been improved in the legacy RPC server. Multiple handler types are now used for methods that do and do not require the RPC client as a dependency. This can statically help prevent nil pointer dereferences, and was very useful for catching bugs during refactoring. To synchronize the wallet loading process between the main package (the default) and through the gRPC WalletLoader service (with the --noinitialload option), as well as increasing the loose coupling of packages, a new wallet.Loader type has been added. All creating and loading of existing wallets is done through a single Loader instance, and callbacks can be attached to the instance to run after the wallet has been opened. This is how the legacy RPC server is associated with a loaded wallet, even after the wallet is loaded by a gRPC method in a completely unrelated package. Documentation for the new RPC server has been added to the rpc/documentation directory. The documentation includes a specification for the new RPC API, addresses how to make changes to the server implementation, and provides short example clients in several different languages. Some of the new RPC methods are not implementated exactly as described by the specification. These are considered bugs with the implementation, not the spec. Known bugs are commented as such.
2015-06-01 21:57:50 +02:00
err := chainClient.Rescan(&batch.bs.Hash, batch.addrs,
batch.outpoints)
if err != nil {
log.Errorf("Rescan for %d %s failed: %v", numAddrs,
noun, err)
}
batch.done(err)
case <-quit:
break out
}
}
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
w.wg.Done()
}
// Rescan begins a rescan for all active addresses and unspent outputs of
// a wallet. This is intended to be used to sync a wallet back up to the
// current best block in the main chain, and is considered an initial sync
// rescan.
2015-04-06 21:03:24 +02:00
func (w *Wallet) Rescan(addrs []btcutil.Address, unspent []wtxmgr.Credit) error {
outpoints := make([]*wire.OutPoint, len(unspent))
for i, output := range unspent {
2015-04-06 21:03:24 +02:00
outpoints[i] = &output.OutPoint
}
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
job := &RescanJob{
InitialSync: true,
Addrs: addrs,
OutPoints: outpoints,
BlockStamp: w.Manager.SyncedTo(),
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
}
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
// Submit merged job and block until rescan completes.
return <-w.SubmitRescan(job)
}