lbcwallet/btcwallet.go

232 lines
6.4 KiB
Go
Raw Normal View History

// Copyright (c) 2013-2015 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
package main
import (
"io/ioutil"
"net"
"net/http"
_ "net/http/pprof"
"os"
"runtime"
Modernize the RPC server. This is a rather monolithic commit that moves the old RPC server to its own package (rpc/legacyrpc), introduces a new RPC server using gRPC (rpc/rpcserver), and provides the ability to defer wallet loading until request at a later time by an RPC (--noinitialload). The legacy RPC server remains the default for now while the new gRPC server is not enabled by default. Enabling the new server requires setting a listen address (--experimenalrpclisten). This experimental flag is used to effectively feature gate the server until it is ready to use as a default. Both RPC servers can be run at the same time, but require binding to different listen addresses. In theory, with the legacy RPC server now living in its own package it should become much easier to unit test the handlers. This will be useful for any future changes to the package, as compatibility with Core's wallet is still desired. Type safety has also been improved in the legacy RPC server. Multiple handler types are now used for methods that do and do not require the RPC client as a dependency. This can statically help prevent nil pointer dereferences, and was very useful for catching bugs during refactoring. To synchronize the wallet loading process between the main package (the default) and through the gRPC WalletLoader service (with the --noinitialload option), as well as increasing the loose coupling of packages, a new wallet.Loader type has been added. All creating and loading of existing wallets is done through a single Loader instance, and callbacks can be attached to the instance to run after the wallet has been opened. This is how the legacy RPC server is associated with a loaded wallet, even after the wallet is loaded by a gRPC method in a completely unrelated package. Documentation for the new RPC server has been added to the rpc/documentation directory. The documentation includes a specification for the new RPC API, addresses how to make changes to the server implementation, and provides short example clients in several different languages. Some of the new RPC methods are not implementated exactly as described by the specification. These are considered bugs with the implementation, not the spec. Known bugs are commented as such.
2015-06-01 21:57:50 +02:00
"sync"
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
"github.com/btcsuite/btcwallet/chain"
"github.com/btcsuite/btcwallet/rpc/legacyrpc"
"github.com/btcsuite/btcwallet/wallet"
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
)
var (
cfg *config
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
)
func main() {
// Use all processor cores.
runtime.GOMAXPROCS(runtime.NumCPU())
// Work around defer not working after os.Exit.
if err := walletMain(); err != nil {
os.Exit(1)
}
}
// walletMain is a work-around main function that is required since deferred
// functions (such as log flushing) are not called with calls to os.Exit.
// Instead, main runs this function and checks for a non-nil error, at which
// point any defers have already run, and if the error is non-nil, the program
// can be exited with an error exit status.
func walletMain() error {
// Load configuration and parse command line. This function also
// initializes logging and configures it accordingly.
tcfg, _, err := loadConfig()
if err != nil {
return err
}
cfg = tcfg
defer func() {
if logRotator != nil {
logRotator.Close()
}
}()
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
// Show version at startup.
log.Infof("Version %s", version())
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
if cfg.Profile != "" {
go func() {
listenAddr := net.JoinHostPort("", cfg.Profile)
log.Infof("Profile server listening on %s", listenAddr)
profileRedirect := http.RedirectHandler("/debug/pprof",
http.StatusSeeOther)
http.Handle("/", profileRedirect)
log.Errorf("%v", http.ListenAndServe(listenAddr, nil))
}()
}
dbDir := networkDir(cfg.AppDataDir.Value, activeNet.Params)
Modernize the RPC server. This is a rather monolithic commit that moves the old RPC server to its own package (rpc/legacyrpc), introduces a new RPC server using gRPC (rpc/rpcserver), and provides the ability to defer wallet loading until request at a later time by an RPC (--noinitialload). The legacy RPC server remains the default for now while the new gRPC server is not enabled by default. Enabling the new server requires setting a listen address (--experimenalrpclisten). This experimental flag is used to effectively feature gate the server until it is ready to use as a default. Both RPC servers can be run at the same time, but require binding to different listen addresses. In theory, with the legacy RPC server now living in its own package it should become much easier to unit test the handlers. This will be useful for any future changes to the package, as compatibility with Core's wallet is still desired. Type safety has also been improved in the legacy RPC server. Multiple handler types are now used for methods that do and do not require the RPC client as a dependency. This can statically help prevent nil pointer dereferences, and was very useful for catching bugs during refactoring. To synchronize the wallet loading process between the main package (the default) and through the gRPC WalletLoader service (with the --noinitialload option), as well as increasing the loose coupling of packages, a new wallet.Loader type has been added. All creating and loading of existing wallets is done through a single Loader instance, and callbacks can be attached to the instance to run after the wallet has been opened. This is how the legacy RPC server is associated with a loaded wallet, even after the wallet is loaded by a gRPC method in a completely unrelated package. Documentation for the new RPC server has been added to the rpc/documentation directory. The documentation includes a specification for the new RPC API, addresses how to make changes to the server implementation, and provides short example clients in several different languages. Some of the new RPC methods are not implementated exactly as described by the specification. These are considered bugs with the implementation, not the spec. Known bugs are commented as such.
2015-06-01 21:57:50 +02:00
loader := wallet.NewLoader(activeNet.Params, dbDir)
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
// Create and start HTTP server to serve wallet client connections.
// This will be updated with the wallet and chain server RPC client
// created below after each is created.
Modernize the RPC server. This is a rather monolithic commit that moves the old RPC server to its own package (rpc/legacyrpc), introduces a new RPC server using gRPC (rpc/rpcserver), and provides the ability to defer wallet loading until request at a later time by an RPC (--noinitialload). The legacy RPC server remains the default for now while the new gRPC server is not enabled by default. Enabling the new server requires setting a listen address (--experimenalrpclisten). This experimental flag is used to effectively feature gate the server until it is ready to use as a default. Both RPC servers can be run at the same time, but require binding to different listen addresses. In theory, with the legacy RPC server now living in its own package it should become much easier to unit test the handlers. This will be useful for any future changes to the package, as compatibility with Core's wallet is still desired. Type safety has also been improved in the legacy RPC server. Multiple handler types are now used for methods that do and do not require the RPC client as a dependency. This can statically help prevent nil pointer dereferences, and was very useful for catching bugs during refactoring. To synchronize the wallet loading process between the main package (the default) and through the gRPC WalletLoader service (with the --noinitialload option), as well as increasing the loose coupling of packages, a new wallet.Loader type has been added. All creating and loading of existing wallets is done through a single Loader instance, and callbacks can be attached to the instance to run after the wallet has been opened. This is how the legacy RPC server is associated with a loaded wallet, even after the wallet is loaded by a gRPC method in a completely unrelated package. Documentation for the new RPC server has been added to the rpc/documentation directory. The documentation includes a specification for the new RPC API, addresses how to make changes to the server implementation, and provides short example clients in several different languages. Some of the new RPC methods are not implementated exactly as described by the specification. These are considered bugs with the implementation, not the spec. Known bugs are commented as such.
2015-06-01 21:57:50 +02:00
rpcs, legacyRPCServer, err := startRPCServers(loader)
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
if err != nil {
Modernize the RPC server. This is a rather monolithic commit that moves the old RPC server to its own package (rpc/legacyrpc), introduces a new RPC server using gRPC (rpc/rpcserver), and provides the ability to defer wallet loading until request at a later time by an RPC (--noinitialload). The legacy RPC server remains the default for now while the new gRPC server is not enabled by default. Enabling the new server requires setting a listen address (--experimenalrpclisten). This experimental flag is used to effectively feature gate the server until it is ready to use as a default. Both RPC servers can be run at the same time, but require binding to different listen addresses. In theory, with the legacy RPC server now living in its own package it should become much easier to unit test the handlers. This will be useful for any future changes to the package, as compatibility with Core's wallet is still desired. Type safety has also been improved in the legacy RPC server. Multiple handler types are now used for methods that do and do not require the RPC client as a dependency. This can statically help prevent nil pointer dereferences, and was very useful for catching bugs during refactoring. To synchronize the wallet loading process between the main package (the default) and through the gRPC WalletLoader service (with the --noinitialload option), as well as increasing the loose coupling of packages, a new wallet.Loader type has been added. All creating and loading of existing wallets is done through a single Loader instance, and callbacks can be attached to the instance to run after the wallet has been opened. This is how the legacy RPC server is associated with a loaded wallet, even after the wallet is loaded by a gRPC method in a completely unrelated package. Documentation for the new RPC server has been added to the rpc/documentation directory. The documentation includes a specification for the new RPC API, addresses how to make changes to the server implementation, and provides short example clients in several different languages. Some of the new RPC methods are not implementated exactly as described by the specification. These are considered bugs with the implementation, not the spec. Known bugs are commented as such.
2015-06-01 21:57:50 +02:00
log.Errorf("Unable to create RPC servers: %v", err)
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
return err
}
Modernize the RPC server. This is a rather monolithic commit that moves the old RPC server to its own package (rpc/legacyrpc), introduces a new RPC server using gRPC (rpc/rpcserver), and provides the ability to defer wallet loading until request at a later time by an RPC (--noinitialload). The legacy RPC server remains the default for now while the new gRPC server is not enabled by default. Enabling the new server requires setting a listen address (--experimenalrpclisten). This experimental flag is used to effectively feature gate the server until it is ready to use as a default. Both RPC servers can be run at the same time, but require binding to different listen addresses. In theory, with the legacy RPC server now living in its own package it should become much easier to unit test the handlers. This will be useful for any future changes to the package, as compatibility with Core's wallet is still desired. Type safety has also been improved in the legacy RPC server. Multiple handler types are now used for methods that do and do not require the RPC client as a dependency. This can statically help prevent nil pointer dereferences, and was very useful for catching bugs during refactoring. To synchronize the wallet loading process between the main package (the default) and through the gRPC WalletLoader service (with the --noinitialload option), as well as increasing the loose coupling of packages, a new wallet.Loader type has been added. All creating and loading of existing wallets is done through a single Loader instance, and callbacks can be attached to the instance to run after the wallet has been opened. This is how the legacy RPC server is associated with a loaded wallet, even after the wallet is loaded by a gRPC method in a completely unrelated package. Documentation for the new RPC server has been added to the rpc/documentation directory. The documentation includes a specification for the new RPC API, addresses how to make changes to the server implementation, and provides short example clients in several different languages. Some of the new RPC methods are not implementated exactly as described by the specification. These are considered bugs with the implementation, not the spec. Known bugs are commented as such.
2015-06-01 21:57:50 +02:00
// Create and start chain RPC client so it's ready to connect to
// the wallet when loaded later.
if !cfg.NoInitialLoad {
go rpcClientConnectLoop(legacyRPCServer, loader)
}
loader.RunAfterLoad(func(w *wallet.Wallet) {
Modernize the RPC server. This is a rather monolithic commit that moves the old RPC server to its own package (rpc/legacyrpc), introduces a new RPC server using gRPC (rpc/rpcserver), and provides the ability to defer wallet loading until request at a later time by an RPC (--noinitialload). The legacy RPC server remains the default for now while the new gRPC server is not enabled by default. Enabling the new server requires setting a listen address (--experimenalrpclisten). This experimental flag is used to effectively feature gate the server until it is ready to use as a default. Both RPC servers can be run at the same time, but require binding to different listen addresses. In theory, with the legacy RPC server now living in its own package it should become much easier to unit test the handlers. This will be useful for any future changes to the package, as compatibility with Core's wallet is still desired. Type safety has also been improved in the legacy RPC server. Multiple handler types are now used for methods that do and do not require the RPC client as a dependency. This can statically help prevent nil pointer dereferences, and was very useful for catching bugs during refactoring. To synchronize the wallet loading process between the main package (the default) and through the gRPC WalletLoader service (with the --noinitialload option), as well as increasing the loose coupling of packages, a new wallet.Loader type has been added. All creating and loading of existing wallets is done through a single Loader instance, and callbacks can be attached to the instance to run after the wallet has been opened. This is how the legacy RPC server is associated with a loaded wallet, even after the wallet is loaded by a gRPC method in a completely unrelated package. Documentation for the new RPC server has been added to the rpc/documentation directory. The documentation includes a specification for the new RPC API, addresses how to make changes to the server implementation, and provides short example clients in several different languages. Some of the new RPC methods are not implementated exactly as described by the specification. These are considered bugs with the implementation, not the spec. Known bugs are commented as such.
2015-06-01 21:57:50 +02:00
startWalletRPCServices(w, rpcs, legacyRPCServer)
})
if !cfg.NoInitialLoad {
// Load the wallet database. It must have been created already
// or this will return an appropriate error.
_, err = loader.OpenExistingWallet([]byte(cfg.WalletPass), true)
if err != nil {
log.Error(err)
return err
}
}
// Add interrupt handlers to shutdown the various process components
// before exiting. Interrupt handlers run in LIFO order, so the wallet
// (which should be closed last) is added first.
addInterruptHandler(func() {
err := loader.UnloadWallet()
if err != nil && err != wallet.ErrNotLoaded {
log.Errorf("Failed to close wallet: %v", err)
}
})
Modernize the RPC server. This is a rather monolithic commit that moves the old RPC server to its own package (rpc/legacyrpc), introduces a new RPC server using gRPC (rpc/rpcserver), and provides the ability to defer wallet loading until request at a later time by an RPC (--noinitialload). The legacy RPC server remains the default for now while the new gRPC server is not enabled by default. Enabling the new server requires setting a listen address (--experimenalrpclisten). This experimental flag is used to effectively feature gate the server until it is ready to use as a default. Both RPC servers can be run at the same time, but require binding to different listen addresses. In theory, with the legacy RPC server now living in its own package it should become much easier to unit test the handlers. This will be useful for any future changes to the package, as compatibility with Core's wallet is still desired. Type safety has also been improved in the legacy RPC server. Multiple handler types are now used for methods that do and do not require the RPC client as a dependency. This can statically help prevent nil pointer dereferences, and was very useful for catching bugs during refactoring. To synchronize the wallet loading process between the main package (the default) and through the gRPC WalletLoader service (with the --noinitialload option), as well as increasing the loose coupling of packages, a new wallet.Loader type has been added. All creating and loading of existing wallets is done through a single Loader instance, and callbacks can be attached to the instance to run after the wallet has been opened. This is how the legacy RPC server is associated with a loaded wallet, even after the wallet is loaded by a gRPC method in a completely unrelated package. Documentation for the new RPC server has been added to the rpc/documentation directory. The documentation includes a specification for the new RPC API, addresses how to make changes to the server implementation, and provides short example clients in several different languages. Some of the new RPC methods are not implementated exactly as described by the specification. These are considered bugs with the implementation, not the spec. Known bugs are commented as such.
2015-06-01 21:57:50 +02:00
if rpcs != nil {
addInterruptHandler(func() {
// TODO: Does this need to wait for the grpc server to
// finish up any requests?
log.Warn("Stopping RPC server...")
rpcs.Stop()
log.Info("RPC server shutdown")
})
}
if legacyRPCServer != nil {
addInterruptHandler(func() {
log.Warn("Stopping legacy RPC server...")
legacyRPCServer.Stop()
log.Info("Legacy RPC server shutdown")
})
go func() {
<-legacyRPCServer.RequestProcessShutdown()
simulateInterrupt()
}()
Modernize the RPC server. This is a rather monolithic commit that moves the old RPC server to its own package (rpc/legacyrpc), introduces a new RPC server using gRPC (rpc/rpcserver), and provides the ability to defer wallet loading until request at a later time by an RPC (--noinitialload). The legacy RPC server remains the default for now while the new gRPC server is not enabled by default. Enabling the new server requires setting a listen address (--experimenalrpclisten). This experimental flag is used to effectively feature gate the server until it is ready to use as a default. Both RPC servers can be run at the same time, but require binding to different listen addresses. In theory, with the legacy RPC server now living in its own package it should become much easier to unit test the handlers. This will be useful for any future changes to the package, as compatibility with Core's wallet is still desired. Type safety has also been improved in the legacy RPC server. Multiple handler types are now used for methods that do and do not require the RPC client as a dependency. This can statically help prevent nil pointer dereferences, and was very useful for catching bugs during refactoring. To synchronize the wallet loading process between the main package (the default) and through the gRPC WalletLoader service (with the --noinitialload option), as well as increasing the loose coupling of packages, a new wallet.Loader type has been added. All creating and loading of existing wallets is done through a single Loader instance, and callbacks can be attached to the instance to run after the wallet has been opened. This is how the legacy RPC server is associated with a loaded wallet, even after the wallet is loaded by a gRPC method in a completely unrelated package. Documentation for the new RPC server has been added to the rpc/documentation directory. The documentation includes a specification for the new RPC API, addresses how to make changes to the server implementation, and provides short example clients in several different languages. Some of the new RPC methods are not implementated exactly as described by the specification. These are considered bugs with the implementation, not the spec. Known bugs are commented as such.
2015-06-01 21:57:50 +02:00
}
<-interruptHandlersDone
log.Info("Shutdown complete")
return nil
}
// rpcClientConnectLoop continuously attempts a connection to the consensus RPC
// server. When a connection is established, the client is used to sync the
// loaded wallet, either immediately or when loaded at a later time.
//
// The legacy RPC is optional. If set, the connected RPC client will be
// associated with the server for RPC passthrough and to enable additional
// methods.
func rpcClientConnectLoop(legacyRPCServer *legacyrpc.Server, loader *wallet.Loader) {
certs := readCAFile()
for {
chainClient, err := startChainRPC(certs)
if err != nil {
log.Errorf("Unable to open connection to consensus RPC server: %v", err)
continue
}
// Rather than inlining this logic directly into the loader
// callback, a function variable is used to avoid running any of
// this after the client disconnects by setting it to nil. This
// prevents the callback from associating a wallet loaded at a
// later time with a client that has already disconnected. A
// mutex is used to make this concurrent safe.
associateRPCClient := func(w *wallet.Wallet) {
w.SynchronizeRPC(chainClient)
if legacyRPCServer != nil {
legacyRPCServer.SetChainServer(chainClient)
}
Modernize the RPC server. This is a rather monolithic commit that moves the old RPC server to its own package (rpc/legacyrpc), introduces a new RPC server using gRPC (rpc/rpcserver), and provides the ability to defer wallet loading until request at a later time by an RPC (--noinitialload). The legacy RPC server remains the default for now while the new gRPC server is not enabled by default. Enabling the new server requires setting a listen address (--experimenalrpclisten). This experimental flag is used to effectively feature gate the server until it is ready to use as a default. Both RPC servers can be run at the same time, but require binding to different listen addresses. In theory, with the legacy RPC server now living in its own package it should become much easier to unit test the handlers. This will be useful for any future changes to the package, as compatibility with Core's wallet is still desired. Type safety has also been improved in the legacy RPC server. Multiple handler types are now used for methods that do and do not require the RPC client as a dependency. This can statically help prevent nil pointer dereferences, and was very useful for catching bugs during refactoring. To synchronize the wallet loading process between the main package (the default) and through the gRPC WalletLoader service (with the --noinitialload option), as well as increasing the loose coupling of packages, a new wallet.Loader type has been added. All creating and loading of existing wallets is done through a single Loader instance, and callbacks can be attached to the instance to run after the wallet has been opened. This is how the legacy RPC server is associated with a loaded wallet, even after the wallet is loaded by a gRPC method in a completely unrelated package. Documentation for the new RPC server has been added to the rpc/documentation directory. The documentation includes a specification for the new RPC API, addresses how to make changes to the server implementation, and provides short example clients in several different languages. Some of the new RPC methods are not implementated exactly as described by the specification. These are considered bugs with the implementation, not the spec. Known bugs are commented as such.
2015-06-01 21:57:50 +02:00
}
mu := new(sync.Mutex)
loader.RunAfterLoad(func(w *wallet.Wallet) {
Modernize the RPC server. This is a rather monolithic commit that moves the old RPC server to its own package (rpc/legacyrpc), introduces a new RPC server using gRPC (rpc/rpcserver), and provides the ability to defer wallet loading until request at a later time by an RPC (--noinitialload). The legacy RPC server remains the default for now while the new gRPC server is not enabled by default. Enabling the new server requires setting a listen address (--experimenalrpclisten). This experimental flag is used to effectively feature gate the server until it is ready to use as a default. Both RPC servers can be run at the same time, but require binding to different listen addresses. In theory, with the legacy RPC server now living in its own package it should become much easier to unit test the handlers. This will be useful for any future changes to the package, as compatibility with Core's wallet is still desired. Type safety has also been improved in the legacy RPC server. Multiple handler types are now used for methods that do and do not require the RPC client as a dependency. This can statically help prevent nil pointer dereferences, and was very useful for catching bugs during refactoring. To synchronize the wallet loading process between the main package (the default) and through the gRPC WalletLoader service (with the --noinitialload option), as well as increasing the loose coupling of packages, a new wallet.Loader type has been added. All creating and loading of existing wallets is done through a single Loader instance, and callbacks can be attached to the instance to run after the wallet has been opened. This is how the legacy RPC server is associated with a loaded wallet, even after the wallet is loaded by a gRPC method in a completely unrelated package. Documentation for the new RPC server has been added to the rpc/documentation directory. The documentation includes a specification for the new RPC API, addresses how to make changes to the server implementation, and provides short example clients in several different languages. Some of the new RPC methods are not implementated exactly as described by the specification. These are considered bugs with the implementation, not the spec. Known bugs are commented as such.
2015-06-01 21:57:50 +02:00
mu.Lock()
associate := associateRPCClient
mu.Unlock()
if associate != nil {
associate(w)
}
Modernize the RPC server. This is a rather monolithic commit that moves the old RPC server to its own package (rpc/legacyrpc), introduces a new RPC server using gRPC (rpc/rpcserver), and provides the ability to defer wallet loading until request at a later time by an RPC (--noinitialload). The legacy RPC server remains the default for now while the new gRPC server is not enabled by default. Enabling the new server requires setting a listen address (--experimenalrpclisten). This experimental flag is used to effectively feature gate the server until it is ready to use as a default. Both RPC servers can be run at the same time, but require binding to different listen addresses. In theory, with the legacy RPC server now living in its own package it should become much easier to unit test the handlers. This will be useful for any future changes to the package, as compatibility with Core's wallet is still desired. Type safety has also been improved in the legacy RPC server. Multiple handler types are now used for methods that do and do not require the RPC client as a dependency. This can statically help prevent nil pointer dereferences, and was very useful for catching bugs during refactoring. To synchronize the wallet loading process between the main package (the default) and through the gRPC WalletLoader service (with the --noinitialload option), as well as increasing the loose coupling of packages, a new wallet.Loader type has been added. All creating and loading of existing wallets is done through a single Loader instance, and callbacks can be attached to the instance to run after the wallet has been opened. This is how the legacy RPC server is associated with a loaded wallet, even after the wallet is loaded by a gRPC method in a completely unrelated package. Documentation for the new RPC server has been added to the rpc/documentation directory. The documentation includes a specification for the new RPC API, addresses how to make changes to the server implementation, and provides short example clients in several different languages. Some of the new RPC methods are not implementated exactly as described by the specification. These are considered bugs with the implementation, not the spec. Known bugs are commented as such.
2015-06-01 21:57:50 +02:00
})
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
Modernize the RPC server. This is a rather monolithic commit that moves the old RPC server to its own package (rpc/legacyrpc), introduces a new RPC server using gRPC (rpc/rpcserver), and provides the ability to defer wallet loading until request at a later time by an RPC (--noinitialload). The legacy RPC server remains the default for now while the new gRPC server is not enabled by default. Enabling the new server requires setting a listen address (--experimenalrpclisten). This experimental flag is used to effectively feature gate the server until it is ready to use as a default. Both RPC servers can be run at the same time, but require binding to different listen addresses. In theory, with the legacy RPC server now living in its own package it should become much easier to unit test the handlers. This will be useful for any future changes to the package, as compatibility with Core's wallet is still desired. Type safety has also been improved in the legacy RPC server. Multiple handler types are now used for methods that do and do not require the RPC client as a dependency. This can statically help prevent nil pointer dereferences, and was very useful for catching bugs during refactoring. To synchronize the wallet loading process between the main package (the default) and through the gRPC WalletLoader service (with the --noinitialload option), as well as increasing the loose coupling of packages, a new wallet.Loader type has been added. All creating and loading of existing wallets is done through a single Loader instance, and callbacks can be attached to the instance to run after the wallet has been opened. This is how the legacy RPC server is associated with a loaded wallet, even after the wallet is loaded by a gRPC method in a completely unrelated package. Documentation for the new RPC server has been added to the rpc/documentation directory. The documentation includes a specification for the new RPC API, addresses how to make changes to the server implementation, and provides short example clients in several different languages. Some of the new RPC methods are not implementated exactly as described by the specification. These are considered bugs with the implementation, not the spec. Known bugs are commented as such.
2015-06-01 21:57:50 +02:00
chainClient.WaitForShutdown()
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
Modernize the RPC server. This is a rather monolithic commit that moves the old RPC server to its own package (rpc/legacyrpc), introduces a new RPC server using gRPC (rpc/rpcserver), and provides the ability to defer wallet loading until request at a later time by an RPC (--noinitialload). The legacy RPC server remains the default for now while the new gRPC server is not enabled by default. Enabling the new server requires setting a listen address (--experimenalrpclisten). This experimental flag is used to effectively feature gate the server until it is ready to use as a default. Both RPC servers can be run at the same time, but require binding to different listen addresses. In theory, with the legacy RPC server now living in its own package it should become much easier to unit test the handlers. This will be useful for any future changes to the package, as compatibility with Core's wallet is still desired. Type safety has also been improved in the legacy RPC server. Multiple handler types are now used for methods that do and do not require the RPC client as a dependency. This can statically help prevent nil pointer dereferences, and was very useful for catching bugs during refactoring. To synchronize the wallet loading process between the main package (the default) and through the gRPC WalletLoader service (with the --noinitialload option), as well as increasing the loose coupling of packages, a new wallet.Loader type has been added. All creating and loading of existing wallets is done through a single Loader instance, and callbacks can be attached to the instance to run after the wallet has been opened. This is how the legacy RPC server is associated with a loaded wallet, even after the wallet is loaded by a gRPC method in a completely unrelated package. Documentation for the new RPC server has been added to the rpc/documentation directory. The documentation includes a specification for the new RPC API, addresses how to make changes to the server implementation, and provides short example clients in several different languages. Some of the new RPC methods are not implementated exactly as described by the specification. These are considered bugs with the implementation, not the spec. Known bugs are commented as such.
2015-06-01 21:57:50 +02:00
mu.Lock()
associateRPCClient = nil
mu.Unlock()
Modernize the RPC server. This is a rather monolithic commit that moves the old RPC server to its own package (rpc/legacyrpc), introduces a new RPC server using gRPC (rpc/rpcserver), and provides the ability to defer wallet loading until request at a later time by an RPC (--noinitialload). The legacy RPC server remains the default for now while the new gRPC server is not enabled by default. Enabling the new server requires setting a listen address (--experimenalrpclisten). This experimental flag is used to effectively feature gate the server until it is ready to use as a default. Both RPC servers can be run at the same time, but require binding to different listen addresses. In theory, with the legacy RPC server now living in its own package it should become much easier to unit test the handlers. This will be useful for any future changes to the package, as compatibility with Core's wallet is still desired. Type safety has also been improved in the legacy RPC server. Multiple handler types are now used for methods that do and do not require the RPC client as a dependency. This can statically help prevent nil pointer dereferences, and was very useful for catching bugs during refactoring. To synchronize the wallet loading process between the main package (the default) and through the gRPC WalletLoader service (with the --noinitialload option), as well as increasing the loose coupling of packages, a new wallet.Loader type has been added. All creating and loading of existing wallets is done through a single Loader instance, and callbacks can be attached to the instance to run after the wallet has been opened. This is how the legacy RPC server is associated with a loaded wallet, even after the wallet is loaded by a gRPC method in a completely unrelated package. Documentation for the new RPC server has been added to the rpc/documentation directory. The documentation includes a specification for the new RPC API, addresses how to make changes to the server implementation, and provides short example clients in several different languages. Some of the new RPC methods are not implementated exactly as described by the specification. These are considered bugs with the implementation, not the spec. Known bugs are commented as such.
2015-06-01 21:57:50 +02:00
loadedWallet, ok := loader.LoadedWallet()
if ok {
// Do not attempt a reconnect when the wallet was
// explicitly stopped.
if loadedWallet.ShuttingDown() {
return
}
Modernize the RPC server. This is a rather monolithic commit that moves the old RPC server to its own package (rpc/legacyrpc), introduces a new RPC server using gRPC (rpc/rpcserver), and provides the ability to defer wallet loading until request at a later time by an RPC (--noinitialload). The legacy RPC server remains the default for now while the new gRPC server is not enabled by default. Enabling the new server requires setting a listen address (--experimenalrpclisten). This experimental flag is used to effectively feature gate the server until it is ready to use as a default. Both RPC servers can be run at the same time, but require binding to different listen addresses. In theory, with the legacy RPC server now living in its own package it should become much easier to unit test the handlers. This will be useful for any future changes to the package, as compatibility with Core's wallet is still desired. Type safety has also been improved in the legacy RPC server. Multiple handler types are now used for methods that do and do not require the RPC client as a dependency. This can statically help prevent nil pointer dereferences, and was very useful for catching bugs during refactoring. To synchronize the wallet loading process between the main package (the default) and through the gRPC WalletLoader service (with the --noinitialload option), as well as increasing the loose coupling of packages, a new wallet.Loader type has been added. All creating and loading of existing wallets is done through a single Loader instance, and callbacks can be attached to the instance to run after the wallet has been opened. This is how the legacy RPC server is associated with a loaded wallet, even after the wallet is loaded by a gRPC method in a completely unrelated package. Documentation for the new RPC server has been added to the rpc/documentation directory. The documentation includes a specification for the new RPC API, addresses how to make changes to the server implementation, and provides short example clients in several different languages. Some of the new RPC methods are not implementated exactly as described by the specification. These are considered bugs with the implementation, not the spec. Known bugs are commented as such.
2015-06-01 21:57:50 +02:00
loadedWallet.SetChainSynced(false)
// TODO: Rework the wallet so changing the RPC client
// does not require stopping and restarting everything.
loadedWallet.Stop()
loadedWallet.WaitForShutdown()
loadedWallet.Start()
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
}
Modernize the RPC server. This is a rather monolithic commit that moves the old RPC server to its own package (rpc/legacyrpc), introduces a new RPC server using gRPC (rpc/rpcserver), and provides the ability to defer wallet loading until request at a later time by an RPC (--noinitialload). The legacy RPC server remains the default for now while the new gRPC server is not enabled by default. Enabling the new server requires setting a listen address (--experimenalrpclisten). This experimental flag is used to effectively feature gate the server until it is ready to use as a default. Both RPC servers can be run at the same time, but require binding to different listen addresses. In theory, with the legacy RPC server now living in its own package it should become much easier to unit test the handlers. This will be useful for any future changes to the package, as compatibility with Core's wallet is still desired. Type safety has also been improved in the legacy RPC server. Multiple handler types are now used for methods that do and do not require the RPC client as a dependency. This can statically help prevent nil pointer dereferences, and was very useful for catching bugs during refactoring. To synchronize the wallet loading process between the main package (the default) and through the gRPC WalletLoader service (with the --noinitialload option), as well as increasing the loose coupling of packages, a new wallet.Loader type has been added. All creating and loading of existing wallets is done through a single Loader instance, and callbacks can be attached to the instance to run after the wallet has been opened. This is how the legacy RPC server is associated with a loaded wallet, even after the wallet is loaded by a gRPC method in a completely unrelated package. Documentation for the new RPC server has been added to the rpc/documentation directory. The documentation includes a specification for the new RPC API, addresses how to make changes to the server implementation, and provides short example clients in several different languages. Some of the new RPC methods are not implementated exactly as described by the specification. These are considered bugs with the implementation, not the spec. Known bugs are commented as such.
2015-06-01 21:57:50 +02:00
}
}
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
Modernize the RPC server. This is a rather monolithic commit that moves the old RPC server to its own package (rpc/legacyrpc), introduces a new RPC server using gRPC (rpc/rpcserver), and provides the ability to defer wallet loading until request at a later time by an RPC (--noinitialload). The legacy RPC server remains the default for now while the new gRPC server is not enabled by default. Enabling the new server requires setting a listen address (--experimenalrpclisten). This experimental flag is used to effectively feature gate the server until it is ready to use as a default. Both RPC servers can be run at the same time, but require binding to different listen addresses. In theory, with the legacy RPC server now living in its own package it should become much easier to unit test the handlers. This will be useful for any future changes to the package, as compatibility with Core's wallet is still desired. Type safety has also been improved in the legacy RPC server. Multiple handler types are now used for methods that do and do not require the RPC client as a dependency. This can statically help prevent nil pointer dereferences, and was very useful for catching bugs during refactoring. To synchronize the wallet loading process between the main package (the default) and through the gRPC WalletLoader service (with the --noinitialload option), as well as increasing the loose coupling of packages, a new wallet.Loader type has been added. All creating and loading of existing wallets is done through a single Loader instance, and callbacks can be attached to the instance to run after the wallet has been opened. This is how the legacy RPC server is associated with a loaded wallet, even after the wallet is loaded by a gRPC method in a completely unrelated package. Documentation for the new RPC server has been added to the rpc/documentation directory. The documentation includes a specification for the new RPC API, addresses how to make changes to the server implementation, and provides short example clients in several different languages. Some of the new RPC methods are not implementated exactly as described by the specification. These are considered bugs with the implementation, not the spec. Known bugs are commented as such.
2015-06-01 21:57:50 +02:00
func readCAFile() []byte {
// Read certificate file if TLS is not disabled.
var certs []byte
if !cfg.DisableClientTLS {
var err error
certs, err = ioutil.ReadFile(cfg.CAFile.Value)
Modernize the RPC server. This is a rather monolithic commit that moves the old RPC server to its own package (rpc/legacyrpc), introduces a new RPC server using gRPC (rpc/rpcserver), and provides the ability to defer wallet loading until request at a later time by an RPC (--noinitialload). The legacy RPC server remains the default for now while the new gRPC server is not enabled by default. Enabling the new server requires setting a listen address (--experimenalrpclisten). This experimental flag is used to effectively feature gate the server until it is ready to use as a default. Both RPC servers can be run at the same time, but require binding to different listen addresses. In theory, with the legacy RPC server now living in its own package it should become much easier to unit test the handlers. This will be useful for any future changes to the package, as compatibility with Core's wallet is still desired. Type safety has also been improved in the legacy RPC server. Multiple handler types are now used for methods that do and do not require the RPC client as a dependency. This can statically help prevent nil pointer dereferences, and was very useful for catching bugs during refactoring. To synchronize the wallet loading process between the main package (the default) and through the gRPC WalletLoader service (with the --noinitialload option), as well as increasing the loose coupling of packages, a new wallet.Loader type has been added. All creating and loading of existing wallets is done through a single Loader instance, and callbacks can be attached to the instance to run after the wallet has been opened. This is how the legacy RPC server is associated with a loaded wallet, even after the wallet is loaded by a gRPC method in a completely unrelated package. Documentation for the new RPC server has been added to the rpc/documentation directory. The documentation includes a specification for the new RPC API, addresses how to make changes to the server implementation, and provides short example clients in several different languages. Some of the new RPC methods are not implementated exactly as described by the specification. These are considered bugs with the implementation, not the spec. Known bugs are commented as such.
2015-06-01 21:57:50 +02:00
if err != nil {
log.Warnf("Cannot open CA file: %v", err)
// If there's an error reading the CA file, continue
// with nil certs and without the client connection.
certs = nil
}
} else {
log.Info("Chain server RPC TLS is disabled")
}
return certs
}
// startChainRPC opens a RPC client connection to a btcd server for blockchain
// services. This function uses the RPC options from the global config and
// there is no recovery in case the server is not available or if there is an
// authentication error. Instead, all requests to the client will simply error.
func startChainRPC(certs []byte) (*chain.RPCClient, error) {
log.Infof("Attempting RPC client connection to %v", cfg.RPCConnect)
rpcc, err := chain.NewRPCClient(activeNet.Params, cfg.RPCConnect,
cfg.BtcdUsername, cfg.BtcdPassword, certs, cfg.DisableClientTLS, 0)
if err != nil {
return nil, err
}
err = rpcc.Start()
return rpcc, err
Remove account support, fix races on btcd connect. This commit is the result of several big changes being made to the wallet. In particular, the "handshake" (initial sync to the chain server) was quite racy and required proper synchronization. To make fixing this race easier, several other changes were made to the internal wallet data structures and much of the RPC server ended up being rewritten. First, all account support has been removed. The previous Account struct has been replaced with a Wallet structure, which includes a keystore for saving keys, and a txstore for storing relevant transactions. This decision has been made since it is the opinion of myself and other developers that bitcoind accounts are fundamentally broken (as accounts implemented by bitcoind support both arbitrary address groupings as well as moving balances between accounts -- these are fundamentally incompatible features), and since a BIP0032 keystore is soon planned to be implemented (at which point, "accounts" can return as HD extended keys). With the keystore handling the grouping of related keys, there is no reason have many different Account structs, and the AccountManager has been removed as well. All RPC handlers that take an account option will only work with "" (the default account) or "*" if the RPC allows specifying all accounts. Second, much of the RPC server has been cleaned up. The global variables for the RPC server and chain server client have been moved to part of the rpcServer struct, and the handlers for each RPC method that are looked up change depending on which components have been set. Passthrough requests are also no longer handled specially, but when the chain server is set, a handler to perform the passthrough will be returned if the method is not otherwise a wallet RPC. The notification system for websocket clients has also been rewritten so wallet components can send notifications through channels, rather than requiring direct access to the RPC server itself, or worse still, sending directly to a websocket client's send channel. In the future, this will enable proper registration of notifications, rather than unsolicited broadcasts to every connected websocket client (see issue #84). Finally, and the main reason why much of this cleanup was necessary, the races during intial sync with the chain server have been fixed. Previously, when the 'Handshake' was run, a rescan would occur which would perform modifications to Account data structures as notifications were received. Synchronization was provided with a single binary semaphore which serialized all access to wallet and account data. However, the Handshake itself was not able to run with this lock (or else notifications would block), and many data races would occur as both notifications were being handled. If GOMAXPROCS was ever increased beyond 1, btcwallet would always immediately crash due to invalid addresses caused by the data races on startup. To fix this, the single lock for all wallet access has been replaced with mutexes for both the keystore and txstore. Handling of btcd notifications and client requests may now occur simultaneously. GOMAXPROCS has also been set to the number of logical CPUs at the beginning of main, since with the data races fixed, there's no reason to prevent the extra parallelism gained by increasing it. Closes #78. Closes #101. Closes #110.
2014-07-09 05:17:38 +02:00
}