lbcwallet/ntfns.go

277 lines
8.3 KiB
Go
Raw Normal View History

/*
* Copyright (c) 2013, 2014 Conformal Systems LLC <info@conformal.com>
*
* Permission to use, copy, modify, and distribute this software for any
* purpose with or without fee is hereby granted, provided that the above
* copyright notice and this permission notice appear in all copies.
*
* THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
* WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
* ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
* WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
* ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
* OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
// This file implements the notification handlers for btcd-side notifications.
package main
import (
"encoding/hex"
"fmt"
"sync"
"time"
"github.com/conformal/btcjson"
"github.com/conformal/btcscript"
"github.com/conformal/btcutil"
"github.com/conformal/btcwallet/txstore"
"github.com/conformal/btcwallet/wallet"
"github.com/conformal/btcwire"
"github.com/conformal/btcws"
)
func parseBlock(block *btcws.BlockDetails) (*txstore.Block, int, error) {
if block == nil {
Another day, another tx store implementation. The last transaction store was a great example of how not to write scalable software. For a variety of reasons, it was very slow at processing transaction inserts. Among them: 1) Every single transaction record being saved in a linked list (container/list), and inserting into this list would be an O(n) operation so that records could be ordered by receive date. 2) Every single transaction in the above mentioned list was iterated over in order to find double spends which must be removed. It is silly to do this check for mined transactions, which already have been checked for this by btcd. Worse yet, if double spends were found, the list would be iterated a second (or third, or fourth) time for each removed transaction. 3) All spend tracking for signed-by-wallet transactions was found on each transaction insert, even if the now spent previous transaction outputs were known by the caller. This list could keep going on, but you get the idea. It was bad. To resolve these issues a new transaction store had to be implemented. The new implementation: 1) Tracks mined and unmined transactions in different data structures. Mined transactions are cheap to track because the required double spend checks have already been performed by the chain server, and double spend checks are only required to be performed on newly-inserted mined transactions which may conflict with previous unmined transactions. 2) Saves mined transactions grouped by block first, and then by their transaction index. Lookup keys for mined transactions are simply the block height (in the best chain, that's all we save) and index of the transaction in the block. This makes looking up any arbitrary transaction almost an O(1) operation (almost, because block height and block indexes are mapped to their slice indexes with a Go map). 3) Saves records in each transaction for whether the outputs are wallet credits (spendable by wallet) and for whether inputs debit from previous credits. Both structures point back to the source or spender (credits point to the transaction that spends them, or nil for unspent credits, and debits include keys to lookup the transaction credits they spent. While complicated to keep track of, this greatly simplifies the spent tracking for transactions across rollbacks and transaction removals. 4) Implements double spend checking as an almost O(1) operation. A Go map is used to map each previous outpoint for all unconfirmed transactions to the unconfirmed tx record itself. Checking for double spends on confirmed transaction inserts only involves looking up each previous outpoint of the inserted tx in this map. If a double spend is found, removal is simplified by only removing the transaction and its spend chain from store maps, rather than iterating a linked list several times over to remove each dead transaction in the spend chain. 5) Allows the caller to specify the previous credits which are spent by a debiting transaction. When a transaction is created by wallet, the previous outputs are already known, and by passing their record types to the AddDebits method, lookups for each previously unspent credit are omitted. 6) Bookkeeps all blocks with transactions with unspent credits, and bookkeeps the transaction indexes of all transactions with unspent outputs for a single block. For the case where the caller adding a debit record does not know what credits a transaction debits from, these bookkeeping structures allow the store to only consider known unspent transactions, rather than searching through both spent and unspents. 7) Saves amount deltas for the entire balance as a result of each block, due to transactions within that block. This improves the performance of calculating the full balance by not needing to iterate over every transaction, and then every credit, to determine if a credit is spent or unspent. When transactions are moved from unconfirmed to a block structure, the amount deltas are incremented by the amount of all transaction credits (both spent and unspent) and debited by the total amount the transaction spends from previous wallet credits. For the common case of calculating a balance with just one confirmation, the only involves iterating over each block structure and adding the (possibly negative) amount delta. Coinbase rewards are saved similarly, but with a different amount variable so they can be seperatly included or excluded. Due to all of the changes in how the store internally works, the serialization format has changed. To simplify the serialization logic, support for reading the last store file version has been removed. Past this change, a rescan (run automatically) will be required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
return nil, btcutil.TxIndexUnknown, nil
}
blksha, err := btcwire.NewShaHashFromStr(block.Hash)
if err != nil {
Another day, another tx store implementation. The last transaction store was a great example of how not to write scalable software. For a variety of reasons, it was very slow at processing transaction inserts. Among them: 1) Every single transaction record being saved in a linked list (container/list), and inserting into this list would be an O(n) operation so that records could be ordered by receive date. 2) Every single transaction in the above mentioned list was iterated over in order to find double spends which must be removed. It is silly to do this check for mined transactions, which already have been checked for this by btcd. Worse yet, if double spends were found, the list would be iterated a second (or third, or fourth) time for each removed transaction. 3) All spend tracking for signed-by-wallet transactions was found on each transaction insert, even if the now spent previous transaction outputs were known by the caller. This list could keep going on, but you get the idea. It was bad. To resolve these issues a new transaction store had to be implemented. The new implementation: 1) Tracks mined and unmined transactions in different data structures. Mined transactions are cheap to track because the required double spend checks have already been performed by the chain server, and double spend checks are only required to be performed on newly-inserted mined transactions which may conflict with previous unmined transactions. 2) Saves mined transactions grouped by block first, and then by their transaction index. Lookup keys for mined transactions are simply the block height (in the best chain, that's all we save) and index of the transaction in the block. This makes looking up any arbitrary transaction almost an O(1) operation (almost, because block height and block indexes are mapped to their slice indexes with a Go map). 3) Saves records in each transaction for whether the outputs are wallet credits (spendable by wallet) and for whether inputs debit from previous credits. Both structures point back to the source or spender (credits point to the transaction that spends them, or nil for unspent credits, and debits include keys to lookup the transaction credits they spent. While complicated to keep track of, this greatly simplifies the spent tracking for transactions across rollbacks and transaction removals. 4) Implements double spend checking as an almost O(1) operation. A Go map is used to map each previous outpoint for all unconfirmed transactions to the unconfirmed tx record itself. Checking for double spends on confirmed transaction inserts only involves looking up each previous outpoint of the inserted tx in this map. If a double spend is found, removal is simplified by only removing the transaction and its spend chain from store maps, rather than iterating a linked list several times over to remove each dead transaction in the spend chain. 5) Allows the caller to specify the previous credits which are spent by a debiting transaction. When a transaction is created by wallet, the previous outputs are already known, and by passing their record types to the AddDebits method, lookups for each previously unspent credit are omitted. 6) Bookkeeps all blocks with transactions with unspent credits, and bookkeeps the transaction indexes of all transactions with unspent outputs for a single block. For the case where the caller adding a debit record does not know what credits a transaction debits from, these bookkeeping structures allow the store to only consider known unspent transactions, rather than searching through both spent and unspents. 7) Saves amount deltas for the entire balance as a result of each block, due to transactions within that block. This improves the performance of calculating the full balance by not needing to iterate over every transaction, and then every credit, to determine if a credit is spent or unspent. When transactions are moved from unconfirmed to a block structure, the amount deltas are incremented by the amount of all transaction credits (both spent and unspent) and debited by the total amount the transaction spends from previous wallet credits. For the common case of calculating a balance with just one confirmation, the only involves iterating over each block structure and adding the (possibly negative) amount delta. Coinbase rewards are saved similarly, but with a different amount variable so they can be seperatly included or excluded. Due to all of the changes in how the store internally works, the serialization format has changed. To simplify the serialization logic, support for reading the last store file version has been removed. Past this change, a rescan (run automatically) will be required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
return nil, btcutil.TxIndexUnknown, err
}
b := &txstore.Block{
Height: block.Height,
Hash: *blksha,
Time: time.Unix(block.Time, 0),
Another day, another tx store implementation. The last transaction store was a great example of how not to write scalable software. For a variety of reasons, it was very slow at processing transaction inserts. Among them: 1) Every single transaction record being saved in a linked list (container/list), and inserting into this list would be an O(n) operation so that records could be ordered by receive date. 2) Every single transaction in the above mentioned list was iterated over in order to find double spends which must be removed. It is silly to do this check for mined transactions, which already have been checked for this by btcd. Worse yet, if double spends were found, the list would be iterated a second (or third, or fourth) time for each removed transaction. 3) All spend tracking for signed-by-wallet transactions was found on each transaction insert, even if the now spent previous transaction outputs were known by the caller. This list could keep going on, but you get the idea. It was bad. To resolve these issues a new transaction store had to be implemented. The new implementation: 1) Tracks mined and unmined transactions in different data structures. Mined transactions are cheap to track because the required double spend checks have already been performed by the chain server, and double spend checks are only required to be performed on newly-inserted mined transactions which may conflict with previous unmined transactions. 2) Saves mined transactions grouped by block first, and then by their transaction index. Lookup keys for mined transactions are simply the block height (in the best chain, that's all we save) and index of the transaction in the block. This makes looking up any arbitrary transaction almost an O(1) operation (almost, because block height and block indexes are mapped to their slice indexes with a Go map). 3) Saves records in each transaction for whether the outputs are wallet credits (spendable by wallet) and for whether inputs debit from previous credits. Both structures point back to the source or spender (credits point to the transaction that spends them, or nil for unspent credits, and debits include keys to lookup the transaction credits they spent. While complicated to keep track of, this greatly simplifies the spent tracking for transactions across rollbacks and transaction removals. 4) Implements double spend checking as an almost O(1) operation. A Go map is used to map each previous outpoint for all unconfirmed transactions to the unconfirmed tx record itself. Checking for double spends on confirmed transaction inserts only involves looking up each previous outpoint of the inserted tx in this map. If a double spend is found, removal is simplified by only removing the transaction and its spend chain from store maps, rather than iterating a linked list several times over to remove each dead transaction in the spend chain. 5) Allows the caller to specify the previous credits which are spent by a debiting transaction. When a transaction is created by wallet, the previous outputs are already known, and by passing their record types to the AddDebits method, lookups for each previously unspent credit are omitted. 6) Bookkeeps all blocks with transactions with unspent credits, and bookkeeps the transaction indexes of all transactions with unspent outputs for a single block. For the case where the caller adding a debit record does not know what credits a transaction debits from, these bookkeeping structures allow the store to only consider known unspent transactions, rather than searching through both spent and unspents. 7) Saves amount deltas for the entire balance as a result of each block, due to transactions within that block. This improves the performance of calculating the full balance by not needing to iterate over every transaction, and then every credit, to determine if a credit is spent or unspent. When transactions are moved from unconfirmed to a block structure, the amount deltas are incremented by the amount of all transaction credits (both spent and unspent) and debited by the total amount the transaction spends from previous wallet credits. For the common case of calculating a balance with just one confirmation, the only involves iterating over each block structure and adding the (possibly negative) amount delta. Coinbase rewards are saved similarly, but with a different amount variable so they can be seperatly included or excluded. Due to all of the changes in how the store internally works, the serialization format has changed. To simplify the serialization logic, support for reading the last store file version has been removed. Past this change, a rescan (run automatically) will be required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
}
return b, block.Index, nil
}
type notificationHandler func(btcjson.Cmd) error
var notificationHandlers = map[string]notificationHandler{
btcws.BlockConnectedNtfnMethod: NtfnBlockConnected,
btcws.BlockDisconnectedNtfnMethod: NtfnBlockDisconnected,
btcws.RecvTxNtfnMethod: NtfnRecvTx,
btcws.RedeemingTxNtfnMethod: NtfnRedeemingTx,
btcws.RescanProgressNtfnMethod: NtfnRescanProgress,
}
// NtfnRecvTx handles the btcws.RecvTxNtfn notification.
func NtfnRecvTx(n btcjson.Cmd) error {
rtx, ok := n.(*btcws.RecvTxNtfn)
if !ok {
return fmt.Errorf("%v handler: unexpected type", n.Method())
}
bs, err := GetCurBlock()
if err != nil {
return fmt.Errorf("%v handler: cannot get current block: %v", n.Method(), err)
}
rawTx, err := hex.DecodeString(rtx.HexTx)
if err != nil {
return fmt.Errorf("%v handler: bad hexstring: %v", n.Method(), err)
}
tx, err := btcutil.NewTxFromBytes(rawTx)
if err != nil {
return fmt.Errorf("%v handler: bad transaction bytes: %v", n.Method(), err)
}
Another day, another tx store implementation. The last transaction store was a great example of how not to write scalable software. For a variety of reasons, it was very slow at processing transaction inserts. Among them: 1) Every single transaction record being saved in a linked list (container/list), and inserting into this list would be an O(n) operation so that records could be ordered by receive date. 2) Every single transaction in the above mentioned list was iterated over in order to find double spends which must be removed. It is silly to do this check for mined transactions, which already have been checked for this by btcd. Worse yet, if double spends were found, the list would be iterated a second (or third, or fourth) time for each removed transaction. 3) All spend tracking for signed-by-wallet transactions was found on each transaction insert, even if the now spent previous transaction outputs were known by the caller. This list could keep going on, but you get the idea. It was bad. To resolve these issues a new transaction store had to be implemented. The new implementation: 1) Tracks mined and unmined transactions in different data structures. Mined transactions are cheap to track because the required double spend checks have already been performed by the chain server, and double spend checks are only required to be performed on newly-inserted mined transactions which may conflict with previous unmined transactions. 2) Saves mined transactions grouped by block first, and then by their transaction index. Lookup keys for mined transactions are simply the block height (in the best chain, that's all we save) and index of the transaction in the block. This makes looking up any arbitrary transaction almost an O(1) operation (almost, because block height and block indexes are mapped to their slice indexes with a Go map). 3) Saves records in each transaction for whether the outputs are wallet credits (spendable by wallet) and for whether inputs debit from previous credits. Both structures point back to the source or spender (credits point to the transaction that spends them, or nil for unspent credits, and debits include keys to lookup the transaction credits they spent. While complicated to keep track of, this greatly simplifies the spent tracking for transactions across rollbacks and transaction removals. 4) Implements double spend checking as an almost O(1) operation. A Go map is used to map each previous outpoint for all unconfirmed transactions to the unconfirmed tx record itself. Checking for double spends on confirmed transaction inserts only involves looking up each previous outpoint of the inserted tx in this map. If a double spend is found, removal is simplified by only removing the transaction and its spend chain from store maps, rather than iterating a linked list several times over to remove each dead transaction in the spend chain. 5) Allows the caller to specify the previous credits which are spent by a debiting transaction. When a transaction is created by wallet, the previous outputs are already known, and by passing their record types to the AddDebits method, lookups for each previously unspent credit are omitted. 6) Bookkeeps all blocks with transactions with unspent credits, and bookkeeps the transaction indexes of all transactions with unspent outputs for a single block. For the case where the caller adding a debit record does not know what credits a transaction debits from, these bookkeeping structures allow the store to only consider known unspent transactions, rather than searching through both spent and unspents. 7) Saves amount deltas for the entire balance as a result of each block, due to transactions within that block. This improves the performance of calculating the full balance by not needing to iterate over every transaction, and then every credit, to determine if a credit is spent or unspent. When transactions are moved from unconfirmed to a block structure, the amount deltas are incremented by the amount of all transaction credits (both spent and unspent) and debited by the total amount the transaction spends from previous wallet credits. For the common case of calculating a balance with just one confirmation, the only involves iterating over each block structure and adding the (possibly negative) amount delta. Coinbase rewards are saved similarly, but with a different amount variable so they can be seperatly included or excluded. Due to all of the changes in how the store internally works, the serialization format has changed. To simplify the serialization logic, support for reading the last store file version has been removed. Past this change, a rescan (run automatically) will be required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
block, txIdx, err := parseBlock(rtx.Block)
if err != nil {
return fmt.Errorf("%v handler: bad block: %v", n.Method(), err)
}
tx.SetIndex(txIdx)
// For transactions originating from this wallet, the sent tx history should
// be recorded before the received history. If wallet created this tx, wait
// for the sent history to finish being recorded before continuing.
//
// TODO(jrick) this is wrong due to tx malleability. Cannot safely use the
// txsha as an identifier.
req := SendTxHistSyncRequest{
txsha: *tx.Sha(),
response: make(chan SendTxHistSyncResponse),
}
SendTxHistSyncChans.access <- req
resp := <-req.response
if resp.ok {
// Wait until send history has been recorded.
<-resp.c
SendTxHistSyncChans.remove <- *tx.Sha()
}
// For every output, find all accounts handling that output address (if any)
// and record the received txout.
for outIdx, txout := range tx.MsgTx().TxOut {
var accounts []*Account
_, addrs, _, _ := btcscript.ExtractPkScriptAddrs(txout.PkScript,
activeNet.Params)
for _, addr := range addrs {
a, err := AcctMgr.AccountByAddress(addr)
if err != nil {
continue
}
accounts = append(accounts, a)
}
for _, a := range accounts {
txr, err := a.TxStore.InsertTx(tx, block)
Another day, another tx store implementation. The last transaction store was a great example of how not to write scalable software. For a variety of reasons, it was very slow at processing transaction inserts. Among them: 1) Every single transaction record being saved in a linked list (container/list), and inserting into this list would be an O(n) operation so that records could be ordered by receive date. 2) Every single transaction in the above mentioned list was iterated over in order to find double spends which must be removed. It is silly to do this check for mined transactions, which already have been checked for this by btcd. Worse yet, if double spends were found, the list would be iterated a second (or third, or fourth) time for each removed transaction. 3) All spend tracking for signed-by-wallet transactions was found on each transaction insert, even if the now spent previous transaction outputs were known by the caller. This list could keep going on, but you get the idea. It was bad. To resolve these issues a new transaction store had to be implemented. The new implementation: 1) Tracks mined and unmined transactions in different data structures. Mined transactions are cheap to track because the required double spend checks have already been performed by the chain server, and double spend checks are only required to be performed on newly-inserted mined transactions which may conflict with previous unmined transactions. 2) Saves mined transactions grouped by block first, and then by their transaction index. Lookup keys for mined transactions are simply the block height (in the best chain, that's all we save) and index of the transaction in the block. This makes looking up any arbitrary transaction almost an O(1) operation (almost, because block height and block indexes are mapped to their slice indexes with a Go map). 3) Saves records in each transaction for whether the outputs are wallet credits (spendable by wallet) and for whether inputs debit from previous credits. Both structures point back to the source or spender (credits point to the transaction that spends them, or nil for unspent credits, and debits include keys to lookup the transaction credits they spent. While complicated to keep track of, this greatly simplifies the spent tracking for transactions across rollbacks and transaction removals. 4) Implements double spend checking as an almost O(1) operation. A Go map is used to map each previous outpoint for all unconfirmed transactions to the unconfirmed tx record itself. Checking for double spends on confirmed transaction inserts only involves looking up each previous outpoint of the inserted tx in this map. If a double spend is found, removal is simplified by only removing the transaction and its spend chain from store maps, rather than iterating a linked list several times over to remove each dead transaction in the spend chain. 5) Allows the caller to specify the previous credits which are spent by a debiting transaction. When a transaction is created by wallet, the previous outputs are already known, and by passing their record types to the AddDebits method, lookups for each previously unspent credit are omitted. 6) Bookkeeps all blocks with transactions with unspent credits, and bookkeeps the transaction indexes of all transactions with unspent outputs for a single block. For the case where the caller adding a debit record does not know what credits a transaction debits from, these bookkeeping structures allow the store to only consider known unspent transactions, rather than searching through both spent and unspents. 7) Saves amount deltas for the entire balance as a result of each block, due to transactions within that block. This improves the performance of calculating the full balance by not needing to iterate over every transaction, and then every credit, to determine if a credit is spent or unspent. When transactions are moved from unconfirmed to a block structure, the amount deltas are incremented by the amount of all transaction credits (both spent and unspent) and debited by the total amount the transaction spends from previous wallet credits. For the common case of calculating a balance with just one confirmation, the only involves iterating over each block structure and adding the (possibly negative) amount delta. Coinbase rewards are saved similarly, but with a different amount variable so they can be seperatly included or excluded. Due to all of the changes in how the store internally works, the serialization format has changed. To simplify the serialization logic, support for reading the last store file version has been removed. Past this change, a rescan (run automatically) will be required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
if err != nil {
return err
}
cred, err := txr.AddCredit(uint32(outIdx), false)
if err != nil {
return err
}
AcctMgr.ds.ScheduleTxStoreWrite(a)
// Notify frontends of tx. If the tx is unconfirmed, it is always
// notified and the outpoint is marked as notified. If the outpoint
// has already been notified and is now in a block, a txmined notifiction
// should be sent once to let frontends that all previous send/recvs
// for this unconfirmed tx are now confirmed.
Another day, another tx store implementation. The last transaction store was a great example of how not to write scalable software. For a variety of reasons, it was very slow at processing transaction inserts. Among them: 1) Every single transaction record being saved in a linked list (container/list), and inserting into this list would be an O(n) operation so that records could be ordered by receive date. 2) Every single transaction in the above mentioned list was iterated over in order to find double spends which must be removed. It is silly to do this check for mined transactions, which already have been checked for this by btcd. Worse yet, if double spends were found, the list would be iterated a second (or third, or fourth) time for each removed transaction. 3) All spend tracking for signed-by-wallet transactions was found on each transaction insert, even if the now spent previous transaction outputs were known by the caller. This list could keep going on, but you get the idea. It was bad. To resolve these issues a new transaction store had to be implemented. The new implementation: 1) Tracks mined and unmined transactions in different data structures. Mined transactions are cheap to track because the required double spend checks have already been performed by the chain server, and double spend checks are only required to be performed on newly-inserted mined transactions which may conflict with previous unmined transactions. 2) Saves mined transactions grouped by block first, and then by their transaction index. Lookup keys for mined transactions are simply the block height (in the best chain, that's all we save) and index of the transaction in the block. This makes looking up any arbitrary transaction almost an O(1) operation (almost, because block height and block indexes are mapped to their slice indexes with a Go map). 3) Saves records in each transaction for whether the outputs are wallet credits (spendable by wallet) and for whether inputs debit from previous credits. Both structures point back to the source or spender (credits point to the transaction that spends them, or nil for unspent credits, and debits include keys to lookup the transaction credits they spent. While complicated to keep track of, this greatly simplifies the spent tracking for transactions across rollbacks and transaction removals. 4) Implements double spend checking as an almost O(1) operation. A Go map is used to map each previous outpoint for all unconfirmed transactions to the unconfirmed tx record itself. Checking for double spends on confirmed transaction inserts only involves looking up each previous outpoint of the inserted tx in this map. If a double spend is found, removal is simplified by only removing the transaction and its spend chain from store maps, rather than iterating a linked list several times over to remove each dead transaction in the spend chain. 5) Allows the caller to specify the previous credits which are spent by a debiting transaction. When a transaction is created by wallet, the previous outputs are already known, and by passing their record types to the AddDebits method, lookups for each previously unspent credit are omitted. 6) Bookkeeps all blocks with transactions with unspent credits, and bookkeeps the transaction indexes of all transactions with unspent outputs for a single block. For the case where the caller adding a debit record does not know what credits a transaction debits from, these bookkeeping structures allow the store to only consider known unspent transactions, rather than searching through both spent and unspents. 7) Saves amount deltas for the entire balance as a result of each block, due to transactions within that block. This improves the performance of calculating the full balance by not needing to iterate over every transaction, and then every credit, to determine if a credit is spent or unspent. When transactions are moved from unconfirmed to a block structure, the amount deltas are incremented by the amount of all transaction credits (both spent and unspent) and debited by the total amount the transaction spends from previous wallet credits. For the common case of calculating a balance with just one confirmation, the only involves iterating over each block structure and adding the (possibly negative) amount delta. Coinbase rewards are saved similarly, but with a different amount variable so they can be seperatly included or excluded. Due to all of the changes in how the store internally works, the serialization format has changed. To simplify the serialization logic, support for reading the last store file version has been removed. Past this change, a rescan (run automatically) will be required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
op := *cred.OutPoint()
previouslyNotifiedReq := NotifiedRecvTxRequest{
Another day, another tx store implementation. The last transaction store was a great example of how not to write scalable software. For a variety of reasons, it was very slow at processing transaction inserts. Among them: 1) Every single transaction record being saved in a linked list (container/list), and inserting into this list would be an O(n) operation so that records could be ordered by receive date. 2) Every single transaction in the above mentioned list was iterated over in order to find double spends which must be removed. It is silly to do this check for mined transactions, which already have been checked for this by btcd. Worse yet, if double spends were found, the list would be iterated a second (or third, or fourth) time for each removed transaction. 3) All spend tracking for signed-by-wallet transactions was found on each transaction insert, even if the now spent previous transaction outputs were known by the caller. This list could keep going on, but you get the idea. It was bad. To resolve these issues a new transaction store had to be implemented. The new implementation: 1) Tracks mined and unmined transactions in different data structures. Mined transactions are cheap to track because the required double spend checks have already been performed by the chain server, and double spend checks are only required to be performed on newly-inserted mined transactions which may conflict with previous unmined transactions. 2) Saves mined transactions grouped by block first, and then by their transaction index. Lookup keys for mined transactions are simply the block height (in the best chain, that's all we save) and index of the transaction in the block. This makes looking up any arbitrary transaction almost an O(1) operation (almost, because block height and block indexes are mapped to their slice indexes with a Go map). 3) Saves records in each transaction for whether the outputs are wallet credits (spendable by wallet) and for whether inputs debit from previous credits. Both structures point back to the source or spender (credits point to the transaction that spends them, or nil for unspent credits, and debits include keys to lookup the transaction credits they spent. While complicated to keep track of, this greatly simplifies the spent tracking for transactions across rollbacks and transaction removals. 4) Implements double spend checking as an almost O(1) operation. A Go map is used to map each previous outpoint for all unconfirmed transactions to the unconfirmed tx record itself. Checking for double spends on confirmed transaction inserts only involves looking up each previous outpoint of the inserted tx in this map. If a double spend is found, removal is simplified by only removing the transaction and its spend chain from store maps, rather than iterating a linked list several times over to remove each dead transaction in the spend chain. 5) Allows the caller to specify the previous credits which are spent by a debiting transaction. When a transaction is created by wallet, the previous outputs are already known, and by passing their record types to the AddDebits method, lookups for each previously unspent credit are omitted. 6) Bookkeeps all blocks with transactions with unspent credits, and bookkeeps the transaction indexes of all transactions with unspent outputs for a single block. For the case where the caller adding a debit record does not know what credits a transaction debits from, these bookkeeping structures allow the store to only consider known unspent transactions, rather than searching through both spent and unspents. 7) Saves amount deltas for the entire balance as a result of each block, due to transactions within that block. This improves the performance of calculating the full balance by not needing to iterate over every transaction, and then every credit, to determine if a credit is spent or unspent. When transactions are moved from unconfirmed to a block structure, the amount deltas are incremented by the amount of all transaction credits (both spent and unspent) and debited by the total amount the transaction spends from previous wallet credits. For the common case of calculating a balance with just one confirmation, the only involves iterating over each block structure and adding the (possibly negative) amount delta. Coinbase rewards are saved similarly, but with a different amount variable so they can be seperatly included or excluded. Due to all of the changes in how the store internally works, the serialization format has changed. To simplify the serialization logic, support for reading the last store file version has been removed. Past this change, a rescan (run automatically) will be required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
op: op,
response: make(chan NotifiedRecvTxResponse),
}
NotifiedRecvTxChans.access <- previouslyNotifiedReq
if <-previouslyNotifiedReq.response {
Another day, another tx store implementation. The last transaction store was a great example of how not to write scalable software. For a variety of reasons, it was very slow at processing transaction inserts. Among them: 1) Every single transaction record being saved in a linked list (container/list), and inserting into this list would be an O(n) operation so that records could be ordered by receive date. 2) Every single transaction in the above mentioned list was iterated over in order to find double spends which must be removed. It is silly to do this check for mined transactions, which already have been checked for this by btcd. Worse yet, if double spends were found, the list would be iterated a second (or third, or fourth) time for each removed transaction. 3) All spend tracking for signed-by-wallet transactions was found on each transaction insert, even if the now spent previous transaction outputs were known by the caller. This list could keep going on, but you get the idea. It was bad. To resolve these issues a new transaction store had to be implemented. The new implementation: 1) Tracks mined and unmined transactions in different data structures. Mined transactions are cheap to track because the required double spend checks have already been performed by the chain server, and double spend checks are only required to be performed on newly-inserted mined transactions which may conflict with previous unmined transactions. 2) Saves mined transactions grouped by block first, and then by their transaction index. Lookup keys for mined transactions are simply the block height (in the best chain, that's all we save) and index of the transaction in the block. This makes looking up any arbitrary transaction almost an O(1) operation (almost, because block height and block indexes are mapped to their slice indexes with a Go map). 3) Saves records in each transaction for whether the outputs are wallet credits (spendable by wallet) and for whether inputs debit from previous credits. Both structures point back to the source or spender (credits point to the transaction that spends them, or nil for unspent credits, and debits include keys to lookup the transaction credits they spent. While complicated to keep track of, this greatly simplifies the spent tracking for transactions across rollbacks and transaction removals. 4) Implements double spend checking as an almost O(1) operation. A Go map is used to map each previous outpoint for all unconfirmed transactions to the unconfirmed tx record itself. Checking for double spends on confirmed transaction inserts only involves looking up each previous outpoint of the inserted tx in this map. If a double spend is found, removal is simplified by only removing the transaction and its spend chain from store maps, rather than iterating a linked list several times over to remove each dead transaction in the spend chain. 5) Allows the caller to specify the previous credits which are spent by a debiting transaction. When a transaction is created by wallet, the previous outputs are already known, and by passing their record types to the AddDebits method, lookups for each previously unspent credit are omitted. 6) Bookkeeps all blocks with transactions with unspent credits, and bookkeeps the transaction indexes of all transactions with unspent outputs for a single block. For the case where the caller adding a debit record does not know what credits a transaction debits from, these bookkeeping structures allow the store to only consider known unspent transactions, rather than searching through both spent and unspents. 7) Saves amount deltas for the entire balance as a result of each block, due to transactions within that block. This improves the performance of calculating the full balance by not needing to iterate over every transaction, and then every credit, to determine if a credit is spent or unspent. When transactions are moved from unconfirmed to a block structure, the amount deltas are incremented by the amount of all transaction credits (both spent and unspent) and debited by the total amount the transaction spends from previous wallet credits. For the common case of calculating a balance with just one confirmation, the only involves iterating over each block structure and adding the (possibly negative) amount delta. Coinbase rewards are saved similarly, but with a different amount variable so they can be seperatly included or excluded. Due to all of the changes in how the store internally works, the serialization format has changed. To simplify the serialization logic, support for reading the last store file version has been removed. Past this change, a rescan (run automatically) will be required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
NotifiedRecvTxChans.remove <- op
} else {
// Notify frontends of new recv tx and mark as notified.
Another day, another tx store implementation. The last transaction store was a great example of how not to write scalable software. For a variety of reasons, it was very slow at processing transaction inserts. Among them: 1) Every single transaction record being saved in a linked list (container/list), and inserting into this list would be an O(n) operation so that records could be ordered by receive date. 2) Every single transaction in the above mentioned list was iterated over in order to find double spends which must be removed. It is silly to do this check for mined transactions, which already have been checked for this by btcd. Worse yet, if double spends were found, the list would be iterated a second (or third, or fourth) time for each removed transaction. 3) All spend tracking for signed-by-wallet transactions was found on each transaction insert, even if the now spent previous transaction outputs were known by the caller. This list could keep going on, but you get the idea. It was bad. To resolve these issues a new transaction store had to be implemented. The new implementation: 1) Tracks mined and unmined transactions in different data structures. Mined transactions are cheap to track because the required double spend checks have already been performed by the chain server, and double spend checks are only required to be performed on newly-inserted mined transactions which may conflict with previous unmined transactions. 2) Saves mined transactions grouped by block first, and then by their transaction index. Lookup keys for mined transactions are simply the block height (in the best chain, that's all we save) and index of the transaction in the block. This makes looking up any arbitrary transaction almost an O(1) operation (almost, because block height and block indexes are mapped to their slice indexes with a Go map). 3) Saves records in each transaction for whether the outputs are wallet credits (spendable by wallet) and for whether inputs debit from previous credits. Both structures point back to the source or spender (credits point to the transaction that spends them, or nil for unspent credits, and debits include keys to lookup the transaction credits they spent. While complicated to keep track of, this greatly simplifies the spent tracking for transactions across rollbacks and transaction removals. 4) Implements double spend checking as an almost O(1) operation. A Go map is used to map each previous outpoint for all unconfirmed transactions to the unconfirmed tx record itself. Checking for double spends on confirmed transaction inserts only involves looking up each previous outpoint of the inserted tx in this map. If a double spend is found, removal is simplified by only removing the transaction and its spend chain from store maps, rather than iterating a linked list several times over to remove each dead transaction in the spend chain. 5) Allows the caller to specify the previous credits which are spent by a debiting transaction. When a transaction is created by wallet, the previous outputs are already known, and by passing their record types to the AddDebits method, lookups for each previously unspent credit are omitted. 6) Bookkeeps all blocks with transactions with unspent credits, and bookkeeps the transaction indexes of all transactions with unspent outputs for a single block. For the case where the caller adding a debit record does not know what credits a transaction debits from, these bookkeeping structures allow the store to only consider known unspent transactions, rather than searching through both spent and unspents. 7) Saves amount deltas for the entire balance as a result of each block, due to transactions within that block. This improves the performance of calculating the full balance by not needing to iterate over every transaction, and then every credit, to determine if a credit is spent or unspent. When transactions are moved from unconfirmed to a block structure, the amount deltas are incremented by the amount of all transaction credits (both spent and unspent) and debited by the total amount the transaction spends from previous wallet credits. For the common case of calculating a balance with just one confirmation, the only involves iterating over each block structure and adding the (possibly negative) amount delta. Coinbase rewards are saved similarly, but with a different amount variable so they can be seperatly included or excluded. Due to all of the changes in how the store internally works, the serialization format has changed. To simplify the serialization logic, support for reading the last store file version has been removed. Past this change, a rescan (run automatically) will be required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
NotifiedRecvTxChans.add <- op
Another day, another tx store implementation. The last transaction store was a great example of how not to write scalable software. For a variety of reasons, it was very slow at processing transaction inserts. Among them: 1) Every single transaction record being saved in a linked list (container/list), and inserting into this list would be an O(n) operation so that records could be ordered by receive date. 2) Every single transaction in the above mentioned list was iterated over in order to find double spends which must be removed. It is silly to do this check for mined transactions, which already have been checked for this by btcd. Worse yet, if double spends were found, the list would be iterated a second (or third, or fourth) time for each removed transaction. 3) All spend tracking for signed-by-wallet transactions was found on each transaction insert, even if the now spent previous transaction outputs were known by the caller. This list could keep going on, but you get the idea. It was bad. To resolve these issues a new transaction store had to be implemented. The new implementation: 1) Tracks mined and unmined transactions in different data structures. Mined transactions are cheap to track because the required double spend checks have already been performed by the chain server, and double spend checks are only required to be performed on newly-inserted mined transactions which may conflict with previous unmined transactions. 2) Saves mined transactions grouped by block first, and then by their transaction index. Lookup keys for mined transactions are simply the block height (in the best chain, that's all we save) and index of the transaction in the block. This makes looking up any arbitrary transaction almost an O(1) operation (almost, because block height and block indexes are mapped to their slice indexes with a Go map). 3) Saves records in each transaction for whether the outputs are wallet credits (spendable by wallet) and for whether inputs debit from previous credits. Both structures point back to the source or spender (credits point to the transaction that spends them, or nil for unspent credits, and debits include keys to lookup the transaction credits they spent. While complicated to keep track of, this greatly simplifies the spent tracking for transactions across rollbacks and transaction removals. 4) Implements double spend checking as an almost O(1) operation. A Go map is used to map each previous outpoint for all unconfirmed transactions to the unconfirmed tx record itself. Checking for double spends on confirmed transaction inserts only involves looking up each previous outpoint of the inserted tx in this map. If a double spend is found, removal is simplified by only removing the transaction and its spend chain from store maps, rather than iterating a linked list several times over to remove each dead transaction in the spend chain. 5) Allows the caller to specify the previous credits which are spent by a debiting transaction. When a transaction is created by wallet, the previous outputs are already known, and by passing their record types to the AddDebits method, lookups for each previously unspent credit are omitted. 6) Bookkeeps all blocks with transactions with unspent credits, and bookkeeps the transaction indexes of all transactions with unspent outputs for a single block. For the case where the caller adding a debit record does not know what credits a transaction debits from, these bookkeeping structures allow the store to only consider known unspent transactions, rather than searching through both spent and unspents. 7) Saves amount deltas for the entire balance as a result of each block, due to transactions within that block. This improves the performance of calculating the full balance by not needing to iterate over every transaction, and then every credit, to determine if a credit is spent or unspent. When transactions are moved from unconfirmed to a block structure, the amount deltas are incremented by the amount of all transaction credits (both spent and unspent) and debited by the total amount the transaction spends from previous wallet credits. For the common case of calculating a balance with just one confirmation, the only involves iterating over each block structure and adding the (possibly negative) amount delta. Coinbase rewards are saved similarly, but with a different amount variable so they can be seperatly included or excluded. Due to all of the changes in how the store internally works, the serialization format has changed. To simplify the serialization logic, support for reading the last store file version has been removed. Past this change, a rescan (run automatically) will be required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
ltr, err := cred.ToJSON(a.Name(), bs.Height, a.Wallet.Net())
if err != nil {
return err
}
NotifyNewTxDetails(allClients, a.Name(), ltr)
}
// Notify frontends of new account balance.
confirmed := a.CalculateBalance(1)
unconfirmed := a.CalculateBalance(0) - confirmed
NotifyWalletBalance(allClients, a.name, confirmed)
NotifyWalletBalanceUnconfirmed(allClients, a.name, unconfirmed)
}
}
return nil
}
// NtfnBlockConnected handles btcd notifications resulting from newly
// connected blocks to the main blockchain.
//
// TODO(jrick): Send block time with notification. This will be used
// to mark wallet files with a possibly-better earliest block height,
// and will greatly reduce rescan times for wallets created with an
// out of sync btcd.
func NtfnBlockConnected(n btcjson.Cmd) error {
bcn, ok := n.(*btcws.BlockConnectedNtfn)
if !ok {
return fmt.Errorf("%v handler: unexpected type", n.Method())
}
hash, err := btcwire.NewShaHashFromStr(bcn.Hash)
if err != nil {
return fmt.Errorf("%v handler: invalid hash string", n.Method())
}
// Update the blockstamp for the newly-connected block.
bs := &wallet.BlockStamp{
Height: bcn.Height,
Hash: *hash,
}
curBlock.Lock()
curBlock.BlockStamp = *bs
curBlock.Unlock()
// btcd notifies btcwallet about transactions first, and then sends
// the new block notification. New balance notifications for txs
// in blocks are therefore sent here after all tx notifications
// have arrived and finished being processed by the handlers.
workers := NotifyBalanceRequest{
block: *hash,
wg: make(chan *sync.WaitGroup),
}
NotifyBalanceSyncerChans.access <- workers
if wg := <-workers.wg; wg != nil {
wg.Wait()
NotifyBalanceSyncerChans.remove <- *hash
}
AcctMgr.BlockNotify(bs)
// Pass notification to frontends too.
marshaled, _ := n.MarshalJSON()
allClients <- marshaled
return nil
}
// NtfnBlockDisconnected handles btcd notifications resulting from
// blocks disconnected from the main chain in the event of a chain
// switch and notifies frontends of the new blockchain height.
func NtfnBlockDisconnected(n btcjson.Cmd) error {
bdn, ok := n.(*btcws.BlockDisconnectedNtfn)
if !ok {
return fmt.Errorf("%v handler: unexpected type", n.Method())
}
hash, err := btcwire.NewShaHashFromStr(bdn.Hash)
if err != nil {
return fmt.Errorf("%v handler: invalid hash string", n.Method())
}
// Rollback Utxo and Tx data stores.
AcctMgr.Rollback(bdn.Height, hash)
// Pass notification to frontends too.
marshaled, _ := n.MarshalJSON()
allClients <- marshaled
return nil
}
// NtfnRedeemingTx handles btcd redeemingtx notifications resulting from a
// transaction spending a watched outpoint.
func NtfnRedeemingTx(n btcjson.Cmd) error {
cn, ok := n.(*btcws.RedeemingTxNtfn)
if !ok {
return fmt.Errorf("%v handler: unexpected type", n.Method())
}
rawTx, err := hex.DecodeString(cn.HexTx)
if err != nil {
return fmt.Errorf("%v handler: bad hexstring: %v", n.Method(), err)
}
tx, err := btcutil.NewTxFromBytes(rawTx)
if err != nil {
return fmt.Errorf("%v handler: bad transaction bytes: %v", n.Method(), err)
}
Another day, another tx store implementation. The last transaction store was a great example of how not to write scalable software. For a variety of reasons, it was very slow at processing transaction inserts. Among them: 1) Every single transaction record being saved in a linked list (container/list), and inserting into this list would be an O(n) operation so that records could be ordered by receive date. 2) Every single transaction in the above mentioned list was iterated over in order to find double spends which must be removed. It is silly to do this check for mined transactions, which already have been checked for this by btcd. Worse yet, if double spends were found, the list would be iterated a second (or third, or fourth) time for each removed transaction. 3) All spend tracking for signed-by-wallet transactions was found on each transaction insert, even if the now spent previous transaction outputs were known by the caller. This list could keep going on, but you get the idea. It was bad. To resolve these issues a new transaction store had to be implemented. The new implementation: 1) Tracks mined and unmined transactions in different data structures. Mined transactions are cheap to track because the required double spend checks have already been performed by the chain server, and double spend checks are only required to be performed on newly-inserted mined transactions which may conflict with previous unmined transactions. 2) Saves mined transactions grouped by block first, and then by their transaction index. Lookup keys for mined transactions are simply the block height (in the best chain, that's all we save) and index of the transaction in the block. This makes looking up any arbitrary transaction almost an O(1) operation (almost, because block height and block indexes are mapped to their slice indexes with a Go map). 3) Saves records in each transaction for whether the outputs are wallet credits (spendable by wallet) and for whether inputs debit from previous credits. Both structures point back to the source or spender (credits point to the transaction that spends them, or nil for unspent credits, and debits include keys to lookup the transaction credits they spent. While complicated to keep track of, this greatly simplifies the spent tracking for transactions across rollbacks and transaction removals. 4) Implements double spend checking as an almost O(1) operation. A Go map is used to map each previous outpoint for all unconfirmed transactions to the unconfirmed tx record itself. Checking for double spends on confirmed transaction inserts only involves looking up each previous outpoint of the inserted tx in this map. If a double spend is found, removal is simplified by only removing the transaction and its spend chain from store maps, rather than iterating a linked list several times over to remove each dead transaction in the spend chain. 5) Allows the caller to specify the previous credits which are spent by a debiting transaction. When a transaction is created by wallet, the previous outputs are already known, and by passing their record types to the AddDebits method, lookups for each previously unspent credit are omitted. 6) Bookkeeps all blocks with transactions with unspent credits, and bookkeeps the transaction indexes of all transactions with unspent outputs for a single block. For the case where the caller adding a debit record does not know what credits a transaction debits from, these bookkeeping structures allow the store to only consider known unspent transactions, rather than searching through both spent and unspents. 7) Saves amount deltas for the entire balance as a result of each block, due to transactions within that block. This improves the performance of calculating the full balance by not needing to iterate over every transaction, and then every credit, to determine if a credit is spent or unspent. When transactions are moved from unconfirmed to a block structure, the amount deltas are incremented by the amount of all transaction credits (both spent and unspent) and debited by the total amount the transaction spends from previous wallet credits. For the common case of calculating a balance with just one confirmation, the only involves iterating over each block structure and adding the (possibly negative) amount delta. Coinbase rewards are saved similarly, but with a different amount variable so they can be seperatly included or excluded. Due to all of the changes in how the store internally works, the serialization format has changed. To simplify the serialization logic, support for reading the last store file version has been removed. Past this change, a rescan (run automatically) will be required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
block, txIdx, err := parseBlock(cn.Block)
if err != nil {
return fmt.Errorf("%v handler: bad block: %v", n.Method(), err)
}
tx.SetIndex(txIdx)
return AcctMgr.RecordSpendingTx(tx, block)
}
// NtfnRescanProgress handles btcd rescanprogress notifications resulting
// from a partially completed rescan.
func NtfnRescanProgress(n btcjson.Cmd) error {
cn, ok := n.(*btcws.RescanProgressNtfn)
if !ok {
return fmt.Errorf("%v handler: unexpected type", n.Method())
}
// Notify the rescan manager of the completed partial progress for
// the current rescan.
AcctMgr.rm.MarkProgress(cn.LastProcessed)
return nil
}