Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
// Copyright (c) 2013, 2014 Conformal Systems LLC <info@conformal.com>
|
|
|
|
//
|
|
|
|
// Permission to use, copy, modify, and distribute this software for any
|
|
|
|
// purpose with or without fee is hereby granted, provided that the above
|
|
|
|
// copyright notice and this permission notice appear in all copies.
|
|
|
|
//
|
|
|
|
// THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
|
|
|
|
// WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
|
|
|
|
// MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
|
|
|
|
// ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
|
|
|
|
// WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
|
|
|
|
// ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
|
|
|
|
// OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
|
2013-08-26 16:48:42 +02:00
|
|
|
|
2014-05-08 21:48:42 +02:00
|
|
|
package txstore_test
|
2013-08-26 16:48:42 +02:00
|
|
|
|
|
|
|
import (
|
|
|
|
"bytes"
|
2014-02-24 20:35:30 +01:00
|
|
|
"encoding/hex"
|
2013-08-26 16:48:42 +02:00
|
|
|
"testing"
|
2014-02-24 20:35:30 +01:00
|
|
|
"time"
|
|
|
|
|
2014-05-27 19:50:51 +02:00
|
|
|
"github.com/conformal/btcnet"
|
2014-02-24 20:35:30 +01:00
|
|
|
"github.com/conformal/btcutil"
|
2014-05-08 21:48:42 +02:00
|
|
|
. "github.com/conformal/btcwallet/txstore"
|
2014-02-24 20:35:30 +01:00
|
|
|
"github.com/conformal/btcwire"
|
2013-08-26 16:48:42 +02:00
|
|
|
)
|
|
|
|
|
2014-02-24 20:35:30 +01:00
|
|
|
// Received transaction output for mainnet outpoint
|
|
|
|
// 61d3696de4c888730cbe06b0ad8ecb6d72d6108e893895aa9bc067bd7eba3fad:0
|
2013-08-26 16:48:42 +02:00
|
|
|
var (
|
2014-02-24 20:35:30 +01:00
|
|
|
TstRecvSerializedTx, _ = hex.DecodeString("010000000114d9ff358894c486b4ae11c2a8cf7851b1df64c53d2e511278eff17c22fb7373000000008c493046022100995447baec31ee9f6d4ec0e05cb2a44f6b817a99d5f6de167d1c75354a946410022100c9ffc23b64d770b0e01e7ff4d25fbc2f1ca8091053078a247905c39fce3760b601410458b8e267add3c1e374cf40f1de02b59213a82e1d84c2b94096e22e2f09387009c96debe1d0bcb2356ffdcf65d2a83d4b34e72c62eccd8490dbf2110167783b2bffffffff0280969800000000001976a914479ed307831d0ac19ebc5f63de7d5f1a430ddb9d88ac38bfaa00000000001976a914dadf9e3484f28b385ddeaa6c575c0c0d18e9788a88ac00000000")
|
|
|
|
TstRecvTx, _ = btcutil.NewTxFromBytes(TstRecvSerializedTx)
|
|
|
|
TstRecvTxSpendingTxBlockHash, _ = btcwire.NewShaHashFromStr("00000000000000017188b968a371bab95aa43522665353b646e41865abae02a4")
|
|
|
|
TstRecvAmt = int64(10000000)
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
TstRecvIndex = 684
|
|
|
|
TstRecvTxBlockDetails = &Block{
|
2014-02-24 20:35:30 +01:00
|
|
|
Height: 276425,
|
|
|
|
Hash: *TstRecvTxSpendingTxBlockHash,
|
|
|
|
Time: time.Unix(1387737310, 0),
|
|
|
|
}
|
|
|
|
|
|
|
|
TstRecvCurrentHeight = int32(284498) // mainnet blockchain height at time of writing
|
|
|
|
TstRecvTxOutConfirms = 8074 // hardcoded number of confirmations given the above block height
|
|
|
|
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
TstSpendingSerializedTx, _ = hex.DecodeString("0100000003ad3fba7ebd67c09baa9538898e10d6726dcb8eadb006be0c7388c8e46d69d361000000006b4830450220702c4fbde5532575fed44f8d6e8c3432a2a9bd8cff2f966c3a79b2245a7c88db02210095d6505a57e350720cb52b89a9b56243c15ddfcea0596aedc1ba55d9fb7d5aa0012103cccb5c48a699d3efcca6dae277fee6b82e0229ed754b742659c3acdfed2651f9ffffffffdbd36173f5610e34de5c00ed092174603761595d90190f790e79cda3e5b45bc2010000006b483045022000fa20735e5875e64d05bed43d81b867f3bd8745008d3ff4331ef1617eac7c44022100ad82261fc57faac67fc482a37b6bf18158da0971e300abf5fe2f9fd39e107f58012102d4e1caf3e022757512c204bf09ff56a9981df483aba3c74bb60d3612077c9206ffffffff65536c9d964b6f89b8ef17e83c6666641bc495cb27bab60052f76cd4556ccd0d040000006a473044022068e3886e0299ffa69a1c3ee40f8b6700f5f6d463a9cf9dbf22c055a131fc4abc02202b58957fe19ff1be7a84c458d08016c53fbddec7184ac5e633f2b282ae3420ae012103b4e411b81d32a69fb81178a8ea1abaa12f613336923ee920ffbb1b313af1f4d2ffffffff02ab233200000000001976a91418808b2fbd8d2c6d022aed5cd61f0ce6c0a4cbb688ac4741f011000000001976a914f081088a300c80ce36b717a9914ab5ec8a7d283988ac00000000")
|
|
|
|
TstSpendingTx, _ = btcutil.NewTxFromBytes(TstSpendingSerializedTx)
|
|
|
|
TstSpendingTxBlockHeight = int32(279143)
|
|
|
|
TstSignedTxBlockHash, _ = btcwire.NewShaHashFromStr("00000000000000017188b968a371bab95aa43522665353b646e41865abae02a4")
|
|
|
|
TstSignedTxIndex = 123
|
|
|
|
TstSignedTxBlockDetails = &Block{
|
2014-02-24 20:35:30 +01:00
|
|
|
Height: TstSpendingTxBlockHeight,
|
|
|
|
Hash: *TstSignedTxBlockHash,
|
|
|
|
Time: time.Unix(1389114091, 0),
|
|
|
|
}
|
|
|
|
)
|
|
|
|
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
func TestInsertsCreditsDebitsRollbacks(t *testing.T) {
|
2014-02-24 20:35:30 +01:00
|
|
|
// Create a double spend of the received blockchain transaction.
|
|
|
|
dupRecvTx, _ := btcutil.NewTxFromBytes(TstRecvSerializedTx)
|
|
|
|
// Switch txout amount to 1 BTC. Transaction store doesn't
|
|
|
|
// validate txs, so this is fine for testing a double spend
|
|
|
|
// removal.
|
|
|
|
TstDupRecvAmount := int64(1e8)
|
|
|
|
newDupMsgTx := dupRecvTx.MsgTx()
|
|
|
|
newDupMsgTx.TxOut[0].Value = TstDupRecvAmount
|
|
|
|
TstDoubleSpendTx := btcutil.NewTx(newDupMsgTx)
|
|
|
|
|
|
|
|
// Create a "signed" (with invalid sigs) tx that spends output 0 of
|
|
|
|
// the double spend.
|
|
|
|
spendingTx := btcwire.NewMsgTx()
|
|
|
|
spendingTxIn := btcwire.NewTxIn(btcwire.NewOutPoint(TstDoubleSpendTx.Sha(), 0), []byte{0, 1, 2, 3, 4})
|
|
|
|
spendingTx.AddTxIn(spendingTxIn)
|
|
|
|
spendingTxOut1 := btcwire.NewTxOut(1e7, []byte{5, 6, 7, 8, 9})
|
|
|
|
spendingTxOut2 := btcwire.NewTxOut(9e7, []byte{10, 11, 12, 13, 14})
|
|
|
|
spendingTx.AddTxOut(spendingTxOut1)
|
|
|
|
spendingTx.AddTxOut(spendingTxOut2)
|
|
|
|
TstSpendingTx := btcutil.NewTx(spendingTx)
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
var _ = TstSpendingTx
|
2014-02-24 20:35:30 +01:00
|
|
|
|
|
|
|
tests := []struct {
|
|
|
|
name string
|
2014-02-28 19:03:23 +01:00
|
|
|
f func(*Store) (*Store, error)
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
bal, unc btcutil.Amount
|
2014-02-24 20:35:30 +01:00
|
|
|
unspents map[btcwire.OutPoint]struct{}
|
|
|
|
unmined map[btcwire.ShaHash]struct{}
|
|
|
|
}{
|
|
|
|
{
|
|
|
|
name: "new store",
|
2014-02-28 19:03:23 +01:00
|
|
|
f: func(_ *Store) (*Store, error) {
|
2014-05-08 21:48:42 +02:00
|
|
|
return New(), nil
|
2014-02-24 20:35:30 +01:00
|
|
|
},
|
|
|
|
bal: 0,
|
|
|
|
unc: 0,
|
|
|
|
unspents: map[btcwire.OutPoint]struct{}{},
|
|
|
|
unmined: map[btcwire.ShaHash]struct{}{},
|
2013-08-26 19:34:18 +02:00
|
|
|
},
|
2014-02-24 20:35:30 +01:00
|
|
|
{
|
|
|
|
name: "txout insert",
|
2014-02-28 19:03:23 +01:00
|
|
|
f: func(s *Store) (*Store, error) {
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
r, err := s.InsertTx(TstRecvTx, nil)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
|
|
|
_, err = r.AddCredit(0, false)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
// Verify that we can create the JSON output without any
|
|
|
|
// errors.
|
2014-05-27 19:50:51 +02:00
|
|
|
_, err = r.ToJSON("", 100, &btcnet.MainNetParams)
|
2014-02-28 19:03:23 +01:00
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
return s, nil
|
|
|
|
},
|
2014-02-24 20:35:30 +01:00
|
|
|
bal: 0,
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
unc: btcutil.Amount(TstRecvTx.MsgTx().TxOut[0].Value),
|
2014-02-24 20:35:30 +01:00
|
|
|
unspents: map[btcwire.OutPoint]struct{}{
|
2014-04-11 20:52:50 +02:00
|
|
|
*btcwire.NewOutPoint(TstRecvTx.Sha(), 0): {},
|
2014-02-24 20:35:30 +01:00
|
|
|
},
|
|
|
|
unmined: map[btcwire.ShaHash]struct{}{},
|
2013-08-26 19:34:18 +02:00
|
|
|
},
|
2014-02-24 20:35:30 +01:00
|
|
|
{
|
2014-02-28 19:03:23 +01:00
|
|
|
name: "insert duplicate unconfirmed",
|
|
|
|
f: func(s *Store) (*Store, error) {
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
r, err := s.InsertTx(TstRecvTx, nil)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
|
|
|
_, err = r.AddCredit(0, false)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
2014-05-27 19:50:51 +02:00
|
|
|
_, err = r.ToJSON("", 100, &btcnet.MainNetParams)
|
2014-02-28 19:03:23 +01:00
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
return s, nil
|
|
|
|
},
|
|
|
|
bal: 0,
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
unc: btcutil.Amount(TstRecvTx.MsgTx().TxOut[0].Value),
|
2014-02-24 20:35:30 +01:00
|
|
|
unspents: map[btcwire.OutPoint]struct{}{
|
2014-04-11 20:52:50 +02:00
|
|
|
*btcwire.NewOutPoint(TstRecvTx.Sha(), 0): {},
|
2014-02-24 20:35:30 +01:00
|
|
|
},
|
|
|
|
unmined: map[btcwire.ShaHash]struct{}{},
|
2013-08-26 19:34:18 +02:00
|
|
|
},
|
2014-02-24 20:35:30 +01:00
|
|
|
{
|
2014-02-28 19:03:23 +01:00
|
|
|
name: "confirmed txout insert",
|
|
|
|
f: func(s *Store) (*Store, error) {
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
TstRecvTx.SetIndex(TstRecvIndex)
|
|
|
|
r, err := s.InsertTx(TstRecvTx, TstRecvTxBlockDetails)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
|
|
|
_, err = r.AddCredit(0, false)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
2014-05-27 19:50:51 +02:00
|
|
|
_, err = r.ToJSON("", 100, &btcnet.MainNetParams)
|
2014-02-28 19:03:23 +01:00
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
return s, nil
|
|
|
|
},
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
bal: btcutil.Amount(TstRecvTx.MsgTx().TxOut[0].Value),
|
2014-02-24 20:35:30 +01:00
|
|
|
unc: 0,
|
|
|
|
unspents: map[btcwire.OutPoint]struct{}{
|
2014-04-11 20:52:50 +02:00
|
|
|
*btcwire.NewOutPoint(TstRecvTx.Sha(), 0): {},
|
2014-02-24 20:35:30 +01:00
|
|
|
},
|
|
|
|
unmined: map[btcwire.ShaHash]struct{}{},
|
2013-08-26 19:34:18 +02:00
|
|
|
},
|
2014-02-24 20:35:30 +01:00
|
|
|
{
|
2014-02-28 19:03:23 +01:00
|
|
|
name: "insert duplicate confirmed",
|
|
|
|
f: func(s *Store) (*Store, error) {
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
TstRecvTx.SetIndex(TstRecvIndex)
|
|
|
|
r, err := s.InsertTx(TstRecvTx, TstRecvTxBlockDetails)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
|
|
|
_, err = r.AddCredit(0, false)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
2014-05-27 19:50:51 +02:00
|
|
|
_, err = r.ToJSON("", 100, &btcnet.MainNetParams)
|
2014-02-28 19:03:23 +01:00
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
return s, nil
|
|
|
|
},
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
bal: btcutil.Amount(TstRecvTx.MsgTx().TxOut[0].Value),
|
2014-02-24 20:35:30 +01:00
|
|
|
unc: 0,
|
|
|
|
unspents: map[btcwire.OutPoint]struct{}{
|
2014-04-11 20:52:50 +02:00
|
|
|
*btcwire.NewOutPoint(TstRecvTx.Sha(), 0): {},
|
2014-02-24 20:35:30 +01:00
|
|
|
},
|
|
|
|
unmined: map[btcwire.ShaHash]struct{}{},
|
2013-08-26 19:34:18 +02:00
|
|
|
},
|
2014-02-28 19:03:23 +01:00
|
|
|
{
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
name: "rollback confirmed credit",
|
2014-02-28 19:03:23 +01:00
|
|
|
f: func(s *Store) (*Store, error) {
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
err := s.Rollback(TstRecvTxBlockDetails.Height)
|
2014-02-28 19:03:23 +01:00
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
return s, nil
|
|
|
|
},
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
bal: 0,
|
|
|
|
unc: btcutil.Amount(TstRecvTx.MsgTx().TxOut[0].Value),
|
|
|
|
unspents: map[btcwire.OutPoint]struct{}{
|
|
|
|
*btcwire.NewOutPoint(TstRecvTx.Sha(), 0): {},
|
|
|
|
},
|
|
|
|
unmined: map[btcwire.ShaHash]struct{}{},
|
2014-02-28 19:03:23 +01:00
|
|
|
},
|
2014-02-24 20:35:30 +01:00
|
|
|
{
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
name: "insert confirmed double spend",
|
2014-02-28 19:03:23 +01:00
|
|
|
f: func(s *Store) (*Store, error) {
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
TstDoubleSpendTx.SetIndex(TstRecvIndex)
|
|
|
|
r, err := s.InsertTx(TstDoubleSpendTx, TstRecvTxBlockDetails)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
|
|
|
_, err = r.AddCredit(0, false)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
2014-05-27 19:50:51 +02:00
|
|
|
_, err = r.ToJSON("", 100, &btcnet.MainNetParams)
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
|
2014-02-28 19:03:23 +01:00
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
return s, nil
|
|
|
|
},
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
bal: btcutil.Amount(TstDoubleSpendTx.MsgTx().TxOut[0].Value),
|
2014-02-24 20:35:30 +01:00
|
|
|
unc: 0,
|
|
|
|
unspents: map[btcwire.OutPoint]struct{}{
|
2014-04-11 20:52:50 +02:00
|
|
|
*btcwire.NewOutPoint(TstDoubleSpendTx.Sha(), 0): {},
|
2014-02-24 20:35:30 +01:00
|
|
|
},
|
|
|
|
unmined: map[btcwire.ShaHash]struct{}{},
|
2013-08-26 19:34:18 +02:00
|
|
|
},
|
2014-02-24 20:35:30 +01:00
|
|
|
{
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
name: "insert unconfirmed debit",
|
2014-02-28 19:03:23 +01:00
|
|
|
f: func(s *Store) (*Store, error) {
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
prev, err := s.InsertTx(TstDoubleSpendTx, TstRecvTxBlockDetails)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
|
|
|
r, err := s.InsertTx(TstSpendingTx, nil)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
|
|
|
_, err = r.AddDebits(prev.Credits())
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
2014-05-27 19:50:51 +02:00
|
|
|
_, err = r.ToJSON("", 100, &btcnet.MainNetParams)
|
2014-02-28 19:03:23 +01:00
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
return s, nil
|
|
|
|
},
|
2014-02-24 20:35:30 +01:00
|
|
|
bal: 0,
|
|
|
|
unc: 0,
|
|
|
|
unspents: map[btcwire.OutPoint]struct{}{},
|
|
|
|
unmined: map[btcwire.ShaHash]struct{}{
|
2014-04-11 20:52:50 +02:00
|
|
|
*TstSpendingTx.Sha(): {},
|
2014-02-24 20:35:30 +01:00
|
|
|
},
|
2013-09-04 15:32:14 +02:00
|
|
|
},
|
2014-02-24 20:35:30 +01:00
|
|
|
{
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
name: "insert unconfirmed debit again",
|
2014-02-28 19:03:23 +01:00
|
|
|
f: func(s *Store) (*Store, error) {
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
prev, err := s.InsertTx(TstDoubleSpendTx, TstRecvTxBlockDetails)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
|
|
|
r, err := s.InsertTx(TstSpendingTx, nil)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
|
|
|
_, err = r.AddDebits(prev.Credits())
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
2014-05-27 19:50:51 +02:00
|
|
|
_, err = r.ToJSON("", 100, &btcnet.MainNetParams)
|
2014-02-28 19:03:23 +01:00
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
return s, nil
|
|
|
|
},
|
2014-02-24 20:35:30 +01:00
|
|
|
bal: 0,
|
|
|
|
unc: 0,
|
|
|
|
unspents: map[btcwire.OutPoint]struct{}{},
|
|
|
|
unmined: map[btcwire.ShaHash]struct{}{
|
2014-04-11 20:52:50 +02:00
|
|
|
*TstSpendingTx.Sha(): {},
|
2013-09-04 22:16:20 +02:00
|
|
|
},
|
2013-08-26 16:48:42 +02:00
|
|
|
},
|
2014-02-24 20:35:30 +01:00
|
|
|
{
|
|
|
|
name: "insert change (index 0)",
|
2014-02-28 19:03:23 +01:00
|
|
|
f: func(s *Store) (*Store, error) {
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
r, err := s.InsertTx(TstSpendingTx, nil)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
|
|
|
_, err = r.AddCredit(0, true)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
2014-05-27 19:50:51 +02:00
|
|
|
_, err = r.ToJSON("", 100, &btcnet.MainNetParams)
|
2014-02-28 19:03:23 +01:00
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
return s, nil
|
|
|
|
},
|
2014-02-24 20:35:30 +01:00
|
|
|
bal: 0,
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
unc: btcutil.Amount(TstSpendingTx.MsgTx().TxOut[0].Value),
|
2014-02-24 20:35:30 +01:00
|
|
|
unspents: map[btcwire.OutPoint]struct{}{
|
2014-04-11 20:52:50 +02:00
|
|
|
*btcwire.NewOutPoint(TstSpendingTx.Sha(), 0): {},
|
2014-02-24 20:35:30 +01:00
|
|
|
},
|
|
|
|
unmined: map[btcwire.ShaHash]struct{}{
|
2014-04-11 20:52:50 +02:00
|
|
|
*TstSpendingTx.Sha(): {},
|
2014-02-24 20:35:30 +01:00
|
|
|
},
|
|
|
|
},
|
|
|
|
{
|
|
|
|
name: "insert output back to this own wallet (index 1)",
|
2014-02-28 19:03:23 +01:00
|
|
|
f: func(s *Store) (*Store, error) {
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
r, err := s.InsertTx(TstSpendingTx, nil)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
|
|
|
_, err = r.AddCredit(1, true)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
2014-05-27 19:50:51 +02:00
|
|
|
_, err = r.ToJSON("", 100, &btcnet.MainNetParams)
|
2014-02-28 19:03:23 +01:00
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
return s, nil
|
|
|
|
},
|
2014-02-24 20:35:30 +01:00
|
|
|
bal: 0,
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
unc: btcutil.Amount(TstSpendingTx.MsgTx().TxOut[0].Value + TstSpendingTx.MsgTx().TxOut[1].Value),
|
2014-02-24 20:35:30 +01:00
|
|
|
unspents: map[btcwire.OutPoint]struct{}{
|
2014-04-11 20:52:50 +02:00
|
|
|
*btcwire.NewOutPoint(TstSpendingTx.Sha(), 0): {},
|
|
|
|
*btcwire.NewOutPoint(TstSpendingTx.Sha(), 1): {},
|
2014-02-24 20:35:30 +01:00
|
|
|
},
|
|
|
|
unmined: map[btcwire.ShaHash]struct{}{
|
2014-04-11 20:52:50 +02:00
|
|
|
*TstSpendingTx.Sha(): {},
|
2014-02-24 20:35:30 +01:00
|
|
|
},
|
|
|
|
},
|
|
|
|
{
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
name: "confirm signed tx",
|
2014-02-28 19:03:23 +01:00
|
|
|
f: func(s *Store) (*Store, error) {
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
TstSpendingTx.SetIndex(TstSignedTxIndex)
|
|
|
|
r, err := s.InsertTx(TstSpendingTx, TstSignedTxBlockDetails)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
2014-05-27 19:50:51 +02:00
|
|
|
_, err = r.ToJSON("", 100, &btcnet.MainNetParams)
|
2014-02-28 19:03:23 +01:00
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
return s, nil
|
|
|
|
},
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
bal: btcutil.Amount(TstSpendingTx.MsgTx().TxOut[0].Value + TstSpendingTx.MsgTx().TxOut[1].Value),
|
2014-02-24 20:35:30 +01:00
|
|
|
unc: 0,
|
|
|
|
unspents: map[btcwire.OutPoint]struct{}{
|
2014-04-11 20:52:50 +02:00
|
|
|
*btcwire.NewOutPoint(TstSpendingTx.Sha(), 0): {},
|
|
|
|
*btcwire.NewOutPoint(TstSpendingTx.Sha(), 1): {},
|
2014-02-24 20:35:30 +01:00
|
|
|
},
|
|
|
|
unmined: map[btcwire.ShaHash]struct{}{},
|
|
|
|
},
|
|
|
|
{
|
|
|
|
name: "rollback after spending tx",
|
2014-02-28 19:03:23 +01:00
|
|
|
f: func(s *Store) (*Store, error) {
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
err := s.Rollback(TstSignedTxBlockDetails.Height + 1)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
2014-02-28 19:03:23 +01:00
|
|
|
return s, nil
|
2014-02-24 20:35:30 +01:00
|
|
|
},
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
bal: btcutil.Amount(TstSpendingTx.MsgTx().TxOut[0].Value + TstSpendingTx.MsgTx().TxOut[1].Value),
|
2014-02-24 20:35:30 +01:00
|
|
|
unc: 0,
|
|
|
|
unspents: map[btcwire.OutPoint]struct{}{
|
2014-04-11 20:52:50 +02:00
|
|
|
*btcwire.NewOutPoint(TstSpendingTx.Sha(), 0): {},
|
|
|
|
*btcwire.NewOutPoint(TstSpendingTx.Sha(), 1): {},
|
2014-02-24 20:35:30 +01:00
|
|
|
},
|
|
|
|
unmined: map[btcwire.ShaHash]struct{}{},
|
|
|
|
},
|
|
|
|
{
|
|
|
|
name: "rollback spending tx block",
|
2014-02-28 19:03:23 +01:00
|
|
|
f: func(s *Store) (*Store, error) {
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
err := s.Rollback(TstSignedTxBlockDetails.Height)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
2014-02-28 19:03:23 +01:00
|
|
|
return s, nil
|
2014-02-24 20:35:30 +01:00
|
|
|
},
|
|
|
|
bal: 0,
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
unc: btcutil.Amount(TstSpendingTx.MsgTx().TxOut[0].Value + TstSpendingTx.MsgTx().TxOut[1].Value),
|
2014-02-24 20:35:30 +01:00
|
|
|
unspents: map[btcwire.OutPoint]struct{}{
|
2014-04-11 20:52:50 +02:00
|
|
|
*btcwire.NewOutPoint(TstSpendingTx.Sha(), 0): {},
|
|
|
|
*btcwire.NewOutPoint(TstSpendingTx.Sha(), 1): {},
|
2014-02-24 20:35:30 +01:00
|
|
|
},
|
|
|
|
unmined: map[btcwire.ShaHash]struct{}{
|
2014-04-11 20:52:50 +02:00
|
|
|
*TstSpendingTx.Sha(): {},
|
2014-02-24 20:35:30 +01:00
|
|
|
},
|
|
|
|
},
|
|
|
|
{
|
|
|
|
name: "rollback double spend tx block",
|
2014-02-28 19:03:23 +01:00
|
|
|
f: func(s *Store) (*Store, error) {
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
err := s.Rollback(TstRecvTxBlockDetails.Height)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
2014-02-28 19:03:23 +01:00
|
|
|
return s, nil
|
2014-02-24 20:35:30 +01:00
|
|
|
},
|
|
|
|
bal: 0,
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
unc: btcutil.Amount(TstSpendingTx.MsgTx().TxOut[0].Value + TstSpendingTx.MsgTx().TxOut[1].Value),
|
2014-02-24 20:35:30 +01:00
|
|
|
unspents: map[btcwire.OutPoint]struct{}{
|
2014-04-11 20:52:50 +02:00
|
|
|
*btcwire.NewOutPoint(TstSpendingTx.Sha(), 0): {},
|
|
|
|
*btcwire.NewOutPoint(TstSpendingTx.Sha(), 1): {},
|
2014-02-24 20:35:30 +01:00
|
|
|
},
|
|
|
|
unmined: map[btcwire.ShaHash]struct{}{
|
2014-04-11 20:52:50 +02:00
|
|
|
*TstSpendingTx.Sha(): {},
|
2014-02-24 20:35:30 +01:00
|
|
|
},
|
|
|
|
},
|
|
|
|
{
|
|
|
|
name: "insert original recv txout",
|
2014-02-28 19:03:23 +01:00
|
|
|
f: func(s *Store) (*Store, error) {
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
TstRecvTx.SetIndex(TstRecvIndex)
|
|
|
|
r, err := s.InsertTx(TstRecvTx, TstRecvTxBlockDetails)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
|
|
|
_, err = r.AddCredit(0, false)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
2014-05-27 19:50:51 +02:00
|
|
|
_, err = r.ToJSON("", 100, &btcnet.MainNetParams)
|
2014-02-28 19:03:23 +01:00
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
return s, nil
|
|
|
|
},
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
bal: btcutil.Amount(TstRecvTx.MsgTx().TxOut[0].Value),
|
2014-02-24 20:35:30 +01:00
|
|
|
unc: 0,
|
|
|
|
unspents: map[btcwire.OutPoint]struct{}{
|
2014-04-11 20:52:50 +02:00
|
|
|
*btcwire.NewOutPoint(TstRecvTx.Sha(), 0): {},
|
2014-02-24 20:35:30 +01:00
|
|
|
},
|
|
|
|
unmined: map[btcwire.ShaHash]struct{}{},
|
|
|
|
},
|
2013-08-26 16:48:42 +02:00
|
|
|
}
|
|
|
|
|
2014-02-24 20:35:30 +01:00
|
|
|
var s *Store
|
|
|
|
for _, test := range tests {
|
2014-02-28 19:03:23 +01:00
|
|
|
tmpStore, err := test.f(s)
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
if err != nil {
|
|
|
|
t.Fatalf("%s: got error: %v", test.name, err)
|
2014-02-28 19:03:23 +01:00
|
|
|
}
|
|
|
|
s = tmpStore
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
bal, err := s.Balance(1, TstRecvCurrentHeight)
|
|
|
|
if err != nil {
|
|
|
|
t.Fatalf("%s: Confirmed Balance() failed: %v", test.name, err)
|
|
|
|
}
|
2014-02-24 20:35:30 +01:00
|
|
|
if bal != test.bal {
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
t.Fatalf("%s: balance mismatch: expected: %d, got: %d", test.name, test.bal, bal)
|
|
|
|
}
|
|
|
|
unc, err := s.Balance(0, TstRecvCurrentHeight)
|
|
|
|
if err != nil {
|
|
|
|
t.Fatalf("%s: Unconfirmed Balance() failed: %v", test.name, err)
|
2013-08-26 16:48:42 +02:00
|
|
|
}
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
unc -= bal
|
2014-02-24 20:35:30 +01:00
|
|
|
if unc != test.unc {
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
t.Errorf("%s: unconfirmed balance mismatch: expected %d, got %d", test.name, test.unc, unc)
|
2013-11-22 19:42:25 +01:00
|
|
|
}
|
2013-08-26 16:48:42 +02:00
|
|
|
|
2014-02-24 20:35:30 +01:00
|
|
|
// Check that unspent outputs match expected.
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
unspent, err := s.UnspentOutputs()
|
|
|
|
if err != nil {
|
|
|
|
t.Fatal(err)
|
|
|
|
}
|
|
|
|
for _, r := range unspent {
|
|
|
|
if r.Spent() {
|
2014-02-24 20:35:30 +01:00
|
|
|
t.Errorf("%s: unspent record marked as spent", test.name)
|
|
|
|
}
|
2013-08-26 16:48:42 +02:00
|
|
|
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
op := *r.OutPoint()
|
2014-02-24 20:35:30 +01:00
|
|
|
if _, ok := test.unspents[op]; !ok {
|
|
|
|
t.Errorf("%s: unexpected unspent output: %v", test.name, op)
|
|
|
|
}
|
|
|
|
delete(test.unspents, op)
|
|
|
|
}
|
|
|
|
if len(test.unspents) != 0 {
|
|
|
|
t.Errorf("%s: missing expected unspent output(s)", test.name)
|
|
|
|
}
|
2013-08-26 19:34:18 +02:00
|
|
|
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
// Check that unmined sent txs match expected.
|
|
|
|
for _, tx := range s.UnminedDebitTxs() {
|
2014-02-24 20:35:30 +01:00
|
|
|
if _, ok := test.unmined[*tx.Sha()]; !ok {
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
t.Fatalf("%s: unexpected unmined signed tx: %v", test.name, *tx.Sha())
|
2014-02-24 20:35:30 +01:00
|
|
|
}
|
|
|
|
delete(test.unmined, *tx.Sha())
|
|
|
|
}
|
|
|
|
if len(test.unmined) != 0 {
|
|
|
|
t.Errorf("%s: missing expected unmined signed tx(s)", test.name)
|
|
|
|
}
|
2013-08-26 19:34:18 +02:00
|
|
|
|
2014-02-24 20:35:30 +01:00
|
|
|
// Pass a re-serialized version of the store to each next test.
|
|
|
|
buf := new(bytes.Buffer)
|
|
|
|
nWritten, err := s.WriteTo(buf)
|
|
|
|
if err != nil {
|
|
|
|
t.Fatalf("%v: serialization failed: %v (wrote %v bytes)", test.name, err, nWritten)
|
|
|
|
}
|
|
|
|
if nWritten != int64(buf.Len()) {
|
|
|
|
t.Errorf("%v: wrote %v bytes but buffer has %v", test.name, nWritten, buf.Len())
|
|
|
|
}
|
|
|
|
nRead, err := s.ReadFrom(buf)
|
|
|
|
if err != nil {
|
|
|
|
t.Fatalf("%v: deserialization failed: %v (read %v bytes after writing %v)",
|
|
|
|
test.name, err, nRead, nWritten)
|
|
|
|
}
|
|
|
|
if nWritten != nRead {
|
|
|
|
t.Errorf("%v: number of bytes written (%v) does not match those read (%v)",
|
|
|
|
test.name, nWritten, nRead)
|
|
|
|
}
|
2013-08-26 19:34:18 +02:00
|
|
|
}
|
|
|
|
}
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
|
|
|
|
func TestFindingSpentCredits(t *testing.T) {
|
2014-05-08 21:48:42 +02:00
|
|
|
s := New()
|
Another day, another tx store implementation.
The last transaction store was a great example of how not to write
scalable software. For a variety of reasons, it was very slow at
processing transaction inserts. Among them:
1) Every single transaction record being saved in a linked list
(container/list), and inserting into this list would be an O(n)
operation so that records could be ordered by receive date.
2) Every single transaction in the above mentioned list was iterated
over in order to find double spends which must be removed. It is
silly to do this check for mined transactions, which already have
been checked for this by btcd. Worse yet, if double spends were
found, the list would be iterated a second (or third, or fourth)
time for each removed transaction.
3) All spend tracking for signed-by-wallet transactions was found on
each transaction insert, even if the now spent previous transaction
outputs were known by the caller.
This list could keep going on, but you get the idea. It was bad.
To resolve these issues a new transaction store had to be implemented.
The new implementation:
1) Tracks mined and unmined transactions in different data structures.
Mined transactions are cheap to track because the required double
spend checks have already been performed by the chain server, and
double spend checks are only required to be performed on
newly-inserted mined transactions which may conflict with previous
unmined transactions.
2) Saves mined transactions grouped by block first, and then by their
transaction index. Lookup keys for mined transactions are simply
the block height (in the best chain, that's all we save) and index
of the transaction in the block. This makes looking up any
arbitrary transaction almost an O(1) operation (almost, because
block height and block indexes are mapped to their slice indexes
with a Go map).
3) Saves records in each transaction for whether the outputs are
wallet credits (spendable by wallet) and for whether inputs debit
from previous credits. Both structures point back to the source
or spender (credits point to the transaction that spends them, or
nil for unspent credits, and debits include keys to lookup the
transaction credits they spent. While complicated to keep track
of, this greatly simplifies the spent tracking for transactions
across rollbacks and transaction removals.
4) Implements double spend checking as an almost O(1) operation. A
Go map is used to map each previous outpoint for all unconfirmed
transactions to the unconfirmed tx record itself. Checking for
double spends on confirmed transaction inserts only involves
looking up each previous outpoint of the inserted tx in this map.
If a double spend is found, removal is simplified by only
removing the transaction and its spend chain from store maps,
rather than iterating a linked list several times over to remove
each dead transaction in the spend chain.
5) Allows the caller to specify the previous credits which are spent
by a debiting transaction. When a transaction is created by
wallet, the previous outputs are already known, and by passing
their record types to the AddDebits method, lookups for each
previously unspent credit are omitted.
6) Bookkeeps all blocks with transactions with unspent credits, and
bookkeeps the transaction indexes of all transactions with unspent
outputs for a single block. For the case where the caller adding a
debit record does not know what credits a transaction debits from,
these bookkeeping structures allow the store to only consider known
unspent transactions, rather than searching through both spent and
unspents.
7) Saves amount deltas for the entire balance as a result of each
block, due to transactions within that block. This improves the
performance of calculating the full balance by not needing to
iterate over every transaction, and then every credit, to determine
if a credit is spent or unspent. When transactions are moved from
unconfirmed to a block structure, the amount deltas are incremented
by the amount of all transaction credits (both spent and unspent)
and debited by the total amount the transaction spends from
previous wallet credits. For the common case of calculating a
balance with just one confirmation, the only involves iterating
over each block structure and adding the (possibly negative)
amount delta. Coinbase rewards are saved similarly, but with a
different amount variable so they can be seperatly included or
excluded.
Due to all of the changes in how the store internally works, the
serialization format has changed. To simplify the serialization
logic, support for reading the last store file version has been
removed. Past this change, a rescan (run automatically) will be
required to rebuild the transaction history.
2014-05-05 23:12:05 +02:00
|
|
|
|
|
|
|
// Insert transaction and credit which will be spent.
|
|
|
|
r, err := s.InsertTx(TstRecvTx, TstRecvTxBlockDetails)
|
|
|
|
if err != nil {
|
|
|
|
t.Fatal(err)
|
|
|
|
}
|
|
|
|
_, err = r.AddCredit(0, false)
|
|
|
|
if err != nil {
|
|
|
|
t.Fatal(err)
|
|
|
|
}
|
|
|
|
|
|
|
|
// Insert confirmed transaction which spends the above credit.
|
|
|
|
TstSpendingTx.SetIndex(TstSignedTxIndex)
|
|
|
|
r2, err := s.InsertTx(TstSpendingTx, TstSignedTxBlockDetails)
|
|
|
|
if err != nil {
|
|
|
|
t.Fatal(err)
|
|
|
|
}
|
|
|
|
_, err = r2.AddCredit(0, false)
|
|
|
|
if err != nil {
|
|
|
|
t.Fatal(err)
|
|
|
|
}
|
|
|
|
_, err = r2.AddDebits(nil)
|
|
|
|
if err != nil {
|
|
|
|
t.Fatal(err)
|
|
|
|
}
|
|
|
|
|
|
|
|
bal, err := s.Balance(1, TstSignedTxBlockDetails.Height)
|
|
|
|
if err != nil {
|
|
|
|
t.Fatal(err)
|
|
|
|
}
|
|
|
|
if bal != btcutil.Amount(TstSpendingTx.MsgTx().TxOut[0].Value) {
|
|
|
|
t.Fatal("bad balance")
|
|
|
|
}
|
|
|
|
unspents, err := s.UnspentOutputs()
|
|
|
|
if err != nil {
|
|
|
|
t.Fatal(err)
|
|
|
|
}
|
|
|
|
op := btcwire.NewOutPoint(TstSpendingTx.Sha(), 0)
|
|
|
|
if *unspents[0].OutPoint() != *op {
|
|
|
|
t.Fatal("unspent outpoint doesn't match expected")
|
|
|
|
}
|
|
|
|
if len(unspents) > 1 {
|
|
|
|
t.Fatal("has more than one unspent credit")
|
|
|
|
}
|
|
|
|
}
|