LBRYCommunity/lbcd
1
0
Fork 0
Code Issues 17 Pull requests 4 Projects Releases 28 Packages Wiki Activity

blockchain: Rework to use new db interface.

Browse source 
This commit is the first stage of several that are planned to convert
the blockchain package into a concurrent safe package that will
ultimately allow support for multi-peer download and concurrent chain
processing.  The goal is to update btcd proper after each step so it can
take advantage of the enhancements as they are developed.

In addition to the aforementioned benefit, this staged approach has been
chosen since it is absolutely critical to maintain consensus.
Separating the changes into several stages makes it easier for reviewers
to logically follow what is happening and therefore helps prevent
consensus bugs.  Naturally there are significant automated tests to help
prevent consensus issues as well.

The main focus of this stage is to convert the blockchain package to use
the new database interface and implement the chain-related functionality
which it no longer handles.  It also aims to improve efficiency in
various areas by making use of the new database and chain capabilities.

The following is an overview of the chain changes:

- Update to use the new database interface
- Add chain-related functionality that the old database used to handle
  - Main chain structure and state
  - Transaction spend tracking
- Implement a new pruned unspent transaction output (utxo) set
  - Provides efficient direct access to the unspent transaction outputs
  - Uses a domain specific compression algorithm that understands the
    standard transaction scripts in order to significantly compress them
  - Removes reliance on the transaction index and paves the way toward
    eventually enabling block pruning
- Modify the New function to accept a Config struct instead of
  inidividual parameters
- Replace the old TxStore type with a new UtxoViewpoint type that makes
  use of the new pruned utxo set
- Convert code to treat the new UtxoViewpoint as a rolling view that is
  used between connects and disconnects to improve efficiency
- Make best chain state always set when the chain instance is created
  - Remove now unnecessary logic for dealing with unset best state
- Make all exported functions concurrent safe
  - Currently using a single chain state lock as it provides a straight
    forward and easy to review path forward however this can be improved
    with more fine grained locking
- Optimize various cases where full blocks were being loaded when only
  the header is needed to help reduce the I/O load
- Add the ability for callers to get a snapshot of the current best
  chain stats in a concurrent safe fashion
  - Does not block callers while new blocks are being processed
- Make error messages that reference transaction outputs consistently
  use <transaction hash>:<output index>
- Introduce a new AssertError type an convert internal consistency
  checks to use it
- Update tests and examples to reflect the changes
- Add a full suite of tests to ensure correct functionality of the new
  code

The following is an overview of the btcd changes:

- Update to use the new database and chain interfaces
- Temporarily remove all code related to the transaction index
- Temporarily remove all code related to the address index
- Convert all code that uses transaction stores to use the new utxo
  view
- Rework several calls that required the block manager for safe
  concurrency to use the chain package directly now that it is
  concurrent safe
- Change all calls to obtain the best hash to use the new best state
  snapshot capability from the chain package
- Remove workaround for limits on fetching height ranges since the new
  database interface no longer imposes them
- Correct the gettxout RPC handler to return the best chain hash as
  opposed the hash the txout was found in
- Optimize various RPC handlers:
  - Change several of the RPC handlers to use the new chain snapshot
    capability to avoid needlessly loading data
  - Update several handlers to use new functionality to avoid accessing
    the block manager so they are able to return the data without
    blocking when the server is busy processing blocks
  - Update non-verbose getblock to avoid deserialization and
    serialization overhead
  - Update getblockheader to request the block height directly from
    chain and only load the header
  - Update getdifficulty to use the new cached data from chain
  - Update getmininginfo to use the new cached data from chain
  - Update non-verbose getrawtransaction to avoid deserialization and
    serialization overhead
  - Update gettxout to use the new utxo store versus loading
    full transactions using the transaction index

The following is an overview of the utility changes:
- Update addblock to use the new database and chain interfaces
- Update findcheckpoint to use the new database and chain interfaces
- Remove the dropafter utility which is no longer supported

NOTE: The transaction index and address index will be reimplemented in
another commit.
This commit is contained in:
Dave Collins 2015-08-25 23:03:18 -05:00
parent 123ff368f4
commit 491acd4ca6
43 changed files with 5960 additions and 3223 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
blockchain/accept.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2013-2015 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -17,6 +17,8 @@ import "github.com/btcsuite/btcutil"
//
// The flags are also passed to checkBlockContext and connectBestChain. See
// their documentation for how the flags modify their behavior.
//
// This function MUST be called with the chain state lock held (for writes).
func (b *BlockChain) maybeAcceptBlock(block *btcutil.Block, flags BehaviorFlags) error {
dryRun := flags&BFDryRun == BFDryRun
@ -74,7 +76,9 @@ func (b *BlockChain) maybeAcceptBlock(block *btcutil.Block, flags BehaviorFlags)
// chain. The caller would typically want to react by relaying the
// inventory to other peers.
if !dryRun {
b.chainLock.Unlock()
b.sendNotification(NTBlockAccepted, block)
b.chainLock.Lock()
}
return nil

153
blockchain/blocklocator.go
View file

@ -1,10 +1,11 @@
// Copyright (c) 2013-2014 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
package blockchain
import (
database "github.com/btcsuite/btcd/database2"
"github.com/btcsuite/btcd/wire"
)
@ -25,7 +26,7 @@ import (
// [17a 16a 15 14 13 12 11 10 9 8 6 2 genesis]
type BlockLocator []*wire.ShaHash
// BlockLocatorFromHash returns a block locator for the passed block hash.
// blockLocatorFromHash returns a block locator for the passed block hash.
// See BlockLocator for details on the algotirhm used to create a block locator.
//
// In addition to the general algorithm referenced above, there are a couple of
@ -35,7 +36,9 @@ type BlockLocator []*wire.ShaHash
// therefore the block locator will only consist of the genesis hash
// - If the passed hash is not currently known, the block locator will only
// consist of the passed hash
func (b *BlockChain) BlockLocatorFromHash(hash *wire.ShaHash) BlockLocator {
//
// This function MUST be called with the chain state lock held (for reads).
func (b *BlockChain) blockLocatorFromHash(hash *wire.ShaHash) BlockLocator {
// The locator contains the requested hash at the very least.
locator := make(BlockLocator, 0, wire.MaxBlockLocatorsPerMsg)
locator = append(locator, hash)
@ -55,7 +58,12 @@ func (b *BlockChain) BlockLocatorFromHash(hash *wire.ShaHash) BlockLocator {
// Try to look up the height for passed block hash. Assume an
// error means it doesn't exist and just return the locator for
// the block itself.
height, err := b.db.FetchBlockHeightBySha(hash)
var height int32
err := b.db.View(func(dbTx database.Tx) error {
var err error
height, err = dbFetchHeightByHash(dbTx, hash)
return err
})
if err != nil {
return locator
}
@ -77,73 +85,94 @@ func (b *BlockChain) BlockLocatorFromHash(hash *wire.ShaHash) BlockLocator {
}
// Generate the block locators according to the algorithm described in
// in the BlockLocator comment and make sure to leave room for the
// final genesis hash.
iterNode := node
increment := int32(1)
for len(locator) < wire.MaxBlockLocatorsPerMsg-1 {
// Once there are 10 locators, exponentially increase the
// distance between each block locator.
if len(locator) > 10 {
increment *= 2
}
blockHeight -= increment
if blockHeight < 1 {
break
// in the BlockLocator comment and make sure to leave room for the final
// genesis hash.
//
// The error is intentionally ignored here since the only way the code
// could fail is if there is something wrong with the database which
// will be caught in short order anyways and it's also safe to ignore
// block locators.
_ = b.db.View(func(dbTx database.Tx) error {
iterNode := node
increment := int32(1)
for len(locator) < wire.MaxBlockLocatorsPerMsg-1 {
// Once there are 10 locators, exponentially increase
// the distance between each block locator.
if len(locator) > 10 {
increment *= 2
}
blockHeight -= increment
if blockHeight < 1 {
break
}
// As long as this is still on the side chain, walk
// backwards along the side chain nodes to each block
// height.
if forkHeight != -1 && blockHeight > forkHeight {
// Intentionally use parent field instead of the
// getPrevNodeFromNode function since we don't
// want to dynamically load nodes when building
// block locators. Side chain blocks should
// always be in memory already, and if they
// aren't for some reason it's ok to skip them.
for iterNode != nil && blockHeight > iterNode.height {
iterNode = iterNode.parent
}
if iterNode != nil && iterNode.height == blockHeight {
locator = append(locator, iterNode.hash)
}
continue
}
// The desired block height is in the main chain, so
// look it up from the main chain database.
h, err := dbFetchHashByHeight(dbTx, blockHeight)
if err != nil {
// This shouldn't happen and it's ok to ignore
// block locators, so just continue to the next
// one.
log.Warnf("Lookup of known valid height failed %v",
blockHeight)
continue
}
locator = append(locator, h)
}
// As long as this is still on the side chain, walk backwards
// along the side chain nodes to each block height.
if forkHeight != -1 && blockHeight > forkHeight {
// Intentionally use parent field instead of the
// getPrevNodeFromNode function since we don't want to
// dynamically load nodes when building block locators.
// Side chain blocks should always be in memory already,
// and if they aren't for some reason it's ok to skip
// them.
for iterNode != nil && blockHeight > iterNode.height {
iterNode = iterNode.parent
}
if iterNode != nil && iterNode.height == blockHeight {
locator = append(locator, iterNode.hash)
}
continue
}
// The desired block height is in the main chain, so look it up
// from the main chain database.
h, err := b.db.FetchBlockShaByHeight(blockHeight)
if err != nil {
// This shouldn't happen and it's ok to ignore block
// locators, so just continue to the next one.
log.Warnf("Lookup of known valid height failed %v",
blockHeight)
continue
}
locator = append(locator, h)
}
return nil
})
// Append the appropriate genesis block.
locator = append(locator, b.chainParams.GenesisHash)
return locator
}
// BlockLocatorFromHash returns a block locator for the passed block hash.
// See BlockLocator for details on the algorithm used to create a block locator.
//
// In addition to the general algorithm referenced above, there are a couple of
// special cases which are handled:
//
// - If the genesis hash is passed, there are no previous hashes to add and
// therefore the block locator will only consist of the genesis hash
// - If the passed hash is not currently known, the block locator will only
// consist of the passed hash
//
// This function is safe for concurrent access.
func (b *BlockChain) BlockLocatorFromHash(hash *wire.ShaHash) BlockLocator {
b.chainLock.RLock()
locator := b.blockLocatorFromHash(hash)
b.chainLock.RUnlock()
return locator
}
// LatestBlockLocator returns a block locator for the latest known tip of the
// main (best) chain.
//
// This function is safe for concurrent access.
func (b *BlockChain) LatestBlockLocator() (BlockLocator, error) {
// Lookup the latest main chain hash if the best chain hasn't been set
// yet.
if b.bestChain == nil {
// Get the latest block hash for the main chain from the
// database.
hash, _, err := b.db.NewestSha()
if err != nil {
return nil, err
}
return b.BlockLocatorFromHash(hash), nil
}
// The best chain is set, so use its hash.
return b.BlockLocatorFromHash(b.bestChain.hash), nil
b.chainLock.RLock()
locator := b.blockLocatorFromHash(b.bestNode.hash)
b.chainLock.RUnlock()
return locator, nil
}

767
blockchain/chain.go
View file

File diff suppressed because it is too large Load diff

1401
blockchain/chainio.go Normal file
View file

File diff suppressed because it is too large Load diff

1030
blockchain/chainio_test.go Normal file
View file

File diff suppressed because it is too large Load diff

218
blockchain/checkpoints.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2013-2014 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -8,6 +8,7 @@ import (
"fmt"
"github.com/btcsuite/btcd/chaincfg"
database "github.com/btcsuite/btcd/database2"
"github.com/btcsuite/btcd/txscript"
"github.com/btcsuite/btcd/wire"
"github.com/btcsuite/btcutil"
@ -29,14 +30,23 @@ func newShaHashFromStr(hexStr string) *wire.ShaHash {
// DisableCheckpoints provides a mechanism to disable validation against
// checkpoints which you DO NOT want to do in production. It is provided only
// for debug purposes.
//
// This function is safe for concurrent access.
func (b *BlockChain) DisableCheckpoints(disable bool) {
b.chainLock.Lock()
b.noCheckpoints = disable
b.chainLock.Unlock()
}
// Checkpoints returns a slice of checkpoints (regardless of whether they are
// already known). When checkpoints are disabled or there are no checkpoints
// for the active network, it will return nil.
//
// This function is safe for concurrent access.
func (b *BlockChain) Checkpoints() []chaincfg.Checkpoint {
b.chainLock.RLock()
defer b.chainLock.RUnlock()
if b.noCheckpoints || len(b.chainParams.Checkpoints) == 0 {
return nil
}
@ -44,10 +54,12 @@ func (b *BlockChain) Checkpoints() []chaincfg.Checkpoint {
return b.chainParams.Checkpoints
}
// LatestCheckpoint returns the most recent checkpoint (regardless of whether it
// latestCheckpoint returns the most recent checkpoint (regardless of whether it
// is already known). When checkpoints are disabled or there are no checkpoints
// for the active network, it will return nil.
func (b *BlockChain) LatestCheckpoint() *chaincfg.Checkpoint {
//
// This function MUST be called with the chain state lock held (for reads).
func (b *BlockChain) latestCheckpoint() *chaincfg.Checkpoint {
if b.noCheckpoints || len(b.chainParams.Checkpoints) == 0 {
return nil
}
@ -56,9 +68,23 @@ func (b *BlockChain) LatestCheckpoint() *chaincfg.Checkpoint {
return &checkpoints[len(checkpoints)-1]
}
// LatestCheckpoint returns the most recent checkpoint (regardless of whether it
// is already known). When checkpoints are disabled or there are no checkpoints
// for the active network, it will return nil.
//
// This function is safe for concurrent access.
func (b *BlockChain) LatestCheckpoint() *chaincfg.Checkpoint {
b.chainLock.RLock()
checkpoint := b.latestCheckpoint()
b.chainLock.RUnlock()
return checkpoint
}
// verifyCheckpoint returns whether the passed block height and hash combination
// match the hard-coded checkpoint data. It also returns true if there is no
// checkpoint data for the passed block height.
//
// This function MUST be called with the chain lock held (for reads).
func (b *BlockChain) verifyCheckpoint(height int32, hash *wire.ShaHash) bool {
if b.noCheckpoints || len(b.chainParams.Checkpoints) == 0 {
return true
@ -83,6 +109,8 @@ func (b *BlockChain) verifyCheckpoint(height int32, hash *wire.ShaHash) bool {
// available in the downloaded portion of the block chain and returns the
// associated block. It returns nil if a checkpoint can't be found (this should
// really only happen for blocks before the first checkpoint).
//
// This function MUST be called with the chain lock held (for reads).
func (b *BlockChain) findPreviousCheckpoint() (*btcutil.Block, error) {
if b.noCheckpoints || len(b.chainParams.Checkpoints) == 0 {
return nil, nil
@ -98,20 +126,21 @@ func (b *BlockChain) findPreviousCheckpoint() (*btcutil.Block, error) {
// Perform the initial search to find and cache the latest known
// checkpoint if the best chain is not known yet or we haven't already
// previously searched.
if b.bestChain == nil || (b.checkpointBlock == nil && b.nextCheckpoint == nil) {
if b.checkpointBlock == nil && b.nextCheckpoint == nil {
// Loop backwards through the available checkpoints to find one
// that we already have.
// that is already available.
checkpointIndex := -1
for i := numCheckpoints - 1; i >= 0; i-- {
exists, err := b.db.ExistsSha(checkpoints[i].Hash)
if err != nil {
return nil, err
}
if exists {
checkpointIndex = i
break
err := b.db.View(func(dbTx database.Tx) error {
for i := numCheckpoints - 1; i >= 0; i-- {
if dbMainChainHasBlock(dbTx, checkpoints[i].Hash) {
checkpointIndex = i
break
}
}
return nil
})
if err != nil {
return nil, err
}
// No known latest checkpoint. This will only happen on blocks
@ -125,19 +154,26 @@ func (b *BlockChain) findPreviousCheckpoint() (*btcutil.Block, error) {
// Cache the latest known checkpoint block for future lookups.
checkpoint := checkpoints[checkpointIndex]
block, err := b.db.FetchBlockBySha(checkpoint.Hash)
err = b.db.View(func(dbTx database.Tx) error {
block, err := dbFetchBlockByHash(dbTx, checkpoint.Hash)
if err != nil {
return err
}
b.checkpointBlock = block
// Set the next expected checkpoint block accordingly.
b.nextCheckpoint = nil
if checkpointIndex < numCheckpoints-1 {
b.nextCheckpoint = &checkpoints[checkpointIndex+1]
}
return nil
})
if err != nil {
return nil, err
}
b.checkpointBlock = block
// Set the next expected checkpoint block accordingly.
b.nextCheckpoint = nil
if checkpointIndex < numCheckpoints-1 {
b.nextCheckpoint = &checkpoints[checkpointIndex+1]
}
return block, nil
return b.checkpointBlock, nil
}
// At this point we've already searched for the latest known checkpoint,
@ -150,7 +186,7 @@ func (b *BlockChain) findPreviousCheckpoint() (*btcutil.Block, error) {
// When there is a next checkpoint and the height of the current best
// chain does not exceed it, the current checkpoint lockin is still
// the latest known checkpoint.
if b.bestChain.height < b.nextCheckpoint.Height {
if b.bestNode.height < b.nextCheckpoint.Height {
return b.checkpointBlock, nil
}
@ -163,11 +199,17 @@ func (b *BlockChain) findPreviousCheckpoint() (*btcutil.Block, error) {
// that if this lookup fails something is very wrong since the chain
// has already passed the checkpoint which was verified as accurate
// before inserting it.
block, err := b.db.FetchBlockBySha(b.nextCheckpoint.Hash)
err := b.db.View(func(tx database.Tx) error {
block, err := dbFetchBlockByHash(tx, b.nextCheckpoint.Hash)
if err != nil {
return err
}
b.checkpointBlock = block
return nil
})
if err != nil {
return nil, err
}
b.checkpointBlock = block
// Set the next expected checkpoint.
checkpointIndex := -1
@ -188,8 +230,6 @@ func (b *BlockChain) findPreviousCheckpoint() (*btcutil.Block, error) {
// isNonstandardTransaction determines whether a transaction contains any
// scripts which are not one of the standard types.
func isNonstandardTransaction(tx *btcutil.Tx) bool {
// TODO(davec): Should there be checks for the input signature scripts?
// Check all of the output public key scripts for non-standard scripts.
for _, txOut := range tx.MsgTx().TxOut {
scriptClass := txscript.GetScriptClass(txOut.PkScript)
@ -215,66 +255,80 @@ func isNonstandardTransaction(tx *btcutil.Tx) bool {
//
// The intent is that candidates are reviewed by a developer to make the final
// decision and then manually added to the list of checkpoints for a network.
//
// This function is safe for concurrent access.
func (b *BlockChain) IsCheckpointCandidate(block *btcutil.Block) (bool, error) {
b.chainLock.RLock()
defer b.chainLock.RUnlock()
// Checkpoints must be enabled.
if b.noCheckpoints {
return false, fmt.Errorf("checkpoints are disabled")
}
// A checkpoint must be in the main chain.
exists, err := b.db.ExistsSha(block.Sha())
if err != nil {
return false, err
}
if !exists {
return false, nil
}
// A checkpoint must be at least CheckpointConfirmations blocks before
// the end of the main chain.
blockHeight := block.Height()
_, mainChainHeight, err := b.db.NewestSha()
if err != nil {
return false, err
}
if blockHeight > (mainChainHeight - CheckpointConfirmations) {
return false, nil
}
// Get the previous block.
prevHash := &block.MsgBlock().Header.PrevBlock
prevBlock, err := b.db.FetchBlockBySha(prevHash)
if err != nil {
return false, err
}
// Get the next block.
nextHash, err := b.db.FetchBlockShaByHeight(blockHeight + 1)
if err != nil {
return false, err
}
nextBlock, err := b.db.FetchBlockBySha(nextHash)
if err != nil {
return false, err
}
// A checkpoint must have timestamps for the block and the blocks on
// either side of it in order (due to the median time allowance this is
// not always the case).
prevTime := prevBlock.MsgBlock().Header.Timestamp
curTime := block.MsgBlock().Header.Timestamp
nextTime := nextBlock.MsgBlock().Header.Timestamp
if prevTime.After(curTime) || nextTime.Before(curTime) {
return false, nil
}
// A checkpoint must have transactions that only contain standard
// scripts.
for _, tx := range block.Transactions() {
if isNonstandardTransaction(tx) {
return false, nil
var isCandidate bool
err := b.db.View(func(dbTx database.Tx) error {
// A checkpoint must be in the main chain.
blockHeight, err := dbFetchHeightByHash(dbTx, block.Sha())
if err != nil {
// Only return an error if it's not due to the block not
// being in the main chain.
if !isNotInMainChainErr(err) {
return err
}
return nil
}
}
return true, nil
// Ensure the height of the passed block and the entry for the
// block in the main chain match. This should always be the
// case unless the caller provided an invalid block.
if blockHeight != block.Height() {
return fmt.Errorf("passed block height of %d does not "+
"match the main chain height of %d",
block.Height(), blockHeight)
}
// A checkpoint must be at least CheckpointConfirmations blocks
// before the end of the main chain.
mainChainHeight := b.bestNode.height
if blockHeight > (mainChainHeight - CheckpointConfirmations) {
return nil
}
// Get the previous block header.
prevHash := &block.MsgBlock().Header.PrevBlock
prevHeader, err := dbFetchHeaderByHash(dbTx, prevHash)
if err != nil {
return err
}
// Get the next block header.
nextHeader, err := dbFetchHeaderByHeight(dbTx, blockHeight+1)
if err != nil {
return err
}
// A checkpoint must have timestamps for the block and the
// blocks on either side of it in order (due to the median time
// allowance this is not always the case).
prevTime := prevHeader.Timestamp
curTime := block.MsgBlock().Header.Timestamp
nextTime := nextHeader.Timestamp
if prevTime.After(curTime) || nextTime.Before(curTime) {
return nil
}
// A checkpoint must have transactions that only contain
// standard scripts.
for _, tx := range block.Transactions() {
if isNonstandardTransaction(tx) {
return nil
}
}
// All of the checks passed, so the block is a candidate.
isCandidate = true
return nil
})
return isCandidate, err
}

164
blockchain/common_test.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2013-2014 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -15,18 +15,21 @@ import (
"github.com/btcsuite/btcd/blockchain"
"github.com/btcsuite/btcd/chaincfg"
"github.com/btcsuite/btcd/database"
_ "github.com/btcsuite/btcd/database/ldb"
_ "github.com/btcsuite/btcd/database/memdb"
database "github.com/btcsuite/btcd/database2"
_ "github.com/btcsuite/btcd/database2/ffldb"
"github.com/btcsuite/btcd/wire"
"github.com/btcsuite/btcutil"
)
// testDbType is the database backend type to use for the tests.
const testDbType = "memdb"
const (
// testDbType is the database backend type to use for the tests.
testDbType = "ffldb"
// testDbRoot is the root directory used to create all test databases.
const testDbRoot = "testdbs"
// testDbRoot is the root directory used to create all test databases.
testDbRoot = "testdbs"
// blockDataNet is the expected network in the test block data.
blockDataNet = wire.MainNet
)
// filesExists returns whether or not the named file or directory exists.
func fileExists(name string) bool {
@ -41,9 +44,9 @@ func fileExists(name string) bool {
// isSupportedDbType returns whether or not the passed database type is
// currently supported.
func isSupportedDbType(dbType string) bool {
supportedDBs := database.SupportedDBs()
for _, sDbType := range supportedDBs {
if dbType == sDbType {
supportedDrivers := database.SupportedDrivers()
for _, driver := range supportedDrivers {
if dbType == driver {
return true
}
}
@ -61,10 +64,10 @@ func chainSetup(dbName string) (*blockchain.BlockChain, func(), error) {
// Handle memory database specially since it doesn't need the disk
// specific handling.
var db database.Db
var db database.DB
var teardown func()
if testDbType == "memdb" {
ndb, err := database.CreateDB(testDbType)
ndb, err := database.Create(testDbType)
if err != nil {
return nil, nil, fmt.Errorf("error creating db: %v", err)
}
@ -88,7 +91,7 @@ func chainSetup(dbName string) (*blockchain.BlockChain, func(), error) {
// Create a new database to store the accepted blocks into.
dbPath := filepath.Join(testDbRoot, dbName)
_ = os.RemoveAll(dbPath)
ndb, err := database.CreateDB(testDbType, dbPath)
ndb, err := database.Create(testDbType, dbPath, blockDataNet)
if err != nil {
return nil, nil, fmt.Errorf("error creating db: %v", err)
}
@ -97,39 +100,34 @@ func chainSetup(dbName string) (*blockchain.BlockChain, func(), error) {
// Setup a teardown function for cleaning up. This function is
// returned to the caller to be invoked when it is done testing.
teardown = func() {
dbVersionPath := filepath.Join(testDbRoot, dbName+".ver")
db.Sync()
db.Close()
os.RemoveAll(dbPath)
os.Remove(dbVersionPath)
os.RemoveAll(testDbRoot)
}
}
// Insert the main network genesis block. This is part of the initial
// database setup.
genesisBlock := btcutil.NewBlock(chaincfg.MainNetParams.GenesisBlock)
_, err := db.InsertBlock(genesisBlock)
// Create the main chain instance.
chain, err := blockchain.New(&blockchain.Config{
DB: db,
ChainParams: &chaincfg.MainNetParams,
})
if err != nil {
teardown()
err := fmt.Errorf("failed to insert genesis block: %v", err)
err := fmt.Errorf("failed to create chain instance: %v", err)
return nil, nil, err
}
chain := blockchain.New(db, &chaincfg.MainNetParams, nil, nil)
return chain, teardown, nil
}
// loadTxStore returns a transaction store loaded from a file.
func loadTxStore(filename string) (blockchain.TxStore, error) {
// The txstore file format is:
// <num tx data entries> <tx length> <serialized tx> <blk height>
// <num spent bits> <spent bits>
// loadUtxoView returns a utxo view loaded from a file.
func loadUtxoView(filename string) (*blockchain.UtxoViewpoint, error) {
// The utxostore file format is:
// <tx hash><serialized utxo len><serialized utxo>
//
// All num and length fields are little-endian uint32s. The spent bits
// field is padded to a byte boundary.
// The serialized utxo len is a little endian uint32 and the serialized
// utxo uses the format described in chainio.go.
filename = filepath.Join("testdata/", filename)
filename = filepath.Join("testdata", filename)
fi, err := os.Open(filename)
if err != nil {
return nil, err
@ -144,80 +142,40 @@ func loadTxStore(filename string) (blockchain.TxStore, error) {
}
defer fi.Close()
// Num of transaction store objects.
var numItems uint32
if err := binary.Read(r, binary.LittleEndian, &numItems); err != nil {
return nil, err
}
txStore := make(blockchain.TxStore)
var uintBuf uint32
for height := uint32(0); height < numItems; height++ {
txD := blockchain.TxData{}
// Serialized transaction length.
err = binary.Read(r, binary.LittleEndian, &uintBuf)
view := blockchain.NewUtxoViewpoint()
for {
// Hash of the utxo entry.
var hash wire.ShaHash
_, err := io.ReadAtLeast(r, hash[:], len(hash[:]))
if err != nil {
return nil, err
}
serializedTxLen := uintBuf
if serializedTxLen > wire.MaxBlockPayload {
return nil, fmt.Errorf("Read serialized transaction "+
"length of %d is larger max allowed %d",
serializedTxLen, wire.MaxBlockPayload)
}
// Transaction.
var msgTx wire.MsgTx
err = msgTx.Deserialize(r)
if err != nil {
return nil, err
}
txD.Tx = btcutil.NewTx(&msgTx)
// Transaction hash.
txHash := msgTx.TxSha()
txD.Hash = &txHash
// Block height the transaction came from.
err = binary.Read(r, binary.LittleEndian, &uintBuf)
if err != nil {
return nil, err
}
txD.BlockHeight = int32(uintBuf)
// Num spent bits.
err = binary.Read(r, binary.LittleEndian, &uintBuf)
if err != nil {
return nil, err
}
numSpentBits := uintBuf
numSpentBytes := numSpentBits / 8
if numSpentBits%8 != 0 {
numSpentBytes++
}
// Packed spent bytes.
spentBytes := make([]byte, numSpentBytes)
_, err = io.ReadFull(r, spentBytes)
if err != nil {
return nil, err
}
// Populate spent data based on spent bits.
txD.Spent = make([]bool, numSpentBits)
for byteNum, spentByte := range spentBytes {
for bit := 0; bit < 8; bit++ {
if uint32((byteNum*8)+bit) < numSpentBits {
if spentByte&(1<<uint(bit)) != 0 {
txD.Spent[(byteNum*8)+bit] = true
}
}
// Expected EOF at the right offset.
if err == io.EOF {
break
}
return nil, err
}
txStore[*txD.Hash] = &txD
// Num of serialize utxo entry bytes.
var numBytes uint32
err = binary.Read(r, binary.LittleEndian, &numBytes)
if err != nil {
return nil, err
}
// Serialized utxo entry.
serialized := make([]byte, numBytes)
_, err = io.ReadAtLeast(r, serialized, int(numBytes))
if err != nil {
return nil, err
}
// Deserialize it and add it to the view.
utxoEntry, err := blockchain.TstDeserializeUtxoEntry(serialized)
if err != nil {
return nil, err
}
view.Entries()[hash] = utxoEntry
}
return txStore, nil
return view, nil
}

603
blockchain/compress.go Normal file
View file

@ -0,0 +1,603 @@
// Copyright (c) 2015-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
package blockchain
import (
"github.com/btcsuite/btcd/btcec"
"github.com/btcsuite/btcd/txscript"
)
// -----------------------------------------------------------------------------
// A variable length quantity (VLQ) is an encoding that uses an arbitrary number
// of binary octets to represent an arbitrarily large integer. The scheme
// employs a most significant byte (MSB) base-128 encoding where the high bit in
// each byte indicates whether or not the byte is the final one. In addition,
// to ensure there are no redundant encodings, an offset is subtracted every
// time a group of 7 bits is shifted out. Therefore each integer can be
// represented in exactly one way, and each representation stands for exactly
// one integer.
//
// Another nice property of this encoding is that it provides a compact
// representation of values that are typically used to indicate sizes. For
// example, the values 0 - 127 are represented with a single byte, 128 - 16511
// with two bytes, and 16512 - 2113663 with three bytes.
//
// While the encoding allows arbitrarily large integers, it is artificially
// limited in this code to an unsigned 64-bit integer for efficiency purposes.
//
// Example encodings:
// 0 -> [0x00]
// 127 -> [0x7f] * Max 1-byte value
// 128 -> [0x80 0x00]
// 129 -> [0x80 0x01]
// 255 -> [0x80 0x7f]
// 256 -> [0x81 0x00]
// 16511 -> [0xff 0x7f] * Max 2-byte value
// 16512 -> [0x80 0x80 0x00]
// 32895 -> [0x80 0xff 0x7f]
// 2113663 -> [0xff 0xff 0x7f] * Max 3-byte value
// 270549119 -> [0xff 0xff 0xff 0x7f] * Max 4-byte value
// 2^64-1 -> [0x80 0xfe 0xfe 0xfe 0xfe 0xfe 0xfe 0xfe 0xfe 0x7f]
//
// References:
// https://en.wikipedia.org/wiki/Variable-length_quantity
// http://www.codecodex.com/wiki/Variable-Length_Integers
// -----------------------------------------------------------------------------
// serializeSizeVLQ returns the number of bytes it would take to serialize the
// passed number as a variable-length quantity according to the format described
// above.
func serializeSizeVLQ(n uint64) int {
size := 1
for ; n > 0x7f; n = (n >> 7) - 1 {
size++
}
return size
}
// putVLQ serializes the provided number to a variable-length quantity according
// to the format described above and returns the number of bytes of the encoded
// value. The result is placed directly into the passed byte slice which must
// be at least large enough to handle the number of bytes returned by the
// serializeSizeVLQ function or it will panic.
func putVLQ(target []byte, n uint64) int {
offset := 0
for ; ; offset++ {
// The high bit is set when another byte follows.
highBitMask := byte(0x80)
if offset == 0 {
highBitMask = 0x00
}
target[offset] = byte(n&0x7f) | highBitMask
if n <= 0x7f {
break
}
n = (n >> 7) - 1
}
// Reverse the bytes so it is MSB-encoded.
for i, j := 0, offset; i < j; i, j = i+1, j-1 {
target[i], target[j] = target[j], target[i]
}
return offset + 1
}
// deserializeVLQ deserializes the provided variable-length quantity according
// to the format described above. It also returns the number of bytes
// deserialized.
func deserializeVLQ(serialized []byte) (uint64, int) {
var n uint64
var size int
for _, val := range serialized {
size++
n = (n << 7) | uint64(val&0x7f)
if val&0x80 != 0x80 {
break
}
n++
}
return n, size
}
// -----------------------------------------------------------------------------
// In order to reduce the size of stored scripts, a domain specific compression
// algorithm is used which recognizes standard scripts and stores them using
// less bytes than the original script. The compression algorithm used here was
// obtained from Bitcoin Core, so all credits for the algorithm go to it.
//
// The general serialized format is:
//
// <script size or type><script data>
//
// Field Type Size
// script size or type VLQ variable
// script data []byte variable
//
// The specific serialized format for each recognized standard script is:
//
// - Pay-to-pubkey-hash: (21 bytes) - <0><20-byte pubkey hash>
// - Pay-to-script-hash: (21 bytes) - <1><20-byte script hash>
// - Pay-to-pubkey**: (33 bytes) - <2, 3, 4, or 5><32-byte pubkey X value>
// 2, 3 = compressed pubkey with bit 0 specifying the y coordinate to use
// 4, 5 = uncompressed pubkey with bit 0 specifying the y coordinate to use
// ** Only valid public keys starting with 0x02, 0x03, and 0x04 are supported.
//
// Any scripts which are not recognized as one of the aforementioned standard
// scripts are encoded using the general serialized format and encode the script
// size as the sum of the actual size of the script and the number of special
// cases.
// -----------------------------------------------------------------------------
// The following constants specify the special constants used to identify a
// special script type in the domain-specific compressed script encoding.
//
// NOTE: This section specifically does not use iota since these values are
// serialized and must be stable for long-term storage.
const (
// cstPayToPubKeyHash identifies a compressed pay-to-pubkey-hash script.
cstPayToPubKeyHash = 0
// cstPayToScriptHash identifies a compressed pay-to-script-hash script.
cstPayToScriptHash = 1
// cstPayToPubKeyComp2 identifies a compressed pay-to-pubkey script to
// a compressed pubkey. Bit 0 specifies which y-coordinate to use
// to reconstruct the full uncompressed pubkey.
cstPayToPubKeyComp2 = 2
// cstPayToPubKeyComp3 identifies a compressed pay-to-pubkey script to
// a compressed pubkey. Bit 0 specifies which y-coordinate to use
// to reconstruct the full uncompressed pubkey.
cstPayToPubKeyComp3 = 3
// cstPayToPubKeyUncomp4 identifies a compressed pay-to-pubkey script to
// an uncompressed pubkey. Bit 0 specifies which y-coordinate to use
// to reconstruct the full uncompressed pubkey.
cstPayToPubKeyUncomp4 = 4
// cstPayToPubKeyUncomp5 identifies a compressed pay-to-pubkey script to
// an uncompressed pubkey. Bit 0 specifies which y-coordinate to use
// to reconstruct the full uncompressed pubkey.
cstPayToPubKeyUncomp5 = 5
// numSpecialScripts is the number of special scripts recognized by the
// domain-specific script compression algorithm.
numSpecialScripts = 6
)
// isPubKeyHash returns whether or not the passed public key script is a
// standard pay-to-pubkey-hash script along with the pubkey hash it is paying to
// if it is.
func isPubKeyHash(script []byte) (bool, []byte) {
if len(script) == 25 && script[0] == txscript.OP_DUP &&
script[1] == txscript.OP_HASH160 &&
script[2] == txscript.OP_DATA_20 &&
script[23] == txscript.OP_EQUALVERIFY &&
script[24] == txscript.OP_CHECKSIG {
return true, script[3:23]
}
return false, nil
}
// isScriptHash returns whether or not the passed public key script is a
// standard pay-to-script-hash script along with the script hash it is paying to
// if it is.
func isScriptHash(script []byte) (bool, []byte) {
if len(script) == 23 && script[0] == txscript.OP_HASH160 &&
script[1] == txscript.OP_DATA_20 &&
script[22] == txscript.OP_EQUAL {
return true, script[2:22]
}
return false, nil
}
// isPubKey returns whether or not the passed public key script is a standard
// pay-to-pubkey script that pays to a valid compressed or uncompressed public
// key along with the serialized pubkey it is paying to if it is.
//
// NOTE: This function ensures the public key is actually valid since the
// compression algorithm requires valid pubkeys. It does not support hybrid
// pubkeys. This means that even if the script has the correct form for a
// pay-to-pubkey script, this function will only return true when it is paying
// to a valid compressed or uncompressed pubkey.
func isPubKey(script []byte) (bool, []byte) {
// Pay-to-compressed-pubkey script.
if len(script) == 35 && script[0] == txscript.OP_DATA_33 &&
script[34] == txscript.OP_CHECKSIG && (script[1] == 0x02 ||
script[1] == 0x03) {
// Ensure the public key is valid.
serializedPubKey := script[1:34]
_, err := btcec.ParsePubKey(serializedPubKey, btcec.S256())
if err == nil {
return true, serializedPubKey
}
}
// Pay-to-uncompressed-pubkey script.
if len(script) == 67 && script[0] == txscript.OP_DATA_65 &&
script[66] == txscript.OP_CHECKSIG && script[1] == 0x04 {
// Ensure the public key is valid.
serializedPubKey := script[1:66]
_, err := btcec.ParsePubKey(serializedPubKey, btcec.S256())
if err == nil {
return true, serializedPubKey
}
}
return false, nil
}
// compressedScriptSize returns the number of bytes the passed script would take
// when encoded with the domain specific compression algorithm described above.
func compressedScriptSize(pkScript []byte, version int32) int {
// Pay-to-pubkey-hash script.
if valid, _ := isPubKeyHash(pkScript); valid {
return 21
}
// Pay-to-script-hash script.
if valid, _ := isScriptHash(pkScript); valid {
return 21
}
// Pay-to-pubkey (compressed or uncompressed) script.
if valid, _ := isPubKey(pkScript); valid {
return 33
}
// When none of the above special cases apply, encode the script as is
// preceded by the sum of its size and the number of special cases
// encoded as a variable length quantity.
return serializeSizeVLQ(uint64(len(pkScript)+numSpecialScripts)) +
len(pkScript)
}
// decodeCompressedScriptSize treats the passed serialized bytes as a compressed
// script, possibly followed by other data, and returns the number of bytes it
// occupies taking into account the special encoding of the script size by the
// domain specific compression algorithm described above.
func decodeCompressedScriptSize(serialized []byte, version int32) int {
scriptSize, bytesRead := deserializeVLQ(serialized)
if bytesRead == 0 {
return 0
}
switch scriptSize {
case cstPayToPubKeyHash:
return 21
case cstPayToScriptHash:
return 21
case cstPayToPubKeyComp2, cstPayToPubKeyComp3, cstPayToPubKeyUncomp4,
cstPayToPubKeyUncomp5:
return 33
}
scriptSize -= numSpecialScripts
scriptSize += uint64(bytesRead)
return int(scriptSize)
}
// putCompressedScript compresses the passed script according to the domain
// specific compression algorithm described above directly into the passed
// target byte slice. The target byte slice must be at least large enough to
// handle the number of bytes returned by the compressedScriptSize function or
// it will panic.
func putCompressedScript(target, pkScript []byte, version int32) int {
// Pay-to-pubkey-hash script.
if valid, hash := isPubKeyHash(pkScript); valid {
target[0] = cstPayToPubKeyHash
copy(target[1:21], hash)
return 21
}
// Pay-to-script-hash script.
if valid, hash := isScriptHash(pkScript); valid {
target[0] = cstPayToScriptHash
copy(target[1:21], hash)
return 21
}
// Pay-to-pubkey (compressed or uncompressed) script.
if valid, serializedPubKey := isPubKey(pkScript); valid {
pubKeyFormat := serializedPubKey[0]
switch pubKeyFormat {
case 0x02, 0x03:
target[0] = pubKeyFormat
copy(target[1:33], serializedPubKey[1:33])
return 33
case 0x04:
// Encode the oddness of the serialized pubkey into the
// compressed script type.
target[0] = pubKeyFormat | (serializedPubKey[64] & 0x01)
copy(target[1:33], serializedPubKey[1:33])
return 33
}
}
// When none of the above special cases apply, encode the unmodified
// script preceded by the sum of its size and the number of special
// cases encoded as a variable length quantity.
encodedSize := uint64(len(pkScript) + numSpecialScripts)
vlqSizeLen := putVLQ(target, encodedSize)
copy(target[vlqSizeLen:], pkScript)
return vlqSizeLen + len(pkScript)
}
// decompressScript returns the original script obtained by decompressing the
// passed compressed script according to the domain specific compression
// algorithm described above.
//
// NOTE: The script parameter must already have been proven to be long enough
// to contain the number of bytes returned by decodeCompressedScriptSize or it
// will panic. This is acceptable since it is only an internal function.
func decompressScript(compressedPkScript []byte, version int32) []byte {
// In practice this function will not be called with a zero-length or
// nil script since the nil script encoding includes the length, however
// the code below assumes the length exists, so just return nil now if
// the function ever ends up being called with a nil script in the
// future.
if len(compressedPkScript) == 0 {
return nil
}
// Decode the script size and examine it for the special cases.
encodedScriptSize, bytesRead := deserializeVLQ(compressedPkScript)
switch encodedScriptSize {
// Pay-to-pubkey-hash script. The resulting script is:
// <OP_DUP><OP_HASH160><20 byte hash><OP_EQUALVERIFY><OP_CHECKSIG>
case cstPayToPubKeyHash:
pkScript := make([]byte, 25)
pkScript[0] = txscript.OP_DUP
pkScript[1] = txscript.OP_HASH160
pkScript[2] = txscript.OP_DATA_20
copy(pkScript[3:], compressedPkScript[bytesRead:bytesRead+20])
pkScript[23] = txscript.OP_EQUALVERIFY
pkScript[24] = txscript.OP_CHECKSIG
return pkScript
// Pay-to-script-hash script. The resulting script is:
// <OP_HASH160><20 byte script hash><OP_EQUAL>
case cstPayToScriptHash:
pkScript := make([]byte, 23)
pkScript[0] = txscript.OP_HASH160
pkScript[1] = txscript.OP_DATA_20
copy(pkScript[2:], compressedPkScript[bytesRead:bytesRead+20])
pkScript[22] = txscript.OP_EQUAL
return pkScript
// Pay-to-compressed-pubkey script. The resulting script is:
// <OP_DATA_33><33 byte compressed pubkey><OP_CHECKSIG>
case cstPayToPubKeyComp2, cstPayToPubKeyComp3:
pkScript := make([]byte, 35)
pkScript[0] = txscript.OP_DATA_33
pkScript[1] = byte(encodedScriptSize)
copy(pkScript[2:], compressedPkScript[bytesRead:bytesRead+32])
pkScript[34] = txscript.OP_CHECKSIG
return pkScript
// Pay-to-uncompressed-pubkey script. The resulting script is:
// <OP_DATA_65><65 byte uncompressed pubkey><OP_CHECKSIG>
case cstPayToPubKeyUncomp4, cstPayToPubKeyUncomp5:
// Change the leading byte to the appropriate compressed pubkey
// identifier (0x02 or 0x03) so it can be decoded as a
// compressed pubkey. This really should never fail since the
// encoding ensures it is valid before compressing to this type.
compressedKey := make([]byte, 33)
compressedKey[0] = byte(encodedScriptSize - 2)
copy(compressedKey[1:], compressedPkScript[1:])
key, err := btcec.ParsePubKey(compressedKey, btcec.S256())
if err != nil {
return nil
}
pkScript := make([]byte, 67)
pkScript[0] = txscript.OP_DATA_65
copy(pkScript[1:], key.SerializeUncompressed())
pkScript[66] = txscript.OP_CHECKSIG
return pkScript
}
// When none of the special cases apply, the script was encoded using
// the general format, so reduce the script size by the number of
// special cases and return the unmodified script.
scriptSize := int(encodedScriptSize - numSpecialScripts)
pkScript := make([]byte, scriptSize)
copy(pkScript, compressedPkScript[bytesRead:bytesRead+scriptSize])
return pkScript
}
// -----------------------------------------------------------------------------
// In order to reduce the size of stored amounts, a domain specific compression
// algorithm is used which relies on there typically being a lot of zeroes at
// end of the amounts. The compression algorithm used here was obtained from
// Bitcoin Core, so all credits for the algorithm go to it.
//
// While this is simply exchanging one uint64 for another, the resulting value
// for typical amounts has a much smaller magnitude which results in fewer bytes
// when encoded as variable length quantity. For example, consider the amount
// of 0.1 BTC which is 10000000 satoshi. Encoding 10000000 as a VLQ would take
// 4 bytes while encoding the compressed value of 8 as a VLQ only takes 1 byte.
//
// Essentially the compression is achieved by splitting the value into an
// exponent in the range [0-9] and a digit in the range [1-9], when possible,
// and encoding them in a way that can be decoded. More specifically, the
// encoding is as follows:
// - 0 is 0
// - Find the exponent, e, as the largest power of 10 that evenly divides the
// value up to a maximum of 9
// - When e < 9, the final digit can't be 0 so store it as d and remove it by
// dividing the value by 10 (call the result n). The encoded value is thus:
// 1 + 10*(9*n + d-1) + e
// - When e==9, the only thing known is the amount is not 0. The encoded value
// is thus:
// 1 + 10*(n-1) + e == 10 + 10*(n-1)
//
// Example encodings:
// (The numbers in parenthesis are the number of bytes when serialized as a VLQ)
// 0 (1) -> 0 (1) * 0.00000000 BTC
// 1000 (2) -> 4 (1) * 0.00001000 BTC
// 10000 (2) -> 5 (1) * 0.00010000 BTC
// 12345678 (4) -> 111111101(4) * 0.12345678 BTC
// 50000000 (4) -> 47 (1) * 0.50000000 BTC
// 100000000 (4) -> 9 (1) * 1.00000000 BTC
// 500000000 (5) -> 49 (1) * 5.00000000 BTC
// 1000000000 (5) -> 10 (1) * 10.00000000 BTC
// -----------------------------------------------------------------------------
// compressTxOutAmount compresses the passed amount according to the domain
// specific compression algorithm described above.
func compressTxOutAmount(amount uint64) uint64 {
// No need to do any work if it's zero.
if amount == 0 {
return 0
}
// Find the largest power of 10 (max of 9) that evenly divides the
// value.
exponent := uint64(0)
for amount%10 == 0 && exponent < 9 {
amount /= 10
exponent++
}
// The compressed result for exponents less than 9 is:
// 1 + 10*(9*n + d-1) + e
if exponent < 9 {
lastDigit := amount % 10
amount /= 10
return 1 + 10*(9*amount+lastDigit-1) + exponent
}
// The compressed result for an exponent of 9 is:
// 1 + 10*(n-1) + e == 10 + 10*(n-1)
return 10 + 10*(amount-1)
}
// decompressTxOutAmount returns the original amount the passed compressed
// amount represents according to the domain specific compression algorithm
// described above.
func decompressTxOutAmount(amount uint64) uint64 {
// No need to do any work if it's zero.
if amount == 0 {
return 0
}
// The decompressed amount is either of the following two equations:
// x = 1 + 10*(9*n + d - 1) + e
// x = 1 + 10*(n - 1) + 9
amount--
// The decompressed amount is now one of the following two equations:
// x = 10*(9*n + d - 1) + e
// x = 10*(n - 1) + 9
exponent := amount % 10
amount /= 10
// The decompressed amount is now one of the following two equations:
// x = 9*n + d - 1 | where e < 9
// x = n - 1 | where e = 9
n := uint64(0)
if exponent < 9 {
lastDigit := amount%9 + 1
amount /= 9
n = amount*10 + lastDigit
} else {
n = amount + 1
}
// Apply the exponent.
for ; exponent > 0; exponent-- {
n *= 10
}
return n
}
// -----------------------------------------------------------------------------
// Compressed transaction outputs consist of an amount and a public key script
// both compressed using the domain specific compression algorithms previously
// described.
//
// The serialized format is:
//
// <compressed amount><compressed script>
//
// Field Type Size
// compressed amount VLQ variable
// compressed script []byte variable
// -----------------------------------------------------------------------------
// compressedTxOutSize returns the number of bytes the passed transaction output
// fields would take when encoded with the format described above. The
// preCompressed flag indicates the provided amount and script are already
// compressed. This is useful since loaded utxo entries are not decompressed
// until the output is accessed.
func compressedTxOutSize(amount uint64, pkScript []byte, version int32, preCompressed bool) int {
if preCompressed {
return serializeSizeVLQ(amount) + len(pkScript)
}
return serializeSizeVLQ(compressTxOutAmount(amount)) +
compressedScriptSize(pkScript, version)
}
// putCompressedTxOut potentially compresses the passed amount and script
// according to their domain specific compression algorithms and encodes them
// directly into the passed target byte slice with the format described above.
// The preCompressed flag indicates the provided amount and script are already
// compressed in which case the values are not modified. This is useful since
// loaded utxo entries are not decompressed until the output is accessed. The
// target byte slice must be at least large enough to handle the number of bytes
// returned by the compressedTxOutSize function or it will panic.
func putCompressedTxOut(target []byte, amount uint64, pkScript []byte, version int32, preCompressed bool) int {
if preCompressed {
offset := putVLQ(target, amount)
copy(target[offset:], pkScript)
return offset + len(pkScript)
}
offset := putVLQ(target, compressTxOutAmount(amount))
offset += putCompressedScript(target[offset:], pkScript, version)
return offset
}
// decodeCompressedTxOut decodes the passed compressed txout, possibly followed
// by other data, into its compressed amount and compressed script and returns
// them along with the number of bytes they occupied.
func decodeCompressedTxOut(serialized []byte, version int32) (uint64, []byte, int, error) {
// Deserialize the compressed amount and ensure there are bytes
// remaining for the compressed script.
compressedAmount, bytesRead := deserializeVLQ(serialized)
if bytesRead >= len(serialized) {
return 0, nil, bytesRead, errDeserialize("unexpected end of " +
"data after compressed amount")
}
// Decode the compressed script size and ensure there are enough bytes
// left in the slice for it.
scriptSize := decodeCompressedScriptSize(serialized[bytesRead:], version)
if len(serialized[bytesRead:]) < scriptSize {
return 0, nil, bytesRead, errDeserialize("unexpected end of " +
"data after script size")
}
// Make a copy of the compressed script so the original serialized data
// can be released as soon as possible.
compressedScript := make([]byte, scriptSize)
copy(compressedScript, serialized[bytesRead:bytesRead+scriptSize])
return compressedAmount, compressedScript, bytesRead + scriptSize, nil
}

491
blockchain/compress_test.go Normal file
View file

@ -0,0 +1,491 @@
// Copyright (c) 2015-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
package blockchain
import (
"bytes"
"encoding/hex"
"testing"
)
// hexToBytes converts the passed hex string into bytes and will panic if there
// is an error. This is only provided for the hard-coded constants so errors in
// the source code can be detected. It will only (and must only) be called with
// hard-coded values.
func hexToBytes(s string) []byte {
b, err := hex.DecodeString(s)
if err != nil {
panic("invalid hex in source file: " + s)
}
return b
}
// TestVLQ ensures the variable length quantity serialization, deserialization,
// and size calculation works as expected.
func TestVLQ(t *testing.T) {
t.Parallel()
tests := []struct {
val uint64
serialized []byte
}{
{0, hexToBytes("00")},
{1, hexToBytes("01")},
{127, hexToBytes("7f")},
{128, hexToBytes("8000")},
{129, hexToBytes("8001")},
{255, hexToBytes("807f")},
{256, hexToBytes("8100")},
{16383, hexToBytes("fe7f")},
{16384, hexToBytes("ff00")},
{16511, hexToBytes("ff7f")}, // Max 2-byte value
{16512, hexToBytes("808000")},
{16513, hexToBytes("808001")},
{16639, hexToBytes("80807f")},
{32895, hexToBytes("80ff7f")},
{2113663, hexToBytes("ffff7f")}, // Max 3-byte value
{2113664, hexToBytes("80808000")},
{270549119, hexToBytes("ffffff7f")}, // Max 4-byte value
{270549120, hexToBytes("8080808000")},
{2147483647, hexToBytes("86fefefe7f")},
{2147483648, hexToBytes("86fefeff00")},
{4294967295, hexToBytes("8efefefe7f")}, // Max uint32, 5 bytes
// Max uint64, 10 bytes
{18446744073709551615, hexToBytes("80fefefefefefefefe7f")},
}
for _, test := range tests {
// Ensure the function to calculate the serialized size without
// actually serializing the value is calculated properly.
gotSize := serializeSizeVLQ(test.val)
if gotSize != len(test.serialized) {
t.Errorf("serializeSizeVLQ: did not get expected size "+
"for %d - got %d, want %d", test.val, gotSize,
len(test.serialized))
continue
}
// Ensure the value serializes to the expected bytes.
gotBytes := make([]byte, gotSize)
gotBytesWritten := putVLQ(gotBytes, test.val)
if !bytes.Equal(gotBytes, test.serialized) {
t.Errorf("putVLQUnchecked: did not get expected bytes "+
"for %d - got %x, want %x", test.val, gotBytes,
test.serialized)
continue
}
if gotBytesWritten != len(test.serialized) {
t.Errorf("putVLQUnchecked: did not get expected number "+
"of bytes written for %d - got %d, want %d",
test.val, gotBytesWritten, len(test.serialized))
continue
}
// Ensure the serialized bytes deserialize to the expected
// value.
gotVal, gotBytesRead := deserializeVLQ(test.serialized)
if gotVal != test.val {
t.Errorf("deserializeVLQ: did not get expected value "+
"for %x - got %d, want %d", test.serialized,
gotVal, test.val)
continue
}
if gotBytesRead != len(test.serialized) {
t.Errorf("deserializeVLQ: did not get expected number "+
"of bytes read for %d - got %d, want %d",
test.serialized, gotBytesRead,
len(test.serialized))
continue
}
}
}
// TestScriptCompression ensures the domain-specific script compression and
// decompression works as expected.
func TestScriptCompression(t *testing.T) {
t.Parallel()
tests := []struct {
name string
version int32
uncompressed []byte
compressed []byte
}{
{
name: "nil",
version: 1,
uncompressed: nil,
compressed: hexToBytes("06"),
},
{
name: "pay-to-pubkey-hash 1",
version: 1,
uncompressed: hexToBytes("76a9141018853670f9f3b0582c5b9ee8ce93764ac32b9388ac"),
compressed: hexToBytes("001018853670f9f3b0582c5b9ee8ce93764ac32b93"),
},
{
name: "pay-to-pubkey-hash 2",
version: 1,
uncompressed: hexToBytes("76a914e34cce70c86373273efcc54ce7d2a491bb4a0e8488ac"),
compressed: hexToBytes("00e34cce70c86373273efcc54ce7d2a491bb4a0e84"),
},
{
name: "pay-to-script-hash 1",
version: 1,
uncompressed: hexToBytes("a914da1745e9b549bd0bfa1a569971c77eba30cd5a4b87"),
compressed: hexToBytes("01da1745e9b549bd0bfa1a569971c77eba30cd5a4b"),
},
{
name: "pay-to-script-hash 2",
version: 1,
uncompressed: hexToBytes("a914f815b036d9bbbce5e9f2a00abd1bf3dc91e9551087"),
compressed: hexToBytes("01f815b036d9bbbce5e9f2a00abd1bf3dc91e95510"),
},
{
name: "pay-to-pubkey compressed 0x02",
version: 1,
uncompressed: hexToBytes("2102192d74d0cb94344c9569c2e77901573d8d7903c3ebec3a957724895dca52c6b4ac"),
compressed: hexToBytes("02192d74d0cb94344c9569c2e77901573d8d7903c3ebec3a957724895dca52c6b4"),
},
{
name: "pay-to-pubkey compressed 0x03",
version: 1,
uncompressed: hexToBytes("2103b0bd634234abbb1ba1e986e884185c61cf43e001f9137f23c2c409273eb16e65ac"),
compressed: hexToBytes("03b0bd634234abbb1ba1e986e884185c61cf43e001f9137f23c2c409273eb16e65"),
},
{
name: "pay-to-pubkey uncompressed 0x04 even",
version: 1,
uncompressed: hexToBytes("4104192d74d0cb94344c9569c2e77901573d8d7903c3ebec3a957724895dca52c6b40d45264838c0bd96852662ce6a847b197376830160c6d2eb5e6a4c44d33f453eac"),
compressed: hexToBytes("04192d74d0cb94344c9569c2e77901573d8d7903c3ebec3a957724895dca52c6b4"),
},
{
name: "pay-to-pubkey uncompressed 0x04 odd",
version: 1,
uncompressed: hexToBytes("410411db93e1dcdb8a016b49840f8c53bc1eb68a382e97b1482ecad7b148a6909a5cb2e0eaddfb84ccf9744464f82e160bfa9b8b64f9d4c03f999b8643f656b412a3ac"),
compressed: hexToBytes("0511db93e1dcdb8a016b49840f8c53bc1eb68a382e97b1482ecad7b148a6909a5c"),
},
{
name: "pay-to-pubkey invalid pubkey",
version: 1,
uncompressed: hexToBytes("3302aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaac"),
compressed: hexToBytes("293302aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaac"),
},
{
name: "null data",
version: 1,
uncompressed: hexToBytes("6a200102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20"),
compressed: hexToBytes("286a200102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20"),
},
{
name: "requires 2 size bytes - data push 200 bytes",
version: 1,
uncompressed: append(hexToBytes("4cc8"), bytes.Repeat([]byte{0x00}, 200)...),
// [0x80, 0x50] = 208 as a variable length quantity
// [0x4c, 0xc8] = OP_PUSHDATA1 200
compressed: append(hexToBytes("80504cc8"), bytes.Repeat([]byte{0x00}, 200)...),
},
}
for _, test := range tests {
// Ensure the function to calculate the serialized size without
// actually serializing the value is calculated properly.
gotSize := compressedScriptSize(test.uncompressed, test.version)
if gotSize != len(test.compressed) {
t.Errorf("compressedScriptSize (%s): did not get "+
"expected size - got %d, want %d", test.name,
gotSize, len(test.compressed))
continue
}
// Ensure the script compresses to the expected bytes.
gotCompressed := make([]byte, gotSize)
gotBytesWritten := putCompressedScript(gotCompressed,
test.uncompressed, test.version)
if !bytes.Equal(gotCompressed, test.compressed) {
t.Errorf("putCompressedScript (%s): did not get "+
"expected bytes - got %x, want %x", test.name,
gotCompressed, test.compressed)
continue
}
if gotBytesWritten != len(test.compressed) {
t.Errorf("putCompressedScript (%s): did not get "+
"expected number of bytes written - got %d, "+
"want %d", test.name, gotBytesWritten,
len(test.compressed))
continue
}
// Ensure the compressed script size is properly decoded from
// the compressed script.
gotDecodedSize := decodeCompressedScriptSize(test.compressed,
test.version)
if gotDecodedSize != len(test.compressed) {
t.Errorf("decodeCompressedScriptSize (%s): did not get "+
"expected size - got %d, want %d", test.name,
gotDecodedSize, len(test.compressed))
continue
}
// Ensure the script decompresses to the expected bytes.
gotDecompressed := decompressScript(test.compressed, test.version)
if !bytes.Equal(gotDecompressed, test.uncompressed) {
t.Errorf("decompressScript (%s): did not get expected "+
"bytes - got %x, want %x", test.name,
gotDecompressed, test.uncompressed)
continue
}
}
}
// TestScriptCompressionErrors ensures calling various functions related to
// script compression with incorrect data returns the expected results.
func TestScriptCompressionErrors(t *testing.T) {
t.Parallel()
// A nil script must result in a decoded size of 0.
if gotSize := decodeCompressedScriptSize(nil, 1); gotSize != 0 {
t.Fatalf("decodeCompressedScriptSize with nil script did not "+
"return 0 - got %d", gotSize)
}
// A nil script must result in a nil decompressed script.
if gotScript := decompressScript(nil, 1); gotScript != nil {
t.Fatalf("decompressScript with nil script did not return nil "+
"decompressed script - got %x", gotScript)
}
// A compressed script for a pay-to-pubkey (uncompressed) that results
// in an invalid pubkey must result in a nil decompressed script.
compressedScript := hexToBytes("04012d74d0cb94344c9569c2e77901573d8d" +
"7903c3ebec3a957724895dca52c6b4")
if gotScript := decompressScript(compressedScript, 1); gotScript != nil {
t.Fatalf("decompressScript with compressed pay-to-"+
"uncompressed-pubkey that is invalid did not return "+
"nil decompressed script - got %x", gotScript)
}
}
// TestAmountCompression ensures the domain-specific transaction output amount
// compression and decompression works as expected.
func TestAmountCompression(t *testing.T) {
t.Parallel()
tests := []struct {
name string
uncompressed uint64
compressed uint64
}{
{
name: "0 BTC (sometimes used in nulldata)",
uncompressed: 0,
compressed: 0,
},
{
name: "546 Satoshi (current network dust value)",
uncompressed: 546,
compressed: 4911,
},
{
name: "0.00001 BTC (typical transaction fee)",
uncompressed: 1000,
compressed: 4,
},
{
name: "0.0001 BTC (typical transaction fee)",
uncompressed: 10000,
compressed: 5,
},
{
name: "0.12345678 BTC",
uncompressed: 12345678,
compressed: 111111101,
},
{
name: "0.5 BTC",
uncompressed: 50000000,
compressed: 48,
},
{
name: "1 BTC",
uncompressed: 100000000,
compressed: 9,
},
{
name: "5 BTC",
uncompressed: 500000000,
compressed: 49,
},
{
name: "21000000 BTC (max minted coins)",
uncompressed: 2100000000000000,
compressed: 21000000,
},
}
for _, test := range tests {
// Ensure the amount compresses to the expected value.
gotCompressed := compressTxOutAmount(test.uncompressed)
if gotCompressed != test.compressed {
t.Errorf("compressTxOutAmount (%s): did not get "+
"expected value - got %d, want %d", test.name,
gotCompressed, test.compressed)
continue
}
// Ensure the value decompresses to the expected value.
gotDecompressed := decompressTxOutAmount(test.compressed)
if gotDecompressed != test.uncompressed {
t.Errorf("decompressTxOutAmount (%s): did not get "+
"expected value - got %d, want %d", test.name,
gotDecompressed, test.uncompressed)
continue
}
}
}
// TestCompressedTxOut ensures the transaction output serialization and
// deserialization works as expected.
func TestCompressedTxOut(t *testing.T) {
t.Parallel()
tests := []struct {
name string
amount uint64
compAmount uint64
pkScript []byte
compPkScript []byte
version int32
compressed []byte
}{
{
name: "nulldata with 0 BTC",
amount: 0,
compAmount: 0,
pkScript: hexToBytes("6a200102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20"),
compPkScript: hexToBytes("286a200102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20"),
version: 1,
compressed: hexToBytes("00286a200102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20"),
},
{
name: "pay-to-pubkey-hash dust",
amount: 546,
compAmount: 4911,
pkScript: hexToBytes("76a9141018853670f9f3b0582c5b9ee8ce93764ac32b9388ac"),
compPkScript: hexToBytes("001018853670f9f3b0582c5b9ee8ce93764ac32b93"),
version: 1,
compressed: hexToBytes("a52f001018853670f9f3b0582c5b9ee8ce93764ac32b93"),
},
{
name: "pay-to-pubkey uncompressed 1 BTC",
amount: 100000000,
compAmount: 9,
pkScript: hexToBytes("4104192d74d0cb94344c9569c2e77901573d8d7903c3ebec3a957724895dca52c6b40d45264838c0bd96852662ce6a847b197376830160c6d2eb5e6a4c44d33f453eac"),
compPkScript: hexToBytes("04192d74d0cb94344c9569c2e77901573d8d7903c3ebec3a957724895dca52c6b4"),
version: 1,
compressed: hexToBytes("0904192d74d0cb94344c9569c2e77901573d8d7903c3ebec3a957724895dca52c6b4"),
},
}
for _, test := range tests {
// Ensure the function to calculate the serialized size without
// actually serializing the txout is calculated properly.
gotSize := compressedTxOutSize(test.amount, test.pkScript,
test.version, false)
if gotSize != len(test.compressed) {
t.Errorf("compressedTxOutSize (%s): did not get "+
"expected size - got %d, want %d", test.name,
gotSize, len(test.compressed))
continue
}
// Ensure the txout compresses to the expected value.
gotCompressed := make([]byte, gotSize)
gotBytesWritten := putCompressedTxOut(gotCompressed,
test.amount, test.pkScript, test.version, false)
if !bytes.Equal(gotCompressed, test.compressed) {
t.Errorf("compressTxOut (%s): did not get expected "+
"bytes - got %x, want %x", test.name,
gotCompressed, test.compressed)
continue
}
if gotBytesWritten != len(test.compressed) {
t.Errorf("compressTxOut (%s): did not get expected "+
"number of bytes written - got %d, want %d",
test.name, gotBytesWritten,
len(test.compressed))
continue
}
// Ensure the serialized bytes are decoded back to the expected
// compressed values.
gotAmount, gotScript, gotBytesRead, err := decodeCompressedTxOut(
test.compressed, test.version)
if err != nil {
t.Errorf("decodeCompressedTxOut (%s): unexpected "+
"error: %v", test.name, err)
continue
}
if gotAmount != test.compAmount {
t.Errorf("decodeCompressedTxOut (%s): did not get "+
"expected amount - got %d, want %d",
test.name, gotAmount, test.compAmount)
continue
}
if !bytes.Equal(gotScript, test.compPkScript) {
t.Errorf("decodeCompressedTxOut (%s): did not get "+
"expected script - got %x, want %x",
test.name, gotScript, test.compPkScript)
continue
}
if gotBytesRead != len(test.compressed) {
t.Errorf("decodeCompressedTxOut (%s): did not get "+
"expected number of bytes read - got %d, want %d",
test.name, gotBytesRead, len(test.compressed))
continue
}
// Ensure the compressed values decompress to the expected
// txout.
gotAmount = decompressTxOutAmount(gotAmount)
if gotAmount != test.amount {
t.Errorf("decompressTxOut (%s): did not get expected "+
"value - got %d, want %d", test.name, gotAmount,
test.amount)
continue
}
gotScript = decompressScript(gotScript, test.version)
if !bytes.Equal(gotScript, test.pkScript) {
t.Errorf("decompressTxOut (%s): did not get expected "+
"script - got %x, want %x", test.name,
gotScript, test.pkScript)
continue
}
}
}
// TestTxOutCompressionErrors ensures calling various functions related to
// txout compression with incorrect data returns the expected results.
func TestTxOutCompressionErrors(t *testing.T) {
t.Parallel()
// A compressed txout with missing compressed script must error.
compressedTxOut := hexToBytes("00")
_, _, _, err := decodeCompressedTxOut(compressedTxOut, 1)
if !isDeserializeErr(err) {
t.Fatalf("decodeCompressedTxOut with missing compressed script "+
"did not return expected error type - got %T, want "+
"errDeserialize", err)
}
// A compressed txout with short compressed script must error.
compressedTxOut = hexToBytes("0010")
_, _, _, err = decodeCompressedTxOut(compressedTxOut, 1)
if !isDeserializeErr(err) {
t.Fatalf("decodeCompressedTxOut with short compressed script "+
"did not return expected error type - got %T, want "+
"errDeserialize", err)
}
}

16
blockchain/difficulty.go
View file

@ -1,11 +1,10 @@
// Copyright (c) 2013-2014 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
package blockchain
import (
"fmt"
"math/big"
"time"
@ -222,6 +221,8 @@ func (b *BlockChain) calcEasiestDifficulty(bits uint32, duration time.Duration)
// findPrevTestNetDifficulty returns the difficulty of the previous block which
// did not have the special testnet minimum difficulty rule applied.
//
// This function MUST be called with the chain state lock held (for writes).
func (b *BlockChain) findPrevTestNetDifficulty(startNode *blockNode) (uint32, error) {
// Search backwards through the chain for the last block without
// the special rule applied.
@ -256,6 +257,8 @@ func (b *BlockChain) findPrevTestNetDifficulty(startNode *blockNode) (uint32, er
// This function differs from the exported CalcNextRequiredDifficulty in that
// the exported version uses the current best chain as the previous block node
// while this function accepts any block node.
//
// This function MUST be called with the chain state lock held (for writes).
func (b *BlockChain) calcNextRequiredDifficulty(lastNode *blockNode, newBlockTime time.Time) (uint32, error) {
// Genesis block.
if lastNode == nil {
@ -309,7 +312,7 @@ func (b *BlockChain) calcNextRequiredDifficulty(lastNode *blockNode, newBlockTim
}
if firstNode == nil {
return 0, fmt.Errorf("unable to obtain previous retarget block")
return 0, AssertError("unable to obtain previous retarget block")
}
// Limit the amount of adjustment that can occur to the previous
@ -355,7 +358,10 @@ func (b *BlockChain) calcNextRequiredDifficulty(lastNode *blockNode, newBlockTim
// after the end of the current best chain based on the difficulty retarget
// rules.
//
// This function is NOT safe for concurrent access.
// This function is safe for concurrent access.
func (b *BlockChain) CalcNextRequiredDifficulty(timestamp time.Time) (uint32, error) {
return b.calcNextRequiredDifficulty(b.bestChain, timestamp)
b.chainLock.Lock()
difficulty, err := b.calcNextRequiredDifficulty(b.bestNode, timestamp)
b.chainLock.Unlock()
return difficulty, err
}

12
blockchain/error.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2014 The btcsuite developers
// Copyright (c) 2014-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -8,6 +8,16 @@ import (
"fmt"
)
// AssertError identifies an error that indicates an internal code consistency
// issue and should be treated as a critical and unrecoverable error.
type AssertError string
// Error returns the assertion error as a huma-readable string and satisfies
// the error interface.
func (e AssertError) Error() string {
return "assertion failed: " + string(e)
}
// ErrorCode identifies a kind of error.
type ErrorCode int

41
blockchain/example_test.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2014 The btcsuite developers
// Copyright (c) 2014-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -7,11 +7,13 @@ package blockchain_test
import (
"fmt"
"math/big"
"os"
"path/filepath"
"github.com/btcsuite/btcd/blockchain"
"github.com/btcsuite/btcd/chaincfg"
"github.com/btcsuite/btcd/database"
_ "github.com/btcsuite/btcd/database/memdb"
database "github.com/btcsuite/btcd/database2"
_ "github.com/btcsuite/btcd/database2/ffldb"
"github.com/btcsuite/btcutil"
)
@ -22,31 +24,33 @@ import (
// block to illustrate how an invalid block is handled.
func ExampleBlockChain_ProcessBlock() {
// Create a new database to store the accepted blocks into. Typically
// this would be opening an existing database and would not use memdb
// which is a memory-only database backend, but we create a new db
// here so this is a complete working example.
db, err := database.CreateDB("memdb")
// this would be opening an existing database and would not be deleting
// and creating a new database like this, but it is done here so this is
// a complete working example and does not leave temporary files laying
// around.
dbPath := filepath.Join(os.TempDir(), "exampleprocessblock")
_ = os.RemoveAll(dbPath)
db, err := database.Create("ffldb", dbPath, chaincfg.MainNetParams.Net)
if err != nil {
fmt.Printf("Failed to create database: %v\n", err)
return
}
defer os.RemoveAll(dbPath)
defer db.Close()
// Insert the main network genesis block. This is part of the initial
// database setup. Like above, this typically would not be needed when
// opening an existing database.
genesisBlock := btcutil.NewBlock(chaincfg.MainNetParams.GenesisBlock)
_, err = db.InsertBlock(genesisBlock)
// Create a new BlockChain instance using the underlying database for
// the main bitcoin network. This example does not demonstrate some
// of the other available configuration options such as specifying a
// notification callback and signature cache.
chain, err := blockchain.New(&blockchain.Config{
DB: db,
ChainParams: &chaincfg.MainNetParams,
})
if err != nil {
fmt.Printf("Failed to insert genesis block: %v\n", err)
fmt.Printf("Failed to create chain instance: %v\n", err)
return
}
// Create a new BlockChain instance without an initialized signature
// verification cache, using the underlying database for the main
// bitcoin network and ignore notifications.
chain := blockchain.New(db, &chaincfg.MainNetParams, nil, nil)
// Create a new median time source that is required by the upcoming
// call to ProcessBlock. Ordinarily this would also add time values
// obtained from other peers on the network so the local time is
@ -56,6 +60,7 @@ func ExampleBlockChain_ProcessBlock() {
// Process a block. For this example, we are going to intentionally
// cause an error by trying to process the genesis block which already
// exists.
genesisBlock := btcutil.NewBlock(chaincfg.MainNetParams.GenesisBlock)
isOrphan, err := chain.ProcessBlock(genesisBlock, timeSource, blockchain.BFNone)
if err != nil {
fmt.Printf("Failed to process block: %v\n", err)

6
blockchain/internal_test.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2013-2014 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -42,3 +42,7 @@ func TstSetMaxMedianTimeEntries(val int) {
// TstCheckBlockScripts makes the internal checkBlockScripts function available
// to the test package.
var TstCheckBlockScripts = checkBlockScripts
// TstDeserializeUtxoEntry makes the internal deserializeUtxoEntry function
// available to the test package.
var TstDeserializeUtxoEntry = deserializeUtxoEntry

2
blockchain/notifications.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2013-2014 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.

22
blockchain/process.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2013-2014 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -7,6 +7,7 @@ package blockchain
import (
"fmt"
database "github.com/btcsuite/btcd/database2"
"github.com/btcsuite/btcd/wire"
"github.com/btcsuite/btcutil"
)
@ -38,14 +39,22 @@ const (
// blockExists determines whether a block with the given hash exists either in
// the main chain or any side chains.
//
// This function MUST be called with the chain state lock held (for reads).
func (b *BlockChain) blockExists(hash *wire.ShaHash) (bool, error) {
// Check memory chain first (could be main chain or side chain blocks).
if _, ok := b.index[*hash]; ok {
return true, nil
}
// Check in database (rest of main chain not in memory).
return b.db.ExistsSha(hash)
// Check in the database.
var exists bool
err := b.db.View(func(dbTx database.Tx) error {
var err error
exists, err = dbTx.HasBlock(hash)
return err
})
return exists, err
}
// processOrphans determines if there are any orphans which depend on the passed
@ -55,6 +64,8 @@ func (b *BlockChain) blockExists(hash *wire.ShaHash) (bool, error) {
//
// The flags do not modify the behavior of this function directly, however they
// are needed to pass along to maybeAcceptBlock.
//
// This function MUST be called with the chain state lock held (for writes).
func (b *BlockChain) processOrphans(hash *wire.ShaHash, flags BehaviorFlags) error {
// Start with processing at least the passed hash. Leave a little room
// for additional orphan blocks that need to be processed without
@ -112,7 +123,12 @@ func (b *BlockChain) processOrphans(hash *wire.ShaHash, flags BehaviorFlags) err
// It returns a bool which indicates whether or not the block is an orphan and
// any errors that occurred during processing. The returned bool is only valid
// when the error is nil.
//
// This function is safe for concurrent access.
func (b *BlockChain) ProcessBlock(block *btcutil.Block, timeSource MedianTimeSource, flags BehaviorFlags) (bool, error) {
b.chainLock.Lock()
defer b.chainLock.Unlock()
fastAdd := flags&BFFastAdd == BFFastAdd
dryRun := flags&BFDryRun == BFDryRun

45
blockchain/scriptval.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2013-2014 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -28,7 +28,7 @@ type txValidator struct {
validateChan chan *txValidateItem
quitChan chan struct{}
resultChan chan error
txStore TxStore
utxoView *UtxoViewpoint
flags txscript.ScriptFlags
sigCache *txscript.SigCache
}
@ -55,8 +55,9 @@ out:
// Ensure the referenced input transaction is available.
txIn := txVI.txIn
originTxHash := &txIn.PreviousOutPoint.Hash
originTx, exists := v.txStore[*originTxHash]
if !exists || originTx.Err != nil || originTx.Tx == nil {
originTxIndex := txIn.PreviousOutPoint.Index
txEntry := v.utxoView.LookupEntry(originTxHash)
if txEntry == nil {
str := fmt.Sprintf("unable to find input "+
"transaction %v referenced from "+
"transaction %v", originTxHash,
@ -65,17 +66,16 @@ out:
v.sendResult(err)
break out
}
originMsgTx := originTx.Tx.MsgTx()
// Ensure the output index in the referenced transaction
// is available.
originTxIndex := txIn.PreviousOutPoint.Index
if originTxIndex >= uint32(len(originMsgTx.TxOut)) {
str := fmt.Sprintf("out of bounds "+
"input index %d in transaction %v "+
"referenced from transaction %v",
originTxIndex, originTxHash,
txVI.tx.Sha())
// Ensure the referenced input transaction public key
// script is available.
pkScript := txEntry.PkScriptByIndex(originTxIndex)
if pkScript == nil {
str := fmt.Sprintf("unable to find unspent "+
"output %v script referenced from "+
"transaction %s:%d",
txIn.PreviousOutPoint, txVI.tx.Sha(),
txVI.txInIndex)
err := ruleError(ErrBadTxInput, str)
v.sendResult(err)
break out
@ -83,7 +83,6 @@ out:
// Create a new script engine for the script pair.
sigScript := txIn.SignatureScript
pkScript := originMsgTx.TxOut[originTxIndex].PkScript
vm, err := txscript.NewEngine(pkScript, txVI.tx.MsgTx(),
txVI.txInIndex, v.flags, v.sigCache)
if err != nil {
@ -180,12 +179,12 @@ func (v *txValidator) Validate(items []*txValidateItem) error {
// newTxValidator returns a new instance of txValidator to be used for
// validating transaction scripts asynchronously.
func newTxValidator(txStore TxStore, flags txscript.ScriptFlags, sigCache *txscript.SigCache) *txValidator {
func newTxValidator(utxoView *UtxoViewpoint, flags txscript.ScriptFlags, sigCache *txscript.SigCache) *txValidator {
return &txValidator{
validateChan: make(chan *txValidateItem),
quitChan: make(chan struct{}),
resultChan: make(chan error),
txStore: txStore,
utxoView: utxoView,
sigCache: sigCache,
flags: flags,
}
@ -193,7 +192,7 @@ func newTxValidator(txStore TxStore, flags txscript.ScriptFlags, sigCache *txscr
// ValidateTransactionScripts validates the scripts for the passed transaction
// using multiple goroutines.
func ValidateTransactionScripts(tx *btcutil.Tx, txStore TxStore, flags txscript.ScriptFlags, sigCache *txscript.SigCache) error {
func ValidateTransactionScripts(tx *btcutil.Tx, utxoView *UtxoViewpoint, flags txscript.ScriptFlags, sigCache *txscript.SigCache) error {
// Collect all of the transaction inputs and required information for
// validation.
txIns := tx.MsgTx().TxIn
@ -213,7 +212,7 @@ func ValidateTransactionScripts(tx *btcutil.Tx, txStore TxStore, flags txscript.
}
// Validate all of the inputs.
validator := newTxValidator(txStore, flags, sigCache)
validator := newTxValidator(utxoView, flags, sigCache)
if err := validator.Validate(txValItems); err != nil {
return err
}
@ -222,10 +221,8 @@ func ValidateTransactionScripts(tx *btcutil.Tx, txStore TxStore, flags txscript.
}
// checkBlockScripts executes and validates the scripts for all transactions in
// the passed block.
func checkBlockScripts(block *btcutil.Block, txStore TxStore,
scriptFlags txscript.ScriptFlags, sigCache *txscript.SigCache) error {
// the passed block using multiple goroutines.
func checkBlockScripts(block *btcutil.Block, utxoView *UtxoViewpoint, scriptFlags txscript.ScriptFlags, sigCache *txscript.SigCache) error {
// Collect all of the transaction inputs and required information for
// validation for all transactions in the block into a single slice.
numInputs := 0
@ -250,7 +247,7 @@ func checkBlockScripts(block *btcutil.Block, txStore TxStore,
}
// Validate all of the inputs.
validator := newTxValidator(txStore, scriptFlags, sigCache)
validator := newTxValidator(utxoView, scriptFlags, sigCache)
if err := validator.Validate(txValItems); err != nil {
return err
}

12
blockchain/scriptval_test.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2013-2015 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -29,18 +29,18 @@ func TestCheckBlockScripts(t *testing.T) {
t.Errorf("The test block file must only have one block in it")
}
txStoreDataFile := fmt.Sprintf("%d.txstore.bz2", testBlockNum)
txStore, err := loadTxStore(txStoreDataFile)
storeDataFile := fmt.Sprintf("%d.utxostore.bz2", testBlockNum)
view, err := loadUtxoView(storeDataFile)
if err != nil {
t.Errorf("Error loading txstore: %v\n", err)
return
}
scriptFlags := txscript.ScriptBip16
err = blockchain.TstCheckBlockScripts(blocks[0], txStore, scriptFlags, nil)
err = blockchain.TstCheckBlockScripts(blocks[0], view, scriptFlags,
nil)
if err != nil {
t.Errorf("Transaction script validation failed: %v\n",
err)
t.Errorf("Transaction script validation failed: %v\n", err)
return
}
}

BIN
blockchain/testdata/277647.txstore.bz2 vendored
View file

Binary file not shown.

BIN
blockchain/testdata/277647.utxostore.bz2 vendored Normal file
View file

Binary file not shown.

318
blockchain/txlookup.go
View file

@ -1,318 +0,0 @@
// Copyright (c) 2013-2014 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
package blockchain
import (
"fmt"
"github.com/btcsuite/btcd/database"
"github.com/btcsuite/btcd/wire"
"github.com/btcsuite/btcutil"
)
// TxData contains contextual information about transactions such as which block
// they were found in and whether or not the outputs are spent.
type TxData struct {
Tx *btcutil.Tx
Hash *wire.ShaHash
BlockHeight int32
Spent []bool
Err error
}
// TxStore is used to store transactions needed by other transactions for things
// such as script validation and double spend prevention. This also allows the
// transaction data to be treated as a view since it can contain the information
// from the point-of-view of different points in the chain.
type TxStore map[wire.ShaHash]*TxData
// connectTransactions updates the passed map by applying transaction and
// spend information for all the transactions in the passed block. Only
// transactions in the passed map are updated.
func connectTransactions(txStore TxStore, block *btcutil.Block) error {
// Loop through all of the transactions in the block to see if any of
// them are ones we need to update and spend based on the results map.
for _, tx := range block.Transactions() {
// Update the transaction store with the transaction information
// if it's one of the requested transactions.
msgTx := tx.MsgTx()
if txD, exists := txStore[*tx.Sha()]; exists {
txD.Tx = tx
txD.BlockHeight = block.Height()
txD.Spent = make([]bool, len(msgTx.TxOut))
txD.Err = nil
}
// Spend the origin transaction output.
for _, txIn := range msgTx.TxIn {
originHash := &txIn.PreviousOutPoint.Hash
originIndex := txIn.PreviousOutPoint.Index
if originTx, exists := txStore[*originHash]; exists {
if originIndex > uint32(len(originTx.Spent)) {
continue
}
originTx.Spent[originIndex] = true
}
}
}
return nil
}
// disconnectTransactions updates the passed map by undoing transaction and
// spend information for all transactions in the passed block. Only
// transactions in the passed map are updated.
func disconnectTransactions(txStore TxStore, block *btcutil.Block) error {
// Loop through all of the transactions in the block to see if any of
// them are ones that need to be undone based on the transaction store.
for _, tx := range block.Transactions() {
// Clear this transaction from the transaction store if needed.
// Only clear it rather than deleting it because the transaction
// connect code relies on its presence to decide whether or not
// to update the store and any transactions which exist on both
// sides of a fork would otherwise not be updated.
if txD, exists := txStore[*tx.Sha()]; exists {
txD.Tx = nil
txD.BlockHeight = 0
txD.Spent = nil
txD.Err = database.ErrTxShaMissing
}
// Unspend the origin transaction output.
for _, txIn := range tx.MsgTx().TxIn {
originHash := &txIn.PreviousOutPoint.Hash
originIndex := txIn.PreviousOutPoint.Index
originTx, exists := txStore[*originHash]
if exists && originTx.Tx != nil && originTx.Err == nil {
if originIndex > uint32(len(originTx.Spent)) {
continue
}
originTx.Spent[originIndex] = false
}
}
}
return nil
}
// fetchTxStoreMain fetches transaction data about the provided set of
// transactions from the point of view of the end of the main chain. It takes
// a flag which specifies whether or not fully spent transaction should be
// included in the results.
func fetchTxStoreMain(db database.Db, txSet map[wire.ShaHash]struct{}, includeSpent bool) TxStore {
// Just return an empty store now if there are no requested hashes.
txStore := make(TxStore)
if len(txSet) == 0 {
return txStore
}
// The transaction store map needs to have an entry for every requested
// transaction. By default, all the transactions are marked as missing.
// Each entry will be filled in with the appropriate data below.
txList := make([]*wire.ShaHash, 0, len(txSet))
for hash := range txSet {
hashCopy := hash
txStore[hash] = &TxData{Hash: &hashCopy, Err: database.ErrTxShaMissing}
txList = append(txList, &hashCopy)
}
// Ask the database (main chain) for the list of transactions. This
// will return the information from the point of view of the end of the
// main chain. Choose whether or not to include fully spent
// transactions depending on the passed flag.
var txReplyList []*database.TxListReply
if includeSpent {
txReplyList = db.FetchTxByShaList(txList)
} else {
txReplyList = db.FetchUnSpentTxByShaList(txList)
}
for _, txReply := range txReplyList {
// Lookup the existing results entry to modify. Skip
// this reply if there is no corresponding entry in
// the transaction store map which really should not happen, but
// be safe.
txD, ok := txStore[*txReply.Sha]
if !ok {
continue
}
// Fill in the transaction details. A copy is used here since
// there is no guarantee the returned data isn't cached and
// this code modifies the data. A bug caused by modifying the
// cached data would likely be difficult to track down and could
// cause subtle errors, so avoid the potential altogether.
txD.Err = txReply.Err
if txReply.Err == nil {
txD.Tx = btcutil.NewTx(txReply.Tx)
txD.BlockHeight = txReply.Height
txD.Spent = make([]bool, len(txReply.TxSpent))
copy(txD.Spent, txReply.TxSpent)
}
}
return txStore
}
// fetchTxStore fetches transaction data about the provided set of transactions
// from the point of view of the given node. For example, a given node might
// be down a side chain where a transaction hasn't been spent from its point of
// view even though it might have been spent in the main chain (or another side
// chain). Another scenario is where a transaction exists from the point of
// view of the main chain, but doesn't exist in a side chain that branches
// before the block that contains the transaction on the main chain.
func (b *BlockChain) fetchTxStore(node *blockNode, txSet map[wire.ShaHash]struct{}) (TxStore, error) {
// Get the previous block node. This function is used over simply
// accessing node.parent directly as it will dynamically create previous
// block nodes as needed. This helps allow only the pieces of the chain
// that are needed to remain in memory.
prevNode, err := b.getPrevNodeFromNode(node)
if err != nil {
return nil, err
}
// If we haven't selected a best chain yet or we are extending the main
// (best) chain with a new block, fetch the requested set from the point
// of view of the end of the main (best) chain without including fully
// spent transactions in the results. This is a little more efficient
// since it means less transaction lookups are needed.
if b.bestChain == nil || (prevNode != nil && prevNode.hash.IsEqual(b.bestChain.hash)) {
txStore := fetchTxStoreMain(b.db, txSet, false)
return txStore, nil
}
// Fetch the requested set from the point of view of the end of the
// main (best) chain including fully spent transactions. The fully
// spent transactions are needed because the following code unspends
// them to get the correct point of view.
txStore := fetchTxStoreMain(b.db, txSet, true)
// The requested node is either on a side chain or is a node on the main
// chain before the end of it. In either case, we need to undo the
// transactions and spend information for the blocks which would be
// disconnected during a reorganize to the point of view of the
// node just before the requested node.
detachNodes, attachNodes := b.getReorganizeNodes(prevNode)
for e := detachNodes.Front(); e != nil; e = e.Next() {
n := e.Value.(*blockNode)
block, err := b.db.FetchBlockBySha(n.hash)
if err != nil {
return nil, err
}
disconnectTransactions(txStore, block)
}
// The transaction store is now accurate to either the node where the
// requested node forks off the main chain (in the case where the
// requested node is on a side chain), or the requested node itself if
// the requested node is an old node on the main chain. Entries in the
// attachNodes list indicate the requested node is on a side chain, so
// if there are no nodes to attach, we're done.
if attachNodes.Len() == 0 {
return txStore, nil
}
// The requested node is on a side chain, so we need to apply the
// transactions and spend information from each of the nodes to attach.
for e := attachNodes.Front(); e != nil; e = e.Next() {
n := e.Value.(*blockNode)
block, exists := b.blockCache[*n.hash]
if !exists {
return nil, fmt.Errorf("unable to find block %v in "+
"side chain cache for transaction search",
n.hash)
}
connectTransactions(txStore, block)
}
return txStore, nil
}
// fetchInputTransactions fetches the input transactions referenced by the
// transactions in the given block from its point of view. See fetchTxList
// for more details on what the point of view entails.
func (b *BlockChain) fetchInputTransactions(node *blockNode, block *btcutil.Block) (TxStore, error) {
// Build a map of in-flight transactions because some of the inputs in
// this block could be referencing other transactions earlier in this
// block which are not yet in the chain.
txInFlight := map[wire.ShaHash]int{}
transactions := block.Transactions()
for i, tx := range transactions {
txInFlight[*tx.Sha()] = i
}
// Loop through all of the transaction inputs (except for the coinbase
// which has no inputs) collecting them into sets of what is needed and
// what is already known (in-flight).
txNeededSet := make(map[wire.ShaHash]struct{})
txStore := make(TxStore)
for i, tx := range transactions[1:] {
for _, txIn := range tx.MsgTx().TxIn {
// Add an entry to the transaction store for the needed
// transaction with it set to missing by default.
originHash := &txIn.PreviousOutPoint.Hash
txD := &TxData{Hash: originHash, Err: database.ErrTxShaMissing}
txStore[*originHash] = txD
// It is acceptable for a transaction input to reference
// the output of another transaction in this block only
// if the referenced transaction comes before the
// current one in this block. Update the transaction
// store acccordingly when this is the case. Otherwise,
// we still need the transaction.
//
// NOTE: The >= is correct here because i is one less
// than the actual position of the transaction within
// the block due to skipping the coinbase.
if inFlightIndex, ok := txInFlight[*originHash]; ok &&
i >= inFlightIndex {
originTx := transactions[inFlightIndex]
txD.Tx = originTx
txD.BlockHeight = node.height
txD.Spent = make([]bool, len(originTx.MsgTx().TxOut))
txD.Err = nil
} else {
txNeededSet[*originHash] = struct{}{}
}
}
}
// Request the input transactions from the point of view of the node.
txNeededStore, err := b.fetchTxStore(node, txNeededSet)
if err != nil {
return nil, err
}
// Merge the results of the requested transactions and the in-flight
// transactions.
for _, txD := range txNeededStore {
txStore[*txD.Hash] = txD
}
return txStore, nil
}
// FetchTransactionStore fetches the input transactions referenced by the
// passed transaction from the point of view of the end of the main chain. It
// also attempts to fetch the transaction itself so the returned TxStore can be
// examined for duplicate transactions.
func (b *BlockChain) FetchTransactionStore(tx *btcutil.Tx, includeSpent bool) (TxStore, error) {
// Create a set of needed transactions from the transactions referenced
// by the inputs of the passed transaction. Also, add the passed
// transaction itself as a way for the caller to detect duplicates.
txNeededSet := make(map[wire.ShaHash]struct{})
txNeededSet[*tx.Sha()] = struct{}{}
for _, txIn := range tx.MsgTx().TxIn {
txNeededSet[txIn.PreviousOutPoint.Hash] = struct{}{}
}
// Request the input transactions from the point of view of the end of
// the main chain with or without without including fully spent transactions
// in the results.
txStore := fetchTxStoreMain(b.db, txNeededSet, includeSpent)
return txStore, nil
}

611
blockchain/utxoviewpoint.go Normal file
View file

@ -0,0 +1,611 @@
// Copyright (c) 2015-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
package blockchain
import (
"fmt"
database "github.com/btcsuite/btcd/database2"
"github.com/btcsuite/btcd/txscript"
"github.com/btcsuite/btcd/wire"
"github.com/btcsuite/btcutil"
)
// utxoOutput houses details about an individual unspent transaction output such
// as whether or not it is spent, its public key script, and how much it pays.
//
// Standard public key scripts are stored in the database using a compressed
// format. Since the vast majority of scripts are of the standard form, a fairly
// significant savings is achieved by discarding the portions of the standard
// scripts that can be reconstructed.
//
// Also, since it is common for only a specific output in a given utxo entry to
// be referenced from a redeeming transaction, the script and amount for a given
// output is not uncompressed until the first time it is accessed. This
// provides a mechanism to avoid the overhead of needlessly uncompressing all
// outputs for a given utxo entry at the time of load.
type utxoOutput struct {
spent bool // Output is spent.
compressed bool // The amount and public key script are compressed.
amount int64 // The amount of the output.
pkScript []byte // The public key script for the output.
}
// maybeDecompress decompresses the amount and public key script fields of the
// utxo and marks it decompressed if needed.
func (o *utxoOutput) maybeDecompress(version int32) {
// Nothing to do if it's not compressed.
if !o.compressed {
return
}
o.amount = int64(decompressTxOutAmount(uint64(o.amount)))
o.pkScript = decompressScript(o.pkScript, version)
o.compressed = false
}
// UtxoEntry contains contextual information about an unspent transaction such
// as whether or not it is a coinbase transaction, which block it was found in,
// and the spent status of its outputs.
type UtxoEntry struct {
modified bool // Entry changed since load.
version int32 // The version of this tx.
isCoinBase bool // Whether entry is a coinbase tx.
blockHeight int32 // Height of block containing tx.
sparseOutputs map[uint32]*utxoOutput // Sparse map of unspent outputs.
}
// Version returns the version of the transaction the utxo represents.
func (entry *UtxoEntry) Version() int32 {
return entry.version
}
// IsCoinBase returns whether or not the transaction the utxo entry represents
// is a coinbase.
func (entry *UtxoEntry) IsCoinBase() bool {
return entry.isCoinBase
}
// BlockHeight returns the height of the block containing the transaction the
// utxo entry represents.
func (entry *UtxoEntry) BlockHeight() int32 {
return entry.blockHeight
}
// IsOutputSpent returns whether or not the provided output index has been
// spent based upon the current state of the unspent transaction output view
// the entry was obtained from.
//
// Returns true if the output index references an output that does not exist
// either due to it being invalid or because the output is not part of the view
// due to previously being spent/pruned.
func (entry *UtxoEntry) IsOutputSpent(outputIndex uint32) bool {
output, ok := entry.sparseOutputs[outputIndex]
if !ok {
return true
}
return output.spent
}
// SpendOutput marks the output at the provided index as spent. Specifying an
// output index that does not exist will not have any effect.
func (entry *UtxoEntry) SpendOutput(outputIndex uint32) {
output, ok := entry.sparseOutputs[outputIndex]
if !ok {
return
}
// Nothing to do if the output is already spent.
if output.spent {
return
}
entry.modified = true
output.spent = true
return
}
// IsFullySpent returns whether or not the transaction the utxo entry represents
// is fully spent.
func (entry *UtxoEntry) IsFullySpent() bool {
// The entry is not fully spent if any of the outputs are unspent.
for _, output := range entry.sparseOutputs {
if !output.spent {
return false
}
}
return true
}
// AmountByIndex returns the amount of the provided output index.
//
// Returns 0 if the output index references an output that does not exist
// either due to it being invalid or because the output is not part of the view
// due to previously being spent/pruned.
func (entry *UtxoEntry) AmountByIndex(outputIndex uint32) int64 {
output, ok := entry.sparseOutputs[outputIndex]
if !ok {
return 0
}
// Ensure the output is decompressed before returning the amount.
output.maybeDecompress(entry.version)
return output.amount
}
// PkScriptByIndex returns the public key script for the provided output index.
//
// Returns nil if the output index references an output that does not exist
// either due to it being invalid or because the output is not part of the view
// due to previously being spent/pruned.
func (entry *UtxoEntry) PkScriptByIndex(outputIndex uint32) []byte {
output, ok := entry.sparseOutputs[outputIndex]
if !ok {
return nil
}
// Ensure the output is decompressed before returning the script.
output.maybeDecompress(entry.version)
return output.pkScript
}
// newUtxoEntry returns a new unspent transaction output entry with the provided
// coinbase flag and block height ready to have unspent outputs added.
func newUtxoEntry(version int32, isCoinBase bool, blockHeight int32) *UtxoEntry {
return &UtxoEntry{
version: version,
isCoinBase: isCoinBase,
blockHeight: blockHeight,
sparseOutputs: make(map[uint32]*utxoOutput),
}
}
// UtxoViewpoint represents a view into the set of unspent transaction outputs
// from a specific point of view in the chain. For example, it could be for
// the end of the main chain, some point in the history of the main chain, or
// down a side chain.
//
// The unspent outputs are needed by other transactions for things such as
// script validation and double spend prevention.
type UtxoViewpoint struct {
entries map[wire.ShaHash]*UtxoEntry
bestHash wire.ShaHash
}
// BestHash returns the hash of the best block in the chain the view currently
// respresents.
func (view *UtxoViewpoint) BestHash() *wire.ShaHash {
return &view.bestHash
}
// SetBestHash sets the hash of the best block in the chain the view currently
// respresents.
func (view *UtxoViewpoint) SetBestHash(hash *wire.ShaHash) {
view.bestHash = *hash
}
// LookupEntry returns information about a given transaction according to the
// current state of the view. It will return nil if the passed transaction
// hash does not exist in the view or is otherwise not available such as when
// it has been disconnected during a reorg.
func (view *UtxoViewpoint) LookupEntry(txHash *wire.ShaHash) *UtxoEntry {
entry, ok := view.entries[*txHash]
if !ok {
return nil
}
return entry
}
// AddTxOuts adds all outputs in the passed transaction which are not provably
// unspendable to the view. When the view already has entries for any of the
// outputs, they are simply marked unspent. All fields will be updated for
// existing entries since it's possible it has changed during a reorg.
func (view *UtxoViewpoint) AddTxOuts(tx *btcutil.Tx, blockHeight int32) {
// When there are not already any utxos associated with the transaction,
// add a new entry for it to the view.
entry := view.LookupEntry(tx.Sha())
if entry == nil {
entry = newUtxoEntry(tx.MsgTx().Version, IsCoinBase(tx),
blockHeight)
view.entries[*tx.Sha()] = entry
} else {
entry.blockHeight = blockHeight
}
entry.modified = true
// Loop all of the transaction outputs and add those which are not
// provably unspendable.
for txOutIdx, txOut := range tx.MsgTx().TxOut {
if txscript.IsUnspendable(txOut.PkScript) {
continue
}
// Update existing entries. All fields are updated because it's
// possible (although extremely unlikely) that the existing
// entry is being replaced by a different transaction with the
// same hash. This is allowed so long as the previous
// transaction is fully spent.
if output, ok := entry.sparseOutputs[uint32(txOutIdx)]; ok {
output.spent = false
output.compressed = false
output.amount = txOut.Value
output.pkScript = txOut.PkScript
continue
}
// Add the unspent transaction output.
entry.sparseOutputs[uint32(txOutIdx)] = &utxoOutput{
spent: false,
compressed: false,
amount: txOut.Value,
pkScript: txOut.PkScript,
}
}
return
}
// connectTransaction updates the view by adding all new utxos created by the
// passed transaction and marking all utxos that the transactions spend as
// spent. In addition, when the 'stxos' argument is not nil, it will be updated
// to append an entry for each spent txout. An error will be returned if the
// view does not contain the required utxos.
func (view *UtxoViewpoint) connectTransaction(tx *btcutil.Tx, blockHeight int32, stxos *[]spentTxOut) error {
// Coinbase transactions don't have any inputs to spend.
if IsCoinBase(tx) {
// Add the transaction's outputs as available utxos.
view.AddTxOuts(tx, blockHeight)
return nil
}
// Spend the referenced utxos by marking them spent in the view and,
// if a slice was provided for the spent txout details, append an entry
// to it.
for _, txIn := range tx.MsgTx().TxIn {
originIndex := txIn.PreviousOutPoint.Index
entry := view.entries[txIn.PreviousOutPoint.Hash]
// Ensure the referenced utxo exists in the view. This should
// never happen unless there is a bug is introduced in the code.
if entry == nil {
return AssertError(fmt.Sprintf("view missing input %v",
txIn.PreviousOutPoint))
}
entry.SpendOutput(originIndex)
// Don't create the stxo details if not requested.
if stxos == nil {
continue
}
// Populate the stxo details using the utxo entry. When the
// transaction is fully spent, set the additional stxo fields
// accordingly since those details will no longer be available
// in the utxo set.
var stxo = spentTxOut{
compressed: false,
version: entry.Version(),
amount: entry.AmountByIndex(originIndex),
pkScript: entry.PkScriptByIndex(originIndex),
}
if entry.IsFullySpent() {
stxo.height = entry.BlockHeight()
stxo.isCoinBase = entry.IsCoinBase()
}
// Append the entry to the provided spent txouts slice.
*stxos = append(*stxos, stxo)
}
// Add the transaction's outputs as available utxos.
view.AddTxOuts(tx, blockHeight)
return nil
}
// connectTransactions updates the view by adding all new utxos created by all
// of the transactions in the passed block, marking all utxos the transactions
// spend as spent, and setting the best hash for the view to the passed block.
// In addition, when the 'stxos' argument is not nil, it will be updated to
// append an entry for each spent txout.
func (view *UtxoViewpoint) connectTransactions(block *btcutil.Block, stxos *[]spentTxOut) error {
for _, tx := range block.Transactions() {
err := view.connectTransaction(tx, block.Height(), stxos)
if err != nil {
return err
}
}
// Update the best hash for view to include this block since all of its
// transactions have been connected.
view.SetBestHash(block.Sha())
return nil
}
// disconnectTransactions updates the view by removing all of the transactions
// created by the passed block, restoring all utxos the transactions spent by
// using the provided spent txo information, and setting the best hash for the
// view to the block before the passed block.
func (view *UtxoViewpoint) disconnectTransactions(block *btcutil.Block, stxos []spentTxOut) error {
// Sanity check the correct number of stxos are provided.
if len(stxos) != countSpentOutputs(block) {
return AssertError("disconnectTransactions called with bad " +
"spent transaction out information")
}
// Loop backwards through all transactions so everything is unspent in
// reverse order. This is necessary since transactions later in a block
// can spend from previous ones.
stxoIdx := len(stxos) - 1
transactions := block.Transactions()
for txIdx := len(transactions) - 1; txIdx > -1; txIdx-- {
tx := transactions[txIdx]
// Clear this transaction from the view if it already exists or
// create a new empty entry for when it does not. This is done
// because the code relies on its existence in the view in order
// to signal modifications have happened.
isCoinbase := txIdx == 0
entry := view.entries[*tx.Sha()]
if entry == nil {
entry = newUtxoEntry(tx.MsgTx().Version, isCoinbase,
block.Height())
view.entries[*tx.Sha()] = entry
}
entry.modified = true
entry.sparseOutputs = make(map[uint32]*utxoOutput)
// Loop backwards through all of the transaction inputs (except
// for the coinbase which has no inputs) and unspend the
// referenced txos. This is necessary to match the order of the
// spent txout entries.
if isCoinbase {
continue
}
for txInIdx := len(tx.MsgTx().TxIn) - 1; txInIdx > -1; txInIdx-- {
// Ensure the spent txout index is decremented to stay
// in sync with the transaction input.
stxo := &stxos[stxoIdx]
stxoIdx--
// When there is not already an entry for the referenced
// transaction in the view, it means it was fully spent,
// so create a new utxo entry in order to resurrect it.
txIn := tx.MsgTx().TxIn[txInIdx]
originHash := &txIn.PreviousOutPoint.Hash
originIndex := txIn.PreviousOutPoint.Index
entry := view.entries[*originHash]
if entry == nil {
entry = newUtxoEntry(stxo.version,
stxo.isCoinBase, stxo.height)
view.entries[*originHash] = entry
}
// Mark the entry as modified since it is either new
// or will be changed below.
entry.modified = true
// Restore the specific utxo using the stxo data from
// the spend journal if it doesn't already exist in the
// view.
output, ok := entry.sparseOutputs[originIndex]
if !ok {
// Add the unspent transaction output.
entry.sparseOutputs[originIndex] = &utxoOutput{
spent: false,
compressed: stxo.compressed,
amount: stxo.amount,
pkScript: stxo.pkScript,
}
continue
}
// Mark the existing referenced transaction output as
// unspent.
output.spent = false
}
}
// Update the best hash for view to the previous block since all of the
// transactions for the current block have been disconnected.
view.SetBestHash(&block.MsgBlock().Header.PrevBlock)
return nil
}
// Entries returns the underlying map that stores of all the utxo entries.
func (view *UtxoViewpoint) Entries() map[wire.ShaHash]*UtxoEntry {
return view.entries
}
// commit prunes all entries marked modified that are now fully spent and marks
// all entries as unmodified.
func (view *UtxoViewpoint) commit() {
for txHash, entry := range view.entries {
if entry == nil || (entry.modified && entry.IsFullySpent()) {
delete(view.entries, txHash)
continue
}
entry.modified = false
}
}
// fetchUtxosMain fetches unspent transaction output data about the provided
// set of transactions from the point of view of the end of the main chain at
// the time of the call.
//
// Upon completion of this function, the view will contain an entry for each
// requested transaction. Fully spent transactions, or those which otherwise
// don't exist, will result in a nil entry in the view.
func (view *UtxoViewpoint) fetchUtxosMain(db database.DB, txSet map[wire.ShaHash]struct{}) error {
// Nothing to do if there are no requested hashes.
if len(txSet) == 0 {
return nil
}
// Load the unspent transaction output information for the requested set
// of transactions from the point of view of the end of the main chain.
//
// NOTE: Missing entries are not considered an error here and instead
// will result in nil entries in the view. This is intentionally done
// since other code uses the presence of an entry in the store as a way
// to optimize spend and unspend updates to apply only to the specific
// utxos that the caller needs access to.
return db.View(func(dbTx database.Tx) error {
for hash := range txSet {
hashCopy := hash
entry, err := dbFetchUtxoEntry(dbTx, &hashCopy)
if err != nil {
return err
}
view.entries[hash] = entry
}
return nil
})
}
// fetchUtxos loads utxo details about provided set of transaction hashes into
// the view from the database as needed unless they already exist in the view in
// which case they are ignored.
func (view *UtxoViewpoint) fetchUtxos(db database.DB, txSet map[wire.ShaHash]struct{}) error {
// Nothing to do if there are no requested hashes.
if len(txSet) == 0 {
return nil
}
// Filter entries that are already in the view.
txNeededSet := make(map[wire.ShaHash]struct{})
for hash := range txSet {
// Already loaded into the current view.
if _, ok := view.entries[hash]; ok {
continue
}
txNeededSet[hash] = struct{}{}
}
// Request the input utxos from the database.
return view.fetchUtxosMain(db, txNeededSet)
}
// fetchInputUtxos loads utxo details about the input transactions referenced
// by the transactions in the given block into the view from the database as
// needed. In particular, referenced entries that are earlier in the block are
// added to the view and entries that are already in the view are not modified.
func (view *UtxoViewpoint) fetchInputUtxos(db database.DB, block *btcutil.Block) error {
// Build a map of in-flight transactions because some of the inputs in
// this block could be referencing other transactions earlier in this
// block which are not yet in the chain.
txInFlight := map[wire.ShaHash]int{}
transactions := block.Transactions()
for i, tx := range transactions {
txInFlight[*tx.Sha()] = i
}
// Loop through all of the transaction inputs (except for the coinbase
// which has no inputs) collecting them into sets of what is needed and
// what is already known (in-flight).
txNeededSet := make(map[wire.ShaHash]struct{})
for i, tx := range transactions[1:] {
for _, txIn := range tx.MsgTx().TxIn {
// It is acceptable for a transaction input to reference
// the output of another transaction in this block only
// if the referenced transaction comes before the
// current one in this block. Add the outputs of the
// referenced transaction as available utxos when this
// is the case. Otherwise, the utxo details are still
// needed.
//
// NOTE: The >= is correct here because i is one less
// than the actual position of the transaction within
// the block due to skipping the coinbase.
originHash := &txIn.PreviousOutPoint.Hash
if inFlightIndex, ok := txInFlight[*originHash]; ok &&
i >= inFlightIndex {
originTx := transactions[inFlightIndex]
view.AddTxOuts(originTx, block.Height())
continue
}
// Don't request entries that are already in the view
// from the database.
if _, ok := view.entries[*originHash]; ok {
continue
}
txNeededSet[*originHash] = struct{}{}
}
}
// Request the input utxos from the database.
return view.fetchUtxosMain(db, txNeededSet)
}
// NewUtxoViewpoint returns a new empty unspent transaction output view.
func NewUtxoViewpoint() *UtxoViewpoint {
return &UtxoViewpoint{
entries: make(map[wire.ShaHash]*UtxoEntry),
}
}
// FetchUtxoView loads utxo details about the input transactions referenced by
// the passed transaction from the point of view of the end of the main chain.
// It also attempts to fetch the utxo details for the transaction itself so the
// returned view can be examined for duplicate unspent transaction outputs.
//
// This function is safe for concurrent access however the returned view is NOT.
func (b *BlockChain) FetchUtxoView(tx *btcutil.Tx) (*UtxoViewpoint, error) {
b.chainLock.RLock()
defer b.chainLock.RUnlock()
// Create a set of needed transactions based on those referenced by the
// inputs of the passed transaction. Also, add the passed transaction
// itself as a way for the caller to detect duplicates that are not
// fully spent.
txNeededSet := make(map[wire.ShaHash]struct{})
txNeededSet[*tx.Sha()] = struct{}{}
if !IsCoinBase(tx) {
for _, txIn := range tx.MsgTx().TxIn {
txNeededSet[txIn.PreviousOutPoint.Hash] = struct{}{}
}
}
// Request the utxos from the point of view of the end of the main
// chain.
view := NewUtxoViewpoint()
err := view.fetchUtxosMain(b.db, txNeededSet)
return view, err
}
// FetchUtxoEntry loads and returns the unspent transaction output entry for the
// passed hash from the point of view of the end of the main chain.
//
// NOTE: Requesting a hash for which there is no data will NOT return an error.
// Instead both the entry and the error will be nil. This is done to allow
// pruning of fully spent transactions. In practice this means the caller must
// check if the returned entry is nil before invoking methods on it.
//
// This function is safe for concurrent access however the returned entry (if
// any) is NOT.
func (b *BlockChain) FetchUtxoEntry(txHash *wire.ShaHash) (*UtxoEntry, error) {
b.chainLock.RLock()
defer b.chainLock.RUnlock()
var entry *UtxoEntry
err := b.db.View(func(dbTx database.Tx) error {
var err error
entry, err = dbFetchUtxoEntry(dbTx, txHash)
return err
})
if err != nil {
return nil, err
}
return entry, nil
}

339
blockchain/validate.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2013-2015 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -12,7 +12,6 @@ import (
"time"
"github.com/btcsuite/btcd/chaincfg"
"github.com/btcsuite/btcd/database"
"github.com/btcsuite/btcd/txscript"
"github.com/btcsuite/btcd/wire"
"github.com/btcsuite/btcutil"
@ -369,7 +368,7 @@ func CountSigOps(tx *btcutil.Tx) int {
// transactions which are of the pay-to-script-hash type. This uses the
// precise, signature operation counting mechanism from the script engine which
// requires access to the input transaction scripts.
func CountP2SHSigOps(tx *btcutil.Tx, isCoinBaseTx bool, txStore TxStore) (int, error) {
func CountP2SHSigOps(tx *btcutil.Tx, isCoinBaseTx bool, utxoView *UtxoViewpoint) (int, error) {
// Coinbase transactions have no interesting inputs.
if isCoinBaseTx {
return 0, nil
@ -379,31 +378,21 @@ func CountP2SHSigOps(tx *btcutil.Tx, isCoinBaseTx bool, txStore TxStore) (int, e
// inputs.
msgTx := tx.MsgTx()
totalSigOps := 0
for _, txIn := range msgTx.TxIn {
for txInIndex, txIn := range msgTx.TxIn {
// Ensure the referenced input transaction is available.
txInHash := &txIn.PreviousOutPoint.Hash
originTx, exists := txStore[*txInHash]
if !exists || originTx.Err != nil || originTx.Tx == nil {
str := fmt.Sprintf("unable to find input transaction "+
"%v referenced from transaction %v", txInHash,
tx.Sha())
return 0, ruleError(ErrMissingTx, str)
}
originMsgTx := originTx.Tx.MsgTx()
// Ensure the output index in the referenced transaction is
// available.
originTxHash := &txIn.PreviousOutPoint.Hash
originTxIndex := txIn.PreviousOutPoint.Index
if originTxIndex >= uint32(len(originMsgTx.TxOut)) {
str := fmt.Sprintf("out of bounds input index %d in "+
"transaction %v referenced from transaction %v",
originTxIndex, txInHash, tx.Sha())
return 0, ruleError(ErrBadTxInput, str)
txEntry := utxoView.LookupEntry(originTxHash)
if txEntry == nil || txEntry.IsOutputSpent(originTxIndex) {
str := fmt.Sprintf("unable to find unspent output "+
"%v referenced from transaction %s:%d",
txIn.PreviousOutPoint, tx.Sha(), txInIndex)
return 0, ruleError(ErrMissingTx, str)
}
// We're only interested in pay-to-script-hash types, so skip
// this input if it's not one.
pkScript := originMsgTx.TxOut[originTxIndex].PkScript
pkScript := txEntry.PkScriptByIndex(originTxIndex)
if !txscript.IsPayToScriptHash(pkScript) {
continue
}
@ -419,10 +408,9 @@ func CountP2SHSigOps(tx *btcutil.Tx, isCoinBaseTx bool, txStore TxStore) (int, e
lastSigOps := totalSigOps
totalSigOps += numSigOps
if totalSigOps < lastSigOps {
str := fmt.Sprintf("the public key script from "+
"output index %d in transaction %v contains "+
"too many signature operations - overflow",
originTxIndex, txInHash)
str := fmt.Sprintf("the public key script from output "+
"%v contains too many signature operations - "+
"overflow", txIn.PreviousOutPoint)
return 0, ruleError(ErrTooManySigOps, str)
}
}
@ -515,7 +503,7 @@ func checkBlockSanity(block *btcutil.Block, powLimit *big.Int, timeSource Median
for i, tx := range transactions[1:] {
if IsCoinBase(tx) {
str := fmt.Sprintf("block contains second coinbase at "+
"index %d", i)
"index %d", i+1)
return ruleError(ErrMultipleCoinbases, str)
}
}
@ -583,12 +571,60 @@ func CheckBlockSanity(block *btcutil.Block, powLimit *big.Int, timeSource Median
return checkBlockSanity(block, powLimit, timeSource, BFNone)
}
// ExtractCoinbaseHeight attempts to extract the height of the block from the
// scriptSig of a coinbase transaction. Coinbase heights are only present in
// blocks of version 2 or later. This was added as part of BIP0034.
func ExtractCoinbaseHeight(coinbaseTx *btcutil.Tx) (int32, error) {
sigScript := coinbaseTx.MsgTx().TxIn[0].SignatureScript
if len(sigScript) < 1 {
str := "the coinbase signature script for blocks of " +
"version %d or greater must start with the " +
"length of the serialized block height"
str = fmt.Sprintf(str, serializedHeightVersion)
return 0, ruleError(ErrMissingCoinbaseHeight, str)
}
serializedLen := int(sigScript[0])
if len(sigScript[1:]) < serializedLen {
str := "the coinbase signature script for blocks of " +
"version %d or greater must start with the " +
"serialized block height"
str = fmt.Sprintf(str, serializedLen)
return 0, ruleError(ErrMissingCoinbaseHeight, str)
}
serializedHeightBytes := make([]byte, 8, 8)
copy(serializedHeightBytes, sigScript[1:serializedLen+1])
serializedHeight := binary.LittleEndian.Uint64(serializedHeightBytes)
return int32(serializedHeight), nil
}
// checkSerializedHeight checks if the signature script in the passed
// transaction starts with the serialized block height of wantHeight.
func checkSerializedHeight(coinbaseTx *btcutil.Tx, wantHeight int32) error {
serializedHeight, err := ExtractCoinbaseHeight(coinbaseTx)
if err != nil {
return err
}
if serializedHeight != wantHeight {
str := fmt.Sprintf("the coinbase signature script serialized "+
"block height is %d when %d was expected",
serializedHeight, wantHeight)
return ruleError(ErrBadCoinbaseHeight, str)
}
return nil
}
// checkBlockHeaderContext peforms several validation checks on the block header
// which depend on its position within the block chain.
//
// The flags modify the behavior of this function as follows:
// - BFFastAdd: All checks except those involving comparing the header against
// the checkpoints are not performed.
//
// This function MUST be called with the chain state lock held (for writes).
func (b *BlockChain) checkBlockHeaderContext(header *wire.BlockHeader, prevNode *blockNode, flags BehaviorFlags) error {
// The genesis block is valid by definition.
if prevNode == nil {
@ -697,6 +733,8 @@ func (b *BlockChain) checkBlockHeaderContext(header *wire.BlockHeader, prevNode
//
// The flags are also passed to checkBlockHeaderContext. See its documentation
// for how the flags modify its behavior.
//
// This function MUST be called with the chain state lock held (for writes).
func (b *BlockChain) checkBlockContext(block *btcutil.Block, prevNode *blockNode, flags BehaviorFlags) error {
// The genesis block is valid by definition.
if prevNode == nil {
@ -746,64 +784,6 @@ func (b *BlockChain) checkBlockContext(block *btcutil.Block, prevNode *blockNode
return nil
}
// ExtractCoinbaseHeight attempts to extract the height of the block from the
// scriptSig of a coinbase transaction. Coinbase heights are only present in
// blocks of version 2 or later. This was added as part of BIP0034.
func ExtractCoinbaseHeight(coinbaseTx *btcutil.Tx) (int32, error) {
sigScript := coinbaseTx.MsgTx().TxIn[0].SignatureScript
if len(sigScript) < 1 {
str := "the coinbase signature script for blocks of " +
"version %d or greater must start with the " +
"length of the serialized block height"
str = fmt.Sprintf(str, serializedHeightVersion)
return 0, ruleError(ErrMissingCoinbaseHeight, str)
}
serializedLen := int(sigScript[0])
if len(sigScript[1:]) < serializedLen {
str := "the coinbase signature script for blocks of " +
"version %d or greater must start with the " +
"serialized block height"
str = fmt.Sprintf(str, serializedHeightVersion)
return 0, ruleError(ErrMissingCoinbaseHeight, str)
}
serializedHeightBytes := make([]byte, 8, 8)
copy(serializedHeightBytes, sigScript[1:serializedLen+1])
serializedHeight := binary.LittleEndian.Uint64(serializedHeightBytes)
return int32(serializedHeight), nil
}
// checkSerializedHeight checks if the signature script in the passed
// transaction starts with the serialized block height of wantHeight.
func checkSerializedHeight(coinbaseTx *btcutil.Tx, wantHeight int32) error {
serializedHeight, err := ExtractCoinbaseHeight(coinbaseTx)
if err != nil {
return err
}
if serializedHeight != wantHeight {
str := fmt.Sprintf("the coinbase signature script serialized "+
"block height is %d when %d was expected",
serializedHeight, wantHeight)
return ruleError(ErrBadCoinbaseHeight, str)
}
return nil
}
// isTransactionSpent returns whether or not the provided transaction data
// describes a fully spent transaction. A fully spent transaction is one where
// all outputs have been spent.
func isTransactionSpent(txD *TxData) bool {
for _, isOutputSpent := range txD.Spent {
if !isOutputSpent {
return false
}
}
return true
}
// checkBIP0030 ensures blocks do not contain duplicate transactions which
// 'overwrite' older transactions that are not fully spent. This prevents an
// attack where a coinbase and all of its dependent transactions could be
@ -812,40 +792,29 @@ func isTransactionSpent(txD *TxData) bool {
//
// For more details, see https://en.bitcoin.it/wiki/BIP_0030 and
// http://r6.ca/blog/20120206T005236Z.html.
func (b *BlockChain) checkBIP0030(node *blockNode, block *btcutil.Block) error {
// Attempt to fetch duplicate transactions for all of the transactions
// in this block from the point of view of the parent node.
//
// This function MUST be called with the chain state lock held (for reads).
func (b *BlockChain) checkBIP0030(node *blockNode, block *btcutil.Block, view *UtxoViewpoint) error {
// Fetch utxo details for all of the transactions in this block.
// Typically, there will not be any utxos for any of the transactions.
fetchSet := make(map[wire.ShaHash]struct{})
for _, tx := range block.Transactions() {
fetchSet[*tx.Sha()] = struct{}{}
}
txResults, err := b.fetchTxStore(node, fetchSet)
err := view.fetchUtxos(b.db, fetchSet)
if err != nil {
return err
}
// Examine the resulting data about the requested transactions.
for _, txD := range txResults {
switch txD.Err {
// A duplicate transaction was not found. This is the most
// common case.
case database.ErrTxShaMissing:
continue
// A duplicate transaction was found. This is only allowed if
// the duplicate transaction is fully spent.
case nil:
if !isTransactionSpent(txD) {
str := fmt.Sprintf("tried to overwrite "+
"transaction %v at block height %d "+
"that is not fully spent", txD.Hash,
txD.BlockHeight)
return ruleError(ErrOverwriteTx, str)
}
// Some other unexpected error occurred. Return it now.
default:
return txD.Err
// Duplicate transactions are only allowed if the previous transaction
// is fully spent.
for _, tx := range block.Transactions() {
txEntry := view.LookupEntry(tx.Sha())
if txEntry != nil && !txEntry.IsFullySpent() {
str := fmt.Sprintf("tried to overwrite transaction %v "+
"at block height %d that is not fully spent",
tx.Sha(), txEntry.blockHeight)
return ruleError(ErrOverwriteTx, str)
}
}
@ -860,7 +829,10 @@ func (b *BlockChain) checkBIP0030(node *blockNode, block *btcutil.Block) error {
// amount, and verifying the signatures to prove the spender was the owner of
// the bitcoins and therefore allowed to spend them. As it checks the inputs,
// it also calculates the total fees for the transaction and returns that value.
func CheckTransactionInputs(tx *btcutil.Tx, txHeight int32, txStore TxStore) (int64, error) {
//
// NOTE: The transaction MUST have already been sanity checked with the
// CheckTransactionSanity function prior to calling this function.
func CheckTransactionInputs(tx *btcutil.Tx, txHeight int32, utxoView *UtxoViewpoint) (int64, error) {
// Coinbase transactions have no inputs.
if IsCoinBase(tx) {
return 0, nil
@ -868,42 +840,39 @@ func CheckTransactionInputs(tx *btcutil.Tx, txHeight int32, txStore TxStore) (in
txHash := tx.Sha()
var totalSatoshiIn int64
for _, txIn := range tx.MsgTx().TxIn {
// Ensure the input is available.
txInHash := &txIn.PreviousOutPoint.Hash
originTx, exists := txStore[*txInHash]
if !exists || originTx.Err != nil || originTx.Tx == nil {
str := fmt.Sprintf("unable to find input transaction "+
"%v for transaction %v", txInHash, txHash)
for txInIndex, txIn := range tx.MsgTx().TxIn {
// Ensure the referenced input transaction is available.
originTxHash := &txIn.PreviousOutPoint.Hash
utxoEntry := utxoView.LookupEntry(originTxHash)
if utxoEntry == nil {
str := fmt.Sprintf("unable to find unspent output "+
"%v referenced from transaction %s:%d",
txIn.PreviousOutPoint, tx.Sha(), txInIndex)
return 0, ruleError(ErrMissingTx, str)
}
// Ensure the transaction is not spending coins which have not
// yet reached the required coinbase maturity.
if IsCoinBase(originTx.Tx) {
originHeight := originTx.BlockHeight
if utxoEntry.IsCoinBase() {
originHeight := int32(utxoEntry.BlockHeight())
blocksSincePrev := txHeight - originHeight
if blocksSincePrev < coinbaseMaturity {
str := fmt.Sprintf("tried to spend coinbase "+
"transaction %v from height %v at "+
"height %v before required maturity "+
"of %v blocks", txInHash, originHeight,
txHeight, coinbaseMaturity)
"of %v blocks", originTxHash,
originHeight, txHeight,
coinbaseMaturity)
return 0, ruleError(ErrImmatureSpend, str)
}
}
// Ensure the transaction is not double spending coins.
originTxIndex := txIn.PreviousOutPoint.Index
if originTxIndex >= uint32(len(originTx.Spent)) {
str := fmt.Sprintf("out of bounds input index %d in "+
"transaction %v referenced from transaction %v",
originTxIndex, txInHash, txHash)
return 0, ruleError(ErrBadTxInput, str)
}
if originTx.Spent[originTxIndex] {
str := fmt.Sprintf("transaction %v tried to double "+
"spend output %v", txHash, txIn.PreviousOutPoint)
if utxoEntry.IsOutputSpent(originTxIndex) {
str := fmt.Sprintf("transaction %s:%d tried to double "+
"spend output %v", txHash, txInIndex,
txIn.PreviousOutPoint)
return 0, ruleError(ErrDoubleSpend, str)
}
@ -913,16 +882,17 @@ func CheckTransactionInputs(tx *btcutil.Tx, txHeight int32, txStore TxStore) (in
// a transaction are in a unit value known as a satoshi. One
// bitcoin is a quantity of satoshi as defined by the
// SatoshiPerBitcoin constant.
originTxSatoshi := originTx.Tx.MsgTx().TxOut[originTxIndex].Value
originTxSatoshi := utxoEntry.AmountByIndex(originTxIndex)
if originTxSatoshi < 0 {
str := fmt.Sprintf("transaction output has negative "+
"value of %v", originTxSatoshi)
"value of %v", btcutil.Amount(originTxSatoshi))
return 0, ruleError(ErrBadTxOutValue, str)
}
if originTxSatoshi > btcutil.MaxSatoshi {
str := fmt.Sprintf("transaction output value of %v is "+
"higher than max allowed value of %v",
originTxSatoshi, btcutil.MaxSatoshi)
btcutil.Amount(originTxSatoshi),
btcutil.MaxSatoshi)
return 0, ruleError(ErrBadTxOutValue, str)
}
@ -939,9 +909,6 @@ func CheckTransactionInputs(tx *btcutil.Tx, txHeight int32, txStore TxStore) (in
btcutil.MaxSatoshi)
return 0, ruleError(ErrBadTxOutValue, str)
}
// Mark the referenced output as spent.
originTx.Spent[originTxIndex] = true
}
// Calculate the total output amount for this transaction. It is safe
@ -968,8 +935,11 @@ func CheckTransactionInputs(tx *btcutil.Tx, txHeight int32, txStore TxStore) (in
}
// checkConnectBlock performs several checks to confirm connecting the passed
// block to the main chain (including whatever reorganization might be necessary
// to get this node to the main chain) does not violate any rules.
// block to the chain represented by the passed view does not violate any rules.
// In addition, the passed view is updated to spend all of the referenced
// outputs and add all of the new utxos created by block. Thus, the view will
// represent the state of the chain as if the block were actually connected and
// consequently the best hash for the view is also updated to passed block.
//
// The CheckConnectBlock function makes use of this function to perform the
// bulk of its work. The only difference is this function accepts a node which
@ -979,7 +949,9 @@ func CheckTransactionInputs(tx *btcutil.Tx, txHeight int32, txStore TxStore) (in
//
// See the comments for CheckConnectBlock for some examples of the type of
// checks performed by this function.
func (b *BlockChain) checkConnectBlock(node *blockNode, block *btcutil.Block) error {
//
// This function MUST be called with the chain state lock held (for writes).
func (b *BlockChain) checkConnectBlock(node *blockNode, block *btcutil.Block, view *UtxoViewpoint, stxos *[]spentTxOut) error {
// If the side chain blocks end up in the database, a call to
// CheckBlockSanity should be done here in case a previous version
// allowed a block that is no longer valid. However, since the
@ -987,32 +959,41 @@ func (b *BlockChain) checkConnectBlock(node *blockNode, block *btcutil.Block) er
// it isn't currently necessary.
// The coinbase for the Genesis block is not spendable, so just return
// now.
if node.hash.IsEqual(b.chainParams.GenesisHash) && b.bestChain == nil {
return nil
// an error now.
if node.hash.IsEqual(b.chainParams.GenesisHash) {
str := "the coinbase for the genesis block is not spendable"
return ruleError(ErrMissingTx, str)
}
// Ensure the view is for the node being checked.
if !view.BestHash().IsEqual(node.parentHash) {
return AssertError(fmt.Sprintf("inconsistent view when "+
"checking block connection: best hash is %v instead "+
"of expected %v", view.BestHash(), node.hash))
}
// BIP0030 added a rule to prevent blocks which contain duplicate
// transactions that 'overwrite' older transactions which are not fully
// spent. See the documentation for checkBIP0030 for more details.
//
// There are two blocks in the chain which violate this
// rule, so the check must be skipped for those blocks. The
// isBIP0030Node function is used to determine if this block is one
// of the two blocks that must be skipped.
// There are two blocks in the chain which violate this rule, so the
// check must be skipped for those blocks. The isBIP0030Node function is
// used to determine if this block is one of the two blocks that must be
// skipped.
enforceBIP0030 := !isBIP0030Node(node)
if enforceBIP0030 {
err := b.checkBIP0030(node, block)
err := b.checkBIP0030(node, block, view)
if err != nil {
return err
}
}
// Request a map that contains all input transactions for the block from
// the point of view of its position within the block chain. These
// transactions are needed for verification of things such as
// Load all of the utxos referenced by the inputs for all transactions
// in the block don't already exist in the utxo view from the database.
//
// These utxo entries are needed for verification of things such as
// transaction inputs, counting pay-to-script-hashes, and scripts.
txInputStore, err := b.fetchInputTransactions(node, block)
err := view.fetchInputUtxos(b.db, block)
if err != nil {
return err
}
@ -1021,10 +1002,7 @@ func (b *BlockChain) checkConnectBlock(node *blockNode, block *btcutil.Block) er
// "standard" type. The rules for this BIP only apply to transactions
// after the timestamp defined by txscript.Bip16Activation. See
// https://en.bitcoin.it/wiki/BIP_0016 for more details.
enforceBIP0016 := false
if node.timestamp.After(txscript.Bip16Activation) {
enforceBIP0016 = true
}
enforceBIP0016 := node.timestamp.After(txscript.Bip16Activation)
// The number of signature operations must be less than the maximum
// allowed per block. Note that the preliminary sanity checks on a
@ -1043,8 +1021,7 @@ func (b *BlockChain) checkConnectBlock(node *blockNode, block *btcutil.Block) er
// countP2SHSigOps for whether or not the transaction is
// a coinbase transaction rather than having to do a
// full coinbase check again.
numP2SHSigOps, err := CountP2SHSigOps(tx, i == 0,
txInputStore)
numP2SHSigOps, err := CountP2SHSigOps(tx, i == 0, view)
if err != nil {
return err
}
@ -1072,7 +1049,7 @@ func (b *BlockChain) checkConnectBlock(node *blockNode, block *btcutil.Block) er
// bounds.
var totalFees int64
for _, tx := range transactions {
txFee, err := CheckTransactionInputs(tx, node.height, txInputStore)
txFee, err := CheckTransactionInputs(tx, node.height, view)
if err != nil {
return err
}
@ -1085,6 +1062,15 @@ func (b *BlockChain) checkConnectBlock(node *blockNode, block *btcutil.Block) er
return ruleError(ErrBadFees, "total fees for block "+
"overflows accumulator")
}
// Add all of the outputs for this transaction which are not
// provably unspendable as available utxos. Also, the passed
// spent txos slice is updated to contain an entry for each
// spent txout in the order each transaction spends them.
err = view.connectTransaction(tx, node.height, stxos)
if err != nil {
return err
}
}
// The total output values of the coinbase transaction must not exceed
@ -1111,7 +1097,7 @@ func (b *BlockChain) checkConnectBlock(node *blockNode, block *btcutil.Block) er
// will therefore be detected by the next checkpoint). This is a huge
// optimization because running the scripts is the most time consuming
// portion of block handling.
checkpoint := b.LatestCheckpoint()
checkpoint := b.latestCheckpoint()
runScripts := !b.noVerify
if checkpoint != nil && node.height <= checkpoint.Height {
runScripts = false
@ -1130,7 +1116,7 @@ func (b *BlockChain) checkConnectBlock(node *blockNode, block *btcutil.Block) er
// Blocks created after the BIP0016 activation time need to have the
// pay-to-script-hash checks enabled.
var scriptFlags txscript.ScriptFlags
if block.MsgBlock().Header.Timestamp.After(txscript.Bip16Activation) {
if enforceBIP0016 {
scriptFlags |= txscript.ScriptBip16
}
@ -1158,12 +1144,16 @@ func (b *BlockChain) checkConnectBlock(node *blockNode, block *btcutil.Block) er
// expensive ECDSA signature check scripts. Doing this last helps
// prevent CPU exhaustion attacks.
if runScripts {
err := checkBlockScripts(block, txInputStore, scriptFlags, b.sigCache)
err := checkBlockScripts(block, view, scriptFlags, b.sigCache)
if err != nil {
return err
}
}
// Update the best hash for view to include this block since all of its
// transactions have been connected.
view.SetBestHash(node.hash)
return nil
}
@ -1175,15 +1165,20 @@ func (b *BlockChain) checkConnectBlock(node *blockNode, block *btcutil.Block) er
// per block, invalid values in relation to the expected block subsidy, or fail
// transaction script validation.
//
// This function is NOT safe for concurrent access.
// This function is safe for concurrent access.
func (b *BlockChain) CheckConnectBlock(block *btcutil.Block) error {
prevNode := b.bestChain
newNode := newBlockNode(&block.MsgBlock().Header, block.Sha(),
block.Height())
if prevNode != nil {
newNode.parent = prevNode
newNode.workSum.Add(prevNode.workSum, newNode.workSum)
}
b.chainLock.Lock()
defer b.chainLock.Unlock()
return b.checkConnectBlock(newNode, block)
prevNode := b.bestNode
newNode := newBlockNode(&block.MsgBlock().Header, block.Sha(),
prevNode.height+1)
newNode.parent = prevNode
newNode.workSum.Add(prevNode.workSum, newNode.workSum)
// Leave the spent txouts entry nil in the state since the information
// is not needed and thus extra work can be avoided.
view := NewUtxoViewpoint()
view.SetBestHash(prevNode.hash)
return b.checkConnectBlock(newNode, block, view, nil)
}

12
blockchain/validate_test.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2013-2014 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -17,7 +17,7 @@ import (
)
// TestCheckConnectBlock tests the CheckConnectBlock function to ensure it
// fails
// fails.
func TestCheckConnectBlock(t *testing.T) {
// Create a new database and chain instance to run tests against.
chain, teardownFunc, err := chainSetup("checkconnectblock")
@ -27,13 +27,7 @@ func TestCheckConnectBlock(t *testing.T) {
}
defer teardownFunc()
err = chain.GenerateInitialIndex()
if err != nil {
t.Errorf("GenerateInitialIndex: %v", err)
}
// The genesis block should fail to connect since it's already
// inserted.
// The genesis block should fail to connect since it's already inserted.
genesisBlock := chaincfg.MainNetParams.GenesisBlock
err = chain.CheckConnectBlock(btcutil.NewBlock(genesisBlock))
if err == nil {

302
blockmanager.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2013-2014 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -15,7 +15,7 @@ import (
"github.com/btcsuite/btcd/blockchain"
"github.com/btcsuite/btcd/chaincfg"
"github.com/btcsuite/btcd/database"
database "github.com/btcsuite/btcd/database2"
"github.com/btcsuite/btcd/wire"
"github.com/btcsuite/btcutil"
)
@ -93,28 +93,6 @@ type getSyncPeerMsg struct {
reply chan *serverPeer
}
// checkConnectBlockMsg is a message type to be sent across the message channel
// for requesting chain to check if a block connects to the end of the current
// main chain.
type checkConnectBlockMsg struct {
block *btcutil.Block
reply chan error
}
// calcNextReqDifficultyResponse is a response sent to the reply channel of a
// calcNextReqDifficultyMsg query.
type calcNextReqDifficultyResponse struct {
difficulty uint32
err error
}
// calcNextReqDifficultyMsg is a message type to be sent across the message
// channel for requesting the required difficulty of the next block.
type calcNextReqDifficultyMsg struct {
timestamp time.Time
reply chan calcNextReqDifficultyResponse
}
// processBlockResponse is a response sent to the reply channel of a
// processBlockMsg.
type processBlockResponse struct {
@ -122,20 +100,6 @@ type processBlockResponse struct {
err error
}
// fetchTransactionStoreResponse is a response sent to the reply channel of a
// fetchTransactionStoreMsg.
type fetchTransactionStoreResponse struct {
TxStore blockchain.TxStore
err error
}
// fetchTransactionStoreMsg is a message type to be sent across the message
// channel fetching the tx input store for some Tx.
type fetchTransactionStoreMsg struct {
tx *btcutil.Tx
reply chan fetchTransactionStoreResponse
}
// processBlockMsg is a message type to be sent across the message channel
// for requested a block is processed. Note this call differs from blockMsg
// above in that blockMsg is intended for blocks that came from peers and have
@ -200,7 +164,7 @@ type blockManager struct {
server *server
started int32
shutdown int32
blockChain *blockchain.BlockChain
chain *blockchain.BlockChain
rejectedTxns map[wire.ShaHash]struct{}
requestedTxns map[wire.ShaHash]struct{}
requestedBlocks map[wire.ShaHash]struct{}
@ -247,7 +211,7 @@ func (b *blockManager) updateChainState(newestHash *wire.ShaHash, newestHeight i
b.chainState.newestHash = newestHash
b.chainState.newestHeight = newestHeight
medianTime, err := b.blockChain.CalcPastMedianTime()
medianTime, err := b.chain.CalcPastMedianTime()
if err != nil {
b.chainState.pastMedianTimeErr = err
} else {
@ -298,13 +262,7 @@ func (b *blockManager) startSync(peers *list.List) {
return
}
// Find the height of the current known best block.
_, height, err := b.server.db.NewestSha()
if err != nil {
bmgrLog.Errorf("%v", err)
return
}
best := b.chain.BestSnapshot()
var bestPeer *serverPeer
var enext *list.Element
for e := peers.Front(); e != nil; e = enext {
@ -317,7 +275,7 @@ func (b *blockManager) startSync(peers *list.List) {
// doesn't have a later block when it's equal, it will likely
// have one soon so it is a reasonable choice. It also allows
// the case where both are at 0 such as during regression test.
if sp.LastBlock() < int32(height) {
if sp.LastBlock() < best.Height {
peers.Remove(e)
continue
}
@ -334,7 +292,7 @@ func (b *blockManager) startSync(peers *list.List) {
// to send.
b.requestedBlocks = make(map[wire.ShaHash]struct{})
locator, err := b.blockChain.LatestBlockLocator()
locator, err := b.chain.LatestBlockLocator()
if err != nil {
bmgrLog.Errorf("Failed to get block locator for the "+
"latest block: %v", err)
@ -361,13 +319,14 @@ func (b *blockManager) startSync(peers *list.List) {
// and fully validate them. Finally, regression test mode does
// not support the headers-first approach so do normal block
// downloads when in regression test mode.
if b.nextCheckpoint != nil && height < b.nextCheckpoint.Height &&
if b.nextCheckpoint != nil &&
best.Height < b.nextCheckpoint.Height &&
!cfg.RegressionTest && !cfg.DisableCheckpoints {
bestPeer.PushGetHeadersMsg(locator, b.nextCheckpoint.Hash)
b.headersFirstMode = true
bmgrLog.Infof("Downloading headers for blocks %d to "+
"%d from peer %s", height+1,
"%d from peer %s", best.Height+1,
b.nextCheckpoint.Height, bestPeer.Addr())
} else {
bestPeer.PushGetBlocksMsg(locator, &zeroHash)
@ -464,16 +423,8 @@ func (b *blockManager) handleDonePeerMsg(peers *list.List, sp *serverPeer) {
if b.syncPeer != nil && b.syncPeer == sp {
b.syncPeer = nil
if b.headersFirstMode {
// This really shouldn't fail. We have a fairly
// unrecoverable database issue if it does.
newestHash, height, err := b.server.db.NewestSha()
if err != nil {
bmgrLog.Warnf("Unable to obtain latest "+
"block information from the database: "+
"%v", err)
return
}
b.resetHeaderState(newestHash, height)
best := b.chain.BestSnapshot()
b.resetHeaderState(best.Hash, best.Height)
}
b.startSync(peers)
}
@ -543,7 +494,7 @@ func (b *blockManager) handleTxMsg(tmsg *txMsg) {
// current returns true if we believe we are synced with our peers, false if we
// still have blocks to check
func (b *blockManager) current() bool {
if !b.blockChain.IsCurrent(b.server.timeSource) {
if !b.chain.IsCurrent(b.server.timeSource) {
return false
}
@ -553,13 +504,9 @@ func (b *blockManager) current() bool {
return true
}
_, height, err := b.server.db.NewestSha()
// No matter what chain thinks, if we are below the block we are
// syncing to we are not current.
// TODO(oga) we can get chain to return the height of each block when we
// parse an orphan, which would allow us to update the height of peers
// from what it was at initial handshake.
if err != nil || height < b.syncPeer.LastBlock() {
// No matter what chain thinks, if we are below the block we are syncing
// to we are not current.
if b.chain.BestSnapshot().Height < b.syncPeer.LastBlock() {
return false
}
return true
@ -615,7 +562,7 @@ func (b *blockManager) handleBlockMsg(bmsg *blockMsg) {
// Process the block to include validation, best chain selection, orphan
// handling, etc.
isOrphan, err := b.blockChain.ProcessBlock(bmsg.block,
isOrphan, err := b.chain.ProcessBlock(bmsg.block,
b.server.timeSource, behaviorFlags)
if err != nil {
// When the error is a rule error, it means the block was simply
@ -672,8 +619,8 @@ func (b *blockManager) handleBlockMsg(bmsg *blockMsg) {
}
}
orphanRoot := b.blockChain.GetOrphanRoot(blockSha)
locator, err := b.blockChain.LatestBlockLocator()
orphanRoot := b.chain.GetOrphanRoot(blockSha)
locator, err := b.chain.LatestBlockLocator()
if err != nil {
bmgrLog.Warnf("Failed to get block locator for the "+
"latest block: %v", err)
@ -685,16 +632,16 @@ func (b *blockManager) handleBlockMsg(bmsg *blockMsg) {
// update the chain state.
b.progressLogger.LogBlockHeight(bmsg.block)
// Query the db for the latest best block since the block
// Query the chain for the latest best block since the block
// that was processed could be on a side chain or have caused
// a reorg.
newestSha, newestHeight, _ := b.server.db.NewestSha()
b.updateChainState(newestSha, newestHeight)
best := b.chain.BestSnapshot()
b.updateChainState(best.Hash, best.Height)
// Update this peer's latest block height, for future
// potential sync node candidacy.
heightUpdate = int32(newestHeight)
blkShaUpdate = newestSha
heightUpdate = best.Height
blkShaUpdate = best.Hash
// Clear the rejected transactions.
b.rejectedTxns = make(map[wire.ShaHash]struct{})
@ -718,8 +665,6 @@ func (b *blockManager) handleBlockMsg(bmsg *blockMsg) {
go b.server.UpdatePeerHeights(blkShaUpdate, int32(heightUpdate), bmsg.peer)
}
}
// Sync the db to disk.
b.server.db.Sync()
// Nothing more to do if we aren't in headers-first mode.
if !b.headersFirstMode {
@ -927,7 +872,7 @@ func (b *blockManager) haveInventory(invVect *wire.InvVect) (bool, error) {
case wire.InvTypeBlock:
// Ask chain if the block is known to it in any form (main
// chain, side chain, or orphan).
return b.blockChain.HaveBlock(&invVect.Hash)
return b.chain.HaveBlock(&invVect.Hash)
case wire.InvTypeTx:
// Ask the transaction memory pool if the transaction is known
@ -938,7 +883,11 @@ func (b *blockManager) haveInventory(invVect *wire.InvVect) (bool, error) {
// Check if the transaction exists from the point of view of the
// end of the main chain.
return b.server.db.ExistsTxSha(&invVect.Hash)
entry, err := b.chain.FetchUtxoEntry(&invVect.Hash)
if err != nil {
return false, err
}
return entry != nil && !entry.IsFullySpent(), nil
}
// The requested inventory is is an unsupported type, so just claim
@ -978,15 +927,9 @@ func (b *blockManager) handleInvMsg(imsg *invMsg) {
// If our chain is current and a peer announces a block we already
// know of, then update their current block height.
if lastBlock != -1 && b.current() {
exists, err := b.server.db.ExistsSha(&invVects[lastBlock].Hash)
if err == nil && exists {
blkHeight, err := b.server.db.FetchBlockHeightBySha(&invVects[lastBlock].Hash)
if err != nil {
bmgrLog.Warnf("Unable to fetch block height for block (sha: %v), %v",
&invVects[lastBlock].Hash, err)
} else {
imsg.peer.UpdateLastBlockHeight(int32(blkHeight))
}
blkHeight, err := b.chain.BlockHeightByHash(&invVects[lastBlock].Hash)
if err == nil {
imsg.peer.UpdateLastBlockHeight(int32(blkHeight))
}
}
@ -994,7 +937,6 @@ func (b *blockManager) handleInvMsg(imsg *invMsg) {
// request parent blocks of orphans if we receive one we already have.
// Finally, attempt to detect potential stalls due to long side chains
// we already have and request more blocks to prevent them.
chain := b.blockChain
for i, iv := range invVects {
// Ignore unsupported inventory types.
if iv.Type != wire.InvTypeBlock && iv.Type != wire.InvTypeTx {
@ -1043,12 +985,12 @@ func (b *blockManager) handleInvMsg(imsg *invMsg) {
// resending the orphan block as an available block
// to signal there are more missing blocks that need to
// be requested.
if chain.IsKnownOrphan(&iv.Hash) {
if b.chain.IsKnownOrphan(&iv.Hash) {
// Request blocks starting at the latest known
// up to the root of the orphan that just came
// in.
orphanRoot := chain.GetOrphanRoot(&iv.Hash)
locator, err := chain.LatestBlockLocator()
orphanRoot := b.chain.GetOrphanRoot(&iv.Hash)
locator, err := b.chain.LatestBlockLocator()
if err != nil {
bmgrLog.Errorf("PEER: Failed to get block "+
"locator for the latest block: "+
@ -1067,7 +1009,7 @@ func (b *blockManager) handleInvMsg(imsg *invMsg) {
// Request blocks after this one up to the
// final one the remote peer knows about (zero
// stop hash).
locator := chain.BlockLocatorFromHash(&iv.Hash)
locator := b.chain.BlockLocatorFromHash(&iv.Hash)
imsg.peer.PushGetBlocksMsg(locator, &zeroHash)
}
}
@ -1171,30 +1113,9 @@ out:
case getSyncPeerMsg:
msg.reply <- b.syncPeer
case checkConnectBlockMsg:
err := b.blockChain.CheckConnectBlock(msg.block)
msg.reply <- err
case calcNextReqDifficultyMsg:
difficulty, err :=
b.blockChain.CalcNextRequiredDifficulty(
msg.timestamp)
msg.reply <- calcNextReqDifficultyResponse{
difficulty: difficulty,
err: err,
}
case fetchTransactionStoreMsg:
txStore, err := b.blockChain.FetchTransactionStore(msg.tx, false)
msg.reply <- fetchTransactionStoreResponse{
TxStore: txStore,
err: err,
}
case processBlockMsg:
isOrphan, err := b.blockChain.ProcessBlock(
msg.block, b.server.timeSource,
msg.flags)
isOrphan, err := b.chain.ProcessBlock(msg.block,
b.server.timeSource, msg.flags)
if err != nil {
msg.reply <- processBlockResponse{
isOrphan: false,
@ -1202,11 +1123,11 @@ out:
}
}
// Query the db for the latest best block since
// the block that was processed could be on a
// side chain or have caused a reorg.
newestSha, newestHeight, _ := b.server.db.NewestSha()
b.updateChainState(newestSha, newestHeight)
// Query the chain for the latest best block
// since the block that was processed could be
// on a side chain or have caused a reorg.
best := b.chain.BestSnapshot()
b.updateChainState(best.Hash, best.Height)
// Allow any clients performing long polling via the
// getblocktemplate RPC to be notified when the new block causes
@ -1301,12 +1222,6 @@ func (b *blockManager) handleNotifyMsg(notification *blockchain.Notification) {
r.ntfnMgr.NotifyBlockConnected(block)
}
// If we're maintaining the address index, and it is up to date
// then update it based off this new block.
if cfg.AddrIndex && b.server.addrIndexer.IsCaughtUp() {
b.server.addrIndexer.UpdateAddressIndex(block)
}
// A block has been disconnected from the main block chain.
case blockchain.NTBlockDisconnected:
block, ok := notification.Data.(*btcutil.Block)
@ -1434,37 +1349,6 @@ func (b *blockManager) SyncPeer() *serverPeer {
return <-reply
}
// CheckConnectBlock performs several checks to confirm connecting the passed
// block to the main chain does not violate any rules. This function makes use
// of CheckConnectBlock on an internal instance of a block chain. It is funneled
// through the block manager since btcchain is not safe for concurrent access.
func (b *blockManager) CheckConnectBlock(block *btcutil.Block) error {
reply := make(chan error)
b.msgChan <- checkConnectBlockMsg{block: block, reply: reply}
return <-reply
}
// CalcNextRequiredDifficulty calculates the required difficulty for the next
// block after the current main chain. This function makes use of
// CalcNextRequiredDifficulty on an internal instance of a block chain. It is
// funneled through the block manager since btcchain is not safe for concurrent
// access.
func (b *blockManager) CalcNextRequiredDifficulty(timestamp time.Time) (uint32, error) {
reply := make(chan calcNextReqDifficultyResponse)
b.msgChan <- calcNextReqDifficultyMsg{timestamp: timestamp, reply: reply}
response := <-reply
return response.difficulty, response.err
}
// FetchTransactionStore makes use of FetchTransactionStore on an internal
// instance of a block chain. It is safe for concurrent access.
func (b *blockManager) FetchTransactionStore(tx *btcutil.Tx) (blockchain.TxStore, error) {
reply := make(chan fetchTransactionStoreResponse, 1)
b.msgChan <- fetchTransactionStoreMsg{tx: tx, reply: reply}
response := <-reply
return response.TxStore, response.err
}
// ProcessBlock makes use of ProcessBlock on an internal instance of a block
// chain. It is funneled through the block manager since btcchain is not safe
// for concurrent access.
@ -1496,11 +1380,6 @@ func (b *blockManager) Pause() chan<- struct{} {
// newBlockManager returns a new bitcoin block manager.
// Use Start to begin processing asynchronous block and inv updates.
func newBlockManager(s *server) (*blockManager, error) {
newestHash, height, err := s.db.NewestSha()
if err != nil {
return nil, err
}
bm := blockManager{
server: s,
rejectedTxns: make(map[wire.ShaHash]struct{}),
@ -1511,31 +1390,33 @@ func newBlockManager(s *server) (*blockManager, error) {
headerList: list.New(),
quit: make(chan struct{}),
}
bm.progressLogger = newBlockProgressLogger("Processed", bmgrLog)
bm.blockChain = blockchain.New(s.db, s.chainParams, bm.handleNotifyMsg,
s.sigCache)
bm.blockChain.DisableCheckpoints(cfg.DisableCheckpoints)
// Create a new block chain instance with the appropriate configuration.
var err error
bm.chain, err = blockchain.New(&blockchain.Config{
DB: s.db,
ChainParams: s.chainParams,
Notifications: bm.handleNotifyMsg,
SigCache: s.sigCache,
})
if err != nil {
return nil, err
}
best := bm.chain.BestSnapshot()
bm.chain.DisableCheckpoints(cfg.DisableCheckpoints)
if !cfg.DisableCheckpoints {
// Initialize the next checkpoint based on the current height.
bm.nextCheckpoint = bm.findNextHeaderCheckpoint(height)
bm.nextCheckpoint = bm.findNextHeaderCheckpoint(best.Height)
if bm.nextCheckpoint != nil {
bm.resetHeaderState(newestHash, height)
bm.resetHeaderState(best.Hash, best.Height)
}
} else {
bmgrLog.Info("Checkpoints are disabled")
}
bmgrLog.Infof("Generating initial block node index. This may " +
"take a while...")
err = bm.blockChain.GenerateInitialIndex()
if err != nil {
return nil, err
}
bmgrLog.Infof("Block index generation complete")
// Initialize the chain state now that the initial block node index has
// been generated.
bm.updateChainState(newestHash, height)
bm.updateChainState(best.Hash, best.Height)
return &bm, nil
}
@ -1586,7 +1467,7 @@ func warnMultipeDBs() {
// This is intentionally not using the known db types which depend
// on the database types compiled into the binary since we want to
// detect legacy db types as well.
dbTypes := []string{"leveldb", "sqlite"}
dbTypes := []string{"ffldb", "leveldb", "sqlite"}
duplicateDbPaths := make([]string, 0, len(dbTypes)-1)
for _, dbType := range dbTypes {
if dbType == cfg.DbType {
@ -1612,18 +1493,18 @@ func warnMultipeDBs() {
}
}
// setupBlockDB loads (or creates when needed) the block database taking into
// account the selected database backend. It also contains additional logic
// such warning the user if there are multiple databases which consume space on
// the file system and ensuring the regression test database is clean when in
// regression test mode.
func setupBlockDB() (database.Db, error) {
// loadBlockDB loads (or creates when needed) the block database taking into
// account the selected database backend and returns a handle to it. It also
// contains additional logic such warning the user if there are multiple
// databases which consume space on the file system and ensuring the regression
// test database is clean when in regression test mode.
func loadBlockDB() (database.DB, error) {
// The memdb backend does not have a file path associated with it, so
// handle it uniquely. We also don't want to worry about the multiple
// database type warnings when running with the memory database.
if cfg.DbType == "memdb" {
btcdLog.Infof("Creating block database in memory.")
db, err := database.CreateDB(cfg.DbType)
db, err := database.Create(cfg.DbType)
if err != nil {
return nil, err
}
@ -1640,11 +1521,13 @@ func setupBlockDB() (database.Db, error) {
removeRegressionDB(dbPath)
btcdLog.Infof("Loading block database from '%s'", dbPath)
db, err := database.OpenDB(cfg.DbType, dbPath)
db, err := database.Open(cfg.DbType, dbPath, activeNetParams.Net)
if err != nil {
// Return the error if it's not because the database
// doesn't exist.
if err != database.ErrDbDoesNotExist {
// Return the error if it's not because the database doesn't
// exist.
if dbErr, ok := err.(database.Error); !ok || dbErr.ErrorCode !=
database.ErrDbDoesNotExist {
return nil, err
}
@ -1653,43 +1536,12 @@ func setupBlockDB() (database.Db, error) {
if err != nil {
return nil, err
}
db, err = database.CreateDB(cfg.DbType, dbPath)
db, err = database.Create(cfg.DbType, dbPath, activeNetParams.Net)
if err != nil {
return nil, err
}
}
return db, nil
}
// loadBlockDB opens the block database and returns a handle to it.
func loadBlockDB() (database.Db, error) {
db, err := setupBlockDB()
if err != nil {
return nil, err
}
// Get the latest block height from the database.
_, height, err := db.NewestSha()
if err != nil {
db.Close()
return nil, err
}
// Insert the appropriate genesis block for the bitcoin network being
// connected to if needed.
if height == -1 {
genesis := btcutil.NewBlock(activeNetParams.GenesisBlock)
_, err := db.InsertBlock(genesis)
if err != nil {
db.Close()
return nil, err
}
btcdLog.Infof("Inserted genesis block %v",
activeNetParams.GenesisHash)
height = 0
}
btcdLog.Infof("Block database loaded with block height %d", height)
btcdLog.Info("Block database loaded")
return db, nil
}

15
btcd.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2013-2014 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -81,21 +81,10 @@ func btcdMain(serverChan chan<- *server) error {
}
defer db.Close()
if cfg.DropAddrIndex {
btcdLog.Info("Deleting entire addrindex.")
err := db.DeleteAddrIndex()
if err != nil {
btcdLog.Errorf("Unable to delete the addrindex: %v", err)
return err
}
btcdLog.Info("Successfully deleted addrindex, exiting")
return nil
}
// Ensure the database is sync'd and closed on Ctrl+C.
addInterruptHandler(func() {
btcdLog.Infof("Gracefully shutting down the database...")
db.RollbackClose()
db.Close()
})
// Create server and start it.

492
chainindexer.go
View file

@ -1,492 +0,0 @@
// Copyright (c) 2013-2014 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
package main
import (
"container/heap"
"fmt"
"runtime"
"sync"
"sync/atomic"
"github.com/btcsuite/btcd/blockchain"
"github.com/btcsuite/btcd/database"
"github.com/btcsuite/btcd/txscript"
"github.com/btcsuite/btcd/wire"
"github.com/btcsuite/btcutil"
"github.com/btcsuite/golangcrypto/ripemd160"
)
type indexState int
const (
// Our two operating modes.
// We go into "CatchUp" mode when, on boot, the current best
// chain height is greater than the last block we've indexed.
// "CatchUp" mode is characterized by several concurrent worker
// goroutines indexing blocks organized by a manager goroutine.
// When in "CatchUp" mode, incoming requests to index newly solved
// blocks are backed up for later processing. Once we've finished
// catching up, we process these queued jobs, and then enter into
// "maintenance" mode.
indexCatchUp indexState = iota
// When in "maintenance" mode, we have a single worker serially
// processing incoming jobs to index newly solved blocks.
indexMaintain
)
// Limit the number of goroutines that concurrently
// build the index to catch up based on the number
// of processor cores. This help ensure the system
// stays reasonably responsive under heavy load.
var numCatchUpWorkers = runtime.NumCPU() * 3
// indexBlockMsg packages a request to have the addresses of a block indexed.
type indexBlockMsg struct {
blk *btcutil.Block
done chan struct{}
}
// writeIndexReq represents a request to have a completed address index
// committed to the database.
type writeIndexReq struct {
blk *btcutil.Block
addrIndex database.BlockAddrIndex
}
// addrIndexer provides a concurrent service for indexing the transactions of
// target blocks based on the addresses involved in the transaction.
type addrIndexer struct {
server *server
started int32
shutdown int32
state indexState
quit chan struct{}
wg sync.WaitGroup
addrIndexJobs chan *indexBlockMsg
writeRequests chan *writeIndexReq
progressLogger *blockProgressLogger
currentIndexTip int32
chainTip int32
sync.Mutex
}
// newAddrIndexer creates a new block address indexer.
// Use Start to begin processing incoming index jobs.
func newAddrIndexer(s *server) (*addrIndexer, error) {
_, chainHeight, err := s.db.NewestSha()
if err != nil {
return nil, err
}
_, lastIndexedHeight, err := s.db.FetchAddrIndexTip()
if err != nil && err != database.ErrAddrIndexDoesNotExist {
return nil, err
}
var state indexState
if chainHeight == lastIndexedHeight {
state = indexMaintain
} else {
state = indexCatchUp
}
ai := &addrIndexer{
server: s,
quit: make(chan struct{}),
state: state,
addrIndexJobs: make(chan *indexBlockMsg),
writeRequests: make(chan *writeIndexReq, numCatchUpWorkers),
currentIndexTip: lastIndexedHeight,
chainTip: chainHeight,
progressLogger: newBlockProgressLogger("Indexed addresses of",
adxrLog),
}
return ai, nil
}
// Start begins processing of incoming indexing jobs.
func (a *addrIndexer) Start() {
// Already started?
if atomic.AddInt32(&a.started, 1) != 1 {
return
}
adxrLog.Trace("Starting address indexer")
a.wg.Add(2)
go a.indexManager()
go a.indexWriter()
}
// Stop gracefully shuts down the address indexer by stopping all ongoing
// worker goroutines, waiting for them to finish their current task.
func (a *addrIndexer) Stop() error {
if atomic.AddInt32(&a.shutdown, 1) != 1 {
adxrLog.Warnf("Address indexer is already in the process of " +
"shutting down")
return nil
}
adxrLog.Infof("Address indexer shutting down")
close(a.quit)
a.wg.Wait()
return nil
}
// IsCaughtUp returns a bool representing if the address indexer has
// caught up with the best height on the main chain.
func (a *addrIndexer) IsCaughtUp() bool {
a.Lock()
defer a.Unlock()
return a.state == indexMaintain
}
// indexManager creates, and oversees worker index goroutines.
// indexManager is the main goroutine for the addresses indexer.
// It creates, and oversees worker goroutines to index incoming blocks, with
// the exact behavior depending on the current index state
// (catch up, vs maintain). Completion of catch-up mode is always proceeded by
// a gracefull transition into "maintain" mode.
// NOTE: Must be run as a goroutine.
func (a *addrIndexer) indexManager() {
if a.state == indexCatchUp {
adxrLog.Infof("Building up address index from height %v to %v.",
a.currentIndexTip+1, a.chainTip)
// Quit semaphores to gracefully shut down our worker tasks.
runningWorkers := make([]chan struct{}, 0, numCatchUpWorkers)
shutdownWorkers := func() {
for _, quit := range runningWorkers {
close(quit)
}
}
criticalShutdown := func() {
shutdownWorkers()
a.server.Stop()
}
// Spin up all of our "catch up" worker goroutines, giving them
// a quit channel and WaitGroup so we can gracefully exit if
// needed.
var workerWg sync.WaitGroup
catchUpChan := make(chan *indexBlockMsg)
for i := 0; i < numCatchUpWorkers; i++ {
quit := make(chan struct{})
runningWorkers = append(runningWorkers, quit)
workerWg.Add(1)
go a.indexCatchUpWorker(catchUpChan, &workerWg, quit)
}
// Starting from the next block after our current index tip,
// feed our workers each successive block to index until we've
// caught up to the current highest block height.
lastBlockIdxHeight := a.currentIndexTip + 1
for lastBlockIdxHeight <= a.chainTip {
targetSha, err := a.server.db.FetchBlockShaByHeight(lastBlockIdxHeight)
if err != nil {
adxrLog.Errorf("Unable to look up the sha of the "+
"next target block (height %v): %v",
lastBlockIdxHeight, err)
criticalShutdown()
goto fin
}
targetBlock, err := a.server.db.FetchBlockBySha(targetSha)
if err != nil {
// Unable to locate a target block by sha, this
// is a critical error, we may have an
// inconsistency in the DB.
adxrLog.Errorf("Unable to look up the next "+
"target block (sha %v): %v", targetSha, err)
criticalShutdown()
goto fin
}
// Send off the next job, ready to exit if a shutdown is
// signalled.
indexJob := &indexBlockMsg{blk: targetBlock}
select {
case catchUpChan <- indexJob:
lastBlockIdxHeight++
case <-a.quit:
shutdownWorkers()
goto fin
}
_, a.chainTip, err = a.server.db.NewestSha()
if err != nil {
adxrLog.Errorf("Unable to get latest block height: %v", err)
criticalShutdown()
goto fin
}
}
a.Lock()
a.state = indexMaintain
a.Unlock()
// We've finished catching up. Signal our workers to quit, and
// wait until they've all finished.
shutdownWorkers()
workerWg.Wait()
}
adxrLog.Infof("Address indexer has caught up to best height, entering " +
"maintainence mode")
// We're all caught up at this point. We now serially process new jobs
// coming in.
for {
select {
case indexJob := <-a.addrIndexJobs:
addrIndex, err := a.indexBlockAddrs(indexJob.blk)
if err != nil {
adxrLog.Errorf("Unable to index transactions of"+
" block: %v", err)
a.server.Stop()
goto fin
}
a.writeRequests <- &writeIndexReq{blk: indexJob.blk,
addrIndex: addrIndex}
case <-a.quit:
goto fin
}
}
fin:
a.wg.Done()
}
// UpdateAddressIndex asynchronously queues a newly solved block to have its
// transactions indexed by address.
func (a *addrIndexer) UpdateAddressIndex(block *btcutil.Block) {
go func() {
job := &indexBlockMsg{blk: block}
a.addrIndexJobs <- job
}()
}
// pendingIndexWrites writes is a priority queue which is used to ensure the
// address index of the block height N+1 is written when our address tip is at
// height N. This ordering is necessary to maintain index consistency in face
// of our concurrent workers, which may not necessarily finish in the order the
// jobs are handed out.
type pendingWriteQueue []*writeIndexReq
// Len returns the number of items in the priority queue. It is part of the
// heap.Interface implementation.
func (pq pendingWriteQueue) Len() int { return len(pq) }
// Less returns whether the item in the priority queue with index i should sort
// before the item with index j. It is part of the heap.Interface implementation.
func (pq pendingWriteQueue) Less(i, j int) bool {
return pq[i].blk.Height() < pq[j].blk.Height()
}
// Swap swaps the items at the passed indices in the priority queue. It is
// part of the heap.Interface implementation.
func (pq pendingWriteQueue) Swap(i, j int) { pq[i], pq[j] = pq[j], pq[i] }
// Push pushes the passed item onto the priority queue. It is part of the
// heap.Interface implementation.
func (pq *pendingWriteQueue) Push(x interface{}) {
*pq = append(*pq, x.(*writeIndexReq))
}
// Pop removes the highest priority item (according to Less) from the priority
// queue and returns it. It is part of the heap.Interface implementation.
func (pq *pendingWriteQueue) Pop() interface{} {
n := len(*pq)
item := (*pq)[n-1]
(*pq)[n-1] = nil
*pq = (*pq)[0 : n-1]
return item
}
// indexWriter commits the populated address indexes created by the
// catch up workers to the database. Since we have concurrent workers, the writer
// ensures indexes are written in ascending order to avoid a possible gap in the
// address index triggered by an unexpected shutdown.
// NOTE: Must be run as a goroutine
func (a *addrIndexer) indexWriter() {
var pendingWrites pendingWriteQueue
minHeightWrite := make(chan *writeIndexReq)
workerQuit := make(chan struct{})
writeFinished := make(chan struct{}, 1)
// Spawn a goroutine to feed our writer address indexes such
// that, if our address tip is at N, the index for block N+1 is always
// written first. We use a priority queue to enforce this condition
// while accepting new write requests.
go func() {
for {
top:
select {
case incomingWrite := <-a.writeRequests:
heap.Push(&pendingWrites, incomingWrite)
// Check if we've found a write request that
// satisfies our condition. If we have, then
// chances are we have some backed up requests
// which wouldn't be written until a previous
// request showed up. If this is the case we'll
// quickly flush our heap of now available in
// order writes. We also accept write requests
// with a block height *before* the current
// index tip, in order to re-index new prior
// blocks added to the main chain during a
// re-org.
writeReq := heap.Pop(&pendingWrites).(*writeIndexReq)
_, addrTip, _ := a.server.db.FetchAddrIndexTip()
for writeReq.blk.Height() == (addrTip+1) ||
writeReq.blk.Height() <= addrTip {
minHeightWrite <- writeReq
// Wait for write to finish so we get a
// fresh view of the addrtip.
<-writeFinished
// Break to grab a new write request
if pendingWrites.Len() == 0 {
break top
}
writeReq = heap.Pop(&pendingWrites).(*writeIndexReq)
_, addrTip, _ = a.server.db.FetchAddrIndexTip()
}
// We haven't found the proper write request yet,
// push back onto our heap and wait for the next
// request which may be our target write.
heap.Push(&pendingWrites, writeReq)
case <-workerQuit:
return
}
}
}()
out:
// Our main writer loop. Here we actually commit the populated address
// indexes to the database.
for {
select {
case nextWrite := <-minHeightWrite:
sha := nextWrite.blk.Sha()
height := nextWrite.blk.Height()
err := a.server.db.UpdateAddrIndexForBlock(sha, height,
nextWrite.addrIndex)
if err != nil {
adxrLog.Errorf("Unable to write index for block, "+
"sha %v, height %v", sha, height)
a.server.Stop()
break out
}
writeFinished <- struct{}{}
a.progressLogger.LogBlockHeight(nextWrite.blk)
case <-a.quit:
break out
}
}
close(workerQuit)
a.wg.Done()
}
// indexCatchUpWorker indexes the transactions of previously validated and
// stored blocks.
// NOTE: Must be run as a goroutine
func (a *addrIndexer) indexCatchUpWorker(workChan chan *indexBlockMsg,
wg *sync.WaitGroup, quit chan struct{}) {
out:
for {
select {
case indexJob := <-workChan:
addrIndex, err := a.indexBlockAddrs(indexJob.blk)
if err != nil {
adxrLog.Errorf("Unable to index transactions of"+
" block: %v", err)
a.server.Stop()
break out
}
a.writeRequests <- &writeIndexReq{blk: indexJob.blk,
addrIndex: addrIndex}
case <-quit:
break out
}
}
wg.Done()
}
// indexScriptPubKey indexes all data pushes greater than 8 bytes within the
// passed SPK. Our "address" index is actually a hash160 index, where in the
// ideal case the data push is either the hash160 of a publicKey (P2PKH) or
// a Script (P2SH).
func indexScriptPubKey(addrIndex database.BlockAddrIndex, scriptPubKey []byte,
locInBlock *wire.TxLoc) error {
dataPushes, err := txscript.PushedData(scriptPubKey)
if err != nil {
adxrLog.Tracef("Couldn't get pushes: %v", err)
return err
}
for _, data := range dataPushes {
// Only index pushes greater than 8 bytes.
if len(data) < 8 {
continue
}
var indexKey [ripemd160.Size]byte
// A perfect little hash160.
if len(data) <= 20 {
copy(indexKey[:], data)
// Otherwise, could be a payToPubKey or an OP_RETURN, so we'll
// make a hash160 out of it.
} else {
copy(indexKey[:], btcutil.Hash160(data))
}
addrIndex[indexKey] = append(addrIndex[indexKey], locInBlock)
}
return nil
}
// indexBlockAddrs returns a populated index of the all the transactions in the
// passed block based on the addresses involved in each transaction.
func (a *addrIndexer) indexBlockAddrs(blk *btcutil.Block) (database.BlockAddrIndex, error) {
addrIndex := make(database.BlockAddrIndex)
txLocs, err := blk.TxLoc()
if err != nil {
return nil, err
}
for txIdx, tx := range blk.Transactions() {
// Tx's offset and length in the block.
locInBlock := &txLocs[txIdx]
// Coinbases don't have any inputs.
if !blockchain.IsCoinBase(tx) {
// Index the SPK's of each input's previous outpoint
// transaction.
for _, txIn := range tx.MsgTx().TxIn {
// Lookup and fetch the referenced output's tx.
prevOut := txIn.PreviousOutPoint
txList, err := a.server.db.FetchTxBySha(&prevOut.Hash)
if len(txList) == 0 {
return nil, fmt.Errorf("transaction %v not found",
prevOut.Hash)
}
if err != nil {
adxrLog.Errorf("Error fetching tx %v: %v",
prevOut.Hash, err)
return nil, err
}
prevOutTx := txList[len(txList)-1]
inputOutPoint := prevOutTx.Tx.TxOut[prevOut.Index]
indexScriptPubKey(addrIndex, inputOutPoint.PkScript, locInBlock)
}
}
for _, txOut := range tx.MsgTx().TxOut {
indexScriptPubKey(addrIndex, txOut.PkScript, locInBlock)
}
}
return addrIndex, nil
}

33
cmd/addblock/addblock.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2013-2014 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -10,8 +10,7 @@ import (
"runtime"
"github.com/btcsuite/btcd/blockchain"
"github.com/btcsuite/btcd/database"
_ "github.com/btcsuite/btcd/database/ldb"
database "github.com/btcsuite/btcd/database2"
"github.com/btcsuite/btcd/limits"
"github.com/btcsuite/btclog"
)
@ -27,20 +26,19 @@ var (
)
// loadBlockDB opens the block database and returns a handle to it.
func loadBlockDB() (database.Db, error) {
func loadBlockDB() (database.DB, error) {
// The database name is based on the database type.
dbName := blockDbNamePrefix + "_" + cfg.DbType
if cfg.DbType == "sqlite" {
dbName = dbName + ".db"
}
dbPath := filepath.Join(cfg.DataDir, dbName)
log.Infof("Loading block database from '%s'", dbPath)
db, err := database.OpenDB(cfg.DbType, dbPath)
db, err := database.Open(cfg.DbType, dbPath, activeNetParams.Net)
if err != nil {
// Return the error if it's not because the database doesn't
// exist.
if err != database.ErrDbDoesNotExist {
if dbErr, ok := err.(database.Error); !ok || dbErr.ErrorCode !=
database.ErrDbDoesNotExist {
return nil, err
}
@ -49,20 +47,13 @@ func loadBlockDB() (database.Db, error) {
if err != nil {
return nil, err
}
db, err = database.CreateDB(cfg.DbType, dbPath)
db, err = database.Create(cfg.DbType, dbPath, activeNetParams.Net)
if err != nil {
return nil, err
}
}
// Get the latest block height from the database.
_, height, err := db.NewestSha()
if err != nil {
db.Close()
return nil, err
}
log.Infof("Block database loaded with block height %d", height)
log.Info("Block database loaded")
return db, nil
}
@ -101,7 +92,11 @@ func realMain() error {
// Create a block importer for the database and input file and start it.
// The done channel returned from start will contain an error if
// anything went wrong.
importer := newBlockImporter(db, fi)
importer, err := newBlockImporter(db, fi)
if err != nil {
log.Errorf("Failed create block importer: %v", err)
return err
}
// Perform the import asynchronously. This allows blocks to be
// processed and read in parallel. The results channel returned from

10
cmd/addblock/config.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2013-2014 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -10,15 +10,15 @@ import (
"path/filepath"
"github.com/btcsuite/btcd/chaincfg"
"github.com/btcsuite/btcd/database"
_ "github.com/btcsuite/btcd/database/ldb"
database "github.com/btcsuite/btcd/database2"
_ "github.com/btcsuite/btcd/database2/ffldb"
"github.com/btcsuite/btcd/wire"
"github.com/btcsuite/btcutil"
flags "github.com/btcsuite/go-flags"
)
const (
defaultDbType = "leveldb"
defaultDbType = "ffldb"
defaultDataFile = "bootstrap.dat"
defaultProgress = 10
)
@ -26,7 +26,7 @@ const (
var (
btcdHomeDir = btcutil.AppDataDir("btcd", false)
defaultDataDir = filepath.Join(btcdHomeDir, "data")
knownDbTypes = database.SupportedDBs()
knownDbTypes = database.SupportedDrivers()
activeNetParams = &chaincfg.MainNetParams
)

25
cmd/addblock/import.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2013-2014 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -12,8 +12,7 @@ import (
"time"
"github.com/btcsuite/btcd/blockchain"
"github.com/btcsuite/btcd/database"
_ "github.com/btcsuite/btcd/database/ldb"
database "github.com/btcsuite/btcd/database2"
"github.com/btcsuite/btcd/wire"
"github.com/btcsuite/btcutil"
)
@ -30,7 +29,7 @@ type importResults struct {
// blockImporter houses information about an ongoing import from a block data
// file to the block database.
type blockImporter struct {
db database.Db
db database.DB
chain *blockchain.BlockChain
medianTime blockchain.MedianTimeSource
r io.ReadSeeker
@ -105,7 +104,7 @@ func (bi *blockImporter) processBlock(serializedBlock []byte) (bool, error) {
// Skip blocks that already exist.
blockSha := block.Sha()
exists, err := bi.db.ExistsSha(blockSha)
exists, err := bi.chain.HaveBlock(blockSha)
if err != nil {
return false, err
}
@ -116,7 +115,7 @@ func (bi *blockImporter) processBlock(serializedBlock []byte) (bool, error) {
// Don't bother trying to process orphans.
prevHash := &block.MsgBlock().Header.PrevBlock
if !prevHash.IsEqual(&zeroHash) {
exists, err := bi.db.ExistsSha(prevHash)
exists, err := bi.chain.HaveBlock(prevHash)
if err != nil {
return false, err
}
@ -295,7 +294,15 @@ func (bi *blockImporter) Import() chan *importResults {
// newBlockImporter returns a new importer for the provided file reader seeker
// and database.
func newBlockImporter(db database.Db, r io.ReadSeeker) *blockImporter {
func newBlockImporter(db database.DB, r io.ReadSeeker) (*blockImporter, error) {
chain, err := blockchain.New(&blockchain.Config{
DB: db,
ChainParams: activeNetParams,
})
if err != nil {
return nil, err
}
return &blockImporter{
db: db,
r: r,
@ -303,8 +310,8 @@ func newBlockImporter(db database.Db, r io.ReadSeeker) *blockImporter {
doneChan: make(chan bool),
errChan: make(chan error),
quit: make(chan struct{}),
chain: blockchain.New(db, activeNetParams, nil, nil),
chain: chain,
medianTime: blockchain.NewMedianTime(),
lastLogTime: time.Now(),
}
}, nil
}

227
cmd/dropafter/dropafter.go
View file

@ -1,227 +0,0 @@
// Copyright (c) 2013 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
package main
import (
"errors"
"fmt"
"os"
"path/filepath"
"strconv"
"github.com/btcsuite/btcd/chaincfg"
"github.com/btcsuite/btcd/database"
_ "github.com/btcsuite/btcd/database/ldb"
"github.com/btcsuite/btcd/wire"
"github.com/btcsuite/btclog"
"github.com/btcsuite/btcutil"
flags "github.com/btcsuite/go-flags"
)
type config struct {
DataDir string `short:"b" long:"datadir" description:"Directory to store data"`
DbType string `long:"dbtype" description:"Database backend"`
TestNet3 bool `long:"testnet" description:"Use the test network"`
RegressionTest bool `long:"regtest" description:"Use the regression test network"`
SimNet bool `long:"simnet" description:"Use the simulation test network"`
ShaString string `short:"s" description:"Block SHA to process" required:"true"`
}
var (
btcdHomeDir = btcutil.AppDataDir("btcd", false)
defaultDataDir = filepath.Join(btcdHomeDir, "data")
log btclog.Logger
activeNetParams = &chaincfg.MainNetParams
)
const (
argSha = iota
argHeight
)
// netName returns the name used when referring to a bitcoin network. At the
// time of writing, btcd currently places blocks for testnet version 3 in the
// data and log directory "testnet", which does not match the Name field of the
// chaincfg parameters. This function can be used to override this directory name
// as "testnet" when the passed active network matches wire.TestNet3.
//
// A proper upgrade to move the data and log directories for this network to
// "testnet3" is planned for the future, at which point this function can be
// removed and the network parameter's name used instead.
func netName(chainParams *chaincfg.Params) string {
switch chainParams.Net {
case wire.TestNet3:
return "testnet"
default:
return chainParams.Name
}
}
func main() {
cfg := config{
DbType: "leveldb",
DataDir: defaultDataDir,
}
parser := flags.NewParser(&cfg, flags.Default)
_, err := parser.Parse()
if err != nil {
if e, ok := err.(*flags.Error); !ok || e.Type != flags.ErrHelp {
parser.WriteHelp(os.Stderr)
}
return
}
backendLogger := btclog.NewDefaultBackendLogger()
defer backendLogger.Flush()
log = btclog.NewSubsystemLogger(backendLogger, "")
database.UseLogger(log)
// Multiple networks can't be selected simultaneously.
funcName := "main"
numNets := 0
// Count number of network flags passed; assign active network params
// while we're at it
if cfg.TestNet3 {
numNets++
activeNetParams = &chaincfg.TestNet3Params
}
if cfg.RegressionTest {
numNets++
activeNetParams = &chaincfg.RegressionNetParams
}
if cfg.SimNet {
numNets++
activeNetParams = &chaincfg.SimNetParams
}
if numNets > 1 {
str := "%s: The testnet, regtest, and simnet params can't be " +
"used together -- choose one of the three"
err := fmt.Errorf(str, funcName)
fmt.Fprintln(os.Stderr, err)
parser.WriteHelp(os.Stderr)
return
}
cfg.DataDir = filepath.Join(cfg.DataDir, netName(activeNetParams))
blockDbNamePrefix := "blocks"
dbName := blockDbNamePrefix + "_" + cfg.DbType
if cfg.DbType == "sqlite" {
dbName = dbName + ".db"
}
dbPath := filepath.Join(cfg.DataDir, dbName)
log.Infof("loading db")
db, err := database.OpenDB(cfg.DbType, dbPath)
if err != nil {
log.Warnf("db open failed: %v", err)
return
}
defer db.Close()
log.Infof("db load complete")
_, height, err := db.NewestSha()
log.Infof("loaded block height %v", height)
sha, err := getSha(db, cfg.ShaString)
if err != nil {
log.Infof("Invalid block hash %v", cfg.ShaString)
return
}
err = db.DropAfterBlockBySha(&sha)
if err != nil {
log.Warnf("failed %v", err)
}
}
func getSha(db database.Db, str string) (wire.ShaHash, error) {
argtype, idx, sha, err := parsesha(str)
if err != nil {
log.Warnf("unable to decode [%v] %v", str, err)
return wire.ShaHash{}, err
}
switch argtype {
case argSha:
// nothing to do
case argHeight:
sha, err = db.FetchBlockShaByHeight(idx)
if err != nil {
return wire.ShaHash{}, err
}
}
if sha == nil {
fmt.Printf("wtf sha is nil but err is %v", err)
}
return *sha, nil
}
var ntxcnt int64
var txspendcnt int64
var txgivecnt int64
var errBadShaPrefix = errors.New("invalid prefix")
var errBadShaLen = errors.New("invalid len")
var errBadShaChar = errors.New("invalid character")
func parsesha(argstr string) (argtype int, height int32, psha *wire.ShaHash, err error) {
var sha wire.ShaHash
var hashbuf string
switch len(argstr) {
case 64:
hashbuf = argstr
case 66:
if argstr[0:2] != "0x" {
log.Infof("prefix is %v", argstr[0:2])
err = errBadShaPrefix
return
}
hashbuf = argstr[2:]
default:
if len(argstr) <= 16 {
// assume value is height
argtype = argHeight
var h int
h, err = strconv.Atoi(argstr)
if err == nil {
height = int32(h)
return
}
log.Infof("Unable to parse height %v, err %v", height, err)
}
err = errBadShaLen
return
}
var buf [32]byte
for idx, ch := range hashbuf {
var val rune
switch {
case ch >= '0' && ch <= '9':
val = ch - '0'
case ch >= 'a' && ch <= 'f':
val = ch - 'a' + rune(10)
case ch >= 'A' && ch <= 'F':
val = ch - 'A' + rune(10)
default:
err = errBadShaChar
return
}
b := buf[31-idx/2]
if idx&1 == 1 {
b |= byte(val)
} else {
b |= (byte(val) << 4)
}
buf[31-idx/2] = b
}
sha.SetBytes(buf[0:32])
psha = &sha
return
}

10
cmd/findcheckpoint/config.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2013 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -10,8 +10,8 @@ import (
"path/filepath"
"github.com/btcsuite/btcd/chaincfg"
"github.com/btcsuite/btcd/database"
_ "github.com/btcsuite/btcd/database/ldb"
database "github.com/btcsuite/btcd/database2"
_ "github.com/btcsuite/btcd/database2/ffldb"
"github.com/btcsuite/btcd/wire"
"github.com/btcsuite/btcutil"
flags "github.com/btcsuite/go-flags"
@ -21,13 +21,13 @@ const (
minCandidates = 1
maxCandidates = 20
defaultNumCandidates = 5
defaultDbType = "leveldb"
defaultDbType = "ffldb"
)
var (
btcdHomeDir = btcutil.AppDataDir("btcd", false)
defaultDataDir = filepath.Join(btcdHomeDir, "data")
knownDbTypes = database.SupportedDBs()
knownDbTypes = database.SupportedDrivers()
activeNetParams = &chaincfg.MainNetParams
)

44
cmd/findcheckpoint/findcheckpoint.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2013 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -11,8 +11,7 @@ import (
"github.com/btcsuite/btcd/blockchain"
"github.com/btcsuite/btcd/chaincfg"
"github.com/btcsuite/btcd/database"
_ "github.com/btcsuite/btcd/database/ldb"
database "github.com/btcsuite/btcd/database2"
"github.com/btcsuite/btcd/wire"
)
@ -23,16 +22,12 @@ var (
)
// loadBlockDB opens the block database and returns a handle to it.
func loadBlockDB() (database.Db, error) {
func loadBlockDB() (database.DB, error) {
// The database name is based on the database type.
dbType := cfg.DbType
dbName := blockDbNamePrefix + "_" + dbType
if dbType == "sqlite" {
dbName = dbName + ".db"
}
dbName := blockDbNamePrefix + "_" + cfg.DbType
dbPath := filepath.Join(cfg.DataDir, dbName)
fmt.Printf("Loading block database from '%s'\n", dbPath)
db, err := database.OpenDB(dbType, dbPath)
db, err := database.Open(cfg.DbType, dbPath, activeNetParams.Net)
if err != nil {
return nil, err
}
@ -44,16 +39,14 @@ func loadBlockDB() (database.Db, error) {
// candidates at the last checkpoint that is already hard coded into btcchain
// since there is no point in finding candidates before already existing
// checkpoints.
func findCandidates(db database.Db, latestHash *wire.ShaHash) ([]*chaincfg.Checkpoint, error) {
func findCandidates(chain *blockchain.BlockChain, latestHash *wire.ShaHash) ([]*chaincfg.Checkpoint, error) {
// Start with the latest block of the main chain.
block, err := db.FetchBlockBySha(latestHash)
block, err := chain.BlockByHash(latestHash)
if err != nil {
return nil, err
}
// Setup chain and get the latest checkpoint. Ignore notifications
// since they aren't needed for this util.
chain := blockchain.New(db, activeNetParams, nil, nil)
// Get the latest known checkpoint.
latestCheckpoint := chain.LatestCheckpoint()
if latestCheckpoint == nil {
// Set the latest checkpoint to the genesis block if there isn't
@ -115,7 +108,7 @@ func findCandidates(db database.Db, latestHash *wire.ShaHash) ([]*chaincfg.Check
}
prevHash := &block.MsgBlock().Header.PrevBlock
block, err = db.FetchBlockBySha(prevHash)
block, err = chain.BlockByHash(prevHash)
if err != nil {
return nil, err
}
@ -155,17 +148,24 @@ func main() {
}
defer db.Close()
// Get the latest block hash and height from the database and report
// status.
latestHash, height, err := db.NewestSha()
// Setup chain. Ignore notifications since they aren't needed for this
// util.
chain, err := blockchain.New(&blockchain.Config{
DB: db,
ChainParams: activeNetParams,
})
if err != nil {
fmt.Fprintln(os.Stderr, err)
fmt.Fprintf(os.Stderr, "failed to initialize chain: %v\n", err)
return
}
fmt.Printf("Block database loaded with block height %d\n", height)
// Get the latest block hash and height from the database and report
// status.
best := chain.BestSnapshot()
fmt.Printf("Block database loaded with block height %d\n", best.Height)
// Find checkpoint candidates.
candidates, err := findCandidates(db, latestHash)
candidates, err := findCandidates(chain, best.Hash)
if err != nil {
fmt.Fprintln(os.Stderr, "Unable to identify candidates:", err)
return

31
config.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2013-2014 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -16,9 +16,8 @@ import (
"strings"
"time"
"github.com/btcsuite/btcd/database"
_ "github.com/btcsuite/btcd/database/ldb"
_ "github.com/btcsuite/btcd/database/memdb"
database "github.com/btcsuite/btcd/database2"
_ "github.com/btcsuite/btcd/database2/ffldb"
"github.com/btcsuite/btcd/wire"
"github.com/btcsuite/btcutil"
flags "github.com/btcsuite/go-flags"
@ -37,7 +36,7 @@ const (
defaultMaxRPCClients = 10
defaultMaxRPCWebsockets = 25
defaultVerifyEnabled = false
defaultDbType = "leveldb"
defaultDbType = "ffldb"
defaultFreeTxRelayLimit = 15.0
defaultBlockMinSize = 0
defaultBlockMaxSize = 750000
@ -45,7 +44,6 @@ const (
blockMaxSizeMax = wire.MaxBlockPayload - 1000
defaultBlockPrioritySize = 50000
defaultGenerate = false
defaultAddrIndex = false
defaultMaxOrphanTransactions = 1000
defaultMaxOrphanTxSize = 5000
defaultSigCacheMaxSize = 50000
@ -55,7 +53,7 @@ var (
btcdHomeDir = btcutil.AppDataDir("btcd", false)
defaultConfigFile = filepath.Join(btcdHomeDir, defaultConfigFilename)
defaultDataDir = filepath.Join(btcdHomeDir, defaultDataDirname)
knownDbTypes = database.SupportedDBs()
knownDbTypes = database.SupportedDrivers()
defaultRPCKeyFile = filepath.Join(btcdHomeDir, "rpc.key")
defaultRPCCertFile = filepath.Join(btcdHomeDir, "rpc.cert")
defaultLogDir = filepath.Join(btcdHomeDir, defaultLogDirname)
@ -130,8 +128,6 @@ type config struct {
BlockMaxSize uint32 `long:"blockmaxsize" description:"Maximum block size in bytes to be used when creating a block"`
BlockPrioritySize uint32 `long:"blockprioritysize" description:"Size in bytes for high-priority/low-fee transactions when creating a block"`
GetWorkKeys []string `long:"getworkkey" description:"DEPRECATED -- Use the --miningaddr option instead"`
AddrIndex bool `long:"addrindex" description:"Build and maintain a full address index. Currently only supported by leveldb."`
DropAddrIndex bool `long:"dropaddrindex" description:"Deletes the address-based transaction index from the database on start up, and then exits."`
NoPeerBloomFilters bool `long:"nopeerbloomfilters" description:"Disable bloom filtering support."`
SigCacheMaxSize uint `long:"sigcachemaxsize" description:"The maximum number of entries in the signature verification cache."`
BlocksOnly bool `long:"blocksonly" description:"Do not accept transactions from remote peers."`
@ -346,7 +342,6 @@ func loadConfig() (*config, []string, error) {
MaxOrphanTxs: defaultMaxOrphanTransactions,
SigCacheMaxSize: defaultSigCacheMaxSize,
Generate: defaultGenerate,
AddrIndex: defaultAddrIndex,
}
// Service options which are only added on Windows.
@ -507,22 +502,6 @@ func loadConfig() (*config, []string, error) {
return nil, nil, err
}
if cfg.AddrIndex && cfg.DropAddrIndex {
err := fmt.Errorf("addrindex and dropaddrindex cannot be " +
"activated at the same")
fmt.Fprintln(os.Stderr, err)
fmt.Fprintln(os.Stderr, usageMessage)
return nil, nil, err
}
// Memdb does not currently support the addrindex.
if cfg.DbType == "memdb" && cfg.AddrIndex {
err := fmt.Errorf("memdb does not currently support the addrindex")
fmt.Fprintln(os.Stderr, err)
fmt.Fprintln(os.Stderr, usageMessage)
return nil, nil, err
}
// Validate profile port number
if cfg.Profile != "" {
profilePort, err := strconv.Atoi(cfg.Profile)

9
doc.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2013-2014 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -73,8 +73,7 @@ Application Options:
--simnet Use the simulation test network
--nocheckpoints Disable built-in checkpoints. Don't do this unless
you know what you're doing.
--dbtype= Database backend to use for the Block Chain
(leveldb)
--dbtype= Database backend to use for the Block Chain (ffldb)
--profile= Enable HTTP profiling on given port -- NOTE port
must be between 1024 and 65536
--cpuprofile= Write CPU profile to the specified file
@ -105,10 +104,6 @@ Application Options:
--blockprioritysize= Size in bytes for high-priority/low-fee transactions
when creating a block (50000)
--getworkkey= DEPRECATED -- Use the --miningaddr option instead
--addrindex Build and maintain a full address index. Currently
only supported by leveldb.
--dropaddrindex Deletes the address-based transaction index from the
database on start up, and the exits.
--nopeerbloomfilters Disable bloom filtering support.
--sigcachemaxsize= The maximum number of entries in the signature
verification cache.

2
log.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2013-2014 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.

261
mempool.go
View file

@ -15,7 +15,6 @@ import (
"time"
"github.com/btcsuite/btcd/blockchain"
"github.com/btcsuite/btcd/database"
"github.com/btcsuite/btcd/mining"
"github.com/btcsuite/btcd/txscript"
"github.com/btcsuite/btcd/wire"
@ -24,7 +23,7 @@ import (
const (
// mempoolHeight is the height used for the "block" height field of the
// contextual transaction information provided in a transaction store.
// contextual transaction information provided in a transaction view.
mempoolHeight = 0x7fffffff
)
@ -40,20 +39,18 @@ type mempoolTxDesc struct {
// mempoolConfig is a descriptor containing the memory pool configuration.
type mempoolConfig struct {
// EnableAddrIndex defines whether the address index should be enabled.
EnableAddrIndex bool
// FetchTransactionStore defines the function to use to fetch
// transacation information.
FetchTransactionStore func(*btcutil.Tx, bool) (blockchain.TxStore, error)
// NewestSha defines the function to retrieve the newest sha
NewestSha func() (*wire.ShaHash, int32, error)
// Policy defines the various mempool configuration options related
// to policy.
Policy mempoolPolicy
// FetchUtxoView defines the function to use to fetch unspent
// transaction output information.
FetchUtxoView func(*btcutil.Tx) (*blockchain.UtxoViewpoint, error)
// Chain defines the concurrent safe block chain instance which houses
// the current best chain.
Chain *blockchain.BlockChain
// RelayNtfnChan defines the channel to send newly accepted transactions
// to. If unset or set to nil, notifications will not be sent.
RelayNtfnChan chan *btcutil.Tx
@ -107,7 +104,6 @@ type txMemPool struct {
pool map[wire.ShaHash]*mempoolTxDesc
orphans map[wire.ShaHash]*btcutil.Tx
orphansByPrev map[wire.ShaHash]map[wire.ShaHash]*btcutil.Tx
addrindex map[string]map[wire.ShaHash]struct{} // maps address to txs
outpoints map[wire.OutPoint]*btcutil.Tx
pennyTotal float64 // exponentially decaying total for penny spends.
lastPennyUnix int64 // unix time of last ``penny spend''
@ -331,10 +327,6 @@ func (mp *txMemPool) removeTransaction(tx *btcutil.Tx, removeRedeemers bool) {
// Remove the transaction and mark the referenced outpoints as unspent
// by the pool.
if txDesc, exists := mp.pool[*txHash]; exists {
if mp.cfg.EnableAddrIndex {
mp.removeTransactionFromAddrIndex(tx)
}
for _, txIn := range txDesc.Tx.MsgTx().TxIn {
delete(mp.outpoints, txIn.PreviousOutPoint)
}
@ -344,52 +336,10 @@ func (mp *txMemPool) removeTransaction(tx *btcutil.Tx, removeRedeemers bool) {
}
// removeTransactionFromAddrIndex removes the passed transaction from our
// address based index.
//
// This function MUST be called with the mempool lock held (for writes).
func (mp *txMemPool) removeTransactionFromAddrIndex(tx *btcutil.Tx) error {
previousOutputScripts, err := mp.fetchReferencedOutputScripts(tx)
if err != nil {
txmpLog.Errorf("Unable to obtain referenced output scripts for "+
"the passed tx (addrindex): %v", err)
return err
}
for _, pkScript := range previousOutputScripts {
mp.removeScriptFromAddrIndex(pkScript, tx)
}
for _, txOut := range tx.MsgTx().TxOut {
mp.removeScriptFromAddrIndex(txOut.PkScript, tx)
}
return nil
}
// removeScriptFromAddrIndex dissociates the address encoded by the
// passed pkScript from the passed tx in our address based tx index.
//
// This function MUST be called with the mempool lock held (for writes).
func (mp *txMemPool) removeScriptFromAddrIndex(pkScript []byte, tx *btcutil.Tx) error {
_, addresses, _, err := txscript.ExtractPkScriptAddrs(pkScript,
activeNetParams.Params)
if err != nil {
txmpLog.Errorf("Unable to extract encoded addresses from script "+
"for addrindex (addrindex): %v", err)
return err
}
for _, addr := range addresses {
delete(mp.addrindex[addr.EncodeAddress()], *tx.Sha())
}
return nil
}
// RemoveTransaction removes the passed transaction from the mempool. If
// RemoveTransaction removes the passed transaction from the mempool. When the
// removeRedeemers flag is set, any transactions that redeem outputs from the
// removed transaction will also be removed recursively from the mempool, as
// they would otherwise become orphan.
// they would otherwise become orphans.
//
// This function is safe for concurrent access.
func (mp *txMemPool) RemoveTransaction(tx *btcutil.Tx, removeRedeemers bool) {
@ -426,7 +376,7 @@ func (mp *txMemPool) RemoveDoubleSpends(tx *btcutil.Tx) {
// helper for maybeAcceptTransaction.
//
// This function MUST be called with the mempool lock held (for writes).
func (mp *txMemPool) addTransaction(txStore blockchain.TxStore, tx *btcutil.Tx, height int32, fee int64) {
func (mp *txMemPool) addTransaction(utxoView *blockchain.UtxoViewpoint, tx *btcutil.Tx, height int32, fee int64) {
// Add the transaction to the pool and mark the referenced outpoints
// as spent by the pool.
mp.pool[*tx.Sha()] = &mempoolTxDesc{
@ -436,85 +386,12 @@ func (mp *txMemPool) addTransaction(txStore blockchain.TxStore, tx *btcutil.Tx,
Height: height,
Fee: fee,
},
StartingPriority: calcPriority(tx.MsgTx(), txStore, height),
StartingPriority: calcPriority(tx.MsgTx(), utxoView, height),
}
for _, txIn := range tx.MsgTx().TxIn {
mp.outpoints[txIn.PreviousOutPoint] = tx
}
atomic.StoreInt64(&mp.lastUpdated, time.Now().Unix())
if mp.cfg.EnableAddrIndex {
mp.addTransactionToAddrIndex(tx)
}
}
// addTransactionToAddrIndex adds all addresses related to the transaction to
// our in-memory address index. Note that this address is only populated when
// we're running with the optional address index activated.
//
// This function MUST be called with the mempool lock held (for writes).
func (mp *txMemPool) addTransactionToAddrIndex(tx *btcutil.Tx) error {
previousOutScripts, err := mp.fetchReferencedOutputScripts(tx)
if err != nil {
txmpLog.Errorf("Unable to obtain referenced output scripts for "+
"the passed tx (addrindex): %v", err)
return err
}
// Index addresses of all referenced previous output tx's.
for _, pkScript := range previousOutScripts {
mp.indexScriptAddressToTx(pkScript, tx)
}
// Index addresses of all created outputs.
for _, txOut := range tx.MsgTx().TxOut {
mp.indexScriptAddressToTx(txOut.PkScript, tx)
}
return nil
}
// fetchReferencedOutputScripts looks up and returns all the scriptPubKeys
// referenced by inputs of the passed transaction.
//
// This function MUST be called with the mempool lock held (for reads).
func (mp *txMemPool) fetchReferencedOutputScripts(tx *btcutil.Tx) ([][]byte, error) {
txStore, err := mp.fetchInputTransactions(tx, false)
if err != nil || len(txStore) == 0 {
return nil, err
}
previousOutScripts := make([][]byte, 0, len(tx.MsgTx().TxIn))
for _, txIn := range tx.MsgTx().TxIn {
outPoint := txIn.PreviousOutPoint
if txStore[outPoint.Hash].Err == nil {
referencedOutPoint := txStore[outPoint.Hash].Tx.MsgTx().TxOut[outPoint.Index]
previousOutScripts = append(previousOutScripts, referencedOutPoint.PkScript)
}
}
return previousOutScripts, nil
}
// indexScriptByAddress alters our address index by indexing the payment address
// encoded by the passed scriptPubKey to the passed transaction.
//
// This function MUST be called with the mempool lock held (for writes).
func (mp *txMemPool) indexScriptAddressToTx(pkScript []byte, tx *btcutil.Tx) error {
_, addresses, _, err := txscript.ExtractPkScriptAddrs(pkScript,
activeNetParams.Params)
if err != nil {
txmpLog.Errorf("Unable to extract encoded addresses from script "+
"for addrindex: %v", err)
return err
}
for _, addr := range addresses {
if mp.addrindex[addr.EncodeAddress()] == nil {
mp.addrindex[addr.EncodeAddress()] = make(map[wire.ShaHash]struct{})
}
mp.addrindex[addr.EncodeAddress()][*tx.Sha()] = struct{}{}
}
return nil
}
// checkPoolDoubleSpend checks whether or not the passed transaction is
@ -536,31 +413,29 @@ func (mp *txMemPool) checkPoolDoubleSpend(tx *btcutil.Tx) error {
return nil
}
// fetchInputTransactions fetches the input transactions referenced by the
// passed transaction. First, it fetches from the main chain, then it tries to
// fetch any missing inputs from the transaction pool.
// fetchInputUtxos loads utxo details about the input transactions referenced by
// the passed transaction. First, it loads the details form the viewpoint of
// the main chain, then it adjusts them based upon the contents of the
// transaction pool.
//
// This function MUST be called with the mempool lock held (for reads).
func (mp *txMemPool) fetchInputTransactions(tx *btcutil.Tx, includeSpent bool) (blockchain.TxStore, error) {
txStore, err := mp.cfg.FetchTransactionStore(tx, includeSpent)
func (mp *txMemPool) fetchInputUtxos(tx *btcutil.Tx) (*blockchain.UtxoViewpoint, error) {
utxoView, err := mp.cfg.FetchUtxoView(tx)
if err != nil {
return nil, err
}
// Attempt to populate any missing inputs from the transaction pool.
for _, txD := range txStore {
if txD.Err == database.ErrTxShaMissing || txD.Tx == nil {
if poolTxDesc, exists := mp.pool[*txD.Hash]; exists {
poolTx := poolTxDesc.Tx
txD.Tx = poolTx
txD.BlockHeight = mempoolHeight
txD.Spent = make([]bool, len(poolTx.MsgTx().TxOut))
txD.Err = nil
}
for originHash, entry := range utxoView.Entries() {
if entry != nil && !entry.IsFullySpent() {
continue
}
if poolTxDesc, exists := mp.pool[originHash]; exists {
utxoView.AddTxOuts(poolTxDesc.Tx, mempoolHeight)
}
}
return txStore, nil
return utxoView, nil
}
// FetchTransaction returns the requested transaction from the transaction pool.
@ -580,27 +455,6 @@ func (mp *txMemPool) FetchTransaction(txHash *wire.ShaHash) (*btcutil.Tx, error)
return nil, fmt.Errorf("transaction is not in the pool")
}
// FilterTransactionsByAddress returns all transactions currently in the
// mempool that either create an output to the passed address or spend a
// previously created output to the address.
func (mp *txMemPool) FilterTransactionsByAddress(addr btcutil.Address) ([]*btcutil.Tx, error) {
// Protect concurrent access.
mp.RLock()
defer mp.RUnlock()
if txs, exists := mp.addrindex[addr.EncodeAddress()]; exists {
addressTxs := make([]*btcutil.Tx, 0, len(txs))
for txHash := range txs {
if txD, exists := mp.pool[txHash]; exists {
addressTxs = append(addressTxs, txD.Tx)
}
}
return addressTxs, nil
}
return nil, fmt.Errorf("address does not have any transactions in the pool")
}
// maybeAcceptTransaction is the internal function which implements the public
// MaybeAcceptTransaction. See the comment for MaybeAcceptTransaction for
// more details.
@ -646,15 +500,10 @@ func (mp *txMemPool) maybeAcceptTransaction(tx *btcutil.Tx, isNew, rateLimit boo
}
// Get the current height of the main chain. A standalone transaction
// will be mined into the next block at best, so it's height is at least
// will be mined into the next block at best, so its height is at least
// one more than the current height.
_, curHeight, err := mp.cfg.NewestSha()
if err != nil {
// This is an unexpected error so don't turn it into a rule
// error.
return nil, err
}
nextBlockHeight := curHeight + 1
best := mp.cfg.Chain.BestSnapshot()
nextBlockHeight := best.Height + 1
// Don't allow non-standard transactions if the network parameters
// forbid their relaying.
@ -688,11 +537,11 @@ func (mp *txMemPool) maybeAcceptTransaction(tx *btcutil.Tx, isNew, rateLimit boo
return nil, err
}
// Fetch all of the transactions referenced by the inputs to this
// transaction. This function also attempts to fetch the transaction
// itself to be used for detecting a duplicate transaction without
// needing to do a separate lookup.
txStore, err := mp.fetchInputTransactions(tx, false)
// Fetch all of the unspent transaction outputs referenced by the inputs
// to this transaction. This function also attempts to fetch the
// transaction itself to be used for detecting a duplicate transaction
// without needing to do a separate lookup.
utxoView, err := mp.fetchInputUtxos(tx)
if err != nil {
if cerr, ok := err.(blockchain.RuleError); ok {
return nil, chainRuleError(cerr)
@ -702,24 +551,26 @@ func (mp *txMemPool) maybeAcceptTransaction(tx *btcutil.Tx, isNew, rateLimit boo
// Don't allow the transaction if it exists in the main chain and is not
// not already fully spent.
if txD, exists := txStore[*txHash]; exists && txD.Err == nil {
for _, isOutputSpent := range txD.Spent {
if !isOutputSpent {
return nil, txRuleError(wire.RejectDuplicate,
"transaction already exists")
}
}
txEntry := utxoView.LookupEntry(txHash)
if txEntry != nil && !txEntry.IsFullySpent() {
return nil, txRuleError(wire.RejectDuplicate,
"transaction already exists")
}
delete(txStore, *txHash)
delete(utxoView.Entries(), *txHash)
// Transaction is an orphan if any of the referenced input transactions
// don't exist. Adding orphans to the orphan pool is not handled by
// this function, and the caller should use maybeAddOrphan if this
// behavior is desired.
var missingParents []*wire.ShaHash
for _, txD := range txStore {
if txD.Err == database.ErrTxShaMissing {
missingParents = append(missingParents, txD.Hash)
for originHash, entry := range utxoView.Entries() {
if entry == nil || entry.IsFullySpent() {
// Must make a copy of the hash here since the iterator
// is replaced and taking its address directly would
// result in all of the entries pointing to the same
// memory location and thus all be the final hash.
hashCopy := originHash
missingParents = append(missingParents, &hashCopy)
}
}
if len(missingParents) > 0 {
@ -730,7 +581,8 @@ func (mp *txMemPool) maybeAcceptTransaction(tx *btcutil.Tx, isNew, rateLimit boo
// rules in btcchain for what transactions are allowed into blocks.
// Also returns the fees associated with the transaction which will be
// used later.
txFee, err := blockchain.CheckTransactionInputs(tx, nextBlockHeight, txStore)
txFee, err := blockchain.CheckTransactionInputs(tx, nextBlockHeight,
utxoView)
if err != nil {
if cerr, ok := err.(blockchain.RuleError); ok {
return nil, chainRuleError(cerr)
@ -741,7 +593,7 @@ func (mp *txMemPool) maybeAcceptTransaction(tx *btcutil.Tx, isNew, rateLimit boo
// Don't allow transactions with non-standard inputs if the network
// parameters forbid their relaying.
if !activeNetParams.RelayNonStdTxs {
err := checkInputsStandard(tx, txStore)
err := checkInputsStandard(tx, utxoView)
if err != nil {
// Attempt to extract a reject code from the error so
// it can be retained. When not possible, fall back to
@ -765,7 +617,7 @@ func (mp *txMemPool) maybeAcceptTransaction(tx *btcutil.Tx, isNew, rateLimit boo
// the coinbase address itself can contain signature operations, the
// maximum allowed signature operations per transaction is less than
// the maximum allowed signature operations per block.
numSigOps, err := blockchain.CountP2SHSigOps(tx, false, txStore)
numSigOps, err := blockchain.CountP2SHSigOps(tx, false, utxoView)
if err != nil {
if cerr, ok := err.(blockchain.RuleError); ok {
return nil, chainRuleError(cerr)
@ -805,7 +657,7 @@ func (mp *txMemPool) maybeAcceptTransaction(tx *btcutil.Tx, isNew, rateLimit boo
// memory pool from blocks that have been disconnected during a reorg
// are exempted.
if isNew && !mp.cfg.Policy.DisableRelayPriority && txFee < minFee {
currentPriority := calcPriority(tx.MsgTx(), txStore,
currentPriority := calcPriority(tx.MsgTx(), utxoView,
nextBlockHeight)
if currentPriority <= minHighPriority {
str := fmt.Sprintf("transaction %v has insufficient "+
@ -841,7 +693,7 @@ func (mp *txMemPool) maybeAcceptTransaction(tx *btcutil.Tx, isNew, rateLimit boo
// Verify crypto signatures for each input and reject the transaction if
// any don't verify.
err = blockchain.ValidateTransactionScripts(tx, txStore,
err = blockchain.ValidateTransactionScripts(tx, utxoView,
txscript.StandardVerifyFlags, mp.cfg.SigCache)
if err != nil {
if cerr, ok := err.(blockchain.RuleError); ok {
@ -851,7 +703,7 @@ func (mp *txMemPool) maybeAcceptTransaction(tx *btcutil.Tx, isNew, rateLimit boo
}
// Add to transaction pool.
mp.addTransaction(txStore, tx, curHeight, txFee)
mp.addTransaction(utxoView, tx, best.Height, txFee)
txmpLog.Debugf("Accepted transaction %v (pool size: %v)", txHash,
len(mp.pool))
@ -1118,8 +970,5 @@ func newTxMemPool(cfg *mempoolConfig) *txMemPool {
orphansByPrev: make(map[wire.ShaHash]map[wire.ShaHash]*btcutil.Tx),
outpoints: make(map[wire.OutPoint]*btcutil.Tx),
}
if cfg.EnableAddrIndex {
memPool.addrindex = make(map[string]map[wire.ShaHash]struct{})
}
return memPool
}

123
mining.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2014 The btcsuite developers
// Copyright (c) 2014-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -11,7 +11,6 @@ import (
"time"
"github.com/btcsuite/btcd/blockchain"
"github.com/btcsuite/btcd/database"
"github.com/btcsuite/btcd/mining"
"github.com/btcsuite/btcd/txscript"
"github.com/btcsuite/btcd/wire"
@ -182,16 +181,17 @@ type BlockTemplate struct {
ValidPayAddress bool
}
// mergeTxStore adds all of the transactions in txStoreB to txStoreA. The
// result is that txStoreA will contain all of its original transactions plus
// all of the transactions in txStoreB.
func mergeTxStore(txStoreA blockchain.TxStore, txStoreB blockchain.TxStore) {
for hash, txDataB := range txStoreB {
if txDataA, exists := txStoreA[hash]; !exists ||
(txDataA.Err == database.ErrTxShaMissing &&
txDataB.Err != database.ErrTxShaMissing) {
// mergeUtxoView adds all of the entries in view to viewA. The result is that
// viewA will contain all of its original entries plus all of the entries
// in viewB. It will replace any entries in viewB which also exist in viewA
// if the entry in viewA is fully spent.
func mergeUtxoView(viewA *blockchain.UtxoViewpoint, viewB *blockchain.UtxoViewpoint) {
viewAEntries := viewA.Entries()
for hash, entryB := range viewB.Entries() {
if entryA, exists := viewAEntries[hash]; !exists ||
entryA == nil || entryA.IsFullySpent() {
txStoreA[hash] = txDataB
viewAEntries[hash] = entryB
}
}
}
@ -249,26 +249,20 @@ func createCoinbaseTx(coinbaseScript []byte, nextBlockHeight int32, addr btcutil
return btcutil.NewTx(tx), nil
}
// spendTransaction updates the passed transaction store by marking the inputs
// to the passed transaction as spent. It also adds the passed transaction to
// the store at the provided height.
func spendTransaction(txStore blockchain.TxStore, tx *btcutil.Tx, height int32) error {
// spendTransaction updates the passed view by marking the inputs to the passed
// transaction as spent. It also adds all outputs in the passed transaction
// which are not provably unspendable as available unspent transaction outputs.
func spendTransaction(utxoView *blockchain.UtxoViewpoint, tx *btcutil.Tx, height int32) error {
for _, txIn := range tx.MsgTx().TxIn {
originHash := &txIn.PreviousOutPoint.Hash
originIndex := txIn.PreviousOutPoint.Index
if originTx, exists := txStore[*originHash]; exists {
originTx.Spent[originIndex] = true
entry := utxoView.LookupEntry(originHash)
if entry != nil {
entry.SpendOutput(originIndex)
}
}
txStore[*tx.Sha()] = &blockchain.TxData{
Tx: tx,
Hash: tx.Sha(),
BlockHeight: height,
Spent: make([]bool, len(tx.MsgTx().TxOut)),
Err: nil,
}
utxoView.AddTxOuts(tx, height)
return nil
}
@ -430,12 +424,12 @@ func NewBlockTemplate(policy *mining.Policy, server *server, payToAddress btcuti
priorityQueue := newTxPriorityQueue(len(sourceTxns), sortedByFee)
// Create a slice to hold the transactions to be included in the
// generated block with reserved space. Also create a transaction
// store to house all of the input transactions so multiple lookups
// can be avoided.
// generated block with reserved space. Also create a utxo view to
// house all of the input transactions so multiple lookups can be
// avoided.
blockTxns := make([]*btcutil.Tx, 0, len(sourceTxns))
blockTxns = append(blockTxns, coinbaseTx)
blockTxStore := make(blockchain.TxStore)
blockUtxos := blockchain.NewUtxoViewpoint()
// dependers is used to track transactions which depend on another
// transaction in the source pool. This, in conjunction with the
@ -474,15 +468,15 @@ mempoolLoop:
continue
}
// Fetch all of the transactions referenced by the inputs to
// this transaction. NOTE: This intentionally does not fetch
// inputs from the mempool since a transaction which depends on
// other transactions in the mempool must come after those
// Fetch all of the utxos referenced by the this transaction.
// NOTE: This intentionally does not fetch inputs from the
// mempool since a transaction which depends on other
// transactions in the mempool must come after those
// dependencies in the final generated block.
txStore, err := blockManager.FetchTransactionStore(tx)
utxos, err := blockManager.chain.FetchUtxoView(tx)
if err != nil {
minrLog.Warnf("Unable to fetch transaction store for "+
"tx %s: %v", tx.Sha(), err)
minrLog.Warnf("Unable to fetch utxo view for tx %s: "+
"%v", tx.Sha(), err)
continue
}
@ -493,13 +487,13 @@ mempoolLoop:
for _, txIn := range tx.MsgTx().TxIn {
originHash := &txIn.PreviousOutPoint.Hash
originIndex := txIn.PreviousOutPoint.Index
txData, exists := txStore[*originHash]
if !exists || txData.Err != nil || txData.Tx == nil {
utxoEntry := utxos.LookupEntry(originHash)
if utxoEntry == nil || utxoEntry.IsOutputSpent(originIndex) {
if !txSource.HaveTransaction(originHash) {
minrLog.Tracef("Skipping tx %s because "+
"it references tx %s which is "+
"not available", tx.Sha,
originHash)
"it references unspent output "+
"%s which is not available",
tx.Sha(), txIn.PreviousOutPoint)
continue mempoolLoop
}
@ -522,23 +516,13 @@ mempoolLoop:
// referenced transaction is available.
continue
}
// Ensure the output index in the referenced transaction
// is available.
msgTx := txData.Tx.MsgTx()
if originIndex > uint32(len(msgTx.TxOut)) {
minrLog.Tracef("Skipping tx %s because "+
"it references output %d of tx %s "+
"which is out of bounds", tx.Sha,
originIndex, originHash)
continue mempoolLoop
}
}
// Calculate the final transaction priority using the input
// value age sum as well as the adjusted transaction size. The
// formula is: sum(inputValue * inputAge) / adjustedTxSize
prioItem.priority = calcPriority(tx.MsgTx(), txStore, nextBlockHeight)
prioItem.priority = calcPriority(tx.MsgTx(), utxos,
nextBlockHeight)
// Calculate the fee in Satoshi/kB.
txSize := tx.MsgTx().SerializeSize()
@ -551,10 +535,10 @@ mempoolLoop:
heap.Push(priorityQueue, prioItem)
}
// Merge the store which contains all of the input transactions
// for this transaction into the input transaction store. This
// allows the code below to avoid a second lookup.
mergeTxStore(blockTxStore, txStore)
// Merge the referenced outputs from the input transactions to
// this transaction into the block utxo view. This allows the
// code below to avoid a second lookup.
mergeUtxoView(blockUtxos, utxos)
}
minrLog.Tracef("Priority queue len %d, dependers len %d",
@ -602,7 +586,7 @@ mempoolLoop:
continue
}
numP2SHSigOps, err := blockchain.CountP2SHSigOps(tx, false,
blockTxStore)
blockUtxos)
if err != nil {
minrLog.Tracef("Skipping tx %s due to error in "+
"CountP2SHSigOps: %v", tx.Sha(), err)
@ -666,14 +650,14 @@ mempoolLoop:
// Ensure the transaction inputs pass all of the necessary
// preconditions before allowing it to be added to the block.
_, err = blockchain.CheckTransactionInputs(tx, nextBlockHeight,
blockTxStore)
blockUtxos)
if err != nil {
minrLog.Tracef("Skipping tx %s due to error in "+
"CheckTransactionInputs: %v", tx.Sha(), err)
logSkippedDeps(tx, deps)
continue
}
err = blockchain.ValidateTransactionScripts(tx, blockTxStore,
err = blockchain.ValidateTransactionScripts(tx, blockUtxos,
txscript.StandardVerifyFlags, server.sigCache)
if err != nil {
minrLog.Tracef("Skipping tx %s due to error in "+
@ -682,11 +666,11 @@ mempoolLoop:
continue
}
// Spend the transaction inputs in the block transaction store
// and add an entry for it to ensure any transactions which
// reference this one have it available as an input and can
// ensure they aren't double spending.
spendTransaction(blockTxStore, tx, nextBlockHeight)
// Spend the transaction inputs in the block utxo view and add
// an entry for it to ensure any transactions which reference
// this one have it available as an input and can ensure they
// aren't double spending.
spendTransaction(blockUtxos, tx, nextBlockHeight)
// Add the transaction to the block, increment counters, and
// save the fees and signature operation counts to the block
@ -733,7 +717,7 @@ mempoolLoop:
if err != nil {
return nil, err
}
requiredDifficulty, err := blockManager.CalcNextRequiredDifficulty(ts)
reqDifficulty, err := blockManager.chain.CalcNextRequiredDifficulty(ts)
if err != nil {
return nil, err
}
@ -746,7 +730,7 @@ mempoolLoop:
PrevBlock: *prevHash,
MerkleRoot: *merkles[len(merkles)-1],
Timestamp: ts,
Bits: requiredDifficulty,
Bits: reqDifficulty,
}
for _, tx := range blockTxns {
if err := msgBlock.AddTransaction(tx.MsgTx()); err != nil {
@ -759,7 +743,7 @@ mempoolLoop:
// chain with no issues.
block := btcutil.NewBlock(&msgBlock)
block.SetHeight(nextBlockHeight)
if err := blockManager.CheckConnectBlock(block); err != nil {
if err := blockManager.chain.CheckConnectBlock(block); err != nil {
return nil, err
}
@ -797,7 +781,8 @@ func UpdateBlockTime(msgBlock *wire.MsgBlock, bManager *blockManager) error {
// If running on a network that requires recalculating the difficulty,
// do so now.
if activeNetParams.ResetMinDifficulty {
difficulty, err := bManager.CalcNextRequiredDifficulty(newTimestamp)
difficulty, err := bManager.chain.CalcNextRequiredDifficulty(
newTimestamp)
if err != nil {
return err
}

40
policy.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2013-2015 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -57,9 +57,9 @@ const (
func calcMinRequiredTxRelayFee(serializedSize int64, minRelayTxFee btcutil.Amount) int64 {
// Calculate the minimum fee for a transaction to be allowed into the
// mempool and relayed by scaling the base fee (which is the minimum
// free transaction relay fee). minTxRelayFee is in Satoshi/kB so
// multiply by serializedSize (which is in bytes) and divide by 1000 to get
// minimum Satoshis.
// free transaction relay fee). minTxRelayFee is in Satoshi/kB so
// multiply by serializedSize (which is in bytes) and divide by 1000 to
// get minimum Satoshis.
minFee := (serializedSize * int64(minRelayTxFee)) / 1000
if minFee == 0 && minRelayTxFee > 0 {
@ -79,7 +79,7 @@ func calcMinRequiredTxRelayFee(serializedSize int64, minRelayTxFee btcutil.Amoun
// of each of its input values multiplied by their age (# of confirmations).
// Thus, the final formula for the priority is:
// sum(inputValue * inputAge) / adjustedTxSize
func calcPriority(tx *wire.MsgTx, txStore blockchain.TxStore, nextBlockHeight int32) float64 {
func calcPriority(tx *wire.MsgTx, utxoView *blockchain.UtxoViewpoint, nextBlockHeight int32) float64 {
// In order to encourage spending multiple old unspent transaction
// outputs thereby reducing the total set, don't count the constant
// overhead for each input as well as enough bytes of the signature
@ -111,7 +111,7 @@ func calcPriority(tx *wire.MsgTx, txStore blockchain.TxStore, nextBlockHeight in
return 0.0
}
inputValueAge := calcInputValueAge(tx, txStore, nextBlockHeight)
inputValueAge := calcInputValueAge(tx, utxoView, nextBlockHeight)
return inputValueAge / float64(serializedTxSize-overhead)
}
@ -121,29 +121,29 @@ func calcPriority(tx *wire.MsgTx, txStore blockchain.TxStore, nextBlockHeight in
// age is the sum of this value for each txin. Any inputs to the transaction
// which are currently in the mempool and hence not mined into a block yet,
// contribute no additional input age to the transaction.
func calcInputValueAge(tx *wire.MsgTx, txStore blockchain.TxStore, nextBlockHeight int32) float64 {
func calcInputValueAge(tx *wire.MsgTx, utxoView *blockchain.UtxoViewpoint, nextBlockHeight int32) float64 {
var totalInputAge float64
for _, txIn := range tx.TxIn {
// Don't attempt to accumulate the total input age if the
// referenced transaction output doesn't exist.
originHash := &txIn.PreviousOutPoint.Hash
originIndex := txIn.PreviousOutPoint.Index
// Don't attempt to accumulate the total input age if the txIn
// in question doesn't exist.
if txData, exists := txStore[*originHash]; exists && txData.Tx != nil {
txEntry := utxoView.LookupEntry(originHash)
if txEntry != nil && !txEntry.IsOutputSpent(originIndex) {
// Inputs with dependencies currently in the mempool
// have their block height set to a special constant.
// Their input age should computed as zero since their
// parent hasn't made it into a block yet.
// Their input age should be computed as zero since
// their parent hasn't made it into a block yet.
var inputAge int32
if txData.BlockHeight == mempoolHeight {
originHeight := txEntry.BlockHeight()
if originHeight == mempoolHeight {
inputAge = 0
} else {
inputAge = nextBlockHeight - txData.BlockHeight
inputAge = nextBlockHeight - originHeight
}
// Sum the input value times age.
originTxOut := txData.Tx.MsgTx().TxOut[originIndex]
inputValue := originTxOut.Value
inputValue := txEntry.AmountByIndex(originIndex)
totalInputAge += float64(inputValue * int64(inputAge))
}
}
@ -158,7 +158,7 @@ func calcInputValueAge(tx *wire.MsgTx, txStore blockchain.TxStore, nextBlockHeig
// exhaustion attacks by "creative" use of scripts that are super expensive to
// process like OP_DUP OP_CHECKSIG OP_DROP repeated a large number of times
// followed by a final OP_TRUE.
func checkInputsStandard(tx *btcutil.Tx, txStore blockchain.TxStore) error {
func checkInputsStandard(tx *btcutil.Tx, utxoView *blockchain.UtxoViewpoint) error {
// NOTE: The reference implementation also does a coinbase check here,
// but coinbases have already been rejected prior to calling this
// function so no need to recheck.
@ -168,8 +168,8 @@ func checkInputsStandard(tx *btcutil.Tx, txStore blockchain.TxStore) error {
// they have already been checked prior to calling this
// function.
prevOut := txIn.PreviousOutPoint
originTx := txStore[prevOut.Hash].Tx.MsgTx()
originPkScript := originTx.TxOut[prevOut.Index].PkScript
entry := utxoView.LookupEntry(&prevOut.Hash)
originPkScript := entry.PkScriptByIndex(prevOut.Index)
// Calculate stats for the script pair.
scriptInfo, err := txscript.CalcScriptInfo(txIn.SignatureScript,

975
rpcserver.go
View file

File diff suppressed because it is too large Load diff

73
rpcwebsocket.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2013-2015 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -14,11 +14,13 @@ import (
"errors"
"fmt"
"io"
"math"
"sync"
"time"
"github.com/btcsuite/btcd/blockchain"
"github.com/btcsuite/btcd/btcjson"
"github.com/btcsuite/btcd/database"
database "github.com/btcsuite/btcd/database2"
"github.com/btcsuite/btcd/txscript"
"github.com/btcsuite/btcd/wire"
"github.com/btcsuite/btcutil"
@ -1783,10 +1785,10 @@ func rescanBlock(wsc *wsClient, lookups *rescanKeys, blk *btcutil.Block) {
// verifies that the new range of blocks is on the same fork as a previous
// range of blocks. If this condition does not hold true, the JSON-RPC error
// for an unrecoverable reorganize is returned.
func recoverFromReorg(db database.Db, minBlock, maxBlock int32,
func recoverFromReorg(chain *blockchain.BlockChain, minBlock, maxBlock int32,
lastBlock *wire.ShaHash) ([]wire.ShaHash, error) {
hashList, err := db.FetchHeightRange(minBlock, maxBlock)
hashList, err := chain.HeightRange(minBlock, maxBlock)
if err != nil {
rpcsLog.Errorf("Error looking up block range: %v", err)
return nil, &btcjson.RPCError{
@ -1797,7 +1799,8 @@ func recoverFromReorg(db database.Db, minBlock, maxBlock int32,
if lastBlock == nil || len(hashList) == 0 {
return hashList, nil
}
blk, err := db.FetchBlockBySha(&hashList[0])
blk, err := chain.BlockByHash(&hashList[0])
if err != nil {
rpcsLog.Errorf("Error looking up possibly reorged block: %v",
err)
@ -1843,12 +1846,13 @@ func handleRescan(wsc *wsClient, icmd interface{}) (interface{}, error) {
outpoints := make([]*wire.OutPoint, 0, len(cmd.OutPoints))
for i := range cmd.OutPoints {
blockHash, err := wire.NewShaHashFromStr(cmd.OutPoints[i].Hash)
cmdOutpoint := &cmd.OutPoints[i]
blockHash, err := wire.NewShaHashFromStr(cmdOutpoint.Hash)
if err != nil {
return nil, rpcDecodeHexError(cmd.OutPoints[i].Hash)
return nil, rpcDecodeHexError(cmdOutpoint.Hash)
}
index := cmd.OutPoints[i].Index
outpoints = append(outpoints, wire.NewOutPoint(blockHash, index))
outpoint := wire.NewOutPoint(blockHash, cmdOutpoint.Index)
outpoints = append(outpoints, outpoint)
}
numAddrs := len(cmd.Addresses)
@ -1916,13 +1920,13 @@ func handleRescan(wsc *wsClient, icmd interface{}) (interface{}, error) {
lookups.unspent[*outpoint] = struct{}{}
}
db := wsc.server.server.db
chain := wsc.server.chain
minBlockSha, err := wire.NewShaHashFromStr(cmd.BeginBlock)
minBlockHash, err := wire.NewShaHashFromStr(cmd.BeginBlock)
if err != nil {
return nil, rpcDecodeHexError(cmd.BeginBlock)
}
minBlock, err := db.FetchBlockHeightBySha(minBlockSha)
minBlock, err := chain.BlockHeightByHash(minBlockHash)
if err != nil {
return nil, &btcjson.RPCError{
Code: btcjson.ErrRPCBlockNotFound,
@ -1930,13 +1934,13 @@ func handleRescan(wsc *wsClient, icmd interface{}) (interface{}, error) {
}
}
maxBlock := database.AllShas
maxBlock := int32(math.MaxInt32)
if cmd.EndBlock != nil {
maxBlockSha, err := wire.NewShaHashFromStr(*cmd.EndBlock)
maxBlockHash, err := wire.NewShaHashFromStr(*cmd.EndBlock)
if err != nil {
return nil, rpcDecodeHexError(*cmd.EndBlock)
}
maxBlock, err = db.FetchBlockHeightBySha(maxBlockSha)
maxBlock, err = chain.BlockHeightByHash(maxBlockHash)
if err != nil {
return nil, &btcjson.RPCError{
Code: btcjson.ErrRPCBlockNotFound,
@ -1955,11 +1959,20 @@ func handleRescan(wsc *wsClient, icmd interface{}) (interface{}, error) {
ticker := time.NewTicker(10 * time.Second)
defer ticker.Stop()
// FetchHeightRange may not return a complete list of block shas for
// the given range, so fetch range as many times as necessary.
// Instead of fetching all block shas at once, fetch in smaller chunks
// to ensure large rescans consume a limited amount of memory.
fetchRange:
for minBlock < maxBlock {
hashList, err := db.FetchHeightRange(minBlock, maxBlock)
// Limit the max number of hashes to fetch at once to the
// maximum number of items allowed in a single inventory.
// This value could be higher since it's not creating inventory
// messages, but this mirrors the limiting logic used in the
// peer-to-peer protocol.
maxLoopBlock := maxBlock
if maxLoopBlock-minBlock > wire.MaxInvPerMsg {
maxLoopBlock = minBlock + wire.MaxInvPerMsg
}
hashList, err := chain.HeightRange(minBlock, maxLoopBlock)
if err != nil {
rpcsLog.Errorf("Error looking up block range: %v", err)
return nil, &btcjson.RPCError{
@ -1971,7 +1984,7 @@ fetchRange:
// The rescan is finished if no blocks hashes for this
// range were successfully fetched and a stop block
// was provided.
if maxBlock != database.AllShas {
if maxBlock != math.MaxInt32 {
break
}
@ -1987,10 +2000,12 @@ fetchRange:
// continuous notifications if necessary. Otherwise,
// continue the fetch loop again to rescan the new
// blocks (or error due to an irrecoverable reorganize).
pauseGuard := wsc.server.server.blockManager.Pause()
curHash, _, err := db.NewestSha()
blockManager := wsc.server.server.blockManager
pauseGuard := blockManager.Pause()
best := blockManager.chain.BestSnapshot()
curHash := best.Hash
again := true
if err == nil && (lastBlockHash == nil || *lastBlockHash == *curHash) {
if lastBlockHash == nil || *lastBlockHash == *curHash {
again = false
n := wsc.server.ntfnMgr
n.RegisterSpentRequests(wsc, lookups.unspentSlice())
@ -2014,11 +2029,13 @@ fetchRange:
loopHashList:
for i := range hashList {
blk, err := db.FetchBlockBySha(&hashList[i])
blk, err := chain.BlockByHash(&hashList[i])
if err != nil {
// Only handle reorgs if a block could not be
// found for the hash.
if err != database.ErrBlockShaMissing {
if dbErr, ok := err.(database.Error); !ok ||
dbErr.ErrorCode != database.ErrBlockNotFound {
rpcsLog.Errorf("Error looking up "+
"block: %v", err)
return nil, &btcjson.RPCError{
@ -2030,7 +2047,7 @@ fetchRange:
// If an absolute max block was specified, don't
// attempt to handle the reorg.
if maxBlock != database.AllShas {
if maxBlock != math.MaxInt32 {
rpcsLog.Errorf("Stopping rescan for "+
"reorged block %v",
cmd.EndBlock)
@ -2048,8 +2065,8 @@ fetchRange:
// before the range was evaluated, as it must be
// reevaluated for the new hashList.
minBlock += int32(i)
hashList, err = recoverFromReorg(db, minBlock,
maxBlock, lastBlockHash)
hashList, err = recoverFromReorg(chain,
minBlock, maxBlock, lastBlockHash)
if err != nil {
return nil, err
}
@ -2118,7 +2135,7 @@ fetchRange:
// is needed to safely inform clients that all rescan notifications have
// been sent.
n := btcjson.NewRescanFinishedNtfn(lastBlockHash.String(),
int32(lastBlock.Height()),
lastBlock.Height(),
lastBlock.MsgBlock().Header.Timestamp.Unix())
if mn, err := btcjson.MarshalCmd(nil, n); err != nil {
rpcsLog.Errorf("Failed to marshal rescan finished "+

5
sample-btcd.conf
View file

@ -153,6 +153,7 @@
; Disable peer bloom filtering. See BIP0111.
; nopeerbloomfilters=1
; ------------------------------------------------------------------------------
; RPC server options - The following options control the built-in RPC server
; which is used to control and query information from a running btcd process.
@ -216,6 +217,7 @@
; the default).
; notls=1
; ------------------------------------------------------------------------------
; Mempool Settings - The following options
; ------------------------------------------------------------------------------
@ -236,6 +238,7 @@
; Do not accept transactions from remote peers.
; blocksonly=1
; ------------------------------------------------------------------------------
; Optional Transaction Indexes
; ------------------------------------------------------------------------------
@ -245,6 +248,7 @@
; Delete the entire address index on start up, then exit.
; dropaddrindex=0
; ------------------------------------------------------------------------------
; Signature Verification Cache
; ------------------------------------------------------------------------------
@ -252,6 +256,7 @@
; Limit the signature cache to a max of 50000 entries.
; sigcachemaxsize=50000
; ------------------------------------------------------------------------------
; Coin Generation (Mining) Settings - The following options control the
; generation of block templates used by external mining applications through RPC

234
server.go
View file

@ -5,6 +5,7 @@
package main
import (
"bytes"
"crypto/rand"
"encoding/binary"
"errors"
@ -22,7 +23,7 @@ import (
"github.com/btcsuite/btcd/addrmgr"
"github.com/btcsuite/btcd/blockchain"
"github.com/btcsuite/btcd/chaincfg"
"github.com/btcsuite/btcd/database"
database "github.com/btcsuite/btcd/database2"
"github.com/btcsuite/btcd/mining"
"github.com/btcsuite/btcd/peer"
"github.com/btcsuite/btcd/txscript"
@ -180,7 +181,6 @@ type server struct {
sigCache *txscript.SigCache
rpcServer *rpcServer
blockManager *blockManager
addrIndexer *addrIndexer
txMemPool *txMemPool
cpuMiner *CPUMiner
relayNtfnChan chan *btcutil.Tx
@ -198,7 +198,7 @@ type server struct {
wg sync.WaitGroup
quit chan struct{}
nat NAT
db database.Db
db database.DB
timeSource blockchain.MedianTimeSource
services wire.ServiceFlag
}
@ -241,6 +241,13 @@ func newServerPeer(s *server, isPersistent bool) *serverPeer {
}
}
// newestBlock returns the current best block hash and height using the format
// required by the configuration for the peer package.
func (sp *serverPeer) newestBlock() (*wire.ShaHash, int32, error) {
best := sp.server.blockManager.chain.BestSnapshot()
return best.Hash, best.Height, nil
}
// addKnownAddresses adds the given addresses to the set of known addreses to
// the peer to prevent sending duplicate addresses.
func (sp *serverPeer) addKnownAddresses(addresses []*wire.NetAddress) {
@ -608,14 +615,13 @@ func (sp *serverPeer) OnGetData(p *peer.Peer, msg *wire.MsgGetData) {
// OnGetBlocks is invoked when a peer receives a getblocks bitcoin
// message.
func (sp *serverPeer) OnGetBlocks(p *peer.Peer, msg *wire.MsgGetBlocks) {
db := sp.server.db
// Return all block hashes to the latest one (up to max per message) if
// no stop hash was specified.
// Attempt to find the ending index of the stop hash if specified.
endIdx := database.AllShas
chain := sp.server.blockManager.chain
endIdx := int32(math.MaxInt32)
if !msg.HashStop.IsEqual(&zeroHash) {
height, err := db.FetchBlockHeightBySha(&msg.HashStop)
height, err := chain.BlockHeightByHash(&msg.HashStop)
if err == nil {
endIdx = height + 1
}
@ -628,7 +634,7 @@ func (sp *serverPeer) OnGetBlocks(p *peer.Peer, msg *wire.MsgGetBlocks) {
// This mirrors the behavior in the reference implementation.
startIdx := int32(1)
for _, hash := range msg.BlockLocatorHashes {
height, err := db.FetchBlockHeightBySha(hash)
height, err := chain.BlockHeightByHash(hash)
if err == nil {
// Start with the next hash since we know this one.
startIdx = height + 1
@ -643,34 +649,18 @@ func (sp *serverPeer) OnGetBlocks(p *peer.Peer, msg *wire.MsgGetBlocks) {
autoContinue = true
}
// Fetch the inventory from the block database.
hashList, err := chain.HeightRange(startIdx, endIdx)
if err != nil {
peerLog.Warnf("Block lookup failed: %v", err)
return
}
// Generate inventory message.
//
// The FetchBlockBySha call is limited to a maximum number of hashes
// per invocation. Since the maximum number of inventory per message
// might be larger, call it multiple times with the appropriate indices
// as needed.
invMsg := wire.NewMsgInv()
for start := startIdx; start < endIdx; {
// Fetch the inventory from the block database.
hashList, err := db.FetchHeightRange(start, endIdx)
if err != nil {
peerLog.Warnf("Block lookup failed: %v", err)
return
}
// The database did not return any further hashes. Break out of
// the loop now.
if len(hashList) == 0 {
break
}
// Add block inventory to the message.
for _, hash := range hashList {
hashCopy := hash
iv := wire.NewInvVect(wire.InvTypeBlock, &hashCopy)
invMsg.AddInvVect(iv)
}
start += int32(len(hashList))
for i := range hashList {
iv := wire.NewInvVect(wire.InvTypeBlock, &hashList[i])
invMsg.AddInvVect(iv)
}
// Send the inventory message if there is anything to send.
@ -696,11 +686,10 @@ func (sp *serverPeer) OnGetHeaders(p *peer.Peer, msg *wire.MsgGetHeaders) {
return
}
db := sp.server.db
// Attempt to look up the height of the provided stop hash.
endIdx := database.AllShas
height, err := db.FetchBlockHeightBySha(&msg.HashStop)
chain := sp.server.blockManager.chain
endIdx := int32(math.MaxInt32)
height, err := chain.BlockHeightByHash(&msg.HashStop)
if err == nil {
endIdx = height + 1
}
@ -711,20 +700,34 @@ func (sp *serverPeer) OnGetHeaders(p *peer.Peer, msg *wire.MsgGetHeaders) {
// No blocks with the stop hash were found so there is nothing
// to do. Just return. This behavior mirrors the reference
// implementation.
if endIdx == database.AllShas {
if endIdx == math.MaxInt32 {
return
}
// Fetch and send the requested block header.
header, err := db.FetchBlockHeaderBySha(&msg.HashStop)
// Fetch the raw block header bytes from the database.
var headerBytes []byte
err := sp.server.db.View(func(dbTx database.Tx) error {
var err error
headerBytes, err = dbTx.FetchBlockHeader(&msg.HashStop)
return err
})
if err != nil {
peerLog.Warnf("Lookup of known block hash failed: %v",
err)
return
}
// Deserialize the block header.
var header wire.BlockHeader
err = header.Deserialize(bytes.NewReader(headerBytes))
if err != nil {
peerLog.Warnf("Block header deserialize failed: %v",
err)
return
}
headersMsg := wire.NewMsgHeaders()
headersMsg.AddBlockHeader(header)
headersMsg.AddBlockHeader(&header)
p.QueueMessage(headersMsg, nil)
return
}
@ -736,7 +739,7 @@ func (sp *serverPeer) OnGetHeaders(p *peer.Peer, msg *wire.MsgGetHeaders) {
// This mirrors the behavior in the reference implementation.
startIdx := int32(1)
for _, hash := range msg.BlockLocatorHashes {
height, err := db.FetchBlockHeightBySha(hash)
height, err := chain.BlockHeightByHash(hash)
if err == nil {
// Start with the next hash since we know this one.
startIdx = height + 1
@ -749,42 +752,37 @@ func (sp *serverPeer) OnGetHeaders(p *peer.Peer, msg *wire.MsgGetHeaders) {
endIdx = startIdx + wire.MaxBlockHeadersPerMsg
}
// Generate headers message and send it.
//
// The FetchHeightRange call is limited to a maximum number of hashes
// per invocation. Since the maximum number of headers per message
// might be larger, call it multiple times with the appropriate indices
// as needed.
headersMsg := wire.NewMsgHeaders()
for start := startIdx; start < endIdx; {
// Fetch the inventory from the block database.
hashList, err := db.FetchHeightRange(start, endIdx)
if err != nil {
peerLog.Warnf("Header lookup failed: %v", err)
return
}
// The database did not return any further hashes. Break out of
// the loop now.
if len(hashList) == 0 {
break
}
// Add headers to the message.
for _, hash := range hashList {
header, err := db.FetchBlockHeaderBySha(&hash)
if err != nil {
peerLog.Warnf("Lookup of known block hash "+
"failed: %v", err)
continue
}
headersMsg.AddBlockHeader(header)
}
// Start at the next block header after the latest one on the
// next loop iteration.
start += int32(len(hashList))
// Fetch the inventory from the block database.
hashList, err := chain.HeightRange(startIdx, endIdx)
if err != nil {
peerLog.Warnf("Header lookup failed: %v", err)
return
}
// Generate headers message and send it.
headersMsg := wire.NewMsgHeaders()
err = sp.server.db.View(func(dbTx database.Tx) error {
for i := range hashList {
headerBytes, err := dbTx.FetchBlockHeader(&hashList[i])
if err != nil {
return err
}
var header wire.BlockHeader
err = header.Deserialize(bytes.NewReader(headerBytes))
if err != nil {
return err
}
headersMsg.AddBlockHeader(&header)
}
return nil
})
if err != nil {
peerLog.Warnf("Failed to build headers: %v", err)
return
}
p.QueueMessage(headersMsg, nil)
}
@ -983,11 +981,30 @@ func (s *server) pushTxMsg(sp *serverPeer, sha *wire.ShaHash, doneChan chan<- st
// pushBlockMsg sends a block message for the provided block hash to the
// connected peer. An error is returned if the block hash is not known.
func (s *server) pushBlockMsg(sp *serverPeer, sha *wire.ShaHash, doneChan chan<- struct{}, waitChan <-chan struct{}) error {
blk, err := s.db.FetchBlockBySha(sha)
func (s *server) pushBlockMsg(sp *serverPeer, hash *wire.ShaHash, doneChan chan<- struct{}, waitChan <-chan struct{}) error {
// Fetch the raw block bytes from the database.
var blockBytes []byte
err := sp.server.db.View(func(dbTx database.Tx) error {
var err error
blockBytes, err = dbTx.FetchBlock(hash)
return err
})
if err != nil {
peerLog.Tracef("Unable to fetch requested block sha %v: %v",
sha, err)
peerLog.Tracef("Unable to fetch requested block hash %v: %v",
hash, err)
if doneChan != nil {
doneChan <- struct{}{}
}
return err
}
// Deserialize the block.
var msgBlock wire.MsgBlock
err = msgBlock.Deserialize(bytes.NewReader(blockBytes))
if err != nil {
peerLog.Tracef("Unable to deserialize requested block hash "+
"%v: %v", hash, err)
if doneChan != nil {
doneChan <- struct{}{}
@ -1004,11 +1021,11 @@ func (s *server) pushBlockMsg(sp *serverPeer, sha *wire.ShaHash, doneChan chan<-
// an inv straight after.
var dc chan<- struct{}
continueHash := sp.continueHash
sendInv := continueHash != nil && continueHash.IsEqual(sha)
sendInv := continueHash != nil && continueHash.IsEqual(hash)
if !sendInv {
dc = doneChan
}
sp.QueueMessage(blk.MsgBlock(), dc)
sp.QueueMessage(&msgBlock, dc)
// When the peer requests the final block that was advertised in
// response to a getblocks message which requested more blocks than
@ -1016,16 +1033,12 @@ func (s *server) pushBlockMsg(sp *serverPeer, sha *wire.ShaHash, doneChan chan<-
// to trigger it to issue another getblocks message for the next
// batch of inventory.
if sendInv {
hash, _, err := s.db.NewestSha()
if err == nil {
invMsg := wire.NewMsgInvSizeHint(1)
iv := wire.NewInvVect(wire.InvTypeBlock, hash)
invMsg.AddInvVect(iv)
sp.QueueMessage(invMsg, doneChan)
sp.continueHash = nil
} else if doneChan != nil {
doneChan <- struct{}{}
}
best := sp.server.blockManager.chain.BestSnapshot()
invMsg := wire.NewMsgInvSizeHint(1)
iv := wire.NewInvVect(wire.InvTypeBlock, best.Hash)
invMsg.AddInvVect(iv)
sp.QueueMessage(invMsg, doneChan)
sp.continueHash = nil
}
return nil
}
@ -1034,7 +1047,7 @@ func (s *server) pushBlockMsg(sp *serverPeer, sha *wire.ShaHash, doneChan chan<-
// the connected peer. Since a merkle block requires the peer to have a filter
// loaded, this call will simply be ignored if there is no filter loaded. An
// error is returned if the block hash is not known.
func (s *server) pushMerkleBlockMsg(sp *serverPeer, sha *wire.ShaHash, doneChan chan<- struct{}, waitChan <-chan struct{}) error {
func (s *server) pushMerkleBlockMsg(sp *serverPeer, hash *wire.ShaHash, doneChan chan<- struct{}, waitChan <-chan struct{}) error {
// Do not send a response if the peer doesn't have a filter loaded.
if !sp.filter.IsLoaded() {
if doneChan != nil {
@ -1043,10 +1056,11 @@ func (s *server) pushMerkleBlockMsg(sp *serverPeer, sha *wire.ShaHash, doneChan
return nil
}
blk, err := s.db.FetchBlockBySha(sha)
// Fetch the raw block bytes from the database.
blk, err := sp.server.blockManager.chain.BlockByHash(hash)
if err != nil {
peerLog.Tracef("Unable to fetch requested block sha %v: %v",
sha, err)
peerLog.Tracef("Unable to fetch requested block hash %v: %v",
hash, err)
if doneChan != nil {
doneChan <- struct{}{}
@ -1477,7 +1491,7 @@ func newPeerConfig(sp *serverPeer) *peer.Config {
// other implementations' alert messages, we will not relay theirs.
OnAlert: nil,
},
NewestBlock: sp.server.db.NewestSha,
NewestBlock: sp.newestBlock,
BestLocalAddress: sp.server.addrManager.GetBestLocalAddress,
HostToNetAddress: sp.server.addrManager.HostToNetAddress,
Proxy: cfg.Proxy,
@ -1815,9 +1829,6 @@ out:
}
}
if cfg.AddrIndex {
s.addrIndexer.Stop()
}
s.blockManager.Stop()
s.addrManager.Stop()
@ -2128,10 +2139,6 @@ func (s *server) Start() {
if cfg.Generate {
s.cpuMiner.Start()
}
if cfg.AddrIndex {
s.addrIndexer.Start()
}
}
// Stop gracefully shuts down the server by stopping and disconnecting all
@ -2319,7 +2326,7 @@ out:
// newServer returns a new btcd server configured to listen on addr for the
// bitcoin network type specified by chainParams. Use start to begin accepting
// connections from peers.
func newServer(listenAddrs []string, db database.Db, chainParams *chaincfg.Params) (*server, error) {
func newServer(listenAddrs []string, db database.DB, chainParams *chaincfg.Params) (*server, error) {
services := defaultServices
if cfg.NoPeerBloomFilters {
services &^= wire.SFNodeBloom
@ -2480,9 +2487,6 @@ func newServer(listenAddrs []string, db database.Db, chainParams *chaincfg.Param
s.blockManager = bm
txC := mempoolConfig{
EnableAddrIndex: cfg.AddrIndex,
FetchTransactionStore: s.blockManager.blockChain.FetchTransactionStore,
NewestSha: s.db.NewestSha,
Policy: mempoolPolicy{
DisableRelayPriority: cfg.NoRelayPriority,
FreeTxRelayLimit: cfg.FreeTxRelayLimit,
@ -2491,6 +2495,8 @@ func newServer(listenAddrs []string, db database.Db, chainParams *chaincfg.Param
MaxSigOpsPerTx: blockchain.MaxSigOpsPerBlock / 5,
MinRelayTxFee: cfg.minRelayTxFee,
},
FetchUtxoView: s.blockManager.chain.FetchUtxoView,
Chain: s.blockManager.chain,
RelayNtfnChan: s.relayNtfnChan,
SigCache: s.sigCache,
TimeSource: s.timeSource,
@ -2508,14 +2514,6 @@ func newServer(listenAddrs []string, db database.Db, chainParams *chaincfg.Param
}
s.cpuMiner = newCPUMiner(&policy, &s)
if cfg.AddrIndex {
ai, err := newAddrIndexer(&s)
if err != nil {
return nil, err
}
s.addrIndexer = ai
}
if !cfg.DisableRPC {
s.rpcServer, err = newRPCServer(cfg.RPCListeners, &policy, &s)
if err != nil {

3
wire/msgheaders.go
View file

@ -1,4 +1,4 @@
// Copyright (c) 2013-2015 The btcsuite developers
// Copyright (c) 2013-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
@ -104,7 +104,6 @@ func (msg *MsgHeaders) BtcEncode(w io.Writer, pver uint32) error {
if err != nil {
return err
}
}
return nil