af3ed803f5
This commit contains a complete redesign and rewrite of the database package that approaches things in a vastly different manner than the previous version. This is the first part of several stages that will be needed to ultimately make use of this new package. Some of the reason for this were discussed in #255, however a quick summary is as follows: - The previous database could only contain blocks on the main chain and reorgs required deleting the blocks from the database. This made it impossible to store orphans and could make external RPC calls for information about blocks during the middle of a reorg fail. - The previous database interface forced a high level of bitcoin-specific intelligence such as spend tracking into each backend driver. - The aforementioned point led to making it difficult to implement new backend drivers due to the need to repeat a lot of non-trivial logic which is better handled at a higher layer, such as the blockchain package. - The old database stored all blocks in leveldb. This made it extremely inefficient to do things such as lookup headers and individual transactions since the entire block had to be loaded from leveldb (which entails it doing data copies) to get access. In order to address all of these concerns, and others not mentioned, the database interface has been redesigned as follows: - Two main categories of functionality are provided: block storage and metadata storage - All block storage and metadata storage are done via read-only and read-write MVCC transactions with both manual and managed modes - Support for multiple concurrent readers and a single writer - Readers use a snapshot and therefore are not blocked by the writer - Some key properties of the block storage and retrieval API: - It is generic and does NOT contain additional bitcoin logic such spend tracking and block linking - Provides access to the raw serialized bytes so deserialization is not forced for callers that don't need it - Support for fetching headers via independent functions which allows implementations to provide significant optimizations - Ability to efficiently retrieve arbitrary regions of blocks (transactions, scripts, etc) - A rich metadata storage API is provided: - Key/value with arbitrary data - Support for buckets and nested buckets - Bucket iteration through a couple of different mechanisms - Cursors for efficient and direct key seeking - Supports registration of backend database implementations - Comprehensive test coverage - Provides strong documentation with example usage This commit also contains an implementation of the previously discussed interface named ffldb (flat file plus leveldb metadata backend). Here is a quick overview: - Highly optimized for read performance with consistent write performance regardless of database size - All blocks are stored in flat files on the file system - Bulk block region fetching is optimized to perform linear reads which improves performance on spindle disks - Anti-corruption mechanisms: - Flat files contain full block checksums to quickly an easily detect database corruption without needing to do expensive merkle root calculations - Metadata checksums - Open reconciliation - Extensive test coverage: - Comprehensive blackbox interface testing - Whitebox testing which uses intimate knowledge to exercise uncommon failure paths such as deleting files out from under the database - Corruption tests (replacing random data in the files) In addition, this commit also contains a new tool under the new database directory named dbtool which provides a few basic commands for testing the database. It is designed around commands, so it could be useful to expand on in the future. Finally, this commit addresses the following issues: - Adds support for and therefore closes #255 - Fixes #199 - Fixes #201 - Implements and closes #256 - Obsoletes and closes #257 - Closes #247 once the required chain and btcd modifications are in place to make use of this new code
197 lines
6.9 KiB
Go
197 lines
6.9 KiB
Go
// Copyright (c) 2015-2016 The btcsuite developers
|
|
// Use of this source code is governed by an ISC
|
|
// license that can be found in the LICENSE file.
|
|
|
|
package database2
|
|
|
|
import "fmt"
|
|
|
|
// ErrorCode identifies a kind of error.
|
|
type ErrorCode int
|
|
|
|
// These constants are used to identify a specific database Error.
|
|
const (
|
|
// **************************************
|
|
// Errors related to driver registration.
|
|
// **************************************
|
|
|
|
// ErrDbTypeRegistered indicates two different database drivers
|
|
// attempt to register with the name database type.
|
|
ErrDbTypeRegistered ErrorCode = iota
|
|
|
|
// *************************************
|
|
// Errors related to database functions.
|
|
// *************************************
|
|
|
|
// ErrDbUnknownType indicates there is no driver registered for
|
|
// the specified database type.
|
|
ErrDbUnknownType
|
|
|
|
// ErrDbDoesNotExist indicates open is called for a database that
|
|
// does not exist.
|
|
ErrDbDoesNotExist
|
|
|
|
// ErrDbExists indicates create is called for a database that
|
|
// already exists.
|
|
ErrDbExists
|
|
|
|
// ErrDbNotOpen indicates a database instance is accessed before
|
|
// it is opened or after it is closed.
|
|
ErrDbNotOpen
|
|
|
|
// ErrDbAlreadyOpen indicates open was called on a database that
|
|
// is already open.
|
|
ErrDbAlreadyOpen
|
|
|
|
// ErrInvalid indicates the specified database is not valid.
|
|
ErrInvalid
|
|
|
|
// ErrCorruption indicates a checksum failure occurred which invariably
|
|
// means the database is corrupt.
|
|
ErrCorruption
|
|
|
|
// ****************************************
|
|
// Errors related to database transactions.
|
|
// ****************************************
|
|
|
|
// ErrTxClosed indicates an attempt was made to commit or rollback a
|
|
// transaction that has already had one of those operations performed.
|
|
ErrTxClosed
|
|
|
|
// ErrTxNotWritable indicates an operation that requires write access to
|
|
// the database was attempted against a read-only transaction.
|
|
ErrTxNotWritable
|
|
|
|
// **************************************
|
|
// Errors related to metadata operations.
|
|
// **************************************
|
|
|
|
// ErrBucketNotFound indicates an attempt to access a bucket that has
|
|
// not been created yet.
|
|
ErrBucketNotFound
|
|
|
|
// ErrBucketExists indicates an attempt to create a bucket that already
|
|
// exists.
|
|
ErrBucketExists
|
|
|
|
// ErrBucketNameRequired indicates an attempt to create a bucket with a
|
|
// blank name.
|
|
ErrBucketNameRequired
|
|
|
|
// ErrKeyRequired indicates at attempt to insert a zero-length key.
|
|
ErrKeyRequired
|
|
|
|
// ErrKeyTooLarge indicates an attmempt to insert a key that is larger
|
|
// than the max allowed key size. The max key size depends on the
|
|
// specific backend driver being used. As a general rule, key sizes
|
|
// should be relatively, so this should rarely be an issue.
|
|
ErrKeyTooLarge
|
|
|
|
// ErrValueTooLarge indicates an attmpt to insert a value that is larger
|
|
// than max allowed value size. The max key size depends on the
|
|
// specific backend driver being used.
|
|
ErrValueTooLarge
|
|
|
|
// ErrIncompatibleValue indicates the value in question is invalid for
|
|
// the specific requested operation. For example, trying create or
|
|
// delete a bucket with an existing non-bucket key, attempting to create
|
|
// or delete a non-bucket key with an existing bucket key, or trying to
|
|
// delete a value via a cursor when it points to a nested bucket.
|
|
ErrIncompatibleValue
|
|
|
|
// ***************************************
|
|
// Errors related to block I/O operations.
|
|
// ***************************************
|
|
|
|
// ErrBlockNotFound indicates a block with the provided hash does not
|
|
// exist in the database.
|
|
ErrBlockNotFound
|
|
|
|
// ErrBlockExists indicates a block with the provided hash already
|
|
// exists in the database.
|
|
ErrBlockExists
|
|
|
|
// ErrBlockRegionInvalid indicates a region that exceeds the bounds of
|
|
// the specified block was requested. When the hash provided by the
|
|
// region does not correspond to an existing block, the error will be
|
|
// ErrBlockNotFound instead.
|
|
ErrBlockRegionInvalid
|
|
|
|
// ***********************************
|
|
// Support for driver-specific errors.
|
|
// ***********************************
|
|
|
|
// ErrDriverSpecific indicates the Err field is a driver-specific error.
|
|
// This provides a mechanism for drivers to plug-in their own custom
|
|
// errors for any situations which aren't already covered by the error
|
|
// codes provided by this package.
|
|
ErrDriverSpecific
|
|
|
|
// numErrorCodes is the maximum error code number used in tests.
|
|
numErrorCodes
|
|
)
|
|
|
|
// Map of ErrorCode values back to their constant names for pretty printing.
|
|
var errorCodeStrings = map[ErrorCode]string{
|
|
ErrDbTypeRegistered: "ErrDbTypeRegistered",
|
|
ErrDbUnknownType: "ErrDbUnknownType",
|
|
ErrDbDoesNotExist: "ErrDbDoesNotExist",
|
|
ErrDbExists: "ErrDbExists",
|
|
ErrDbNotOpen: "ErrDbNotOpen",
|
|
ErrDbAlreadyOpen: "ErrDbAlreadyOpen",
|
|
ErrInvalid: "ErrInvalid",
|
|
ErrCorruption: "ErrCorruption",
|
|
ErrTxClosed: "ErrTxClosed",
|
|
ErrTxNotWritable: "ErrTxNotWritable",
|
|
ErrBucketNotFound: "ErrBucketNotFound",
|
|
ErrBucketExists: "ErrBucketExists",
|
|
ErrBucketNameRequired: "ErrBucketNameRequired",
|
|
ErrKeyRequired: "ErrKeyRequired",
|
|
ErrKeyTooLarge: "ErrKeyTooLarge",
|
|
ErrValueTooLarge: "ErrValueTooLarge",
|
|
ErrIncompatibleValue: "ErrIncompatibleValue",
|
|
ErrBlockNotFound: "ErrBlockNotFound",
|
|
ErrBlockExists: "ErrBlockExists",
|
|
ErrBlockRegionInvalid: "ErrBlockRegionInvalid",
|
|
ErrDriverSpecific: "ErrDriverSpecific",
|
|
}
|
|
|
|
// String returns the ErrorCode as a human-readable name.
|
|
func (e ErrorCode) String() string {
|
|
if s := errorCodeStrings[e]; s != "" {
|
|
return s
|
|
}
|
|
return fmt.Sprintf("Unknown ErrorCode (%d)", int(e))
|
|
}
|
|
|
|
// Error provides a single type for errors that can happen during database
|
|
// operation. It is used to indicate several types of failures including errors
|
|
// with caller requests such as specifying invalid block regions or attempting
|
|
// to access data against closed database transactions, driver errors, errors
|
|
// retrieving data, and errors communicating with database servers.
|
|
//
|
|
// The caller can use type assertions to determine if an error is an Error and
|
|
// access the ErrorCode field to ascertain the specific reason for the failure.
|
|
//
|
|
// The ErrDriverSpecific error code will also have the Err field set with the
|
|
// underlying error. Depending on the backend driver, the Err field might be
|
|
// set to the underlying error for other error codes as well.
|
|
type Error struct {
|
|
ErrorCode ErrorCode // Describes the kind of error
|
|
Description string // Human readable description of the issue
|
|
Err error // Underlying error
|
|
}
|
|
|
|
// Error satisfies the error interface and prints human-readable errors.
|
|
func (e Error) Error() string {
|
|
if e.Err != nil {
|
|
return e.Description + ": " + e.Err.Error()
|
|
}
|
|
return e.Description
|
|
}
|
|
|
|
// makeError creates an Error given a set of arguments. The error code must
|
|
// be one of the error codes provided by this package.
|
|
func makeError(c ErrorCode, desc string, err error) Error {
|
|
return Error{ErrorCode: c, Description: desc, Err: err}
|
|
}
|