lbcd/database2/internal/treap/treapiter.go
Dave Collins af3ed803f5 database: Major redesign of database package.
This commit contains a complete redesign and rewrite of the database
package that approaches things in a vastly different manner than the
previous version.  This is the first part of several stages that will be
needed to ultimately make use of this new package.

Some of the reason for this were discussed in #255, however a quick
summary is as follows:

- The previous database could only contain blocks on the main chain and
  reorgs required deleting the blocks from the database.  This made it
  impossible to store orphans and could make external RPC calls for
  information about blocks during the middle of a reorg fail.
- The previous database interface forced a high level of bitcoin-specific
  intelligence such as spend tracking into each backend driver.
- The aforementioned point led to making it difficult to implement new
  backend drivers due to the need to repeat a lot of non-trivial logic
  which is better handled at a higher layer, such as the blockchain
  package.
- The old database stored all blocks in leveldb.  This made it extremely
  inefficient to do things such as lookup headers and individual
  transactions since the entire block had to be loaded from leveldb (which
  entails it doing data copies) to get access.

In order to address all of these concerns, and others not mentioned, the
database interface has been redesigned as follows:

- Two main categories of functionality are provided: block storage and
  metadata storage
- All block storage and metadata storage are done via read-only and
  read-write MVCC transactions with both manual and managed modes
  - Support for multiple concurrent readers and a single writer
  - Readers use a snapshot and therefore are not blocked by the writer
- Some key properties of the block storage and retrieval API:
  - It is generic and does NOT contain additional bitcoin logic such spend
    tracking and block linking
  - Provides access to the raw serialized bytes so deserialization is not
    forced for callers that don't need it
  - Support for fetching headers via independent functions which allows
    implementations to provide significant optimizations
  - Ability to efficiently retrieve arbitrary regions of blocks
    (transactions, scripts, etc)
- A rich metadata storage API is provided:
  - Key/value with arbitrary data
  - Support for buckets and nested buckets
  - Bucket iteration through a couple of different mechanisms
  - Cursors for efficient and direct key seeking
- Supports registration of backend database implementations
- Comprehensive test coverage
- Provides strong documentation with example usage

This commit also contains an implementation of the previously discussed
interface named ffldb (flat file plus leveldb metadata backend).  Here
is a quick overview:

- Highly optimized for read performance with consistent write performance
  regardless of database size
- All blocks are stored in flat files on the file system
- Bulk block region fetching is optimized to perform linear reads which
  improves performance on spindle disks
- Anti-corruption mechanisms:
  - Flat files contain full block checksums to quickly an easily detect
    database corruption without needing to do expensive merkle root
    calculations
  - Metadata checksums
  - Open reconciliation
- Extensive test coverage:
  - Comprehensive blackbox interface testing
  - Whitebox testing which uses intimate knowledge to exercise uncommon
    failure paths such as deleting files out from under the database
  - Corruption tests (replacing random data in the files)

In addition, this commit also contains a new tool under the new database
directory named dbtool which provides a few basic commands for testing the
database.  It is designed around commands, so it could be useful to expand
on in the future.

Finally, this commit addresses the following issues:

- Adds support for and therefore closes #255
- Fixes #199
- Fixes #201
- Implements and closes #256
- Obsoletes and closes #257
- Closes #247 once the required chain and btcd modifications are in place
  to make use of this new code
2016-02-03 11:42:04 -06:00

354 lines
11 KiB
Go

// Copyright (c) 2015-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
package treap
import "bytes"
// Iterator represents an iterator for forwards and backwards iteration over
// the contents of a treap (mutable or immutable).
type Iterator struct {
t *Mutable // Mutable treap iterator is associated with or nil
root *treapNode // Root node of treap iterator is associated with
node *treapNode // The node the iterator is positioned at
parents parentStack // The stack of parents needed to iterate
isNew bool // Whether the iterator has been positioned
seekKey []byte // Used to handle dynamic updates for mutable treap
startKey []byte // Used to limit the iterator to a range
limitKey []byte // Used to limit the iterator to a range
}
// limitIterator clears the current iterator node if it is outside of the range
// specified when the iterator was created. It returns whether the iterator is
// valid.
func (iter *Iterator) limitIterator() bool {
if iter.node == nil {
return false
}
node := iter.node
if iter.startKey != nil && bytes.Compare(node.key, iter.startKey) < 0 {
iter.node = nil
return false
}
if iter.limitKey != nil && bytes.Compare(node.key, iter.limitKey) >= 0 {
iter.node = nil
return false
}
return true
}
// seek moves the iterator based on the provided key and flags.
//
// When the exact match flag is set, the iterator will either be moved to first
// key in the treap that exactly matches the provided key, or the one
// before/after it depending on the greater flag.
//
// When the exact match flag is NOT set, the iterator will be moved to the first
// key in the treap before/after the provided key depending on the greater flag.
//
// In all cases, the limits specified when the iterator was created are
// respected.
func (iter *Iterator) seek(key []byte, exactMatch bool, greater bool) bool {
iter.node = nil
iter.parents = parentStack{}
var selectedNodeDepth int
for node := iter.root; node != nil; {
iter.parents.Push(node)
// Traverse left or right depending on the result of the
// comparison. Also, set the iterator to the node depending on
// the flags so the iterator is positioned properly when an
// exact match isn't found.
compareResult := bytes.Compare(key, node.key)
if compareResult < 0 {
if greater {
iter.node = node
selectedNodeDepth = iter.parents.Len() - 1
}
node = node.left
continue
}
if compareResult > 0 {
if !greater {
iter.node = node
selectedNodeDepth = iter.parents.Len() - 1
}
node = node.right
continue
}
// The key is an exact match. Set the iterator and return now
// when the exact match flag is set.
if exactMatch {
iter.node = node
iter.parents.Pop()
return iter.limitIterator()
}
// The key is an exact match, but the exact match is not set, so
// choose which direction to go based on whether the larger or
// smaller key was requested.
if greater {
node = node.right
} else {
node = node.left
}
}
// There was either no exact match or there was an exact match but the
// exact match flag was not set. In any case, the parent stack might
// need to be adjusted to only include the parents up to the selected
// node. Also, ensure the selected node's key does not exceed the
// allowed range of the iterator.
for i := iter.parents.Len(); i > selectedNodeDepth; i-- {
iter.parents.Pop()
}
return iter.limitIterator()
}
// First moves the iterator to the first key/value pair. When there is only a
// single key/value pair both First and Last will point to the same pair.
// Returns false if there are no key/value pairs.
func (iter *Iterator) First() bool {
// Seek the start key if the iterator was created with one. This will
// result in either an exact match, the first greater key, or an
// exhausted iterator if no such key exists.
iter.isNew = false
if iter.startKey != nil {
return iter.seek(iter.startKey, true, true)
}
// The smallest key is in the left-most node.
iter.parents = parentStack{}
for node := iter.root; node != nil; node = node.left {
if node.left == nil {
iter.node = node
return true
}
iter.parents.Push(node)
}
return false
}
// Last moves the iterator to the last key/value pair. When there is only a
// single key/value pair both First and Last will point to the same pair.
// Returns false if there are no key/value pairs.
func (iter *Iterator) Last() bool {
// Seek the limit key if the iterator was created with one. This will
// result in the first key smaller than the limit key, or an exhausted
// iterator if no such key exists.
iter.isNew = false
if iter.limitKey != nil {
return iter.seek(iter.limitKey, false, false)
}
// The highest key is in the right-most node.
iter.parents = parentStack{}
for node := iter.root; node != nil; node = node.right {
if node.right == nil {
iter.node = node
return true
}
iter.parents.Push(node)
}
return false
}
// Next moves the iterator to the next key/value pair and returns false when the
// iterator is exhausted. When invoked on a newly created iterator it will
// position the iterator at the first item.
func (iter *Iterator) Next() bool {
if iter.isNew {
return iter.First()
}
if iter.node == nil {
return false
}
// Reseek the previous key without allowing for an exact match if a
// force seek was requested. This results in the key greater than the
// previous one or an exhausted iterator if there is no such key.
if seekKey := iter.seekKey; seekKey != nil {
iter.seekKey = nil
return iter.seek(seekKey, false, true)
}
// When there is no right node walk the parents until the parent's right
// node is not equal to the previous child. This will be the next node.
if iter.node.right == nil {
parent := iter.parents.Pop()
for parent != nil && parent.right == iter.node {
iter.node = parent
parent = iter.parents.Pop()
}
iter.node = parent
return iter.limitIterator()
}
// There is a right node, so the next node is the left-most node down
// the right sub-tree.
iter.parents.Push(iter.node)
iter.node = iter.node.right
for node := iter.node.left; node != nil; node = node.left {
iter.parents.Push(iter.node)
iter.node = node
}
return iter.limitIterator()
}
// Prev moves the iterator to the previous key/value pair and returns false when
// the iterator is exhausted. When invoked on a newly created iterator it will
// position the iterator at the last item.
func (iter *Iterator) Prev() bool {
if iter.isNew {
return iter.Last()
}
if iter.node == nil {
return false
}
// Reseek the previous key without allowing for an exact match if a
// force seek was requested. This results in the key smaller than the
// previous one or an exhausted iterator if there is no such key.
if seekKey := iter.seekKey; seekKey != nil {
iter.seekKey = nil
return iter.seek(seekKey, false, false)
}
// When there is no left node walk the parents until the parent's left
// node is not equal to the previous child. This will be the previous
// node.
for iter.node.left == nil {
parent := iter.parents.Pop()
for parent != nil && parent.left == iter.node {
iter.node = parent
parent = iter.parents.Pop()
}
iter.node = parent
return iter.limitIterator()
}
// There is a left node, so the previous node is the right-most node
// down the left sub-tree.
iter.parents.Push(iter.node)
iter.node = iter.node.left
for node := iter.node.right; node != nil; node = node.right {
iter.parents.Push(iter.node)
iter.node = node
}
return iter.limitIterator()
}
// Seek moves the iterator to the first key/value pair with a key that is
// greater than or equal to the given key and returns true if successful.
func (iter *Iterator) Seek(key []byte) bool {
iter.isNew = false
return iter.seek(key, true, true)
}
// Key returns the key of the current key/value pair or nil when the iterator
// is exhausted. The caller should not modify the contents of the returned
// slice.
func (iter *Iterator) Key() []byte {
if iter.node == nil {
return nil
}
return iter.node.key
}
// Value returns the value of the current key/value pair or nil when the
// iterator is exhausted. The caller should not modify the contents of the
// returned slice.
func (iter *Iterator) Value() []byte {
if iter.node == nil {
return nil
}
return iter.node.value
}
// Valid indicates whether the iterator is positioned at a valid key/value pair.
// It will be considered invalid when the iterator is newly created or exhausted.
func (iter *Iterator) Valid() bool {
return iter.node != nil
}
// ForceReseek notifies the iterator that the underlying mutable treap has been
// updated, so the next call to Prev or Next needs to reseek in order to allow
// the iterator to continue working properly.
//
// NOTE: Calling this function when the iterator is associated with an immutable
// treap has no effect as you would expect.
func (iter *Iterator) ForceReseek() {
// Nothing to do when the iterator is associated with an immutable
// treap.
if iter.t == nil {
return
}
// Update the iterator root to the mutable treap root in case it
// changed.
iter.root = iter.t.root
// Set the seek key to the current node. This will force the Next/Prev
// functions to reseek, and thus properly reconstruct the iterator, on
// their next call.
if iter.node == nil {
iter.seekKey = nil
return
}
iter.seekKey = iter.node.key
}
// Iterator returns a new iterator for the mutable treap. The newly returned
// iterator is not pointing to a valid item until a call to one of the methods
// to position it is made.
//
// The start key and limit key parameters cause the iterator to be limited to
// a range of keys. The start key is inclusive and the limit key is exclusive.
// Either or both can be nil if the functionality is not desired.
//
// WARNING: The ForceSeek method must be called on the returned iterator if
// the treap is mutated. Failure to do so will cause the iterator to return
// unexpected keys and/or values.
//
// For example:
// iter := t.Iterator(nil, nil)
// for iter.Next() {
// if someCondition {
// t.Delete(iter.Key())
// iter.ForceReseek()
// }
// }
func (t *Mutable) Iterator(startKey, limitKey []byte) *Iterator {
iter := &Iterator{
t: t,
root: t.root,
isNew: true,
startKey: startKey,
limitKey: limitKey,
}
return iter
}
// Iterator returns a new iterator for the immutable treap. The newly returned
// iterator is not pointing to a valid item until a call to one of the methods
// to position it is made.
//
// The start key and limit key parameters cause the iterator to be limited to
// a range of keys. The start key is inclusive and the limit key is exclusive.
// Either or both can be nil if the functionality is not desired.
func (t *Immutable) Iterator(startKey, limitKey []byte) *Iterator {
iter := &Iterator{
root: t.root,
isNew: true,
startKey: startKey,
limitKey: limitKey,
}
return iter
}