LBRYCommunity/lbrycrd
1
0
Fork 0
Code Issues 29 Pull requests 1 Projects Releases 32 Packages Wiki Activity
lbrycrd/src/txdb.cpp

427 lines
14 KiB
C++
Raw Normal View History

LevelDB block and coin databases Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files, implemented by either LevelDB or BDB. Based on code from earlier commits by Mike Hearn in his leveldb branch.
2012-09-03 21:14:03 +02:00
// Copyright (c) 2009-2010 Satoshi Nakamoto
Increment MIT Licence copyright header year on files modified in 2016 Edited via: $ contrib/devtools/copyright_header.py update .
2016-12-31 19:01:21 +01:00
// Copyright (c) 2009-2016 The Bitcoin Core developers
Remove references to X11 licence
2014-12-13 05:09:33 +01:00
// Distributed under the MIT software license, see the accompanying
LevelDB block and coin databases Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files, implemented by either LevelDB or BDB. Based on code from earlier commits by Mike Hearn in his leveldb branch.
2012-09-03 21:14:03 +02:00
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
Remove BDB block database support
2012-10-16 22:23:39 +02:00
#include "txdb.h"
Cleanup code using forward declarations. Use misc methods of avoiding unnecesary header includes. Replace int typedefs with int##_t from stdint.h. Replace PRI64[xdu] with PRI[xdu]64 from inttypes.h. Normalize QT_VERSION ifs where possible. Resolve some indirect dependencies as direct ones. Remove extern declarations from .cpp files.
2013-04-13 07:13:08 +02:00
Consensus: Refactor: Decouple pow.o from chainparams.o
2015-02-15 02:21:42 +01:00
#include "chainparams.h"
#include "hash.h"
Random db flush crash simulator
2017-04-05 10:37:09 +02:00
#include "random.h"
Refactor proof of work related functions out of main
2014-03-10 16:46:53 +01:00
#include "pow.h"
Cleanup code using forward declarations. Use misc methods of avoiding unnecesary header includes. Replace int typedefs with int##_t from stdint.h. Replace PRI64[xdu] with PRI[xdu]64 from inttypes.h. Normalize QT_VERSION ifs where possible. Resolve some indirect dependencies as direct ones. Remove extern declarations from .cpp files.
2013-04-13 07:13:08 +02:00
#include "uint256.h"
Random db flush crash simulator
2017-04-05 10:37:09 +02:00
#include "util.h"
Allow to cancel the txdb upgrade via splashscreen callback
2017-06-23 09:35:24 +02:00
#include "ui_interface.h"
#include "init.h"
Cleanup code using forward declarations. Use misc methods of avoiding unnecesary header includes. Replace int typedefs with int##_t from stdint.h. Replace PRI64[xdu] with PRI[xdu]64 from inttypes.h. Normalize QT_VERSION ifs where possible. Resolve some indirect dependencies as direct ones. Remove extern declarations from .cpp files.
2013-04-13 07:13:08 +02:00
#include <stdint.h>
LevelDB block and coin databases Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files, implemented by either LevelDB or BDB. Based on code from earlier commits by Mike Hearn in his leveldb branch.
2012-09-03 21:14:03 +02:00
header include cleanup - ensures alphabetical ordering for includes etc. in source file headers
2014-09-14 12:43:56 +02:00
#include <boost/thread.hpp>
Switch CCoinsView and chainstate db from per-txid to per-txout This patch makes several related changes: * Changes the CCoinsView virtual methods (GetCoins, HaveCoins, ...) to be COutPoint/Coin-based rather than txid/CCoins-based. * Changes the chainstate db to a new incompatible format that is also COutPoint/Coin based. * Implements reconstruction code for hash_serialized_2. * Adapts the coins_tests unit tests (thanks to Russell Yanofsky). A side effect of the new CCoinsView model is that we can no longer use the (unreliable) test for transaction outputs in the UTXO set to determine whether we already have a particular transaction.
2017-04-25 20:29:39 +02:00
static const char DB_COIN = 'C';
change hardcoded character constants to a set of descriptive named constants for database keys
2015-01-25 16:27:54 +01:00
static const char DB_COINS = 'c';
static const char DB_BLOCK_FILES = 'f';
static const char DB_TXINDEX = 't';
static const char DB_BLOCK_INDEX = 'b';
static const char DB_BEST_BLOCK = 'B';
Non-atomic flushing using the blockchain as replay journal
2017-04-19 18:34:30 +02:00
static const char DB_HEAD_BLOCKS = 'H';
change hardcoded character constants to a set of descriptive named constants for database keys
2015-01-25 16:27:54 +01:00
static const char DB_FLAG = 'F';
static const char DB_REINDEX_FLAG = 'R';
static const char DB_LAST_BLOCK = 'l';
Switch CCoinsView and chainstate db from per-txid to per-txout This patch makes several related changes: * Changes the CCoinsView virtual methods (GetCoins, HaveCoins, ...) to be COutPoint/Coin-based rather than txid/CCoins-based. * Changes the chainstate db to a new incompatible format that is also COutPoint/Coin based. * Implements reconstruction code for hash_serialized_2. * Adapts the coins_tests unit tests (thanks to Russell Yanofsky). A side effect of the new CCoinsView model is that we can no longer use the (unreliable) test for transaction outputs in the UTXO set to determine whether we already have a particular transaction.
2017-04-25 20:29:39 +02:00
namespace {
scripted-diff: various renames for per-utxo consistency Thanks to John Newberry for pointing these out. -BEGIN VERIFY SCRIPT- sed -i 's/\<GetCoins\>/GetCoin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<HaveCoins\>/HaveCoin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<HaveCoinsInCache\>/HaveCoinInCache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<IsPruned\>/IsSpent/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<FetchCoins\>/FetchCoin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<CoinsEntry\>/CoinEntry/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<vHashTxnToUncache\>/coins_to_uncache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<vHashTxToUncache\>/coins_to_uncache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<fHadTxInCache\>/had_coin_in_cache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<coinbaseids\>/coinbase_coins/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<disconnectedids\>/disconnected_coins/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<duplicateids\>/duplicate_coins/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<oldcoins\>/old_coin/g' src/test/coins_tests.cpp sed -i 's/\<origcoins\>/orig_coin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h -END VERIFY SCRIPT-
2017-05-31 02:58:54 +02:00
struct CoinEntry {
Switch CCoinsView and chainstate db from per-txid to per-txout This patch makes several related changes: * Changes the CCoinsView virtual methods (GetCoins, HaveCoins, ...) to be COutPoint/Coin-based rather than txid/CCoins-based. * Changes the chainstate db to a new incompatible format that is also COutPoint/Coin based. * Implements reconstruction code for hash_serialized_2. * Adapts the coins_tests unit tests (thanks to Russell Yanofsky). A side effect of the new CCoinsView model is that we can no longer use the (unreliable) test for transaction outputs in the UTXO set to determine whether we already have a particular transaction.
2017-04-25 20:29:39 +02:00
COutPoint* outpoint;
char key;
scripted-diff: various renames for per-utxo consistency Thanks to John Newberry for pointing these out. -BEGIN VERIFY SCRIPT- sed -i 's/\<GetCoins\>/GetCoin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<HaveCoins\>/HaveCoin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<HaveCoinsInCache\>/HaveCoinInCache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<IsPruned\>/IsSpent/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<FetchCoins\>/FetchCoin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<CoinsEntry\>/CoinEntry/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<vHashTxnToUncache\>/coins_to_uncache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<vHashTxToUncache\>/coins_to_uncache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<fHadTxInCache\>/had_coin_in_cache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<coinbaseids\>/coinbase_coins/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<disconnectedids\>/disconnected_coins/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<duplicateids\>/duplicate_coins/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<oldcoins\>/old_coin/g' src/test/coins_tests.cpp sed -i 's/\<origcoins\>/orig_coin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h -END VERIFY SCRIPT-
2017-05-31 02:58:54 +02:00
CoinEntry(const COutPoint* ptr) : outpoint(const_cast<COutPoint*>(ptr)), key(DB_COIN) {}
Switch CCoinsView and chainstate db from per-txid to per-txout This patch makes several related changes: * Changes the CCoinsView virtual methods (GetCoins, HaveCoins, ...) to be COutPoint/Coin-based rather than txid/CCoins-based. * Changes the chainstate db to a new incompatible format that is also COutPoint/Coin based. * Implements reconstruction code for hash_serialized_2. * Adapts the coins_tests unit tests (thanks to Russell Yanofsky). A side effect of the new CCoinsView model is that we can no longer use the (unreliable) test for transaction outputs in the UTXO set to determine whether we already have a particular transaction.
2017-04-25 20:29:39 +02:00
template<typename Stream>
void Serialize(Stream &s) const {
s << key;
s << outpoint->hash;
s << VARINT(outpoint->n);
}
template<typename Stream>
void Unserialize(Stream& s) {
s >> key;
s >> outpoint->hash;
s >> VARINT(outpoint->n);
}
};
}
change hardcoded character constants to a set of descriptive named constants for database keys
2015-01-25 16:27:54 +01:00
Add chainstate obfuscation to avoid spurious antivirus detection Adds an `obfuscate` parameter to `CLevelDBWrapper` and makes use of it for all new chainstate stores built via `CCoinsViewDB`. Also adds an `Xor` method to `CDataStream`. Thanks to @sipa @laanwj @pstratem @dexX7 @KyrosKrane @gmaxwell.
2015-09-08 00:22:23 +02:00
CCoinsViewDB::CCoinsViewDB(size_t nCacheSize, bool fMemory, bool fWipe) : db(GetDataDir() / "chainstate", nCacheSize, fMemory, fWipe, true)
{
LevelDB block and coin databases Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files, implemented by either LevelDB or BDB. Based on code from earlier commits by Mike Hearn in his leveldb branch.
2012-09-03 21:14:03 +02:00
}
scripted-diff: various renames for per-utxo consistency Thanks to John Newberry for pointing these out. -BEGIN VERIFY SCRIPT- sed -i 's/\<GetCoins\>/GetCoin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<HaveCoins\>/HaveCoin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<HaveCoinsInCache\>/HaveCoinInCache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<IsPruned\>/IsSpent/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<FetchCoins\>/FetchCoin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<CoinsEntry\>/CoinEntry/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<vHashTxnToUncache\>/coins_to_uncache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<vHashTxToUncache\>/coins_to_uncache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<fHadTxInCache\>/had_coin_in_cache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<coinbaseids\>/coinbase_coins/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<disconnectedids\>/disconnected_coins/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<duplicateids\>/duplicate_coins/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<oldcoins\>/old_coin/g' src/test/coins_tests.cpp sed -i 's/\<origcoins\>/orig_coin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h -END VERIFY SCRIPT-
2017-05-31 02:58:54 +02:00
bool CCoinsViewDB::GetCoin(const COutPoint &outpoint, Coin &coin) const {
return db.Read(CoinEntry(&outpoint), coin);
LevelDB block and coin databases Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files, implemented by either LevelDB or BDB. Based on code from earlier commits by Mike Hearn in his leveldb branch.
2012-09-03 21:14:03 +02:00
}
scripted-diff: various renames for per-utxo consistency Thanks to John Newberry for pointing these out. -BEGIN VERIFY SCRIPT- sed -i 's/\<GetCoins\>/GetCoin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<HaveCoins\>/HaveCoin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<HaveCoinsInCache\>/HaveCoinInCache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<IsPruned\>/IsSpent/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<FetchCoins\>/FetchCoin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<CoinsEntry\>/CoinEntry/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<vHashTxnToUncache\>/coins_to_uncache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<vHashTxToUncache\>/coins_to_uncache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<fHadTxInCache\>/had_coin_in_cache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<coinbaseids\>/coinbase_coins/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<disconnectedids\>/disconnected_coins/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<duplicateids\>/duplicate_coins/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<oldcoins\>/old_coin/g' src/test/coins_tests.cpp sed -i 's/\<origcoins\>/orig_coin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h -END VERIFY SCRIPT-
2017-05-31 02:58:54 +02:00
bool CCoinsViewDB::HaveCoin(const COutPoint &outpoint) const {
return db.Exists(CoinEntry(&outpoint));
LevelDB block and coin databases Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files, implemented by either LevelDB or BDB. Based on code from earlier commits by Mike Hearn in his leveldb branch.
2012-09-03 21:14:03 +02:00
}
Make appropriate getter-routines "const" in CCoinsView. Mark the "Get"/"Have" routines in CCoinsView and subclasses as "const".
2014-07-19 16:42:48 +02:00
uint256 CCoinsViewDB::GetBestBlock() const {
LevelDB block and coin databases Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files, implemented by either LevelDB or BDB. Based on code from earlier commits by Mike Hearn in his leveldb branch.
2012-09-03 21:14:03 +02:00
uint256 hashBestChain;
change hardcoded character constants to a set of descriptive named constants for database keys
2015-01-25 16:27:54 +01:00
if (!db.Read(DB_BEST_BLOCK, hashBestChain))
Replace direct use of 0 with SetNull and IsNull Replace x=0 with .SetNull(), x==0 with IsNull(), x!=0 with !IsNull(). Replace uses of uint256(0) with uint256().
2014-12-15 09:11:16 +01:00
return uint256();
Make CCoinsView use block hashes instead of indices
2013-11-05 02:27:39 +01:00
return hashBestChain;
LevelDB block and coin databases Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files, implemented by either LevelDB or BDB. Based on code from earlier commits by Mike Hearn in his leveldb branch.
2012-09-03 21:14:03 +02:00
}
Non-atomic flushing using the blockchain as replay journal
2017-04-19 18:34:30 +02:00
std::vector<uint256> CCoinsViewDB::GetHeadBlocks() const {
std::vector<uint256> vhashHeadBlocks;
if (!db.Read(DB_HEAD_BLOCKS, vhashHeadBlocks)) {
return std::vector<uint256>();
}
return vhashHeadBlocks;
}
Allow BatchWrite to destroy its input, reducing copying
2014-08-24 02:08:05 +02:00
bool CCoinsViewDB::BatchWrite(CCoinsMap &mapCoins, const uint256 &hashBlock) {
dbwrapper: Pass parent CDBWrapper into CDBBatch and CDBIterator Pass parent wrapper directly instead of obfuscation key. This makes it possible for other databases which re-use this code to use other properties from the database. Add a namespace dbwrapper_private for private functions to be used only in dbwrapper.h/cpp and dbwrapper_tests.
2016-04-20 11:46:01 +02:00
CDBBatch batch(db);
Do not keep fully spent but unwritten CCoins entries cached. Instead of storing CCoins entries directly in CCoinsMap, store a CCoinsCacheEntry which additionally keeps track of whether a particular entry is: * dirty: potentially different from its parent view. * fresh: the parent view is known to not have a non-pruned version. This allows us to skip non-dirty cache entries when pushing batches of changes up, and to remove CCoins entries about transactions that are fully spent before the parent cache learns about them.
2014-09-03 09:37:47 +02:00
size_t count = 0;
size_t changed = 0;
Non-atomic flushing using the blockchain as replay journal
2017-04-19 18:34:30 +02:00
size_t batch_size = (size_t)GetArg("-dbbatchsize", nDefaultDbBatchSize);
Random db flush crash simulator
2017-04-05 10:37:09 +02:00
int crash_simulate = GetArg("-dbcrashratio", 0);
Dont create pcoinsTip until after ReplayBlocks. This requires that we not access pcoinsTip in InitBlockIndex's FlushStateToDisk (so we just skip it until later in AppInitMain) and the LoadChainTip in LoadBlockIndex (which there is already one later in AppinitMain, after ReplayBlocks, so skipping it there is fine). Includes some simplifications by Suhas Daftuar and Pieter Wuille.
2017-04-17 17:41:10 +02:00
assert(!hashBlock.IsNull());
Non-atomic flushing using the blockchain as replay journal
2017-04-19 18:34:30 +02:00
uint256 old_tip = GetBestBlock();
if (old_tip.IsNull()) {
// We may be in the middle of replaying.
std::vector<uint256> old_heads = GetHeadBlocks();
if (old_heads.size() == 2) {
assert(old_heads[0] == hashBlock);
old_tip = old_heads[1];
}
}
// In the first batch, mark the database as being in the middle of a
// transition from old_tip to hashBlock.
// A vector is used for future extensibility, as we may want to support
// interrupting after partial writes from multiple independent reorgs.
batch.Erase(DB_BEST_BLOCK);
batch.Write(DB_HEAD_BLOCKS, std::vector<uint256>{hashBlock, old_tip});
Allow BatchWrite to destroy its input, reducing copying
2014-08-24 02:08:05 +02:00
for (CCoinsMap::iterator it = mapCoins.begin(); it != mapCoins.end();) {
Do not keep fully spent but unwritten CCoins entries cached. Instead of storing CCoins entries directly in CCoinsMap, store a CCoinsCacheEntry which additionally keeps track of whether a particular entry is: * dirty: potentially different from its parent view. * fresh: the parent view is known to not have a non-pruned version. This allows us to skip non-dirty cache entries when pushing batches of changes up, and to remove CCoins entries about transactions that are fully spent before the parent cache learns about them.
2014-09-03 09:37:47 +02:00
if (it->second.flags & CCoinsCacheEntry::DIRTY) {
scripted-diff: various renames for per-utxo consistency Thanks to John Newberry for pointing these out. -BEGIN VERIFY SCRIPT- sed -i 's/\<GetCoins\>/GetCoin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<HaveCoins\>/HaveCoin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<HaveCoinsInCache\>/HaveCoinInCache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<IsPruned\>/IsSpent/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<FetchCoins\>/FetchCoin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<CoinsEntry\>/CoinEntry/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<vHashTxnToUncache\>/coins_to_uncache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<vHashTxToUncache\>/coins_to_uncache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<fHadTxInCache\>/had_coin_in_cache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<coinbaseids\>/coinbase_coins/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<disconnectedids\>/disconnected_coins/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<duplicateids\>/duplicate_coins/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<oldcoins\>/old_coin/g' src/test/coins_tests.cpp sed -i 's/\<origcoins\>/orig_coin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h -END VERIFY SCRIPT-
2017-05-31 02:58:54 +02:00
CoinEntry entry(&it->first);
if (it->second.coin.IsSpent())
Switch CCoinsView and chainstate db from per-txid to per-txout This patch makes several related changes: * Changes the CCoinsView virtual methods (GetCoins, HaveCoins, ...) to be COutPoint/Coin-based rather than txid/CCoins-based. * Changes the chainstate db to a new incompatible format that is also COutPoint/Coin based. * Implements reconstruction code for hash_serialized_2. * Adapts the coins_tests unit tests (thanks to Russell Yanofsky). A side effect of the new CCoinsView model is that we can no longer use the (unreliable) test for transaction outputs in the UTXO set to determine whether we already have a particular transaction.
2017-04-25 20:29:39 +02:00
batch.Erase(entry);
Add chainstate obfuscation to avoid spurious antivirus detection Adds an `obfuscate` parameter to `CLevelDBWrapper` and makes use of it for all new chainstate stores built via `CCoinsViewDB`. Also adds an `Xor` method to `CDataStream`. Thanks to @sipa @laanwj @pstratem @dexX7 @KyrosKrane @gmaxwell.
2015-09-08 00:22:23 +02:00
else
Rename CCoinsCacheEntry::coins to coin
2017-05-04 09:15:36 +02:00
batch.Write(entry, it->second.coin);
Do not keep fully spent but unwritten CCoins entries cached. Instead of storing CCoins entries directly in CCoinsMap, store a CCoinsCacheEntry which additionally keeps track of whether a particular entry is: * dirty: potentially different from its parent view. * fresh: the parent view is known to not have a non-pruned version. This allows us to skip non-dirty cache entries when pushing batches of changes up, and to remove CCoins entries about transactions that are fully spent before the parent cache learns about them.
2014-09-03 09:37:47 +02:00
changed++;
}
count++;
Allow BatchWrite to destroy its input, reducing copying
2014-08-24 02:08:05 +02:00
CCoinsMap::iterator itOld = it++;
mapCoins.erase(itOld);
Non-atomic flushing using the blockchain as replay journal
2017-04-19 18:34:30 +02:00
if (batch.SizeEstimate() > batch_size) {
LogPrint(BCLog::COINDB, "Writing partial batch of %.2f MiB\n", batch.SizeEstimate() * (1.0 / 1048576.0));
db.WriteBatch(batch);
batch.Clear();
Random db flush crash simulator
2017-04-05 10:37:09 +02:00
if (crash_simulate) {
static FastRandomContext rng;
if (rng.randrange(crash_simulate) == 0) {
LogPrintf("Simulating a crash. Goodbye.\n");
[qa] Test non-atomic chainstate writes Adds new functional test, dbcrash.py, which uses -dbcrashratio to exercise the logic for recovering from a crash during chainstate flush. dbcrash.py is added to the extended tests, as it may take ~10 minutes to run Use _Exit() instead of exit() for crash simulation This eliminates stderr output such as: terminate called without an active exception or Assertion failed: (!pthread_mutex_destroy(&m)), function ~recursive_mutex, file /usr/local/include/boost/thread/pthread/recursive_mutex.hpp, line 104. Eliminating the stderr output on crash simulation allows testing with test_runner.py, which reports a test as failed if stderr is produced.
2017-06-15 23:08:48 +02:00
_Exit(0);
Random db flush crash simulator
2017-04-05 10:37:09 +02:00
}
}
Non-atomic flushing using the blockchain as replay journal
2017-04-19 18:34:30 +02:00
}
Allow BatchWrite to destroy its input, reducing copying
2014-08-24 02:08:05 +02:00
}
LevelDB block and coin databases Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files, implemented by either LevelDB or BDB. Based on code from earlier commits by Mike Hearn in his leveldb branch.
2012-09-03 21:14:03 +02:00
Non-atomic flushing using the blockchain as replay journal
2017-04-19 18:34:30 +02:00
// In the last batch, mark the database as consistent with hashBlock again.
batch.Erase(DB_HEAD_BLOCKS);
batch.Write(DB_BEST_BLOCK, hashBlock);
LogPrint(BCLog::COINDB, "Writing final batch of %.2f MiB\n", batch.SizeEstimate() * (1.0 / 1048576.0));
Switch CCoinsView and chainstate db from per-txid to per-txout This patch makes several related changes: * Changes the CCoinsView virtual methods (GetCoins, HaveCoins, ...) to be COutPoint/Coin-based rather than txid/CCoins-based. * Changes the chainstate db to a new incompatible format that is also COutPoint/Coin based. * Implements reconstruction code for hash_serialized_2. * Adapts the coins_tests unit tests (thanks to Russell Yanofsky). A side effect of the new CCoinsView model is that we can no longer use the (unreliable) test for transaction outputs in the UTXO set to determine whether we already have a particular transaction.
2017-04-25 20:29:39 +02:00
bool ret = db.WriteBatch(batch);
LogPrint(BCLog::COINDB, "Committed %u changed transaction outputs (out of %u) to coin database...\n", (unsigned int)changed, (unsigned int)count);
return ret;
LevelDB block and coin databases Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files, implemented by either LevelDB or BDB. Based on code from earlier commits by Mike Hearn in his leveldb branch.
2012-09-03 21:14:03 +02:00
}
Report on-disk size in gettxoutsetinfo
2017-05-13 00:19:19 +02:00
size_t CCoinsViewDB::EstimateSize() const
{
Switch CCoinsView and chainstate db from per-txid to per-txout This patch makes several related changes: * Changes the CCoinsView virtual methods (GetCoins, HaveCoins, ...) to be COutPoint/Coin-based rather than txid/CCoins-based. * Changes the chainstate db to a new incompatible format that is also COutPoint/Coin based. * Implements reconstruction code for hash_serialized_2. * Adapts the coins_tests unit tests (thanks to Russell Yanofsky). A side effect of the new CCoinsView model is that we can no longer use the (unreliable) test for transaction outputs in the UTXO set to determine whether we already have a particular transaction.
2017-04-25 20:29:39 +02:00
return db.EstimateSize(DB_COIN, (char)(DB_COIN+1));
Report on-disk size in gettxoutsetinfo
2017-05-13 00:19:19 +02:00
}
leveldbwrapper symbol rename: Remove "Level" from class, etc. names
2015-10-23 03:02:20 +02:00
CBlockTreeDB::CBlockTreeDB(size_t nCacheSize, bool fMemory, bool fWipe) : CDBWrapper(GetDataDir() / "blocks" / "index", nCacheSize, fMemory, fWipe) {
Add LevelDB MemEnv support Support LevelDB memory-backed environments, and use them in unit tests.
2012-09-04 18:12:00 +02:00
}
LevelDB block and coin databases Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files, implemented by either LevelDB or BDB. Based on code from earlier commits by Mike Hearn in his leveldb branch.
2012-09-03 21:14:03 +02:00
bool CBlockTreeDB::ReadBlockFileInfo(int nFile, CBlockFileInfo &info) {
Refactor: Remove using namespace <xxx> from src/*.cpp.
2017-01-27 09:43:41 +01:00
return Read(std::make_pair(DB_BLOCK_FILES, nFile), info);
LevelDB block and coin databases Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files, implemented by either LevelDB or BDB. Based on code from earlier commits by Mike Hearn in his leveldb branch.
2012-09-03 21:14:03 +02:00
}
Add -reindex, to perform in-place reindexing of block chain files Flushes the blktree/ and coins/ databases, and reindexes the block chain files, as if their contents was loaded via -loadblock. Based on earlier work by Jeff Garzik.
2012-10-21 21:23:13 +02:00
bool CBlockTreeDB::WriteReindexing(bool fReindexing) {
if (fReindexing)
change hardcoded character constants to a set of descriptive named constants for database keys
2015-01-25 16:27:54 +01:00
return Write(DB_REINDEX_FLAG, '1');
Add -reindex, to perform in-place reindexing of block chain files Flushes the blktree/ and coins/ databases, and reindexes the block chain files, as if their contents was loaded via -loadblock. Based on earlier work by Jeff Garzik.
2012-10-21 21:23:13 +02:00
else
change hardcoded character constants to a set of descriptive named constants for database keys
2015-01-25 16:27:54 +01:00
return Erase(DB_REINDEX_FLAG);
Add -reindex, to perform in-place reindexing of block chain files Flushes the blktree/ and coins/ databases, and reindexes the block chain files, as if their contents was loaded via -loadblock. Based on earlier work by Jeff Garzik.
2012-10-21 21:23:13 +02:00
}
bool CBlockTreeDB::ReadReindexing(bool &fReindexing) {
change hardcoded character constants to a set of descriptive named constants for database keys
2015-01-25 16:27:54 +01:00
fReindexing = Exists(DB_REINDEX_FLAG);
Add -reindex, to perform in-place reindexing of block chain files Flushes the blktree/ and coins/ databases, and reindexes the block chain files, as if their contents was loaded via -loadblock. Based on earlier work by Jeff Garzik.
2012-10-21 21:23:13 +02:00
return true;
}
LevelDB block and coin databases Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files, implemented by either LevelDB or BDB. Based on code from earlier commits by Mike Hearn in his leveldb branch.
2012-09-03 21:14:03 +02:00
bool CBlockTreeDB::ReadLastBlockFile(int &nFile) {
change hardcoded character constants to a set of descriptive named constants for database keys
2015-01-25 16:27:54 +01:00
return Read(DB_LAST_BLOCK, nFile);
LevelDB block and coin databases Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files, implemented by either LevelDB or BDB. Based on code from earlier commits by Mike Hearn in his leveldb branch.
2012-09-03 21:14:03 +02:00
}
txdb: Add Cursor() method to CCoinsView to iterate over UTXO set Add a method Cursor() to CCoinsView that returns a cursor which can be used to iterate over the whole UTXO set. - rpc: Change gettxoutsetinfo to use new Cursor method - txdb: Remove GetStats method - Now that GetStats is implemented in terms of Cursor, remove it.
2016-03-28 18:18:30 +02:00
CCoinsViewCursor *CCoinsViewDB::Cursor() const
{
Avoid dereference-of-casted-pointer
2017-07-07 09:54:42 +02:00
CCoinsViewDBCursor *i = new CCoinsViewDBCursor(const_cast<CDBWrapper&>(db).NewIterator(), GetBestBlock());
Make appropriate getter-routines "const" in CCoinsView. Mark the "Get"/"Have" routines in CCoinsView and subclasses as "const".
2014-07-19 16:42:48 +02:00
/* It seems that there are no "const iterators" for LevelDB. Since we
only need read operations on it, use a const-cast to get around
that restriction. */
Switch CCoinsView and chainstate db from per-txid to per-txout This patch makes several related changes: * Changes the CCoinsView virtual methods (GetCoins, HaveCoins, ...) to be COutPoint/Coin-based rather than txid/CCoins-based. * Changes the chainstate db to a new incompatible format that is also COutPoint/Coin based. * Implements reconstruction code for hash_serialized_2. * Adapts the coins_tests unit tests (thanks to Russell Yanofsky). A side effect of the new CCoinsView model is that we can no longer use the (unreliable) test for transaction outputs in the UTXO set to determine whether we already have a particular transaction.
2017-04-25 20:29:39 +02:00
i->pcursor->Seek(DB_COIN);
txdb: Add Cursor() method to CCoinsView to iterate over UTXO set Add a method Cursor() to CCoinsView that returns a cursor which can be used to iterate over the whole UTXO set. - rpc: Change gettxoutsetinfo to use new Cursor method - txdb: Remove GetStats method - Now that GetStats is implemented in terms of Cursor, remove it.
2016-03-28 18:18:30 +02:00
// Cache key of first record
Fix: make CCoinsViewDbCursor::Seek work for missing keys Thanks to Suhas Daftuar for figuring this out.
2017-05-23 19:04:14 +02:00
if (i->pcursor->Valid()) {
scripted-diff: various renames for per-utxo consistency Thanks to John Newberry for pointing these out. -BEGIN VERIFY SCRIPT- sed -i 's/\<GetCoins\>/GetCoin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<HaveCoins\>/HaveCoin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<HaveCoinsInCache\>/HaveCoinInCache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<IsPruned\>/IsSpent/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<FetchCoins\>/FetchCoin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<CoinsEntry\>/CoinEntry/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<vHashTxnToUncache\>/coins_to_uncache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<vHashTxToUncache\>/coins_to_uncache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<fHadTxInCache\>/had_coin_in_cache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<coinbaseids\>/coinbase_coins/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<disconnectedids\>/disconnected_coins/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<duplicateids\>/duplicate_coins/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<oldcoins\>/old_coin/g' src/test/coins_tests.cpp sed -i 's/\<origcoins\>/orig_coin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h -END VERIFY SCRIPT-
2017-05-31 02:58:54 +02:00
CoinEntry entry(&i->keyTmp.second);
Switch CCoinsView and chainstate db from per-txid to per-txout This patch makes several related changes: * Changes the CCoinsView virtual methods (GetCoins, HaveCoins, ...) to be COutPoint/Coin-based rather than txid/CCoins-based. * Changes the chainstate db to a new incompatible format that is also COutPoint/Coin based. * Implements reconstruction code for hash_serialized_2. * Adapts the coins_tests unit tests (thanks to Russell Yanofsky). A side effect of the new CCoinsView model is that we can no longer use the (unreliable) test for transaction outputs in the UTXO set to determine whether we already have a particular transaction.
2017-04-25 20:29:39 +02:00
i->pcursor->GetKey(entry);
i->keyTmp.first = entry.key;
Fix: make CCoinsViewDbCursor::Seek work for missing keys Thanks to Suhas Daftuar for figuring this out.
2017-05-23 19:04:14 +02:00
} else {
i->keyTmp.first = 0; // Make sure Valid() and GetKey() return false
}
txdb: Add Cursor() method to CCoinsView to iterate over UTXO set Add a method Cursor() to CCoinsView that returns a cursor which can be used to iterate over the whole UTXO set. - rpc: Change gettxoutsetinfo to use new Cursor method - txdb: Remove GetStats method - Now that GetStats is implemented in terms of Cursor, remove it.
2016-03-28 18:18:30 +02:00
return i;
}
Switch CCoinsView and chainstate db from per-txid to per-txout This patch makes several related changes: * Changes the CCoinsView virtual methods (GetCoins, HaveCoins, ...) to be COutPoint/Coin-based rather than txid/CCoins-based. * Changes the chainstate db to a new incompatible format that is also COutPoint/Coin based. * Implements reconstruction code for hash_serialized_2. * Adapts the coins_tests unit tests (thanks to Russell Yanofsky). A side effect of the new CCoinsView model is that we can no longer use the (unreliable) test for transaction outputs in the UTXO set to determine whether we already have a particular transaction.
2017-04-25 20:29:39 +02:00
bool CCoinsViewDBCursor::GetKey(COutPoint &key) const
txdb: Add Cursor() method to CCoinsView to iterate over UTXO set Add a method Cursor() to CCoinsView that returns a cursor which can be used to iterate over the whole UTXO set. - rpc: Change gettxoutsetinfo to use new Cursor method - txdb: Remove GetStats method - Now that GetStats is implemented in terms of Cursor, remove it.
2016-03-28 18:18:30 +02:00
{
// Return cached key
Switch CCoinsView and chainstate db from per-txid to per-txout This patch makes several related changes: * Changes the CCoinsView virtual methods (GetCoins, HaveCoins, ...) to be COutPoint/Coin-based rather than txid/CCoins-based. * Changes the chainstate db to a new incompatible format that is also COutPoint/Coin based. * Implements reconstruction code for hash_serialized_2. * Adapts the coins_tests unit tests (thanks to Russell Yanofsky). A side effect of the new CCoinsView model is that we can no longer use the (unreliable) test for transaction outputs in the UTXO set to determine whether we already have a particular transaction.
2017-04-25 20:29:39 +02:00
if (keyTmp.first == DB_COIN) {
txdb: Add Cursor() method to CCoinsView to iterate over UTXO set Add a method Cursor() to CCoinsView that returns a cursor which can be used to iterate over the whole UTXO set. - rpc: Change gettxoutsetinfo to use new Cursor method - txdb: Remove GetStats method - Now that GetStats is implemented in terms of Cursor, remove it.
2016-03-28 18:18:30 +02:00
key = keyTmp.second;
return true;
Add gettxout and gettxoutsetinfo RPCs
2012-09-25 23:04:54 +02:00
}
txdb: Add Cursor() method to CCoinsView to iterate over UTXO set Add a method Cursor() to CCoinsView that returns a cursor which can be used to iterate over the whole UTXO set. - rpc: Change gettxoutsetinfo to use new Cursor method - txdb: Remove GetStats method - Now that GetStats is implemented in terms of Cursor, remove it.
2016-03-28 18:18:30 +02:00
return false;
}
Switch CCoinsView and chainstate db from per-txid to per-txout This patch makes several related changes: * Changes the CCoinsView virtual methods (GetCoins, HaveCoins, ...) to be COutPoint/Coin-based rather than txid/CCoins-based. * Changes the chainstate db to a new incompatible format that is also COutPoint/Coin based. * Implements reconstruction code for hash_serialized_2. * Adapts the coins_tests unit tests (thanks to Russell Yanofsky). A side effect of the new CCoinsView model is that we can no longer use the (unreliable) test for transaction outputs in the UTXO set to determine whether we already have a particular transaction.
2017-04-25 20:29:39 +02:00
bool CCoinsViewDBCursor::GetValue(Coin &coin) const
txdb: Add Cursor() method to CCoinsView to iterate over UTXO set Add a method Cursor() to CCoinsView that returns a cursor which can be used to iterate over the whole UTXO set. - rpc: Change gettxoutsetinfo to use new Cursor method - txdb: Remove GetStats method - Now that GetStats is implemented in terms of Cursor, remove it.
2016-03-28 18:18:30 +02:00
{
Switch CCoinsView and chainstate db from per-txid to per-txout This patch makes several related changes: * Changes the CCoinsView virtual methods (GetCoins, HaveCoins, ...) to be COutPoint/Coin-based rather than txid/CCoins-based. * Changes the chainstate db to a new incompatible format that is also COutPoint/Coin based. * Implements reconstruction code for hash_serialized_2. * Adapts the coins_tests unit tests (thanks to Russell Yanofsky). A side effect of the new CCoinsView model is that we can no longer use the (unreliable) test for transaction outputs in the UTXO set to determine whether we already have a particular transaction.
2017-04-25 20:29:39 +02:00
return pcursor->GetValue(coin);
txdb: Add Cursor() method to CCoinsView to iterate over UTXO set Add a method Cursor() to CCoinsView that returns a cursor which can be used to iterate over the whole UTXO set. - rpc: Change gettxoutsetinfo to use new Cursor method - txdb: Remove GetStats method - Now that GetStats is implemented in terms of Cursor, remove it.
2016-03-28 18:18:30 +02:00
}
unsigned int CCoinsViewDBCursor::GetValueSize() const
{
return pcursor->GetValueSize();
}
bool CCoinsViewDBCursor::Valid() const
{
Switch CCoinsView and chainstate db from per-txid to per-txout This patch makes several related changes: * Changes the CCoinsView virtual methods (GetCoins, HaveCoins, ...) to be COutPoint/Coin-based rather than txid/CCoins-based. * Changes the chainstate db to a new incompatible format that is also COutPoint/Coin based. * Implements reconstruction code for hash_serialized_2. * Adapts the coins_tests unit tests (thanks to Russell Yanofsky). A side effect of the new CCoinsView model is that we can no longer use the (unreliable) test for transaction outputs in the UTXO set to determine whether we already have a particular transaction.
2017-04-25 20:29:39 +02:00
return keyTmp.first == DB_COIN;
txdb: Add Cursor() method to CCoinsView to iterate over UTXO set Add a method Cursor() to CCoinsView that returns a cursor which can be used to iterate over the whole UTXO set. - rpc: Change gettxoutsetinfo to use new Cursor method - txdb: Remove GetStats method - Now that GetStats is implemented in terms of Cursor, remove it.
2016-03-28 18:18:30 +02:00
}
void CCoinsViewDBCursor::Next()
{
pcursor->Next();
scripted-diff: various renames for per-utxo consistency Thanks to John Newberry for pointing these out. -BEGIN VERIFY SCRIPT- sed -i 's/\<GetCoins\>/GetCoin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<HaveCoins\>/HaveCoin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<HaveCoinsInCache\>/HaveCoinInCache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<IsPruned\>/IsSpent/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<FetchCoins\>/FetchCoin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<CoinsEntry\>/CoinEntry/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<vHashTxnToUncache\>/coins_to_uncache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<vHashTxToUncache\>/coins_to_uncache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<fHadTxInCache\>/had_coin_in_cache/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<coinbaseids\>/coinbase_coins/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<disconnectedids\>/disconnected_coins/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<duplicateids\>/duplicate_coins/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h sed -i 's/\<oldcoins\>/old_coin/g' src/test/coins_tests.cpp sed -i 's/\<origcoins\>/orig_coin/g' src/*.cpp src/*.h src/*/*.cpp src/*/*.h -END VERIFY SCRIPT-
2017-05-31 02:58:54 +02:00
CoinEntry entry(&keyTmp.second);
Switch CCoinsView and chainstate db from per-txid to per-txout This patch makes several related changes: * Changes the CCoinsView virtual methods (GetCoins, HaveCoins, ...) to be COutPoint/Coin-based rather than txid/CCoins-based. * Changes the chainstate db to a new incompatible format that is also COutPoint/Coin based. * Implements reconstruction code for hash_serialized_2. * Adapts the coins_tests unit tests (thanks to Russell Yanofsky). A side effect of the new CCoinsView model is that we can no longer use the (unreliable) test for transaction outputs in the UTXO set to determine whether we already have a particular transaction.
2017-04-25 20:29:39 +02:00
if (!pcursor->Valid() || !pcursor->GetKey(entry)) {
txdb: Add Cursor() method to CCoinsView to iterate over UTXO set Add a method Cursor() to CCoinsView that returns a cursor which can be used to iterate over the whole UTXO set. - rpc: Change gettxoutsetinfo to use new Cursor method - txdb: Remove GetStats method - Now that GetStats is implemented in terms of Cursor, remove it.
2016-03-28 18:18:30 +02:00
keyTmp.first = 0; // Invalidate cached key after last record so that Valid() and GetKey() return false
Switch CCoinsView and chainstate db from per-txid to per-txout This patch makes several related changes: * Changes the CCoinsView virtual methods (GetCoins, HaveCoins, ...) to be COutPoint/Coin-based rather than txid/CCoins-based. * Changes the chainstate db to a new incompatible format that is also COutPoint/Coin based. * Implements reconstruction code for hash_serialized_2. * Adapts the coins_tests unit tests (thanks to Russell Yanofsky). A side effect of the new CCoinsView model is that we can no longer use the (unreliable) test for transaction outputs in the UTXO set to determine whether we already have a particular transaction.
2017-04-25 20:29:39 +02:00
} else {
keyTmp.first = entry.key;
}
Add gettxout and gettxoutsetinfo RPCs
2012-09-25 23:04:54 +02:00
}
Do all block index writes in a batch
2014-11-25 16:26:20 +01:00
bool CBlockTreeDB::WriteBatchSync(const std::vector<std::pair<int, const CBlockFileInfo*> >& fileInfo, int nLastFile, const std::vector<const CBlockIndex*>& blockinfo) {
dbwrapper: Pass parent CDBWrapper into CDBBatch and CDBIterator Pass parent wrapper directly instead of obfuscation key. This makes it possible for other databases which re-use this code to use other properties from the database. Add a namespace dbwrapper_private for private functions to be used only in dbwrapper.h/cpp and dbwrapper_tests.
2016-04-20 11:46:01 +02:00
CDBBatch batch(*this);
Do all block index writes in a batch
2014-11-25 16:26:20 +01:00
for (std::vector<std::pair<int, const CBlockFileInfo*> >::const_iterator it=fileInfo.begin(); it != fileInfo.end(); it++) {
Refactor: Remove using namespace <xxx> from src/*.cpp.
2017-01-27 09:43:41 +01:00
batch.Write(std::make_pair(DB_BLOCK_FILES, it->first), *it->second);
Do all block index writes in a batch
2014-11-25 16:26:20 +01:00
}
change hardcoded character constants to a set of descriptive named constants for database keys
2015-01-25 16:27:54 +01:00
batch.Write(DB_LAST_BLOCK, nLastFile);
Do all block index writes in a batch
2014-11-25 16:26:20 +01:00
for (std::vector<const CBlockIndex*>::const_iterator it=blockinfo.begin(); it != blockinfo.end(); it++) {
Refactor: Remove using namespace <xxx> from src/*.cpp.
2017-01-27 09:43:41 +01:00
batch.Write(std::make_pair(DB_BLOCK_INDEX, (*it)->GetBlockHash()), CDiskBlockIndex(*it));
Do all block index writes in a batch
2014-11-25 16:26:20 +01:00
}
return WriteBatch(batch, true);
}
Add optional transaction index to databases By specifying -txindex when initializing the database, a txid-to-diskpos index is maintained in the blktree database. This database is used to help answering getrawtransaction() RPC queries, when enabled. Changing the -txindex value requires a -reindex; the client will abort at startup if the database and the specified -txindex mismatch.
2013-01-11 01:47:57 +01:00
bool CBlockTreeDB::ReadTxIndex(const uint256 &txid, CDiskTxPos &pos) {
Refactor: Remove using namespace <xxx> from src/*.cpp.
2017-01-27 09:43:41 +01:00
return Read(std::make_pair(DB_TXINDEX, txid), pos);
Add optional transaction index to databases By specifying -txindex when initializing the database, a txid-to-diskpos index is maintained in the blktree database. This database is used to help answering getrawtransaction() RPC queries, when enabled. Changing the -txindex value requires a -reindex; the client will abort at startup if the database and the specified -txindex mismatch.
2013-01-11 01:47:57 +01:00
}
bool CBlockTreeDB::WriteTxIndex(const std::vector<std::pair<uint256, CDiskTxPos> >&vect) {
dbwrapper: Pass parent CDBWrapper into CDBBatch and CDBIterator Pass parent wrapper directly instead of obfuscation key. This makes it possible for other databases which re-use this code to use other properties from the database. Add a namespace dbwrapper_private for private functions to be used only in dbwrapper.h/cpp and dbwrapper_tests.
2016-04-20 11:46:01 +02:00
CDBBatch batch(*this);
Add optional transaction index to databases By specifying -txindex when initializing the database, a txid-to-diskpos index is maintained in the blktree database. This database is used to help answering getrawtransaction() RPC queries, when enabled. Changing the -txindex value requires a -reindex; the client will abort at startup if the database and the specified -txindex mismatch.
2013-01-11 01:47:57 +01:00
for (std::vector<std::pair<uint256,CDiskTxPos> >::const_iterator it=vect.begin(); it!=vect.end(); it++)
Refactor: Remove using namespace <xxx> from src/*.cpp.
2017-01-27 09:43:41 +01:00
batch.Write(std::make_pair(DB_TXINDEX, it->first), it->second);
Add optional transaction index to databases By specifying -txindex when initializing the database, a txid-to-diskpos index is maintained in the blktree database. This database is used to help answering getrawtransaction() RPC queries, when enabled. Changing the -txindex value requires a -reindex; the client will abort at startup if the database and the specified -txindex mismatch.
2013-01-11 01:47:57 +01:00
return WriteBatch(batch);
}
bool CBlockTreeDB::WriteFlag(const std::string &name, bool fValue) {
change hardcoded character constants to a set of descriptive named constants for database keys
2015-01-25 16:27:54 +01:00
return Write(std::make_pair(DB_FLAG, name), fValue ? '1' : '0');
Add optional transaction index to databases By specifying -txindex when initializing the database, a txid-to-diskpos index is maintained in the blktree database. This database is used to help answering getrawtransaction() RPC queries, when enabled. Changing the -txindex value requires a -reindex; the client will abort at startup if the database and the specified -txindex mismatch.
2013-01-11 01:47:57 +01:00
}
bool CBlockTreeDB::ReadFlag(const std::string &name, bool &fValue) {
char ch;
change hardcoded character constants to a set of descriptive named constants for database keys
2015-01-25 16:27:54 +01:00
if (!Read(std::make_pair(DB_FLAG, name), ch))
Add optional transaction index to databases By specifying -txindex when initializing the database, a txid-to-diskpos index is maintained in the blktree database. This database is used to help answering getrawtransaction() RPC queries, when enabled. Changing the -txindex value requires a -reindex; the client will abort at startup if the database and the specified -txindex mismatch.
2013-01-11 01:47:57 +01:00
return false;
fValue = ch == '1';
return true;
}
Globals: Pass Consensus::Params through CBlockTreeDB::LoadBlockIndexGuts()
2016-11-07 23:31:55 +01:00
bool CBlockTreeDB::LoadBlockIndexGuts(const Consensus::Params& consensusParams, std::function<CBlockIndex*(const uint256&)> insertBlockIndex)
LevelDB block and coin databases Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files, implemented by either LevelDB or BDB. Based on code from earlier commits by Mike Hearn in his leveldb branch.
2012-09-03 21:14:03 +02:00
{
C++11: s/boost::scoped_ptr/std::unique_ptr/
2016-08-30 22:41:56 +02:00
std::unique_ptr<CDBIterator> pcursor(NewIterator());
LevelDB block and coin databases Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files, implemented by either LevelDB or BDB. Based on code from earlier commits by Mike Hearn in his leveldb branch.
2012-09-03 21:14:03 +02:00
Refactor: Remove using namespace <xxx> from src/*.cpp.
2017-01-27 09:43:41 +01:00
pcursor->Seek(std::make_pair(DB_BLOCK_INDEX, uint256()));
LevelDB block and coin databases Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files, implemented by either LevelDB or BDB. Based on code from earlier commits by Mike Hearn in his leveldb branch.
2012-09-03 21:14:03 +02:00
// Load mapBlockIndex
while (pcursor->Valid()) {
Clean up shutdown process
2013-03-09 18:02:57 +01:00
boost::this_thread::interruption_point();
Encapsulate CLevelDB iterators cleanly Conflicts: src/leveldb.cpp src/leveldb.h src/txdb.cpp
2015-10-08 02:12:24 +02:00
std::pair<char, uint256> key;
trivial: use constants for db keys Replace literal occurances of the key "prefixes" 'c' and 'b' in txdb.cpp by the respective constants.
2015-10-13 20:25:57 +02:00
if (pcursor->GetKey(key) && key.first == DB_BLOCK_INDEX) {
Encapsulate CLevelDB iterators cleanly Conflicts: src/leveldb.cpp src/leveldb.h src/txdb.cpp
2015-10-08 02:12:24 +02:00
CDiskBlockIndex diskindex;
if (pcursor->GetValue(diskindex)) {
LevelDB block and coin databases Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files, implemented by either LevelDB or BDB. Based on code from earlier commits by Mike Hearn in his leveldb branch.
2012-09-03 21:14:03 +02:00
// Construct block index object
Break circular dependency main ↔ txdb Break the circular dependency between main and txdb by: - Moving `CBlockFileInfo` from `main.h` to `chain.h`. I think this makes sense, as the other block-file stuff is there too. - Moving `CDiskTxPos` from `main.h` to `txdb.h`. This type seems specific to txdb. - Pass a functor `insertBlockIndex` to `LoadBlockIndexGuts`. This leaves it up to the caller how to insert block indices.
2016-04-05 15:14:48 +02:00
CBlockIndex* pindexNew = insertBlockIndex(diskindex.GetBlockHash());
pindexNew->pprev = insertBlockIndex(diskindex.hashPrev);
LevelDB block and coin databases Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files, implemented by either LevelDB or BDB. Based on code from earlier commits by Mike Hearn in his leveldb branch.
2012-09-03 21:14:03 +02:00
pindexNew->nHeight = diskindex.nHeight;
pindexNew->nFile = diskindex.nFile;
pindexNew->nDataPos = diskindex.nDataPos;
pindexNew->nUndoPos = diskindex.nUndoPos;
pindexNew->nVersion = diskindex.nVersion;
pindexNew->hashMerkleRoot = diskindex.hashMerkleRoot;
pindexNew->nTime = diskindex.nTime;
pindexNew->nBits = diskindex.nBits;
pindexNew->nNonce = diskindex.nNonce;
pindexNew->nStatus = diskindex.nStatus;
pindexNew->nTx = diskindex.nTx;
Globals: Pass Consensus::Params through CBlockTreeDB::LoadBlockIndexGuts()
2016-11-07 23:31:55 +01:00
if (!CheckProofOfWork(pindexNew->GetBlockHash(), pindexNew->nBits, consensusParams))
return error("%s: CheckProofOfWork failed: %s", __func__, pindexNew->ToString());
LevelDB block and coin databases Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files, implemented by either LevelDB or BDB. Based on code from earlier commits by Mike Hearn in his leveldb branch.
2012-09-03 21:14:03 +02:00
pcursor->Next();
} else {
Globals: Pass Consensus::Params through CBlockTreeDB::LoadBlockIndexGuts()
2016-11-07 23:31:55 +01:00
return error("%s: failed to read value", __func__);
LevelDB block and coin databases Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files, implemented by either LevelDB or BDB. Based on code from earlier commits by Mike Hearn in his leveldb branch.
2012-09-03 21:14:03 +02:00
}
Encapsulate CLevelDB iterators cleanly Conflicts: src/leveldb.cpp src/leveldb.h src/txdb.cpp
2015-10-08 02:12:24 +02:00
} else {
break;
LevelDB block and coin databases Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files, implemented by either LevelDB or BDB. Based on code from earlier commits by Mike Hearn in his leveldb branch.
2012-09-03 21:14:03 +02:00
}
}
return true;
}
Upgrade from per-tx database to per-txout
2017-04-25 20:29:46 +02:00
[MOVEONLY] Move old CCoins class to txdb.cpp It's only used for upgrading from the old database anymore.
2017-04-25 20:29:48 +02:00
namespace {
//! Legacy class to deserialize pre-pertxout database entries without reindex.
class CCoins
{
public:
//! whether transaction is a coinbase
bool fCoinBase;
//! unspent transaction outputs; spent outputs are .IsNull(); spent outputs at the end of the array are dropped
std::vector<CTxOut> vout;
//! at which height this transaction was included in the active block chain
int nHeight;
//! empty constructor
CCoins() : fCoinBase(false), vout(0), nHeight(0) { }
template<typename Stream>
void Unserialize(Stream &s) {
unsigned int nCode = 0;
// version
int nVersionDummy;
::Unserialize(s, VARINT(nVersionDummy));
// header code
::Unserialize(s, VARINT(nCode));
fCoinBase = nCode & 1;
std::vector<bool> vAvail(2, false);
vAvail[0] = (nCode & 2) != 0;
vAvail[1] = (nCode & 4) != 0;
unsigned int nMaskCode = (nCode / 8) + ((nCode & 6) != 0 ? 0 : 1);
// spentness bitmask
while (nMaskCode > 0) {
unsigned char chAvail = 0;
::Unserialize(s, chAvail);
for (unsigned int p = 0; p < 8; p++) {
bool f = (chAvail & (1 << p)) != 0;
vAvail.push_back(f);
}
if (chAvail != 0)
nMaskCode--;
}
// txouts themself
vout.assign(vAvail.size(), CTxOut());
for (unsigned int i = 0; i < vAvail.size(); i++) {
if (vAvail[i])
::Unserialize(s, REF(CTxOutCompressor(vout[i])));
}
// coinbase height
::Unserialize(s, VARINT(nHeight));
}
};
}
Upgrade from per-tx database to per-txout
2017-04-25 20:29:46 +02:00
/** Upgrade the database from older formats.
*
* Currently implemented: from the per-tx utxo model (0.8..0.14.x) to per-txout.
*/
bool CCoinsViewDB::Upgrade() {
std::unique_ptr<CDBIterator> pcursor(db.NewIterator());
pcursor->Seek(std::make_pair(DB_COINS, uint256()));
if (!pcursor->Valid()) {
return true;
}
Allow to cancel the txdb upgrade via splashscreen callback
2017-06-23 09:35:24 +02:00
int64_t count = 0;
Show txdb upgrade progress in debug log
2017-06-23 09:51:59 +02:00
LogPrintf("Upgrading utxo-set database...\n");
LogPrintf("[0%%]...");
Upgrade from per-tx database to per-txout
2017-04-25 20:29:46 +02:00
size_t batch_size = 1 << 24;
CDBBatch batch(db);
Allow to cancel the txdb upgrade via splashscreen callback
2017-06-23 09:35:24 +02:00
uiInterface.SetProgressBreakAction(StartShutdown);
Report txdb upgrade not more often then every 10%
2017-06-28 21:05:39 +02:00
int reportDone = 0;
Force on-the-fly compaction during pertxout upgrade
2017-06-03 02:52:37 +02:00
std::pair<unsigned char, uint256> key;
std::pair<unsigned char, uint256> prev_key = {DB_COINS, uint256()};
Upgrade from per-tx database to per-txout
2017-04-25 20:29:46 +02:00
while (pcursor->Valid()) {
boost::this_thread::interruption_point();
Allow to shut down during txdb upgrade
2017-06-23 09:33:31 +02:00
if (ShutdownRequested()) {
break;
}
Upgrade from per-tx database to per-txout
2017-04-25 20:29:46 +02:00
if (pcursor->GetKey(key) && key.first == DB_COINS) {
Allow to cancel the txdb upgrade via splashscreen callback
2017-06-23 09:35:24 +02:00
if (count++ % 256 == 0) {
uint32_t high = 0x100 * *key.second.begin() + *(key.second.begin() + 1);
Report txdb upgrade not more often then every 10%
2017-06-28 21:05:39 +02:00
int percentageDone = (int)(high * 100.0 / 65536.0 + 0.5);
uiInterface.ShowProgress(_("Upgrading UTXO database") + "\n"+ _("(press q to shutdown and continue later)") + "\n", percentageDone);
if (reportDone < percentageDone/10) {
// report max. every 10% step
LogPrintf("[%d%%]...", percentageDone);
reportDone = percentageDone/10;
}
Allow to cancel the txdb upgrade via splashscreen callback
2017-06-23 09:35:24 +02:00
}
Upgrade from per-tx database to per-txout
2017-04-25 20:29:46 +02:00
CCoins old_coins;
if (!pcursor->GetValue(old_coins)) {
return error("%s: cannot parse CCoins record", __func__);
}
COutPoint outpoint(key.second, 0);
for (size_t i = 0; i < old_coins.vout.size(); ++i) {
if (!old_coins.vout[i].IsNull() && !old_coins.vout[i].scriptPubKey.IsUnspendable()) {
Coin newcoin(std::move(old_coins.vout[i]), old_coins.nHeight, old_coins.fCoinBase);
outpoint.n = i;
CoinEntry entry(&outpoint);
batch.Write(entry, newcoin);
}
}
batch.Erase(key);
if (batch.SizeEstimate() > batch_size) {
db.WriteBatch(batch);
batch.Clear();
Force on-the-fly compaction during pertxout upgrade
2017-06-03 02:52:37 +02:00
db.CompactRange(prev_key, key);
prev_key = key;
Upgrade from per-tx database to per-txout
2017-04-25 20:29:46 +02:00
}
pcursor->Next();
} else {
break;
}
}
db.WriteBatch(batch);
Force on-the-fly compaction during pertxout upgrade
2017-06-03 02:52:37 +02:00
db.CompactRange({DB_COINS, uint256()}, key);
Allow to cancel the txdb upgrade via splashscreen callback
2017-06-23 09:35:24 +02:00
uiInterface.SetProgressBreakAction(std::function<void(void)>());
Report [CANCELLED] instead of [DONE] when shut down during txdb upgrade
2017-06-28 21:28:11 +02:00
LogPrintf("[%s].\n", ShutdownRequested() ? "CANCELLED" : "DONE");
Upgrade from per-tx database to per-txout
2017-04-25 20:29:46 +02:00
return true;
}
Reference in a new issue Copy permalink