bc679829e2
937bf4335
Use std:🧵:hardware_concurrency, instead of Boost, to determine available cores (fanquake)
Pull request description:
Following discussion on IRC about replacing Boost usage for detecting available system cores, I've opened this to collect some benchmarks + further discussion.
The current method for detecting available cores was introduced in #6361.
Recap of the IRC chat:
```
21:14:08 fanquake: Since we seem to be giving Boost removal a good shot for 0.15, does anyone have suggestions for replacing GetNumCores?
21:14:26 fanquake: There is std:🧵:hardware_concurrency(), but that seems to count virtual cores, which I don't think we want.
21:14:51 BlueMatt: fanquake: I doubt we'll do boost removal for 0.15
21:14:58 BlueMatt: shit like BOOST_FOREACH, sure
21:15:07 BlueMatt: but all of boost? doubtful, there are still things we need
21:16:36 fanquake: Yea sorry, not the whole lot, but we can remove a decent chunk. Just looking into what else needs to be done to replace some of the less involved Boost usage.
21:16:43 BlueMatt: fair
21:17:14 wumpus: yes, it makes sense to plan ahead a bit, without immediately doing it
21:18:12 wumpus: right, don't count virtual cores, that used to be the case but it makes no sense for our usage
21:19:15 wumpus: it'd create a swarm of threads overwhelming any machine with hyperthreading (+accompanying thread stack overhead), for script validation, and there was no gain at all for that
21:20:03 sipa: BlueMatt: don't worry, there is no hurry
21:59:10 morcos: wumpus: i don't think that is correct
21:59:24 morcos: suppose you have 4 cores (8 virtual cores)
21:59:24 wumpus: fanquake: indeed seems that std has no equivalent to physical_concurrency, on any standard. That's annoying as it is non-trivial to implement
21:59:35 morcos: i think running par=8 (if it let you) would be notably faster
21:59:59 morcos: jeremyrubin and i discussed this at length a while back... i think i commented about it on irc at the time
22:00:21 wumpus: morcos: I think the conclusion at the time was that it made no difference, but sure would make sense to benchmark
22:00:39 morcos: perhaps historical testing on the virtual vs actual cores was polluted by concurrency issues that have now improved
22:00:47 wumpus: I think there are not more ALUs, so there is not really a point in having more threads
22:01:40 wumpus: hyperthreads are basically just a stored register state right?
22:02:23 sipa: wumpus: yes but it helps the scheduler
22:02:27 wumpus: in which case the only speedup using "number of cores" threads would give you is, possibly, excluding other software from running on the cores on the same time
22:02:37 morcos: well this is where i get out of my depth
22:02:50 sipa: if one of the threads is waiting on a read from ram, the other can use the arithmetic unit for example
22:02:54 morcos: wumpus: i'm pretty sure though that the speed up is considerably more than what you might expect from that
22:02:59 wumpus: sipa: ok, I back down, I didn't want to argue this at all
22:03:35 morcos: the reason i haven't tested it myself, is the machine i usually use has 16 cores... so not easy due to remaining concurrency issues to get much more speedup
22:03:36 wumpus: I'm fine with restoring it to number of virtual threads if that's faster
22:03:54 morcos: we should have somene with 4 cores (and  actually test it though, i agree
22:03:58 sipa: i would expect (but we should benchmark...) that if 8 scriot validation threads instead of 4 on a quadcore hyperthreading is not faster, it's due to lock contention
22:04:20 morcos: sipa: yeah thats my point, i think lock contention isn't that bad with 8 now
22:04:22 wumpus: on 64-bit systems the additional thread overhead wouldn't be important at least
22:04:23 gmaxwell: I previously benchmarked, a long time ago, it was faster.
22:04:33 gmaxwell: (to use the HT core count)
22:04:44 wumpus: why was this changed at all then?
22:04:47 wumpus: I'm confused
22:05:04 sipa: good question!
22:05:06 gmaxwell: I had no idea we changed it.
22:05:25 wumpus: sigh 
22:05:54 gmaxwell: What PR changed it?
22:06:51 gmaxwell: In any case, on 32-bit it's probably a good tradeoff... the extra ram overhead is worth avoiding.
22:07:22 wumpus: https://github.com/bitcoin/bitcoin/pull/6361
22:07:28 gmaxwell: PR 6461 btw.
22:07:37 gmaxwell: er lol at least you got it right.
22:07:45 wumpus: the complaint was that systems became unsuably slow when using that many thread
22:07:51 wumpus: so at least I got one thing right, woohoo
22:07:55 sipa: seems i even acked it!
22:07:57 BlueMatt: wumpus: there are more alus
22:08:38 BlueMatt: but we need to improve lock contention first
22:08:40 morcos: anywya, i think in the past the lock contention made 8 threads regardless of cores a bit dicey.. now that is much better (although more still to be done)
22:09:01 BlueMatt: or we can just merge #10192, thats fee
22:09:04 gribble: https://github.com/bitcoin/bitcoin/issues/10192 | Cache full script execution results in addition to signatures by TheBlueMatt · Pull Request #10192 · bitcoin/bitcoin · GitHub
22:09:11 BlueMatt: s/fee/free/
22:09:21 morcos: no, we do not need to improve lock contention first. but we should probably do that before we increase the max beyond 16
22:09:26 BlueMatt: then we can toss concurrency issues out the window and get more speedup anyway
22:09:35 gmaxwell: wumpus: yea, well in QT I thought we also diminished the count by 1 or something? but yes, if the motivation was to reduce how heavily the machine was used, thats fair.
22:09:56 sipa: the benefit of using HT cores is certainly not a factor 2
22:09:58 wumpus: gmaxwell: for the default I think this makes a lot of sense, yes
22:10:10 gmaxwell: morcos: right now on my 24/28 physical core hosts going beyond 16 still reduces performance.
22:10:11 wumpus: gmaxwell: do we also restrict the maximum par using this? that'd make less sense
22:10:51 wumpus: if someone *wants* to use the virtual cores they should be able to by setting -par=
22:10:51 sipa: *flies to US*
22:10:52 BlueMatt: sipa: sure, but the shared cache helps us get more out of it than some others, as morcos points out
22:11:30 BlueMatt: (because it means our thread contention issues are less)
22:12:05 morcos: gmaxwell: yeah i've been bogged down in fee estimation as well (and the rest of life) for a while now.. otherwise i would have put more effort into jeremy's checkqueue
22:12:36 BlueMatt: morcos: heh, well now you can do other stuff while the rest of us get bogged down in understanding fee estimation enough to review it 
22:12:37 wumpus: [to answer my own question: no, the limit for par is MAX_SCRIPTCHECK_THREADS, or 16]
22:12:54 morcos: but to me optimizing for more than 16 cores is pretty valuable as miners could use beefy machines and be less concerned by block validation time
22:14:38 BlueMatt: morcos: i think you may be surprised by the number of mining pools that are on VPSes that do not have 16 cores 
22:15:34 gmaxwell: I assume right now most of the time block validation is bogged in the parts that are not as concurrent. simple because caching makes the concurrent parts so fast. (and soon to hopefully increase with bluematt's patch)
22:17:55 gmaxwell: improving sha2 speed, or transaction malloc overhead are probably bigger wins now for connection at the tip than parallelism beyond 16 (though I'd like that too).
22:18:21 BlueMatt: sha2 speed is big
22:18:27 morcos: yeah lots of things to do actually...
22:18:57 gmaxwell: BlueMatt: might be a tiny bit less big if we didn't hash the block header 8 times for every block. 
22:21:27 BlueMatt: ehh, probably, but I'm less rushed there
22:21:43 BlueMatt: my new cache thing is about to add a bunch of hashing
22:21:50 BlueMatt: 1 sha round per tx
22:22:25 BlueMatt: and sigcache is obviously a ton
```
Tree-SHA512: a594430e2a77d8cc741ea8c664a2867b1e1693e5050a4bbc8511e8d66a2bffe241a9965f6dff1e7fbb99f21dd1fdeb95b826365da8bd8f9fab2d0ffd80d5059c
954 lines
28 KiB
C++
954 lines
28 KiB
C++
// Copyright (c) 2009-2010 Satoshi Nakamoto
|
|
// Copyright (c) 2009-2017 The Bitcoin Core developers
|
|
// Distributed under the MIT software license, see the accompanying
|
|
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
|
|
|
#include <util.h>
|
|
#include <fs.h>
|
|
|
|
#include <chainparamsbase.h>
|
|
#include <random.h>
|
|
#include <serialize.h>
|
|
#include <utilstrencodings.h>
|
|
|
|
#include <stdarg.h>
|
|
|
|
#if (defined(__FreeBSD__) || defined(__OpenBSD__) || defined(__DragonFly__))
|
|
#include <pthread.h>
|
|
#include <pthread_np.h>
|
|
#endif
|
|
|
|
#ifndef WIN32
|
|
// for posix_fallocate
|
|
#ifdef __linux__
|
|
|
|
#ifdef _POSIX_C_SOURCE
|
|
#undef _POSIX_C_SOURCE
|
|
#endif
|
|
|
|
#define _POSIX_C_SOURCE 200112L
|
|
|
|
#endif // __linux__
|
|
|
|
#include <algorithm>
|
|
#include <fcntl.h>
|
|
#include <sys/resource.h>
|
|
#include <sys/stat.h>
|
|
|
|
#else
|
|
|
|
#ifdef _MSC_VER
|
|
#pragma warning(disable:4786)
|
|
#pragma warning(disable:4804)
|
|
#pragma warning(disable:4805)
|
|
#pragma warning(disable:4717)
|
|
#endif
|
|
|
|
#ifdef _WIN32_WINNT
|
|
#undef _WIN32_WINNT
|
|
#endif
|
|
#define _WIN32_WINNT 0x0501
|
|
|
|
#ifdef _WIN32_IE
|
|
#undef _WIN32_IE
|
|
#endif
|
|
#define _WIN32_IE 0x0501
|
|
|
|
#define WIN32_LEAN_AND_MEAN 1
|
|
#ifndef NOMINMAX
|
|
#define NOMINMAX
|
|
#endif
|
|
|
|
#include <io.h> /* for _commit */
|
|
#include <shlobj.h>
|
|
#endif
|
|
|
|
#ifdef HAVE_SYS_PRCTL_H
|
|
#include <sys/prctl.h>
|
|
#endif
|
|
|
|
#ifdef HAVE_MALLOPT_ARENA_MAX
|
|
#include <malloc.h>
|
|
#endif
|
|
|
|
#include <boost/algorithm/string/case_conv.hpp> // for to_lower()
|
|
#include <boost/algorithm/string/predicate.hpp> // for startswith() and endswith()
|
|
#include <boost/interprocess/sync/file_lock.hpp>
|
|
#include <boost/program_options/detail/config_file.hpp>
|
|
#include <boost/thread.hpp>
|
|
#include <openssl/crypto.h>
|
|
#include <openssl/rand.h>
|
|
#include <openssl/conf.h>
|
|
#include <thread>
|
|
|
|
// Application startup time (used for uptime calculation)
|
|
const int64_t nStartupTime = GetTime();
|
|
|
|
const char * const BITCOIN_CONF_FILENAME = "bitcoin.conf";
|
|
const char * const BITCOIN_PID_FILENAME = "bitcoind.pid";
|
|
const char * const DEFAULT_DEBUGLOGFILE = "debug.log";
|
|
|
|
ArgsManager gArgs;
|
|
bool fPrintToConsole = false;
|
|
bool fPrintToDebugLog = true;
|
|
|
|
bool fLogTimestamps = DEFAULT_LOGTIMESTAMPS;
|
|
bool fLogTimeMicros = DEFAULT_LOGTIMEMICROS;
|
|
bool fLogIPs = DEFAULT_LOGIPS;
|
|
std::atomic<bool> fReopenDebugLog(false);
|
|
CTranslationInterface translationInterface;
|
|
|
|
/** Log categories bitfield. */
|
|
std::atomic<uint32_t> logCategories(0);
|
|
|
|
/** Init OpenSSL library multithreading support */
|
|
static std::unique_ptr<CCriticalSection[]> ppmutexOpenSSL;
|
|
void locking_callback(int mode, int i, const char* file, int line) NO_THREAD_SAFETY_ANALYSIS
|
|
{
|
|
if (mode & CRYPTO_LOCK) {
|
|
ENTER_CRITICAL_SECTION(ppmutexOpenSSL[i]);
|
|
} else {
|
|
LEAVE_CRITICAL_SECTION(ppmutexOpenSSL[i]);
|
|
}
|
|
}
|
|
|
|
// Singleton for wrapping OpenSSL setup/teardown.
|
|
class CInit
|
|
{
|
|
public:
|
|
CInit()
|
|
{
|
|
// Init OpenSSL library multithreading support
|
|
ppmutexOpenSSL.reset(new CCriticalSection[CRYPTO_num_locks()]);
|
|
CRYPTO_set_locking_callback(locking_callback);
|
|
|
|
// OpenSSL can optionally load a config file which lists optional loadable modules and engines.
|
|
// We don't use them so we don't require the config. However some of our libs may call functions
|
|
// which attempt to load the config file, possibly resulting in an exit() or crash if it is missing
|
|
// or corrupt. Explicitly tell OpenSSL not to try to load the file. The result for our libs will be
|
|
// that the config appears to have been loaded and there are no modules/engines available.
|
|
OPENSSL_no_config();
|
|
|
|
#ifdef WIN32
|
|
// Seed OpenSSL PRNG with current contents of the screen
|
|
RAND_screen();
|
|
#endif
|
|
|
|
// Seed OpenSSL PRNG with performance counter
|
|
RandAddSeed();
|
|
}
|
|
~CInit()
|
|
{
|
|
// Securely erase the memory used by the PRNG
|
|
RAND_cleanup();
|
|
// Shutdown OpenSSL library multithreading support
|
|
CRYPTO_set_locking_callback(nullptr);
|
|
// Clear the set of locks now to maintain symmetry with the constructor.
|
|
ppmutexOpenSSL.reset();
|
|
}
|
|
}
|
|
instance_of_cinit;
|
|
|
|
/**
|
|
* LogPrintf() has been broken a couple of times now
|
|
* by well-meaning people adding mutexes in the most straightforward way.
|
|
* It breaks because it may be called by global destructors during shutdown.
|
|
* Since the order of destruction of static/global objects is undefined,
|
|
* defining a mutex as a global object doesn't work (the mutex gets
|
|
* destroyed, and then some later destructor calls OutputDebugStringF,
|
|
* maybe indirectly, and you get a core dump at shutdown trying to lock
|
|
* the mutex).
|
|
*/
|
|
|
|
static boost::once_flag debugPrintInitFlag = BOOST_ONCE_INIT;
|
|
|
|
/**
|
|
* We use boost::call_once() to make sure mutexDebugLog and
|
|
* vMsgsBeforeOpenLog are initialized in a thread-safe manner.
|
|
*
|
|
* NOTE: fileout, mutexDebugLog and sometimes vMsgsBeforeOpenLog
|
|
* are leaked on exit. This is ugly, but will be cleaned up by
|
|
* the OS/libc. When the shutdown sequence is fully audited and
|
|
* tested, explicit destruction of these objects can be implemented.
|
|
*/
|
|
static FILE* fileout = nullptr;
|
|
static boost::mutex* mutexDebugLog = nullptr;
|
|
static std::list<std::string>* vMsgsBeforeOpenLog;
|
|
|
|
static int FileWriteStr(const std::string &str, FILE *fp)
|
|
{
|
|
return fwrite(str.data(), 1, str.size(), fp);
|
|
}
|
|
|
|
static void DebugPrintInit()
|
|
{
|
|
assert(mutexDebugLog == nullptr);
|
|
mutexDebugLog = new boost::mutex();
|
|
vMsgsBeforeOpenLog = new std::list<std::string>;
|
|
}
|
|
|
|
fs::path GetDebugLogPath()
|
|
{
|
|
fs::path logfile(gArgs.GetArg("-debuglogfile", DEFAULT_DEBUGLOGFILE));
|
|
return AbsPathForConfigVal(logfile);
|
|
}
|
|
|
|
bool OpenDebugLog()
|
|
{
|
|
boost::call_once(&DebugPrintInit, debugPrintInitFlag);
|
|
boost::mutex::scoped_lock scoped_lock(*mutexDebugLog);
|
|
|
|
assert(fileout == nullptr);
|
|
assert(vMsgsBeforeOpenLog);
|
|
fs::path pathDebug = GetDebugLogPath();
|
|
|
|
fileout = fsbridge::fopen(pathDebug, "a");
|
|
if (!fileout) {
|
|
return false;
|
|
}
|
|
|
|
setbuf(fileout, nullptr); // unbuffered
|
|
// dump buffered messages from before we opened the log
|
|
while (!vMsgsBeforeOpenLog->empty()) {
|
|
FileWriteStr(vMsgsBeforeOpenLog->front(), fileout);
|
|
vMsgsBeforeOpenLog->pop_front();
|
|
}
|
|
|
|
delete vMsgsBeforeOpenLog;
|
|
vMsgsBeforeOpenLog = nullptr;
|
|
return true;
|
|
}
|
|
|
|
struct CLogCategoryDesc
|
|
{
|
|
uint32_t flag;
|
|
std::string category;
|
|
};
|
|
|
|
const CLogCategoryDesc LogCategories[] =
|
|
{
|
|
{BCLog::NONE, "0"},
|
|
{BCLog::NONE, "none"},
|
|
{BCLog::NET, "net"},
|
|
{BCLog::TOR, "tor"},
|
|
{BCLog::MEMPOOL, "mempool"},
|
|
{BCLog::HTTP, "http"},
|
|
{BCLog::BENCH, "bench"},
|
|
{BCLog::ZMQ, "zmq"},
|
|
{BCLog::DB, "db"},
|
|
{BCLog::RPC, "rpc"},
|
|
{BCLog::ESTIMATEFEE, "estimatefee"},
|
|
{BCLog::ADDRMAN, "addrman"},
|
|
{BCLog::SELECTCOINS, "selectcoins"},
|
|
{BCLog::REINDEX, "reindex"},
|
|
{BCLog::CMPCTBLOCK, "cmpctblock"},
|
|
{BCLog::RAND, "rand"},
|
|
{BCLog::PRUNE, "prune"},
|
|
{BCLog::PROXY, "proxy"},
|
|
{BCLog::MEMPOOLREJ, "mempoolrej"},
|
|
{BCLog::LIBEVENT, "libevent"},
|
|
{BCLog::COINDB, "coindb"},
|
|
{BCLog::QT, "qt"},
|
|
{BCLog::LEVELDB, "leveldb"},
|
|
{BCLog::ALL, "1"},
|
|
{BCLog::ALL, "all"},
|
|
};
|
|
|
|
bool GetLogCategory(uint32_t *f, const std::string *str)
|
|
{
|
|
if (f && str) {
|
|
if (*str == "") {
|
|
*f = BCLog::ALL;
|
|
return true;
|
|
}
|
|
for (unsigned int i = 0; i < ARRAYLEN(LogCategories); i++) {
|
|
if (LogCategories[i].category == *str) {
|
|
*f = LogCategories[i].flag;
|
|
return true;
|
|
}
|
|
}
|
|
}
|
|
return false;
|
|
}
|
|
|
|
std::string ListLogCategories()
|
|
{
|
|
std::string ret;
|
|
int outcount = 0;
|
|
for (unsigned int i = 0; i < ARRAYLEN(LogCategories); i++) {
|
|
// Omit the special cases.
|
|
if (LogCategories[i].flag != BCLog::NONE && LogCategories[i].flag != BCLog::ALL) {
|
|
if (outcount != 0) ret += ", ";
|
|
ret += LogCategories[i].category;
|
|
outcount++;
|
|
}
|
|
}
|
|
return ret;
|
|
}
|
|
|
|
std::vector<CLogCategoryActive> ListActiveLogCategories()
|
|
{
|
|
std::vector<CLogCategoryActive> ret;
|
|
for (unsigned int i = 0; i < ARRAYLEN(LogCategories); i++) {
|
|
// Omit the special cases.
|
|
if (LogCategories[i].flag != BCLog::NONE && LogCategories[i].flag != BCLog::ALL) {
|
|
CLogCategoryActive catActive;
|
|
catActive.category = LogCategories[i].category;
|
|
catActive.active = LogAcceptCategory(LogCategories[i].flag);
|
|
ret.push_back(catActive);
|
|
}
|
|
}
|
|
return ret;
|
|
}
|
|
|
|
/**
|
|
* fStartedNewLine is a state variable held by the calling context that will
|
|
* suppress printing of the timestamp when multiple calls are made that don't
|
|
* end in a newline. Initialize it to true, and hold it, in the calling context.
|
|
*/
|
|
static std::string LogTimestampStr(const std::string &str, std::atomic_bool *fStartedNewLine)
|
|
{
|
|
std::string strStamped;
|
|
|
|
if (!fLogTimestamps)
|
|
return str;
|
|
|
|
if (*fStartedNewLine) {
|
|
int64_t nTimeMicros = GetTimeMicros();
|
|
strStamped = DateTimeStrFormat("%Y-%m-%d %H:%M:%S", nTimeMicros/1000000);
|
|
if (fLogTimeMicros)
|
|
strStamped += strprintf(".%06d", nTimeMicros%1000000);
|
|
int64_t mocktime = GetMockTime();
|
|
if (mocktime) {
|
|
strStamped += " (mocktime: " + DateTimeStrFormat("%Y-%m-%d %H:%M:%S", mocktime) + ")";
|
|
}
|
|
strStamped += ' ' + str;
|
|
} else
|
|
strStamped = str;
|
|
|
|
if (!str.empty() && str[str.size()-1] == '\n')
|
|
*fStartedNewLine = true;
|
|
else
|
|
*fStartedNewLine = false;
|
|
|
|
return strStamped;
|
|
}
|
|
|
|
int LogPrintStr(const std::string &str)
|
|
{
|
|
int ret = 0; // Returns total number of characters written
|
|
static std::atomic_bool fStartedNewLine(true);
|
|
|
|
std::string strTimestamped = LogTimestampStr(str, &fStartedNewLine);
|
|
|
|
if (fPrintToConsole)
|
|
{
|
|
// print to console
|
|
ret = fwrite(strTimestamped.data(), 1, strTimestamped.size(), stdout);
|
|
fflush(stdout);
|
|
}
|
|
else if (fPrintToDebugLog)
|
|
{
|
|
boost::call_once(&DebugPrintInit, debugPrintInitFlag);
|
|
boost::mutex::scoped_lock scoped_lock(*mutexDebugLog);
|
|
|
|
// buffer if we haven't opened the log yet
|
|
if (fileout == nullptr) {
|
|
assert(vMsgsBeforeOpenLog);
|
|
ret = strTimestamped.length();
|
|
vMsgsBeforeOpenLog->push_back(strTimestamped);
|
|
}
|
|
else
|
|
{
|
|
// reopen the log file, if requested
|
|
if (fReopenDebugLog) {
|
|
fReopenDebugLog = false;
|
|
fs::path pathDebug = GetDebugLogPath();
|
|
if (fsbridge::freopen(pathDebug,"a",fileout) != nullptr)
|
|
setbuf(fileout, nullptr); // unbuffered
|
|
}
|
|
|
|
ret = FileWriteStr(strTimestamped, fileout);
|
|
}
|
|
}
|
|
return ret;
|
|
}
|
|
|
|
/** A map that contains all the currently held directory locks. After
|
|
* successful locking, these will be held here until the global destructor
|
|
* cleans them up and thus automatically unlocks them, or ReleaseDirectoryLocks
|
|
* is called.
|
|
*/
|
|
static std::map<std::string, std::unique_ptr<boost::interprocess::file_lock>> dir_locks;
|
|
/** Mutex to protect dir_locks. */
|
|
static std::mutex cs_dir_locks;
|
|
|
|
bool LockDirectory(const fs::path& directory, const std::string lockfile_name, bool probe_only)
|
|
{
|
|
std::lock_guard<std::mutex> ulock(cs_dir_locks);
|
|
fs::path pathLockFile = directory / lockfile_name;
|
|
|
|
// If a lock for this directory already exists in the map, don't try to re-lock it
|
|
if (dir_locks.count(pathLockFile.string())) {
|
|
return true;
|
|
}
|
|
|
|
// Create empty lock file if it doesn't exist.
|
|
FILE* file = fsbridge::fopen(pathLockFile, "a");
|
|
if (file) fclose(file);
|
|
|
|
try {
|
|
auto lock = MakeUnique<boost::interprocess::file_lock>(pathLockFile.string().c_str());
|
|
if (!lock->try_lock()) {
|
|
return false;
|
|
}
|
|
if (!probe_only) {
|
|
// Lock successful and we're not just probing, put it into the map
|
|
dir_locks.emplace(pathLockFile.string(), std::move(lock));
|
|
}
|
|
} catch (const boost::interprocess::interprocess_exception& e) {
|
|
return error("Error while attempting to lock directory %s: %s", directory.string(), e.what());
|
|
}
|
|
return true;
|
|
}
|
|
|
|
void ReleaseDirectoryLocks()
|
|
{
|
|
std::lock_guard<std::mutex> ulock(cs_dir_locks);
|
|
dir_locks.clear();
|
|
}
|
|
|
|
/** Interpret string as boolean, for argument parsing */
|
|
static bool InterpretBool(const std::string& strValue)
|
|
{
|
|
if (strValue.empty())
|
|
return true;
|
|
return (atoi(strValue) != 0);
|
|
}
|
|
|
|
/** Turn -noX into -X=0 */
|
|
static void InterpretNegativeSetting(std::string& strKey, std::string& strValue)
|
|
{
|
|
if (strKey.length()>3 && strKey[0]=='-' && strKey[1]=='n' && strKey[2]=='o')
|
|
{
|
|
strKey = "-" + strKey.substr(3);
|
|
strValue = InterpretBool(strValue) ? "0" : "1";
|
|
}
|
|
}
|
|
|
|
void ArgsManager::ParseParameters(int argc, const char* const argv[])
|
|
{
|
|
LOCK(cs_args);
|
|
mapArgs.clear();
|
|
mapMultiArgs.clear();
|
|
|
|
for (int i = 1; i < argc; i++)
|
|
{
|
|
std::string str(argv[i]);
|
|
std::string strValue;
|
|
size_t is_index = str.find('=');
|
|
if (is_index != std::string::npos)
|
|
{
|
|
strValue = str.substr(is_index+1);
|
|
str = str.substr(0, is_index);
|
|
}
|
|
#ifdef WIN32
|
|
boost::to_lower(str);
|
|
if (boost::algorithm::starts_with(str, "/"))
|
|
str = "-" + str.substr(1);
|
|
#endif
|
|
|
|
if (str[0] != '-')
|
|
break;
|
|
|
|
// Interpret --foo as -foo.
|
|
// If both --foo and -foo are set, the last takes effect.
|
|
if (str.length() > 1 && str[1] == '-')
|
|
str = str.substr(1);
|
|
InterpretNegativeSetting(str, strValue);
|
|
|
|
mapArgs[str] = strValue;
|
|
mapMultiArgs[str].push_back(strValue);
|
|
}
|
|
}
|
|
|
|
std::vector<std::string> ArgsManager::GetArgs(const std::string& strArg) const
|
|
{
|
|
LOCK(cs_args);
|
|
auto it = mapMultiArgs.find(strArg);
|
|
if (it != mapMultiArgs.end()) return it->second;
|
|
return {};
|
|
}
|
|
|
|
bool ArgsManager::IsArgSet(const std::string& strArg) const
|
|
{
|
|
LOCK(cs_args);
|
|
return mapArgs.count(strArg);
|
|
}
|
|
|
|
std::string ArgsManager::GetArg(const std::string& strArg, const std::string& strDefault) const
|
|
{
|
|
LOCK(cs_args);
|
|
auto it = mapArgs.find(strArg);
|
|
if (it != mapArgs.end()) return it->second;
|
|
return strDefault;
|
|
}
|
|
|
|
int64_t ArgsManager::GetArg(const std::string& strArg, int64_t nDefault) const
|
|
{
|
|
LOCK(cs_args);
|
|
auto it = mapArgs.find(strArg);
|
|
if (it != mapArgs.end()) return atoi64(it->second);
|
|
return nDefault;
|
|
}
|
|
|
|
bool ArgsManager::GetBoolArg(const std::string& strArg, bool fDefault) const
|
|
{
|
|
LOCK(cs_args);
|
|
auto it = mapArgs.find(strArg);
|
|
if (it != mapArgs.end()) return InterpretBool(it->second);
|
|
return fDefault;
|
|
}
|
|
|
|
bool ArgsManager::SoftSetArg(const std::string& strArg, const std::string& strValue)
|
|
{
|
|
LOCK(cs_args);
|
|
if (IsArgSet(strArg)) return false;
|
|
ForceSetArg(strArg, strValue);
|
|
return true;
|
|
}
|
|
|
|
bool ArgsManager::SoftSetBoolArg(const std::string& strArg, bool fValue)
|
|
{
|
|
if (fValue)
|
|
return SoftSetArg(strArg, std::string("1"));
|
|
else
|
|
return SoftSetArg(strArg, std::string("0"));
|
|
}
|
|
|
|
void ArgsManager::ForceSetArg(const std::string& strArg, const std::string& strValue)
|
|
{
|
|
LOCK(cs_args);
|
|
mapArgs[strArg] = strValue;
|
|
mapMultiArgs[strArg] = {strValue};
|
|
}
|
|
|
|
|
|
|
|
static const int screenWidth = 79;
|
|
static const int optIndent = 2;
|
|
static const int msgIndent = 7;
|
|
|
|
std::string HelpMessageGroup(const std::string &message) {
|
|
return std::string(message) + std::string("\n\n");
|
|
}
|
|
|
|
std::string HelpMessageOpt(const std::string &option, const std::string &message) {
|
|
return std::string(optIndent,' ') + std::string(option) +
|
|
std::string("\n") + std::string(msgIndent,' ') +
|
|
FormatParagraph(message, screenWidth - msgIndent, msgIndent) +
|
|
std::string("\n\n");
|
|
}
|
|
|
|
static std::string FormatException(const std::exception* pex, const char* pszThread)
|
|
{
|
|
#ifdef WIN32
|
|
char pszModule[MAX_PATH] = "";
|
|
GetModuleFileNameA(nullptr, pszModule, sizeof(pszModule));
|
|
#else
|
|
const char* pszModule = "bitcoin";
|
|
#endif
|
|
if (pex)
|
|
return strprintf(
|
|
"EXCEPTION: %s \n%s \n%s in %s \n", typeid(*pex).name(), pex->what(), pszModule, pszThread);
|
|
else
|
|
return strprintf(
|
|
"UNKNOWN EXCEPTION \n%s in %s \n", pszModule, pszThread);
|
|
}
|
|
|
|
void PrintExceptionContinue(const std::exception* pex, const char* pszThread)
|
|
{
|
|
std::string message = FormatException(pex, pszThread);
|
|
LogPrintf("\n\n************************\n%s\n", message);
|
|
fprintf(stderr, "\n\n************************\n%s\n", message.c_str());
|
|
}
|
|
|
|
fs::path GetDefaultDataDir()
|
|
{
|
|
// Windows < Vista: C:\Documents and Settings\Username\Application Data\Bitcoin
|
|
// Windows >= Vista: C:\Users\Username\AppData\Roaming\Bitcoin
|
|
// Mac: ~/Library/Application Support/Bitcoin
|
|
// Unix: ~/.bitcoin
|
|
#ifdef WIN32
|
|
// Windows
|
|
return GetSpecialFolderPath(CSIDL_APPDATA) / "Bitcoin";
|
|
#else
|
|
fs::path pathRet;
|
|
char* pszHome = getenv("HOME");
|
|
if (pszHome == nullptr || strlen(pszHome) == 0)
|
|
pathRet = fs::path("/");
|
|
else
|
|
pathRet = fs::path(pszHome);
|
|
#ifdef MAC_OSX
|
|
// Mac
|
|
return pathRet / "Library/Application Support/Bitcoin";
|
|
#else
|
|
// Unix
|
|
return pathRet / ".bitcoin";
|
|
#endif
|
|
#endif
|
|
}
|
|
|
|
static fs::path pathCached;
|
|
static fs::path pathCachedNetSpecific;
|
|
static CCriticalSection csPathCached;
|
|
|
|
const fs::path &GetDataDir(bool fNetSpecific)
|
|
{
|
|
|
|
LOCK(csPathCached);
|
|
|
|
fs::path &path = fNetSpecific ? pathCachedNetSpecific : pathCached;
|
|
|
|
// This can be called during exceptions by LogPrintf(), so we cache the
|
|
// value so we don't have to do memory allocations after that.
|
|
if (!path.empty())
|
|
return path;
|
|
|
|
if (gArgs.IsArgSet("-datadir")) {
|
|
path = fs::system_complete(gArgs.GetArg("-datadir", ""));
|
|
if (!fs::is_directory(path)) {
|
|
path = "";
|
|
return path;
|
|
}
|
|
} else {
|
|
path = GetDefaultDataDir();
|
|
}
|
|
if (fNetSpecific)
|
|
path /= BaseParams().DataDir();
|
|
|
|
if (fs::create_directories(path)) {
|
|
// This is the first run, create wallets subdirectory too
|
|
fs::create_directories(path / "wallets");
|
|
}
|
|
|
|
return path;
|
|
}
|
|
|
|
void ClearDatadirCache()
|
|
{
|
|
LOCK(csPathCached);
|
|
|
|
pathCached = fs::path();
|
|
pathCachedNetSpecific = fs::path();
|
|
}
|
|
|
|
fs::path GetConfigFile(const std::string& confPath)
|
|
{
|
|
return AbsPathForConfigVal(fs::path(confPath), false);
|
|
}
|
|
|
|
void ArgsManager::ReadConfigFile(const std::string& confPath)
|
|
{
|
|
fs::ifstream streamConfig(GetConfigFile(confPath));
|
|
if (!streamConfig.good())
|
|
return; // No bitcoin.conf file is OK
|
|
|
|
{
|
|
LOCK(cs_args);
|
|
std::set<std::string> setOptions;
|
|
setOptions.insert("*");
|
|
|
|
for (boost::program_options::detail::config_file_iterator it(streamConfig, setOptions), end; it != end; ++it)
|
|
{
|
|
// Don't overwrite existing settings so command line settings override bitcoin.conf
|
|
std::string strKey = std::string("-") + it->string_key;
|
|
std::string strValue = it->value[0];
|
|
InterpretNegativeSetting(strKey, strValue);
|
|
if (mapArgs.count(strKey) == 0)
|
|
mapArgs[strKey] = strValue;
|
|
mapMultiArgs[strKey].push_back(strValue);
|
|
}
|
|
}
|
|
// If datadir is changed in .conf file:
|
|
ClearDatadirCache();
|
|
if (!fs::is_directory(GetDataDir(false))) {
|
|
throw std::runtime_error(strprintf("specified data directory \"%s\" does not exist.", gArgs.GetArg("-datadir", "").c_str()));
|
|
}
|
|
}
|
|
|
|
#ifndef WIN32
|
|
fs::path GetPidFile()
|
|
{
|
|
return AbsPathForConfigVal(fs::path(gArgs.GetArg("-pid", BITCOIN_PID_FILENAME)));
|
|
}
|
|
|
|
void CreatePidFile(const fs::path &path, pid_t pid)
|
|
{
|
|
FILE* file = fsbridge::fopen(path, "w");
|
|
if (file)
|
|
{
|
|
fprintf(file, "%d\n", pid);
|
|
fclose(file);
|
|
}
|
|
}
|
|
#endif
|
|
|
|
bool RenameOver(fs::path src, fs::path dest)
|
|
{
|
|
#ifdef WIN32
|
|
return MoveFileExA(src.string().c_str(), dest.string().c_str(),
|
|
MOVEFILE_REPLACE_EXISTING) != 0;
|
|
#else
|
|
int rc = std::rename(src.string().c_str(), dest.string().c_str());
|
|
return (rc == 0);
|
|
#endif /* WIN32 */
|
|
}
|
|
|
|
/**
|
|
* Ignores exceptions thrown by Boost's create_directories if the requested directory exists.
|
|
* Specifically handles case where path p exists, but it wasn't possible for the user to
|
|
* write to the parent directory.
|
|
*/
|
|
bool TryCreateDirectories(const fs::path& p)
|
|
{
|
|
try
|
|
{
|
|
return fs::create_directories(p);
|
|
} catch (const fs::filesystem_error&) {
|
|
if (!fs::exists(p) || !fs::is_directory(p))
|
|
throw;
|
|
}
|
|
|
|
// create_directories didn't create the directory, it had to have existed already
|
|
return false;
|
|
}
|
|
|
|
void FileCommit(FILE *file)
|
|
{
|
|
fflush(file); // harmless if redundantly called
|
|
#ifdef WIN32
|
|
HANDLE hFile = (HANDLE)_get_osfhandle(_fileno(file));
|
|
FlushFileBuffers(hFile);
|
|
#else
|
|
#if defined(__linux__) || defined(__NetBSD__)
|
|
fdatasync(fileno(file));
|
|
#elif defined(__APPLE__) && defined(F_FULLFSYNC)
|
|
fcntl(fileno(file), F_FULLFSYNC, 0);
|
|
#else
|
|
fsync(fileno(file));
|
|
#endif
|
|
#endif
|
|
}
|
|
|
|
bool TruncateFile(FILE *file, unsigned int length) {
|
|
#if defined(WIN32)
|
|
return _chsize(_fileno(file), length) == 0;
|
|
#else
|
|
return ftruncate(fileno(file), length) == 0;
|
|
#endif
|
|
}
|
|
|
|
/**
|
|
* this function tries to raise the file descriptor limit to the requested number.
|
|
* It returns the actual file descriptor limit (which may be more or less than nMinFD)
|
|
*/
|
|
int RaiseFileDescriptorLimit(int nMinFD) {
|
|
#if defined(WIN32)
|
|
return 2048;
|
|
#else
|
|
struct rlimit limitFD;
|
|
if (getrlimit(RLIMIT_NOFILE, &limitFD) != -1) {
|
|
if (limitFD.rlim_cur < (rlim_t)nMinFD) {
|
|
limitFD.rlim_cur = nMinFD;
|
|
if (limitFD.rlim_cur > limitFD.rlim_max)
|
|
limitFD.rlim_cur = limitFD.rlim_max;
|
|
setrlimit(RLIMIT_NOFILE, &limitFD);
|
|
getrlimit(RLIMIT_NOFILE, &limitFD);
|
|
}
|
|
return limitFD.rlim_cur;
|
|
}
|
|
return nMinFD; // getrlimit failed, assume it's fine
|
|
#endif
|
|
}
|
|
|
|
/**
|
|
* this function tries to make a particular range of a file allocated (corresponding to disk space)
|
|
* it is advisory, and the range specified in the arguments will never contain live data
|
|
*/
|
|
void AllocateFileRange(FILE *file, unsigned int offset, unsigned int length) {
|
|
#if defined(WIN32)
|
|
// Windows-specific version
|
|
HANDLE hFile = (HANDLE)_get_osfhandle(_fileno(file));
|
|
LARGE_INTEGER nFileSize;
|
|
int64_t nEndPos = (int64_t)offset + length;
|
|
nFileSize.u.LowPart = nEndPos & 0xFFFFFFFF;
|
|
nFileSize.u.HighPart = nEndPos >> 32;
|
|
SetFilePointerEx(hFile, nFileSize, 0, FILE_BEGIN);
|
|
SetEndOfFile(hFile);
|
|
#elif defined(MAC_OSX)
|
|
// OSX specific version
|
|
fstore_t fst;
|
|
fst.fst_flags = F_ALLOCATECONTIG;
|
|
fst.fst_posmode = F_PEOFPOSMODE;
|
|
fst.fst_offset = 0;
|
|
fst.fst_length = (off_t)offset + length;
|
|
fst.fst_bytesalloc = 0;
|
|
if (fcntl(fileno(file), F_PREALLOCATE, &fst) == -1) {
|
|
fst.fst_flags = F_ALLOCATEALL;
|
|
fcntl(fileno(file), F_PREALLOCATE, &fst);
|
|
}
|
|
ftruncate(fileno(file), fst.fst_length);
|
|
#elif defined(__linux__)
|
|
// Version using posix_fallocate
|
|
off_t nEndPos = (off_t)offset + length;
|
|
posix_fallocate(fileno(file), 0, nEndPos);
|
|
#else
|
|
// Fallback version
|
|
// TODO: just write one byte per block
|
|
static const char buf[65536] = {};
|
|
fseek(file, offset, SEEK_SET);
|
|
while (length > 0) {
|
|
unsigned int now = 65536;
|
|
if (length < now)
|
|
now = length;
|
|
fwrite(buf, 1, now, file); // allowed to fail; this function is advisory anyway
|
|
length -= now;
|
|
}
|
|
#endif
|
|
}
|
|
|
|
void ShrinkDebugFile()
|
|
{
|
|
// Amount of debug.log to save at end when shrinking (must fit in memory)
|
|
constexpr size_t RECENT_DEBUG_HISTORY_SIZE = 10 * 1000000;
|
|
// Scroll debug.log if it's getting too big
|
|
fs::path pathLog = GetDebugLogPath();
|
|
FILE* file = fsbridge::fopen(pathLog, "r");
|
|
// If debug.log file is more than 10% bigger the RECENT_DEBUG_HISTORY_SIZE
|
|
// trim it down by saving only the last RECENT_DEBUG_HISTORY_SIZE bytes
|
|
if (file && fs::file_size(pathLog) > 11 * (RECENT_DEBUG_HISTORY_SIZE / 10))
|
|
{
|
|
// Restart the file with some of the end
|
|
std::vector<char> vch(RECENT_DEBUG_HISTORY_SIZE, 0);
|
|
fseek(file, -((long)vch.size()), SEEK_END);
|
|
int nBytes = fread(vch.data(), 1, vch.size(), file);
|
|
fclose(file);
|
|
|
|
file = fsbridge::fopen(pathLog, "w");
|
|
if (file)
|
|
{
|
|
fwrite(vch.data(), 1, nBytes, file);
|
|
fclose(file);
|
|
}
|
|
}
|
|
else if (file != nullptr)
|
|
fclose(file);
|
|
}
|
|
|
|
#ifdef WIN32
|
|
fs::path GetSpecialFolderPath(int nFolder, bool fCreate)
|
|
{
|
|
char pszPath[MAX_PATH] = "";
|
|
|
|
if(SHGetSpecialFolderPathA(nullptr, pszPath, nFolder, fCreate))
|
|
{
|
|
return fs::path(pszPath);
|
|
}
|
|
|
|
LogPrintf("SHGetSpecialFolderPathA() failed, could not obtain requested path.\n");
|
|
return fs::path("");
|
|
}
|
|
#endif
|
|
|
|
void runCommand(const std::string& strCommand)
|
|
{
|
|
if (strCommand.empty()) return;
|
|
int nErr = ::system(strCommand.c_str());
|
|
if (nErr)
|
|
LogPrintf("runCommand error: system(%s) returned %d\n", strCommand, nErr);
|
|
}
|
|
|
|
void RenameThread(const char* name)
|
|
{
|
|
#if defined(PR_SET_NAME)
|
|
// Only the first 15 characters are used (16 - NUL terminator)
|
|
::prctl(PR_SET_NAME, name, 0, 0, 0);
|
|
#elif (defined(__FreeBSD__) || defined(__OpenBSD__) || defined(__DragonFly__))
|
|
pthread_set_name_np(pthread_self(), name);
|
|
|
|
#elif defined(MAC_OSX)
|
|
pthread_setname_np(name);
|
|
#else
|
|
// Prevent warnings for unused parameters...
|
|
(void)name;
|
|
#endif
|
|
}
|
|
|
|
void SetupEnvironment()
|
|
{
|
|
#ifdef HAVE_MALLOPT_ARENA_MAX
|
|
// glibc-specific: On 32-bit systems set the number of arenas to 1.
|
|
// By default, since glibc 2.10, the C library will create up to two heap
|
|
// arenas per core. This is known to cause excessive virtual address space
|
|
// usage in our usage. Work around it by setting the maximum number of
|
|
// arenas to 1.
|
|
if (sizeof(void*) == 4) {
|
|
mallopt(M_ARENA_MAX, 1);
|
|
}
|
|
#endif
|
|
// On most POSIX systems (e.g. Linux, but not BSD) the environment's locale
|
|
// may be invalid, in which case the "C" locale is used as fallback.
|
|
#if !defined(WIN32) && !defined(MAC_OSX) && !defined(__FreeBSD__) && !defined(__OpenBSD__)
|
|
try {
|
|
std::locale(""); // Raises a runtime error if current locale is invalid
|
|
} catch (const std::runtime_error&) {
|
|
setenv("LC_ALL", "C", 1);
|
|
}
|
|
#endif
|
|
// The path locale is lazy initialized and to avoid deinitialization errors
|
|
// in multithreading environments, it is set explicitly by the main thread.
|
|
// A dummy locale is used to extract the internal default locale, used by
|
|
// fs::path, which is then used to explicitly imbue the path.
|
|
std::locale loc = fs::path::imbue(std::locale::classic());
|
|
fs::path::imbue(loc);
|
|
}
|
|
|
|
bool SetupNetworking()
|
|
{
|
|
#ifdef WIN32
|
|
// Initialize Windows Sockets
|
|
WSADATA wsadata;
|
|
int ret = WSAStartup(MAKEWORD(2,2), &wsadata);
|
|
if (ret != NO_ERROR || LOBYTE(wsadata.wVersion ) != 2 || HIBYTE(wsadata.wVersion) != 2)
|
|
return false;
|
|
#endif
|
|
return true;
|
|
}
|
|
|
|
int GetNumCores()
|
|
{
|
|
return std::thread::hardware_concurrency();
|
|
}
|
|
|
|
std::string CopyrightHolders(const std::string& strPrefix)
|
|
{
|
|
std::string strCopyrightHolders = strPrefix + strprintf(_(COPYRIGHT_HOLDERS), _(COPYRIGHT_HOLDERS_SUBSTITUTION));
|
|
|
|
// Check for untranslated substitution to make sure Bitcoin Core copyright is not removed by accident
|
|
if (strprintf(COPYRIGHT_HOLDERS, COPYRIGHT_HOLDERS_SUBSTITUTION).find("Bitcoin Core") == std::string::npos) {
|
|
strCopyrightHolders += "\n" + strPrefix + "The Bitcoin Core developers";
|
|
}
|
|
return strCopyrightHolders;
|
|
}
|
|
|
|
// Obtain the application startup time (used for uptime calculation)
|
|
int64_t GetStartupTime()
|
|
{
|
|
return nStartupTime;
|
|
}
|
|
|
|
fs::path AbsPathForConfigVal(const fs::path& path, bool net_specific)
|
|
{
|
|
return fs::absolute(path, GetDataDir(net_specific));
|
|
}
|