lbrycrd/src/util.cpp
Wladimir J. van der Laan bc679829e2
Merge #10271: Use std:🧵:hardware_concurrency, instead of Boost, to determine available cores
937bf4335 Use std:🧵:hardware_concurrency, instead of Boost, to determine available cores (fanquake)

Pull request description:

  Following discussion on IRC about replacing Boost usage for detecting available system cores, I've opened this to collect some benchmarks + further discussion.

  The current method for detecting available cores was introduced in #6361.

  Recap of the IRC chat:
  ```
  21:14:08 fanquake: Since we seem to be giving Boost removal a good shot for 0.15, does anyone have suggestions for replacing GetNumCores?
  21:14:26 fanquake: There is std:🧵:hardware_concurrency(), but that seems to count virtual cores, which I don't think we want.
  21:14:51 BlueMatt: fanquake: I doubt we'll do boost removal for 0.15
  21:14:58 BlueMatt: shit like BOOST_FOREACH, sure
  21:15:07 BlueMatt: but all of boost? doubtful, there are still things we need
  21:16:36 fanquake: Yea sorry, not the whole lot, but we can remove a decent chunk. Just looking into what else needs to be done to replace some of the less involved Boost usage.
  21:16:43 BlueMatt: fair
  21:17:14 wumpus: yes, it makes sense to plan ahead a bit, without immediately doing it
  21:18:12 wumpus: right, don't count virtual cores, that used to be the case but it makes no sense for our usage
  21:19:15 wumpus: it'd create a swarm of threads overwhelming any machine with hyperthreading (+accompanying thread stack overhead), for script validation, and there was no gain at all for that
  21:20:03 sipa: BlueMatt: don't worry, there is no hurry
  21:59:10 morcos: wumpus: i don't think that is correct
  21:59:24 morcos: suppose you have 4 cores (8 virtual cores)
  21:59:24 wumpus: fanquake: indeed seems that std has no equivalent to physical_concurrency, on any standard. That's annoying as it is non-trivial to implement
  21:59:35 morcos: i think running par=8 (if it let you) would be notably faster
  21:59:59 morcos: jeremyrubin and i discussed this at length a while back... i think i commented about it on irc at the time
  22:00:21 wumpus: morcos: I think the conclusion at the time was that it made no difference, but sure would make sense to benchmark
  22:00:39 morcos: perhaps historical testing on the virtual vs actual cores was polluted by concurrency issues that have now improved
  22:00:47 wumpus: I think there are not more ALUs, so there is not really a point in having more threads
  22:01:40 wumpus: hyperthreads are basically just a stored register state right?
  22:02:23 sipa: wumpus: yes but it helps the scheduler
  22:02:27 wumpus: in which case the only speedup using "number of cores" threads would give you is, possibly, excluding other software from running on the cores on the same time
  22:02:37 morcos: well this is where i get out of my depth
  22:02:50 sipa: if one of the threads is waiting on a read from ram, the other can use the arithmetic unit for example
  22:02:54 morcos: wumpus: i'm pretty sure though that the speed up is considerably more than what you might expect from that
  22:02:59 wumpus: sipa: ok, I back down, I didn't want to argue this at all
  22:03:35 morcos: the reason i haven't tested it myself, is the machine i usually use has 16 cores... so not easy due to remaining concurrency issues to get much more speedup
  22:03:36 wumpus: I'm fine with restoring it to number of virtual threads if that's faster
  22:03:54 morcos: we should have somene with 4 cores (and  actually test it though, i agree
  22:03:58 sipa: i would expect (but we should benchmark...) that if 8 scriot validation threads instead of 4 on a quadcore hyperthreading is not faster, it's due to lock contention
  22:04:20 morcos: sipa: yeah thats my point, i think lock contention isn't that bad with 8 now
  22:04:22 wumpus: on 64-bit systems the additional thread overhead wouldn't be important at least
  22:04:23 gmaxwell: I previously benchmarked, a long time ago, it was faster.
  22:04:33 gmaxwell: (to use the HT core count)
  22:04:44 wumpus: why was this changed at all then?
  22:04:47 wumpus: I'm confused
  22:05:04 sipa: good question!
  22:05:06 gmaxwell: I had no idea we changed it.
  22:05:25 wumpus: sigh 
  22:05:54 gmaxwell: What PR changed it?
  22:06:51 gmaxwell: In any case, on 32-bit it's probably a good tradeoff... the extra ram overhead is worth avoiding.
  22:07:22 wumpus: https://github.com/bitcoin/bitcoin/pull/6361
  22:07:28 gmaxwell: PR 6461 btw.
  22:07:37 gmaxwell: er lol at least you got it right.
  22:07:45 wumpus: the complaint was that systems became unsuably slow when using that many thread
  22:07:51 wumpus: so at least I got one thing right, woohoo
  22:07:55 sipa: seems i even acked it!
  22:07:57 BlueMatt: wumpus: there are more alus
  22:08:38 BlueMatt: but we need to improve lock contention first
  22:08:40 morcos: anywya, i think in the past the lock contention made 8 threads regardless of cores a bit dicey.. now that is much better (although more still to be done)
  22:09:01 BlueMatt: or we can just merge #10192, thats fee
  22:09:04 gribble: https://github.com/bitcoin/bitcoin/issues/10192 | Cache full script execution results in addition to signatures by TheBlueMatt · Pull Request #10192 · bitcoin/bitcoin · GitHub
  22:09:11 BlueMatt: s/fee/free/
  22:09:21 morcos: no, we do not need to improve lock contention first.   but we should probably do that before we increase the max beyond 16
  22:09:26 BlueMatt: then we can toss concurrency issues out the window and get more speedup anyway
  22:09:35 gmaxwell: wumpus: yea, well in QT I thought we also diminished the count by 1 or something?  but yes, if the motivation was to reduce how heavily the machine was used, thats fair.
  22:09:56 sipa: the benefit of using HT cores is certainly not a factor 2
  22:09:58 wumpus: gmaxwell: for the default I think this makes a lot of sense, yes
  22:10:10 gmaxwell: morcos: right now on my 24/28 physical core hosts going beyond 16 still reduces performance.
  22:10:11 wumpus: gmaxwell: do we also restrict the maximum par using this? that'd make less sense
  22:10:51 wumpus: if someone *wants* to use the virtual cores they should be able to by setting -par=
  22:10:51 sipa: *flies to US*
  22:10:52 BlueMatt: sipa: sure, but the shared cache helps us get more out of it than some others, as morcos points out
  22:11:30 BlueMatt: (because it means our thread contention issues are less)
  22:12:05 morcos: gmaxwell: yeah i've been bogged down in fee estimation as well (and the rest of life) for a while now.. otherwise i would have put more effort into jeremy's checkqueue
  22:12:36 BlueMatt: morcos: heh, well now you can do other stuff while the rest of us get bogged down in understanding fee estimation enough to review it 
  22:12:37 wumpus: [to answer my own question: no, the limit for par is MAX_SCRIPTCHECK_THREADS, or 16]
  22:12:54 morcos: but to me optimizing for more than 16 cores is pretty valuable as miners could use beefy machines and be less concerned by block validation time
  22:14:38 BlueMatt: morcos: i think you may be surprised by the number of mining pools that are on VPSes that do not have 16 cores 
  22:15:34 gmaxwell: I assume right now most of the time block validation is bogged in the parts that are not as concurrent. simple because caching makes the concurrent parts so fast. (and soon to hopefully increase with bluematt's patch)
  22:17:55 gmaxwell: improving sha2 speed, or transaction malloc overhead are probably bigger wins now for connection at the tip than parallelism beyond 16 (though I'd like that too).
  22:18:21 BlueMatt: sha2 speed is big
  22:18:27 morcos: yeah lots of things to do actually...
  22:18:57 gmaxwell: BlueMatt: might be a tiny bit less big if we didn't hash the block header 8 times for every block. 
  22:21:27 BlueMatt: ehh, probably, but I'm less rushed there
  22:21:43 BlueMatt: my new cache thing is about to add a bunch of hashing
  22:21:50 BlueMatt: 1 sha round per tx
  22:22:25 BlueMatt: and sigcache is obviously a ton
  ```

Tree-SHA512: a594430e2a77d8cc741ea8c664a2867b1e1693e5050a4bbc8511e8d66a2bffe241a9965f6dff1e7fbb99f21dd1fdeb95b826365da8bd8f9fab2d0ffd80d5059c
2018-03-06 19:21:34 +01:00

954 lines
28 KiB
C++

// Copyright (c) 2009-2010 Satoshi Nakamoto
// Copyright (c) 2009-2017 The Bitcoin Core developers
// Distributed under the MIT software license, see the accompanying
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
#include <util.h>
#include <fs.h>
#include <chainparamsbase.h>
#include <random.h>
#include <serialize.h>
#include <utilstrencodings.h>
#include <stdarg.h>
#if (defined(__FreeBSD__) || defined(__OpenBSD__) || defined(__DragonFly__))
#include <pthread.h>
#include <pthread_np.h>
#endif
#ifndef WIN32
// for posix_fallocate
#ifdef __linux__
#ifdef _POSIX_C_SOURCE
#undef _POSIX_C_SOURCE
#endif
#define _POSIX_C_SOURCE 200112L
#endif // __linux__
#include <algorithm>
#include <fcntl.h>
#include <sys/resource.h>
#include <sys/stat.h>
#else
#ifdef _MSC_VER
#pragma warning(disable:4786)
#pragma warning(disable:4804)
#pragma warning(disable:4805)
#pragma warning(disable:4717)
#endif
#ifdef _WIN32_WINNT
#undef _WIN32_WINNT
#endif
#define _WIN32_WINNT 0x0501
#ifdef _WIN32_IE
#undef _WIN32_IE
#endif
#define _WIN32_IE 0x0501
#define WIN32_LEAN_AND_MEAN 1
#ifndef NOMINMAX
#define NOMINMAX
#endif
#include <io.h> /* for _commit */
#include <shlobj.h>
#endif
#ifdef HAVE_SYS_PRCTL_H
#include <sys/prctl.h>
#endif
#ifdef HAVE_MALLOPT_ARENA_MAX
#include <malloc.h>
#endif
#include <boost/algorithm/string/case_conv.hpp> // for to_lower()
#include <boost/algorithm/string/predicate.hpp> // for startswith() and endswith()
#include <boost/interprocess/sync/file_lock.hpp>
#include <boost/program_options/detail/config_file.hpp>
#include <boost/thread.hpp>
#include <openssl/crypto.h>
#include <openssl/rand.h>
#include <openssl/conf.h>
#include <thread>
// Application startup time (used for uptime calculation)
const int64_t nStartupTime = GetTime();
const char * const BITCOIN_CONF_FILENAME = "bitcoin.conf";
const char * const BITCOIN_PID_FILENAME = "bitcoind.pid";
const char * const DEFAULT_DEBUGLOGFILE = "debug.log";
ArgsManager gArgs;
bool fPrintToConsole = false;
bool fPrintToDebugLog = true;
bool fLogTimestamps = DEFAULT_LOGTIMESTAMPS;
bool fLogTimeMicros = DEFAULT_LOGTIMEMICROS;
bool fLogIPs = DEFAULT_LOGIPS;
std::atomic<bool> fReopenDebugLog(false);
CTranslationInterface translationInterface;
/** Log categories bitfield. */
std::atomic<uint32_t> logCategories(0);
/** Init OpenSSL library multithreading support */
static std::unique_ptr<CCriticalSection[]> ppmutexOpenSSL;
void locking_callback(int mode, int i, const char* file, int line) NO_THREAD_SAFETY_ANALYSIS
{
if (mode & CRYPTO_LOCK) {
ENTER_CRITICAL_SECTION(ppmutexOpenSSL[i]);
} else {
LEAVE_CRITICAL_SECTION(ppmutexOpenSSL[i]);
}
}
// Singleton for wrapping OpenSSL setup/teardown.
class CInit
{
public:
CInit()
{
// Init OpenSSL library multithreading support
ppmutexOpenSSL.reset(new CCriticalSection[CRYPTO_num_locks()]);
CRYPTO_set_locking_callback(locking_callback);
// OpenSSL can optionally load a config file which lists optional loadable modules and engines.
// We don't use them so we don't require the config. However some of our libs may call functions
// which attempt to load the config file, possibly resulting in an exit() or crash if it is missing
// or corrupt. Explicitly tell OpenSSL not to try to load the file. The result for our libs will be
// that the config appears to have been loaded and there are no modules/engines available.
OPENSSL_no_config();
#ifdef WIN32
// Seed OpenSSL PRNG with current contents of the screen
RAND_screen();
#endif
// Seed OpenSSL PRNG with performance counter
RandAddSeed();
}
~CInit()
{
// Securely erase the memory used by the PRNG
RAND_cleanup();
// Shutdown OpenSSL library multithreading support
CRYPTO_set_locking_callback(nullptr);
// Clear the set of locks now to maintain symmetry with the constructor.
ppmutexOpenSSL.reset();
}
}
instance_of_cinit;
/**
* LogPrintf() has been broken a couple of times now
* by well-meaning people adding mutexes in the most straightforward way.
* It breaks because it may be called by global destructors during shutdown.
* Since the order of destruction of static/global objects is undefined,
* defining a mutex as a global object doesn't work (the mutex gets
* destroyed, and then some later destructor calls OutputDebugStringF,
* maybe indirectly, and you get a core dump at shutdown trying to lock
* the mutex).
*/
static boost::once_flag debugPrintInitFlag = BOOST_ONCE_INIT;
/**
* We use boost::call_once() to make sure mutexDebugLog and
* vMsgsBeforeOpenLog are initialized in a thread-safe manner.
*
* NOTE: fileout, mutexDebugLog and sometimes vMsgsBeforeOpenLog
* are leaked on exit. This is ugly, but will be cleaned up by
* the OS/libc. When the shutdown sequence is fully audited and
* tested, explicit destruction of these objects can be implemented.
*/
static FILE* fileout = nullptr;
static boost::mutex* mutexDebugLog = nullptr;
static std::list<std::string>* vMsgsBeforeOpenLog;
static int FileWriteStr(const std::string &str, FILE *fp)
{
return fwrite(str.data(), 1, str.size(), fp);
}
static void DebugPrintInit()
{
assert(mutexDebugLog == nullptr);
mutexDebugLog = new boost::mutex();
vMsgsBeforeOpenLog = new std::list<std::string>;
}
fs::path GetDebugLogPath()
{
fs::path logfile(gArgs.GetArg("-debuglogfile", DEFAULT_DEBUGLOGFILE));
return AbsPathForConfigVal(logfile);
}
bool OpenDebugLog()
{
boost::call_once(&DebugPrintInit, debugPrintInitFlag);
boost::mutex::scoped_lock scoped_lock(*mutexDebugLog);
assert(fileout == nullptr);
assert(vMsgsBeforeOpenLog);
fs::path pathDebug = GetDebugLogPath();
fileout = fsbridge::fopen(pathDebug, "a");
if (!fileout) {
return false;
}
setbuf(fileout, nullptr); // unbuffered
// dump buffered messages from before we opened the log
while (!vMsgsBeforeOpenLog->empty()) {
FileWriteStr(vMsgsBeforeOpenLog->front(), fileout);
vMsgsBeforeOpenLog->pop_front();
}
delete vMsgsBeforeOpenLog;
vMsgsBeforeOpenLog = nullptr;
return true;
}
struct CLogCategoryDesc
{
uint32_t flag;
std::string category;
};
const CLogCategoryDesc LogCategories[] =
{
{BCLog::NONE, "0"},
{BCLog::NONE, "none"},
{BCLog::NET, "net"},
{BCLog::TOR, "tor"},
{BCLog::MEMPOOL, "mempool"},
{BCLog::HTTP, "http"},
{BCLog::BENCH, "bench"},
{BCLog::ZMQ, "zmq"},
{BCLog::DB, "db"},
{BCLog::RPC, "rpc"},
{BCLog::ESTIMATEFEE, "estimatefee"},
{BCLog::ADDRMAN, "addrman"},
{BCLog::SELECTCOINS, "selectcoins"},
{BCLog::REINDEX, "reindex"},
{BCLog::CMPCTBLOCK, "cmpctblock"},
{BCLog::RAND, "rand"},
{BCLog::PRUNE, "prune"},
{BCLog::PROXY, "proxy"},
{BCLog::MEMPOOLREJ, "mempoolrej"},
{BCLog::LIBEVENT, "libevent"},
{BCLog::COINDB, "coindb"},
{BCLog::QT, "qt"},
{BCLog::LEVELDB, "leveldb"},
{BCLog::ALL, "1"},
{BCLog::ALL, "all"},
};
bool GetLogCategory(uint32_t *f, const std::string *str)
{
if (f && str) {
if (*str == "") {
*f = BCLog::ALL;
return true;
}
for (unsigned int i = 0; i < ARRAYLEN(LogCategories); i++) {
if (LogCategories[i].category == *str) {
*f = LogCategories[i].flag;
return true;
}
}
}
return false;
}
std::string ListLogCategories()
{
std::string ret;
int outcount = 0;
for (unsigned int i = 0; i < ARRAYLEN(LogCategories); i++) {
// Omit the special cases.
if (LogCategories[i].flag != BCLog::NONE && LogCategories[i].flag != BCLog::ALL) {
if (outcount != 0) ret += ", ";
ret += LogCategories[i].category;
outcount++;
}
}
return ret;
}
std::vector<CLogCategoryActive> ListActiveLogCategories()
{
std::vector<CLogCategoryActive> ret;
for (unsigned int i = 0; i < ARRAYLEN(LogCategories); i++) {
// Omit the special cases.
if (LogCategories[i].flag != BCLog::NONE && LogCategories[i].flag != BCLog::ALL) {
CLogCategoryActive catActive;
catActive.category = LogCategories[i].category;
catActive.active = LogAcceptCategory(LogCategories[i].flag);
ret.push_back(catActive);
}
}
return ret;
}
/**
* fStartedNewLine is a state variable held by the calling context that will
* suppress printing of the timestamp when multiple calls are made that don't
* end in a newline. Initialize it to true, and hold it, in the calling context.
*/
static std::string LogTimestampStr(const std::string &str, std::atomic_bool *fStartedNewLine)
{
std::string strStamped;
if (!fLogTimestamps)
return str;
if (*fStartedNewLine) {
int64_t nTimeMicros = GetTimeMicros();
strStamped = DateTimeStrFormat("%Y-%m-%d %H:%M:%S", nTimeMicros/1000000);
if (fLogTimeMicros)
strStamped += strprintf(".%06d", nTimeMicros%1000000);
int64_t mocktime = GetMockTime();
if (mocktime) {
strStamped += " (mocktime: " + DateTimeStrFormat("%Y-%m-%d %H:%M:%S", mocktime) + ")";
}
strStamped += ' ' + str;
} else
strStamped = str;
if (!str.empty() && str[str.size()-1] == '\n')
*fStartedNewLine = true;
else
*fStartedNewLine = false;
return strStamped;
}
int LogPrintStr(const std::string &str)
{
int ret = 0; // Returns total number of characters written
static std::atomic_bool fStartedNewLine(true);
std::string strTimestamped = LogTimestampStr(str, &fStartedNewLine);
if (fPrintToConsole)
{
// print to console
ret = fwrite(strTimestamped.data(), 1, strTimestamped.size(), stdout);
fflush(stdout);
}
else if (fPrintToDebugLog)
{
boost::call_once(&DebugPrintInit, debugPrintInitFlag);
boost::mutex::scoped_lock scoped_lock(*mutexDebugLog);
// buffer if we haven't opened the log yet
if (fileout == nullptr) {
assert(vMsgsBeforeOpenLog);
ret = strTimestamped.length();
vMsgsBeforeOpenLog->push_back(strTimestamped);
}
else
{
// reopen the log file, if requested
if (fReopenDebugLog) {
fReopenDebugLog = false;
fs::path pathDebug = GetDebugLogPath();
if (fsbridge::freopen(pathDebug,"a",fileout) != nullptr)
setbuf(fileout, nullptr); // unbuffered
}
ret = FileWriteStr(strTimestamped, fileout);
}
}
return ret;
}
/** A map that contains all the currently held directory locks. After
* successful locking, these will be held here until the global destructor
* cleans them up and thus automatically unlocks them, or ReleaseDirectoryLocks
* is called.
*/
static std::map<std::string, std::unique_ptr<boost::interprocess::file_lock>> dir_locks;
/** Mutex to protect dir_locks. */
static std::mutex cs_dir_locks;
bool LockDirectory(const fs::path& directory, const std::string lockfile_name, bool probe_only)
{
std::lock_guard<std::mutex> ulock(cs_dir_locks);
fs::path pathLockFile = directory / lockfile_name;
// If a lock for this directory already exists in the map, don't try to re-lock it
if (dir_locks.count(pathLockFile.string())) {
return true;
}
// Create empty lock file if it doesn't exist.
FILE* file = fsbridge::fopen(pathLockFile, "a");
if (file) fclose(file);
try {
auto lock = MakeUnique<boost::interprocess::file_lock>(pathLockFile.string().c_str());
if (!lock->try_lock()) {
return false;
}
if (!probe_only) {
// Lock successful and we're not just probing, put it into the map
dir_locks.emplace(pathLockFile.string(), std::move(lock));
}
} catch (const boost::interprocess::interprocess_exception& e) {
return error("Error while attempting to lock directory %s: %s", directory.string(), e.what());
}
return true;
}
void ReleaseDirectoryLocks()
{
std::lock_guard<std::mutex> ulock(cs_dir_locks);
dir_locks.clear();
}
/** Interpret string as boolean, for argument parsing */
static bool InterpretBool(const std::string& strValue)
{
if (strValue.empty())
return true;
return (atoi(strValue) != 0);
}
/** Turn -noX into -X=0 */
static void InterpretNegativeSetting(std::string& strKey, std::string& strValue)
{
if (strKey.length()>3 && strKey[0]=='-' && strKey[1]=='n' && strKey[2]=='o')
{
strKey = "-" + strKey.substr(3);
strValue = InterpretBool(strValue) ? "0" : "1";
}
}
void ArgsManager::ParseParameters(int argc, const char* const argv[])
{
LOCK(cs_args);
mapArgs.clear();
mapMultiArgs.clear();
for (int i = 1; i < argc; i++)
{
std::string str(argv[i]);
std::string strValue;
size_t is_index = str.find('=');
if (is_index != std::string::npos)
{
strValue = str.substr(is_index+1);
str = str.substr(0, is_index);
}
#ifdef WIN32
boost::to_lower(str);
if (boost::algorithm::starts_with(str, "/"))
str = "-" + str.substr(1);
#endif
if (str[0] != '-')
break;
// Interpret --foo as -foo.
// If both --foo and -foo are set, the last takes effect.
if (str.length() > 1 && str[1] == '-')
str = str.substr(1);
InterpretNegativeSetting(str, strValue);
mapArgs[str] = strValue;
mapMultiArgs[str].push_back(strValue);
}
}
std::vector<std::string> ArgsManager::GetArgs(const std::string& strArg) const
{
LOCK(cs_args);
auto it = mapMultiArgs.find(strArg);
if (it != mapMultiArgs.end()) return it->second;
return {};
}
bool ArgsManager::IsArgSet(const std::string& strArg) const
{
LOCK(cs_args);
return mapArgs.count(strArg);
}
std::string ArgsManager::GetArg(const std::string& strArg, const std::string& strDefault) const
{
LOCK(cs_args);
auto it = mapArgs.find(strArg);
if (it != mapArgs.end()) return it->second;
return strDefault;
}
int64_t ArgsManager::GetArg(const std::string& strArg, int64_t nDefault) const
{
LOCK(cs_args);
auto it = mapArgs.find(strArg);
if (it != mapArgs.end()) return atoi64(it->second);
return nDefault;
}
bool ArgsManager::GetBoolArg(const std::string& strArg, bool fDefault) const
{
LOCK(cs_args);
auto it = mapArgs.find(strArg);
if (it != mapArgs.end()) return InterpretBool(it->second);
return fDefault;
}
bool ArgsManager::SoftSetArg(const std::string& strArg, const std::string& strValue)
{
LOCK(cs_args);
if (IsArgSet(strArg)) return false;
ForceSetArg(strArg, strValue);
return true;
}
bool ArgsManager::SoftSetBoolArg(const std::string& strArg, bool fValue)
{
if (fValue)
return SoftSetArg(strArg, std::string("1"));
else
return SoftSetArg(strArg, std::string("0"));
}
void ArgsManager::ForceSetArg(const std::string& strArg, const std::string& strValue)
{
LOCK(cs_args);
mapArgs[strArg] = strValue;
mapMultiArgs[strArg] = {strValue};
}
static const int screenWidth = 79;
static const int optIndent = 2;
static const int msgIndent = 7;
std::string HelpMessageGroup(const std::string &message) {
return std::string(message) + std::string("\n\n");
}
std::string HelpMessageOpt(const std::string &option, const std::string &message) {
return std::string(optIndent,' ') + std::string(option) +
std::string("\n") + std::string(msgIndent,' ') +
FormatParagraph(message, screenWidth - msgIndent, msgIndent) +
std::string("\n\n");
}
static std::string FormatException(const std::exception* pex, const char* pszThread)
{
#ifdef WIN32
char pszModule[MAX_PATH] = "";
GetModuleFileNameA(nullptr, pszModule, sizeof(pszModule));
#else
const char* pszModule = "bitcoin";
#endif
if (pex)
return strprintf(
"EXCEPTION: %s \n%s \n%s in %s \n", typeid(*pex).name(), pex->what(), pszModule, pszThread);
else
return strprintf(
"UNKNOWN EXCEPTION \n%s in %s \n", pszModule, pszThread);
}
void PrintExceptionContinue(const std::exception* pex, const char* pszThread)
{
std::string message = FormatException(pex, pszThread);
LogPrintf("\n\n************************\n%s\n", message);
fprintf(stderr, "\n\n************************\n%s\n", message.c_str());
}
fs::path GetDefaultDataDir()
{
// Windows < Vista: C:\Documents and Settings\Username\Application Data\Bitcoin
// Windows >= Vista: C:\Users\Username\AppData\Roaming\Bitcoin
// Mac: ~/Library/Application Support/Bitcoin
// Unix: ~/.bitcoin
#ifdef WIN32
// Windows
return GetSpecialFolderPath(CSIDL_APPDATA) / "Bitcoin";
#else
fs::path pathRet;
char* pszHome = getenv("HOME");
if (pszHome == nullptr || strlen(pszHome) == 0)
pathRet = fs::path("/");
else
pathRet = fs::path(pszHome);
#ifdef MAC_OSX
// Mac
return pathRet / "Library/Application Support/Bitcoin";
#else
// Unix
return pathRet / ".bitcoin";
#endif
#endif
}
static fs::path pathCached;
static fs::path pathCachedNetSpecific;
static CCriticalSection csPathCached;
const fs::path &GetDataDir(bool fNetSpecific)
{
LOCK(csPathCached);
fs::path &path = fNetSpecific ? pathCachedNetSpecific : pathCached;
// This can be called during exceptions by LogPrintf(), so we cache the
// value so we don't have to do memory allocations after that.
if (!path.empty())
return path;
if (gArgs.IsArgSet("-datadir")) {
path = fs::system_complete(gArgs.GetArg("-datadir", ""));
if (!fs::is_directory(path)) {
path = "";
return path;
}
} else {
path = GetDefaultDataDir();
}
if (fNetSpecific)
path /= BaseParams().DataDir();
if (fs::create_directories(path)) {
// This is the first run, create wallets subdirectory too
fs::create_directories(path / "wallets");
}
return path;
}
void ClearDatadirCache()
{
LOCK(csPathCached);
pathCached = fs::path();
pathCachedNetSpecific = fs::path();
}
fs::path GetConfigFile(const std::string& confPath)
{
return AbsPathForConfigVal(fs::path(confPath), false);
}
void ArgsManager::ReadConfigFile(const std::string& confPath)
{
fs::ifstream streamConfig(GetConfigFile(confPath));
if (!streamConfig.good())
return; // No bitcoin.conf file is OK
{
LOCK(cs_args);
std::set<std::string> setOptions;
setOptions.insert("*");
for (boost::program_options::detail::config_file_iterator it(streamConfig, setOptions), end; it != end; ++it)
{
// Don't overwrite existing settings so command line settings override bitcoin.conf
std::string strKey = std::string("-") + it->string_key;
std::string strValue = it->value[0];
InterpretNegativeSetting(strKey, strValue);
if (mapArgs.count(strKey) == 0)
mapArgs[strKey] = strValue;
mapMultiArgs[strKey].push_back(strValue);
}
}
// If datadir is changed in .conf file:
ClearDatadirCache();
if (!fs::is_directory(GetDataDir(false))) {
throw std::runtime_error(strprintf("specified data directory \"%s\" does not exist.", gArgs.GetArg("-datadir", "").c_str()));
}
}
#ifndef WIN32
fs::path GetPidFile()
{
return AbsPathForConfigVal(fs::path(gArgs.GetArg("-pid", BITCOIN_PID_FILENAME)));
}
void CreatePidFile(const fs::path &path, pid_t pid)
{
FILE* file = fsbridge::fopen(path, "w");
if (file)
{
fprintf(file, "%d\n", pid);
fclose(file);
}
}
#endif
bool RenameOver(fs::path src, fs::path dest)
{
#ifdef WIN32
return MoveFileExA(src.string().c_str(), dest.string().c_str(),
MOVEFILE_REPLACE_EXISTING) != 0;
#else
int rc = std::rename(src.string().c_str(), dest.string().c_str());
return (rc == 0);
#endif /* WIN32 */
}
/**
* Ignores exceptions thrown by Boost's create_directories if the requested directory exists.
* Specifically handles case where path p exists, but it wasn't possible for the user to
* write to the parent directory.
*/
bool TryCreateDirectories(const fs::path& p)
{
try
{
return fs::create_directories(p);
} catch (const fs::filesystem_error&) {
if (!fs::exists(p) || !fs::is_directory(p))
throw;
}
// create_directories didn't create the directory, it had to have existed already
return false;
}
void FileCommit(FILE *file)
{
fflush(file); // harmless if redundantly called
#ifdef WIN32
HANDLE hFile = (HANDLE)_get_osfhandle(_fileno(file));
FlushFileBuffers(hFile);
#else
#if defined(__linux__) || defined(__NetBSD__)
fdatasync(fileno(file));
#elif defined(__APPLE__) && defined(F_FULLFSYNC)
fcntl(fileno(file), F_FULLFSYNC, 0);
#else
fsync(fileno(file));
#endif
#endif
}
bool TruncateFile(FILE *file, unsigned int length) {
#if defined(WIN32)
return _chsize(_fileno(file), length) == 0;
#else
return ftruncate(fileno(file), length) == 0;
#endif
}
/**
* this function tries to raise the file descriptor limit to the requested number.
* It returns the actual file descriptor limit (which may be more or less than nMinFD)
*/
int RaiseFileDescriptorLimit(int nMinFD) {
#if defined(WIN32)
return 2048;
#else
struct rlimit limitFD;
if (getrlimit(RLIMIT_NOFILE, &limitFD) != -1) {
if (limitFD.rlim_cur < (rlim_t)nMinFD) {
limitFD.rlim_cur = nMinFD;
if (limitFD.rlim_cur > limitFD.rlim_max)
limitFD.rlim_cur = limitFD.rlim_max;
setrlimit(RLIMIT_NOFILE, &limitFD);
getrlimit(RLIMIT_NOFILE, &limitFD);
}
return limitFD.rlim_cur;
}
return nMinFD; // getrlimit failed, assume it's fine
#endif
}
/**
* this function tries to make a particular range of a file allocated (corresponding to disk space)
* it is advisory, and the range specified in the arguments will never contain live data
*/
void AllocateFileRange(FILE *file, unsigned int offset, unsigned int length) {
#if defined(WIN32)
// Windows-specific version
HANDLE hFile = (HANDLE)_get_osfhandle(_fileno(file));
LARGE_INTEGER nFileSize;
int64_t nEndPos = (int64_t)offset + length;
nFileSize.u.LowPart = nEndPos & 0xFFFFFFFF;
nFileSize.u.HighPart = nEndPos >> 32;
SetFilePointerEx(hFile, nFileSize, 0, FILE_BEGIN);
SetEndOfFile(hFile);
#elif defined(MAC_OSX)
// OSX specific version
fstore_t fst;
fst.fst_flags = F_ALLOCATECONTIG;
fst.fst_posmode = F_PEOFPOSMODE;
fst.fst_offset = 0;
fst.fst_length = (off_t)offset + length;
fst.fst_bytesalloc = 0;
if (fcntl(fileno(file), F_PREALLOCATE, &fst) == -1) {
fst.fst_flags = F_ALLOCATEALL;
fcntl(fileno(file), F_PREALLOCATE, &fst);
}
ftruncate(fileno(file), fst.fst_length);
#elif defined(__linux__)
// Version using posix_fallocate
off_t nEndPos = (off_t)offset + length;
posix_fallocate(fileno(file), 0, nEndPos);
#else
// Fallback version
// TODO: just write one byte per block
static const char buf[65536] = {};
fseek(file, offset, SEEK_SET);
while (length > 0) {
unsigned int now = 65536;
if (length < now)
now = length;
fwrite(buf, 1, now, file); // allowed to fail; this function is advisory anyway
length -= now;
}
#endif
}
void ShrinkDebugFile()
{
// Amount of debug.log to save at end when shrinking (must fit in memory)
constexpr size_t RECENT_DEBUG_HISTORY_SIZE = 10 * 1000000;
// Scroll debug.log if it's getting too big
fs::path pathLog = GetDebugLogPath();
FILE* file = fsbridge::fopen(pathLog, "r");
// If debug.log file is more than 10% bigger the RECENT_DEBUG_HISTORY_SIZE
// trim it down by saving only the last RECENT_DEBUG_HISTORY_SIZE bytes
if (file && fs::file_size(pathLog) > 11 * (RECENT_DEBUG_HISTORY_SIZE / 10))
{
// Restart the file with some of the end
std::vector<char> vch(RECENT_DEBUG_HISTORY_SIZE, 0);
fseek(file, -((long)vch.size()), SEEK_END);
int nBytes = fread(vch.data(), 1, vch.size(), file);
fclose(file);
file = fsbridge::fopen(pathLog, "w");
if (file)
{
fwrite(vch.data(), 1, nBytes, file);
fclose(file);
}
}
else if (file != nullptr)
fclose(file);
}
#ifdef WIN32
fs::path GetSpecialFolderPath(int nFolder, bool fCreate)
{
char pszPath[MAX_PATH] = "";
if(SHGetSpecialFolderPathA(nullptr, pszPath, nFolder, fCreate))
{
return fs::path(pszPath);
}
LogPrintf("SHGetSpecialFolderPathA() failed, could not obtain requested path.\n");
return fs::path("");
}
#endif
void runCommand(const std::string& strCommand)
{
if (strCommand.empty()) return;
int nErr = ::system(strCommand.c_str());
if (nErr)
LogPrintf("runCommand error: system(%s) returned %d\n", strCommand, nErr);
}
void RenameThread(const char* name)
{
#if defined(PR_SET_NAME)
// Only the first 15 characters are used (16 - NUL terminator)
::prctl(PR_SET_NAME, name, 0, 0, 0);
#elif (defined(__FreeBSD__) || defined(__OpenBSD__) || defined(__DragonFly__))
pthread_set_name_np(pthread_self(), name);
#elif defined(MAC_OSX)
pthread_setname_np(name);
#else
// Prevent warnings for unused parameters...
(void)name;
#endif
}
void SetupEnvironment()
{
#ifdef HAVE_MALLOPT_ARENA_MAX
// glibc-specific: On 32-bit systems set the number of arenas to 1.
// By default, since glibc 2.10, the C library will create up to two heap
// arenas per core. This is known to cause excessive virtual address space
// usage in our usage. Work around it by setting the maximum number of
// arenas to 1.
if (sizeof(void*) == 4) {
mallopt(M_ARENA_MAX, 1);
}
#endif
// On most POSIX systems (e.g. Linux, but not BSD) the environment's locale
// may be invalid, in which case the "C" locale is used as fallback.
#if !defined(WIN32) && !defined(MAC_OSX) && !defined(__FreeBSD__) && !defined(__OpenBSD__)
try {
std::locale(""); // Raises a runtime error if current locale is invalid
} catch (const std::runtime_error&) {
setenv("LC_ALL", "C", 1);
}
#endif
// The path locale is lazy initialized and to avoid deinitialization errors
// in multithreading environments, it is set explicitly by the main thread.
// A dummy locale is used to extract the internal default locale, used by
// fs::path, which is then used to explicitly imbue the path.
std::locale loc = fs::path::imbue(std::locale::classic());
fs::path::imbue(loc);
}
bool SetupNetworking()
{
#ifdef WIN32
// Initialize Windows Sockets
WSADATA wsadata;
int ret = WSAStartup(MAKEWORD(2,2), &wsadata);
if (ret != NO_ERROR || LOBYTE(wsadata.wVersion ) != 2 || HIBYTE(wsadata.wVersion) != 2)
return false;
#endif
return true;
}
int GetNumCores()
{
return std::thread::hardware_concurrency();
}
std::string CopyrightHolders(const std::string& strPrefix)
{
std::string strCopyrightHolders = strPrefix + strprintf(_(COPYRIGHT_HOLDERS), _(COPYRIGHT_HOLDERS_SUBSTITUTION));
// Check for untranslated substitution to make sure Bitcoin Core copyright is not removed by accident
if (strprintf(COPYRIGHT_HOLDERS, COPYRIGHT_HOLDERS_SUBSTITUTION).find("Bitcoin Core") == std::string::npos) {
strCopyrightHolders += "\n" + strPrefix + "The Bitcoin Core developers";
}
return strCopyrightHolders;
}
// Obtain the application startup time (used for uptime calculation)
int64_t GetStartupTime()
{
return nStartupTime;
}
fs::path AbsPathForConfigVal(const fs::path& path, bool net_specific)
{
return fs::absolute(path, GetDataDir(net_specific));
}