LBRYCommunity/lbrycrd
1
0
Fork 0
Code Issues 29 Pull requests 1 Packages Projects Releases 32 Wiki Activity
The blockchain that provides the digital content namespace for the LBRY protocol
16552 commits 22 branches 42 tags 99 MiB
C++ 66.5%
Python 15.1%
C 12.7%
M4 2.1%
Makefile 1.2%
Other 2.3%
Go to file
Wladimir J. van der Laan 047865e8d1
Merge #12495: Increase LevelDB max_open_files 
ccedbaf Increase LevelDB max_open_files unless on 32-bit Unix. (Evan Klitzke)

Pull request description:

  Currently we set `max_open_files = 64` on all architectures due to concerns about file descriptor exhaustion. This is extremely expensive due to how LevelDB is designed.

  When a LevelDB file handle is opened, a bloom filter and block index are decoded, and some CRCs are checked. Bloom filters and block indexes in open table handles can be checked purely in memory. This means that when doing a key lookup, if a given table file may contain a given key, all of the lookup operations can happen completely in RAM until the block itself is fetched. In the common case fetching the block is one disk seek, because the block index stores its physical offset. This is the ideal case, and what we want to happen as often as possible.

  If a table file handle is not open in the table cache, then in addition to the regular system calls to open the file, the block index and bloom filter need to be decoded before they can be checked. This is expensive and is something we want to avoid.

  The current setting of 64 file handles means that on a synced node, only about 4% of key lookups can be satisifed by table file handles that are actually open and in memory.

  The original concerns about file descriptor exhaustion are unwarranted on most systems because:
   * On 64-bit POSIX hosts LevelDB will open up to 1000 file descriptors using `mmap()`, and it does not retain an open file descriptor for such files.
   * On Windows non-socket files do not interfere with the main network `select()` loop, so the same fd exhaustion issues do not apply there.

  This change keeps the default `max_open_files` value (which is 1000) on all systems except 32-bit POSIX hosts (which do not use `mmap()`). Open file handles use about 20 KB of memory (for the block index), so the extra file handles do not cause much memory overhead. At most 1000 will be open, and a fully synced node right now has about 1500 such files.

  Profile of `loadblk` thread before changes: https://monad.io/maxopenfiles-master.svg
  Profile of `loadblk` thread after changes: https://monad.io/maxopenfiles-increase.svg

Tree-SHA512: de54f77d57e9f8999eaf8d12592aab5b02f5877be8fa727a1f42cf02da2693ce25846445eb19eb138ce4e5045d1c65e14054df72faf3ff32c7655c9cfadd27a9
2018-03-29 14:59:59 +02:00
.github Make default issue text all comments to make issues more readable 2017-11-16 11:50:56 -05:00
.tx tx: Update transifex slug for 0.16 2018-01-24 16:35:40 +01:00
build-aux/m4 ax_boost_{chrono,unit_test_framework}.m4: take changes from upstream 2018-03-15 19:59:11 +01:00
contrib contrib: Remove unused import string 2018-03-28 14:35:28 -04:00
depends Merge #12625: depends: biplist 1.0.3 2018-03-14 14:29:27 +01:00
doc Increase LevelDB max_open_files unless on 32-bit Unix. 2018-03-28 22:34:37 -07:00
share Change all python files to use Python3 2018-03-26 16:49:33 -04:00
src Increase LevelDB max_open_files unless on 32-bit Unix. 2018-03-28 22:34:37 -07:00
test Merge #12811: test: Make summary row bold-red if any test failed and show failed tests at end of table 2018-03-28 16:00:33 +02:00
.gitattributes Separate protocol versioning from clientversion 2014-10-29 00:24:40 -04:00
.gitignore [build] .gitignore: add QT Creator artifacts 2017-12-22 12:37:00 +01:00
.travis.yml [CI]: bump travis timeout for make check to 50m 2018-03-23 15:58:43 -04:00
autogen.sh Add MIT license to autogen.sh and share/genbuild.sh 2016-09-21 23:01:36 +00:00
configure.ac Remove redundant checks for MSG_* from configure.ac 2018-03-15 20:02:00 +01:00
CONTRIBUTING.md Docs: Improve documentation on standard communication channels 2018-03-22 12:58:57 -07:00
COPYING [Trivial] Update license year range to 2018 2018-01-01 04:33:09 +09:00
INSTALL.md Update INSTALL landing redirection notice for build instructions. 2016-10-06 12:27:23 +13:00
libbitcoinconsensus.pc.in Unify package name to as few places as possible without major changes 2015-12-14 02:11:10 +00:00
Makefile.am Build: Add a makefile target for Doxygen documentation 2018-01-25 19:43:19 +01:00
README.md Docs: Improve documentation on standard communication channels 2018-03-22 12:58:57 -07:00

README.md

Bitcoin Core integration/staging tree

Build Status

https://bitcoincore.org

What is Bitcoin?

Bitcoin is an experimental digital currency that enables instant payments to anyone, anywhere in the world. Bitcoin uses peer-to-peer technology to operate with no central authority: managing transactions and issuing money are carried out collectively by the network. Bitcoin Core is the name of open source software which enables the use of this currency.

For more information, as well as an immediately useable, binary version of the Bitcoin Core software, see https://bitcoin.org/en/download, or read the original whitepaper.

License

Bitcoin Core is released under the terms of the MIT license. See COPYING for more information or see https://opensource.org/licenses/MIT.

Development Process

The master branch is regularly built and tested, but is not guaranteed to be completely stable. Tags are created regularly to indicate new official, stable release versions of Bitcoin Core.

The contribution workflow is described in CONTRIBUTING.md.

Testing

Testing and code review is the bottleneck for development; we get more pull requests than we can review and test on short notice. Please be patient and help out by testing other people's pull requests, and remember this is a security-critical project where any mistake might cost people lots of money.

Automated Testing

Developers are strongly encouraged to write unit tests for new code, and to submit new unit tests for old code. Unit tests can be compiled and run (assuming they weren't disabled in configure) with: make check. Further details on running and extending unit tests can be found in /src/test/README.md.

There are also regression and integration tests, written in Python, that are run automatically on the build server. These tests can be run (if the test dependencies are installed) with: test/functional/test_runner.py

The Travis CI system makes sure that every pull request is built for Windows, Linux, and OS X, and that unit/sanity tests are run automatically.

Manual Quality Assurance (QA) Testing

Changes should be tested by somebody other than the developer who wrote the code. This is especially important for large or high-risk changes. It is useful to add a test plan to the pull request description if testing the changes is not straightforward.

Translations

Changes to translations as well as new translations can be submitted to Bitcoin Core's Transifex page.

Translations are periodically pulled from Transifex and merged into the git repository. See the translation process for details on how this works.

Important: We do not accept translation changes as GitHub pull requests because the next pull from Transifex would automatically overwrite them again.

Translators should also subscribe to the mailing list.