Commit graph

20990 commits

Author SHA1 Message Date
Andrew Chow
c88e87c3b2 Remove nFileVersion from CWalletScanState
nFileVersion is not the actual file version and is not used except
in one place. So it is removed from CWalletScanState and changed so
that it is just read at the place it is needed. Furthermore, the
"version" record now only indicates the version of the highest
versioned client that has opened a wallet file so the variable
name is changed accordingly
2019-07-22 13:02:03 -04:00
Daniel Kraft
29ee4c417d Specify AM_CPPFLAGS for ZMQ.
When building the ZMQ static library, add AM_CPPFLAGS to the library
CPPFLAGS.  Otherwise, we may miss important flags that are specified
elsewhere.  For instance, if --enable-debug is passed and
-DDEBUG_LOCKORDER set, then that would not apply to the ZMQ library
before (causing potential for hard-to-find bugs).
2019-07-22 14:26:18 +02:00
MarcoFalke
0000ff0aa7
txmempool: Remove unused default value MemPoolRemovalReason::UNKNOWN 2019-07-22 07:40:24 -04:00
James O'Beirne
4f050b91c7 move-onlyish: move CCoinsViewErrorCatcher out of init.cpp
and into coins.cpp. This move is necessary so that we can later include a
CCoinsViewErrorCatcher instance under CChainState.

Co-authored-by: MarcoFalke <falke.marco@gmail.com>
2019-07-21 21:00:31 -04:00
MarcoFalke
fa56b21c74
doc: Update bips 35, 37 and 111 status 2019-07-21 10:21:40 -04:00
MarcoFalke
fabfcb5d8e
build: Treat -Wswitch as error when --enable-werror 2019-07-19 15:40:48 -04:00
Fabian Jahr
e967cae8fa Use switch on status in RpcWallet 2019-07-19 14:34:53 -04:00
Fabian Jahr
ba1f128d6c Return error for ignored passphrase through disable private keys option 2019-07-19 14:34:33 -04:00
Wladimir J. van der Laan
51a6e2c419
Merge #15681: [mempool] Allow one extra single-ancestor transaction per package
50cede3f5a [mempool] Allow one extra single-ancestor transaction per package (Matt Corallo)

Pull request description:

  This implements the proposed policy change from [1], which allows
  certain classes of contract protocols involving revocation
  punishments to use CPFP. Note that some such use-cases may still
  want some form of one-deep package relay, though even this alone
  may greatly simplify some lightning fee negotiation.

  [1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-November/016518.html

ACKs for top commit:
  ajtowns:
    ACK 50cede3f5a -- looked over code again, compared with previous commit, compiles, etc.
  sdaftuar:
    ACK 50cede3f5a
  ryanofsky:
    utACK 50cede3f5a. Changes since last review: adding EXTRA_DESCENDANT_TX_SIZE_LIMIT constant, changing max ancestor size from 1,000,000 to nLimitAncestorSize constant (101,000), fixing test comment and getting rid of unused test node.

Tree-SHA512: b052c2a0f384855572b4579310131897b612201214b5abbb225167224e4f550049e300b471dbf320928652571e92ca2d650050b7cf39ac92b3bc1d2bcd386c1c
2019-07-19 20:00:12 +02:00
Wladimir J. van der Laan
f4b1fe7165
Merge #16412: net: Make poll in InterruptibleRecv only filter for POLLIN events.
a52818cc56 net: Make poll in InterruptibleRecv only filter for POLLIN events. poll should block until there is data to be read or the timeout expires. (tecnovert)

Pull request description:

  poll should block until there is data to be read or the timeout expires.

  Filtering for the POLLOUT event causes poll to return immediately which leads to high CPU usage when trying to connect to non-responding peers through tor.

  When USE_POLL is not defined select is used with the writefds parameter set to nullptr.
  Removing POLLOUT causes the behavior of poll to match that of select.

  Fixes: #16004.

ACKs for top commit:
  laanwj:
    code review ACK a52818cc56
  jonasschnelli:
    utACK a52818cc56

Tree-SHA512: 69934cc14e3327c7ff7f6c5942af8761e865220b2540d74ea1e176adad326307a73860417dddfd32d601b5c0e9e2ada1848bd7e3d27b0b7a9b42f11129af8eb1
2019-07-19 17:20:23 +02:00
MarcoFalke
c7b7cf299a
Merge #16422: test: remove redundant setup in addrman_tests
5c3c24cf9e test: remove redundant setup in addrman_tests (zenosage)

Pull request description:

  #10765 make this default behavior. No reason to keep these line.

Top commit has no ACKs.

Tree-SHA512: 545eea9c2d0741a75708f288f2c8752534ecaa6d54a9d014ef9afa295b0d075007704b64809eec090023703f47753e8ec755d22c9ccecf57b75f6898f6b708dd
2019-07-19 10:18:23 -04:00
fanquake
59ce537a49
Merge #16152: Disable bloom filtering by default.
bead32e31e Add release notes for DEFAULT_BLOOM change (Matt Corallo)
f27309f55c Move DEFAULT_PEERBLOOMFILTERS from validation.h to net_processing.h (Matt Corallo)
5efcb77283 Disable bloom filtering by default. (Matt Corallo)

Pull request description:

  BIP 37 bloom filters have been well-known to be a significant DoS
  target for some time. However, in order to provide continuity for
  SPV clients relying on it, the NODE_BLOOM service flag was added,
  and left as a default, to ensure sufficient nodes exist with such a
  flag.

  NODE_BLOOM is, at this point, well-established and, as long as
  there exist 0.18 nodes with default config (which I'd anticipate
  will be true for many years), will be available from some peers. By
  that time, the continued slowdown of BIP 37-based filtering will
  likely have rendered it useless (though this is already largely the
  case). Further, BIP 37 was deliberately never updated to support
  witness-based filtering as newer wallets are expected to migrate to
  some yet-to-be-network-exposed filters.

ACKs for top commit:
  jnewbery:
    ACK bead32e31e
  kallewoof:
    ACK bead32e31e

Tree-SHA512: ecd901898e8efe1a7c82b471af0acc2373c2282ac633eb58d9aae7c35deda1999d0f79fb0485e6cecbda7246aeda00206cd82c7fa36866e2ac64705ba93f9390
2019-07-19 17:33:56 +08:00
fanquake
89d7229c9c
Merge #16420: QA: Fix race condition in wallet_encryption test
024ecd7e01 QA: Fix race condition in wallet_encryption test (Jonas Schnelli)

Pull request description:

  There is some imprecision probably in the internal HTTPRPCTimer class (haven't exactly figured out where).
  But we can't expect that waiting excatly 2 seconds right after calling `walletpassphrase(2)` will result in a locked wallet due to the nature how we internally handle threads/timers.

  The wallet_encryption test fails regularely in CIs.

  Here is a logged session:
  ```shell
   node0 2019-07-18T18:51:22.569739Z [] ThreadRPCServer method=walletpassphrase user=__cookie__ 
   node0 2019-07-18T18:51:22.628656Z [] queue run of timer lockwallet() in 2 seconds (using HTTP) 
   node0 2019-07-18T18:51:22.629002Z [] Received a POST request for / from 127.0.0.1:46898 
   node0 2019-07-18T18:51:22.629081Z [] ThreadRPCServer method=dumpprivkey user=__cookie__ 
   node0 2019-07-18T18:51:24.445620Z [] Flushing wallet.dat 
   node0 2019-07-18T18:51:24.451421Z [] Flushed wallet.dat 6ms 
   node0 2019-07-18T18:51:24.631703Z [] Received a POST request for / from 127.0.0.1:46898 
   node0 2019-07-18T18:51:24.631737Z [] ThreadRPCServer method=dumpprivkey user=__cookie__ 
   test  2019-07-18T18:51:24.632000Z TestFramework (ERROR): Assertion failed 
                                     Traceback (most recent call last):
                                       File "/home/ubuntu/src/test/functional/test_framework/test_framework.py", line 193, in main
                                         self.run_test()
                                       File "/home/ubuntu/src/test/functional/wallet_encryption.py", line 53, in run_test
                                         assert_raises_rpc_error(-13, "Please enter the wallet passphrase with walletpassphrase first", self.nodes[0].dumpprivkey, address)
  ```

ACKs for top commit:
  promag:
    ACK 024ecd7e01, simple fix, one second shouldn't hurt.
  MarcoFalke:
    ACK 024ecd7e01
  fanquake:
    ACK 024ecd7e01

Tree-SHA512: 0cda1b8969b084bb765d2b35e90a8611c565ee458a7be1f2dde675f8ddbd9b9e421514547a7683f836e2c996e0538eb66b8c5b935b5a81e9319fb2be27624374
2019-07-19 09:06:49 +08:00
Andrew Chow
c6a8274247 Have importprivkey use CWallet's ImportPrivKeys, ImportScripts, and ImportScriptPubKeys
Behavior changes:
* If we already have the key, it's wpkh script will still be added, although it should already be there
2019-07-18 20:35:51 -04:00
Andrew Chow
fae7a5befd Log when an import is being skipped because we already have it
Behavior Changes:
* Those pubkeys being imported with add_keypool set and are already in the wallet will no longer be added to the keypool
2019-07-18 20:34:53 -04:00
zenosage
5c3c24cf9e test: remove redundant setup in addrman_tests 2019-07-18 17:31:46 -07:00
Carl Dong
0c55d8b581
depends: qt: Patch to remove dep on libX11
We can actually patch QT to remove its dependency on libX11's headers.
It turns it this wasn't that hard.
2019-07-18 17:43:06 -04:00
Carl Dong
222e6cc520
gitignore: Actually pay attention to depends patches
There was a previous attempt to achieve this, and it was bad. This
works.
2019-07-18 17:41:50 -04:00
Matt Corallo
bead32e31e Add release notes for DEFAULT_BLOOM change 2019-07-18 17:29:24 -04:00
Jonas Schnelli
024ecd7e01
QA: Fix race condition in wallet_encryption test 2019-07-18 22:31:24 +02:00
Russell Yanofsky
4d94916f0d Get rid of PendingWalletTx class.
No reason for this class to exist if it doesn't have any code to run in the
destructor. e10e1e8db0 from
https://github.com/bitcoin/bitcoin/pull/16208 recently removed code destructor
code that would return an unused key if the transaction wasn't committed.
2019-07-18 12:06:23 -04:00
Wladimir J. van der Laan
e5abb59a9a
Merge #16379: Fix autostart filenames on Linux for testnet/regtest
ae311bc036 Fix autostart filenames on Linux (Hennadii Stepanov)

Pull request description:

  Currently, on master the `bitcoin-test.lnk` and `bitcoin-regtest.lnk` files do not work as autostart application `.desktop` files.

  This PR fixes it.

  Refs:
  - #7045
  - [Autostart Of Applications During Startup](https://standards.freedesktop.org/autostart-spec/autostart-spec-latest.html)

ACKs for top commit:
  promag:
    utACK ae311bc, weird why extension `.lnk` was used in #7045.
  laanwj:
    Code review ACK ae311bc036

Tree-SHA512: 210cc346600d52b0a262c81ed5f258365a3cea2e5522f4b5f4798fd3b54f45ed82aba68eefae59a6b6f1d8e4d00221476c23bdffc038f16f2f45c1acc837f522
2019-07-18 14:27:25 +02:00
Wladimir J. van der Laan
65d12110d4
Merge #16405: fix: tor: Call event_base_loopbreak from the event's callback
a981e749e6 fix: tor: Call event_base_loopbreak from the event's callback (João Barbosa)

Pull request description:

  Calling `event_base_loopbreak` before `event_base_dispatch` has no effect. Fix this by calling `event_base_loopbreak` from the event's callback. From the [documentation](http://www.wangafu.net/~nickm/libevent-2.0/doxygen/html/event_8h.html#a07a7599e478e4031fa8cf52e26d8aa1e):

  > event_base_loop() will abort the loop after the next event is completed; event_base_loopbreak() is typically invoked from this event's callback. This behavior is analogous to the "break;" statement.

  This can be tested by running the following with and without this change:
  ```sh
  bitcoind -- -regtest -proxy=127.0.0.1:9050 -listen=1 -bind=127.0.0.1 -whitebind=127.0.0.1:0
  ```

  Fixes #16376.

ACKs for top commit:
  laanwj:
    code review ACK a981e749e6
  fanquake:
    ACK a981e749e6

Tree-SHA512: 328fe71366404d5be8177d7081d5b4868cee73412df631a1865d24fb1c153427210762738109e06b737f037f4c68966812fba041831bb9e8129861f19ce61a63
2019-07-18 13:58:12 +02:00
tecnovert
a52818cc56
net: Make poll in InterruptibleRecv only filter for POLLIN events.
poll should block until there is data to be read or the timeout expires.

Filtering for the POLLOUT event causes poll to return immediately which leads to high CPU usage when trying to connect to non-responding peers through tor.

Removing POLLOUT matches how select is used when USE_POLL isn't defined.
2019-07-18 13:04:16 +02:00
fanquake
0515406acb
Merge #16374: test: Enable passing wildcard test names to test runner from root
e142ee03e7 doc: describe how to pass wildcard names to test runner (Jon Atack)
6a7a70b8cf test: enable passing wildcards with path to test runner (Jon Atack)

Pull request description:

  Currently, passing wildcard testname args to the test runner from outside the test/functional/ directory does not work, even though developers expect it to. See these recent IRC discussions for more background: http://www.erisian.com.au/bitcoin-core-dev/log-2019-07-10.html#l-262 (lines 262 to 323) and http://www.erisian.com.au/bitcoin-core-dev/log-2019-07-11.html#l-134.

  1. [BUGFIX] Enable passing wildcards with paths. Examples:
      - `test/functional/test_runner.py test/functional/wallet*`
      - `functional/test_runner.py functional/wallet*`
      - `test/functional/test_runner.py ./test/functional/tool* test/functional/mempool*`
      - A current limitation this PR does not change: 9 test files with arguments in their filename are not picked up by wildcard search.

  2. [Docs] Describe how to pass wildcard names (multiple and with paths) to the test runner in test/README.md.

ACKs for top commit:
  jnewbery:
    tested ACK e142ee03e7
  jachiang:
    Tested ACK e142ee03e7. Thanks a lot for this fix!
  MarcoFalke:
    ACK e142ee03e7, fine with me

Tree-SHA512: cb3d994880cdc9b8918546b573a25faa5b4c7339826ac7cfe20f076aac6e731a34271609c0cf5a7ee5e4a2d5ae205298319d24bf36ef5b5d569a1a0c57883e54
2019-07-18 10:05:08 +08:00
Carl Dong
65f8da08df
symbol-check: Disallow libX11-*.so.* shared libraries
They should no longer be needed as we build QT without libX11/XLib
libraries now.
2019-07-17 17:09:48 -04:00
Carl Dong
924569914e
depends: libXext isn't needed by anyone
libXext was only needed (as a library) by QT when it was using
XLib/libX11 (as a library), now that we're building QT without
XLib/libX11, we can safely remove libXext.
2019-07-17 17:04:42 -04:00
Carl Dong
689d3b4a03
build-aux: Remove check for x11-xcb
We're no longer building QT with libX11/XLib, so it doesn't make sense
to check for the x11-xcb package.
2019-07-17 17:04:41 -04:00
Carl Dong
aa53cb7a2f
depends: libX11: Make package headers-only
We're no longer building QT with libX11/XLib, however, libX11/XLib
headers are still required for parts of QT. In this commit we add a
minimal configure.ac for libX11/XLib that is headers-only.

This change allows us to remove all of libX11/XLib's dependencies.
2019-07-17 17:04:40 -04:00
Carl Dong
9a01ab04e1
depends: qt: Explicitly stop using Xlib/libX11
Previously, in 683b7d7a3f and
0e752637a2, we accidentally broke QT's
ability to pick up Xlib thru the config.gui.tests.xlib configuration
test, which also means that config.gui.libraries.xcb_xlib wasn't run.

This resulted in a QT build that was implicitly -no-xcb-lib and
-no-feature-xlib.

This is actually a desired behaviour, as it means less required shared
objects for our final bitcoin-qt binary. Specifically, it eliminated the
libX11-xcb.so.1 and libX11.so.6 requirements.

In this commit, we explicitly build without Xlib. We should continue to
track upstream ticket https://bugreports.qt.io/browse/QTBUG-61452 which
talks about adding a -no-xlib (non-hidden) flag instead of the
-no-feature-xlib (hidden) flag.
2019-07-17 17:04:39 -04:00
Carl Dong
1ec30b8fbe
depends: xproto is only directly needed by libXau 2019-07-17 17:04:38 -04:00
João Barbosa
a981e749e6 fix: tor: Call event_base_loopbreak from the event's callback 2019-07-17 15:32:38 +01:00
MeshCollider
459baa1756
Merge #16208: wallet: Consume ReserveDestination on successful CreateTransaction
e10e1e8db0 Restrict lifetime of ReserveDestination to CWallet::CreateTransaction (Gregory Sanders)
d9ff862f2d CreateTransaction calls KeepDestination on ReserveDestination before success (Gregory Sanders)

Pull request description:

  The typical usage pattern of `ReserveDestination` is to explicitly `KeepDestination`, or `ReturnDestination` when it's detected it will not be used.

  Implementers such as myself may fail to complete this pattern, and could result in key re-use: https://github.com/bitcoin/bitcoin/pull/15557#discussion_r271956393

  Since ReserveDestination is currently only used directly in the `CreateTransaction`/`CommitTransaction` flow(or fee bumping where it's just used in `CreateTransaction`), I instead make the assumption that if a transaction is returned by `CreateTransaction` it's highly likely that it will be accepted by the caller, and the `ReserveDestination` kept. This simplifies the API as well. There are very few cases where this would not be the case which may result in keys being burned.

  Those failure cases appear to be:
  `CommitTransaction` failing to get the transaction into the mempool
  Belt and suspenders check in `WalletModel::prepareTransaction`

  Alternative to https://github.com/bitcoin/bitcoin/pull/15796

ACKs for top commit:
  achow101:
    ACK e10e1e8db0 Reviewed the diff
  stevenroose:
    utACK e10e1e8db0
  meshcollider:
    utACK e10e1e8db0

Tree-SHA512: 78d047a00f39ab41cfa297052cc1e9c224d5f47d3d2299face650d71827635de077ac33fb4ab9f7dc6fc5a27f4a68415a1bc9ca33a3cb09a78f4f15b2a48411b
2019-07-17 19:45:55 +12:00
Fabian Jahr
d6649d16b5 Use strong enum for WalletCreationStatus 2019-07-16 17:33:22 -04:00
Fabian Jahr
3199610ad3 Place out args at the end for CreateWallet 2019-07-16 17:27:50 -04:00
MarcoFalke
fa4a605a4c
Remove wallet settings from chainparams 2019-07-16 16:22:14 -04:00
MarcoFalke
24dbcf3808
Merge #15891: test: Require standard txs in regtest by default
fa89badf88 test: Require standard txs in regtest (MarcoFalke)
fa9b419160 test: Add test that mainnet requires standard txs (MarcoFalke)
fa613ca0a8 chainparams: Remove unused fMineBlocksOnDemand (MarcoFalke)

Pull request description:

  I don't see a reason why regtest should allow non-standard txs, as it makes testing mainnet behaviour such as #15846 unnecessarily hard and unintuitive.

  Of course, testnet policy remains unchanged to allow propagation of non-standard txs.

ACKs for top commit:
  ajtowns:
    ACK fa89badf88

Tree-SHA512: c4c675affb054868850bd2683aa07f4c741a448cbacb2ea8334191e105f426b0790fe6a468be61e9c5880d24154f7bf1c7075051697172dce92180c1bc3a1c90
2019-07-16 16:10:17 -04:00
Wladimir J. van der Laan
8f604361eb
Merge #16194: refactor: share blockmetadata with BlockManager
682a1d0f20 refactoring: remove mapBlockIndex global (James O'Beirne)
55d525ab90 refactoring: make pindexBestInvalid internal to validation.cpp (James O'Beirne)
4ed55dfcd7 refactoring: add block_index_candidates arg to LoadBlockIndex (James O'Beirne)
613c46fe9e refactoring: move block metadata structures into BlockManager (James O'Beirne)

Pull request description:

  This is part of the [assumeutxo project](https://github.com/bitcoin/bitcoin/projects/11):

  Parent PR: #15606
  Issue: #15605
  Specification: https://github.com/jamesob/assumeutxo-docs/tree/2019-04-proposal/proposal

  ---

  Under an assumeutxo model, we have multiple CChainState instances in use at once in order to support background validation. Currently, each CChainState instance has its own mapBlockIndex, a collection of linked block headers, in addition to a few other data structures that are related to maintenance of the block tree but not necessarily to any given chainstate.

  In order to avoid duplicating this data across chainstates, this change moves chainstate-agnostic block metadata (and related behavior) into a class, `BlockManager`. Chainstates are parameterized with a reference to a blockmanager instance and in practice they share the same instance.

  Most of this change is conceptually move-only, though the diff is somewhat muddled. The first commit can be reviewed slightly more easily with `--color-moved=dimmed_zebra`. Admittedly, that commit is pretty unwieldy; I tried to split it up after the fact with `git add --patch`, but that was difficult because of git's inability to split hunks past a certain point. Some of the moves also ended up being obscured when done over separate commits.

ACKs for top commit:
  MarcoFalke:
    ACK 682a1d0f20
  ryanofsky:
    utACK 682a1d0f20, only changes since last review were rebase and fixing conflict on a moved line
  ariard:
    utACK 682a1d0. Most of the changes are move-only, with main problem being to avoid creating circular dependencies between `BlockManager` and `CChainState`. Tested, comments are mostly nits, feel free to ignore them

Tree-SHA512: 738d8d06539ba53acf4bd2d48ae000473e645bbc4e63d798d55d247a4d5a4f781b73538ed590f6407be9ab402ea9d395570ea20bff0a4b9ce747bcc1600c5108
2019-07-16 18:48:07 +02:00
MarcoFalke
8f9725c83f
Merge #16390: qa: Add --filter option to test_runner.py
1a62425260 qa: Add --filter option to test_runner.py (João Barbosa)

Pull request description:

  Allows to run functional tests like:
  ```sh
  test/functional/test_runner.py --filter wallet
  ```

ACKs for top commit:
  jonatack:
    ACK 1a62425260

Tree-SHA512: 53199e01da3b2e0112843c1c68c69d8fd7fc9bb6a6cb45a81c324973c4824ebf5fef574f9efab81a64d52e397e25d979ae40f0eaba35afb771e80012768f0b08
2019-07-16 09:01:34 -04:00
fanquake
29082e8f40
Merge #16380: Remove unused bits from the service flags enum
fa0d0ff6e1 Remove unused bits from the service flags enum (MarcoFalke)

Pull request description:

  Remove all bits that have no BIP specification nor can be observed on the active network

ACKs for top commit:
  practicalswift:
    utACK fa0d0ff6e1
  LarryRuane:
    utACK fa0d0ff6e1
  promag:
    ACK fa0d0ff6e1.
  laanwj:
    ACK fa0d0ff6e1

Tree-SHA512: 6342017bfd4c2a39c998fbb02497931b11892e1cb60fc13b948b91812f281b605a25a3fdc0d5358dff18da4e82eb4eb4de95c43c7e76ecb331c1c3985443dd21
2019-07-16 10:02:04 +08:00
João Barbosa
1a62425260 qa: Add --filter option to test_runner.py 2019-07-16 01:00:27 +01:00
Wladimir J. van der Laan
6d37ed888e
Merge #15824: docs: Improve netbase comments
c7f6ce74d3 docs: Improve netbase comments (Carl Dong)

Pull request description:

  Second in a series of PRs documenting the net stack. Contributed with sincere thanks to sipa, laanwj, and gmaxwell for providing much of the history, context, and rationale.

ACKs for top commit:
  laanwj:
    ACK c7f6ce74d3

Tree-SHA512: ad83054d3b8d0c8c3fb55be011bcf294176e7509513bf61326866afd53e8159644e0d59bb3a2f404717f525cbf736096d4c1990e61cfd89845d51fa6b5394b7c
2019-07-15 21:25:09 +02:00
Carl Dong
c7f6ce74d3
docs: Improve netbase comments
- Improve and add various Lookup* docs
- Improve InterruptibleRecv docs
- Improve Socks5 docs
- Add CreateSocket docs
- Add ConnectSocketDirectly docs
- Add SetNameProxy docs
- Add ConnectThroughProxy docs
- Add LookupSubNet docs
2019-07-15 14:46:15 -04:00
MarcoFalke
0822b44d8a
Merge #15282: test: Replace hard-coded hex tx with class in test framework
8f250ab788 TEST: Replace hard-coded hex tx with classes (Steven Roose)

Pull request description:

  Came across these breaking Elements.

ACKs for top commit:
  MarcoFalke:
    ACK 8f250ab788
  instagibbs:
    utACK 8f250ab788

Tree-SHA512: e8615dad4cda0beea4b0c7d4951a467fb9882a0a64d49c9b5ecf167369ea62a3fe5348e2401153162b0ccadecdb128492c94be36ebb881c3c42659626d86eda8
2019-07-15 08:02:07 -04:00
Jon Atack
e142ee03e7
doc: describe how to pass wildcard names to test runner 2019-07-15 10:14:23 +02:00
Jon Atack
6a7a70b8cf
test: enable passing wildcards with path to test runner
Currently, passing wildcard testname args to the test runner from outside the `test/functional/` directory does not work. See this recent IRC discussion for more background: http://www.erisian.com.au/bitcoin-core-dev/log-2019-07-10.html#l-262 (lines 262 to 323).

This small change enables passing multiple wildcards with paths, as long as the paths are coherent. Examples:
  - test/functional/test_runner.py test/functional/wallet*
  - functional/test_runner.py functional/wallet*
  - test/functional/test_runner.py ./test/functional/tool* test/functional/mempool*

A current limitation that this PR does not change: 9 test files with arguments in their name are not picked up by wildcard search.

- Squashed commit: non-mutating version

- Squashed commit: minor code optimisation
2019-07-15 10:13:39 +02:00
Wladimir J. van der Laan
536590f358
Merge #16334: test: rpc_users: Also test rpcauth.py with password.
e263a343d4 test: rpc_users: Make variable names more clear. (Carl Dong)
830dc2dd0f test: rpc_users: Also test rpcauth.py with specified password. (Carl Dong)
c73d871799 test: rpc_users: Add function for testing auth params. (Carl Dong)
604e2a997f test: rpc_users: Add function for auth'd requests. (Carl Dong)

Pull request description:

  Fixes #14758

  First two commits are tidy-ups which I feel are worthwhile as they are very straightforward, cut down the file by 50%, and made the final diff more minimal. Happy to squash after review.

ACKs for top commit:
  laanwj:
    ACK e263a343d4

Tree-SHA512: aa75c48570a87060238932d4c68e17234e158077f6195fb4917367e1ecc565e3cd8dd0ae51f9159ddd3d03742739680391bc1246454302db22d4a608c0633e80
2019-07-12 21:28:47 +02:00
MarcoFalke
fa0d0ff6e1
Remove unused bits from the service flags enum 2019-07-12 14:14:54 -04:00
Wladimir J. van der Laan
3453cf26db
Merge #15277: contrib: Enable building in Guix containers
751549b52a contrib: guix: Additional clarifications re: substitutes (Carl Dong)
cd3e947f50 contrib: guix: Various improvements. (Carl Dong)
8dff3e48a9 contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong)
3e80ec3ea9 contrib: Add deterministic Guix builds. (Carl Dong)

Pull request description:

  ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~

  Please read the `README.md`.

  -----

  ### Guix Introduction

  This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin.

  ### Guix Build Walkthrough

  Please read the `README.md`.

  [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718)

  [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011)

  [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)

  <details>
  <summary>Old instructions no. 1</summary>
  In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment.

  We can then invoke
  ```
  guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
  ```
  To have Guix:
  1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details).
  2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in.
  3. Drop you into a shell in that container.

  > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>`

  Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following:

  ```
  make -C depends -j"$(nproc)" download && \
      cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
  ```

  We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing).
  </details>

  ### Rationale

  I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because:

  1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](72bd4ab867/contrib/gitian-descriptors/gitian-linux.yml (L10))), because Guix builds everything about the container, we can perform this on almost any Linux distro/system.
  2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)).
  3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler).

  ### Steps to Completion

  - [x] Successfully build bitcoin inside the Guix environment
  - [x] Make `check-symbols` pass
  - [x] Do the above but without nasty hacks
  - [x] Solve some of the more innocuous hacks
  - [ ] Make it cross-compile (HELP WANTED HERE)
    - [x] Linux
      - [x] x86_64-linux-gnu
      - [x] i686-linux-gnu
      - [x] aarch64-linux-gnu
      - [x] arm-linux-gnueabihf
      - [x] riscv64-linux-gnu
    - [ ] OS X
      - [ ] x86_64-apple-darwin14
    - [ ] Windows
      - [ ] x86_64-w64-mingw32
  - [ ] Maybe make importer for depends syntax
  - [ ] Document build process for future releases
  - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html)

  ### Help Wanted

  [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210)

  [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)

  <details>
  <summary>Old content no. 1</summary>
  As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be.

  Here's what `ldd src/bitcoind` looks like when built in a Guix container:
  ```
  	linux-vdso.so.1 (0x00007ffcc2d90000)
  	libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000)
  	librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000)
  	libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000)
  	libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000)
  	libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000)
  	libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000)
  	libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000)
  	/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000)
  ```

  And here's what it looks in one of our releases:
  ```
  	linux-vdso.so.1 (0x00007ffff52cd000)
  	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000)
  	librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000)
  	libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000)
  	libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000)
  	libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000)
  	/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000)
  ```

  ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely.

  Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](d6e700e40f/contrib/gitian-descriptors/gitian-win-signer.yml (L14))
  </details>

  ### How to Help

  1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation)
  2. Try out my branch and the command described above!

ACKs for top commit:
  MarcoFalke:
    Thanks for the replies. ACK 751549b52a
  laanwj:
    ACK 751549b52a

Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:24:45 +02:00
Carl Dong
751549b52a
contrib: guix: Additional clarifications re: substitutes 2019-07-12 12:31:55 -04:00