Commit graph

20800 commits

Author SHA1 Message Date
Hennadii Stepanov 4057b7acb7
wallet: Recognize -disablewallet option early 2019-07-23 15:38:10 +03:00
Wladimir J. van der Laan 51a6e2c419
Merge #15681: [mempool] Allow one extra single-ancestor transaction per package
50cede3f5a [mempool] Allow one extra single-ancestor transaction per package (Matt Corallo)

Pull request description:

  This implements the proposed policy change from [1], which allows
  certain classes of contract protocols involving revocation
  punishments to use CPFP. Note that some such use-cases may still
  want some form of one-deep package relay, though even this alone
  may greatly simplify some lightning fee negotiation.

  [1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-November/016518.html

ACKs for top commit:
  ajtowns:
    ACK 50cede3f5a -- looked over code again, compared with previous commit, compiles, etc.
  sdaftuar:
    ACK 50cede3f5a
  ryanofsky:
    utACK 50cede3f5a. Changes since last review: adding EXTRA_DESCENDANT_TX_SIZE_LIMIT constant, changing max ancestor size from 1,000,000 to nLimitAncestorSize constant (101,000), fixing test comment and getting rid of unused test node.

Tree-SHA512: b052c2a0f384855572b4579310131897b612201214b5abbb225167224e4f550049e300b471dbf320928652571e92ca2d650050b7cf39ac92b3bc1d2bcd386c1c
2019-07-19 20:00:12 +02:00
Wladimir J. van der Laan f4b1fe7165
Merge #16412: net: Make poll in InterruptibleRecv only filter for POLLIN events.
a52818cc56 net: Make poll in InterruptibleRecv only filter for POLLIN events. poll should block until there is data to be read or the timeout expires. (tecnovert)

Pull request description:

  poll should block until there is data to be read or the timeout expires.

  Filtering for the POLLOUT event causes poll to return immediately which leads to high CPU usage when trying to connect to non-responding peers through tor.

  When USE_POLL is not defined select is used with the writefds parameter set to nullptr.
  Removing POLLOUT causes the behavior of poll to match that of select.

  Fixes: #16004.

ACKs for top commit:
  laanwj:
    code review ACK a52818cc56
  jonasschnelli:
    utACK a52818cc56

Tree-SHA512: 69934cc14e3327c7ff7f6c5942af8761e865220b2540d74ea1e176adad326307a73860417dddfd32d601b5c0e9e2ada1848bd7e3d27b0b7a9b42f11129af8eb1
2019-07-19 17:20:23 +02:00
MarcoFalke c7b7cf299a
Merge #16422: test: remove redundant setup in addrman_tests
5c3c24cf9e test: remove redundant setup in addrman_tests (zenosage)

Pull request description:

  #10765 make this default behavior. No reason to keep these line.

Top commit has no ACKs.

Tree-SHA512: 545eea9c2d0741a75708f288f2c8752534ecaa6d54a9d014ef9afa295b0d075007704b64809eec090023703f47753e8ec755d22c9ccecf57b75f6898f6b708dd
2019-07-19 10:18:23 -04:00
fanquake 59ce537a49
Merge #16152: Disable bloom filtering by default.
bead32e31e Add release notes for DEFAULT_BLOOM change (Matt Corallo)
f27309f55c Move DEFAULT_PEERBLOOMFILTERS from validation.h to net_processing.h (Matt Corallo)
5efcb77283 Disable bloom filtering by default. (Matt Corallo)

Pull request description:

  BIP 37 bloom filters have been well-known to be a significant DoS
  target for some time. However, in order to provide continuity for
  SPV clients relying on it, the NODE_BLOOM service flag was added,
  and left as a default, to ensure sufficient nodes exist with such a
  flag.

  NODE_BLOOM is, at this point, well-established and, as long as
  there exist 0.18 nodes with default config (which I'd anticipate
  will be true for many years), will be available from some peers. By
  that time, the continued slowdown of BIP 37-based filtering will
  likely have rendered it useless (though this is already largely the
  case). Further, BIP 37 was deliberately never updated to support
  witness-based filtering as newer wallets are expected to migrate to
  some yet-to-be-network-exposed filters.

ACKs for top commit:
  jnewbery:
    ACK bead32e31e
  kallewoof:
    ACK bead32e31e

Tree-SHA512: ecd901898e8efe1a7c82b471af0acc2373c2282ac633eb58d9aae7c35deda1999d0f79fb0485e6cecbda7246aeda00206cd82c7fa36866e2ac64705ba93f9390
2019-07-19 17:33:56 +08:00
fanquake 89d7229c9c
Merge #16420: QA: Fix race condition in wallet_encryption test
024ecd7e01 QA: Fix race condition in wallet_encryption test (Jonas Schnelli)

Pull request description:

  There is some imprecision probably in the internal HTTPRPCTimer class (haven't exactly figured out where).
  But we can't expect that waiting excatly 2 seconds right after calling `walletpassphrase(2)` will result in a locked wallet due to the nature how we internally handle threads/timers.

  The wallet_encryption test fails regularely in CIs.

  Here is a logged session:
  ```shell
   node0 2019-07-18T18:51:22.569739Z [] ThreadRPCServer method=walletpassphrase user=__cookie__ 
   node0 2019-07-18T18:51:22.628656Z [] queue run of timer lockwallet() in 2 seconds (using HTTP) 
   node0 2019-07-18T18:51:22.629002Z [] Received a POST request for / from 127.0.0.1:46898 
   node0 2019-07-18T18:51:22.629081Z [] ThreadRPCServer method=dumpprivkey user=__cookie__ 
   node0 2019-07-18T18:51:24.445620Z [] Flushing wallet.dat 
   node0 2019-07-18T18:51:24.451421Z [] Flushed wallet.dat 6ms 
   node0 2019-07-18T18:51:24.631703Z [] Received a POST request for / from 127.0.0.1:46898 
   node0 2019-07-18T18:51:24.631737Z [] ThreadRPCServer method=dumpprivkey user=__cookie__ 
   test  2019-07-18T18:51:24.632000Z TestFramework (ERROR): Assertion failed 
                                     Traceback (most recent call last):
                                       File "/home/ubuntu/src/test/functional/test_framework/test_framework.py", line 193, in main
                                         self.run_test()
                                       File "/home/ubuntu/src/test/functional/wallet_encryption.py", line 53, in run_test
                                         assert_raises_rpc_error(-13, "Please enter the wallet passphrase with walletpassphrase first", self.nodes[0].dumpprivkey, address)
  ```

ACKs for top commit:
  promag:
    ACK 024ecd7e01, simple fix, one second shouldn't hurt.
  MarcoFalke:
    ACK 024ecd7e01
  fanquake:
    ACK 024ecd7e01

Tree-SHA512: 0cda1b8969b084bb765d2b35e90a8611c565ee458a7be1f2dde675f8ddbd9b9e421514547a7683f836e2c996e0538eb66b8c5b935b5a81e9319fb2be27624374
2019-07-19 09:06:49 +08:00
zenosage 5c3c24cf9e test: remove redundant setup in addrman_tests 2019-07-18 17:31:46 -07:00
Matt Corallo bead32e31e Add release notes for DEFAULT_BLOOM change 2019-07-18 17:29:24 -04:00
Jonas Schnelli 024ecd7e01
QA: Fix race condition in wallet_encryption test 2019-07-18 22:31:24 +02:00
Wladimir J. van der Laan e5abb59a9a
Merge #16379: Fix autostart filenames on Linux for testnet/regtest
ae311bc036 Fix autostart filenames on Linux (Hennadii Stepanov)

Pull request description:

  Currently, on master the `bitcoin-test.lnk` and `bitcoin-regtest.lnk` files do not work as autostart application `.desktop` files.

  This PR fixes it.

  Refs:
  - #7045
  - [Autostart Of Applications During Startup](https://standards.freedesktop.org/autostart-spec/autostart-spec-latest.html)

ACKs for top commit:
  promag:
    utACK ae311bc, weird why extension `.lnk` was used in #7045.
  laanwj:
    Code review ACK ae311bc036

Tree-SHA512: 210cc346600d52b0a262c81ed5f258365a3cea2e5522f4b5f4798fd3b54f45ed82aba68eefae59a6b6f1d8e4d00221476c23bdffc038f16f2f45c1acc837f522
2019-07-18 14:27:25 +02:00
Wladimir J. van der Laan 65d12110d4
Merge #16405: fix: tor: Call event_base_loopbreak from the event's callback
a981e749e6 fix: tor: Call event_base_loopbreak from the event's callback (João Barbosa)

Pull request description:

  Calling `event_base_loopbreak` before `event_base_dispatch` has no effect. Fix this by calling `event_base_loopbreak` from the event's callback. From the [documentation](http://www.wangafu.net/~nickm/libevent-2.0/doxygen/html/event_8h.html#a07a7599e478e4031fa8cf52e26d8aa1e):

  > event_base_loop() will abort the loop after the next event is completed; event_base_loopbreak() is typically invoked from this event's callback. This behavior is analogous to the "break;" statement.

  This can be tested by running the following with and without this change:
  ```sh
  bitcoind -- -regtest -proxy=127.0.0.1:9050 -listen=1 -bind=127.0.0.1 -whitebind=127.0.0.1:0
  ```

  Fixes #16376.

ACKs for top commit:
  laanwj:
    code review ACK a981e749e6
  fanquake:
    ACK a981e749e6

Tree-SHA512: 328fe71366404d5be8177d7081d5b4868cee73412df631a1865d24fb1c153427210762738109e06b737f037f4c68966812fba041831bb9e8129861f19ce61a63
2019-07-18 13:58:12 +02:00
tecnovert a52818cc56
net: Make poll in InterruptibleRecv only filter for POLLIN events.
poll should block until there is data to be read or the timeout expires.

Filtering for the POLLOUT event causes poll to return immediately which leads to high CPU usage when trying to connect to non-responding peers through tor.

Removing POLLOUT matches how select is used when USE_POLL isn't defined.
2019-07-18 13:04:16 +02:00
fanquake 0515406acb
Merge #16374: test: Enable passing wildcard test names to test runner from root
e142ee03e7 doc: describe how to pass wildcard names to test runner (Jon Atack)
6a7a70b8cf test: enable passing wildcards with path to test runner (Jon Atack)

Pull request description:

  Currently, passing wildcard testname args to the test runner from outside the test/functional/ directory does not work, even though developers expect it to. See these recent IRC discussions for more background: http://www.erisian.com.au/bitcoin-core-dev/log-2019-07-10.html#l-262 (lines 262 to 323) and http://www.erisian.com.au/bitcoin-core-dev/log-2019-07-11.html#l-134.

  1. [BUGFIX] Enable passing wildcards with paths. Examples:
      - `test/functional/test_runner.py test/functional/wallet*`
      - `functional/test_runner.py functional/wallet*`
      - `test/functional/test_runner.py ./test/functional/tool* test/functional/mempool*`
      - A current limitation this PR does not change: 9 test files with arguments in their filename are not picked up by wildcard search.

  2. [Docs] Describe how to pass wildcard names (multiple and with paths) to the test runner in test/README.md.

ACKs for top commit:
  jnewbery:
    tested ACK e142ee03e7
  jachiang:
    Tested ACK e142ee03e7. Thanks a lot for this fix!
  MarcoFalke:
    ACK e142ee03e7, fine with me

Tree-SHA512: cb3d994880cdc9b8918546b573a25faa5b4c7339826ac7cfe20f076aac6e731a34271609c0cf5a7ee5e4a2d5ae205298319d24bf36ef5b5d569a1a0c57883e54
2019-07-18 10:05:08 +08:00
João Barbosa a981e749e6 fix: tor: Call event_base_loopbreak from the event's callback 2019-07-17 15:32:38 +01:00
MeshCollider 459baa1756
Merge #16208: wallet: Consume ReserveDestination on successful CreateTransaction
e10e1e8db0 Restrict lifetime of ReserveDestination to CWallet::CreateTransaction (Gregory Sanders)
d9ff862f2d CreateTransaction calls KeepDestination on ReserveDestination before success (Gregory Sanders)

Pull request description:

  The typical usage pattern of `ReserveDestination` is to explicitly `KeepDestination`, or `ReturnDestination` when it's detected it will not be used.

  Implementers such as myself may fail to complete this pattern, and could result in key re-use: https://github.com/bitcoin/bitcoin/pull/15557#discussion_r271956393

  Since ReserveDestination is currently only used directly in the `CreateTransaction`/`CommitTransaction` flow(or fee bumping where it's just used in `CreateTransaction`), I instead make the assumption that if a transaction is returned by `CreateTransaction` it's highly likely that it will be accepted by the caller, and the `ReserveDestination` kept. This simplifies the API as well. There are very few cases where this would not be the case which may result in keys being burned.

  Those failure cases appear to be:
  `CommitTransaction` failing to get the transaction into the mempool
  Belt and suspenders check in `WalletModel::prepareTransaction`

  Alternative to https://github.com/bitcoin/bitcoin/pull/15796

ACKs for top commit:
  achow101:
    ACK e10e1e8db0 Reviewed the diff
  stevenroose:
    utACK e10e1e8db0
  meshcollider:
    utACK e10e1e8db0

Tree-SHA512: 78d047a00f39ab41cfa297052cc1e9c224d5f47d3d2299face650d71827635de077ac33fb4ab9f7dc6fc5a27f4a68415a1bc9ca33a3cb09a78f4f15b2a48411b
2019-07-17 19:45:55 +12:00
MarcoFalke 24dbcf3808
Merge #15891: test: Require standard txs in regtest by default
fa89badf88 test: Require standard txs in regtest (MarcoFalke)
fa9b419160 test: Add test that mainnet requires standard txs (MarcoFalke)
fa613ca0a8 chainparams: Remove unused fMineBlocksOnDemand (MarcoFalke)

Pull request description:

  I don't see a reason why regtest should allow non-standard txs, as it makes testing mainnet behaviour such as #15846 unnecessarily hard and unintuitive.

  Of course, testnet policy remains unchanged to allow propagation of non-standard txs.

ACKs for top commit:
  ajtowns:
    ACK fa89badf88

Tree-SHA512: c4c675affb054868850bd2683aa07f4c741a448cbacb2ea8334191e105f426b0790fe6a468be61e9c5880d24154f7bf1c7075051697172dce92180c1bc3a1c90
2019-07-16 16:10:17 -04:00
Wladimir J. van der Laan 8f604361eb
Merge #16194: refactor: share blockmetadata with BlockManager
682a1d0f20 refactoring: remove mapBlockIndex global (James O'Beirne)
55d525ab90 refactoring: make pindexBestInvalid internal to validation.cpp (James O'Beirne)
4ed55dfcd7 refactoring: add block_index_candidates arg to LoadBlockIndex (James O'Beirne)
613c46fe9e refactoring: move block metadata structures into BlockManager (James O'Beirne)

Pull request description:

  This is part of the [assumeutxo project](https://github.com/bitcoin/bitcoin/projects/11):

  Parent PR: #15606
  Issue: #15605
  Specification: https://github.com/jamesob/assumeutxo-docs/tree/2019-04-proposal/proposal

  ---

  Under an assumeutxo model, we have multiple CChainState instances in use at once in order to support background validation. Currently, each CChainState instance has its own mapBlockIndex, a collection of linked block headers, in addition to a few other data structures that are related to maintenance of the block tree but not necessarily to any given chainstate.

  In order to avoid duplicating this data across chainstates, this change moves chainstate-agnostic block metadata (and related behavior) into a class, `BlockManager`. Chainstates are parameterized with a reference to a blockmanager instance and in practice they share the same instance.

  Most of this change is conceptually move-only, though the diff is somewhat muddled. The first commit can be reviewed slightly more easily with `--color-moved=dimmed_zebra`. Admittedly, that commit is pretty unwieldy; I tried to split it up after the fact with `git add --patch`, but that was difficult because of git's inability to split hunks past a certain point. Some of the moves also ended up being obscured when done over separate commits.

ACKs for top commit:
  MarcoFalke:
    ACK 682a1d0f20
  ryanofsky:
    utACK 682a1d0f20, only changes since last review were rebase and fixing conflict on a moved line
  ariard:
    utACK 682a1d0. Most of the changes are move-only, with main problem being to avoid creating circular dependencies between `BlockManager` and `CChainState`. Tested, comments are mostly nits, feel free to ignore them

Tree-SHA512: 738d8d06539ba53acf4bd2d48ae000473e645bbc4e63d798d55d247a4d5a4f781b73538ed590f6407be9ab402ea9d395570ea20bff0a4b9ce747bcc1600c5108
2019-07-16 18:48:07 +02:00
MarcoFalke 8f9725c83f
Merge #16390: qa: Add --filter option to test_runner.py
1a62425260 qa: Add --filter option to test_runner.py (João Barbosa)

Pull request description:

  Allows to run functional tests like:
  ```sh
  test/functional/test_runner.py --filter wallet
  ```

ACKs for top commit:
  jonatack:
    ACK 1a62425260

Tree-SHA512: 53199e01da3b2e0112843c1c68c69d8fd7fc9bb6a6cb45a81c324973c4824ebf5fef574f9efab81a64d52e397e25d979ae40f0eaba35afb771e80012768f0b08
2019-07-16 09:01:34 -04:00
fanquake 29082e8f40
Merge #16380: Remove unused bits from the service flags enum
fa0d0ff6e1 Remove unused bits from the service flags enum (MarcoFalke)

Pull request description:

  Remove all bits that have no BIP specification nor can be observed on the active network

ACKs for top commit:
  practicalswift:
    utACK fa0d0ff6e1
  LarryRuane:
    utACK fa0d0ff6e1
  promag:
    ACK fa0d0ff6e1.
  laanwj:
    ACK fa0d0ff6e1

Tree-SHA512: 6342017bfd4c2a39c998fbb02497931b11892e1cb60fc13b948b91812f281b605a25a3fdc0d5358dff18da4e82eb4eb4de95c43c7e76ecb331c1c3985443dd21
2019-07-16 10:02:04 +08:00
João Barbosa 1a62425260 qa: Add --filter option to test_runner.py 2019-07-16 01:00:27 +01:00
Wladimir J. van der Laan 6d37ed888e
Merge #15824: docs: Improve netbase comments
c7f6ce74d3 docs: Improve netbase comments (Carl Dong)

Pull request description:

  Second in a series of PRs documenting the net stack. Contributed with sincere thanks to sipa, laanwj, and gmaxwell for providing much of the history, context, and rationale.

ACKs for top commit:
  laanwj:
    ACK c7f6ce74d3

Tree-SHA512: ad83054d3b8d0c8c3fb55be011bcf294176e7509513bf61326866afd53e8159644e0d59bb3a2f404717f525cbf736096d4c1990e61cfd89845d51fa6b5394b7c
2019-07-15 21:25:09 +02:00
Carl Dong c7f6ce74d3
docs: Improve netbase comments
- Improve and add various Lookup* docs
- Improve InterruptibleRecv docs
- Improve Socks5 docs
- Add CreateSocket docs
- Add ConnectSocketDirectly docs
- Add SetNameProxy docs
- Add ConnectThroughProxy docs
- Add LookupSubNet docs
2019-07-15 14:46:15 -04:00
MarcoFalke 0822b44d8a
Merge #15282: test: Replace hard-coded hex tx with class in test framework
8f250ab788 TEST: Replace hard-coded hex tx with classes (Steven Roose)

Pull request description:

  Came across these breaking Elements.

ACKs for top commit:
  MarcoFalke:
    ACK 8f250ab788
  instagibbs:
    utACK 8f250ab788

Tree-SHA512: e8615dad4cda0beea4b0c7d4951a467fb9882a0a64d49c9b5ecf167369ea62a3fe5348e2401153162b0ccadecdb128492c94be36ebb881c3c42659626d86eda8
2019-07-15 08:02:07 -04:00
Jon Atack e142ee03e7
doc: describe how to pass wildcard names to test runner 2019-07-15 10:14:23 +02:00
Jon Atack 6a7a70b8cf
test: enable passing wildcards with path to test runner
Currently, passing wildcard testname args to the test runner from outside the `test/functional/` directory does not work. See this recent IRC discussion for more background: http://www.erisian.com.au/bitcoin-core-dev/log-2019-07-10.html#l-262 (lines 262 to 323).

This small change enables passing multiple wildcards with paths, as long as the paths are coherent. Examples:
  - test/functional/test_runner.py test/functional/wallet*
  - functional/test_runner.py functional/wallet*
  - test/functional/test_runner.py ./test/functional/tool* test/functional/mempool*

A current limitation that this PR does not change: 9 test files with arguments in their name are not picked up by wildcard search.

- Squashed commit: non-mutating version

- Squashed commit: minor code optimisation
2019-07-15 10:13:39 +02:00
Wladimir J. van der Laan 536590f358
Merge #16334: test: rpc_users: Also test rpcauth.py with password.
e263a343d4 test: rpc_users: Make variable names more clear. (Carl Dong)
830dc2dd0f test: rpc_users: Also test rpcauth.py with specified password. (Carl Dong)
c73d871799 test: rpc_users: Add function for testing auth params. (Carl Dong)
604e2a997f test: rpc_users: Add function for auth'd requests. (Carl Dong)

Pull request description:

  Fixes #14758

  First two commits are tidy-ups which I feel are worthwhile as they are very straightforward, cut down the file by 50%, and made the final diff more minimal. Happy to squash after review.

ACKs for top commit:
  laanwj:
    ACK e263a343d4

Tree-SHA512: aa75c48570a87060238932d4c68e17234e158077f6195fb4917367e1ecc565e3cd8dd0ae51f9159ddd3d03742739680391bc1246454302db22d4a608c0633e80
2019-07-12 21:28:47 +02:00
MarcoFalke fa0d0ff6e1
Remove unused bits from the service flags enum 2019-07-12 14:14:54 -04:00
Wladimir J. van der Laan 3453cf26db
Merge #15277: contrib: Enable building in Guix containers
751549b52a contrib: guix: Additional clarifications re: substitutes (Carl Dong)
cd3e947f50 contrib: guix: Various improvements. (Carl Dong)
8dff3e48a9 contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong)
3e80ec3ea9 contrib: Add deterministic Guix builds. (Carl Dong)

Pull request description:

  ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~

  Please read the `README.md`.

  -----

  ### Guix Introduction

  This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin.

  ### Guix Build Walkthrough

  Please read the `README.md`.

  [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718)

  [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011)

  [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)

  <details>
  <summary>Old instructions no. 1</summary>
  In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment.

  We can then invoke
  ```
  guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
  ```
  To have Guix:
  1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details).
  2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in.
  3. Drop you into a shell in that container.

  > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>`

  Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following:

  ```
  make -C depends -j"$(nproc)" download && \
      cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
  ```

  We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing).
  </details>

  ### Rationale

  I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because:

  1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](72bd4ab867/contrib/gitian-descriptors/gitian-linux.yml (L10))), because Guix builds everything about the container, we can perform this on almost any Linux distro/system.
  2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)).
  3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler).

  ### Steps to Completion

  - [x] Successfully build bitcoin inside the Guix environment
  - [x] Make `check-symbols` pass
  - [x] Do the above but without nasty hacks
  - [x] Solve some of the more innocuous hacks
  - [ ] Make it cross-compile (HELP WANTED HERE)
    - [x] Linux
      - [x] x86_64-linux-gnu
      - [x] i686-linux-gnu
      - [x] aarch64-linux-gnu
      - [x] arm-linux-gnueabihf
      - [x] riscv64-linux-gnu
    - [ ] OS X
      - [ ] x86_64-apple-darwin14
    - [ ] Windows
      - [ ] x86_64-w64-mingw32
  - [ ] Maybe make importer for depends syntax
  - [ ] Document build process for future releases
  - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html)

  ### Help Wanted

  [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210)

  [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)

  <details>
  <summary>Old content no. 1</summary>
  As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be.

  Here's what `ldd src/bitcoind` looks like when built in a Guix container:
  ```
  	linux-vdso.so.1 (0x00007ffcc2d90000)
  	libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000)
  	librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000)
  	libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000)
  	libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000)
  	libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000)
  	libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000)
  	libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000)
  	/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000)
  ```

  And here's what it looks in one of our releases:
  ```
  	linux-vdso.so.1 (0x00007ffff52cd000)
  	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000)
  	librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000)
  	libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000)
  	libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000)
  	libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000)
  	/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000)
  ```

  ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely.

  Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](d6e700e40f/contrib/gitian-descriptors/gitian-win-signer.yml (L14))
  </details>

  ### How to Help

  1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation)
  2. Try out my branch and the command described above!

ACKs for top commit:
  MarcoFalke:
    Thanks for the replies. ACK 751549b52a
  laanwj:
    ACK 751549b52a

Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:24:45 +02:00
Carl Dong 751549b52a
contrib: guix: Additional clarifications re: substitutes 2019-07-12 12:31:55 -04:00
Hennadii Stepanov ae311bc036
Fix autostart filenames on Linux 2019-07-12 19:01:05 +03:00
Carl Dong cd3e947f50
contrib: guix: Various improvements.
- Clearer and more accurate prose
- Pin `guix pull' to commit rather than branch
- Just use `use-module' instead of `define-module'
- Use `bash-minimal' instead of `bash'
- Remove unneeded `tcsh' from manifest
- Explicitly use `python-3.7'
- Add comments about how {native,cross}-toolchains are produced and
  why
2019-07-12 11:42:36 -04:00
Carl Dong 8dff3e48a9
contrib: guix: Clarify SOURCE_DATE_EPOCH. 2019-07-12 11:42:09 -04:00
Carl Dong 3e80ec3ea9
contrib: Add deterministic Guix builds. 2019-07-12 00:48:39 -04:00
Wladimir J. van der Laan 735d6b57e7
Merge #16227: Refactor CWallet's inheritance chain
93ce4a0b6f Move WatchOnly stuff from SigningProvider to CWallet (Andrew Chow)
8f5b81e6ed Remove CCryptoKeyStore and move all of it's functionality into CWallet (Andrew Chow)
37a79a4fcc Move various SigningProviders to signingprovider.{cpp,h} (Andrew Chow)
16f8096e91 Move KeyOriginInfo to its own header file (Andrew Chow)
d9becff4e1 scripted-diff: rename CBasicKeyStore to FillableSigningProvider (Andrew Chow)
a913e3f2fb Move HaveKey static function from keystore to rpcwallet where it is used (Andrew Chow)
c7797ec655 Remove CKeyStore and squash into CBasicKeyStore (Andrew Chow)
1b699a5083 Add HaveKey and HaveCScript to SigningProvider (Andrew Chow)

Pull request description:

  This PR compresses the `CWallet` chain of inheritance from 5 classes to 3 classes. `CBasicKeyStore` is renamed to `FillableSigningProvider` and some parts of it (the watchonly parts) are moved into `CWallet`. `CKeyStore` and `CCrypoKeyStore` are completely removed. `CKeyStore`'s `Have*` functions are moved into `SigningProvider` and the `Add*` moved into `FillableSigningProvider`, thus allowing it to go away entirely. `CCryptoKeyStore`'s functionality is moved into `CWallet`. The new inheritance chain is:

  ```
  SigningProvider -> FillableSigningProvider -> CWallet
  ```

  `SigningProvider` now is the class the provides keys and scripts and indicates whether keys and scripts are present. `FillableSigningProvider` allows keys and scripts to be added to the signing provider via `Add*` functions. `CWallet` handles all of the watchonly stuff (`AddWatchOnly`, `HaveWatchOnly`, `RemoveWatchOnly` which were previously in `CKeyStore`) and key encryption (previously in `CCryptoKeyStore`).

  Implements the 2nd [prerequisite](https://github.com/bitcoin-core/bitcoin-devwiki/wiki/Wallet-Class-Structure-Changes#cwallet-subclass-stack) from the wallet restructure.

ACKs for top commit:
  Sjors:
    re-ACK 93ce4a0; it keeps `EncryptSecret`, `DecryptSecret` and `DecryptKey` in `wallet/crypter.cpp`, but makes them not static. It improves alphabetical includes, reorders some function definitions, fixes commit message, brings back lost code comment.
  instagibbs:
    utACK 93ce4a0b6f

Tree-SHA512: 393dfd0623ad2dac38395eb89b862424318d6072f0b7083c92a0d207fd032c48b284f5f2cb13bc492f34557de350c5fee925da02e47daf011c5c6930a721b6d3
2019-07-11 22:42:39 +02:00
Wladimir J. van der Laan 28d1353f48
Merge #15649: Add ChaCha20Poly1305@Bitcoin AEAD
bb326add9f Add ChaCha20Poly1305@Bitcoin AEAD benchmark (Jonas Schnelli)
99aea045d6 Add ChaCha20Poly1305@Bitcoin tests (Jonas Schnelli)
af5d1b5f4a Add ChaCha20Poly1305@Bitcoin AEAD implementation (Jonas Schnelli)

Pull request description:

  This adds a new AEAD (authenticated encryption with additional data) construct optimised for small messages (like used in Bitcoins p2p network).

  Includes: #15519, #15512 (please review those first).

  The construct is specified here.
  https://gist.github.com/jonasschnelli/c530ea8421b8d0e80c51486325587c52#ChaCha20Poly1305Bitcoin_Cipher_Suite

  This aims for being used in v2 peer-to-peer messages.

ACKs for top commit:
  laanwj:
    code review ACK bb326add9f

Tree-SHA512: 15bcb86c510fce7abb7a73536ff2ae89893b24646bf108c6cf18f064d672dbbbea8b1dd0868849fdac0c6854e498f1345d01dab56d1c92031afd728302234686
2019-07-11 22:00:16 +02:00
MarcoFalke 4fcccdac78
Merge #16244: Move wallet creation out of the createwallet rpc into its own function
1aecdf2063 Move wallet creation out of the createwallet rpc into its own function (Andrew Chow)

Pull request description:

  Moves the wallet creation logic from within the `createwallet` rpc and into its own function within wallet.cpp.

ACKs for top commit:
  jnewbery:
    ACK 1aecdf2063
  MarcoFalke:
    ACK 1aecdf2063
  Sjors:
    ACK 1aecdf2 with some suggestions for followup.

Tree-SHA512: 8d26d7ff48db4f8fac12408a5a294f788b7f50a72e7eb4008fb74ff14d7400eb3970f8038a19f989eff55198fc11c0cf86f52231c62b9015eb777132edc8ea88
2019-07-10 13:51:25 -04:00
MarcoFalke ff0aad8a40
Merge #16361: Remove redundant pre-TopUpKeypool check
96b6dd468a Remove redundant pre-TopUpKeypool checks (Gregory Sanders)

Pull request description:

  TopUpKeypool already has a quick check for `IsLocked()`

ACKs for top commit:
  achow101:
    ACK 96b6dd468a Reviewed the diff and checked that the `if (!IsLocked()) TopUpKeypool()` pattern is changed everywhere.

Tree-SHA512: 36f5ae1be611404656ac855763e569fd3b5e932db8170f39ebda74300aa02062774b2c28ce6cf00f2ccc0e3550de58df36efa9097e24f0a51f2809b8a489c95a
2019-07-10 13:31:24 -04:00
Gregory Sanders e10e1e8db0 Restrict lifetime of ReserveDestination to CWallet::CreateTransaction 2019-07-10 11:38:37 -04:00
Gregory Sanders d9ff862f2d CreateTransaction calls KeepDestination on ReserveDestination before success 2019-07-10 11:38:37 -04:00
Gregory Sanders 96b6dd468a Remove redundant pre-TopUpKeypool checks 2019-07-10 09:39:26 -04:00
Wladimir J. van der Laan 6c1e45c4c4
Merge #16322: wallet: Fix -maxtxfee check by moving it to CWallet::CreateTransaction
0d101a340c test: Add test for maxtxfee option (MarcoFalke)
177550101b wallet: Remove unreachable code in CreateTransaction (MarcoFalke)
5c1b9714cb wallet: Fix -maxtxfee check by moving it to CWallet::CreateTransaction (João Barbosa)

Pull request description:

  Follow up to #16257, this PR makes `bumpfee` aware of `-maxtxfee`.

  It also prevents dangling locked unspents when calling `fundrawtransaction` - because the previous check was after `LockCoin`.

ACKs for top commit:
  MarcoFalke:
    re-ACK 0d101a340c, only change is small test fixup

Tree-SHA512: 3464b24ae7cd4e72ed41438c6661828ba1304af020f05da62720b23668ae734e16cf47c6d97e150cc84ef631ee099b16fc786c858f3d089905845437338fd512
2019-07-10 14:00:52 +02:00
MarcoFalke 0d101a340c test: Add test for maxtxfee option 2019-07-10 11:58:46 +01:00
Wladimir J. van der Laan d1fc827300
Merge #16270: depends: expat 2.2.7
0512f0521a depends: expat 2.2.7 (fanquake)

Pull request description:

  Major changes in expat 2.2.7:

  * [#186](https://github.com/libexpat/libexpat/issues/186) [#262](https://github.com/libexpat/libexpat/pull/262)  Fix extraction of namespace prefixes from XML names;
                      XML names with multiple colons could end up in the
                      wrong namespace, and take a high amount of RAM and CPU
                      resources while processing, opening the door to use for denial-of-service attacks
  * [#227](https://github.com/libexpat/libexpat/pull/227) Autotools: Add --without-examples and --without-tests

  Full changelog is available [here](https://github.com/libexpat/libexpat/blob/R_2_2_7/expat/Changes#L5).

ACKs for top commit:
  laanwj:
    ACK 0512f0521a

Tree-SHA512: 45162a9b0011107fd59a97dae7b5eb61989dafbec26b1ee497d1b11bf5c6a119971096899caa2998648b82a62db57c629a1560453557146c2496b39a7f3f8de9
2019-07-10 12:41:55 +02:00
Wladimir J. van der Laan 5859b7dc6f
Merge #16338: test: Disable other targets when enable-fuzz is set
84edfc72e5 Update doc and CI config (qmma)
48bcb2ac24 Disable other targets when enable-fuzz is set (qmma)

Pull request description:

  This is to fix https://github.com/bitcoin/bitcoin/issues/16094

  When the `enable-fuzz` flag is set, disable all other binary targets.

ACKs for top commit:
  MarcoFalke:
    ACK 84edfc72e5 (only checked that travis compiled this)

Tree-SHA512: f4ac80526388a67709986b22de88b00bf93ab44ae31a20bd4d8923a4982ab97e015a9f13010081d6ecf6c23ae8afeac7ca9d849d198ce6ebe239aa3127151efc
2019-07-10 12:23:35 +02:00
Wladimir J. van der Laan 8d1286014c
Merge #16237: Have the wallet give out destinations instead of keys
8e7f930828 Add GetNewChangeDestination for getting new change Destinations (Andrew Chow)
33d13edd2b Replace CReserveKey with ReserveDestinatoin (Andrew Chow)
172213be5b Add GetNewDestination to CWallet to fetch new destinations (Andrew Chow)

Pull request description:

  The wallet should give out destinations instead of keys. It should be the one that handles the conversion from key to destination and the setting of the label, not the caller. In order to do this, two new member functions are introduced `GetNewDestination()` and `GetNewChangeDestination()`. Additionally, `CReserveKey` is changed to be `ReserveDestination` and represents destinations whose keys can be returned to the keypool.

ACKs for top commit:
  instagibbs:
    re-utACK 8e7f930828
  sipa:
    ACK 8e7f930828. Concept ACK as this gives a much cleaner abstraction to work with, and light code review ACK.
  laanwj:
    ACK 8e7f930828

Tree-SHA512: 5be7051409232b71e0ef2c1fd1a3e76964ed2f5b14d47d06edc2ad3b3687abd0be2803a1adc45c0433aa2c3bed172e14f8a7e9f4a23bff70f86260b5a0497500
2019-07-10 11:45:55 +02:00
Andrew Chow 1aecdf2063 Move wallet creation out of the createwallet rpc into its own function 2019-07-09 19:50:16 -04:00
MarcoFalke 357488f660
Merge #16240: JSONRPCRequest-aware RPCHelpMan
b6fb617aaa rpc: switch to using RPCHelpMan.Check() (Karl-Johan Alm)
c7a9fc234f Make the RPCHelpMan aware of JSONRPCRequest and add Check() helper (Karl-Johan Alm)
5c5e32bbe3 rpc: migrate JSONRPCRequest functionality into request.cpp (Karl-Johan Alm)
0ab8ba1ac6 rpc: fix RPC help requirements for getblocktemplate (Karl-Johan Alm)

Pull request description:

  Every single RPC call has a helper-section at the start, which throws a help string if the user asks for help or if the user provided too few/many arguments.

  ```C++
  const RPCHelpMan help{...};
  if (request.fHelp || !help.IsValidNumArgs(request.params.size())) {
      throw std::runtime_error(help.ToString());
  }
  ```

  or (older version)

  ```C++
  if (request.fHelp || request.params.size() < min || request.params.size() > max)
      throw std::runtime_error(
          RPCHelpMan{...}.ToString()
      );
  ```

  It seems like an obvious improvement, and less copy-pasting, to make `RPCHelpMan` aware of `JSONRPCRequest`, and to let it handle the checks instead. Both of the above become

  ```C++
  RPCHelpMan{...}.Check(request);
  ```

  which means we save roughly 3 lines per RPC command, and the `RPCHelpMan` instance is never referenced afterwards, so the approach is a tiny fraction cleaner.

  This is a complete update, sans a few special case locations that had special rules. 623 lines turn into 284 (which includes the addition to `RPCHelpMan`).

ACKs for top commit:
  laanwj:
    code rview and lightly tested ACK b6fb617aaa
  MarcoFalke:
    ACK b6fb617aaa, looked at the diff, verified move-only where applicable

Tree-SHA512: eb73f47f812512905b852e313281d1c8df803db40a6188aa39d5a7586631664db6764491152a8a96769946c796dc56d38c6e3a66ddd06ba3fb9d20050e6274e1
2019-07-09 19:31:52 -04:00
Andrew Chow 8e7f930828 Add GetNewChangeDestination for getting new change Destinations
Adds a GetNewChangeDestination that has the same objective as GetNewDestination
2019-07-09 16:43:10 -04:00
Andrew Chow 33d13edd2b Replace CReserveKey with ReserveDestinatoin
Instead of reserving keys, reserve destinations which are backed by keys
2019-07-09 16:43:10 -04:00
Andrew Chow 172213be5b Add GetNewDestination to CWallet to fetch new destinations
Instead of having the same multiple lines of code everywhere
that new destinations are fetched, introduce GetNewDestination as
a member function of CWallet which does the key fetching, label
setting, script generation, and destination generation.
2019-07-09 16:43:10 -04:00