Commit graph

566 commits

Author SHA1 Message Date
Pieter Wuille
940a21932b SegWit wallet support
This introduces two command line flags (-addresstype and -changetype) which control
the type of addresses/outputs created by the GUI and RPCs. Certain RPCs allow
overriding these (`getnewaddress` and `getrawchangeaddress`). Supported types
are "legacy" (P2PKH and P2SH-multisig), "p2sh-segwit" (P2SH-P2WPKH and P2SH-P2WSH-multisig),
and "bech32" (P2WPKH and P2WSH-multisig).

A few utility functions are added to the wallet to construct different address type
and to add the necessary entries to the wallet file to be compatible with earlier
versions (see `CWallet::LearnRelatedScripts`, `GetDestinationForKey`,
`GetAllDestinationsForKey`, `CWallet::AddAndGetDestinationForScript`).
2018-01-09 15:35:31 -08:00
Pieter Wuille
f37c64e477 Implicitly know about P2WPKH redeemscripts
Make CKeyStore automatically known about the redeemscripts necessary for P2SH-P2WPKH
(and due to the extra checks in IsMine, also P2WPKH) spending.
2018-01-09 15:35:27 -08:00
Pieter Wuille
57273f2b30 [test] Serialize CTransaction with witness by default 2018-01-09 15:27:55 -08:00
Suhas Daftuar
35c2b1fe3b Fix rare failure in p2p-segwit.py
Avoid creating very small utxos that would violate an assumption in
test_non_standard_witness.
2018-01-09 14:13:10 -05:00
Russell Yanofsky
a14dbff39e Allow multiwallet.py to be used with --usecli
Add test coverage for bitcoin-cli multiwallet calls.
2018-01-08 17:35:27 -05:00
John Newbery
f6ade9ce1a [tests] allow tests to be run with --usecli
test_framework accepts a new --usecli parameter. Running the test with
this parameter will cause all RPCs to be sent through bitcoin-cli rather
than directly over http. By default, individual test cases do not
support --usecli, and self.supports_cli must be set to True in the
set_test_params method.

We can make supports_cli default to True in future once we know which
tests will fail with use_cli.
2018-01-08 17:35:27 -05:00
Russell Yanofsky
ff9a363ff7 TestNodeCLI batch emulation
Support same get_request and batch methods as AuthServiceProxy
2018-01-08 15:17:14 -05:00
Russell Yanofsky
ca9085afc5 Prevent TestNodeCLI.args mixups
Change TestNodeCLI.__call__() to return a new instance instead of modifying the
existing instance. This way, it's possible to create different cli objects that
have their own options (for example -rpcwallet options to connect to different
wallets), and options set for a single call (`node.cli(options).method(args)`)
will no longer leak into future calls.
2018-01-08 15:17:14 -05:00
Russell Yanofsky
fcfb952bca Improve TestNodeCLI output parsing
Parse JSONRPCException errors, and avoid JSON decode exception if RPC method
returns a plain string.
2018-01-08 15:17:14 -05:00
MarcoFalke
45173fa6fc
Merge #12079: Improve prioritisetransaction test coverage
7f67dd0aa6 [qa] Improve prioritisetransaction functional test (João Barbosa)

Pull request description:

Tree-SHA512: 7a5c446772069cd9ace085ae2635e1f61870c597e2216614628f4b6ebfe209b29f381a182a6f60d09f43f22bb82b59bb573b5441fa8e7b958a5fd0d5aad80d86
2018-01-06 03:28:23 -09:00
João Barbosa
7f67dd0aa6 [qa] Improve prioritisetransaction functional test 2018-01-05 15:34:43 +00:00
practicalswift
4aa64557b5 Fix incorrect Markdown link 2018-01-04 23:55:13 +01:00
Wladimir J. van der Laan
a9a49e6e7e
Merge #12001: [RPC] Adding ::minRelayTxFee amount to getmempoolinfo and updating help
aad3090 [rpc] Adding ::minRelayTxFee amount to getmempoolinfo and updating mempoolminfee help description (Jeff Rade)

Pull request description:

  These are RPC document changes from #11475 which is now merged.  Took into consideration comments from #11475 and #6941 for this PR.

  Biggest change here is when calling `getmempoolinfo`, will now show the `minrelaytxfee` in the JSON reponse (see below):

  ```
  $ bitcoin-cli getmempoolinfo
  {
    "size": 50,
    "bytes": 13102,
    "usage": 70480,
    "maxmempool": 300000000,
    "mempoolminfee": 0.00001000,
    "minrelaytxfee": 0.00001000
  }
  ```

  Fixes #8953

Tree-SHA512: 5ca583961365ee1cfe6e0d19afb0b41d542e179efee3b3c5f3fcf7d3ebca9cc3eedfd1434a0da40c5eed84fba98b35646fda201e6e61c689b58bee9cbea44b9e
2018-01-04 09:22:53 +01:00
Wladimir J. van der Laan
5961b23898
Merge #12062: Increment MIT Licence copyright header year on files modified in 2017
595a7ba Increment MIT Licence copyright header year on files modified in 2017 (Akira Takizawa)

Pull request description:

  Edited via:

  $ contrib/devtools/copyright_header.py update .

  ps) It is the same commit as #9450

Tree-SHA512: 274bfcd6cf2914315ed52f6db773a68800ce9d6bd225a3142654483f0bbc3fd865009e62f9d954f65765d038c626e55d2a64e37e16843809adc2f67abe659b6d
2018-01-03 14:33:28 +01:00
MarcoFalke
d38d1a3e75
Merge #11965: qa: Note on test order in test_runner
fadf60e381 qa: Note on test order in test_runner (MarcoFalke)

Pull request description:

  C.f. #11964

Tree-SHA512: 5f087965093722d9e7a3febddcc187e412bd0636a7ed2da60111668fe3bba6668110e25a38ddcccc0d0aae132611c56fa72f3f0c473fb3fb59e38be445edfcd5
2018-01-03 12:07:52 +02:00
Anthony Towns
5fecd842a6 [tests] Remove redundant import in blocktools.py test 2018-01-03 16:16:56 +10:00
Akira Takizawa
595a7bab23 Increment MIT Licence copyright header year on files modified in 2017 2018-01-03 02:26:56 +09:00
Jeff Rade
aad309065d [rpc] Adding ::minRelayTxFee amount to getmempoolinfo and updating mempoolminfee help description 2017-12-29 08:56:44 -06:00
Wladimir J. van der Laan
9bad8d6472
Merge #11475: [rpc] mempoolinfo should take ::minRelayTxFee into account
149dffd [rpc] mempoolinfo should take ::minRelayTxFee into account (Cristian Mircea Messel)

Pull request description:

  Fixes #6941 following https://github.com/bitcoin/bitcoin/pull/11410#issuecomment-332991078 's suggestion

  This takes care of the mentioned ticket without changing the behavior of https://github.com/bitcoin/bitcoin/pull/11410/files#diff-24efdb00bfbe56b140fb006b562cc70bL629

  By modifying 5a9da37fb3/src/txmempool.cpp (L984) the syncing mempools becomes problematic as per https://github.com/bitcoin/bitcoin/pull/11410#issuecomment-333868390

  ~~Same code causes different tests to fail: https://travis-ci.org/bitcoin/bitcoin/jobs/286128241 https://travis-ci.org/bitcoin/bitcoin/jobs/286128241 . I can't reproduce the problems locally, am I doing something wrong?~~ travis sometimes fails unexpectedly

Tree-SHA512: fd81628da6a3eff51bd09e5342d781bac0710f79d6b330b1df3662756ecaceb2e1682bf9768b5f8edbcba6479a3223dfa6604d37c9e9d37d00d077172da4f6ea
2017-12-23 10:00:24 +01:00
MarcoFalke
fadf60e381 qa: Note on test order in test_runner 2017-12-21 08:56:40 -05:00
Wladimir J. van der Laan
711d16ca4a
Merge #11667: Add scripts to dumpwallet RPC
656fde5 Add script birthtime metadata to dump and import wallet (MeshCollider)
1bab9b2 Add script dump note to RPC help text and release notes (MeshCollider)
68c1e00 Add test for importwallet (MeshCollider)
9e1184d Add dumpwallet scripts test (MeshCollider)
ef0c730 Add scripts to importwallet RPC (MeshCollider)
b702ae8 Add CScripts to dumpwallet RPC (MeshCollider)
cdc260a Add GetCScripts to CBasicKeyStore (MeshCollider)

Pull request description:

  As discussed in https://github.com/bitcoin/bitcoin/pull/11289#issuecomment-334600457, adds the CScripts from the wallet to the `dumpwallet` RPC and then allows them to be imported with the `importwallet` RPC. Includes a basic test, and modifies the helptext of the dumpwallet RPC.

  Notes:
  - Reviewers: use `?w=1` to avoid the indentation-only change in commit `Add scripts to importwallet RPC `
  - currently the scripts are followed with `# addr=` comments just as the other keys are, unsure if this might confuse users into thinking all the scripts are for valid P2SH addresses though, but I don't think that should be an issue.
  - there are no birthtimes for scripts, so script imports don't affect rescans
  - `importwallet` imports the CScripts but I'm not sure how to approach specifying whether scripts are for P2SH addresses, BIP173 addresses, etc. whether that matters or not. Otherwise the RPC helptext might just need modification.

  Fixes #11715

Tree-SHA512: 36c55837b3a58b9d3499d4c0c2ae82153d62aa71919e751574651b63a1d2b8ecc83796db4553cc65dad9b5341c3a42ae2fcf4d62598c30af267f8e1461ba8272
2017-12-21 13:03:26 +01:00
MarcoFalke
604e08c83c
Merge #11726: Cleanups + nit fixes for walletdir PR
aac6b3f067 Update files.md for new wallets/ subdirectory (MeshCollider)
b67342906c Cleanups for walletdir PR (MeshCollider)

Pull request description:

  This addresses the remaining nits from https://github.com/bitcoin/bitcoin/pull/11466

  - Updates `doc/files.md` with respect to the new default wallet directory
  - Fixes @promag and @laanwj's error message nit, and Jonas' release notes nit
  - ~Addresses @laanwj's net-specific wallet subdirectory concern in the case that a walletdir is specified~
  - Changes the #includes from "" to <> style after #11651

Tree-SHA512: b86bf5fdc4de54c1b0f65b60a83af3cf82b35d216ce9c0de724803bfba6934796238b6c412659dcc29ae2e3e856d4eb97ae777c80f36f4089d8acecfddefe9aa
2017-12-20 17:37:57 -05:00
Wladimir J. van der Laan
cfd99ddc3c
Merge #11883: Add configuration file/argument testing
be9a13c Add configuration/argument testing (MeshCollider)

Pull request description:

  Adds a new functional test for testing various command line and configuration file argument interactions, that aren't specific enough to other functionality to be placed in other tests.

  Currently this tests the error messages for non-existent datadir, which would have caught the bug fixed in https://github.com/bitcoin/bitcoin/pull/11829. It also tests that command line arguments override the ones in the config file.

  I plan on working on a fix for https://github.com/bitcoin/bitcoin/issues/11819 / https://github.com/bitcoin/bitcoin/issues/1044 and then expanding this test with cases for that.

Tree-SHA512: 97aea18c67d331db3ca3d0c99c79267cf012df67fddafc9fac63d392f5c3a6469aa14d93b5865c3bbe561461648d2485367978a77446483b8df53d1916f1c8e8
2017-12-20 09:37:36 +01:00
MeshCollider
68c1e00a00 Add test for importwallet 2017-12-20 18:47:56 +13:00
MeshCollider
9e1184dd54 Add dumpwallet scripts test 2017-12-20 18:47:56 +13:00
MeshCollider
be9a13c8a0 Add configuration/argument testing 2017-12-20 12:23:12 +13:00
John Newbery
ee5efad6cf [tests] refactor node_network_limited 2017-12-19 11:20:12 -05:00
John Newbery
b425131f5a [tests] remove redundant duplicate tests from node_network_limited 2017-12-19 11:20:12 -05:00
John Newbery
2e02984591 [tests] node_network_limited - remove race condition
node_network_limited had a race condition, since wait_for_block()
doesn't do what you might expect. It only checks the most recent block
received over the P2P interface (perhaps we should rename the method
wait_for_most_recent_block() to avoid future confusion). The test can
fail if the node sends us invs for other blocks, we respond with a
getdata, and the node sends us one of those blocks in the 0.05 second
wait_until loop window.

Fix this by not responding to inv messages with getdata messages.
2017-12-19 11:20:12 -05:00
John Newbery
dbfe294805 [tests] define NODE_NETWORK_LIMITED in test framework 2017-12-19 11:19:55 -05:00
John Newbery
1285312048 [tests] fix flake8 warnings in node_network_limited.py 2017-12-19 11:19:55 -05:00
MarcoFalke
8049241e22
Merge #11946: tests: Remove unused variable (firstAddrnServices)
bdaed47558 tests: Remove unused variable (firstAddrnServices) (practicalswift)

Pull request description:

  Remove unused variable (`firstAddrnServices`).

Tree-SHA512: 7075952b392227c00fcb22adb1fd3f7ef31d7fd7101fbeff632beeea814c70beaef5c9957736eff2451976c8b43b950fc949642e608b610c5fefd150282eafb0
2017-12-19 08:12:53 -05:00
Wladimir J. van der Laan
4508519250
test: Fix rawtransactions test
Looks like another `assert_raises_jsonrpc` snuck in with #11178.
Change it to `assert_raises_rpc_error`.
2017-12-19 13:24:51 +01:00
practicalswift
bdaed47558 tests: Remove unused variable (firstAddrnServices) 2017-12-19 13:17:58 +01:00
Wladimir J. van der Laan
fee0370fd6
Merge #11178: Add iswitness parameter to decode- and fundrawtransaction RPCs
6f39ac0 Add test for decoderawtransaction bool (MeshCollider)
bbdbe80 Add iswitness parameter to decode- and fundrawtransaction RPCs (MeshCollider)

Pull request description:

  Suggested in https://github.com/bitcoin/bitcoin/pull/10481#issuecomment-325244946, this adds the option to explicitly choose whether a serialized transaction should be decoded as a witness or non-witness transaction rather than relying on the heuristic checks in #10481. The parameter defaults to relying on #10481 if not included, but it overrides that if included.

Tree-SHA512: d4846a5bb7d64dc19c516445488b00af329fc1f4181d9dfdf9f2382a086568edc98250a4ac7594e24a1bc231dfdee53c699b12c8380c355b920a67cc6770b7a9
2017-12-19 09:55:11 +01:00
João Barbosa
88af5028ad test: Add createrawtransaction functional tests 2017-12-13 14:51:46 +00:00
João Barbosa
27c6199373 test: Add multidict to support dictionary with duplicate key (laanwj) 2017-12-13 14:51:46 +00:00
MeshCollider
b67342906c Cleanups for walletdir PR 2017-12-13 23:36:43 +13:00
Wladimir J. van der Laan
ad1820cbad
Merge #11849: [tests] Assert that only one NetworkThread exists
5c8ff26 [tests] Add NetworkThread assertions (John Newbery)
34e08b3 [tests] Fix network threading in functional tests (John Newbery)
74e64f2 [tests] Use network_thread_start() in tests. (John Newbery)
5fc6e71 [tests] Add network_thread_ utility functions. (John Newbery)

Pull request description:

  Add assert that only one NetworkThread exists at any time in functional tests, and fix cases where that wasn't true.

  fixes #11776

Tree-SHA512: fe5d1c59005f94bf66e11bb23ccf274b1cd9913741b56ea11dbcd21db4cc0b53b4413c0c4c16dbcd6ac611adad5e5cc2baaa39720598ce7b6393889945d06298
2017-12-12 13:09:52 +01:00
Wladimir J. van der Laan
8ab6c0b09e
Merge #11839: don't attempt mempool entry for wallet transactions on startup if alr…
6697a70 add test for unconfirmed balance between restarts (Gregory Sanders)
6ba8f30 don't attempt mempool entry for wallet transactions on startup if already in mempool (Gregory Sanders)

Pull request description:

  …eady in mempool

  Mempool loads first, wallet second. Second attempt fails, marking that transaction !fInMempool. Those funds will disappear until confirmation is reached.

Tree-SHA512: 955f0565ec1dc1ba395e0b803a98c07b7cd00c8cac5ec618ed832fed259a856fb7bbbe41310cf6a4e43c0435e09b156109d2a4467d403811dc8379d2caebeede
2017-12-11 16:21:03 +01:00
John Newbery
5c8ff26b2f [tests] Add NetworkThread assertions
Check that P2PConnections aren't created after the NetworkThread
has started and that at any time only one NetworkThread is running.
2017-12-11 09:17:21 -05:00
John Newbery
34e08b3510 [tests] Fix network threading in functional tests
assumevalid.py, example_test.py and p2p-acceptblocks.py add
p2p_connections after the NetworkThread has been started. This isn't
permitted. Fix test to restart the network thread when adding new
connections.

p2p-leaktest.py had a potential race condition if the NetworkThread
hadn't terminated by the time we tried to restart it.
2017-12-11 09:17:21 -05:00
John Newbery
74e64f24b8 [tests] Use network_thread_start() in tests. 2017-12-11 09:17:21 -05:00
John Newbery
5fc6e71d19 [tests] Add network_thread_ utility functions.
Add network thread_start(), network_thread_running() and
network_thread_join() utility functions in mininode.py and use
network_thread_running() in network thread assertions.
2017-12-11 09:16:44 -05:00
Gregory Sanders
6697a70894 add test for unconfirmed balance between restarts 2017-12-11 09:14:50 -05:00
practicalswift
c7399e7082 Remove unused Python imports 2017-12-10 11:48:36 +01:00
Wladimir J. van der Laan
59d3dc85b6
Merge #11740: Implement BIP159 NODE_NETWORK_LIMITED (pruned peers) *signaling only*
de74c62 [Doc] Update bip.md, add support for BIP 159 (Jonas Schnelli)
e054d0e [QA] Add node_network_limited test (Jonas Schnelli)
bd09416 Avoid leaking the prune height through getdata (fingerprinting countermeasure) (Jonas Schnelli)
27df193 Always set NODE_NETWORK_LIMITED bit (Jonas Schnelli)
7caba38 Add NODE_NETWORK_LIMITED flags and min block amount constants (Jonas Schnelli)

Pull request description:

  Extracted from #10387.
  Does implement BIP159, but only the signalling part. No connections are made to NODE_NETWORK_LIMITED in this PR.

  The address relay and connection work (the more complicated part) can then be separated (probably in #10387).

Tree-SHA512: e3218eb4789a9320b0f42dc10f62d30c13c49bdef00443fbe653bee22933477adcfc1cf8f6a95269324560b5721203ed41f3c5e2dd8a98ec2791f6a9d8346b1a
2017-12-09 08:38:36 +01:00
Anthony Towns
9b20bb40fb [tests] Check tests conform to naming convention
Extra-Author: John Newbery <john@johnnewbery.com>
2017-12-08 01:50:02 +10:00
Jonas Schnelli
e054d0e532
[QA] Add node_network_limited test 2017-12-06 21:13:48 -10:00
MarcoFalke
fa4c16d2e7 qa: Add getrawtransaction in_active_chain=False test 2017-12-06 11:00:08 -05:00
Wladimir J. van der Laan
497d0e014c
Merge #10275: [rpc] Allow fetching tx directly from specified block in getrawtransaction
434526a [test] Add tests for getrawtransaction with block hash. (Karl-Johan Alm)
b167951 [rpc] Allow getrawtransaction to take optional blockhash to fetch transaction from a block directly. (Karl-Johan Alm)
a5f5a2c [rpc] Fix fVerbose parsing (remove excess if cases). (Karl-Johan Alm)

Pull request description:

  [Reviewer hint: use [?w=1](https://github.com/bitcoin/bitcoin/pull/10275/files?w=1) to avoid seeing a bunch of indentation changes.]

  Presuming a user knows the block hash of the block containing a given transaction, this PR allows them to fetch the raw transaction, even without `-txindex`. It also enables support for getting transactions that are in orphaned blocks.

  Note that supplying a block hash will override mempool and txindex support in `GetTransaction`. The rationale behind this is that a transaction may be in multiple places (orphaned blocks) and if the user supplies an explicit block hash it should be adhered to.

  ```Bash
  $ # a41.. is a tx inside an orphan block ..3c6f.. -- first try getting it normally
  $ ./bitcoin-cli getrawtransaction a41e66ee1341aa9fb9475b98cfdc1fe1261faa56c0a49254f33065ec90f7cd79 1
  error code: -5
  error message:
  No such mempool transaction. Use -txindex to enable blockchain transaction queries. Use gettransaction for wallet transactions.
  $ # now try with block hash
  $ ./bitcoin-cli getrawtransaction a41e66ee1341aa9fb9475b98cfdc1fe1261faa56c0a49254f33065ec90f7cd79 1 0000000000000000003c6fe479122bfa4a9187493937af1734e1e5cd9f198ec7
  {
    "hex": "01000000014e7e81144e42f6d65550e59b715d470c9301fd7ac189[...]90488ac00000000",
    "inMainChain": false,
    "txid": "a41e66ee1341aa9fb9475b98cfdc1fe1261faa56c0a49254f33065ec90f7cd79",
    "hash": "a41e66ee1341aa9fb9475b98cfdc1fe1261faa56c0a49254f33065ec90f7cd79",
    "size": 225,
  [...]
  }
  $ # another tx 6c66... in block 462000
  $ ./bitcoin-cli getrawtransaction 6c66b98191e9d6cc671f6817142152ebf6c5cab2ef008397b5a71ac13255a735 1 00000000000000000217f2c12922e321f6d4aa933ce88005a9a493c503054a40
  {
    "hex": "0200000004d157[...]88acaf0c0700",
    "inMainChain": true,
    "txid": "6c66b98191e9d6cc671f6817142152ebf6c5cab2ef008397b5a71ac13255a735",
    "hash": "6c66b98191e9d6cc671f6817142152ebf6c5cab2ef008397b5a71ac13255a735",
    "size": 666,
  [...]
  }
  $
  ```

Tree-SHA512: 279be3818141edd3cc194a9ee65929331920afb30297ab2d6da07293a2d7311afee5c8b00c6457477d9f1f86e86786a9b56878ea3ee19fa2629b829d042d0cda
2017-12-06 12:10:21 +01:00
Karl-Johan Alm
434526aba6
[test] Add tests for getrawtransaction with block hash. 2017-12-05 12:28:06 +09:00
Wladimir J. van der Laan
24df9af816
Merge #11781: Add -debuglogfile option
5a7c09a test: Add tests for `-debuglogfile` with subdirs (Anthony Towns)
4158734 doc: Update release notes for `-debuglogfile` (Wladimir J. van der Laan)
2323242 test: Add test for `-debuglogfile` (Wladimir J. van der Laan)
cf5f432 Add `-debuglogfile` option (Wladimir J. van der Laan)

Pull request description:

  This patch adds an option to configure the name and/or directory of the debug log file.

  The user can specify either a relative path, in which case the path is relative to the (network specific) data directory. They can also specify an absolute path to put the log anywhere else in the file system.

  Alternative to #11741 that gets rid of the concept of a "log directory" by specifying the path for the specific kind of log, the debug log. Which happens to be the only kind of log we have at this point*, but a hypothetical new kind of log (say, an audit log) would get a new option. This has more flexibility than specifying a directory which has to contain all of them.

  \* excluding `db.log` which is internally generated by the wallet database library, but that one moves along with `-walletdir`.

Tree-SHA512: 4434d0e598dc23504e5c9e67fdbaef56db4f0fd490f9f54fd503e69d4dda9b5b69c539e1794ed841e72161b7b1dc3374d2f1193dd431b057566750e56fd8f24b
2017-12-04 18:55:17 +01:00
Anthony Towns
5a7c09aebf test: Add tests for -debuglogfile with subdirs 2017-12-01 11:28:23 +01:00
Wladimir J. van der Laan
23232422e5 test: Add test for -debuglogfile 2017-12-01 11:28:13 +01:00
MarcoFalke
13e31dd654
Merge #11791: [tests] Rename NodeConn and NodeConnCB
873beca6d [tests] Rename NodeConn and NodeConnCB (John Newbery)

Pull request description:

  Final step in #11518

  NodeConn -> P2PConnection
  NodeConnCB -> P2PInterface

  This is basically just a rename. Should be an easy review.

Tree-SHA512: fe1204b2b3d8182c5e324ffa7cb4099a47ef8536380e0bb9d37a5fccf76a24f548d1f1a7988ab8f830986a3058b670696de3fc891af5e5f75dbeb4e3273005d7
2017-11-30 18:01:51 -05:00
Wladimir J. van der Laan
60d739eb49
Merge #11789: [tests] [travis-ci] Combine logs on failure
ff8a9b0 [tests] Add combinedlogslen argument to test_runner.py (John Newbery)
dba94ea [tests] [travis-ci] Move Travis functional test log post processing to test_runner (John Newbery)
bba1c54 [tests] Improve logging shutdown and add hint for combine_logs (John Newbery)

Pull request description:

  Replaces #11779 . Notes from that PR:

  > Currently, when a functional test fails, the debug logs are printed sequentially to the travis log. This makes debugging race conditions based on the travis log hard. Instead, all logs events should be combined and sorted by their timestamp, then appended to the travis log.

  @MarcoFalke

Tree-SHA512: 56c80067d6a2c92f7e6a35e3ae5160637a0de052d9da593c7be6e02233544a93c66d62456f903f85e2edc09e31ab4bdafd1aed1d9897ae48c634f82631f856f7
2017-11-30 13:41:11 +01:00
Anthony Towns
7250b4e563 [tests] README.md nit fixes 2017-11-30 19:59:39 +10:00
John Newbery
873beca6de [tests] Rename NodeConn and NodeConnCB
NodeConn -> P2PConnection
NodeConnCB -> P2PInterface
2017-11-29 17:14:48 -05:00
MarcoFalke
9f2c2dba21
Merge #11712: [tests] Split NodeConn from NodeConnCB
e9dfa9bcc [tests] Move version message sending from NodeConn to NodeConnCB (John Newbery)
dad596fc3 [tests] Make NodeConnCB a subclass of NodeConn (John Newbery)
e30d40438 [tests] Move only: move NodeConnCB below NodeConn (John Newbery)
4d5059856 [tests] Tidy up mininode (John Newbery)
f2ae6f32a [tests] Remove mininode periodic (half-hour) ping messages (John Newbery)
ec59523c5 [tests] Remove rpc property from TestNode in p2p-segwit.py. (John Newbery)

Pull request description:

  This is the final step in #11518, except for possibly renaming - for motivation, please see that PR.

  If this is merged, then migrating the test framework from asyncore to asyncio should be easier (I say should because I haven't dug too deeply into what would be required).

  Requesting review from @ryanofsky , since he always has good feedback on these refactor PRs, and I'd appreciate his take on this refactor. Note particularly that I've reverted the change suggested here: https://github.com/bitcoin/bitcoin/pull/11182#discussion_r148859555 . The idea, as always, is to present a simple interface to the test writer.

Tree-SHA512: 94dd467a13ec799b101108cf47d4dccb6f6240b601e375e3d785313333bbb389c26072a50759aca663bbf3d6c8b867b99e36ae8800ab8ea115e0496c151926ce
2017-11-29 16:29:19 -05:00
John Newbery
ff8a9b068a [tests] Add combinedlogslen argument to test_runner.py
Instead of calling combine_logs from test_runner when the environment variable
PYTHON_DEBUG is set, add a combinedlogslen argument. This makes it easier
to call when running locally, and allows a different length of combined logs
to be printed.
2017-11-29 15:10:51 -05:00
John Newbery
dba94ead0c [tests] [travis-ci] Move Travis functional test log post processing to test_runner 2017-11-29 15:10:05 -05:00
John Newbery
bba1c54edd [tests] Improve logging shutdown and add hint for combine_logs 2017-11-29 13:29:15 -05:00
Karl-Johan Alm
a5f5a2ce53
[rpc] Fix fVerbose parsing (remove excess if cases). 2017-11-29 17:52:13 +09:00
John Newbery
e9dfa9bccc [tests] Move version message sending from NodeConn to NodeConnCB
This commit moves the logic that sends a version message
on connection from NodeConn to NodeConnCB. NodeConn should
not be aware of the semantics or meaning of the P2P payloads.
2017-11-28 12:47:02 -05:00
John Newbery
dad596fc37 [tests] Make NodeConnCB a subclass of NodeConn
This makes NodeConnCB a subclass of NodeConn, and
removes the need for the client code to know
anything about the implementation details of NodeConnCB.

NodeConn can now be swapped out for any other implementation
of a low-level connection without changing client code.
2017-11-28 12:44:16 -05:00
John Newbery
e30d404385 [tests] Move only: move NodeConnCB below NodeConn
This is required since NodeConnCB will inherit from NodeConn
after the next commit.
2017-11-28 12:31:37 -05:00
John Newbery
4d50598569 [tests] Tidy up mininode
Add docstrings and renames some methods.
Also removes the redundant NodeConn.readable() method override.
2017-11-28 12:31:31 -05:00
MarcoFalke
a89221873a
Merge #11683: tests: Remove unused mininode functions {ser,deser}_int_vector(...). Remove unused imports.
f522fb7c9 tests: Remove unused mininode functions deser_int_vector(f) and ser_int_vector(l) (practicalswift)
0f3b752ec Remove unused imports (practicalswift)

Pull request description:

  * Remove unused mininode functions `deser_int_vector(f)` and `ser_int_vector(l)`. Last use removed in 3858aabbd0. Friendly ping @jnewbery :-)
  * Remove unused imports.

Tree-SHA512: 840c5623eae9f929561f6e86816883c5904ec1af82fc8d5e56dee1c0b1fe22e8600c10f7358ed8b556b3aec8c65c4910f6eee30e8015a573c4df8bef91124d3e
2017-11-25 21:54:45 -05:00
John Newbery
f2ae6f32a6 [tests] Remove mininode periodic (half-hour) ping messages 2017-11-23 10:03:22 -05:00
John Newbery
ec59523c59 [tests] Remove rpc property from TestNode in p2p-segwit.py.
Change the helper methods to functions which take a node and a p2p
connection as arguments.
2017-11-22 18:08:37 -05:00
MarcoFalke
3d6ad40777
Merge #11743: qa: Add multiwallet prefix test
fa61c6f6a qa: Add multiwallet prefix test (MarcoFalke)

Pull request description:

  Fixes https://github.com/bitcoin/bitcoin/pull/10849#discussion_r127847938

Tree-SHA512: 7967be04e76d935398b3bea60c864ffc9e38dbb4cfb55890bb146a6f16c28d81ca5d89736275e2d0b03642806f6f7093beeea979f5257c464f437c4e5a9684f1
2017-11-22 11:31:35 -05:00
MarcoFalke
0d89fa0877
Merge #11707: [tests] Fix sendheaders
9d42cc333 [tests] address review comments (John Newbery)
f0c4ab9a7 [tests] fix flakiness in sendheaders.py (John Newbery)
25fd6e2c2 [tests] refactor check_last_announcement() in sendheaders.py (John Newbery)
f39d4bbd1 [tests] tidy up BaseNode in sendheaders.py (John Newbery)
2613c545f [tests] fix flake8 warnings in sendheaders.py (John Newbery)

Pull request description:

  This PR should fix the intermittent failure of sendheaders.py described in #11673. The first three commits are tidying up and refactoring the file. The final commit _fix flakiness in sendheaders.py_ fixes the intermittent failures. The commit message for that commit describes the problems that are being fixed.

  I think @laanwj @MeshCollider @MarcoFalke have seen these failures.

  fixes #11673

Tree-SHA512: 278e1af85f2eae00f970f2d8ef33686dd52b4f62180dea4cfdaff7bcf3287c6f1c2930355d99461a12f0c51c4d42cc3b1cb3275174134028ca4d06ffc24c18dd
2017-11-18 17:30:24 -05:00
Wladimir J. van der Laan
d080a7d503
Merge #11466: Specify custom wallet directory with -walletdir param
c1e5d40 Make debugging test crash easier (MeshCollider)
8263f6a Create walletdir if datadir doesn't exist and fix tests (MeshCollider)
9587a9c Default walletdir is wallets/ if it exists (MeshCollider)
d987889 Add release notes for -walletdir and wallets/ dir (MeshCollider)
80c5cbc Add test for -walletdir (MeshCollider)
0530ba0 Add -walletdir parameter to specify custom wallet dir (MeshCollider)

Pull request description:

  Closes #11348

  Adds a `-walletdir` parameter which specifies a directory to use for wallets, allowing them to be stored separately from the 'main' data directory. Creates a new `wallets/` directory in datadir if this is the first time running, and defaults to using it if it exists.

  Includes tests and release notes. Things which might need to be considered more:
  - there is no 'lock' on the wallets directory, which might be needed?
  - because this uses a new wallets/ directory by default, downgrading to an earlier version won't see the wallets in that directory (not a big deal though, users can just copy them up to the main dir)
  - jnewbery suggested putting each wallet in its own directory, which is a good idea, but out of scope for this PR IMO. EDIT: this is being done in https://github.com/bitcoin/bitcoin/pull/11687
  - doc/files.md needs updating (will do soon)

  I also considered including  a cleanup by removing caching of data directory paths and instead just initialise them once on startup (c.f. #3073), but decided it wasn't super relevant here will just complicate review.

Tree-SHA512: c8ac04bfe9a810c32055f2c8b8fa0d535e56125ceb8d96f12447dd3538bf3e5ee992b60b1cd2173bf5f3fa023a9feab12c9963593bf27ed419df929bb413398d
2017-11-18 14:41:15 +01:00
John Newbery
9d42cc3331 [tests] address review comments 2017-11-17 17:15:28 -05:00
practicalswift
f522fb7c9c tests: Remove unused mininode functions deser_int_vector(f) and ser_int_vector(l) 2017-11-17 15:35:15 +01:00
practicalswift
0f3b752ec7 Remove unused imports 2017-11-17 15:34:28 +01:00
Wladimir J. van der Laan
dcfef277b4 cli: Reject arguments to -getinfo
Currently it's possible to accidentally type e.g.

    bitcoin-cli -getinfo getbalance

and get an answer which can be confusing; the trialing arguments are
just ignored.

To avoid this, throw an error if the user provides arguments to
`-getinfo`.
2017-11-17 15:11:50 +01:00
Wladimir J. van der Laan
ccc70a295f
Merge #11648: [tests] Add messages.py
1135c79 [tests] Tidy up mininode.py module (John Newbery)
f9cd9b1 [tests] Move test_framework Bitcoin primitives into separate module (John Newbery)

Pull request description:

  Second part of #11518.

  Moves the primitive Bitcoin datastructures and message classes into their own module, and tidies up the mininode.py module.

  - First commit is almost entirely move-only
  - Second commit is mostly move-only, but also does a little tidying.

Tree-SHA512: 5d74802677f1ab788e43188653106a96fffd9ab1fe3aa6a4eb94e5807de5dd5c8ee212296f45e8d16c7e3d95cfc4891677e812b7944bd3ab604e04b3b88aa06e
2017-11-17 13:16:40 +01:00
MeshCollider
c1e5d40e16 Make debugging test crash easier 2017-11-18 00:51:00 +13:00
MeshCollider
8263f6a5ac Create walletdir if datadir doesn't exist and fix tests 2017-11-18 00:50:59 +13:00
MeshCollider
80c5cbc14f Add test for -walletdir 2017-11-18 00:50:58 +13:00
John Newbery
f0c4ab9a70 [tests] fix flakiness in sendheaders.py
Fixes to sources of intermittent failure in sendheaders.py

- at the start of test_null_locators(), a new block is generated and
then a getheaders is sent. check_last_accouncement() is called to assert
that the headers message is received. However, the new block triggers an
inv to be sent over both P2P connections, so there's a race. If the inv
is received at the wrong time, the test fails.

- test_null_locators() ends by sending a block to the node under test.
At the start of test_nonnull_locators(), a block is mined and
check_last_announcement() is called to assert that the inv received is
for the same block. That means there's a race: if the inv from the block
sent in test_null_locators() is received at the wrong time, the test
fails.
2017-11-16 13:45:46 -05:00
John Newbery
25fd6e2c20 [tests] refactor check_last_announcement() in sendheaders.py
All calls of check_last_announcement() asserted that the return
value was True. Just assert inside the function instead. This
gives better debug information if the assert fails.

Also only check the contents of the most recent inv and header if
check_last_announcement() is called with the relevant argument.
2017-11-16 12:05:27 -05:00
John Newbery
f39d4bbd1e [tests] tidy up BaseNode in sendheaders.py 2017-11-16 11:58:01 -05:00
John Newbery
2613c545f5 [tests] fix flake8 warnings in sendheaders.py 2017-11-16 09:53:35 -05:00
Wladimir J. van der Laan
99bc0b428b
Merge #11087: Diagnose unsuitable outputs in lockunspent().
28f8b66 Diagnose unsuitable outputs in lockunspent(). (Eelis)

Pull request description:

  Fixes #2667.

  This is a simplified version of pull request #3574, which was abandoned by its author.

  I added some tests as well.

Tree-SHA512: e63e00dec8b1b232079380183805cb0b0b18c78ea6bea769837949aab984689d7f68b2ccfe66b1873517b040b9e616ce0eb058575c3d4382aa8c26eebcf1f14e
2017-11-16 12:24:05 +01:00
MarcoFalke
fa61c6f6a6 qa: Add multiwallet prefix test 2017-11-15 11:02:41 -05:00
MarcoFalke
3bdf242fc6
Merge #11677: qa: Remove unused NodeConn members
fafdad0d4 qa: Remove unused NodeConn members (MarcoFalke)

Pull request description:

  * `ver_send` and `ver_recv` were completely unused
  * `rpc` was only used once, in p2p-segwit. Imo better only pass it to the constructor in that single test

Tree-SHA512: 7f85554d6d0fd2096516ca3c608811d5370da66cde35d8031bdc921607a7a4efdb26355896012f75f713f8df09e28d46ba46be69fd96a5898fabb1a25cbcb8ad
2017-11-14 17:44:09 -05:00
Wladimir J. van der Laan
7adeea3b0f
Merge #11641: qa: Only allow disconnecting all NodeConns
faaa7db qa: Only allow disconnecting all NodeConns (MarcoFalke)

Pull request description:

  Disconnecting the connection with `index=0` makes no sense when there are more than one connections, as the list "rotates around" and populates index 0 after `del`.

  Just disconnect all NodeConns in any case.

Tree-SHA512: e5cf540823fccb31634b5a11501f54222be89862e80ccafc28bc06726480f8d2153b8c1b6f859fa6a6d087876251d48a6c6035bccdaaf16831e300bc17ff613d
2017-11-14 08:56:37 +01:00
MarcoFalke
fafdad0d46 qa: Remove unused NodeConn members 2017-11-13 17:43:39 -05:00
MarcoFalke
faaa7db25e qa: Only allow disconnecting all NodeConns 2017-11-13 15:27:10 -05:00
MarcoFalke
95e14dc81d
Merge #11055: [wallet] [rpc] getreceivedbyaddress should return error if called with address not owned by the wallet
5e0ba8f8c [wallet] getreceivedbyaddress should return error if address is not mine (John Newbery)
ea0cd24f7 [tests] Tidy up receivedby.py (John Newbery)

Pull request description:

  Two commits:

  - First commit tidies up the `receivedby.py` test (and speeds it up by factor of two)
  - Second commit changes getreceivedbyaddress to return error if the address is not found in wallet, and adds test to `receivedby.py`

Tree-SHA512: e41342dcbd037a6b440cbe4ecd3b8ed589e18e477333f0d866f3564e948e0f5231e497d5ffb66da4e6680eb772d9f0cf839125098bb68b92d04a5ee35c6c0a81
2017-11-11 12:35:44 -05:00
Pieter Wuille
033c78671b
Merge #11258: [rpc] Add initialblockdownload to getblockchaininfo
11413646b [trivial] (whitespace only) fix getblockchaininfo alignment (John Newbery)
bd9c18171 [rpc] Add initialblockdownload to getblockchaininfo (John Newbery)

Pull request description:

  Exposing whether the node is in IBD would help for testing, and may be useful in general, particularly for developers.

  First discussed in #10357 here: https://github.com/bitcoin/bitcoin/pull/10357#pullrequestreview-59963870

  > ... we could simplify this (and possibly other) tests by just adding a way to know if a node is in IBD. I'd like to do that, but I'm not sure it makes sense to complicate this PR with discussion over how that information should be made available. Eg it's not clear to me that the notion of being in IBD is worth exposing to the casual user, versus a hidden rpc call or something, since the definition has changed over time, and may continue to change in the future. But I still do agree that at least for testing purposes it would be far simpler to expose the field somehow...

   This PR currently implements the simplest way of doing this: adding an `initialblockdownload` field to `getblockchaininfo`. Other approaches we could take:

  1. add a new debug RPC method that exposes `IBD` and potentially other information.
  2. add a parameter to `getblockchaininfo`, eg `debug_info`, which would cause it to return debug information including IBD
  3. add a query string to the url `?debug=true` which would cause RPCs to return additional debug information.

  I quite like the idea of (3). Feedback on these and other approaches very much welcomed!

  @sdaftuar @laanwj

Tree-SHA512: a6dedd47f8c9bd38769cc597524466250041136feb33500644b9c48d0ffe4e3eeeb2587b5bbc6420364ebdd2667df807fbb50416f9a7913bbf11a14ea86dc0d4
2017-11-10 16:12:22 -08:00
John Newbery
5e0ba8f8cd [wallet] getreceivedbyaddress should return error if address is not mine 2017-11-10 16:07:37 -05:00
John Newbery
ea0cd24f7d [tests] Tidy up receivedby.py
- Fix flake8 warnings
- Remove the useless get_sub_array_from_array() function
- Reduce runtime for receivedby.py by about half by only using two nodes
2017-11-10 15:23:17 -05:00
John Newbery
1135c796a0 [tests] Tidy up mininode.py module
Mostly move only. Adds a few extra comments.
2017-11-10 11:56:59 -05:00
John Newbery
f9cd9b1bfa [tests] Move test_framework Bitcoin primitives into separate module
mininode.py wildcard imports all names from primitives.py. This is
to avoid having to change all test scripts that import from mininode.py.
2017-11-10 11:56:45 -05:00
MarcoFalke
76b3349159
Merge #11468: [tests] Make comp test framework more debuggable
748157913 [tests] Make comp test framework more debuggable (John Newbery)

Pull request description:

  We should remove the comparison test framework entirely (see #10603).

  Until we do that, let's make it a bit more debuggable. Currently, if there's an assert in the framework, it's very difficult to track down where we are in the test generator. Make the logging a bit better to help with debugging.

  Before this PR:

  ```
  → ./p2p-fullblocktest.py
  2017-10-09 14:05:11.302000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testzdnax_yr
  2017-10-09 14:05:11.557000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11975
  2017-10-09 14:05:11.712000 TestFramework.comptool (INFO): Test 1: PASS
  2017-10-09 14:05:11.947000 TestFramework.comptool (INFO): Test 2: PASS
  2017-10-09 14:05:12.057000 TestFramework.comptool (INFO): Test 3: PASS
  2017-10-09 14:05:12.058000 TestFramework (ERROR): Assertion failed
  Traceback (most recent call last):
    File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main
      self.run_test()
    File "./p2p-fullblocktest.py", line 72, in run_test
      self.test.run()
    File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 306, in run
      assert test_number != 4
  AssertionError
  2017-10-09 14:05:12.059000 TestFramework (INFO): Stopping nodes
  2017-10-09 14:05:14.203000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testzdnax_yr
  2017-10-09 14:05:14.204000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testzdnax_yr/test_framework.log
  ```

  With this PR:

  ```
  → ./p2p-fullblocktest.py
  2017-10-09 14:03:54.069000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testuey7t3tf
  2017-10-09 14:03:54.329000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11783
  2017-10-09 14:03:54.383000 TestFramework.comptool (INFO): Running test 1: ./p2p-fullblocktest.py line 184
  2017-10-09 14:03:54.496000 TestFramework.comptool (INFO): Running test 2: ./p2p-fullblocktest.py line 193
  2017-10-09 14:03:54.758000 TestFramework.comptool (INFO): Running test 3: ./p2p-fullblocktest.py line 205
  2017-10-09 14:03:54.867000 TestFramework.comptool (INFO): Running test 4: ./p2p-fullblocktest.py line 208
  2017-10-09 14:03:54.867000 TestFramework (ERROR): Assertion failed
  Traceback (most recent call last):
    File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main
      self.run_test()
    File "./p2p-fullblocktest.py", line 72, in run_test
      self.test.run()
    File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 309, in run
      assert test_number != 4
  AssertionError
  2017-10-09 14:03:54.868000 TestFramework (INFO): Stopping nodes
  2017-10-09 14:03:56.950000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testuey7t3tf
  2017-10-09 14:03:56.950000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testuey7t3tf/test_framework.log
  ```

Tree-SHA512: 5525958b0098d661c281bd955c92c72bf81359464376e96d44e6c88f18aea57ba08270ecd564edda4b47d674e3b27d20e5b1060544bf8dd5c6a68103d2bb35b8
2017-11-10 10:08:24 -05:00
Wladimir J. van der Laan
1f4375f8e7
Merge #11580: Do not send (potentially) invalid headers in response to getheaders
725b79a [test] Verify node doesn't send headers that haven't been fully validated (Russell Yanofsky)
3788a84 Do not send (potentially) invalid headers in response to getheaders (Matt Corallo)

Pull request description:

  Nowhere else in the protocol do we send headers which are for
  blocks we have not fully validated except in response to getheaders
  messages with a null locator. On my public node I have not seen any
  such request (whether for an invalid block or not) in at least two
  years of debug.log output, indicating that this should have minimal
  impact.

Tree-SHA512: c1f6e0cdcdfb78ea577d555f9b3ceb1b4b60eff4f6cf313bfd8b576c9562d797bea73abc23f7011f249ae36dd539c715f3d20487ac03ace60e84e1b77c0c1e1a
2017-11-09 19:57:47 +01:00