Commit graph

402 commits

Author SHA1 Message Date
MarcoFalke
fa178a6385
[rpc] mining: Omit uninitialized currentblockweight, currentblocktx 2019-02-12 11:34:57 -05:00
João Barbosa
6440e61375 qa: Drop RPC connection if --usecli 2019-02-06 00:04:28 +00:00
MarcoFalke
5029e94f85
Merge #14519: tests: add utility to easily profile node performance with perf
13782b8ba8 docs: add perf section to developer docs (James O'Beirne)
58180b5fd4 tests: add utility to easily profile node performance with perf (James O'Beirne)

Pull request description:

  Adds a context manager to easily (and selectively) profile node performance during functional test execution using `perf`.

  While writing some tests, I encountered some odd bitcoind slowness. I wrote up a utility (`TestNode.profile_with_perf`) that generates performance diagnostics for a node by running `perf` during the execution of a particular region of test code.

  `perf` usage is detailed in the excellent (and sadly unmerged) https://github.com/bitcoin/bitcoin/pull/12649; all due props to @eklitzke.

  ### Example

  ```python
  with node.profile_with_perf("large-msgs"):
      for i in range(200):
          node.p2p.send_message(some_large_msg)
      node.p2p.sync_with_ping()
  ```

  This generates a perf data file in the test node's datadir (`/tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data`).

  Running `perf report` generates nice output about where the node spent most of its time while running that part of the test:

  ```bash
  $ perf report -i /tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data --stdio \
    | c++filt \
    | less

  # To display the perf.data header info, please use --header/--header-only options.
  #
  #
  # Total Lost Samples: 0
  #
  # Samples: 135  of event 'cycles:pp'
  # Event count (approx.): 1458205679493582
  #
  # Children      Self  Command          Shared Object        Symbol
  # ........  ........  ...............  ...................  ........................................................................................................................................................................................................................................................................
  #
      70.14%     0.00%  bitcoin-net      bitcoind             [.] CNode::ReceiveMsgBytes(char const*, unsigned int, bool&)
                  |
                  ---CNode::ReceiveMsgBytes(char const*, unsigned int, bool&)

      70.14%     0.00%  bitcoin-net      bitcoind             [.] CNetMessage::readData(char const*, unsigned int)
                  |
                  ---CNetMessage::readData(char const*, unsigned int)
                     CNode::ReceiveMsgBytes(char const*, unsigned int, bool&)

      35.52%     0.00%  bitcoin-net      bitcoind             [.] std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&)
                  |
                  ---std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&)
                     CNetMessage::readData(char const*, unsigned int)
                     CNode::ReceiveMsgBytes(char const*, unsigned int, bool&)

  ...
  ```

Tree-SHA512: 9ac4ceaa88818d5eca00994e8e3c8ad42ae019550d6583972a0a4f7b0c4f61032e3d0c476b4ae58756bc5eb8f8015a19a7fc26c095bd588f31d49a37ed0c6b3e
2019-02-05 17:40:16 -05:00
Andrew Chow
2e023419c5 tests: unify RPC argument to cli argument conversion and handle dicts and lists
When running tests with --usecli, unify the conversion from argument objects to
strings using a new function arg_to_cli(). This fixes boolean arguments when
using named arguments.

Also use json.dumps() to get the string values for arguments that are dicts and
lists so that bitcoind's JSON parser does not become confused.
2019-01-31 12:37:33 -05:00
MarcoFalke
252fd15add
Merge #13926: [Tools] bitcoin-wallet - a tool for creating and managing wallets offline
3c3e31c3a4 [tests] Add wallet-tool test (João Barbosa)
49d2374acf [tools] Add wallet inspection and modification tool (Jonas Schnelli)

Pull request description:

  Adds an offline tool `bitcoin-wallet-tool` for wallet creation and maintenance.

  Currently this tool can create a new wallet file, display information on an existing wallet, and run the salvage and zapwallettxes maintenance tasks on an existing wallet. It can later be extended to support other common wallet maintenance tasks.

  Doing wallet maintenance tasks in an offline tool makes much more sense (and is potentially safer) than having to spin up a full node.

Tree-SHA512: 75a28b8a58858d9d76c7532db40eacdefc5714ea5aab536fb1dc9756e2f7d750d69d68d59c50a68e633ce38fb5b8c3e3d4880db30fe01561e07ce58d42bceb2b
2019-01-31 11:07:51 -05:00
João Barbosa
3c3e31c3a4 [tests] Add wallet-tool test
Original tests by João Barbosa <joao.paulo.barbosa@gmail.com>

Additional contribution by John Newbery <john@johnnewbery.com>
2019-01-30 16:26:56 -05:00
MarcoFalke
fa3745bda8
qa: Add tests for invalid message headers 2019-01-24 17:08:22 -05:00
Gregory Sanders
1e7f741745 remove some magic mining constants in functional tests 2019-01-24 09:48:34 -05:00
James O'Beirne
58180b5fd4 tests: add utility to easily profile node performance with perf
Introduces `TestNode.profile_with_perf()` context manager which
samples node execution to produce profiling data.

Also introduces a test framework flag, `--perf`, which will run
perf on all nodes for the duration of a given test.
2019-01-22 08:55:55 -05:00
MarcoFalke
1cfbb16b5d
Merge #15108: [tests] tidy up wallet_importmulti.py
2d5f1ea2e3 [tests] move wallet util functions to wallet_util.py (John Newbery)
6be64ef02c [tests] tidy up wallet_importmulti.py (John Newbery)

Pull request description:

  Cherry picks un-merged commits from #14952, which "fixes review comments from @ryanofsky here: https://github.com/bitcoin/bitcoin/pull/14886#pullrequestreview-183772779"

Tree-SHA512: 5f389196b0140d013a533d500f1812786a3a5cfb65980e13eaeacc459fddb55f43d05da3ab5e7cc8c997f26c0b667eed081ab6de2d125e631c70a7dd4c06e350
2019-01-09 16:58:46 -05:00
Wladimir J. van der Laan
c6806ee869
Merge #15059: test: Add basic test for BIP34
fab17e8272 test: Add basic test for BIP34 (MarcoFalke)

Pull request description:

  BIP34 was disabled for testing, which explains why it had no test.

  Fix that by enabling it and adding a test.

Tree-SHA512: 9cb5702d474117ce6420226eb93ee09d6fb5fc856fabc8b67abe56a088cd727674e0e5462000e1afa83b911374036f90abdbdde56a8c236a75572ed47e10a00f
2019-01-08 15:53:39 +01:00
John Newbery
2d5f1ea2e3 [tests] move wallet util functions to wallet_util.py
Adds a new wallet_util.py module and moves generic helper functions
there:

- get_key
- get_multisig
- test_address
2019-01-04 17:49:17 -08:00
Wladimir J. van der Laan
df894fa69a
Merge #14457: test: add invalid tx templates for use in functional tests
59e387705c test: add invalid tx templates for use in functional tests (James O'Beirne)

Pull request description:

  This change adds a list of `CTransaction`-generating templates which each correspond to a specific type of invalid transaction. We then use this list to test for a wider variety of invalid tx types in `p2p_invalid_tx.py` and `feature_block.py`.

  Consolidating all invalid tx types will allow us to more easily cover all tx reject cases from a variety of tests without repeating ourselves. Validation logic doesn't differ much between mempool and block acceptance, but there *is* a difference and we should be sure we're testing both comprehensively.

  Right now, I've only added templates covering the tx reject types listed below but if this approach seems worthwhile I will expand the list to be fully comprehensive.
  ```
  bad-txns-in-belowout
  bad-txns-inputs-duplicate
  bad-txns-too-many-sigops
  bad-txns-vin-empty
  bad-txns-vout-empty
  bad-txns-vout-negative
  ```

Tree-SHA512: 05407f4a953fbd7c44c08bb49bb989cefd39a2b05ea00f5b3c92197a3f05e1b302f789e33832445734220e1c333d133aba385740b77b84139b170c583471ce20
2019-01-02 14:44:14 +01:00
MarcoFalke
2741b2b6f4
Merge #15026: [test] Rename rpc_timewait to rpc_timeout
4999992c34 whitespace: Split ~300 char line into multiple ones (MarcoFalke)
fa71b38168 scripted-diff: Rename rpc_timewait to rpc_timeout (MarcoFalke)
fa3e5786d0 scripted-diff: Remove unused 'split' parameter to setup_network (MarcoFalke)

Pull request description:

  This is a bugfix, since wallet_dump currently uses the wrong name:

  18857b4c40/test/functional/wallet_dump.py (L89-L92)

  Rename all to the same name with a scripted diff (and some unrelated cleanups).

Tree-SHA512: 338ddd20dae12e6cf7aa7adbcfb239cf648017a1572b373f8431fecb184bd2a65492846d81e75a023864d9e41c94afb53044c16b79651a5937d34a5a6b772f81
2018-12-29 20:18:50 +01:00
MarcoFalke
fab17e8272
test: Add basic test for BIP34 2018-12-29 19:39:37 +01:00
MarcoFalke
96d912bea8
Merge #14738: Tests: Fix running wallet_listtransactions.py individually through test_runner.py
2474de0265 Fix running individually through test_runner.py, as suggested by @MarcoFalke (#14732) (Kristaps Kaupe)

Pull request description:

  As suggested by @MarcoFalke. Resolves #14732.

Tree-SHA512: b4e400ba06075e218dbd97d0390845f1c55be42a2b6fd70513381318cfc2693473ba1d0f9d7f379a96939d1960b53801fad7c02e06bddc50c5a835ad024c37ef
2018-12-29 18:19:27 +01:00
MarcoFalke
4999992c34
whitespace: Split ~300 char line into multiple ones 2018-12-22 17:26:49 +01:00
MarcoFalke
fa71b38168
scripted-diff: Rename rpc_timewait to rpc_timeout
-BEGIN VERIFY SCRIPT-
sed -i -e 's/self.rpc_timewait/self.rpc_timeout/g' $(git grep -l self.rpc_timewait)
-END VERIFY SCRIPT-
2018-12-22 17:23:25 +01:00
MarcoFalke
b53573e5c6
Merge #14951: Revert "tests: Support calling add_nodes more than once"
fa4b8c90d3 test: add_nodes can only be called once after set_test_params (MarcoFalke)
faa831102a Revert "tests: Support calling add_nodes more than once" (MarcoFalke)

Pull request description:

  Writing tests should be straightforward and with little side-effects as possible.

  I don't see how this is needed and can not be achieved with `self.num_nodes` (and `self.extra_args` et al.)

Tree-SHA512: 83a67f2cba9d97e21d80847ff405a4633fcb0d5674486efa57ee1813e46efe8709ae0fb462b8339a01ebeca5c4f2d29ecb1807d648b8fd9ee8ce336b08d580a8
2018-12-14 13:02:10 -05:00
MarcoFalke
fa4b8c90d3
test: add_nodes can only be called once after set_test_params 2018-12-13 15:08:38 -05:00
MarcoFalke
faa831102a
Revert "tests: Support calling add_nodes more than once"
This reverts commit 98a1846b00.
2018-12-13 12:06:38 -05:00
Wladimir J. van der Laan
914faf1784
Merge #14805: tests: Support calling add_nodes more than once
98a1846b00 tests: Support calling add_nodes more than once (Steven Roose)

Pull request description:

  Ran into this while writing [a multi-chain test for Elements](https://github.com/ElementsProject/elements/pull/458) where I call this method more than once.

Tree-SHA512: f2d698fcb560552aa5d81a4c3fbf40b7269b228b34d85a118291649ef83f8c0a30cd82a28d418237b55893bcecd538046b704e64a4d8a41f2c0aef8033dc83e5
2018-12-13 15:04:47 +01:00
Wladimir J. van der Laan
7202ae2e1b
Merge #14926: test: consensus: Check that final transactions are valid
aaaa8eb1ed test: consensus: Check that final transactions are valid (MarcoFalke)
fae3617d79 test: Correctly deserialize without witness (MarcoFalke)

Pull request description:

  There is no check that checks that final transactions are valid, i.e. the consensus rules could be changed (accidentally) with none of the tests failing.

Tree-SHA512: 48f4c24bfcc525ddbc1bfe8c37131953b464823428c1f7a278ba6d98b98827b6b84a8eb2b33396bfb5b8cc4012b7cc1cd771637f405ea20beddae001c22aa290
2018-12-13 14:22:31 +01:00
Wladimir J. van der Laan
84dc252a02
Merge #14884: Travis: enforce Python 3.4 support through linter
31926ee8cf [test] functional framework: add CScript hex() for Python 3.4 (Sjors Provoost)
74ce326831 [test] Travis: enforce Python 3.4 support in functional tests (Sjors Provoost)

Pull request description:

  The minimum supported version of Python is 3.4 according to [dependencies.md](https://github.com/bitcoin/bitcoin/blob/master/doc/dependencies.md). This PR makes the Travis linter use this version in order to catch accidental use of modern syntax.

Tree-SHA512: 71b2c102be72b135a8ba049378d66875760f20a04a657102a399240c5c2b2ddbdfa7d5ab4c0c0242ecc3259e0ee8eb2273f331bc5eb824f4ae4c3cc58aea37ac
2018-12-13 13:17:19 +01:00
Sjors Provoost
31926ee8cf
[test] functional framework: add CScript hex() for Python 3.4
test/functional/wallet_importmulti.py failed with:
AttributeError: 'CScript' object has no attribute 'hex'
2018-12-12 11:09:04 +01:00
MarcoFalke
fae3617d79
test: Correctly deserialize without witness 2018-12-11 13:23:24 -05:00
Daniel Ingram
17b55202da Compare to None with is/is not 2018-12-10 15:11:37 -05:00
MarcoFalke
2753285be7
Merge #14788: tests: Possible fix the permission error when the tests open the cookie file
d6b3790d1a tests: check readability of cookie file (Chun Kuan Lee)

Pull request description:

  This PR would wait until the `.cookie` file is readable
  Possible fix no. 5 `PermissionError` in #14446

Tree-SHA512: e7055c7ca26a6eadbbe19e4eef08ffee61cd17de79b30af2f0d090f0ad81ca24815e3c7e034e5e30d47c580bb0b221b3955e9ff2fcec2274fbf7b9232ab0cdc7
2018-12-07 13:03:47 -05:00
Wladimir J. van der Laan
a88bd3186d
Merge #14670: http: Fix HTTP server shutdown
28479f926f qa: Test bitcond shutdown (João Barbosa)
8d3f46ec39 http: Remove timeout to exit event loop (João Barbosa)
e98a9eede2 http: Remove unnecessary event_base_loopexit call (João Barbosa)
6b13580f4e http: Unlisten sockets after all workers quit (João Barbosa)
18e9685816 http: Send "Connection: close" header if shutdown is requested (João Barbosa)
02e1e4eff6 rpc: Add wait argument to stop (João Barbosa)

Pull request description:

  Fixes #11777. Reverts #11006. Replaces #13501.

  With this change the HTTP server will exit gracefully, meaning that all requests will finish processing and sending the response, even if this means to wait more than 2 seconds (current time allowed to exit the event loop).

  Another small change is that connections are accepted even when the server is stopping, but HTTP requests are rejected. This can be improved later, especially if chunked replies are implemented.

  Briefly, before this PR, this is the order or events when a request arrives (RPC `stop`):
   1. `bufferevent_disable(..., EV_READ)`
   2. `StartShutdown()`
   3. `evhttp_del_accept_socket(...)`
   4. `ThreadHTTP` terminates (event loop exits) because there are no active or pending events thanks to 1. and 3.
   5. client doesn't get the response thanks to 4.

  This can be verified by applying
  ```diff
       // Event loop will exit after current HTTP requests have been handled, so
       // this reply will get back to the client.
       StartShutdown();
  +    MilliSleep(2000);
       return "Bitcoin server stopping";
   }
  ```
  and checking the log output:
  ```
      Received a POST request for / from 127.0.0.1:62443
      ThreadRPCServer method=stop user=__cookie__
      Interrupting HTTP server
  **  Exited http event loop
      Interrupting HTTP RPC server
      Interrupting RPC
      tor: Thread interrupt
      Shutdown: In progress...
      torcontrol thread exit
      Stopping HTTP RPC server
      addcon thread exit
      opencon thread exit
      Unregistering HTTP handler for / (exactmatch 1)
      Unregistering HTTP handler for /wallet/ (exactmatch 0)
      Stopping RPC
      RPC stopped.
      Stopping HTTP server
      Waiting for HTTP worker threads to exit
      msghand thread exit
      net thread exit

      ... sleep 2 seconds ...

      Waiting for HTTP event thread to exit
      Stopped HTTP server
  ```

  For this reason point 3. is moved right after all HTTP workers quit. In that moment HTTP replies are queued in the event loop which keeps spinning util all connections are closed. In order to trigger the server side close with keep alive connections (implicit in HTTP/1.1) the header `Connection: close` is sent if shutdown was requested. This can be tested by
  ```
  bitcoind -regtest
  nc localhost 18443
  POST / HTTP/1.1
  Authorization: Basic ...
  Content-Type: application/json
  Content-Length: 44

  {"jsonrpc": "2.0","method":"stop","id":123}
  ```

  Summing up, this PR:
   - removes explicit event loop exit — event loop exits once there are no active or pending events
   - changes the moment the listening sockets are removed — explained above
   - sends header `Connection: close` on active requests when shutdown was requested which is relevant when it's a persistent connection (default in HTTP 1.1) — libevent is aware of this header and closes the connection gracefully
   - removes event loop explicit break after 2 seconds timeout

Tree-SHA512: 4dac1e86abe388697c1e2dedbf31fb36a394cfafe5e64eadbf6ed01d829542785a8c3b91d1ab680d3f03f912d14fc87176428041141441d25dcb6c98a1e069d8
2018-12-06 17:43:07 +01:00
Wladimir J. van der Laan
0257062e50
Merge #14816: Add CScriptNum decode python implementation in functional suite
2012d4df2 Add CScriptNum decode python implementation in functional suite (Gregory Sanders)

Pull request description:

  I needed this for reasons and thought it'd be good to upsteam it.

Tree-SHA512: 6ea89fa2a5f5a7759ba722f2b4ed5cd6423ebfff4e83ac8b8b5c935e6aa479684e626c5f41fa020816d2a9079a99af5564e30808594d5c13e3b51ec9b474926d
2018-12-04 11:28:53 +01:00
James O'Beirne
4aabadbf44 tests: have combine_logs default to most recent test dir 2018-11-29 17:41:12 -05:00
Gregory Sanders
2012d4df23 Add CScriptNum decode python implementation in functional suite 2018-11-29 08:31:48 -05:00
James O'Beirne
59e387705c test: add invalid tx templates for use in functional tests
Add templates for easily constructing different kinds of invalid
transactions and use them in feature_block and p2p_invalid_tx.
2018-11-27 17:53:53 -05:00
James O'Beirne
5a1f57646b qa: clean up assert_memory_usage_stable utility 2018-11-26 15:47:30 -05:00
Chun Kuan Lee
d6b3790d1a tests: check readability of cookie file 2018-11-24 02:32:38 +08:00
João Barbosa
28479f926f qa: Test bitcond shutdown 2018-11-23 16:54:22 +00:00
Steven Roose
98a1846b00 tests: Support calling add_nodes more than once 2018-11-21 18:03:01 +00:00
Kristaps Kaupe
2474de0265 Fix running individually through test_runner.py, as suggested by @MarcoFalke (#14732) 2018-11-16 22:26:07 +02:00
MarcoFalke
fa7da0617c
qa: Check specific reject reasons in feature_block 2018-11-13 11:05:24 -05:00
MarcoFalke
fa9ed38d57
test_node: get_mem_rss fixups 2018-11-08 18:05:09 -05:00
practicalswift
590a57fdec tests: Remove unused testing code 2018-11-07 18:07:49 +01:00
Wladimir J. van der Laan
024816d6cf
Merge #14522: tests: add invalid P2P message tests
d20a9fa13d tests: add tests for invalid P2P messages (James O'Beirne)
62f94d39f8 tests: add P2PConnection.send_raw_message (James O'Beirne)
5aa31f6ef2 tests: add utility to assert node memory usage hasn't increased (James O'Beirne)

Pull request description:

  - Adds `p2p_invalid_messages.py`: tests based on behavior for dealing with invalid and malformed P2P messages. Includes a test verifying that we can't DoS a node by spamming it with large invalid messages.
  - Adds `TestNode.assert_memory_usage_stable`: a context manager that allows us to ensure memory usage doesn't significantly increase on a node during some test.
  - Adds `P2PConnection.send_raw_message`: which allows us to construct and send messages with tweaked headers.

Tree-SHA512: 720a4894c1e6d8f1551b2ae710e5b06c9e4f281524623957cb01599be9afea82671dc26d6152281de0acb87720f0c53b61e2b27d40434d30e525dd9e31fa671f
2018-11-06 11:48:14 +01:00
James O'Beirne
62f94d39f8 tests: add P2PConnection.send_raw_message 2018-11-01 14:52:49 -04:00
James O'Beirne
5aa31f6ef2 tests: add utility to assert node memory usage hasn't increased
Adds a utility to get resident set size memory usage for a test
node and a context manager that allows assertions based upon
maximum memory use increase.
2018-11-01 14:52:46 -04:00
John Newbery
3fd7e76f6d [tests] Move deterministic address import to setup_nodes
This requires a small changes to a few tests, but means that
deterministic addresses will always be imported (unless setup_nodes
behaviour is explicitly overridden).
2018-11-01 12:53:49 -04:00
Wladimir J. van der Laan
b312579c69
Merge #14454: Add SegWit support to importmulti
c11875c590 Add segwit address tests for importmulti (MeshCollider)
201451b1ca Make getaddressinfo return solvability (MeshCollider)
1753d217ea Add release notes for importmulti segwit change (MeshCollider)
353c064596 Fix typo in test_framework/blocktools (MeshCollider)
f6ed748cf0 Add SegWit support to importmulti with some ProcessImport cleanup (MeshCollider)

Pull request description:

  Add support for segwit to importmulti, supports P2WSH, P2WPKH, P2SH-P2WPKH, P2SH-P2WSH. Adds a new `witnessscript` parameter which must be used for the witness scripts in the relevant situations.

  Also includes some tests for the various import types.

  ~Also makes the change in #14019 redundant, but cherry-picks the test from that PR to test the behavior (@achow101).~

  Fixes #12253, also addresses the second point in #12703, and fixes #14407

Tree-SHA512: 775a755c524d1c387a99acddd772f677d2073876b72403dcfb92c59f9b405ae13ceedcf4dbd2ee1d7a8db91c494f67ca137161032ee3a2071282eeb411be090a
2018-10-31 17:44:31 +01:00
MarcoFalke
fa78a2fc67
[tests] Test that nodes respond to getdata with notfound
If a node has not announced a tx at all, then it should respond to
getdata messages for that tx with notfound, to avoid leaking tx
origination privacy.
2018-10-26 07:45:12 -04:00
MeshCollider
353c064596 Fix typo in test_framework/blocktools 2018-10-25 09:30:57 +13:00
MarcoFalke
3668bb335c
Merge #14468: [wallet] Deprecate generate RPC method
ab9aca2bdf [rpc] add 'getnewaddress' hint to 'generatetoaddress' help text. (John Newbery)
c9f02955b2 [wallet] Deprecate the generate RPC method (John Newbery)
aab81720de [tests] Add generate method to TestNode (John Newbery)
c269209336 [tests] Small fixups before deprecating generate (John Newbery)

Pull request description:

  Deprecates the `generate` RPC method.

  For concept discussion, see #14299.

  Fixes #14299.

Tree-SHA512: 16a3b8b742932e4f0476c06b23de07a34d9d215b41d9272c1c9d1e39966b0c2406f17c5ab3cc568947620c08171ebe5eb74fd7ed4b62151363e305ee2937cc80
2018-10-23 18:15:21 -04:00
John Newbery
aab81720de [tests] Add generate method to TestNode
Adds a generate() method to the TestNode class in the test framework.
This method intercepts calls to generate, imports a dewterministic
private key to the node and then calls generatetoaddress to generate the
block to that address.

Note that repeated calls to importprivkey for the same private keys are
no-ops, so it's fine to call the generate() method many times.
2018-10-18 17:46:48 -04:00
James O'Beirne
94e21c1501 test: forward timeouts properly in send_blocks_and_test 2018-10-10 02:51:19 -04:00
MarcoFalke
d12e3d9acc
Merge #14324: qa: Run more tests with wallet disabled
faa4043c66 qa: Run more tests with wallet disabled (MarcoFalke)

Pull request description:

  Instead of skipping the whole test, only skip the wallet specific section of a test if the wallet is not compiled in. This is mostly an indentation change, so can be reviewed with `--ignore-all-space`.

Tree-SHA512: 5941a8b6b00dca5cf9438c5f6f010ba812115188a69e427d7ade4c1ab8cfe7a57c73daf52c66235dbb24b1cd9ab7c7a17c49bc23d931e041b605d79116a71f66
2018-10-08 01:42:02 -03:00
Chun Kuan Lee
62c304ea48 tests: Allow closed http server in assert_start_raises_init_error 2018-10-08 11:48:19 +08:00
practicalswift
db01839361 test: Add missing call to skip_if_no_cli() 2018-10-03 10:10:35 +02:00
MarcoFalke
b88dd7c2db
Merge #12246: Bugfix: Only run bitcoin-tx tests when bitcoin-tx is enabled
a2a04a5abb Bugfix: Only run bitcoin-tx tests when bitcoin-tx is enabled (Luke Dashjr)
92af71cea9 configure: Make it possible to build only one of bitcoin-cli or bitcoin-tx (Luke Dashjr)

Pull request description:

  Includes #5618 (which the reasons for rejecting no longer hold true)

Tree-SHA512: f30a8e4a2f70166b7cabef77c4674163b3a9da14c6a547d34f00d1056a19bf4d23e22851eea726fad2afc8735d5473ae91122c770b65ac3886663dc20e2c5b70
2018-09-27 11:13:51 -04:00
MarcoFalke
edcf29c9da
Merge #14305: Tests: enforce critical class instance attributes in functional tests, fix segwit test specificity
e460232876 Document fixed attribute behavior in critical test framework classes. (Justin Turner Arthur)
17b42f4122 Check for specific tx acceptance failures based on script signature (Justin Turner Arthur)
3a4449e9ad Strictly enforce instance attrs in critical functional test classes. (Justin Turner Arthur)
1d0ce94a54 Fix for incorrect version attr set on functional test segwit block. (Justin Turner Arthur)
ba923e32a0 test: Fix broken segwit test (practicalswift)

Pull request description:

  No extra attributes will be able to be added to instances of the C++ class ports or of other critical classes without causing an exception. Helps prevent adding or depending on attributes that aren't in the intended object structure. It may prevent issues such as the one fixed in bitcoin/bitcoin#14300.

  This request fixes the erroneous version attribute used in the p2p_segwit.py functional tests. This pull includes the commit from bitcoin/bitcoin#14300.

Tree-SHA512: 1b8c58e7aa0f71075ed5ff3e5be0a5182599108d8cd9bce682feac3b1842508124288e9335432b16a43f40f159c9710899e6d84af1b5868f48c947bc6f3e07ec
2018-09-27 11:09:46 -04:00
Justin Turner Arthur
3a4449e9ad Strictly enforce instance attrs in critical functional test classes.
Additionally, removed redundant parentheses and added PEP-8 compliant
spacing around those classes.
2018-09-26 21:50:42 -05:00
MarcoFalke
faa4043c66
qa: Run more tests with wallet disabled 2018-09-25 18:08:08 -04:00
MarcoFalke
990fc0de1a
Merge #14007: tests: Run functional test on Windows and enable it on Appveyor
661ac15a4a appveyor: Run functional tests on appveyor (Chun Kuan Lee)
2148c36b6e tests: Make it possible to run functional tests on Windows (Chun Kuan Lee)

Pull request description:

  This PR do the following things:
  - Make functional tests compatible with Windows
  - Print color output in functional tests for Windows 10
  - Run util and functional tests on appveyor
  - Do not run symlink tests on Windows

  Note:
  - The wallet_multiwallet.py fail is unrelated to the test framework, it's a bug related to c++ code or maybe dependencies. `bitcoind` would exit with 0xC0000005(Access violation) during shutdown occasionally. Disable this for now.
  - Not using `--failfast` because this is still in experimental. We should track if there is any other error.
  - Disable ZMQ tests because the python zmq library could cause access violation sometimes.
  - Disable `feature_notifications` because Bitcoin Core handles the command in different thread, whicha can cause a race condition.

Tree-SHA512: b76db137d264e62a5c130e1cbca7a2ca002a7a0f4153fa0b92c1ea6c9c09ef0533e11c49bdbd566c472d8ff59f245758feb5e5a6ec6cb6bb66a1c67bab5fa48a
2018-09-24 16:13:17 -04:00
sanket1729
0ca4c8b3c6 Changed functional tests which do not require wallets to run without
skipping  .Addreses #14216. Changed get_deterministic_priv_key() to a

named tuple
2018-09-17 08:25:10 -05:00
Chun Kuan Lee
2148c36b6e tests: Make it possible to run functional tests on Windows 2018-09-17 09:56:49 +08:00
Wladimir J. van der Laan
f09bc7ec98
Merge #12493: [wallet] Reopen CDBEnv after encryption instead of shutting down
c1dde3a949 No longer shutdown after encrypting the wallet (Andrew Chow)
d7637c5a3f After encrypting the wallet, reload the database environment (Andrew Chow)
5d296ac810 Add function to close all Db's and reload the databae environment (Andrew Chow)
a769461d5e Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow)

Pull request description:

  This is the replacement for #11678 which implements @ryanofsky's [suggestion](https://github.com/bitcoin/bitcoin/pull/11678#pullrequestreview-76464511).

  Shutting down the software was to prevent the BDB environment from writing unencrypted private keys to disk in the database log files, as was noted [here](https://bitcointalk.org/index.php?topic=51474.msg616068#msg616068). This PR replaces the shutdown behavior with a CDBEnv flush, close, and reopen which achieves the same effect: everything is cleanly flushed and closed, the log files are removed, and then the environment reopened to continue normal operation.

  To ensure that no unencrypted private keys are in the log files after encrypting the wallet, I wrote [this script](https://gist.github.com/achow101/7f7143e6c3d3fdc034d3470e72823e9d) to pull private keys from the original wallet file and searches for these keys in the log files (note that you will have to change your file paths to make it work on your own machine).

  As for concerns about private keys being written to slack space or being kept in memory, these behaviors no longer exist after the original wallet encryption PR and the shutting down solution from 2011.

  cc @ryanofsky

Tree-SHA512: 34b894283b0677a873d06dee46dff8424dec85a2973009ac9b84bcf3d22d05f227c494168c395219d9aee3178e420cf70d4b3eeacc9785aa86b6015d25758e75
2018-09-14 10:43:35 +02:00
MarcoFalke
efb11d7c05
Merge #14179: qa: Fixups to "Run all tests even if wallet is not compiled"
fa8433e379 qa: Remove unneded import_deterministic_coinbase_privkeys overwrite, add comments (MarcoFalke)
e413c2ddd1 qa: Fix codespell error and have lint-spelling error instead of warn (MarcoFalke)

Pull request description:

  Currently the functional tests require the wallet module to be compiled into the Bitcoin Core executable. For example the premine (or datadir cache) to speed up tests when run in parallel would mine a bunch of blocks and store the private keys to sign the coinbase tx outputs in a wallet. There is no need to have the overhead of the whole wallet module by using keys that are deterministic for all runs.

  Note that this change most likely requires the `./test/cache/` to be cleared.

Tree-SHA512: 9ce26036b0e10f0f888f66a1e50be6a357343f9ffb302ae24a7bb3df2f083a31702ef308b738a03b08a1b623aeddac5d6563dc1b15078c0357b7dafad7808ec3
2018-09-13 17:45:42 -04:00
MarcoFalke
e413c2ddd1 qa: Fix codespell error and have lint-spelling error instead of warn 2018-09-13 08:53:05 -04:00
Luke Dashjr
92af71cea9 configure: Make it possible to build only one of bitcoin-cli or bitcoin-tx 2018-09-13 11:48:23 +00:00
Wladimir J. van der Laan
288ddf4ff5
Merge #14180: qa: Run all tests even if wallet is not compiled
fac9539836 qa: Run all tests even if wallet is not compiled (MarcoFalke)
faa669cbcd qa: Premine to deterministic address with -disablewallet (MarcoFalke)

Pull request description:

  Currently the test_runner would exit if the wallet was not compiled into the Bitcoin Core executable. However, a lot of the tests run without the wallet just fine and there is no need to globally require the wallet to run the tests.

Tree-SHA512: 63177260aa29126fd20f0be217a82b10b62288ab846f96f1cbcc3bd2c52702437703475d91eae3f8d821a3149fc62b725a4c5b2a7b3657b67ffcbc81532a03bb
2018-09-13 13:06:07 +02:00
MarcoFalke
fac9539836
qa: Run all tests even if wallet is not compiled 2018-09-10 17:53:21 -04:00
MarcoFalke
faa669cbcd
qa: Premine to deterministic address with -disablewallet 2018-09-10 17:51:46 -04:00
Wladimir J. van der Laan
cb25cd6aa1
Merge #14119: qa: Read reject reasons from debug log, not p2p messages
fac3e22b18 qa: Read reject reasons from debug log, not p2p messages (MarcoFalke)

Pull request description:

  For local testing we don't need to rely on p2p messages just to assert a reject reason.

  Replace reading p2p messages with reading from the debug log file.

Tree-SHA512: fa59598ecf5e00cfb420ef1892d90aa415501fd882e1c608894dc577b0d00e93a442326d3a9167fef77d26aafbe345b730b49109982ccad68a5942384564a90b
2018-09-08 07:09:07 +02:00
MarcoFalke
adf27b531a
Merge #13954: Warn (don't fail!) on spelling errors. Fix typos reported by codespell.
f8a81f73ac lint: Add spell check linter (codespell) (practicalswift)
ada356208e Fix typos reported by codespell (practicalswift)

Pull request description:

  * Check for common misspellings using `codespell`.
  * Fix recently introduced typos reported by `codespell`.

Tree-SHA512: 9974c0e640b411c7d0ebc5b45de253c19bac7fe3002cd98601ff8da8db584224c2fd7d331aee3df612c9f2cfef540d647a9b4c5a1a73fd208dc93ce4bf9e5e3e
2018-09-05 18:13:05 -04:00
practicalswift
ada356208e Fix typos reported by codespell 2018-09-04 13:11:26 +02:00
Kvaciral
8dfc2f30de Test rpc_help.py failed: Check whether ZMQ is enabled or not. 2018-09-01 01:25:34 +02:00
MarcoFalke
fac3e22b18
qa: Read reject reasons from debug log, not p2p messages 2018-08-31 10:27:19 -04:00
MarcoFalke
fa782a308d
qa: Use named args in some tests 2018-08-29 13:06:02 -04:00
MarcoFalke
55c18a4530
Merge #14024: qa: Add TestNode::assert_debug_log
fa3e9f7627 qa: Add TestNode::assert_debug_log (MarcoFalke)

Pull request description:

  Closes #13006

Tree-SHA512: 8e2d2331039d70525a425aad65a4eaf9b83fb1f85a4260d69453176f04dbede6dd9b7bb4d5f089b46cf8f8c98571aa6ba7fac8fa6847bb3bdf6a6ad21a27b1a7
2018-08-24 14:07:24 -04:00
MarcoFalke
fa3e9f7627
qa: Add TestNode::assert_debug_log 2018-08-24 12:30:13 -04:00
MarcoFalke
fa74d3d720
qa: Remove unused deserialization code in msg_version 2018-08-23 10:05:43 -04:00
MarcoFalke
fa5099ceb7
p2p: Remove dead code for nVersion=10300 2018-08-22 09:22:04 -04:00
Carl Dong
8845c8aea6 tests: Replace usage of tostring() with tobytes()
tostring() is deprecated as of python 3.7 and results in stderr output
causing tests to fail
2018-08-13 19:05:31 -07:00
MarcoFalke
bffb35f876
Merge #13054: tests: Enable automatic detection of undefined names in Python tests scripts. Remove wildcard imports.
68400d8b96 tests: Use explicit imports (practicalswift)

Pull request description:

  Enable automatic detection of undefined names in Python tests scripts. Remove wildcard imports.

  Wildcard imports make it unclear which names are present in the namespace, confusing both readers and many automated tools.

  An additional benefit of not using wildcard imports in tests scripts is that readers of a test script then can infer the rough testing scope just by looking at the imports.

  Before this commit:

  ```
  $ contrib/devtools/lint-python.sh | head -10
  ./test/functional/feature_rbf.py:8:1: F403 'from test_framework.util import *' used; unable to detect undefined names
  ./test/functional/feature_rbf.py:9:1: F403 'from test_framework.script import *' used; unable to detect undefined names
  ./test/functional/feature_rbf.py:10:1: F403 'from test_framework.mininode import *' used; unable to detect undefined names
  ./test/functional/feature_rbf.py:15:12: F405 bytes_to_hex_str may be undefined, or defined from star imports: test_framework.mininode, test_framework.script, test_framework.util
  ./test/functional/feature_rbf.py:17:58: F405 CScript may be undefined, or defined from star imports: test_framework.mininode, test_framework.script, test_framework.util
  ./test/functional/feature_rbf.py:25:13: F405 COIN may be undefined, or defined from star imports: test_framework.mininode, test_framework.script, test_framework.util
  ./test/functional/feature_rbf.py:26:31: F405 satoshi_round may be undefined, or defined from star imports: test_framework.mininode, test_framework.script, test_framework.util
  ./test/functional/feature_rbf.py:26:60: F405 COIN may be undefined, or defined from star imports: test_framework.mininode, test_framework.script, test_framework.util
  ./test/functional/feature_rbf.py:30:41: F405 satoshi_round may be undefined, or defined from star imports: test_framework.mininode, test_framework.script, test_framework.util
  ./test/functional/feature_rbf.py:30:68: F405 COIN may be undefined, or defined from star imports: test_framework.mininode, test_framework.script, test_framework.util
  $
  ```

  After this commit:

  ```
  $ contrib/devtools/lint-python.sh | head -10
  $
  ```

Tree-SHA512: 3f826d39cffb6438388e5efcb20a9622ff8238247e882d68f7b38609877421b2a8e10e9229575f8eb6a8fa42dec4256986692e92922c86171f750a0e887438d9
2018-08-13 08:28:06 -04:00
practicalswift
68400d8b96 tests: Use explicit imports 2018-08-13 14:13:39 +02:00
MarcoFalke
b8eb0dfde4
Merge #13928: qa: blocktools enforce named args for amount
cf9ed307e6 qa: blocktools enforce named args for amount (MarcoFalke)

Pull request description:

  Since  #13669 changed some signatures, I think it might be worthwhile to enforce named args for primitive types such as amounts.

Tree-SHA512: 2733e7b6a20590b54bd54e81a09e3f5e2fadf4390bed594916b70729bcf485b048266012c1203369e0968032a2c6a2719107ac17ee925d8939af3df916eab1a6
2018-08-13 07:30:38 -04:00
Kvaciral
5654efb187 Ported usage of deprecated optparse module to argparse module 2018-08-12 00:23:53 +02:00
MarcoFalke
1b04b55f2d
Merge #13867: qa: Make extended tests pass on native Windows
fafe73a626 qa: Raise feature_help timeout to 5s (MarcoFalke)
faabd7bc47 qa: Use files for stdout/stderr to support Windows (MarcoFalke)
facb56ffaf qa: Run gen_rpcauth with sys.executable (MarcoFalke)
fada8966c5 qa: Close stdout and stderr file when node stops (MarcoFalke)

Pull request description:

  ### qa: Close stdout and stderr file when node stops

  Since these files are potentially deleted by the test framework for cleanup, they should be closed first. Otherwise this will lead to errors on Windows when the tests finish successfully.

  Side note: After the patch, it is no longer possible to reopen the file on Windows (see https://docs.python.org/3/library/tempfile.html#tempfile.NamedTemporaryFile)

  ### qa: Run gen_rpcauth with sys.executable

  Similar to `test_runner.py`, the `sys.executable` needs to be passed down into subprocesses to pass on native Windows. (Should have no effect on Linux)

  ###   qa: Use files for stdout/stderr to support Windows

  It seems that using PIPE is not supported on Windows. Also, it is easier to just use the files that capture the stdout and stderr within the test node class.

Tree-SHA512: ec675012b10705978606b7fcbdb287c39a8e6e3732aae2fa4041d963a3c6993c6eac6a9a3cbd5479514e7d8017fe74c12235d1ed6fed2e8af8f3c71981e91864
2018-08-11 07:01:37 -04:00
MarcoFalke
a04888a075
Merge #13915: [qa] Add test for max number of entries in locator
fa85c985ed qa: Add p2p_invalid_locator test (MarcoFalke)

Pull request description:

  Should not be merged *before* #13907

Tree-SHA512: a67ca407854c421ed20a184d0b0dc90085aed3e3431d9652a107fa3022244767e67f67e50449b7e95721f56906836b134615875f28a21e8a012eb22cfe6a66a5
2018-08-11 06:59:14 -04:00
MarcoFalke
fa85c985ed
qa: Add p2p_invalid_locator test 2018-08-10 14:09:19 -04:00
Andrew Chow
c1dde3a949 No longer shutdown after encrypting the wallet
Since the database environment is flushed, closed, and reopened during
EncryptWallet, there is no need to shut down the software anymore.
2018-08-09 11:28:33 -07:00
MarcoFalke
cf9ed307e6
qa: blocktools enforce named args for amount 2018-08-09 13:25:36 -04:00
MarcoFalke
f66e1c793e
Merge #13669: Tests: Cleanup create_transaction implementations
44bbceeef1 [Tests] Cleanup feature_block.py, remove unnecessary PreviousSpendableOutput object (Conor Scott)
736f941424 [Tests] Cleanup extra instances of create_transaction (Conor Scott)
157651855f [Tests] Rename create_tx and move to blocktools.py (Conor Scott)

Pull request description:

  There currently exist seven ([1](https://github.com/bitcoin/bitcoin/blob/master/test/functional/feature_cltv.py#L52-L60), [2](https://github.com/bitcoin/bitcoin/blob/master/test/functional/feature_csv_activation.py#L88-L95) [3](https://github.com/bitcoin/bitcoin/blob/master/test/functional/feature_dersig.py#L40-L48), [4](https://github.com/bitcoin/bitcoin/blob/master/test/functional/feature_nulldummy.py#L100-L108), [5](https://github.com/bitcoin/bitcoin/blob/master/test/functional/test_framework/util.py#L529-L535), [6](https://github.com/bitcoin/bitcoin/blob/master/test/functional/test_framework/blocktools.py#L120-L129), [7](https://github.com/bitcoin/bitcoin/blob/master/test/functional/feature_block.py#L1218-L1220)) implementations of a function called something similar to `create_transaction` in the functional tests, some of which are exact copies of each other.

  This PR aims to clean this up into [three different cases implemented in blocktools.py](https://github.com/conscott/bitcoin/blob/create_tx_cleanup/test/functional/test_framework/blocktools.py#L121-L149)
  1. `create_tx_with_script`: Return transaction object spending generic tx output optionally specifying scriptSig and scriptPubKey
  2. `create_transaction`: Return transaction object spending coinbase tx
  2. `create_raw_transaction`: Return raw transaction (hex string) spending coinbase tx

  I am not committed to any of these function names, so I'll gladly take suggestions on there.

  Additionally there are some related cleanups to feature_block.py tests, specifically removing the [PreviousSpendableOutput](https://github.com/conscott/bitcoin/blob/master/test/functional/feature_block.py#L51-L54) object, which seems like an unnecessary layer given that every instance spends the 0 output.

Tree-SHA512: 63c6233b6f0942c81ba1ca67ea6770809b8c9409314c6d4cf8e5a3991cb9ee92b22bebe88c0dde45cd71e754eb351230c4c404b70ff118f5f43c034452ada65c
2018-08-09 12:09:37 -04:00
MarcoFalke
8ac7125d59
Merge #13916: qa: wait_for_verack by default
fa5587fe71 qa: wait_for_verack by default (MarcoFalke)

Pull request description:

  This removes the need to do so manually every time a connection is added.

Tree-SHA512: a46c92cb4df41e30778b42b9fd3dcbd8d2d82aa7503d1213cb1c1165034f648d8caee01c292e2d87d05b0f71696996eef5be8a753f35ab49e5f66b0e3bf29f21
2018-08-09 08:07:28 -04:00
Conor Scott
736f941424 [Tests] Cleanup extra instances of create_transaction 2018-08-09 12:58:36 +02:00
Conor Scott
157651855f [Tests] Rename create_tx and move to blocktools.py 2018-08-09 12:58:35 +02:00
MarcoFalke
fa5587fe71
qa: wait_for_verack by default 2018-08-08 17:22:35 -04:00
Wladimir J. van der Laan
78dae8cacc
Merge #13780: 0.17: Pre-branch maintenance
3fc20632a3 qt: Set BLOCK_CHAIN_SIZE = 220 (DrahtBot)
2b6a2f4a28 Regenerate manpages (DrahtBot)
eb7daf4d60 Update copyright headers to 2018 (DrahtBot)

Pull request description:

  Some trivial maintenance to avoid having to do it again after the 0.17 branch off.

  (The scripts to do this are in `./contrib/`)

Tree-SHA512: 16b2af45e0351b1c691c5311d48025dc6828079e98c2aa2e600dc5910ee8aa01858ca6c356538150dc46fe14c8819ed8ec8e4ec9a0f682b9950dd41bc50518fa
2018-08-08 13:55:27 +02:00
MarcoFalke
faabd7bc47
qa: Use files for stdout/stderr to support Windows 2018-08-03 17:29:46 -04:00
MarcoFalke
fada8966c5
qa: Close stdout and stderr file when node stops 2018-08-03 13:43:59 -04:00
Wladimir J. van der Laan
489b51b08e
Merge #13823: qa: quote path in authproxy for external multiwallets
fa67505e1e qa: Quote wallet name for rpc path (MarcoFalke)

Pull request description:

  When using external multiwallets they are specified by their full path which might contain non-ascii characters (e.g. umlauts or emojis).

  Fix this by url-quoting the path.

Tree-SHA512: 7cc66514579d9f602f88a6817c5ab43a44c6d3711df452dc904173f0bc34e2c0b2c34877497f06b61f6720c532fa183053f54622dc454e316c89cee7eaa72463
2018-08-02 16:00:04 +02:00
MarcoFalke
fa67505e1e
qa: Quote wallet name for rpc path 2018-08-02 08:25:19 -04:00
MarcoFalke
fa5b440971
qa: Extract rpc_timewait as test param
Also increase it for wallet_dump and wallet_groups
2018-08-01 14:37:07 -04:00
DrahtBot
eb7daf4d60 Update copyright headers to 2018 2018-07-27 07:15:02 -04:00