Commit graph

55 commits

Author SHA1 Message Date
John Newbery
9c92c8c827 [tests] Remove Comparison Test Framework 2018-04-02 14:04:07 -04:00
Wladimir J. van der Laan
6d36f599f8
Merge #12076: qa: Use node.datadir instead of tmpdir in test framework
c8330d4 qa: Use node.datadir instead of tmpdir in test framework (MarcoFalke)

Pull request description:

  Commit c53c9831ee introduced the utility function `get_datadir_path`, however not all places in the code use this util function. Using the util function everywhere makes it easier to review pull requests related to the datadir.

  This commit replaces datadir path creation with the `datadir` member of `TestNode`, which itself uses `get_datadir_path`.

Tree-SHA512: c75707ab7149d732a6d56152a5813138a33459d3d07577b60b89f2a207c83b7663fac5f203593677c9892d1c23a5eba4bd45c5c4ababf040d720b437240fcddf
2018-03-22 11:10:56 +01:00
Wladimir J. van der Laan
185d48473e
Merge #12718: [Tests] Require exact match in assert_start_raises_init_eror (jnewbery)
fae1374 qa: Allow for partial_match when checking init error (MarcoFalke)
5812273 [Tests] Require exact match in assert_start_raises_init_eror() (John Newbery)
0ec08a6 [Tests] Move assert_start_raises_init_error method to TestNode (John Newbery)

Pull request description:

  Extracted from #12379, because the changes are important on their own.

  This allows for exact testing, since the match can be specified with a strict regex. Internal details (such as exact formatting of the error message) can still be fuzzed away by regex wildcards.

Tree-SHA512: 605d2c9c42362a32d42321b066637577a026d0bb8cfc1c9f5737a4ca6503ffe85457a5122cea6e1101053ccc6c8aa1bbae3602e1fa7d2988bf7d5c275f412f66
2018-03-22 10:31:44 +01:00
Dimitris Apostolou
4d9b4256d8 Fix typos 2018-03-21 08:34:44 +02:00
MarcoFalke
c8330d4216 qa: Use node.datadir instead of tmpdir in test framework 2018-03-19 12:26:00 -04:00
John Newbery
0ec08a672d [Tests] Move assert_start_raises_init_error method to TestNode 2018-03-18 12:29:02 -04:00
Wladimir J. van der Laan
ce6ffe196e
Merge #12638: qa: Cache only chain and wallet for regtest datadir
fa23105 qa: Cache only chain and wallet for regtest datadir (MarcoFalke)

Pull request description:

  mempool.dat should be empty and I don't see a need to copy it around when restoring from the cache.

Tree-SHA512: f11ab69732db4dee0e9a0900570464e49085532b0cebc963877057112a7b985c477da3d32eb2093daabac9ada9e73b7c49881681ec5efa6101919b0af76001cf
2018-03-14 14:21:49 +01:00
practicalswift
a7324bd799 Format timestamps using ISO 8601 formatting (e.g. "2018-02-28T12:34:56Z")
* Z is the zone designator for the zero UTC offset.
* T is the delimiter used to separate date and time.

This makes it clear for the end-user that the date/time logged is
specified in UTC and not in the local time zone.
2018-03-09 15:02:01 +01:00
MarcoFalke
fa2310572f
qa: Cache only chain and wallet for regtest datadir 2018-03-07 15:53:30 -05:00
Sjors Provoost
b156ff7c30
[tests] bind functional test nodes to 127.0.0.1
Prevents OSX firewall allow-this-application-to-accept-inbound-connections
permission popups and is generally safer.

To prevent binding to 127.0.0.1, set self.bind_to_localhost_only = False.
2018-03-06 16:48:15 -05:00
MarcoFalke
face7220b7
qa: Move common args to bitcoin.conf 2018-02-15 14:00:28 -05:00
MarcoFalke
faefd2923a
qa: Prepare functional tests for Windows
* Pass `sys.executable` when calling a python script via the subprocess
  module
* Don't remove the log file while it is still open and written to
* Properly use os.pathsep and os.path.sep when modifying the PATH
  environment variable
* util-tests: Use os.path.join for Windows compatibility
2018-01-24 15:52:07 -05:00
Russell Yanofsky
ec527c6c88 Don't allow relative -walletdir paths
Also warn if bitcoind is configured to use a relative -datadir path.

Specifying paths relative to the current working directory in a daemon process
can be dangerous, because files can fail to be located even if the
configuration doesn't change, but the daemon is started up differently.

Specifying a relative -datadir now adds a warning to the debug log. It would
not be backwards-compatible to forbid relative -datadir paths entirely, and it
could also be also inconvenient for command line testing.

Specifying a relative -walletdir now results in a startup error. But since the
-walletdir option is new in 0.16.0, there should be no compatibility issues.
Another reason not to use working directory paths for -walletdir specifically
is that the default -walletdir is a "wallets" subdirectory inside the datadir,
so it could be surprising that setting -walletdir manually would choose a
directory rooted in a completely different location.
2018-01-18 15:09:27 -05:00
John Newbery
f6ade9ce1a [tests] allow tests to be run with --usecli
test_framework accepts a new --usecli parameter. Running the test with
this parameter will cause all RPCs to be sent through bitcoin-cli rather
than directly over http. By default, individual test cases do not
support --usecli, and self.supports_cli must be set to True in the
set_test_params method.

We can make supports_cli default to True in future once we know which
tests will fail with use_cli.
2018-01-08 17:35:27 -05:00
Akira Takizawa
595a7bab23 Increment MIT Licence copyright header year on files modified in 2017 2018-01-03 02:26:56 +09:00
practicalswift
c7399e7082 Remove unused Python imports 2017-12-10 11:48:36 +01:00
John Newbery
dba94ead0c [tests] [travis-ci] Move Travis functional test log post processing to test_runner 2017-11-29 15:10:05 -05:00
John Newbery
bba1c54edd [tests] Improve logging shutdown and add hint for combine_logs 2017-11-29 13:29:15 -05:00
MeshCollider
8263f6a5ac Create walletdir if datadir doesn't exist and fix tests 2017-11-18 00:50:59 +13:00
Wladimir J. van der Laan
6759a24eaa
Merge #11472: qa: Make tmpdir option an absolute path, misc cleanup
fafa003 qa: Remove never used return value of sync_with_ping (MarcoFalke)
fa9de37 qa: Make tmpdir option an absolute path (MarcoFalke)

Pull request description:

  This should fix issues with the multiwallet test and its symlinks
  when the tmpdir is a relative path.

  Rather than fixing os.symlink to work with paths relative to a
  directory descriptor, which does not work on Windows, normalize
  the path instead.

Tree-SHA512: 189690f3d065ea2f0f48e06775c86d513d0916c7c86312432e8e16df160e65539e288c2bd53d49a4180735fa940f6fcd52b506ccd7d9815651a9b1a69850dda6
2017-10-18 16:52:48 +02:00
John Newbery
f893085325 [tests] Don't subclass from object for Python 3 2017-10-16 21:46:23 -04:00
MarcoFalke
fa9de370b1 qa: Make tmpdir option an absolute path
This should fix issues with the multiwallet test and symlinks
when the tmpdir is a relative path.

Rather than fixing os.symlink to work with paths relative to a
directory descriptor, which does not work on Windows, normalize
the path instead.
2017-10-13 15:28:56 +02:00
João Barbosa
fd8f45fe88 [test] Add restart_node to BitcoinTestFramework 2017-09-20 15:38:09 +01:00
MarcoFalke
faa8d9581a [qa] TestNode: Add wait_until_stopped helper method 2017-09-06 09:41:21 -07:00
John Newbery
7148b74dc3 [tests] Functional tests must explicitly set num_nodes 2017-09-01 12:25:55 -04:00
John Newbery
5448a1471d [tests] don't override __init__() in individual tests
Almost all test scripts currently need to override the __init__()
method. When they do that they need to call into super().__init__() as
the base class does some generic initialization.

This commit makes the base class __init__() call into set_test_params()
method. Individual test cases can override set_test_params() to setup
their test parameters.
2017-09-01 12:25:55 -04:00
John Newbery
6cf094a022 [tests] Avoid passing around member variables in test_framework 2017-09-01 12:25:54 -04:00
John Newbery
36b6268670 [tests] TestNode: separate add_node from start_node
Separates the act of creating a TestNode object from starting the node.
The test_framework now keeps track of its list of TestNodes, and test
writers can call start_node() and stop_node() without having to update
the self.nodes list.
2017-09-01 12:25:54 -04:00
MarcoFalke
85aec87b11
Merge #10711: [tests] Introduce TestNode
789733891 [tests] Introduce TestNode (John Newbery)

Pull request description:

  Continues #10082

  TestNode is a class responsible for all state related to a bitcoind node
  under test. It stores local state, is responsible for tracking the
  bitcoind process and delegates unrecognised messages to the RPC
  connection.

  This commit changes start_nodes and stop_nodes to start and stop the
  bitcoind nodes in parallel, making test setup and teardown much faster.

  On my vm, this changeset reduces total test_runner runtime for the base set of tests
  (including building the cache) from 263s to 195s (a 25% speedup). Note that the time
  reported by test_runner does not include time spent building the cache:

  *with TestNode*:
  ```
  → date +"%T" ; ./test_runner.py -q ; date +"%T"
  12:48:04
  ..................................................................................................................................................................................................................................................................................................................................
  TEST                           | STATUS    | DURATION

  abandonconflict.py             | ✓ Passed  | 12 s
  bip68-112-113-p2p.py           | ✓ Passed  | 19 s
  blockchain.py                  | ✓ Passed  | 8 s
  bumpfee.py                     | ✓ Passed  | 13 s
  decodescript.py                | ✓ Passed  | 3 s
  disablewallet.py               | ✓ Passed  | 3 s
  disconnect_ban.py              | ✓ Passed  | 6 s
  fundrawtransaction.py          | ✓ Passed  | 37 s
  getchaintips.py                | ✓ Passed  | 4 s
  httpbasics.py                  | ✓ Passed  | 3 s
  import-rescan.py               | ✓ Passed  | 4 s
  importmulti.py                 | ✓ Passed  | 6 s
  importprunedfunds.py           | ✓ Passed  | 3 s
  invalidblockrequest.py         | ✓ Passed  | 4 s
  invalidtxrequest.py            | ✓ Passed  | 4 s
  keypool.py                     | ✓ Passed  | 7 s
  listsinceblock.py              | ✓ Passed  | 4 s
  listtransactions.py            | ✓ Passed  | 33 s
  mempool_limit.py               | ✓ Passed  | 4 s
  mempool_persist.py             | ✓ Passed  | 15 s
  mempool_reorg.py               | ✓ Passed  | 4 s
  mempool_resurrect_test.py      | ✓ Passed  | 3 s
  mempool_spendcoinbase.py       | ✓ Passed  | 3 s
  merkle_blocks.py               | ✓ Passed  | 3 s
  multi_rpc.py                   | ✓ Passed  | 4 s
  net.py                         | ✓ Passed  | 3 s
  nulldummy.py                   | ✓ Passed  | 3 s
  p2p-compactblocks.py           | ✓ Passed  | 28 s
  p2p-fullblocktest.py           | ✓ Passed  | 126 s
  p2p-leaktests.py               | ✓ Passed  | 8 s
  p2p-mempool.py                 | ✓ Passed  | 3 s
  p2p-segwit.py                  | ✓ Passed  | 59 s
  p2p-versionbits-warning.py     | ✓ Passed  | 8 s
  preciousblock.py               | ✓ Passed  | 3 s
  prioritise_transaction.py      | ✓ Passed  | 5 s
  proxy_test.py                  | ✓ Passed  | 3 s
  rawtransactions.py             | ✓ Passed  | 9 s
  receivedby.py                  | ✓ Passed  | 19 s
  reindex.py                     | ✓ Passed  | 12 s
  rest.py                        | ✓ Passed  | 9 s
  rpcnamedargs.py                | ✓ Passed  | 3 s
  segwit.py                      | ✓ Passed  | 7 s
  sendheaders.py                 | ✓ Passed  | 24 s
  signmessages.py                | ✓ Passed  | 3 s
  signrawtransactions.py         | ✓ Passed  | 3 s
  txn_clone.py                   | ✓ Passed  | 4 s
  txn_doublespend.py --mineblock | ✓ Passed  | 4 s
  uptime.py                      | ✓ Passed  | 3 s
  wallet-accounts.py             | ✓ Passed  | 3 s
  wallet-dump.py                 | ✓ Passed  | 7 s
  wallet-encryption.py           | ✓ Passed  | 8 s
  wallet-hd.py                   | ✓ Passed  | 15 s
  wallet.py                      | ✓ Passed  | 31 s
  walletbackup.py                | ✓ Passed  | 104 s
  zapwallettxes.py               | ✓ Passed  | 9 s
  zmq_test.py                    | ○ Skipped | 0 s

  ALL                            | ✓ Passed  | 735 s (accumulated)
  Runtime: 189 s

  12:51:19
  ```

  *master*:
  ```
  → date +"%T" ; ./test_runner.py -q ; date +"%T"
  12:40:13
  ..........................................................................................................................................................................................................................................................................................................................................................................................................................................
  TEST                           | STATUS    | DURATION

  abandonconflict.py             | ✓ Passed  | 15 s
  bip68-112-113-p2p.py           | ✓ Passed  | 19 s
  blockchain.py                  | ✓ Passed  | 8 s
  bumpfee.py                     | ✓ Passed  | 20 s
  decodescript.py                | ✓ Passed  | 3 s
  disablewallet.py               | ✓ Passed  | 3 s
  disconnect_ban.py              | ✓ Passed  | 8 s
  fundrawtransaction.py          | ✓ Passed  | 36 s
  getchaintips.py                | ✓ Passed  | 11 s
  httpbasics.py                  | ✓ Passed  | 7 s
  import-rescan.py               | ✓ Passed  | 16 s
  importmulti.py                 | ✓ Passed  | 10 s
  importprunedfunds.py           | ✓ Passed  | 5 s
  invalidblockrequest.py         | ✓ Passed  | 4 s
  invalidtxrequest.py            | ✓ Passed  | 3 s
  keypool.py                     | ✓ Passed  | 7 s
  listsinceblock.py              | ✓ Passed  | 11 s
  listtransactions.py            | ✓ Passed  | 37 s
  mempool_limit.py               | ✓ Passed  | 4 s
  mempool_persist.py             | ✓ Passed  | 23 s
  mempool_reorg.py               | ✓ Passed  | 7 s
  mempool_resurrect_test.py      | ✓ Passed  | 3 s
  mempool_spendcoinbase.py       | ✓ Passed  | 3 s
  merkle_blocks.py               | ✓ Passed  | 10 s
  multi_rpc.py                   | ✓ Passed  | 6 s
  net.py                         | ✓ Passed  | 6 s
  nulldummy.py                   | ✓ Passed  | 3 s
  p2p-compactblocks.py           | ✓ Passed  | 30 s
  p2p-fullblocktest.py           | ✓ Passed  | 126 s
  p2p-leaktests.py               | ✓ Passed  | 8 s
  p2p-mempool.py                 | ✓ Passed  | 3 s
  p2p-segwit.py                  | ✓ Passed  | 62 s
  p2p-versionbits-warning.py     | ✓ Passed  | 8 s
  preciousblock.py               | ✓ Passed  | 8 s
  prioritise_transaction.py      | ✓ Passed  | 7 s
  proxy_test.py                  | ✓ Passed  | 10 s
  rawtransactions.py             | ✓ Passed  | 15 s
  receivedby.py                  | ✓ Passed  | 28 s
  reindex.py                     | ✓ Passed  | 12 s
  rest.py                        | ✓ Passed  | 12 s
  rpcnamedargs.py                | ✓ Passed  | 3 s
  segwit.py                      | ✓ Passed  | 12 s
  sendheaders.py                 | ✓ Passed  | 26 s
  signmessages.py                | ✓ Passed  | 3 s
  signrawtransactions.py         | ✓ Passed  | 3 s
  txn_clone.py                   | ✓ Passed  | 10 s
  txn_doublespend.py --mineblock | ✓ Passed  | 10 s
  uptime.py                      | ✓ Passed  | 3 s
  wallet-accounts.py             | ✓ Passed  | 3 s
  wallet-dump.py                 | ✓ Passed  | 6 s
  wallet-encryption.py           | ✓ Passed  | 8 s
  wallet-hd.py                   | ✓ Passed  | 18 s
  wallet.py                      | ✓ Passed  | 69 s
  walletbackup.py                | ✓ Passed  | 130 s
  zapwallettxes.py               | ✓ Passed  | 15 s
  zmq_test.py                    | ○ Skipped | 0 s

  ALL                            | ✓ Passed  | 936 s (accumulated)
  Runtime: 242 s

  12:44:36
  ```

Tree-SHA512: 6dfc4c11fd0caf7de6954c93679cf22c3df0acc6f432e616d1151062a61f456faa8ae2fe670b427868af55bb564802df84c8fd76e90b4b338750dbc23f46ad88
2017-08-15 23:35:07 +02:00
John Newbery
7897338918 [tests] Introduce TestNode
TestNode is a class responsible for all state related to a bitcoind node
under test. It stores local state, is responsible for tracking the
bitcoind process and delegates unrecognised messages to the RPC
connection.

This commit changes start_nodes and stop_nodes to start and stop the
bitcoind nodes in parallel, making test setup and teardown much faster.
2017-08-15 13:12:15 -04:00
John Newbery
cc5d38f4b8 Add option to attach a python debugger if test fails 2017-08-10 12:50:51 -04:00
John Newbery
5ba83c1d43 [tests] fix nits.
Thanks to Marco Falke.
2017-06-29 15:06:38 +01:00
John Newbery
0d473c539e [tests] move mocktime property and functions to BitcoinTestFramework 2017-06-29 11:55:23 +01:00
John Newbery
cad967a892 [tests] Move stop_node and start_node methods to BitcoinTestFramework
This commit moves functions start_node, start_nodes, stop_node and
stop_nodes functions into the BitcoinTestFramework class. It also moves
the bitcoind_processes dict and coverage variables into BitcoinTestFramework.
2017-06-29 11:55:19 +01:00
John Newbery
f1fe5368f1 [tests] fix flake8 warnings in test_framework.py and util.py 2017-06-29 11:19:06 +01:00
MarcoFalke
416af3edf5
Merge #10690: [qa] Bugfix: allow overriding extra_args in ComparisonTestFramework
4ed3653 [qa] Bugfix: allow overriding extra_args in ComparisonTestFramework (Suhas Daftuar)

Tree-SHA512: d8f724b3324aad73a7b15cf87ff394e8d615bf3cd5a394d5715347d833f2ae9ac745a944202986866255eca5fc105ea06ab3abe12e168b67de34482f751c68e2
2017-06-28 18:09:58 +02:00
Suhas Daftuar
4ed36539bc [qa] Bugfix: allow overriding extra_args in ComparisonTestFramework 2017-06-28 08:53:20 -04:00
Andrew Chow
c53c9831ee Replace cookie auth in tests
Since rpcuser and rpcpassword are now deprecated, replace them with cookie auth.

Fix test failures with cookie auth
2017-06-08 11:22:52 -07:00
MarcoFalke
329fc1dce7
Merge #10359: [tests] functional tests should call BitcoinTestFramework start/stop node methods
53f6775 fixup: fix nits (John Newbery)
a433d8a [tests] Update start/stop node functions to be private module functions (John Newbery)
d8c218f [tests] Functional tests call self.start_node(s) and self.stop_node(s) (John Newbery)

Tree-SHA512: 9cc01584a5e57686b7e7cb1c4c5186ad8cc7eb650d6d4f27b06bdb5e249a10966705814bdfb22d9ff2d5d3326911e489bf3d22257d751a299c0b24b7f40bffb5
2017-06-02 12:10:41 +02:00
John Newbery
a433d8a15e [tests] Update start/stop node functions to be private module functions
This commit marks the start/stop functions in util.py as private module
functions. A future PR will remove these entirely and move the
functionality directly into the BitcoinTestFramework class, but setting them as
private in this PR will prevent anyone from accidentally calling them
before that future PR is merged.
2017-05-31 16:59:38 -04:00
John Newbery
930deb9b2c [tests] skipped tests should clean up after themselves 2017-05-22 14:46:18 -04:00
John Newbery
b040243cc3 [tests] improve tmpdir structure 2017-05-18 18:11:32 -04:00
MarcoFalke
fff72de5bf
Merge #10171: [tests] Add node methods to test framework
4550049 Reorganize BitcoinTestFramework class (John Newbery)
b7dd44c Add start and stop node methods to BitcoinTestFramework (John Newbery)
b111324 move initialize_chain() and initialize_chain_clean() to be methods of BitcoinTestFramework (John Newbery)

Tree-SHA512: 17e541aea8ca4c0d1189701499384e26239e2d5905de8adb0f042d3cf4c0bbed79fcaad61d563e1743bf4c62ad4915cebb4714783db839d9c53dfbbedcae6e9a
2017-05-07 15:13:53 +02:00
John Newbery
4550049903 Reorganize BitcoinTestFramework class 2017-05-05 18:05:42 -04:00
Wladimir J. van der Laan
83073de4bd
Merge #10307: [tests] allow zmq test to be run in out-of-tree builds
b8251f6 [tests] allow zmq test to be run in out-of-tree builds (John Newbery)

Tree-SHA512: 6946d23bc8a0b57e841a6811989182732d0534989e0e3b94421387f7971379b25a25d238cad22272e04076293275f6e980c8a713fce87ba48c1c4463d9243051
2017-05-03 08:47:37 +02:00
John Newbery
b7dd44c528 Add start and stop node methods to BitcoinTestFramework 2017-05-02 14:20:13 -04:00
John Newbery
b111324acd move initialize_chain() and initialize_chain_clean() to be methods of BitcoinTestFramework 2017-05-02 14:18:55 -04:00
John Newbery
c9cc76dcaa [tests] Remove is_network_split from funtional test cases 2017-05-02 13:31:54 -04:00
John Newbery
b8251f6337 [tests] allow zmq test to be run in out-of-tree builds 2017-05-01 15:12:49 -04:00
John Newbery
8c7288c06b Print out the final 1000 lines of test_framework.log if test fails 2017-03-27 19:34:25 -04:00