John Newbery
7148b74dc3
[tests] Functional tests must explicitly set num_nodes
2017-09-01 12:25:55 -04:00
John Newbery
5448a1471d
[tests] don't override __init__() in individual tests
...
Almost all test scripts currently need to override the __init__()
method. When they do that they need to call into super().__init__() as
the base class does some generic initialization.
This commit makes the base class __init__() call into set_test_params()
method. Individual test cases can override set_test_params() to setup
their test parameters.
2017-09-01 12:25:55 -04:00
John Newbery
6cf094a022
[tests] Avoid passing around member variables in test_framework
2017-09-01 12:25:54 -04:00
John Newbery
36b6268670
[tests] TestNode: separate add_node from start_node
...
Separates the act of creating a TestNode object from starting the node.
The test_framework now keeps track of its list of TestNodes, and test
writers can call start_node() and stop_node() without having to update
the self.nodes list.
2017-09-01 12:25:54 -04:00
MarcoFalke
85aec87b11
Merge #10711 : [tests] Introduce TestNode
...
789733891#10082
TestNode is a class responsible for all state related to a bitcoind node
under test. It stores local state, is responsible for tracking the
bitcoind process and delegates unrecognised messages to the RPC
connection.
This commit changes start_nodes and stop_nodes to start and stop the
bitcoind nodes in parallel, making test setup and teardown much faster.
On my vm, this changeset reduces total test_runner runtime for the base set of tests
(including building the cache) from 263s to 195s (a 25% speedup). Note that the time
reported by test_runner does not include time spent building the cache:
*with TestNode*:
```
→ date +"%T" ; ./test_runner.py -q ; date +"%T"
12:48:04
..................................................................................................................................................................................................................................................................................................................................
TEST | STATUS | DURATION
abandonconflict.py | ✓ Passed | 12 s
bip68-112-113-p2p.py | ✓ Passed | 19 s
blockchain.py | ✓ Passed | 8 s
bumpfee.py | ✓ Passed | 13 s
decodescript.py | ✓ Passed | 3 s
disablewallet.py | ✓ Passed | 3 s
disconnect_ban.py | ✓ Passed | 6 s
fundrawtransaction.py | ✓ Passed | 37 s
getchaintips.py | ✓ Passed | 4 s
httpbasics.py | ✓ Passed | 3 s
import-rescan.py | ✓ Passed | 4 s
importmulti.py | ✓ Passed | 6 s
importprunedfunds.py | ✓ Passed | 3 s
invalidblockrequest.py | ✓ Passed | 4 s
invalidtxrequest.py | ✓ Passed | 4 s
keypool.py | ✓ Passed | 7 s
listsinceblock.py | ✓ Passed | 4 s
listtransactions.py | ✓ Passed | 33 s
mempool_limit.py | ✓ Passed | 4 s
mempool_persist.py | ✓ Passed | 15 s
mempool_reorg.py | ✓ Passed | 4 s
mempool_resurrect_test.py | ✓ Passed | 3 s
mempool_spendcoinbase.py | ✓ Passed | 3 s
merkle_blocks.py | ✓ Passed | 3 s
multi_rpc.py | ✓ Passed | 4 s
net.py | ✓ Passed | 3 s
nulldummy.py | ✓ Passed | 3 s
p2p-compactblocks.py | ✓ Passed | 28 s
p2p-fullblocktest.py | ✓ Passed | 126 s
p2p-leaktests.py | ✓ Passed | 8 s
p2p-mempool.py | ✓ Passed | 3 s
p2p-segwit.py | ✓ Passed | 59 s
p2p-versionbits-warning.py | ✓ Passed | 8 s
preciousblock.py | ✓ Passed | 3 s
prioritise_transaction.py | ✓ Passed | 5 s
proxy_test.py | ✓ Passed | 3 s
rawtransactions.py | ✓ Passed | 9 s
receivedby.py | ✓ Passed | 19 s
reindex.py | ✓ Passed | 12 s
rest.py | ✓ Passed | 9 s
rpcnamedargs.py | ✓ Passed | 3 s
segwit.py | ✓ Passed | 7 s
sendheaders.py | ✓ Passed | 24 s
signmessages.py | ✓ Passed | 3 s
signrawtransactions.py | ✓ Passed | 3 s
txn_clone.py | ✓ Passed | 4 s
txn_doublespend.py --mineblock | ✓ Passed | 4 s
uptime.py | ✓ Passed | 3 s
wallet-accounts.py | ✓ Passed | 3 s
wallet-dump.py | ✓ Passed | 7 s
wallet-encryption.py | ✓ Passed | 8 s
wallet-hd.py | ✓ Passed | 15 s
wallet.py | ✓ Passed | 31 s
walletbackup.py | ✓ Passed | 104 s
zapwallettxes.py | ✓ Passed | 9 s
zmq_test.py | ○ Skipped | 0 s
ALL | ✓ Passed | 735 s (accumulated)
Runtime: 189 s
12:51:19
```
*master*:
```
→ date +"%T" ; ./test_runner.py -q ; date +"%T"
12:40:13
..........................................................................................................................................................................................................................................................................................................................................................................................................................................
TEST | STATUS | DURATION
abandonconflict.py | ✓ Passed | 15 s
bip68-112-113-p2p.py | ✓ Passed | 19 s
blockchain.py | ✓ Passed | 8 s
bumpfee.py | ✓ Passed | 20 s
decodescript.py | ✓ Passed | 3 s
disablewallet.py | ✓ Passed | 3 s
disconnect_ban.py | ✓ Passed | 8 s
fundrawtransaction.py | ✓ Passed | 36 s
getchaintips.py | ✓ Passed | 11 s
httpbasics.py | ✓ Passed | 7 s
import-rescan.py | ✓ Passed | 16 s
importmulti.py | ✓ Passed | 10 s
importprunedfunds.py | ✓ Passed | 5 s
invalidblockrequest.py | ✓ Passed | 4 s
invalidtxrequest.py | ✓ Passed | 3 s
keypool.py | ✓ Passed | 7 s
listsinceblock.py | ✓ Passed | 11 s
listtransactions.py | ✓ Passed | 37 s
mempool_limit.py | ✓ Passed | 4 s
mempool_persist.py | ✓ Passed | 23 s
mempool_reorg.py | ✓ Passed | 7 s
mempool_resurrect_test.py | ✓ Passed | 3 s
mempool_spendcoinbase.py | ✓ Passed | 3 s
merkle_blocks.py | ✓ Passed | 10 s
multi_rpc.py | ✓ Passed | 6 s
net.py | ✓ Passed | 6 s
nulldummy.py | ✓ Passed | 3 s
p2p-compactblocks.py | ✓ Passed | 30 s
p2p-fullblocktest.py | ✓ Passed | 126 s
p2p-leaktests.py | ✓ Passed | 8 s
p2p-mempool.py | ✓ Passed | 3 s
p2p-segwit.py | ✓ Passed | 62 s
p2p-versionbits-warning.py | ✓ Passed | 8 s
preciousblock.py | ✓ Passed | 8 s
prioritise_transaction.py | ✓ Passed | 7 s
proxy_test.py | ✓ Passed | 10 s
rawtransactions.py | ✓ Passed | 15 s
receivedby.py | ✓ Passed | 28 s
reindex.py | ✓ Passed | 12 s
rest.py | ✓ Passed | 12 s
rpcnamedargs.py | ✓ Passed | 3 s
segwit.py | ✓ Passed | 12 s
sendheaders.py | ✓ Passed | 26 s
signmessages.py | ✓ Passed | 3 s
signrawtransactions.py | ✓ Passed | 3 s
txn_clone.py | ✓ Passed | 10 s
txn_doublespend.py --mineblock | ✓ Passed | 10 s
uptime.py | ✓ Passed | 3 s
wallet-accounts.py | ✓ Passed | 3 s
wallet-dump.py | ✓ Passed | 6 s
wallet-encryption.py | ✓ Passed | 8 s
wallet-hd.py | ✓ Passed | 18 s
wallet.py | ✓ Passed | 69 s
walletbackup.py | ✓ Passed | 130 s
zapwallettxes.py | ✓ Passed | 15 s
zmq_test.py | ○ Skipped | 0 s
ALL | ✓ Passed | 936 s (accumulated)
Runtime: 242 s
12:44:36
```
Tree-SHA512: 6dfc4c11fd0caf7de6954c93679cf22c3df0acc6f432e616d1151062a61f456faa8ae2fe670b427868af55bb564802df84c8fd76e90b4b338750dbc23f46ad88
2017-08-15 23:35:07 +02:00
John Newbery
7897338918
[tests] Introduce TestNode
...
TestNode is a class responsible for all state related to a bitcoind node
under test. It stores local state, is responsible for tracking the
bitcoind process and delegates unrecognised messages to the RPC
connection.
This commit changes start_nodes and stop_nodes to start and stop the
bitcoind nodes in parallel, making test setup and teardown much faster.
2017-08-15 13:12:15 -04:00
John Newbery
cc5d38f4b8
Add option to attach a python debugger if test fails
2017-08-10 12:50:51 -04:00
John Newbery
5ba83c1d43
[tests] fix nits.
...
Thanks to Marco Falke.
2017-06-29 15:06:38 +01:00
John Newbery
0d473c539e
[tests] move mocktime property and functions to BitcoinTestFramework
2017-06-29 11:55:23 +01:00
John Newbery
cad967a892
[tests] Move stop_node and start_node methods to BitcoinTestFramework
...
This commit moves functions start_node, start_nodes, stop_node and
stop_nodes functions into the BitcoinTestFramework class. It also moves
the bitcoind_processes dict and coverage variables into BitcoinTestFramework.
2017-06-29 11:55:19 +01:00
John Newbery
f1fe5368f1
[tests] fix flake8 warnings in test_framework.py and util.py
2017-06-29 11:19:06 +01:00
MarcoFalke
416af3edf5
Merge #10690 : [qa] Bugfix: allow overriding extra_args in ComparisonTestFramework
...
4ed3653
2017-06-28 18:09:58 +02:00
Suhas Daftuar
4ed36539bc
[qa] Bugfix: allow overriding extra_args in ComparisonTestFramework
2017-06-28 08:53:20 -04:00
Andrew Chow
c53c9831ee
Replace cookie auth in tests
...
Since rpcuser and rpcpassword are now deprecated, replace them with cookie auth.
Fix test failures with cookie auth
2017-06-08 11:22:52 -07:00
MarcoFalke
329fc1dce7
Merge #10359 : [tests] functional tests should call BitcoinTestFramework start/stop node methods
...
53f6775a433d8ad8c218f
2017-06-02 12:10:41 +02:00
John Newbery
a433d8a15e
[tests] Update start/stop node functions to be private module functions
...
This commit marks the start/stop functions in util.py as private module
functions. A future PR will remove these entirely and move the
functionality directly into the BitcoinTestFramework class, but setting them as
private in this PR will prevent anyone from accidentally calling them
before that future PR is merged.
2017-05-31 16:59:38 -04:00
John Newbery
930deb9b2c
[tests] skipped tests should clean up after themselves
2017-05-22 14:46:18 -04:00
John Newbery
b040243cc3
[tests] improve tmpdir structure
2017-05-18 18:11:32 -04:00
MarcoFalke
fff72de5bf
Merge #10171 : [tests] Add node methods to test framework
...
4550049b7dd44cb111324
2017-05-07 15:13:53 +02:00
John Newbery
4550049903
Reorganize BitcoinTestFramework class
2017-05-05 18:05:42 -04:00
Wladimir J. van der Laan
83073de4bd
Merge #10307 : [tests] allow zmq test to be run in out-of-tree builds
...
b8251f6
2017-05-03 08:47:37 +02:00
John Newbery
b7dd44c528
Add start and stop node methods to BitcoinTestFramework
2017-05-02 14:20:13 -04:00
John Newbery
b111324acd
move initialize_chain() and initialize_chain_clean() to be methods of BitcoinTestFramework
2017-05-02 14:18:55 -04:00
John Newbery
c9cc76dcaa
[tests] Remove is_network_split from funtional test cases
2017-05-02 13:31:54 -04:00
John Newbery
b8251f6337
[tests] allow zmq test to be run in out-of-tree builds
2017-05-01 15:12:49 -04:00
John Newbery
8c7288c06b
Print out the final 1000 lines of test_framework.log if test fails
2017-03-27 19:34:25 -04:00
MarcoFalke
ca209230c8
Merge #10053 : [test] Allow functional test cases to be skipped
...
0c1ade6232b666
2017-03-24 13:58:03 +01:00
practicalswift
72163d4585
[tests] Remove unused and duplicate imports
2017-03-22 17:49:23 +01:00
John Newbery
232b6665bc
Allow test cases to be skipped
...
Currently, functional test cases can either pass or fail. There are
occasions when it is helpful to skip tests, for example if the
system they are running on does not meet the requirements for the test.
The rest of the test suite can run without being marked as a failure.
This commit adds framework for tests to skip if their requirements
aren't met.
2017-03-22 10:26:02 -04:00
John Newbery
8317a45161
Python functional tests should log in UTC
...
bitcoind logs use UTC. Python functional tests should also log in UTC.
2017-03-21 17:30:13 -04:00
John Newbery
c28ee91db0
Rename rpc-tests directory to functional
2017-03-20 10:40:31 -04:00
