The current getblocktxn implementation drops and ignores requests for old
blocks, which causes occasional sync_block timeouts during the
p2p-compactblocks.py test as reported in
https://github.com/bitcoin/bitcoin/issues/8842.
The p2p-compactblocks.py test setup creates many new blocks in a short
period of time, which can lead to getblocktxn requests for blocks below the
hardcoded depth limit of 10 blocks. This commit changes the getblocktxn
handler not to ignore these requests, so the peer nodes in the test setup
will reliably be able to sync.
The protocol change is documented in BIP-152 update "Allow block responses
to getblocktxn requests" at https://github.com/bitcoin/bips/pull/469.
The protocol change is not expected to affect nodes running outside the test
environment, because there shouldn't normally be lots of new blocks being
rapidly added that need to be synced.
Increase wallet-dump RPC timeout from 30 seconds to 1 minute. This avoids a
timeout error that seemed to happen regularly (around 50% of builds) on a
particular jenkins server during the first getnewaddress RPC call made by the
test.
The failing stack trace looked like:
Unexpected exception caught during testing: timeout('timed out',)
File ".../bitcoin/qa/rpc-tests/test_framework/test_framework.py", line 146, in main
self.run_test()
File ".../bitcoin/qa/rpc-tests/wallet-dump.py", line 73, in run_test
addr = self.nodes[0].getnewaddress()
File ".../bitcoin/qa/rpc-tests/test_framework/coverage.py", line 49, in __call__
return_val = self.auth_service_proxy_instance.__call__(*args, **kwargs)
File ".../bitcoin/qa/rpc-tests/test_framework/authproxy.py", line 145, in __call__
response = self._request('POST', self.__url.path, postdata.encode('utf-8'))
File ".../bitcoin/qa/rpc-tests/test_framework/authproxy.py", line 121, in _request
return self._get_response()
File ".../bitcoin/qa/rpc-tests/test_framework/authproxy.py", line 160, in _get_response
http_response = self.__conn.getresponse()
File "/usr/lib/python3.4/http/client.py", line 1171, in getresponse
response.begin()
File "/usr/lib/python3.4/http/client.py", line 351, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.4/http/client.py", line 313, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.4/socket.py", line 374, in readinto
return self._sock.recv_into(b)
Three categories of modifications:
1)
1 instance of 'The Bitcoin Core developers \n',
1 instance of 'the Bitcoin Core developers\n',
3 instances of 'Bitcoin Core Developers\n', and
12 instances of 'The Bitcoin developers\n'
are made uniform with the 443 instances of 'The Bitcoin Core developers\n'
2)
3 instances of 'BitPay, Inc\.\n' are made uniform with the other 6
instances of 'BitPay Inc\.\n'
3)
4 instances where there was no '(c)' between the 'Copyright' and the year
where it deviates from the style of the local directory.
Clear test_node.last_block before requesting blocks in the
compactblocks_not_at_tip test so comparisons won't fail if a blocks were received
before the test started.
The bug doesn't currently cause any problems due to the order tests run, but
this will change in the next commit.
Bug caused the wait_for_block_announcement to be called on the wrong node,
leading to nondeterminism and occasional test failures. Bug was introduced in
merge commit:
d075479 Merge #8882: [qa] Fix race conditions in p2p-compactblocks.py and sendheaders.py
Underlying commits which conflicted were:
27acfc1 [qa] Update p2p-compactblocks.py for compactblocks v2
6976db2 [qa] Another attempt to fix race condition in p2p-compactblocks.py
The first commit changed the test_compactblock_construction function signature
and second commit added code which wasn't updated during the merge to use the
new arguments.
Suhas Daftuar <sdaftuar@chaincode.com> noticed the bug and suggested the fix.
67d6ee1 remove redundant tests in p2p-segwit.py (Johnson Lau)
9260085 test segwit uncompressed key fixes (Johnson Lau)
248f3a7 Fix ismine and addwitnessaddress: no uncompressed keys in segwit (Pieter Wuille)
b811124 [qa] Add tests for uncompressed pubkeys in segwit (Suhas Daftuar)
9f0397a Make test framework produce lowS signatures (Johnson Lau)
4c0c25a Require compressed keys in segwit as policy and disable signing with uncompressed keys for segwit scripts (Johnson Lau)
3ade2f6 Add standard limits for P2WSH with tests (Johnson Lau)
sync_with_ping() only guarantees that the node has processed messages
it's received from the peer, not that block announcements from the node have
made it back to the peer. Replace sync_with_ping() with an explicit check that
the node's tip has been announced.
These are text files but their encoding does not depend on the locale.
Not all of them require utf8 but it is better to fix it at something
to remove potential unpredictability.
This is necessary on FreeBSD where no locale is set by default,
and apparently Python defaults not only the terminal encoding to the locale
but that of every text file. So without LOCALE environment it defaults text
file encoding to ASCII. This causes problems with e.g. `bitcoin.conf`.
Luckily the locale doesn't affect the default encoding for str.encode() and
bytes.decode() on Python 3, so this is the only change necessary.
This improves error reporting if `JSONRPCException` is not specifically caught
and ends up in Python's default backtrace handler.
Before:
```
Traceback (most recent call last):
File "/.../projects/bitcoin/bitcoin/qa/rpc-tests/test_framework/authproxy.py", line 153, in __call__
raise JSONRPCException(response['error'])
test_framework.authproxy.JSONRPCException
```
After:
```
Traceback (most recent call last):
File "/.../projects/bitcoin/bitcoin/qa/rpc-tests/test_framework/authproxy.py", line 152, in __call__
raise JSONRPCException(response['error'])
test_framework.authproxy.JSONRPCException: Unknown named parameter random (-8)
```
Python lambda use was incorrect.
sendcmpct messages need to be synchronized with RPC calls to generate().
Headers need to be synced (eg with getheaders) for cmpctblock announcements
to start.
Last test omitted sending a sendcmpct message.
5547aeb p2psegwit.py transaction is rejected due to premature witness not size (instagibbs)
bc1d1f2 Update p2p-segwit.py to reflect correct AskFor behavior (instagibbs)
waitfornewblock waits until a new block is received, or the timeout expires, then
returns the current block height/hash.
waitforblock waits for a specific blockhash, or until the timeout expires, then
returns the current block height/hash. If the target blockhash is the current
tip, it will return immediately.
waitforblockheight waits until the tip has reached a certain height or higher,
then returns the current height and hash.
waitforblockheight is used to avoid polling in the rpc tests.
fad8cf6 [qa] Use single cache dir for chains (MarcoFalke)
fa2d68f [qa] Adjust timeouts for micro-optimization of run time (MarcoFalke)
fae596f [qa] Sort scripts by time for pull_tester and don't overwrite setup_chain (MarcoFalke)
Forward-ports two commits from 0.13:
- [0.13] Create a new HD seed after encrypting the wallet
- [Wallet] Add CKeyMetadata record for HDMasterKey(s), factor out HD key generation
Github-Pull: #8389
Rebased-From: f142c11ac634df487cc4bc65a5f1c9a3e3563dd9 de45c065f0648c4c41b57cb492420ceeed29dd11
c59c434 qa: Add test for standardness of segwit v0 outputs (Suhas Daftuar)
1ffaff2 Make witness v0 outputs non-standard before segwit activation (Johnson Lau)
7945088 [Wallet] comsetic non-code changes for the HD feature (Jonas Schnelli)
68d7682 [Wallet] ensure CKeyMetadata.hdMasterKeyID will be cleared during SetNull() (Jonas Schnelli)
f708085 [QA] extend wallet-hd test to cover HD metadata (Jonas Schnelli)
986c223 [Wallet] print hd masterkeyid in getwalletinfo (Jonas Schnelli)
b1c7b24 [Wallet] report optional HDKeypath/HDMasterKeyId in validateaddress (Jonas Schnelli)
5b95dd2 [Wallet] extend CKeyMetadata with HD keypath (Jonas Schnelli)
46c9620 Test that unnecessary witnesses can't be used for mempool DoS (Suhas Daftuar)
bb66a11 Fix DoS vulnerability in mempool acceptance (Suhas Daftuar)
mininode now supports witness transactions/blocks, blocktools
has a helper for adding witness commitments to blocks, and script
has a function to calculate hashes for signature under sigversion
1, used by segwit.
Py3 conversion by Marco Falke
Test to make sure upgraded nodes don't ask for non-wit blocks by
Gregory Sanders.
7982fce doc: Mention full UTF-8 support in release notes (Wladimir J. van der Laan)
6bbb4ef test: test utf-8 for labels in wallet (Wladimir J. van der Laan)
a406fcb test: add ensure_ascii setting to AuthServiceProxy (Wladimir J. van der Laan)
60ab9b2 Squashed 'src/univalue/' changes from 2740c4f..f32df99 (Wladimir J. van der Laan)
ff2dcf2 Tests: Edit bloated varint test and add option for 'barely expensive' tests (mrbandrews)
12c5a16 Catch exceptions from non-canonical encoding and print only to log (mrbandrews)
291f8aa Continuing port of java comptool (mrbandrews)
8c9e681 Tests: Rework blockstore to avoid re-serialization. (mrbandrews)
Add a setting ensure_ascii to AuthServiceProxy. This setting,
defaulting to True (backwards compatible),
is passed through to json.dumps. If set to False, non-ASCII characters
>0x80 are not escaped. This is useful for testing server
input processing, as well as slightly more bandwidth friendly in case of
heavy unicode usage.
3d3602f Add RPC test for the p2p mempool command in conjunction with disabled bloomfilters (Jonas Schnelli)
beceac9 Disable the mempool P2P command when bloom filters disabled (Peter Todd)
ae357d5 [Bitcoin-Tx] Add tests for sequence number support (Jonas Schnelli)
e59336f [bitcoin-tx] allow to set nSequence number over the in= command (Jonas Schnelli)
a946bb6 [RPC] createrawtransaction: add option to set the sequence number per input (Jonas Schnelli)
Also introduce UniValueType
UniValueType is a wrapper for UniValue::VType which allows setting
a typeAny flag. This flag indicates the type does not matter.
(Used by RPCTypeCheckObj)
Previously these functions would infinitely loop if sync failed;
now they have a default timeout of 60 seconds, after which an
AssertionError is raised.
sync_blocks() has also been improved and now compares the tip
hash of each node, rather than just using block count.
b4d24e1 Report reindexing progress in GUI (Pieter Wuille)
d3d7547 Add -reindex-chainstate that does not rebuild block index (Pieter Wuille)
fb8fad1 Optimize ActivateBestChain for long chains (Pieter Wuille)
316623f Switch reindexing to AcceptBlock in-loop and ActivateBestChain afterwards (Pieter Wuille)
d253ec4 Make ProcessNewBlock dbp const and update comment (Pieter Wuille)
fa83a5d [qa] wallet: Temporarily disable salvagewallet test (MarcoFalke)
fadd048 [doc] Link to clang-format in the developer notes (MarcoFalke)
fa72f7d [doc] Remove outdated line from listunspent RPC help, fix typo (MarcoFalke)
ac40ed7 Increase timeout waiting for pruned blk00000.dat (error10)
In my ever-growing list of test failures, I was seeing this one intermittently.
```
Running 2nd level testscript pruning.py...
Initializing test directory /tmp/testY5ypCv
Warning! This test requires 4GB of disk space and takes over 30 mins (up to 2 hours)
Mining a big blockchain of 995 blocks
Check that we haven't started pruning yet because we're below PruneAfterHeight
Success
Though we're already using more than 550MB, current usage: 587
Mining 25 more blocks should cause the first block file to be pruned
Assertion failed: blk00000.dat not pruned when it should be
File "/home/error/bitcoinxt-0.11D/qa/rpc-tests/test_framework/test_framework.py", line 118, in main
self.run_test()
File "/home/error/bitcoinxt-0.11D/qa/rpc-tests/pruning.py", line 272, in run_test
self.test_height_min()
File "/home/error/bitcoinxt-0.11D/qa/rpc-tests/pruning.py", line 94, in test_height_min
raise AssertionError("blk00000.dat not pruned when it should be")
Stopping nodes
Failed
```
After digging into the test, I found that the code is waiting 10 seconds for blk00000.dat to be deleted, and then throwing this failure if it still exists after 10 seconds.
I increased this amount, had the script print the actual time taken, and ran the test a few more times. The time taken ranged between 8 to 12 seconds. So, I feel that this timeout is too short.
After changing the timeout to 30 seconds, the test passes consistently.
(cherry picked from commit 3469911c89a48dd2fefe4d1c2a0c176256e14ee0)
