de9a1db Acquire cs_main lock before cs_wallet during wallet initialization (Russell Yanofsky)
Pull request description:
`CWallet::MarkConflicted` may acquire the `cs_main` lock after `CWalletDB::LoadWallet` acquires the `cs_wallet` lock during wallet initialization. (`CWalletDB::LoadWallet` calls `ReadKeyValue` which calls `CWallet::LoadToWallet` which calls `CWallet::MarkConflicted`). This is the opposite order that `cs_main` and `cs_wallet` locks are acquired in the rest of the code, and so leads to `POTENTIAL DEADLOCK DETECTED` errors if bitcoin is built with `-DDEBUG_LOCKORDER`.
This commit changes `CWallet::LoadWallet` (which calls `CWalletDB::LoadWallet`) to acquire both locks in the standard order.
Error was reported by @luke-jr in https://botbot.me/freenode/bitcoin-core-dev/msg/90244330/
Tree-SHA512: 353fe21bc0a4a2828b41876897001a3c414d4b115ee7430925bd391d8bc396fca81661145d00996c1ba1a01516d9acf8b89fb5c3da27092f5f3aa7e37ef26ffa
6af49dd Output a bit more information for fee calculation report. (Alex Morcos)
a54c7b9 Fix rounding errors in calculation of minimum change size (Alex Morcos)
Pull request description:
Thanks to @juscamarena for reporting this.
Please backport to 0.15.
There was a potential rounding error where the fee for the change added to the fee for the original tx could be less than the fee for the tx including change.
This is fixed in the first commit. The second commit adds one more snippet of information in the fee calculation report. I actually realized that there is more information that would be nice to report, but we can add that post 0.15.
An open question is whether we should be returning failure if the test in line 2885 is hit or just resetting pick_new_inputs and continuing. Originally I made it a failure to avoid any possible infinite loops. But the case hit here is an example of where that logic possibly backfired.
Tree-SHA512: efe049781acc1f6a8ad429a689359ac6f7b7c44cdfc9578a866dff4a2f6596e8de474a89d25c704f31ef4f8c89af770e98b75ef06c25419d5a6dfc87247bf274
f01103c MOVEONLY: Init functions wallet/wallet.cpp -> wallet/init.cpp (Russell Yanofsky)
e7fe320 MOVEONLY: Fee functions wallet/wallet.cpp -> wallet/fees.cpp (Russell Yanofsky)
d97fe20 Move some static functions out of wallet.h/cpp (Russell Yanofsky)
Pull request description:
This just moves some static wallet fee and init functions out of `wallet/wallet.cpp` and into new `wallet/fees.cpp` and `wallet/init.cpp` source files. There is one commit updating declarations and callers, followed by two MOVEONLY commits actually moving the function bodies.
This change is desirable because wallet.h/cpp are monolithic and hard to navigate, so pulling things out and grouping together pieces of related functionality should improve the organization.
Another motivation is the wallet process separation work in https://github.com/bitcoin/bitcoin/pull/10973, where (at least initially) parameter parsing and fee estimation are still done in the main process rather than the wallet process, and having functions that run in different processes scrambled up throughout wallet.cpp is unnecessarily confusing.
Tree-SHA512: 6e6982ff82b2ab4e681c043907e2b1801ceb9513394730070f16c46ad338278a863f5b3759aa13db76a259b268b1c919c81f4e339f0796a3cfb990161e8c316d
CWallet::MarkConflicted may acquire the cs_main lock after
CWalletDB::LoadWallet acquires the cs_wallet lock during wallet initialization.
(CWalletDB::LoadWallet calls ReadKeyValue which calls CWallet::LoadToWallet
which calls CWallet::MarkConflicted). This is the opposite order that cs_main
and cs_wallet locks are acquired in the rest of the code, and so leads to
POTENTIAL DEADLOCK DETECTED errors if bitcoin is built with -DDEBUG_LOCKORDER.
This commit changes CWallet::LoadWallet (which calls CWalletDB::LoadWallet) to
acquire both locks in the standard order. It also fixes some tests that were
acquiring wallet and main locks out of order and failed with the new locking in
CWallet::LoadWallet.
Error was reported by Luke Dashjr <luke-jr@utopios.org> in
https://botbot.me/freenode/bitcoin-core-dev/msg/90244330/
e53615b Remove vchDefaultKey and have better first run detection (Andrew Chow)
Pull request description:
Removes vchDefaultKey which was only used for first run detection. Improves wallet first run detection by checking to see if any keys were read from the database.
This also fixes a (rather contrived) case where an encrypted non-HD wallet has corruption such that the default key is no longer valid and is loaded into a Core version that supports HD wallets. This causes a runtime exception since a new hd master key is generated as the software believes the wallet file is newly created but cannot add the generated key to the wallet since it is encrypted. I was only able to replicate this error by creating a non-hd wallet, encrypting it, then editing the wallet using `db_dump` and `db_load` before loading the wallet with hd enabled. This problem has been reported by [two](https://bitcointalk.org/index.php?topic=1993244.0) [users](https://bitcointalk.org/index.php?topic=1746976.msg17511261#msg17511261) so it is something that can happen, although that raises the question of "what corrupted the default key".
~P.S. I don't know what's up with the whitespace changes. I think my text editor is doing something stupid but I don't think those are important enough to attempt undoing them.~ Undid those
Tree-SHA512: 63b485f356566e8ffa033ad9b7101f7f6b56372b29ec2a43b947b0eeb1ada4c2cfe24740515d013aedd5f51aa1890dfbe499d2c5c062fc1b5d272324728a7d55
67ceff4 [wallet] Add logging to MarkReserveKeysAsUsed (John Newbery)
1221f60 [wallet] Remove keypool_topup_cleanups (John Newbery)
Pull request description:
A couple of minor cleanups suggested by @ryanofsky here: https://github.com/bitcoin/bitcoin/pull/11022#pullrequestreview-55598940
Does not affect functionality. Not required for v0.15.
Tree-SHA512: d8d0698fd26ea49a4157e68669d5511095760c3a1ecfa3f917e3f273efbafb55c51a202d677614216eae3f796b6e8d17506b2ec2d4799a94f18981b396e65eec
8f2f1e0 wallet: Avoid second mapWallet lookup (João Barbosa)
Pull request description:
All calls to `mapWallet.count()` have the intent to detect if a `txid` exists and most are followed by a second lookup to retrieve the `CWalletTx`.
This PR replaces all `mapWallet.count()` calls with `mapWallet.find()` to avoid the second lookup.
Tree-SHA512: 96b7de7f5520ebf789a1aec1949a4e9c74e13683869cee012f717e5be8e51097d068e2347a36e89097c9a89f1ed1a1529db71760dac9b572e36a3e9ac1155f29
Removes vchDefaultKey which was only used for first run detection.
Improves wallet first run detection by checking to see if any keys
were read from the database.
This will now also check for a valid defaultkey for backwards
compatibility reasons and to check for any corruption.
Keys will stil be generated on the first one, but there won't be
any shown in the address book as was previously done.
This commit just moves a few function declarations and updates callers.
Function bodies are moved in two followup MOVEONLY commits.
This change is desirable because wallet.h/cpp are monolithic and hard to
navigate, so pulling things out and grouping together pieces of related
functionality should improve the organization.
Another proximate motivation is the wallet process separation work in
https://github.com/bitcoin/bitcoin/pull/10973, where (at least initially)
parameter parsing and fee estimation are still done in the main process rather
than the wallet process, and having functions that run in different processes
scrambled up throughout wallet.cpp is unnecessarily confusing.
90d4d89 scripted-diff: Use the C++11 keyword nullptr to denote the pointer literal instead of the macro NULL (practicalswift)
Pull request description:
Since C++11 the macro `NULL` may be:
* an integer literal with value zero, or
* a prvalue of type `std::nullptr_t`
By using the C++11 keyword `nullptr` we are guaranteed a prvalue of type `std::nullptr_t`.
For a more thorough discussion, see "A name for the null pointer: nullptr" (Sutter &
Stroustrup), http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2431.pdf
With this patch applied there are no `NULL` macro usages left in the repo:
```
$ git grep NULL -- "*.cpp" "*.h" | egrep -v '(/univalue/|/secp256k1/|/leveldb/|_NULL|NULLDUMMY|torcontrol.*NULL|NULL cert)' | wc -l
0
```
The road towards `nullptr` (C++11) is split into two PRs:
* `NULL` → `nullptr` is handled in PR #10483 (scripted, this PR)
* `0` → `nullptr` is handled in PR #10645 (manual)
Tree-SHA512: 3c395d66f2ad724a8e6fed74b93634de8bfc0c0eafac94e64e5194c939499fefd6e68f047de3083ad0b4eff37df9a8a3a76349aa17d55eabbd8e0412f140a297
d34957e [wallet] [tests] Add keypool topup functional test (Jonas Schnelli)
095142d [wallet] keypool mark-used and topup (John Newbery)
c25d90f [wallet] Add HasUnusedKeys() helper (John Newbery)
f2123e3 [wallet] Cache keyid -> keypool id mappings (John Newbery)
83f1ec3 [wallet] Don't hold cs_LastBlockFile while calling setBestChain (John Newbery)
2376bfc [wallet] [moveonly] Move LoadKeyPool to cpp (Matt Corallo)
cab8557 [wallet] [moveonly] Move CAffectedKeysVisitor (Jonas Schnelli)
Pull request description:
This PR contains the first part of #10882 :
- if a key from the keypool is used, mark all keys up to that key as used, and then try to top up the keypool
- top up the keypool on startup
Notably, it does not stop the node or prevent the best block from advancing if the keypool drops below a threshold (which means that transactions may be missed and funds lost if restoring from an old HD wallet backup).
Tree-SHA512: ac681fefeaf7ec2aab2fa1da93d12273ea80bd05eb48d7b3b551ea6e5d975dd97ba7de52b7fba52993823280ac4079cc36cf78a27dac708107ebf8fb6326142b
This commit adds basic keypool mark-used and topup:
- try to topup the keypool on initial load
- if a key in the keypool is used, mark all keys before that as used and
try to top up
49d903e Eliminate fee overpaying edge case when subtracting fee from recipients (Alex Morcos)
Pull request description:
I'm not sure if this is the cause of the issue in #10034 , but this was a known edge case. I just didn't realize how simple the fix is.
Could use a couple more eyes to make sure nothing silly can go wrong here, but if we all agree it's this simple, we can add this as another 0.15 bug fix.
Tree-SHA512: db1dd1e83363a3c231267b626d3a388893ee70ba1972056fe2c339c5c9e4fbfd30f7fe837c30cc7be884d454797fd4c619b9d631a8d5eeb55cdb07402a83acb3
095b917 Avoid using sizes on non-fixed-width types to derive protocol constants. (Gregory Maxwell)
Pull request description:
Thanks to awemany for pointing this out.
This replaces #10172 which appears to be abandoned, but uses the constants as requested on that PR.
Tree-SHA512: 032c0d75b3aaf807a7d0c7fb8ff5515acc45ad58bd00fe81413f900fe02bad900534a970403b9bb568e132c9eddea6043e958daf625e8acc84375bd41ee2e2ef
f4d00e6 Add a discard_rate (Alex Morcos)
b138585 Remove factor of 3 from definition of dust. (Alex Morcos)
Pull request description:
The definition of dust is redefined to remove the factor of 3.
Dust is redefined to be the value of an output such that it would
cost that value in fees to (create and) spend the output at the dust
relay rate. The previous definition was that it would cost 1/3 of the
value. The default dust relay rate is correspondingly increased to
3000 sat/kB so the actual default dust output value of 546 satoshis
for a non-segwit output remains unchanged. This commit is a refactor
only unless a dustrelayfee is passed on the commandline in which case
that number now needs to be increased by a factor of 3 to get the same
behavior. -dustrelayfee is a hidden command line option.
Note: It's not exactly a refactor due to edge case changes in rounding
as evidenced by the required change to the unit test.
A discard_rate is added which defaults to 10,000 sat/kB
Any change output which would be dust at the discard_rate you are
willing to discard completely and add to fee (as well as continuing to
pay the fee that would have been needed for creating the change)
This would be a nice addition for 0.15 and I think will remain useful for 0.16 with the new coin selection algorithms in discussion, but its not crucial.
It does add translation strings, but we could (should?) avoid that by hiding the option
Tree-SHA512: 5b6f655354d0ab6b8b6cac1e8d1fe3136d10beb15c6d948fb15bfb105155a9d03684c6240624039b3eed6428b7e60e54216cc8b2f90c4600701e39f646284a9b
fundrawtransaction allows users to add a change output and then
not have it removed from keypool. While it would be nice to have
users follow the normal CreateTransaction/CommitTransaction process
we use internally, there isnt much benefit in exposing this option,
especially with HD wallets, while there is ample room for users to
misunderstand or misuse this option.
This could be particularly nasty in some use-cases (especially
pre-HD-split) - eg a user might fundrawtransaction, then call
getnewaddress, hand out the address for someone to pay them, then
sendrawtransaction. This may result in the user thinking they have
received payment, even though it was really just their own change!
This could obviously result in needless key-reuse.
Any change output which would be dust at the discard_rate you are
willing to discard completely and add to fee (as well as continuing to
pay the fee that would have been needed for creating the change).
This fixes an issue where you could reserve a keypool entry, then
top up the keypool, writing out a new key at the given index, then
return they key from the pool. This isnt likely to cause issues,
but given there is no reason to ever re-use keypool indexes
(they're 64 bits...), best to avoid it alltogether.
b0e8e2d Print one log message per keypool top-up, not one per key. (Gregory Maxwell)
41dc163 Increase wallet default keypool size to 1000. (Gregory Maxwell)
30d8f3a Pushdown walletdb though CWallet::AddKeyPubKey to avoid flushes. (Gregory Maxwell)
3a53f19 Pushdown walletdb object through GenerateNewKey/DeriveNewChildKey. (Gregory Maxwell)
Pull request description:
This carries the walletdb object from top-up through GenerateNewKey/DeriveNewChildKey/CWallet::AddKeyPubKey, which allows us to avoid the flush on destruction until the top up finishes instead of flushing the wallet for every key.
This speeds up adding keys by well over 10x on my laptop (actually something like 17x), I wouldn't be surprised if it were an even bigger speedup on spinning rust.
Then it increases the keypool size to 1000. I would have preferred to use 10,000 but in the case where the user creates a new wallet and then turns on encryption it seems kind of dumb to have >400KB of marked-used born unencrypted keys just laying around.
(Thanks to Matt for cluesticking me on how to bypass the crypter spaghetti)
Tree-SHA512: 868303de38fce4c3f67d7fe133f765f15435c94b39d252d7450b5fee5c607a3cc2f5e531861a69d8c8877bf130e0ff4c539f97500a6bc0ff6d67e4a42c9385c7
11590d3 Properly bound check conf_target in wallet RPC calls (Alex Morcos)
fd29d3d Remove checking of mempool min fee from estimateSmartFee. (Alex Morcos)
2fffaa9 Make QT fee displays use GetMinimumFee instead of estimateSmartFee (Alex Morcos)
1983ca6 Use CoinControl to pass custom fee setting from QT. (Alex Morcos)
03ee701 Refactor to use CoinControl in GetMinimumFee and FeeBumper (Alex Morcos)
ecd81df Make CoinControl a required argument to CreateTransaction (Alex Morcos)
Pull request description:
This builds on #10589 (first 5 commits from that PR, last 5 commits are new)
The first couple commits refactor to use the CCoinControl class to pass fee calculation parameters around.
This allows for fixing the buggy interaction in QT between the global payTxFee which can be modified by the RPC call settxfee or temporarily modified by the QT custom fee settings. Before these changes the GUI could sometimes send a transaction with a recently set payTxFee and not respect the settings displayed in the GUI. After these changes, using the GUI does not involve the global transaction confirm target or payTxFee.
The prospective fee displays in the smart fee slider and the coin control dialog are changed to use the fee calculation from GetMinimumFee, this simplifies the code and makes them slightly more correct in edge cases.
Maxing the fee calculation with the mempool min fee is move from estimateSmartFee to GetMinimumFee.
This fixes a long standing bug, and should be tagged for 0.15 as it is holding up finalizing the estimatesmartfee RPC API before release.
Tree-SHA512: 4d36a1bd5934aa62f3806d380fcafbef73e9fe5bdf190fc5259a3e3a13349e5ce796e50e7068c46dc630ccf56d061bce5804f0bfe2e082bb01ca725b63efd4c1
4c3b538 [logs] fix zapwallettxes startup logs (John Newbery)
e7a2181 [wallet] fix zapwallettxes interaction with persistent mempool (John Newbery)
ff7365e [tests] fix flake8 warnings in zapwallettxes.py (John Newbery)
Pull request description:
zapwallettxes previously did not interact well with persistent mempool.
zapwallettxes would cause wallet transactions to be zapped, but they
would then be reloaded from the mempool on startup. This commit softsets
persistmempool to false if zapwallettxes is enabled so transactions are
actually zapped.
This PR also fixes the zapwallettxes.py functional test, which did not properly test this feature. The test line:
```py
assert_raises(JSONRPCException, self.nodes[0].gettransaction, [txid3])
#there must be a expection because the unconfirmed wallettx0 must be gone by now
```
is not actually testing the presence of the transaction since the RPC is being called incorrectly (with an array instead of a string). The `assert_raises()` passes since an assert is raised, but it's not the one the test writer had in mind!
Fixes#9710 .
Tree-SHA512: e3236efc7a2fd2b3bf1d9e2e8a7726d470c57f5d95cf41b7bde264edc8817bd36a6f3feff52f8de8db0ef64b7247c88b24e7ff7cefaa706cba86fe4e2135a508
