38bc1ec Make more json-like output from estimaterawfee (Alex Morcos)
2d2e170 Comments and improved documentation (Alex Morcos)
ef589f8 minor cleanup: remove unnecessary variable (Alex Morcos)
3ee76d6 Introduce a scale factor (Alex Morcos)
5f1f0c6 Historical block span (Alex Morcos)
aa19b8e Clean up fee estimate debug printing (Alex Morcos)
10f7cbd Track first recorded height (Alex Morcos)
3810e97 Rewrite estimateSmartFee (Alex Morcos)
c7447ec Track failures in fee estimation. (Alex Morcos)
4186d3f Expose estimaterawfee (Alex Morcos)
2681153 minor refactor: explicitly track start of new bucket range and don't update curNearBucket on final loop. (Alex Morcos)
1ba43cc Make EstimateMedianVal smarter about small failures. (Alex Morcos)
d3e30bc Refactor to update moving average on fly (Alex Morcos)
e5007ba Change parameters for fee estimation and estimates on all 3 time horizons. (Alex Morcos)
c0a273f Change file format for fee estimates. (Alex Morcos)
Tree-SHA512: 186e7508d86a1f351bb656edcd84ee9091f5f2706331eda9ee29da9c8eb5bf67b8c1f2abf6662835560e7f613b1377099054f20767f41ddcdbc89c4f9e78946d
For the per confirmation number tracking of data, introduce a scale factor so that in the longer horizones confirmations are bucketed together at a resolution of the scale. (instead of 1008 individual data points for each fee bucket, have 42 data points each covering 24 different confirmation values.. (1-24), (25-48), etc.. )
Store in fee estimate file the block span for which we were tracking estimates, so we know what targets we can successfully evaluate with the data in the file. When restarting use either this historical block span to set valid range of targets until our current span of tracking estimates is just as long.
Track the first time we seen txs in a block that we have been tracking in our mempool. Used to evaluate validity of fee estimates for different targets.
Change the logic of estimateSmartFee to check a 60% threshold at half the target, a 85% threshold at the target and a 95% threshold at double the target. Always check the shortest time horizon possible and ensure that estimates are monotonically decreasing. Add a conservative mode, which makes sure that the 95% threshold is also met at longer time horizons as well.
Track information the ranges of fee rates that were used to calculate the fee estimates (the last range of fee rates in which the data points met the threshold and the first to fail) and provide an RPC call to return this information.
Instead of stopping if it encounters a "sufficient" number of transactions which don't meet the threshold for being confirmed within the target, it keeps looking to add more transactions to see if there is a temporary blip in the data. This allows a smaller number of required data points.
Move buckets and bucketMap to be stored as part of overall serialization of estimator.
Add some placeholder data so file format is only changed once.
Maintain 3 different TxConfirmStats with potential for different decays and scales.
This changes the logging categories to boolean flags instead of strings.
This simplifies the acceptance testing by avoiding accessing a scoped
static thread local pointer to a thread local set of strings. It
eliminates the only use of boost::thread_specific_ptr outside of
lockorder debugging.
This change allows log entries to be directed to multiple categories
and makes it easy to change the logging flags at runtime (e.g. via
an RPC, though that isn't done by this commit.)
It also eliminates the fDebug global.
Configuration of unknown logging categories now produces a warning.
All decisions about whether the transactions are valid data points are made at the time the transaction arrives. Updating on blocks all the time will now cause stale fee estimates to decay quickly when we restart a node.
Fee estimation can just check its own mapMemPoolTxs to determine the same information. Note that now fee estimation for block processing must happen before those transactions are removed, but this shoudl be a speedup.
Once priority estimation was removed, not all transactions in the mempool are tracked in the fee estimation mempool tracking. So there is no error if a transaction is not found for removal.
f8d43b8 Avoid rollingMinimumFeeRate never being able to decay below half (Alex Morcos)
eab8e1b fix a bug if the min fee is 0 for FeeFilterRounder (Alex Morcos)
76faa3c Rename the remaining main.{h,cpp} to validation.{h,cpp} (Matt Corallo)
e736772 Move network-msg-processing code out of main to its own file (Matt Corallo)
87c35f5 Remove orphan state wipe from UnloadBlockIndex. (Matt Corallo)
Three categories of modifications:
1)
1 instance of 'The Bitcoin Core developers \n',
1 instance of 'the Bitcoin Core developers\n',
3 instances of 'Bitcoin Core Developers\n', and
12 instances of 'The Bitcoin developers\n'
are made uniform with the 443 instances of 'The Bitcoin Core developers\n'
2)
3 instances of 'BitPay, Inc\.\n' are made uniform with the other 6
instances of 'BitPay Inc\.\n'
3)
4 instances where there was no '(c)' between the 'Copyright' and the year
where it deviates from the style of the local directory.
There are only a few uses of `insecure_random` outside the tests.
This PR replaces uses of insecure_random (and its accompanying global
state) in the core code with an FastRandomContext that is automatically
seeded on creation.
This is meant to be used for inner loops. The FastRandomContext
can be in the outer scope, or the class itself, then rand32() is used
inside the loop. Useful e.g. for pushing addresses in CNode or the fee
rounding, or randomization for coin selection.
As a context is created per purpose, thus it gets rid of
cross-thread unprotected shared usage of a single set of globals, this
should also get rid of the potential race conditions.
- I'd say TxMempool::check is not called enough to warrant using a special
fast random context, this is switched to GetRand() (open for
discussion...)
- The use of `insecure_rand` in ConnectThroughProxy has been replaced by
an atomic integer counter. The only goal here is to have a different
credentials pair for each connection to go on a different Tor circuit,
it does not need to be random nor unpredictable.
- To avoid having a FastRandomContext on every CNode, the context is
passed into PushAddress as appropriate.
There remains an insecure_random for test usage in `test_random.h`.
c59c434 qa: Add test for standardness of segwit v0 outputs (Suhas Daftuar)
1ffaff2 Make witness v0 outputs non-standard before segwit activation (Johnson Lau)
The "feefilter" p2p message is used to inform other nodes of your mempool min fee which is the feerate that any new transaction must meet to be accepted to your mempool. This will allow them to filter invs to you according to this feerate.
- Replace NOP3 with CHECKSEQUENCEVERIFY (BIP112)
<nSequence> CHECKSEQUENCEVERIFY -> <nSequence>
- Fails if txin.nSequence < nSequence, allowing funds of a txout to be locked for a number of blocks or a duration of time after its inclusion in a block.
- Pull most of CheckLockTime() out into VerifyLockTime(), a local function that will be reused for CheckSequence()
- Add bitwise AND operator to CScriptNum
- Enable CHECKSEQUENCEVERIFY as a standard script verify flag
- Transactions that fail CSV verification will be rejected from the mempool, making it easy to test the feature. However blocks containing "invalid" CSV-using transactions will still be accepted; this is *not* the soft-fork required to actually enable CSV for production use.
SequenceLocks functions are used to evaluate sequence lock times or heights per BIP 68.
The majority of this code is copied from maaku in #6312
Further credit: btcdrak, sipa, NicolasDorier
Add "bip125-replaceable" output field to listtransactions and gettransaction
which indicates if an unconfirmed transaction, or any unconfirmed parent, is
signaling opt-in RBF according to BIP 125.
Make RPC tests have a default block priority size of 50000 (the old default) so we can still use free transactions in RPC tests. When priority is eliminated, we will have to make a different change if we want to continue allowing free txs.
This provides more conservative estimates and reacts more quickly to a backlog.
Unfortunately the unit test for fee estimation depends on the success threshold (and the decay) chosen; also modify the unit test for the new default success thresholds.
These are more useful fee and priority estimation functions. If there is no fee/pri high enough for the target you are aiming for, it will give you the estimate for the lowest target that you can reliably obtain. This is better than defaulting to the minimum. It will also pass back the target for which it returned an answer.
Revert "Revert "Add rules--presently disabled--for using GetMedianTimePast as endpoint for lock-time calculations""
This reverts commit 40cd32e835.
After careful analysis it was determined that the change was, in fact, safe and several people were suffering
momentary confusion about locktime semantics.
This reverts commit 9d55050773.
As noted by Luke-Jr, under some conditions this will accept transactions which are invalid by the network
rules. This happens when the current block time is head of the median time past and a transaction's
locktime is in the middle.
This could be addressed by changing the rule to MAX(this_block_time, MTP+offset) but this solution and
the particular offset used deserve some consideration.
Transactions are not allowed in the memory pool or selected for inclusion in a block until their lock times exceed chainActive.Tip()->GetMedianTimePast(). However blocks including transactions which are only mature under the old rules are still accepted; this is *not* the soft-fork required to actually rely on the new constraint in production.
The lock-time code currently uses CBlock::nTime as the cutoff point for time based locked transactions. This has the unfortunate outcome of creating a perverse incentive for miners to lie about the time of a block in order to collect more fees by including transactions that by wall clock determination have not yet matured. By using CBlockIndex::GetMedianTimePast from the prior block instead, the self-interested miner no longer gains from generating blocks with fraudulent timestamps. Users can compensate for this change by simply adding an hour (3600 seconds) to their time-based lock times.
If enforced, this would be a soft-fork change. This commit only adds the functionality on an unexecuted code path, without changing the behaviour of Bitcoin Core.
This adds SCRIPT_VERIFY_LOW_S to STANDARD_SCRIPT_VERIFY_FLAGS which
will make the node require the canonical 'low-s' encoding for
ECDSA signatures when relaying or mining.
Consensus behavior is unchanged.
The rational is explained in a81cd96805:
Absent this kind of test ECDSA is not a strong signature as given
a valid signature {r, s} both that value and {r, -s mod n} are valid.
These two encodings have different hashes allowing third parties a
vector to change users txids. These attacks are avoided by picking
a particular form as canonical and rejecting the other form(s); in
the of the LOW_S rule, the smaller of the two possible S values is
used.
If widely deployed this change would eliminate the last remaining
known vector for nuisance malleability on boring SIGHASH_ALL
p2pkh transactions. On the down-side it will block most
transactions made by sufficiently out of date software.
Unlike the other avenues to change txids on boring transactions this
one was randomly violated by all deployed bitcoin software prior to
its discovery. So, while other malleability vectors where made
non-standard as soon as they were discovered, this one has remained
permitted. Even BIP62 did not propose applying this rule to
old version transactions, but conforming implementations have become
much more common since BIP62 was initially written.
Bitcoin Core has produced compatible signatures since a28fb70e in
September 2013, but this didn't make it into a release until 0.9
in March 2014; Bitcoinj has done so for a similar span of time.
Bitcoinjs and electrum have been more recently updated.
This does not replace the need for BIP62 or similar, as miners can
still cooperate to break transactions. Nor does it replace the
need for wallet software to handle malleability sanely[1]. This
only eliminates the cheap and irritating DOS attack.
[1] On the Malleability of Bitcoin Transactions
Marcin Andrychowicz, Stefan Dziembowski, Daniel Malinowski, Łukasz Mazurek
http://fc15.ifca.ai/preproceedings/bitcoin/paper_9.pdf
Previously only one PUSHDATA was allowed, needlessly limiting
applications such as matching OP_RETURN contents with bloom filters that
operate on a per-PUSHDATA level. Now any combination that passes
IsPushOnly() is allowed, so long as the total size of the scriptPubKey
is less than 42 bytes. (unchanged modulo non-minimal PUSHDATA encodings)
Also, this fixes the odd bug where previously the PUSHDATA could be
replaced by any single opcode, even sigops consuming opcodes such as
CHECKMULTISIG. (20 sigops!)
This class groups transactions that have been confirmed in blocks into buckets, based on either their fee or their priority. Then for each bucket, the class calculates what percentage of the transactions were confirmed within various numbers of blocks. It does this by keeping an exponentially decaying moving history for each bucket and confirm block count of the percentage of transactions in that bucket that were confirmed within that number of blocks.
-Eliminate txs which didn't have all inputs available at entry from fee/pri calcs
-Add dynamic breakpoints and tracking of confirmation delays in mempool transactions
-Remove old CMinerPolicyEstimator and CBlockAverage code
-New smartfees.py
-Pass a flag to the estimation code, using IsInitialBlockDownload as a proxy for when we are still catching up and we shouldn't be counting how many blocks it takes for transactions to be included.
-Add a policyestimator unit test
