This converts the IsUnspendable function to make use of a combination of
raw script analysis and the new tokenizer instead of the far less
efficient parseScript thereby significantly optimizing the function.
It is important to note that this new implementation intentionally has a
semantic difference from the existing implementation in that it will now
report scripts that are larger than the max allowed script size are
unspendable as well.
Finally, the comment is modified to explicitly call out the script
version semantics.
Note: this function was recently optimized in master, so the gains here
are less noticable than other optimizations.
The following is a before and after comparison of analyzing a large
script:
benchmark old ns/op new ns/op delta
BenchmarkIsUnspendable-8 656 584 -10.98%
benchmark old allocs new allocs delta
BenchmarkIsUnspendable-8 1 0 -100.00%
benchmark old bytes new bytes delta
BenchmarkIsUnspendable-8 1 0 -100.00%
This converts the IsNullData function to analyze the raw script instead
of using the far less efficient parseScript, thereby significantly
optimizing the function.
The following is a before and after comparison of analyzing a large
script:
benchmark old ns/op new ns/op delta
BenchmarkIsNullDataScript-8 62495 2.65 -100.00%
benchmark old allocs new allocs delta
BenchmarkIsNullDataScript-8 1 0 -100.00%
benchmark old bytes new bytes delta
BenchmarkIsNullDataScript-8 311299 0 -100.00%
This converts the IsPayToWitnessScriptHash function to analyze the raw
script instead of using the far less efficient parseScript, thereby
significantly optimizing the function.
In order to accomplish this, it introduces two new functions. The first
one is named extractWitnessScriptHash and works with the raw script byte
to simultaneously deteremine if the script is a p2wsh script, and in the
case that is is, extract and return the hash. The second new function is
named isWitnessScriptHashScript and is defined in terms of the former.
The extract function approach was chosed because it is common for
callers to want to only extract relevant details from a script if the
script is of the specific type. Extracting those details requires
performing the exact same checks to ensure the script is of the correct
type, so it is more efficient to combine the two into one and define the
type determination in terms of the result, so long as the extraction
does not require allocations.
Finally, this also deprecates the isWitnessScriptHash function that
requires opcodes in favor of the new functions and modifies the comment
on IsPayToWitnessScriptHash to call out the script version semantics.
The following is a before and after comparison of executing
IsPayToWitnessScriptHash on a large script:
benchmark old ns/op new ns/op delta
BenchmarkIsWitnessScriptHash-8 62774 0.63 -100.00%
benchmark old allocs new allocs delta
BenchmarkIsWitnessScriptHash-8 1 0 -100.00%
benchmark old bytes new bytes delta
BenchmarkIsWitnessScriptHash-8 311299 0 -100.00%
This converts the IsPayToWitnessPubKeyHash function to analyze the raw
script instead of the far less efficient parseScript, thereby
significantly optimizing the function.
In order to accomplish this, it introduces two new functions. The first
one is named extractWitnessPubKeyHash and works with the raw script
bytes to simultaneously deteremine if the script is a p2wkh, and in case
it is, extract and return the hash. The second new function is name
isWitnessPubKeyHashScript which is defined in terms of the former.
The extract function is approach was chosen because it is common for
callers to want to only extract relevant details from the script if the
script is of the specific type. Extracting those details requires the
exact same checks to ensure the script is of the correct type, so it is
more efficient to combine the two and define the type determination in
terms of the result so long as the extraction does not require
allocations.
Finally, this deprecates the isWitnessPubKeyHash function that requires
opcodes in favor of the new functions and modifies the comment on
IsPayToWitnessPubKeyHash to explicitly call out the script version
semantics.
The following is a before and after comparison of executing
IsPayToWitnessPubKeyHash on a large script:
benchmark old ns/op new ns/op delta
BenchmarkIsWitnessPubKeyHash-8 68927 0.53 -100.00%
benchmark old allocs new allocs delta
BenchmarkIsWitnessPubKeyHash-8 1 0 -100.00%
benchmark old bytes new bytes delta
BenchmarkIsWitnessPubKeyHash-8 311299 0 -100.00%
This converts the IsPushOnlyScript function to make use of the new
tokenizer instead of the far less efficient parseScript thereby
significantly optimizing the function.
It also deprecates the isPushOnly function that requires opcodes in
favor of the new function and modifies the comment on IsPushOnlyScript
to explicitly call out the script version semantics.
The following is a before and after comparison of analyzing a large
script:
benchmark old ns/op new ns/op delta
BenchmarkIsPushOnlyScript-8 62412 622 -99.00%
benchmark old allocs new allocs delta
BenchmarkIsPushOnlyScript-8 1 0 -100.00%
benchmark old bytes new bytes delta
BenchmarkIsPushOnlyScript-8 311299 0 -100.00%
This converts the IsMultisigSigScript function to analyze the raw script
and make use of the new tokenizer instead of the far less efficient
parseScript thereby significantly optimizing the function.
In order to accomplish this, it first rejects scripts that can't
possibly fit the bill due to the final byte of what would be the redeem
script not being the appropriate opcode or the overall script not having
enough bytes. Then, it uses a new function that is introduced named
finalOpcodeData that uses the tokenizer to return any data associated
with the final opcode in the signature script (which will be nil for
non-push opcodes or if the script fails to parse) and analyzes it as if
it were a redeem script when it is non nil.
It is also worth noting that this new implementation intentionally has
the same semantic difference from the existing implementation as the
updated IsMultisigScript function in regards to allowing zero pubkeys
whereas previously it incorrectly required at least one pubkey.
Finally, the comment is modified to explicitly call out the script
version semantics.
The following is a before and after comparison of analyzing a large
script that is not a multisig script and both a 1-of-2 multisig public
key script (which should be false) and a signature script comprised of a
pay-to-script-hash 1-of-2 multisig redeem script (which should be true):
benchmark old ns/op new ns/op delta
BenchmarkIsMultisigSigScriptLarge-8 69328 2.93 -100.00%
BenchmarkIsMultisigSigScript-8 2375 146 -93.85%
benchmark old allocs new allocs delta
BenchmarkIsMultisigSigScriptLarge-8 5 0 -100.00%
BenchmarkIsMultisigSigScript-8 3 0 -100.00%
benchmark old bytes new bytes delta
BenchmarkIsMultisigSigScriptLarge-8 330035 0 -100.00%
BenchmarkIsMultisigSigScript-8 9472 0 -100.00%
This converts the IsPayToScriptHash function to analyze the raw script
instead of using the far less efficient parseScript thereby
significantly optimizing the function.
In order to accomplish this, it introduces two new functions. The first
one is named extractScriptHash and works with the raw script bytes to
simultaneously determine if the script is a p2sh script, and in the case
it is, extract and return the hash. The second new function is named
isScriptHashScript and is defined in terms of the former.
The extract function approach was chosen because it is common for
callers to want to only extract relevant details from a script if the
script is of the specific type. Extracting those details requires
performing the exact same checks to ensure the script is of the correct
type, so it is more efficient to combine the two into one and define the
type determination in terms of the result so long as the extraction does
not require allocations.
Finally, this also deprecates the isScriptHash function that requires
opcodes in favor of the new functions and modifies the comment on
IsPayToScriptHash to explicitly call out the script version semantics.
The following is a before and after comparison of analyzing a large
script that is not a p2sh script:
benchmark old ns/op new ns/op delta
BenchmarkIsPayToScriptHash-8 62393 0.60 -100.00%
benchmark old allocs new allocs delta
BenchmarkIsPayToScriptHash-8 1 0 -100.00%
benchmark old bytes new bytes delta
BenchmarkIsPayToScriptHash-8 311299 0 -100.00%
This converts the IsPayToPubKeyHash function to analyze the raw script
instead of using the far less efficient parseScript, thereby
significantly optimization the function.
In order to accomplish this, it introduces two new functions. The first
one is named extractPubKeyHash and works with the raw script bytes
to simultaneously determine if the script is a pay-to-pubkey-hash script,
and in the case it is, extract and return the hash. The second new
function is named isPubKeyHashScript and is defined in terms of the
former.
The extract function approach was chosen because it is common for
callers to want to only extract relevant details from a script if the
script is of the specific type. Extracting those details requires
performing the exact same checks to ensure the script is of the correct
type, so it is more efficient to combine the two into one and define the
type determination in terms of the result so long as the extraction does
not require allocations.
The following is a before and after comparison of analyzing a large
script:
benchmark old ns/op new ns/op delta
BenchmarkIsPubKeyHashScript-8 62228 0.45 -100.00%
benchmark old allocs new allocs delta
BenchmarkIsPubKeyHashScript-8 1 0 -100.00%
benchmark old bytes new bytes delta
BenchmarkIsPubKeyHashScript-8 311299 0 -100.00%
This converts the IsPayToScriptHash function to analyze the raw script
instead of using the far less efficient parseScript, thereby
significantly optimizing the function.
In order to accomplish this, it introduces four new functions:
extractCompressedPubKey, extractUncompressedPubKey, extractPubKey, and
isPubKeyScript. The extractPubKey function makes use of
extractCompressedPubKey and extractUncompressedPubKey to combine their
functionality as a convenience and isPubKeyScript is defined in terms of
extractPubKey.
The extractCompressedPubKey works with the raw script bytes to
simultaneously determine if the script is a pay-to-compressed-pubkey
script, and in the case it is, extract and return the raw compressed
pubkey bytes.
Similarly, the extractUncompressedPubKey works in the same way except it
determines if the script is a pay-to-uncompressed-pubkey script and
returns the raw uncompressed pubkey bytes in the case it is.
The extract function approach was chosen because it is common for
callers to want to only extract relevant details from a script if the
script is of the specific type. Extracting those details requires
performing the exact same checks to ensure the script is of the correct
type, so it is more efficient to combine the two into one and define the
type determination in terms of the result so long as the extraction does
not require allocations.
The following is a before and after comparison of analyzing a large
script:
benchmark old ns/op new ns/op delta
BenchmarkIsPubKeyScript-8 62323 2.97 -100.00%
benchmark old allocs new allocs delta
BenchmarkIsPubKeyScript-8 1 0 -100.00%
benchmark old bytes new bytes delta
BenchmarkIsPubKeyScript-8 311299 0 -100.00%
This converts the asSmallInt function to accept an opcode as a byte
instead of the internal opcode data struct in order to make it more
flexible for raw script analysis.
It also updates all callers accordingly.
This converts the isSmallInt function to accept an opcode as a byte
instead of the internal opcode data struct in order to make it more
flexible for raw script analysis.
The comment is modified to explicitly call out the script version
semantics.
Finally, it updates all callers accordingly.
This modifies the CalcSignatureHash function to make use of the new
signature hash calculation function that accepts raw scripts without
needing to first parse them. Consequently, it also doubles as a slight
optimization to the execution time and a significant reduction in the
number of allocations.
In order to convert the CalcScriptHash function and keep the same
semantics, a new function named checkScriptParses is introduced which
will quickly determine if a script can be fully parsed without failure
and return the parse failure in the case it can't.
The following is a before and after comparison of analyzing a large
multiple input transaction:
benchmark old ns/op new ns/op delta
BenchmarkCalcSigHash-8 3627895 3619477 -0.23%
benchmark old allocs new allocs delta
BenchmarkCalcSigHash-8 1335 801 -40.00%
benchmark old bytes new bytes delta
BenchmarkCalcSigHash-8 1373812 1293354 -5.86%
This introduces a new function named calcSignatureHashRaw which accepts
the raw script bytes to calculate the script hash versus requiring the
parsed opcode only to unparse them later in order to make it more
flexible for working with raw scripts.
Since there are several places in the rest of the code that currently
only have access to the parsed opcodes, this modifies the existing
calcSignatureHash to first unparse the script before calling the new
function.
Backport of decred/dcrd:f306a72a16eaabfb7054a26f9d9f850b87b00279
This converts the DisasmString function to make use of the new
zero-allocation script tokenizer instead of the far less efficient
parseScript thereby significantly optimizing the function.
In order to facilitate this, the opcode disassembly functionality is
split into a separate function called disasmOpcode that accepts the
opcode struct and data independently as opposed to requiring a parsed
opcode. The new function also accepts a pointer to a string builder so
the disassembly can be more efficiently be built.
While here, the comment is modified to explicitly call out the script
version semantics.
The following is a before and after comparison of a large script:
benchmark old ns/op new ns/op delta
BenchmarkDisasmString-8 102902 40124 -61.01%
benchmark old allocs new allocs delta
BenchmarkDisasmString-8 46 51 +10.87%
benchmark old bytes new bytes delta
BenchmarkDisasmString-8 389324 130552 -66.47%
- create benchmarks to measure allocations
- add test for benchmark input
- create a low alloc parseScriptTemplate
- refactor parsing logic for a single opcode
This modifies calcSignatureHash to use a shallow copy of the transaction
versus a deep copy since the actual scripts themselves are not modified
and therefore don't need to be copied.
This is being done because profiling the most overall allocated space
shows that the deep copy performed in calcSignatureHash accounts for
nearly 20% of all allocations on a synced running instance. Also,
copying all of the additional data makes it more time consuming as well.
With this change, that figure drops from ~20% to ~5% of all allocations.
The following benchmark shows the relative speedups and allocation
reduction as a result of the optimization on my system. In particular,
the changes result in approximately a 15% speedup and a whopping 99.89%
reduction in allocations when using a large transaction with thousands
of inputs which was the worst case scenario.
benchmark old allocs new allocs delta
--------------------------------------------------------------------
BenchmarkCalcSignatureHash 11151 12 -99.89%
benchmark old ns/op new ns/op delta
--------------------------------------------------------------------
BenchmarkCalcSignatureHash 3599845 3056359 -15.10%
This commit introduces a series of internal and external helper
functions which enable the txscript package to be aware of the new
standard script templates introduced as part of BIP0141. The two new
standard script templates recognized are pay-to-witness-key-hash
(P2WKH) and pay-to-witness-script-hash (P2WSH).
This commit implements most of BIP0143 by adding logic to implement the
new sighash calculation, signing, and additionally introduces the
HashCache optimization which eliminates the O(N^2) computational
complexity for the SIGHASH_ALL sighash type.
The HashCache struct is the equivalent to the existing SigCache struct,
but for caching the reusable midstate for transactions which are
spending segwitty outputs.
This converts the majority of script errors from generic errors created
via errors.New and fmt.Errorf to use a concrete type that implements the
error interface with an error code and description.
This allows callers to programmatically detect the type of error via
type assertions and an error code while still allowing the errors to
provide more context.
For example, instead of just having an error the reads "disabled opcode"
as would happen prior to these changes when a disabled opcode is
encountered, the error will now read "attempt to execute disabled opcode
OP_FOO".
While it was previously possible to programmatically detect many errors
due to them being exported, they provided no additional context and
there were also various instances that were just returning errors
created on the spot which callers could not reliably detect without
resorting to looking at the actual error message, which is nearly always
bad practice.
Also, while here, export the MaxStackSize and MaxScriptSize constants
since they can be useful for consumers of the package and perform some
minor cleanup of some of the tests.
This is mostly a backport of some of the same modifications made in
Decred along with a few additional things cleaned up. In particular,
this updates the code to make use of the new chainhash package.
Also, since this required API changes anyways and the hash algorithm is
no longer tied specifically to SHA, all other functions throughout the
code base which had "Sha" in their name have been changed to Hash so
they are not incorrectly implying the hash algorithm.
The following is an overview of the changes:
- Remove the wire.ShaHash type
- Update all references to wire.ShaHash to the new chainhash.Hash type
- Rename the following functions and update all references:
- wire.BlockHeader.BlockSha -> BlockHash
- wire.MsgBlock.BlockSha -> BlockHash
- wire.MsgBlock.TxShas -> TxHashes
- wire.MsgTx.TxSha -> TxHash
- blockchain.ShaHashToBig -> HashToBig
- peer.ShaFunc -> peer.HashFunc
- Rename all variables that included sha in their name to include hash
instead
- Update for function name changes in other dependent packages such as
btcutil
- Update copyright dates on all modified files
- Update glide.lock file to use the required version of btcutil
This changes the script template parsing function to use a pointer into
the constant global opcode array for parsed opcodes as opposed to making
a copy of the opcode entries which causes unnecessary allocations.
Profiling showed that after roughly 48 hours of operation, this
copy was the culprit of 207 million unnecessary allocations.
500 tests with various transactions and scripts, verifying that
calcSignatureHash generates the expected hash in each case.
This requires changing SigHashType to uint32; that won't affect the
standard use-cases, but will make calcSignatureHash behave more like the
Core counterpart for non-standard SigHashType settings, like those in
some of these tests.
This commit modifies the DisasmString function to use a bytes buffer for
constructing the disassembled string instead of naive string
concatenation. This makes a huge difference when disassembling scripts
with large numbers of opcodes.
IsUnspendable takes a public key script and returns whether it is
spendable.
Additionally, hook this into the mempool isDust function, since
unspendable outputs can't be spent.
This mimics Bitcoin Core commit 0aad1f13b2430165062bf9436036c1222a8724da
This commit moves all code related to standard scripts into a separate
file named standard.go as well as the associated tests into
standard_test.go. Since the code in address.go and address_test.go is
only related to standard scripts, it has been combined into the new
files and the old files deleted.
The intent here is to make it clear that the code in standard.go is not
related to consensus.
This commit contains a lot of cleanup on the txscript code to make it
more consistent with the code throughout the rest of the project. It
doesn't change any operational logic.
The following is an overview of the changes:
- Add a significant number of comments throughout in order to better
explain what the code is doing
- Fix several comment typos
- Move a couple of constants only used by the engine to engine.go
- Move a variable only used by the engine to engine.go
- Fix a couple of format specifiers in the test prints
- Reorder functions so they're defined before/closer to use
- Make the code lint clean with the exception of the opcode definitions
This commit converts the opcode map to an array to improve performance.
Benchmark of executing a standard p2pk transaction:
New: BenchmarkExecute 2000 784349 ns/op
Old: BenchmarkExecute 2000 792600 ns/op
The time is dominated by the signature checking as expected, however there
is still an increase in speed.
This commit renames the Script type to Engine to better reflect its
purpose. It also renames the NewScript function to NewEngine to match.
This is being done because name Script for the engine is confusing since
it implies it is an actual script rather than the execution environment
for the script. It also paves the way for eventually supplying a
ParsedScript type which will be less likely to be confused with the
execution environment.
While moving the code, some additional variable names and comments have
been updated to better match the style used throughout the rest of the
code base. In addition, an attempt has been made to use consistent naming
of the engine as 'vm' instead of using different variables names as it was
previously.
Finally, the relevant engine code has been moved into a new file named
engine.go and related tests moved to engine_test.go.
This commit removes the unnecessary sigScript parameter from the
txscript.NewScript function. This has bothered me for a while because it
can and really should be obtained from the provided transaction and input
index. The way it was, the passed script could technically be different
than what is in the transaction. Obviously that would be an improper use
of the API, but it's safer and more convenient to simply pull it from the
provided transaction and index.
Also, since the function signature is changing anyways, make the input
index parameter come after the transaction which it references.
The ScriptVerifyLowS flag defines that script signatures must
comply with the DER format as well as have an S value less than
or equal to the half order.
The ScriptVerifyCleanStack flag requires that only a single
stack element remains after evaluation and that when interpreted
as a bool, it must be true. This is BIP0062, rule 6.
This mimics Bitcoin Core commit b6e03cc59208305681745ad06f2056ffe6690597
This change increases the maximum allowed bytes allowed in pushed
data to be considered a nulldata transaction. This matches the current
value the reference implementation uses by default.
By exporting StandardVerifyFlags, clients can ensure they create
transactions that btcd will accept into its mempool.
This flag doesn't belong in txscript. It belongs in a
policy package. However, this is currently the least worse place.
The ScriptVerifyMinimalData enforces that all push operations use the
minimal data push required. This is part of BIP0062.
This commit mimics Bitcoin Core commit
698c6abb25c1fbbc7fa4ba46b60e9f17d97332ef
