Commit graph

3483 commits

Author SHA1 Message Date
Dave Collins
c39e35318d Update for recent chainhash-related API changes. (#81)
This updates all code to make use of the new chainhash package since the
old wire.ShaHash type and related functions have been removed in favor
of the abstracted package.

Also, while here, rename all variables that included sha in their name
to include hash instead.
2016-08-08 14:16:04 -05:00
Dave Collins
bd4e64d1d4 chainhash: Abstract hash logic to new package. (#729)
This is mostly a backport of some of the same modifications made in
Decred along with a few additional things cleaned up.  In particular,
this updates the code to make use of the new chainhash package.

Also, since this required API changes anyways and the hash algorithm is
no longer tied specifically to SHA, all other functions throughout the
code base which had "Sha" in their name have been changed to Hash so
they are not incorrectly implying the hash algorithm.

The following is an overview of the changes:

- Remove the wire.ShaHash type
- Update all references to wire.ShaHash to the new chainhash.Hash type
- Rename the following functions and update all references:
  - wire.BlockHeader.BlockSha -> BlockHash
  - wire.MsgBlock.BlockSha -> BlockHash
  - wire.MsgBlock.TxShas -> TxHashes
  - wire.MsgTx.TxSha -> TxHash
  - blockchain.ShaHashToBig -> HashToBig
  - peer.ShaFunc -> peer.HashFunc
- Rename all variables that included sha in their name to include hash
  instead
- Update for function name changes in other dependent packages such as
  btcutil
- Update copyright dates on all modified files
- Update glide.lock file to use the required version of btcutil
2016-08-08 14:04:33 -05:00
Dave Collins
b6b1e55d1e TravisCI: Set vendor experiment variable earlier. (#731)
This sets the GO15VENDOREXPERIMENT environment variable before glide is
installed so vendoring is used when installing it as well.
2016-08-08 13:31:17 -05:00
Dave Collins
711f33450c chainhash: Implement a new chainhash package. (#730)
This is a backport of the chainhash package made in Decred along with a
few additional things cleaned up, finished test coverage, and rewording
of some documentation to make it more generic.

In particular, the new package provides the definition of the hash type
and associated hashing functions which will allow the rest of the code to be
agnostic to the specific hash algorithm.

This only implements the package and does not change any of the code
base over to use it.
2016-08-08 12:05:51 -05:00
Dave Collins
d406d9e52b wire: Consolidate tests into the wire pkg. (#728)
Putting the test code in the same package makes it easier for forks
since they don't have to change the import paths as much and it also
gets rid of the need for internal_test.go to bridge.

This same thing should probably be done for the majority of the code
base.
2016-08-08 11:42:54 -05:00
Calvin McAnarney
5f19113422 Allow RPC identifiers in E notation (#80) 2016-08-07 22:47:07 -05:00
David Hill
6e644855f5 Bump travis to use the latest golang - 1.6.3 (#723) 2016-07-25 13:48:19 -05:00
David Hill
61a15f6f1b blockchain: optimize HaveBlock (#720)
If a block is known to exist in the memory chain or database then
there is no need to check the orphan pool.
2016-07-25 10:16:57 -05:00
Dave Collins
00ebb9d14d blockchain: Associate time src with chain instance.
Rather than making the caller to pass in the median time source on
ProcessBlock and IsCurrent, modify the Config struct to include the
median time source and associate it with the chain instance when it is
created.

This is being done because both the ProcessBlock and IsCurrent functions
require access to the blockchain state already, it is a little bit safer
to ensure the time source matches the chain instance state, it
simplifies the caller logic, and it also allows its use within the logic
of the blockchain package itself which will be required by upcoming
rule change warning logic that is part of BIP9.
2016-07-14 13:10:47 -05:00
Jonathan Gillham
1ffc3dc18d peer: Fix logging of connected peer. 2016-06-24 13:39:50 +01:00
Jonathan Gillham
f3d759d783 peer: Extract protocol negotiation from main read and write code paths.
This allows cleaner separation of the half-duplex version negotiation from the fully duplex message passing between peers.
2016-06-24 13:18:33 +01:00
Jonathan Gillham
777ccdade3 peer: Remove error return from Connect. 2016-06-24 13:12:01 +01:00
Jonathan Gillham
5cbd1f85bf peer: Remove potential race when calling Connect. 2016-06-24 13:12:01 +01:00
David Hill
7de7bddba9 peer: use atomics instead of mutexes (#670) 2016-06-20 14:34:21 -05:00
Hector Jusforgues
ff4ada0b0e Add automatic RPC configuration. 2016-06-03 21:14:15 -05:00
Dave Collins
6229e35835 wire: Further reduce transaction allocs.
This commit drastically reduces the number of allocations needed to
deserialize a transaction and its scripts by using the combination of a
free list for initially deserializing the individual scripts along with
copying them into a single contiguous byte slice after the final size is
known and modifying each script in the transaction to point to its
location within the contiguous blob.

The end result is only a single allocation that holds all of the scripts
for a transaction regardless of the total number of scripts it has.

The script free list allows a maximum of 12,500 items with each buffer
being 512 bytes.  This implies it will have a peak usage of 6.1MB.  The
values were chosen based on profiling data and a desire to allow at
least 100 scripts per transaction to be simultaneously deserialized by
125 peers.

Also, while optimizing, decode directly into the existing previous
outpoint structure of each transaction input in order to avoid the extra
allocation per input that is otherwise caused when the local escapes to
the heap.

The following is a before and after comparison of the allocations
with the benchmarks that did not change removed:

benchmark              old allocs     new allocs     delta
-----------------------------------------------------------
ReadTxOut              1              0              -100.00%
ReadTxIn               2              0              -100.00%
DeserializeTxSmall     7              5              -28.57%
DeserializeTxLarge     11146          6              -99.95%
2016-06-03 17:09:14 -05:00
Dave Collins
2adfb3b56a wire: Reduce allocs with contiguous slices.
The current code involves a ton of small allocations which is harsh on
the garbage collector and in turn causes a lot of addition runtime
overhead both in terms of additional memory and processing time.

In order to improve the situation, this drasticially reduces the number
of allocations by creating contiguous slices of objects and
deserializing into them.  Since the final data structures consist of
slices of pointers to the objects, they are constructed by pointing them
into the appropriate offset of the contiguous slice.

This could be improved upon even further by converting all of the data
structures provided the wire package to be slices of contiguous objects
directly, however that would be a major breaking API change and would
end up requiring updating a lot more code in every caller.  I do think
that ultimately the API should be changed, but the changes in this
commit already makes a massive difference and it doesn't require
touching any of the callers, so it is a good place to begin.

The following is a before and after comparison of the allocations
with the benchmarks that did not change removed:

benchmark              old allocs     new allocs     delta
-----------------------------------------------------------
DeserializeTxLarge     16715          11146          -33.32%
DecodeGetHeaders       501            2              -99.60%
DecodeHeaders          2001           2              -99.90%
DecodeGetBlocks        501            2              -99.60%
DecodeAddr             3001           2002           -33.29%
DecodeInv              50003          3              -99.99%
DecodeNotFound         50002          3              -99.99%
DecodeMerkleBlock      107            3              -97.20%
2016-06-03 17:08:31 -05:00
Dave Collins
5de5b7354c wire: Avoid allocation on timestamp decodes.
Since the protocol encodes timestamps differently depending on the
message, the code currently decodes into a local variable and then
converts it to a time.Time.  However, this causes an allocation due to
the local having to escape to the heap in order for the readElement
function to write to it.

So, in order to avoid that, this introduces two new types for a
timestamp named uint32Time and int64Time that are encoded as the
respective type on the read.  When calling the readElements function,
the time.Time field in the message is cast to a pointer of the
appropriate type which effectively allows the allocations to be avoided.

The following is a before and after comparison of the allocations
with the benchmarks that did not change removed:

benchmark              old allocs     new allocs     delta
----------------------------------------------------------------------
ReadBlockHeader        1              0              -100.00%
DecodeHeaders          4001           2001           -49.99%
DecodeAddr             4001           3001           -24.99%
DecodeMerkleBlock      108            107            -0.93%
2016-06-03 17:08:31 -05:00
Dave Collins
f68cd7422d wire: Reduce allocs with a binary free list.
This introduces a new binary free list which provides a concurrent safe
list of unused buffers for the purpose of serializing and deserializing
primitive integers to their raw binary bytes.

For convenience, the type also provides functions for each of the
primitive unsigned integers that automatically obtain a buffer from the
free list, perform the necessary binary conversion, read from or write
to the given io.Reader or io.Writer, and return the buffer to the free
list.

A global instance of the type has been introduced with a maximum number
of 1024 items. Since each buffer is 8 bytes, it will consume a maximum
of 8KB.  Theoretically, this value would only allow up to 1024 peers
simultaneously reading and writing without having to resort to burdening
the garbage collector with additional allocations.  However, due to the
fact the code is designed in such a way that the buffers are quickly
used and returned to the free list, in practice it can support much more
than 1024 peers without involving the garbage collector since it is
highly unlikely every peer would need a buffer at the exact same time.

The following is a before and after comparison of the allocations
with the benchmarks that did not change removed:

benchmark              old allocs     new allocs     delta
-------------------------------------------------------------
WriteVarInt1           1              0              -100.00%
WriteVarInt3           1              0              -100.00%
WriteVarInt5           1              0              -100.00%
WriteVarInt9           1              0              -100.00%
ReadVarInt1            1              0              -100.00%
ReadVarInt3            1              0              -100.00%
ReadVarInt5            1              0              -100.00%
ReadVarInt9            1              0              -100.00%
ReadVarStr4            3              2              -33.33%
ReadVarStr10           3              2              -33.33%
WriteVarStr4           2              1              -50.00%
WriteVarStr10          2              1              -50.00%
ReadOutPoint           1              0              -100.00%
WriteOutPoint          1              0              -100.00%
ReadTxOut              3              1              -66.67%
WriteTxOut             2              0              -100.00%
ReadTxIn               5              2              -60.00%
WriteTxIn              3              0              -100.00%
DeserializeTxSmall     15             7              -53.33%
DeserializeTxLarge     33428          16715          -50.00%
SerializeTx            8              0              -100.00%
ReadBlockHeader        7              1              -85.71%
WriteBlockHeader       10             4              -60.00%
DecodeGetHeaders       1004           501            -50.10%
DecodeHeaders          18002          4001           -77.77%
DecodeGetBlocks        1004           501            -50.10%
DecodeAddr             9002           4001           -55.55%
DecodeInv              150005         50003          -66.67%
DecodeNotFound         150004         50002          -66.67%
DecodeMerkleBlock      222            108            -51.35%
TxSha                  10             2              -80.00%
2016-06-03 17:08:31 -05:00
Dave Collins
dc83f4ee6a addblock: Add support for generating indexes.
This adds two new flags, --txindex and --addrindex, to the addblock
utility which mirror the flags on btcd.  They serve to to specify that
the transaction index and/or address index, respectively, should be
built while importing from the bootstrap file.

This is technically not 100% required since btcd will build the indexes
on the first load (when enabled) if they aren't already built, however
it is much faster to build the indexes as the blocks are being validated
(particularly for the address index), so this makes the capability
available.
2016-06-03 17:06:16 -05:00
Mawueli Kofi Adzoe
7f07fb1093 txscript: Fix typo. (#700)
* Fix tiny typo. Bump copyright year.
* Clarify documentation.
2016-05-22 23:23:20 -05:00
Mawueli Kofi Adzoe
e8e2167a1a Bump up copyright. Reflect recent update. (#699) 2016-05-22 23:22:42 -05:00
Steven Roose
24e41c843b Update installation instructions using Glide (#698)
The main README.md file had newer installation that uses Glide, while these did not.
2016-05-21 17:30:26 -05:00
Nathan Bass
f893558d78 Minor correction to GenerateSharedSecret documentation. (#696) 2016-05-14 22:56:29 -05:00
Dave Collins
2554caee59 build: Convert project to use glide. (#689)
This converts the project to allow btcd to be used with the glide
package manager in order to provide stable and reproducible builds
without the user having to jump through all of the hoops as they do
today.

It consists of adding a glide.yaml file which identifies the project
dependencies and locations along with a glide.lock file which contains
the complete dependency tree pinned to specific versions.  Glide uses
these files to download the packages (or updates) to a local vendor
directory and checkout the correct pinned versions.  The go tool, in
turn, is used to build/install btcd and will use the pinned versions in
the vendor directory.

This also updates TravisCI to build using glide, removes some of the
exceptions in the lint checks which are no longer required, and updates
the README.md with the new instructions needed to build the project with
glide.
2016-05-06 10:47:53 -05:00
Dave Collins
128366734f main: Limit garbage collection percentage. (#686)
This reduces the target ratio of freshly allocated data to live data to
10% in order to limit excessive overallocations by the garbage collector
during data bursts such as processing complex blocks or rapidly
receiving a lot of large transactions.
2016-05-05 14:16:58 -05:00
David Hill
1a0e7452f3 btcd: handle signal SIGTERM (#688)
When an OS reboots or shuts down, it sends all processes SIGTERM before
sending SIGKILL.  This allows btcd to do a proper shutdown which most
importantly closes the database.
2016-05-05 14:16:42 -05:00
Jonathan Gillham
0d7f526600 mining: Correctly format log messages. (#685)
Type feePerKB is an int64 and therefore is not correctly interpreted by
format verb f.
2016-04-27 14:09:23 -05:00
David Hill
1b23410214 btcd: sendheaders server support (#671)
This adds support for serving headers instead of inventory messages in
accordance with BIP0130.  btcd itself does not yet make use of the
feature when receiving data.
2016-04-26 13:24:03 -05:00
Dave Collins
b14032487f wire: Add several decode benchmarks. (#682)
This adds decode benchmarks for several of the messages that profiling
has identified to cause a lot of allocations in addition to those that
already exist.  By adding these benchmarks, it makes it easier to get
allocation and speed statistics which can in turn be used to compare
future improvements.

The following bencharmarks have been added:

DecodeGetHeaders, DecodeHeaders, DecodeGetBlocks, DecodeAddr, DecodeInv,
DecodeNotFound, and DecodeMerkleBlock

For reference, here is the benchmark data as of this commit.

DecodeGetHeaders     93261 ns/op     24120 B/op     1004 allocs/op
DecodeHeaders      2071263 ns/op    368399 B/op    18002 allocs/op
DecodeGetBlocks      92486 ns/op     24120 B/op     1004 allocs/op
DecodeAddr          850608 ns/op    136202 B/op     9002 allocs/op
DecodeInv         17107172 ns/op   3601447 B/op   150004 allocs/op
DecodeNotFound    17522225 ns/op   3601444 B/op   150004 allocs/op
DecodeMerkleBlock    21062 ns/op      5192 B/op      222 allocs/op
2016-04-25 17:32:29 -05:00
Dave Collins
e7ddaa468e wire: Don't allocate new readers in benchmarks. (#679)
This modifies the benchmarks in the wire package to avoid creating a new
reader for each iteration.  This is useful since it means that showing
the memory allocations will only show the function under test instead of
the allocation for the benchmark setup as well.

The following is a before and after comparison of the allocations
with the benchmarks that did not change removed:

benchmark              old allocs     new allocs     delta
------------------------------------------------------------
ReadVarInt1            2              1              -50.00%
ReadVarInt3            2              1              -50.00%
ReadVarInt5            2              1              -50.00%
ReadVarInt9            2              1              -50.00%
ReadVarStr4            4              3              -25.00%
ReadVarStr10           4              3              -25.00%
ReadOutPoint           2              1              -50.00%
ReadTxOut              4              3              -25.00%
ReadTxIn               6              5              -16.67%
DeserializeTxSmall     16             15             -6.25%
DeserializeTxLarge     33430          33428          -0.01%
ReadBlockHeader        8              7              -12.50%
2016-04-25 16:57:44 -05:00
Dave Collins
27c0f9f8d1 wire: Add large tx deserialize benchmark. (#678)
This adds a benchmark for deserializing a large transaction that is
often referred to as the megatransaction since it is the largest Bitcoin
transaction mined to date.  It consists of 5569 inputs and 1 output and
its hash is:

bb41a757f405890fb0f5856228e23b715702d714d59bf2b1feb70d8b2b4e3e08.

This is being done so there is a benchmark that tests more of a
worst-case scenario which is a better candidate for identifying and
testing improvements.

The following benchmark results shows the how much more intensive this
transaction is over the existing mock transaction:

DeserializeTxSmall  1000000    1751 ns/op      376 B/op     16 allocs/op
DeserializeTxLarge  300     5093980 ns/op  1672829 B/op  33430 allocs/op
2016-04-25 16:51:27 -05:00
Dave Collins
de4fb24389 blockchain: Remove unused root field. (#680)
This removes the root field and all references to it from the BlockChain
since it is no longer required.

It was previously required because the chain state was not initialized
when the instance was created.  However, that is no longer the case, so
there is no reason to keep it around any longer.
2016-04-25 16:17:29 -05:00
Dave Collins
644570487f txscript: Reduce script parse opcode allocs. (#677)
This changes the script template parsing function to use a pointer into
the constant global opcode array for parsed opcodes as opposed to making
a copy of the opcode entries which causes unnecessary allocations.

Profiling showed that after roughly 48 hours of operation, this
copy was the culprit of 207 million unnecessary allocations.
2016-04-25 16:17:07 -05:00
Dave Collins
b87723cd94 btcd: Remove peer-specific logging funcs. (#675)
This removes the logging functions that are now implemented in the peer
package as they are no longer used by btcd itself and should have been
removed when they were copied into the peer package.
2016-04-20 23:58:31 -05:00
David Hill
474547b211 travis: run tests on latest golang (1.5.4 and 1.6.1) (#669) 2016-04-14 13:38:25 -05:00
David Hill
a1bb291b28 mempool: Have ProcessTransaction return accepted transactions. (#547)
It is not the responsibility of mempool to relay transactions, so
return a slice of transactions accepted to the mempool due to the
passed transaction to the caller.
2016-04-14 12:58:09 -05:00
Dave Collins
e15d3008cf mining: Improve tests for prio queue. (#667)
This improves the tests of the priority queue to include the secondary
sort ordering as well as adds some manual entries to ensure the edge
conditions are properly tested.

This also brings the priority queue test coverage up to 100%.
2016-04-14 00:19:23 -05:00
Tadge Dryja
432ad76952 fix memory allignment for 32-bit architectures (#668)
having 3 int32s above the uint64s in the struct
will cause misalignment for some 32-bit architectures.
see https://golang.org/pkg/sync/atomic/#pkg-note-BUG
This aligns bytesReceived and bytesSent.
2016-04-13 22:51:02 -05:00
Olaoluwa Osuntokun
3b39edcaa1 txscript: optimize sigcache lookup (#598)
Profiles discovered that lookups into the signature cache included an
expensive comparison to the stored `sigInfo` struct. This lookup had the
potential to be more expensive than directly verifying the signature
itself!

In addition, evictions were rather expensive because they involved
reading from /dev/urandom, or equivalent, for each eviction once the
signature cache was full as well as potentially iterating over every
item in the cache in the worst-case.

To remedy this poor performance several changes have been made:
* Change the lookup key to the fixed sized 32-byte signature hash
* Perform a full equality check only if there is a cache hit which
    results in a significant  speed up for both insertions and existence
checks
* Override entries in the case of a colliding hash on insert Add an
* .IsEqual() method to the Signature and PublicKey types in the
  btcec package to facilitate easy equivalence testing
* Allocate the signature cache map with the max number of entries in
  order to avoid unnecessary map re-sizes/allocations
* Optimize evictions from the signature cache Delete the first entry
* seen which is safe from manipulation due to
    the pre image resistance of the hash function
* Double the default maximum number of entries within the signature
  cache due to the reduction in the size of a cache entry
  * With this eviction scheme, removals are effectively O(1)

Fixes #575.
2016-04-13 21:56:10 -05:00
Dave Collins
5a1e77bd2d blockchain: Remove unneeded unspentness size check. (#665)
The current code is needlessly checking the number of bytes needed to
serialize the unspentness bitmap in the utxo against a maximum value
that could never be returned because the function takes a uint32 output
index which is treated as a bit offset, and converts it bytes, which
will necessarily be less than a max uint32.

This check also causes a compile error on arm where native integers are
32 bits.

This simply removes the unneeded check.
2016-04-13 21:43:16 -05:00
Dave Collins
5b14e157ea mining: Add tests for prio queue. (#666)
This adds a test to ensure the priority queue works properly both for
sorting by fee per KB and priorities.

Thanks to @ceejep for the original test code and idea which was
subsequently modified and cleaned up a bit to the code here.
2016-04-13 14:36:53 -05:00
Dave Collins
b580cdb7d3 database: Replace with new version.
This commit removes the old database package, moves the new package into
its place, and updates all imports accordingly.
2016-04-12 14:55:15 -05:00
Dave Collins
7c174620f7 indexers: Implement optional tx/address indexes.
This introduces a new indexing infrastructure for supporting optional
indexes using the new database and blockchain infrastructure along with
two concrete indexer implementations which provide both a
transaction-by-hash and a transaction-by-address index.

The new infrastructure is mostly separated into a package named indexers
which is housed under the blockchain package.  In order to support this,
a new interface named IndexManager has been introduced in the blockchain
package which provides methods to be notified when the chain has been
initialized and when blocks are connected and disconnected from the main
chain.  A concrete implementation of an index manager is provided by the
new indexers package.

The new indexers package also provides a new interface named Indexer
which allows the index manager to manage concrete index implementations
which conform to the interface.

The following is high level overview of the main index infrastructure
changes:

- Define a new IndexManager interface in the blockchain package and
  modify the package to make use of the interface when specified
- Create a new indexers package
  - Provides an Index interface which allows concrete indexes to plugin
    to an index manager
  - Provides a concrete IndexManager implementation
    - Handles the lifecycle of all indexes it manages
    - Tracks the index tips
    - Handles catching up disabled indexes that have been reenabled
    - Handles reorgs while the index was disabled
    - Invokes the appropriate methods for all managed indexes to allow
      them to index and deindex the blocks and transactions
  - Implement a transaction-by-hash index
    - Makes use of internal block IDs to save a significant amount of
      space and indexing costs over the old transaction index format
  - Implement a transaction-by-address index
    - Makes use of a leveling scheme in order to provide a good tradeoff
      between space required and indexing costs
- Supports enabling and disabling indexes at will
- Support the ability to drop indexes if they are no longer desired

The following is an overview of the btcd changes:

- Add a new index logging subsystem
- Add new options --txindex and --addrindex in order to enable the
  optional indexes
  - NOTE: The transaction index will automatically be enabled when the
    address index is enabled because it depends on it
- Add new options --droptxindex and --dropaddrindex to allow the indexes
  to be removed
  - NOTE: The address index will also be removed when the transaction
    index is dropped because it depends on it
- Update getrawtransactions RPC to make use of the transaction index
- Reimplement the searchrawtransaction RPC that makes use of the address
  index
- Update sample-btcd.conf to include sample usage for the new optional
  index flags
2016-04-11 17:16:42 -05:00
Dave Collins
491acd4ca6 blockchain: Rework to use new db interface.
This commit is the first stage of several that are planned to convert
the blockchain package into a concurrent safe package that will
ultimately allow support for multi-peer download and concurrent chain
processing.  The goal is to update btcd proper after each step so it can
take advantage of the enhancements as they are developed.

In addition to the aforementioned benefit, this staged approach has been
chosen since it is absolutely critical to maintain consensus.
Separating the changes into several stages makes it easier for reviewers
to logically follow what is happening and therefore helps prevent
consensus bugs.  Naturally there are significant automated tests to help
prevent consensus issues as well.

The main focus of this stage is to convert the blockchain package to use
the new database interface and implement the chain-related functionality
which it no longer handles.  It also aims to improve efficiency in
various areas by making use of the new database and chain capabilities.

The following is an overview of the chain changes:

- Update to use the new database interface
- Add chain-related functionality that the old database used to handle
  - Main chain structure and state
  - Transaction spend tracking
- Implement a new pruned unspent transaction output (utxo) set
  - Provides efficient direct access to the unspent transaction outputs
  - Uses a domain specific compression algorithm that understands the
    standard transaction scripts in order to significantly compress them
  - Removes reliance on the transaction index and paves the way toward
    eventually enabling block pruning
- Modify the New function to accept a Config struct instead of
  inidividual parameters
- Replace the old TxStore type with a new UtxoViewpoint type that makes
  use of the new pruned utxo set
- Convert code to treat the new UtxoViewpoint as a rolling view that is
  used between connects and disconnects to improve efficiency
- Make best chain state always set when the chain instance is created
  - Remove now unnecessary logic for dealing with unset best state
- Make all exported functions concurrent safe
  - Currently using a single chain state lock as it provides a straight
    forward and easy to review path forward however this can be improved
    with more fine grained locking
- Optimize various cases where full blocks were being loaded when only
  the header is needed to help reduce the I/O load
- Add the ability for callers to get a snapshot of the current best
  chain stats in a concurrent safe fashion
  - Does not block callers while new blocks are being processed
- Make error messages that reference transaction outputs consistently
  use <transaction hash>:<output index>
- Introduce a new AssertError type an convert internal consistency
  checks to use it
- Update tests and examples to reflect the changes
- Add a full suite of tests to ensure correct functionality of the new
  code

The following is an overview of the btcd changes:

- Update to use the new database and chain interfaces
- Temporarily remove all code related to the transaction index
- Temporarily remove all code related to the address index
- Convert all code that uses transaction stores to use the new utxo
  view
- Rework several calls that required the block manager for safe
  concurrency to use the chain package directly now that it is
  concurrent safe
- Change all calls to obtain the best hash to use the new best state
  snapshot capability from the chain package
- Remove workaround for limits on fetching height ranges since the new
  database interface no longer imposes them
- Correct the gettxout RPC handler to return the best chain hash as
  opposed the hash the txout was found in
- Optimize various RPC handlers:
  - Change several of the RPC handlers to use the new chain snapshot
    capability to avoid needlessly loading data
  - Update several handlers to use new functionality to avoid accessing
    the block manager so they are able to return the data without
    blocking when the server is busy processing blocks
  - Update non-verbose getblock to avoid deserialization and
    serialization overhead
  - Update getblockheader to request the block height directly from
    chain and only load the header
  - Update getdifficulty to use the new cached data from chain
  - Update getmininginfo to use the new cached data from chain
  - Update non-verbose getrawtransaction to avoid deserialization and
    serialization overhead
  - Update gettxout to use the new utxo store versus loading
    full transactions using the transaction index

The following is an overview of the utility changes:
- Update addblock to use the new database and chain interfaces
- Update findcheckpoint to use the new database and chain interfaces
- Remove the dropafter utility which is no longer supported

NOTE: The transaction index and address index will be reimplemented in
another commit.
2016-04-11 16:47:27 -05:00
David Hill
123ff368f4 mempool: Create and use mempoolPolicy. (#571)
mempoolPolicy contains the values that configure the mempool policy.
This decouples the values from the internals of btcd to move closer
to a mempool package.
2016-04-11 16:37:52 -05:00
Dave Collins
5ff5fc5fa2 txscript: Correct comments on alt stack methods. (#657) 2016-04-11 14:22:25 -05:00
Dave Collins
1d83cd5721 chaincfg: Consolidate tests into the chaincfg pkg. (#662)
Putting the test code in the same package makes it easier for forks
since they don't have to change the import paths as much and it also
gets rid of the need for internal_test.go to bridge.

This same thing should probably be done for the majority of the code
base.
2016-04-11 14:21:40 -05:00
Dave Collins
c7e5d56b58 chaincfg: Register networks instead of hard coding. (#660)
This modifies the chaincfg package to register the default network
params via the init function instead of manually hard coding their data
into the maps.  This is less error prone when adding new default
networks.

A new function named mustRegister has been introduced that panics if
there are any errors when registering the network that the new code
makes use of and appropriate tests have been added.
2016-04-11 12:34:28 -05:00
Dave Collins
23f59144c7 server: Optimize map limiting in block manager. (#658)
This optimizes the way in which the maps are limited by the block
manager.

Previously the code would read a cryptographically random value large
enough to construct a hash, find the first entry larger than that value,
and evict it.

That approach is quite inefficient and could easily become a bottleneck
when processing transactions due to the need to read from a source such
as /dev/urandom and all of the subsequent hash comparisons.

Luckily, strong cryptographic randomness is not needed here. The primary
intent of limiting the maps is to control memory usage with a secondary
concern of making it difficult for adversaries to force eviction of
specific entries.

Consequently, this changes the code to make use of the pseudorandom
iteration order of Go's maps along with the preimage resistance of the
hashing function to provide the desired functionality.  It has
previously been discussed that the specific pseudorandom iteration order
is not guaranteed by the Go spec even though in practice that is how it
is implemented.  This is not a concern however because even if the
specific compiler doesn't implement that, the preimage resistance of the
hashing function alone is enough.

Thanks to @Roasbeef for pointing out the efficiency concerns and the
fact that strong cryptographic randomness is not necessary.
2016-04-11 10:29:07 -05:00