Commit graph

3338 commits

Author SHA1 Message Date
Olaoluwa Osuntokun
088eec67ec
peer: ensure the version negotiation goroutine will always exit
In this commit, we patch a goroutine leak within the peer struct. This
goroutine leak can happen, if the remote side fails to actually finish
negotiation the protocol before our timeout ticker ticks. In this case,
the goroutine will be blocked on a send, as the channel is unfired. An
example trace from a btcd node I had on testnet showed:
```
3183 @ 0x42e78a 0x42e83e 0x40540b 0x4051a5 0x872f76 0x45bfc1
```

So, all instances of the goroutine failing to exit due to the remote
peer not finishing the p2p version negotiation handshake.

Our fix is simple: make the `negotiateErr' channel unbuffered. With this
simple change, we ensure that the goroutine will always exit even in the
case that the parent goroutine exists due to a timeout.  # Please enter
the commit message for your changes. Lines starting
2018-07-21 17:41:23 -07:00
Olaoluwa Osuntokun
f673a4b563
Merge pull request #1215 from davecgh/blockchain_generalize_reorganizechain
blockchain: Generalize chain reorg.
2018-07-17 20:36:26 -07:00
Olaoluwa Osuntokun
bc3861a5a2
Merge pull request #1218 from wpaulino/filter-type-verification
server: ensure we only fetch filters we know of
2018-07-16 20:16:23 -07:00
Olaoluwa Osuntokun
5e86c37411
server: fix bug in cf checkpoint serving
In this commit, we fix a bug in the way that we previously attempted to
server cfcheckpoints. In the prior version we would never actually
fetch the current length of the cache. As a result, after the first time
the checkpoints were fetched, we would always continually grow the
cache rather than using what's there if sufficient.

In this commit, we fix this behavior by always checking the length, then
either keeping the rite lock, or downgrading to a read lock if the size
was sufficient.
2018-07-16 16:31:11 -07:00
Olaoluwa Osuntokun
f7366fb51b
server: ensure we only fetch filters we know of 2018-07-14 12:20:20 -07:00
Dave Collins
b091d71646
blockchain: Generalize chain reorg.
This generalizes the reorganizeChain function in the blockchain package
to allow it to rewind the chain in the case no attach nodes are provided
or extend the chain in the case no detach nodes are provided.

It also adds several assertions to ensure the assumptions about the
state hold and cleans up the handling of setting invalid ancestor nodes
in the case of a failed block validation.
2018-07-11 19:46:30 -05:00
Olaoluwa Osuntokun
fdfc19097e
Merge pull request #1211 from wpaulino/gcs-modifications
multi: update gcs indexing and serving code to match latest BIP 158 instance
2018-07-06 18:25:21 -05:00
Olaoluwa Osuntokun
21b0303341
wire: remove instances of the extended gcs filter 2018-07-06 16:15:25 -07:00
Olaoluwa Osuntokun
a05f62fabd
btcjson: remove instances of the extended gcs filter 2018-07-06 16:15:24 -07:00
Olaoluwa Osuntokun
298efd8359
blockchain/indexers: ensure the cfindex gets inputs 2018-07-06 16:15:24 -07:00
Olaoluwa Osuntokun
102ca293f6
blockchain: pass in prevScript for constructing the basic filter 2018-07-06 16:15:24 -07:00
Olaoluwa Osuntokun
576800a99e
blockchain/indexer: remove extended filter 2018-07-06 16:15:23 -07:00
Olaoluwa Osuntokun
942116c5ae
rpc: remove extended fitler from help 2018-07-06 16:15:23 -07:00
Olaoluwa Osuntokun
66d632f32e
wire: remove the extended filter 2018-07-06 16:15:23 -07:00
Olaoluwa Osuntokun
28d4261c2d build: update glide for btcutil w/ latest BIP 158 2018-07-06 16:08:15 -07:00
Olaoluwa Osuntokun
991d32e72f
Merge pull request #1210 from wpaulino/indexing-use-spend-journal
blockchain: modify indexers to use the stxo set for a block rather than the utxo set
2018-07-06 17:02:24 -05:00
Wilmer Paulino
1d99af29c7
btcjson: fix formatting directives 2018-07-06 13:06:00 -07:00
Tzu-Jung Lee
98769ddbdf docs: Update example links in README 2018-07-06 14:41:20 -05:00
Steven Roose
fbe1bfa659 server: print full name of cf index in main log 2018-07-06 14:40:27 -05:00
Olaoluwa Osuntokun
c32d30c567
Merge pull request #1193 from cfromknecht/btcec-decompression-fix
btcec: verify square root during point decompression
2018-07-06 14:39:47 -05:00
Conner Fromknecht
347cd3839f
btcec/signature_test: adds small pubkey recovery tests 2018-06-12 17:31:03 -07:00
Olaoluwa Osuntokun
a26e2634fa
blockchain/indexers: update indexing to use stxos instead of utxo view for blocks
In this commit, we update all the indexers to use the stxo set for a
particular block rather than the utxo view for the block. We do this as
we can eliminate a large number of random reads for each block, and can
instead deserialize a single instance of all the outputs spent in that
block and feed in the prev input scripts to each indexer.
2018-05-30 20:47:40 -07:00
Olaoluwa Osuntokun
ad69a7121b blockchain: update IndexManager interface to use spent txos
In this commit, we update the IndexManager interface to use spent txos
rather than the unspent output set for a particualr block. We do this in
order to improve the performance of the current address index which
requires reconstructing the utxo view from the PoV of that new block. In
practice, this is very slow as we need to perform a series of random
reads in order to reconstruct the utxo set. Instead, we can use the set
of SpentTxOut's for that block as this already contains the previous
output scripts which is what all of the current indexers really need.
2018-05-30 20:46:12 -07:00
Olaoluwa Osuntokun
e4d82bd6e2 blockchain: publicly export spentTxOut and all attributes
In this commit, we publicly export the spentTxOut struct and all its
attributes. This is the first in a set of commits to optimize the
existing address index by using the spend journal rather than manually
re-creating the utxoViewPoint each time.
2018-05-30 20:46:12 -07:00
Olaoluwa Osuntokun
4bd5b1a43a blockchain: add new FetchSpendJournal method 2018-05-30 20:46:12 -07:00
Olaoluwa Osuntokun
86fed78113
Merge pull request #1045 from davecgh/multi_utxoentry_outpoints
multi: Rework utxoset/view to use outpoints.
2018-05-30 19:59:44 -07:00
Conner Fromknecht
bd1d6c9148
btcec/pubkey: verify decompressed y-coord is sqroot 2018-05-29 17:06:32 -07:00
Dave Collins
a59ac5b18f
multi: Rework utxoset/view to use outpoints.
This modifies the utxoset in the database and related UtxoViewpoint to
store and work with unspent transaction outputs on a per-output basis
instead of at a transaction level.  This was inspired by similar recent
changes in Bitcoin Core.

The primary motivation is to simplify the code, pave the way for a
utxo cache, and generally focus on optimizing runtime performance.

The tradeoff is that this approach does somewhat increase the size of
the serialized utxoset since it means that the transaction hash is
duplicated for each output as a part of the key and some additional
details such as whether the containing transaction is a coinbase and the
block height it was a part of are duplicated in each output.

However, in practice, the size difference isn't all that large, disk
space is relatively cheap, certainly cheaper than memory, and it is much
more important to provide more efficient runtime operation since that is
the ultimate purpose of the daemon.

While performing this conversion, it also simplifies the code to remove
the transaction version information from the utxoset as well as the
spend journal.  The logic for only serializing it under certain
circumstances is complicated and it isn't actually used anywhere aside
from the gettxout RPC where it also isn't used by anything important
either.  Consequently, this also removes the version field of the
gettxout RPC result.

The utxos in the database are automatically migrated to the new format
with this commit and it is possible to interrupt and resume the
migration process.

Finally, it also updates the tests for the new format and adds a new
function to the tests to convert the old test data to the new format for
convenience.  The data has already been converted and updated in the
commit.

An overview of the changes are as follows:

- Remove transaction version from both spent and unspent output entries
  - Update utxo serialization format to exclude the version
  - Modify the spend journal serialization format
    - The old version field is now reserved and always stores zero and
      ignores it when reading
    - This allows old entries to be used by new code without having to
      migrate the entire spend journal
  - Remove version field from gettxout RPC result
- Convert UtxoEntry to represent a specific utxo instead of a
  transaction with all remaining utxos
  - Optimize for memory usage with an eye towards a utxo cache
    - Combine details such as whether the txout was contained in a
      coinbase, is spent, and is modified into a single packed field of
      bit flags
    - Align entry fields to eliminate extra padding since ultimately
      there will be a lot of these in memory
    - Introduce a free list for serializing an outpoint to the database
      key format to significantly reduce pressure on the GC
  - Update all related functions that previously dealt with transaction
    hashes to accept outpoints instead
  - Update all callers accordingly
  - Only add individually requested outputs from the mempool when
    constructing a mempool view
- Modify the spend journal to always store the block height and coinbase
  information with every spent txout
  - Introduce code to handle fetching the missing information from
    another utxo from the same transaction in the event an old style
    entry is encountered
    - Make use of a database cursor with seek to do this much more
      efficiently than testing every possible output
- Always decompress data loaded from the database now that a utxo entry
  only consists of a specific output
- Introduce upgrade code to migrate the utxo set to the new format
  - Store versions of the utxoset and spend journal buckets
  - Allow migration process to be interrupted and resumed
- Update all tests to expect the correct encodings, remove tests that no
  longer apply, and add new ones for the new expected behavior
  - Convert old tests for the legacy utxo format deserialization code to
    test the new function that is used during upgrade
  - Update the utxostore test data and add function that was used to
    convert it
- Introduce a few new functions on UtxoViewpoint
  - AddTxOut for adding an individual txout versus all of them
  - addTxOut to handle the common code between the new AddTxOut and
    existing AddTxOuts
  - RemoveEntry for removing an individual txout
  - fetchEntryByHash for fetching any remaining utxo for a given
    transaction hash
2018-05-27 03:07:41 -05:00
Conner Fromknecht
bc09449045 glide: update btcutil deps 2018-05-23 20:51:14 -07:00
Olaoluwa Osuntokun
d52471044a build: update glide to point to latest btcutil 2018-05-23 16:46:15 -07:00
Olaoluwa Osuntokun
4cb1c950e0 rpc: add help for new gcs commands 2018-05-23 16:46:15 -07:00
Olaoluwa Osuntokun
5596b63846 multi: fix linter warnings 2018-05-23 16:46:15 -07:00
Olaoluwa Osuntokun
2b7326ae52 blockchain/indexers: use latest API for cf indexing 2018-05-23 16:46:15 -07:00
Olaoluwa Osuntokun
f48bc43421 build: restore glide.lock 2018-05-23 16:46:15 -07:00
Olaoluwa Osuntokun
b72e16f0d6 multi: correct all import paths 2018-05-23 16:46:15 -07:00
Johan T. Halseth
3082ae92c3 rpcwebsocket: notify spends already in the mempool
This commit adds a logic to the addSpentRequests that inspects the
mempool for any spends of the outputs. Before this commit, a spend would
only be checked when a transaction was first accepted into the mempool.
2018-05-23 16:46:15 -07:00
Johan T. Halseth
c758834800 mempool_test: add TestCheckSpend 2018-05-23 16:46:15 -07:00
Johan T. Halseth
5394ca1afe mempool: add CheckSpend method
This commit adds a new method CheckSpend to the mempool, which takes an
outpoint and returns any transaction in the mempool that spends this
outpoint.
2018-05-23 16:46:15 -07:00
Olaoluwa Osuntokun
ceb1caeb24 wire: increase MaxGetCFiltersReqRange from 100 to 1000 2018-05-23 16:46:15 -07:00
Jim Posen
7b67f61fa6 rpcclient: Put CFHeader in PrevFilterHeader field in RPC response. 2018-05-23 16:46:15 -07:00
Jim Posen
4d0e856ea1 server: Handler for getcfcheckpt messages. 2018-05-23 16:46:15 -07:00
Jim Posen
0581e18840 wire: Define CFCheckpt message. 2018-05-23 16:46:15 -07:00
Jim Posen
336b18c584 wire: Define GetCFCheckpt message. 2018-05-23 16:46:15 -07:00
Jim Posen
4c991c8783 wire: Populate cfheaders message with filter hashes instead of headers. 2018-05-23 16:46:15 -07:00
Jim Posen
175af18043 multi: Modify CFHeaders message to have a PrevFilterHeader field. 2018-05-23 16:46:15 -07:00
Jim Posen
7a53a05878 multi: Redefine GetCFHeaders to have StartHeight and StopHash. 2018-05-23 16:46:15 -07:00
Jim Posen
daac60675e multi: Redefine GetCFilters to have StartHeight and StopHash. 2018-05-23 16:46:15 -07:00
Jim Posen
3425d33506 blockchain/indexers: Add methods on CfIndex for batch retrieval.
Fetching all items in one db transaction will save time when
responding to getcfilters or getcfheaders requests.
2018-05-23 16:46:15 -07:00
Jim Posen
e617483b44 blockchain/indexers: Store filter hashes with cfindex. 2018-05-23 16:46:15 -07:00
Jim Posen
d07fd2f333 blockchain/indexers: Reduce duplication in cfindex.
The index will hold three types of entries for each filter type, block
pair: filter, header, and hash. Since they all have similar methods
and implementations, refactor to reduce duplication.
2018-05-23 16:46:15 -07:00