Commit graph

3378 commits

Author SHA1 Message Date
Dave Collins
8965d88893
peer: Strictly enforce bloom filter service bit.
This makes the enforcement of the bloom filter service bit much more
strict.  In particular, it does the following:

- Moves the enforcement of the bloom filter service bit out of the peer
  package and into the server so the server can ban as necessary
- Disconnect peers that send filter commands when the server is
  configured to disable them regardless of the protocol version
- Bans peers that are a high enough protocol version that they are
  supposed to observe the service bit is disabled, but ignore it and
  send filter commands regardless.

As an added bonus, this fixes the old logic which had a bug in that it
was examining the *remote* peer's supported services in order to choose
whether or not to disconnect instead of the *local* server's supported
services.
2016-10-16 02:19:28 -05:00
Dave Collins
77913ad2e8
blockchain: Expose main chain flag on ProcessBlock.
This modifies the blockchain.ProcessBlock function to return an
additional boolean as the first parameter which indicates whether or not
the block ended up on the main chain.

This is primarily useful for upcoming test code that needs to be able to
tell the difference between a block accepted to a side chain and a block
that either extends the main chain or causes a reorganize that causes it
to become the main chain.  However, it is also useful for the addblock
utility since it allows a better error in the case a file with out of
order blocks is provided.
2016-10-13 16:47:50 -05:00
David Hill
42a4366ba8 addrmgr: Fix AddressCache to not include nils
allAddr was being allocated with counters instead of the actual size
of the address map.  This led to the possibility of including nils
in the returned slice, which resulted in a panic.
2016-09-26 12:16:40 -05:00
Dave Collins
754c4fbe0c
rpctest: Gate rpctest-based behind a build tag.
This adds a new build tag named rpctest which must be set in order for
rpctest-based tests to be executed.  The new build tag is also added to
the goclean.sh script which is executed by Travis during continuous
integration builds.

This change is being made because the rpctest framework requires
additional careful user configuration to ensure the version of btcd
under test can be programmatically launched from the system path with
all of the necessary ports open whereas all of the other tests are
self-contained within the test binary itself.

Since said additional configuration is typically not done, it leads to a
lot of false positives.  Putting the tests behind a build tag allows
them to remain to be available and run during continuous integration
without imposing the additional configuration requirements on users.
2016-09-26 01:20:31 -05:00
David Hill
99165eb558 rpctest: Fix typo
Use os.Getpid() to get the process ID, not os.Getppid(), which returns
the parent process ID.  This resulted in multiple calls to
generateListeningAddresses() getting the same listening ports.
2016-09-22 00:14:51 -04:00
David Hill
5ec83d23f3 Update dependencies and API usage. 2016-09-21 20:50:55 -04:00
Dave Collins
7cf9ec8190
rpctest: Use ports based on the process id.
This modifies the ports that are selected for use for the p2p and rpc
ports to start with a port that is based on the process id instead of a
hard-coded value.  The chosen ports are incremented for each running
instance similar to the previous code except the p2p and rpc ports and
now split into ranges instead of being 2 apart.

This is being done because the previous code only worked for a single
process which means it prevented the ability to run tests in parallel.

The new approach will work with multiple processes, however it must be
stated that there is still a very small probability that the stars could
align resulting in the same ports being selected.

Finally, this also reverts the recent change to run tests serially since
this fixes the underlying cause for that change.
2016-09-20 16:59:37 -05:00
Olaoluwa Osuntokun
daac24626e build: execute tests across all packages serially
This modifies the goclean.sh to execute all the tests amongst
the packages serially. The default behavior of the `go test` command is
to execute all tests in parallel amongst the listed packages. This
behavior can at times cause tests which use the `rpctest` package to
fail due to multiple `btcd` nodes attempting to bind to the same port
simultaneously. As only one node can successfully bind to the port, the
btcd processes for the other concurrent harness instances exit silently
causing the RPC clients to fail with connection timeouts as their
target process no longer exists. Executing all tests serially
eliminates such a race condition which can cause non-deterministic test
failures.
2016-09-20 01:16:08 -05:00
Dave Collins
2b780d16b0 Improve HTTP error handling.
This modifies the code which handles failed server responses to attempt
to unmarshal the response bytes as a regular JSON-RPC response
regardless of the HTTP status code and, if that fails, return an error
that includes the status code as well as the raw response bytes.

This is being done because some JSON-RPC servers, such as Bitcoin Core,
don't follow the intended HTTP status code meanings.  For example, if
you request a block that doesn't exist, it returns a status code of 500
which is supposed to mean internal server error instead of a standard
200 status code (since the HTTP request itself was successful) along
with the JSON-RPC error response.

The result is that errors from Core will now show up as an actual
RPCError instead of an error with the raw JSON bytes.

This also has the benefit of returning the HTTP status code in the
error for real HTTP failure cases such as 401 authentication failures,
which previously would just be an empty error when used against Core
since it doesn't return the actual response along with the status code
as it should.
2016-09-16 14:52:50 -05:00
Chris Martin
c20db1cf14 blockchain: make scriptval_test fail more nicely
Return early when failing on len(blocks) > 1.

Check for len(blocks) == 0 so we can fail on that condition with an
error message instead of a panic.
2016-09-16 15:44:27 -04:00
David Hill
6e5c0a7904 Fix GetBlockVerbose API.
This changes the GetBlockVersion API to not send a third parameter.
The third parameter is a boolean that expands the transaction data
structures as well.  However, Bitcore Core does not recognize this
feature.

GetBlockVerbose now only takes a hash value.

Users of the GetBlockVerbose(hash, true) must now use
GetBlockVerboseTx(hash).
2016-09-16 12:35:52 -04:00
Dave Collins
982229439a Return wire.MsgBlock from GetBlock(Async).
This modifies the return value of the GetBlock and GetBlockAsync
functions to a raw *wire.MsgBlock instead of *btcutil.Block.

This is being done because a btcutil.Block assumes there is a height
associated with the block and the block height is not part of a raw
serialized block as returned by the underlying non-verbose RPC API
getblock call and thus is not available to populate a btcutil.Block
correctly.

A raw wire.Block correctly matches what the underlying RPC call is
actually returning and thus avoids the confusion that could occur.
2016-09-15 14:06:36 -05:00
David Hill
96ed4cb5b0 Update installation section in README.md
Have users use -u along with go get to pull the latest
dependencies.
2016-09-15 14:25:23 -04:00
David Hill
8ff9623a8c Add API for getblockheader
GetBlockHeader returns the blockheader of the specified blockhash.
GetBlockHeaderVerbose returns the data structure with information about
the blockheader of the specified blockhash.

Fixes #89
2016-09-15 13:39:27 -04:00
Dave Collins
fb90c334df
rpctest: Cleanup resources on failed setup.
This modifies the rpctest harness to call the TearDown function in the
SetUp error path.  This ensures any resources, such as temp directories,
that were created during setup before whatever the failure that caused
the error are properly cleaned up.
2016-09-13 17:19:32 -05:00
Chris Martin
0ddd10add6 Remove extra backtick in README.md 2016-09-13 15:10:48 -04:00
Marco Peereboom
5e93b1664e config: Replace log outputs with fmt.Fprintln.
This corrects the case where any errors that might have happened when
creating the config file were not being displayed due to the logging
system not being initialized yet.

While here, consistently use fmt.Fprint{f,ln} throughout the loadConfig
function even after the logging system is initialized since it will
help prevent copy/paste errors such as the one that induced this change
to begin with.
2016-09-12 02:21:37 -05:00
Dave Collins
2ef82e7db3
mempool: Improve tx input standard checks.
This changes the transaction input standardness checks as follows:

- Allow any script in a pay-to-script-hash transaction to be relayed and
  mined so long as it has no more than 15 signature operations
- Remove the obsolete checks which naively calculated the number of
  expected inputs in favor of the more accurate ScriptVerifyCleanStack
  and ScriptVerifySigPushOnly functionality of the script engine that was
  added after these checks.
2016-08-24 23:16:23 -05:00
Olaoluwa Osuntokun
c6d50b7abf
rpctest: ensure the main harness rejects non-standard transactions
This commit modifies the current set of integration tests to ensure
that that the main harness always rejects non-standard transactions.

With this in place, even though we’re using simnet parameters, we
ensure that transaction acceptance/validation is identical to that of
main net.
2016-08-24 15:43:35 -07:00
Olaoluwa Osuntokun
815ded348e
config: introduce new flags to accept/reject non-std transactions
This commit adds two new cli flags: one for accepting non-std
transactions, and the other for rejecting non-std transactions.

The two flag are rejected when using concurrently. Config parsing is
set up such that, the desired policy expressed via the config always
overrides the policy set by default for a particular chain.

The doc.go files and the sample-btcd.conf file have been updated to document
the new flags exposing further policy control.
2016-08-24 15:43:26 -07:00
Olaoluwa Osuntokun
dc5486a579
mempool: add non-standard tx relaying to policy config 2016-08-24 15:43:22 -07:00
Dave Collins
15bace88dc
mempool: Add basic test harness infrastructure.
This adds a basic test harness infrastructure for the mempool package
which aims to make writing tests for it much easier.

The harness provides functionality for creating and signing transactions
as well as a fake chain that provides utxos for use in generating valid
transactions and allows an arbitrary chain height to be set.  In order
to simplify transaction creation, a single signing key and payment
address is used throughout and a convenience type for spendable outputs
is provided.

The harness is initialized with a spendable coinbase output by default
and the fake chain height set to the maturity height needed to ensure
the provided output is in fact spendable as well as a policy that is
suitable for testing.

Since tests are in the same package and each harness provides a unique
pool and fake chain instance, the tests can safely reach into the pool
policy, or any other state, and change it for a given harness without
affecting the others.

In order to be able to make use of the existing blockchain.Viewpoint
type, a Clone method has been to the UtxoEntry type which allows the
fake chain instance to keep a single view with the actual available
unspent utxos while the mempool ends up fetching a subset of the view
with the specifically requested entries cloned.

To demo the harness, this also contains a couple of tests which make use
of it:

- TestSimpleOrphanChain -- Ensures an entire chain of orphans is
  properly accepted and connects up when the missing parent transaction
  is added
- TestOrphanRejects -- Ensure orphans are actually rejected when the
  flag on ProcessTransactions is set to reject them
2016-08-24 10:04:40 -05:00
David Hill
a109bea3f1 mempool: unexport the mutex
callers should not need to lock/unlock the mempool themselves.
2016-08-23 14:59:48 -04:00
Dave Collins
641182b2ad
mempool: Break dependency on chain instance.
This modifies the config for the new mempool package such that it takes
a callback function to obtain the best chain height instead of requiring
a fully initialized blockchain.BlockChain instance.

This will make it much easier to test the mempool since the tests will
be able to provide their own height function to test various
functionality without having create and manipulate full blocks and chain
instances.
2016-08-23 12:29:45 -05:00
Dave Collins
c57c18c8e2
blockchain: Add median time to state snapshot.
This adds a new field to the best chain state snapshot for the
calculated past median time as returned by the calcPastMedianTime
function.  This is useful since it provides fast access to it without
having to acquire the chain lock which is needed to recalculate it.

This will ultimately allow the associated exported function to be
removed since it only exists to be able to calculate this exact value,
however this commit only introduces the new field in order to keep the
changes minimal.
2016-08-23 01:29:11 -05:00
Olaoluwa Osuntokun
bfe2ba4191
rpctest: prevent leaking processes in tests due to panics
This commit adds a `defer` statement at the top of  `TestRpcServer`
which will attempt a `recover` which tears down all active harnesses in
the event that one of the tests causes a panic in the main goroutine.

Before this commit, if a buggy test caused a panic while all integration
tests were being executed, then any active harnesses would fail to be
properly torn down. This would cause the running btcd processes to be
leaked, possibly interfering with future test runs until the process was
manually killed. This commit fixes such behavior.

In order to aide in debugging, when a test panics, the test number is
printed out along with a full stack-trace from the start of the test to
the panic point.
2016-08-22 17:09:18 -07:00
Dave Collins
6cf60b5fae
rpctest: Correct several issues in tests and joins.
This corrects several issues with the new rpctest package.

The following is an overview of the issues that have been corrected:

- The JoinNodes code was incorrect for both mempool and blocks.
  - For mempool it was only checking the first node against itself
  - For blocks it was only testing the height instead of hash and height
    which means the chains could be different and it would claim they're
    synced
- The test for mempool joins was inaccurate in a few ways:
  - It was generating separate chains such that the generated
    transaction was invalid (an orphan) on one chain, but not
    the other
  - Mempools are not automatically synced when nodes are connected and
    there is a large random delay before any transaction rebroadcast
    happens, so it can't be relied on for the purposes of this test
- The test for block joins was generating two independent chains of the
  same height with the same difficulty and was only passing due to the
  aforementioned bug in JoinNodes
- All of the ConnectNode calls were connecting the main harness outbound
  to the local test harness instances
  - This is not correct because ConnectNode makes the outbound
    connection persistent, which means once the local test harness is
    gone, it would keep trying to connect for the remainder of the tests
    to a node that is never coming back and instead ends up connecting to
    an independent test harness.
2016-08-21 15:10:23 -05:00
jadeblaquiere
47ced81d44 don't spin forever looking for peers (#724) 2016-08-20 22:11:12 -05:00
Olaoluwa Osuntokun
7c4b169faa
build: install btcd binary within Travis container
This commit modifies the existing Travis configuration to also install
the btcd binary within the container’s PATH.

This change is necessary in order for tests written against the new
rpctest package to work properly within the Travis testing environment.
2016-08-19 17:43:42 -05:00
Olaoluwa Osuntokun
c3d5371615
build: update dependancies required by rpctest 2016-08-19 17:42:25 -05:00
Olaoluwa Osuntokun
2d86fbface
btcd: add basic RPC tests using the rpctest package
This commit introduces a new file: rpcserver_test.go dedicated for
including integration tests for btcd using the new rpctest package.

The tests are created using a TestMain instance first creates a single
main harness which is intended to be re-used across tests instances.
Afterwards all registered RPC tests are executed, with proper clean up
being executed regardless of the passing state of the tests.

The following RPC calls are excessed by the initial set of tests added:
    * getbestblock
    * getblockcount
    * getblockhash
2016-08-19 17:41:37 -05:00
Olaoluwa Osuntokun
b86df0ba91
rpctest: create new rpctest package
This commit adds a new package (rpctest) which provides functionality
for writing automated black box tests to exercise the RPC interface.

An instance of a rpctest consists of an active btcd process running in
(typically) --simnet mode, a btcrpcclient instance connected to said
node, and finally an embedded in-memory wallet instance (the memWallet)
which manages any created coinbase outputs created by the mining btcd
node.

As part of the SetUp process for an RPC test, a test author can
optionally opt to have a test blockchain created. The second argument
to SetUp dictates the number of mature coinbase outputs desired. The
btcd process will then be directed to generate a test chain of length:
100 + numMatureOutputs.

The embedded memWallet instance acts as a minimal, simple wallet for
each Harness instance. The memWallet itself is a BIP 32 HD wallet
capable of creating new addresses, creating fully signed transactions,
creating+broadcasting a transaction paying to an arbitrary set of
outputs, and querying the currently confirmed balance.

In order to test various scenarios of blocks containing arbitrary
transactions, one can use the Generate rpc call via the exposed
btcrpcclient connected to the active btcd node. Additionally, the
Harness also exposes a secondary block generation API allowing callers
to create blocks with a set of hand-selected transactions, and an
arbitrary BlockVersion or Timestamp.

After execution of test logic TearDown should be called, allowing the
test instance to clean up created temporary directories, and shut down
the running processes.

Running multiple concurrent rpctest.Harness instances is supported in
order to allow for test authors to exercise complex scenarios. As a
result, the primary interface to create, and initialize an
rpctest.Harness instance is concurrent safe, with shared package level
private global variables protected by a sync.Mutex.

Fixes #116.
2016-08-19 17:37:08 -05:00
Dave Collins
763f731c5c wire: Lower MaxUserAgentLen to 256. (#744)
The protocol was silently upgrade in Core some time ago to enforce a
limit of 256 for the user agent in version messages.  This updates wire
to coincide with that change and consequently includes a wire version
bump to 0.4.1.
2016-08-19 12:40:24 -05:00
Dave Collins
7fac099bee mempool: Refactor mempool code to its own package. (#737)
This does the minimum work necessary to refactor the mempool code into
its own package.  The idea is that separating this code into its own
package will greatly improve its testability, allow independent
benchmarking and profiling, and open up some interesting opportunities
for future development related to the memory pool.

There are likely some areas related to policy that could be further
refactored, however it is better to do that in future commits in order
to keep the changeset as small as possible during this refactor.

Overview of the major changes:

- Create the new package
- Move several files into the new package:
  - mempool.go -> mempool/mempool.go
  - mempoolerror.go -> mempool/error.go
  - policy.go -> mempool/policy.go
  - policy_test.go -> mempool/policy_test.go
- Update mempool logging to use the new mempool package logger
- Rename mempoolPolicy to Policy (so it's now mempool.Policy)
- Rename mempoolConfig to Config (so it's now mempool.Config)
- Rename mempoolTxDesc to TxDesc (so it's now mempool.TxDesc)
- Rename txMemPool to TxPool (so it's now mempool.TxPool)
- Move defaultBlockPrioritySize to the new package and export it
- Export DefaultMinRelayTxFee from the mempool package
- Export the CalcPriority function from the mempool package
- Introduce a new RawMempoolVerbose function on the TxPool and update
  the RPC server to use it
- Update all references to the mempool to use the package.
- Add a skeleton README.md
2016-08-19 11:08:37 -05:00
Dave Collins
87b3756c8c server: Remove superfluous check in OnMemPool. (#736)
This reduces the mempool lock contention by removing an unnecessary
check when responding to a "mempool" request.

In particular, the code first gets a list of all transactions from the
mempool and then iterates them in order to construct the inventory
vectors and apply bloom filtering if it is enabled.  Since it is
possible that the transaction was removed from the mempool by another
thread while that list is being iterated, the code was checking if each
transaction was still in the mempool.  This is a pointless check because
the transaction might still be removed at any point after the check
anyways.  For example, it might be removed after the mempool response
has been sent to the remote peer or even while the loop is still
iterating.
2016-08-19 11:04:16 -05:00
David Hill
05ab7141d3 travis: Add go 1.7 and drop go 1.5 support. (#740) 2016-08-18 11:06:26 -05:00
Dave Collins
b89c91b9d6 Refactor ntfnstate mutex. (#85)
This refactors the notification state mutex out of the state itself to
the client.  This is being done since the code makes a copy of the
notification state and accesses that copy immutably, and therefore there
is no need for it to have its own mutex.
2016-08-17 13:12:27 -05:00
David Hill
8c8d4453ad travis: Add golang 1.7 and drop golang 1.5 support. (#83) 2016-08-17 13:09:36 -05:00
Dave Collins
2de61b2d9a Manually copy relevant fields in ntfn state copy. (#84)
This makes vet happy by manually copying the notification state fields
during the deep copy instead of copying the entire struct which contains
an embedded mutex.
2016-08-17 12:39:34 -05:00
Janus Troelsen
4a5223266c docs: Add chainhash to README.md (#739) 2016-08-17 10:41:50 -05:00
Dave Collins
cee207c64c txscript: Expose AddOps on ScriptBuilder. (#734)
This exposes a new function on the ScriptBuilder type named AddOps that
allows multiple opcodes to be added via a single call and adds tests to
exercise the new function.

Finally, it updates a couple of places in the signing code that were
abusing the interface by setting its private script directly to use the
new public function instead.
2016-08-12 19:29:28 -05:00
Waldir Pimenta
f584dbd2e5 add license title (#82)
background: https://github.com/btcsuite/btcd/pull/717
2016-08-11 20:44:30 -05:00
Waldir Pimenta
fb9b640ef2 add license title (#717)
It's not strictly required, but it's useful metadata, and part of the recommended license template text (see http://choosealicense.com/licenses/isc/ and https://opensource.org/licenses/isc-license)
2016-08-11 16:58:15 -05:00
Dave Collins
044a11c9fc btcd: Simplify shutdown signal handling logic. (#733)
This rewrites the shutdown logic to simplify the shutdown signalling.
All cleanup is now run from deferred functions in the main function and
channels are used to signal shutdown either from OS signals or from
other subsystems such as the RPC server and windows service controller.

The RPC server has been modified to use a new channel for signalling
shutdown that is exposed via the RequestedProcessShutdown function
instead of directly calling Stop on the server as it previously did.

Finally, it adds a few checks for early termination during the main
start sequence so the process can be stopped without starting all the
subsystems if desired.

This is a backport of the equivalent logic from Decred with a few slight
modifications.  Credits go to @jrick.
2016-08-11 13:39:23 -05:00
Dave Collins
a7b35d9f9e chaincfg/blockchain: Parameterize more chain consts. (#732)
This moves several of the chain constants to the Params struct in the
chaincfg package which is intended for that purpose.  This is mostly a
backport of the same modifications made in Decred along with a few
additional things cleaned up.

The following is an overview of the changes:

- Comment all fields in the Params struct definition
- Add locals to BlockChain instance for the calculated values based on
  the provided chain params
- Rename the following param fields:
  - SubsidyHalvingInterval -> SubsidyReductionInterval
  - ResetMinDifficulty -> ReduceMinDifficulty
- Add new Param fields:
  - CoinbaseMaturity
  - TargetTimePerBlock
  - TargetTimespan
  - BlocksPerRetarget
  - RetargetAdjustmentFactor
  - MinDiffReductionTime
2016-08-10 16:02:23 -05:00
Dave Collins
c39e35318d Update for recent chainhash-related API changes. (#81)
This updates all code to make use of the new chainhash package since the
old wire.ShaHash type and related functions have been removed in favor
of the abstracted package.

Also, while here, rename all variables that included sha in their name
to include hash instead.
2016-08-08 14:16:04 -05:00
Dave Collins
bd4e64d1d4 chainhash: Abstract hash logic to new package. (#729)
This is mostly a backport of some of the same modifications made in
Decred along with a few additional things cleaned up.  In particular,
this updates the code to make use of the new chainhash package.

Also, since this required API changes anyways and the hash algorithm is
no longer tied specifically to SHA, all other functions throughout the
code base which had "Sha" in their name have been changed to Hash so
they are not incorrectly implying the hash algorithm.

The following is an overview of the changes:

- Remove the wire.ShaHash type
- Update all references to wire.ShaHash to the new chainhash.Hash type
- Rename the following functions and update all references:
  - wire.BlockHeader.BlockSha -> BlockHash
  - wire.MsgBlock.BlockSha -> BlockHash
  - wire.MsgBlock.TxShas -> TxHashes
  - wire.MsgTx.TxSha -> TxHash
  - blockchain.ShaHashToBig -> HashToBig
  - peer.ShaFunc -> peer.HashFunc
- Rename all variables that included sha in their name to include hash
  instead
- Update for function name changes in other dependent packages such as
  btcutil
- Update copyright dates on all modified files
- Update glide.lock file to use the required version of btcutil
2016-08-08 14:04:33 -05:00
Dave Collins
b6b1e55d1e TravisCI: Set vendor experiment variable earlier. (#731)
This sets the GO15VENDOREXPERIMENT environment variable before glide is
installed so vendoring is used when installing it as well.
2016-08-08 13:31:17 -05:00
Dave Collins
711f33450c chainhash: Implement a new chainhash package. (#730)
This is a backport of the chainhash package made in Decred along with a
few additional things cleaned up, finished test coverage, and rewording
of some documentation to make it more generic.

In particular, the new package provides the definition of the hash type
and associated hashing functions which will allow the rest of the code to be
agnostic to the specific hash algorithm.

This only implements the package and does not change any of the code
base over to use it.
2016-08-08 12:05:51 -05:00
Dave Collins
d406d9e52b wire: Consolidate tests into the wire pkg. (#728)
Putting the test code in the same package makes it easier for forks
since they don't have to change the import paths as much and it also
gets rid of the need for internal_test.go to bridge.

This same thing should probably be done for the majority of the code
base.
2016-08-08 11:42:54 -05:00