Previously a new goroutine was launched for each notification in order to
avoid blocking chain from continuing while the notification is being
processed. This approach had a couple of issues.
First, since goroutines are not guaranteed to execute in any given order,
the notifications were no longer handled in the same order as they were
sent. For the current code, this is not a problem, but upcoming code that
handles a transaction memory pool, the order needs to be correct.
Second, goroutines are relatively cheap, but it's still quite a bit of
overhead to launch 3-4 goroutines per block.
This commit modifies the handling code to have a single sink executing in
a separate goroutine. The main handler then adds the notifications to a
queue which is processed by the sink. This approach retains the
non-blocking behavior of the previous approach, but also keeps the order
correct and, as an additional benefit, is also more efficient.
This removes a horrible case of reach-around from per into the guts of
the blockmaanger to frob the chain. Soon, when we try to deduplicate the
fetching of blocks from multiple peers this will need decisions made in
a central point.
Discussed at length with davec.
This commit modifies the code to choose sane defaults for the backing
arrays for slices that involve a lot of appends such block locators, hash
processing, and needed transactions. This is an optimization to avoid
the overhead of growing the backing arrays and copying the data multiple
times in the most common case. This also prevents a leak in Go GC which
will likely ultimatley be fixed, but the efficiecy gains alone are worth
the change.
Removing from the bottom of a stack (nipN(depth)) would leak the first
entry in the array by slicing [1:], leaving array[0] dangling an
inaccessible (but unable to be freed until the whole slice is gone). We
left it like this for a while, but best not to leak the memory. Happens
rarely so the performance hit shouldn't matter that much. Do the same
thing for condstack.
Several of the messages store the parts that have a variable number of
elements as slices. This commit modifies the code to choose sane defaults
for the backing arrays for the slices so when the entries are actually
appended, a lot of the overhead of growing the backing arrays and copying
the data multiple times is avoided.
Along the same lines, when decoding messages, the actual size is known and
now is pre-allocated instead of dynamically growing the backing array
thereby avoiding some overhead.
When we do append loops, make an educated guess as to the size and make an array
with that capacity to avoid extra copying.
Doesn't affect the speed of the tests, over 4 runs the difference was lost in
the noise.
After discussion with others and thinking about the notification channel
some more, we've decided to leave it up to the caller to quickly handle
notifications. While it is true that notification should be handled
quickly to not block the chain processing code unnecessarily, launching a
goroutine in chain means the notifications are no longer necessarily in
order. Also, if the caller is not properly handling the notifications,
the goroutines end up sicking around forever. By leaving it up to the
caller to quickly handle the notification or launch a goroutine as
necessary for the caller, it provides the flexibility to ensure proper
notification ordering as well as control over other things such as
how to handle backpressure.
This commit adds detection and filtering for back-to-back duplicate
getblocks requests. This is needed because the trigger for requesting
more blocks is receiving an orphan. When the peer is further behind than
the number of blocks advertised via a single inventory message, the same
orphan block will be sent multiple times. When the peer receives the
final inventory message, it too contains the orphan that was previously
sent. This leads to a duplicate getblocks request that must be filtered
to prevent requesting the final series of blocks again.
This commit adds environment variable expansion and path cleaning to the
data directory. This allows the user to specify data paths in the config
file such as datadir=~/.btcd/data and datadir=$SOMEVAR/btcd. It also
adds usage instructions and an example to the sample btcd.conf file.
- Remove leftover debug log prints
- Increment waitgroup outside of goroutine
- Various comment and log message consistency
- Combine peer setup and newPeer -> newInboundPeer
- Save and load peers.json to/from cfg.DataDir
- Only claim addrmgr needs more addresses when it has less than 1000
- Add warning if unkown peer on orphan block.
Use it to add multiple peer support. We try and keep 8 outbound peers
active at all times.
This address manager is not as complete as the one in bitcoind yet, but
additional functionality is being worked on.
We currently handle (in a similar manner to bitcoind):
- biasing between new and already tried addresses based on number of connected
peers.
- rejection of non-default ports until desparate
- address selection probabilities based on last successful connection and number
of failures.
- routability checks based on known unroutable subnets.
- only connecting to each network `group' once at any one time.
We currently lack support for:
- tor ``addresses'' (an .onion address encoded in 64 bytes of ip address)
- full state save and restore (we just save a json with the list of known
addresses in it)
- multiple buckets for new and tried addresses selected by a hash of address and
source. The current algorithm functions the same as bitcoind would with only
one bucket for new and tried (making the address cache rather smaller than it
otherwise would be).
The regression test mode is special in that the 'official' block test
suite requires an empty database to work properly. Rather than having to
manual go delete it before each test, add code to automatically delete the
old regression test database when in regression test mode.
This commit modifies the way the data paths are handled. Since there will
ultimately be more data associated with each network than just the block
database, the data path has been modified to be "namespaced" based on the
network. This allows all data associated with a specific network to
simply use the data path without having to worry about conflicts with data
from other networks.
In addition, this commit renames the block database to "blocks" plus a
suffix which denotes the database type. This prevents issues that would
otherwise arise if the user decides to use a different database type and
a file/folder with the same name already eixsts but is of the old database
type. For most users this won't matter, but it does provide nice
properties for testing and development as well since it makes it easy to
go back and forth between database types.
This commit also includes code to upgrade the old database paths to the
new ones so the change is seamless for the user.
Finally, bump the version to 0.2.0.
This commit adds a basic infrastructure to allow upgrades to happen to
btcd as needed. This paves the way for the upcoming data path changes to
be automatically updated for the user as needed and also ensures any
future changes that might require upgrades already have an established
way of performing the needed upgrades.
This change paves the way for saving more than just the block database to
the filesystem (such as address manager data, index data, etc) where the
name "dbdir" no longer makes sense.
This commit changes the code so that all calls to .Add on waitgroups
happen before the associated goroutines are launched. Doing this after
the goroutine could technically cause a race where the goroutine started
and finished before the main goroutine has a chance to increment the
counter. In our particular case none of the goroutines exit quickly
enough for this to be an issue, but nevertheless the correct way should be
used.
