Commit graph

120 commits

Author SHA1 Message Date
Jack Robison
9920ff59d4
force KBucket refreshes 2018-06-07 15:02:47 -04:00
Jack Robison
aee7a3aa38
simplify announceHaveBlob, remove unused getPeersForBlob 2018-06-07 15:02:47 -04:00
Jack Robison
372fb45e06
refactor dht bootstrap
after finding the closest nodes try to populate the buckets out by looking up random ids in their key ranges
2018-06-07 15:02:47 -04:00
Jack Robison
95ed1e044b
raise TransportNotConnected instead of logging a warning
-add a _listening Deferred to KademliaProtocol which is called when the protocol is started
2018-06-07 15:02:47 -04:00
Jack Robison
f1e0a784d9
refactor iterativeFind, move to own file 2018-06-07 15:02:47 -04:00
Jack Robison
23c202b5e4
refactor Contact class, DHT RPCs, and Contact addition/removal
-track contact failures, last replied, and last requested. use this to provide a 'contact_is_good' property on Contact objects
-ensure no duplicate contact objects are created
-remove confusing conflation of node id strings with Contact objects, update docstrings
-move RPC failure tracking to a callback/errback pair in sendRPC (so the contact is only updated once)
-handle seed nodes during the join sequence by setting their node ids after they initially reply to our ping
-name all of the kademlia RPC keyword args, remove confusing **kwargs and dictionary parsing
-add host ip/port to DHT send/receive logging to make the results comprehensible when running many nodes at once
2018-06-07 15:02:47 -04:00
Jack Robison
406ddaa4ef
use base class to simplify Node init
-add looping call helpers which use the same clock as the Node
2018-06-07 15:02:47 -04:00
Jack Robison
e52689a33d
remove OptimizedTreeRoutingTable for now, use TreeRoutingTable 2018-06-07 15:02:47 -04:00
Jack Robison
d65dc0aec3
disable loading DictDataStore in Node.__init__
-to be re-done when the datastore uses sqlite
2018-06-07 15:02:47 -04:00
Jack Robison
4bd9f3bd68
remove popular hash tracking, simplify DHTPeerFinder 2018-06-07 15:02:47 -04:00
Jack Robison
e6ffd7caf8
remove hashwatcher 2018-06-07 15:02:47 -04:00
Jack Robison
b89823c968
fix blob announce locking up the daemon process 2018-04-19 15:47:19 -04:00
Jack Robison
5cea031f38
return list of dictionaries from peer_list, include peer node ids 2018-04-03 13:10:30 -04:00
Jack Robison
5bab6f7d39
remove bandwidth stats 2018-03-28 18:50:24 -04:00
Jack Robison
a41bbd5e27
pylint and tests 2018-03-28 18:50:18 -04:00
Jack Robison
c5bf64cf0a
refactor DHTHashAnnouncer
-remove hash_announcer from Node and DiskBlobManager
-remove announcement related functions from DiskBlobManager
-update SQLiteStorage to store announcement times and provide blob hashes needing to be announced
-use dataExpireTimeout from lbrynet.dht.constants for re-announce timing
-use DeferredSemaphore for concurrent blob announcement
2018-03-28 15:53:51 -04:00
Jack Robison
ea0ea704a2
refactor iterativeAnnounceHaveBlob
-change to only self_store if the number of contacts to store to is less than k and we are the closest node to the hash
2018-03-28 15:53:51 -04:00
Jack Robison
a8025b02c6
log invalid vs missing token 2018-03-28 15:53:51 -04:00
Jack Robison
4eab77fa10
safe start_listening 2018-03-28 15:53:50 -04:00
Jack Robison
5013426e02
logging and docstring 2018-03-28 15:53:50 -04:00
Jack Robison
cb09be5336
remove unused stuff 2018-03-28 15:53:50 -04:00
Jack Robison
5628d0825b
add CallLaterManager 2018-03-28 15:53:49 -04:00
Jack Robison
88970cb0a8
move peer manager class to lbrynet.core 2018-03-28 15:53:49 -04:00
Jack Robison
df78f7ff9f
add response assertion to announce_to_peer 2018-03-28 15:53:48 -04:00
Jack Robison
bdba263224
catch TimeoutError in _IterativeFindHelper 2018-03-28 15:53:48 -04:00
Jack Robison
16fcc3f5c1
findValue inlinecallbacks refactor 2018-03-28 15:53:48 -04:00
Jack Robison
43896c8d17
refactor joinNetwork into smaller functions
-try to re-join network if no contacts are known
2018-03-28 15:53:47 -04:00
Jack Robison
e6caedac91
remove DHTHashSupplier class, move former functions into DHTHashAnnouncer 2018-03-28 15:53:47 -04:00
Jack Robison
6666468640
add reactor arguments to Node
-adds reactor (clock) and reactor functions listenUDP, callLater, and resolve as arguments to Node.__init__
-set the reactor clock on LoopingCalls to make them easily testable
-convert callLater manage loops to LoopingCalls
2018-03-28 15:53:47 -04:00
Jack Robison
efaa97216f
move dht node setup back into node class 2018-03-28 15:53:47 -04:00
Jack Robison
3296c0fb3d
move dht related classes to lbrynet.dht 2018-03-28 15:53:47 -04:00
Jack Robison
446c3a88dc
refactor DHTHashAnnouncer and iterativeAnnounceHaveBlob
-use looping call for running manage function rather than a scheduled
callLater

-track announce speed

-retry store requests that failed up to 3 times

-return a dict of {blob_hash: [storing_node_id]} results from
_announce_hashes

_refreshRoutingTable inline cb refactor

-add and use DeferredLockContextManager

-don't trap errback from iterativeFindNode in iterativeAnnounceHaveBlob
2018-03-28 15:53:45 -04:00
Jack Robison
67ef8be7b7
convert node manage function to a looping call 2018-03-28 15:53:45 -04:00
Jack Robison
ecbe4113ce
move Distance to own file 2018-03-28 15:53:45 -04:00
Antonio Quartulli
4f7885e499
dht_node: add hasContacts method
This method can be used by other components to check
if in the Node routing table there is at least one peer.

Signed-off-by: Antonio Quartulli <antonio@mandelbit.com>
2018-02-06 12:33:10 +08:00
Antonio Quartulli
ce0af77aa9
dht_node: split network start and join procedures
In order to attempt to join the DHT several times
(i.e. when the first attempt has failed) we need to
split the components initialization from the real
joining operation.

Create node.startNetwork() to initialize the node
and keep the rest in node.joinNetwork()

Signed-off-by: Antonio Quartulli <antonio@mandelbit.com>
2018-02-06 12:19:33 +08:00
Kay Kurokawa
b380f5d344 completely remove unused from_peer field 2017-11-07 09:56:40 -05:00
Kay Kurokawa
9776655c3c remove unneeded if statement in getPeersForBlob 2017-11-07 09:56:40 -05:00
Kay Kurokawa
7e8f3254b1 Add to DHT Node class initialization argument peerPort where it serves blobs, instead of specifying it in announceHaveBlob 2017-11-07 09:56:40 -05:00
Antonio Quartulli
965969b856 Don't filter out local node ID when returning peer list
If a node is returning a peer list for a given blob hash
(being this been requested via CLI or via DHT) and it is
part of the resulting peer list, it will filter itself out
before returning the list.

This makes the results across the DHT inconsistent as
different nodes won't include themselves when
responding a findValue/findNode query.

Remove such filtering so that the local node ID is always
included when needed.

Signed-off-by: Antonio Quartulli <antonio@mandelbit.com>
2017-11-07 09:56:40 -05:00
Jack Robison
0b771c16ba
fix lbry id for dead contact to replace in _IterativeFindHelper 2017-10-24 19:13:45 -04:00
Jack Robison
d23d4c1d73
fix lbryid length comparison 2017-10-23 01:23:04 -04:00
Jack Robison
e9fd8eb096
update tests and scripts 2017-10-10 13:55:31 -04:00
Jack Robison
9919fd06c6
uncomment error for store request where contact is not known 2017-10-10 13:29:54 -04:00
Jack Robison
c9515f8fb6
logging and whitespace 2017-10-10 13:29:29 -04:00
Jack Robison
5937ead17c
add Node.contacts helper property 2017-10-10 13:28:57 -04:00
Jack Robison
ab956d4a8e
use count parameter in findCloseNodes 2017-10-10 13:20:19 -04:00
Jack Robison
fcaca05a81
fix redundant lbryid 2017-10-10 13:15:25 -04:00
Jack Robison
46e31d5b45
getPeersForBlob inlinecallbacks refactor 2017-10-10 13:12:47 -04:00
Jack Robison
8e9f3c90a1
use looping call for Node.change_token to avoid hanging delayedCalls 2017-10-10 13:10:47 -04:00
Jack Robison
02475ff985 add dht bandwidth tracking 2017-05-30 11:41:54 -04:00
Alex Grintsvayg
cdf67de46c pylint type checking 2017-04-25 14:36:14 -04:00
Alex Grintsvayg
fd51d8ddf0 pylint 2017-04-25 14:36:14 -04:00
Alex Grintsvayg
8db7c37fa7 exploring dht 2017-04-10 13:45:43 -04:00
Alex Grintsvayg
850f51140e formatting 2017-03-31 13:32:43 -04:00
Job Evers-Meltzer
4f3b5cd802 Better logging on DHT errors
Timeout errors are common on the dht so log those
at debug, but other errors need to (potentially) recieve
more attention
2016-12-15 18:58:16 -06:00
Job Evers-Meltzer
323bccb0ae another distance optimization 2016-12-15 18:58:15 -06:00
Job Evers-Meltzer
0084d4684f Add distance optimization 2016-12-15 18:58:15 -06:00
Job Evers-Meltzer
740fad5cbe Refactor dht.node.iterativeFind
Move nested functions into a helper class.
Add new, smaller functions to increase readability
2016-12-15 18:58:15 -06:00
Job Evers-Meltzer
c30ea04959 Remove commented code
This is the result of running eradicate (https://github.com/myint/eradicate)
on the code and double-checking the changes.
2016-12-15 18:58:06 -06:00
Job Evers-Meltzer
14ac2bac39 pylint: fix trailing-whitespace 2016-12-14 19:21:59 -06:00
Job Evers-Meltzer
b64fa51567 fixup long lines 2016-12-07 09:38:34 -05:00
Job Evers-Meltzer
61a354fa60 add bad-whitespace check to pylint 2016-11-09 11:27:07 -06:00
Jack
6b4519e959 squelch very verbose dht error log 2016-08-12 10:46:05 -05:00
Job Evers-Meltzer
fccdf32c9f Remove all unused imports; add check to pylint 2016-08-08 15:42:35 -05:00
Job Evers
a2eab1577d Have TimeoutError hexlify blobs 2016-07-26 13:54:46 -05:00
Jack
49ae029572 Don't try to process blank IPs 2016-01-26 16:07:33 -05:00
Jimmy Kiselak
7df6e99e28 fix typo and formatting errors, stop console's shutdown from running twice on exit, give more informative message if ports can't be bound, only set upnp if there isn't already a redirect on that port/protocol, and only unset upnp if the redirect was set by the program 2015-10-19 15:05:17 -04:00
Jimmy Kiselak
bb5cd49327 Use named loggers, fix some pep8 formatting mistakes 2015-09-08 15:42:56 -04:00
Jimmy Kiselak
7240ff6b1c initial commit 2015-08-20 11:27:15 -04:00