Madiator2011/lbry-sdk
1
0
Fork 0
forked from LBRYCommunity/lbry-sdk
Code Pull requests Activity
4254 commits 111 branches 641 tags 30 MiB
Commit graph

211 commits

Author SHA1 Message Date
Jack Robison ae631f05c3
fix teardown error 2018-06-07 15:02:47 -04:00
Jack Robison bc0da5e2d1
only use seeds in iterative bootstrap if no contacts are known yet 2018-06-07 15:02:47 -04:00
Jack Robison 44644673d7
add profiler 2018-06-07 15:02:47 -04:00
Jack Robison e8b402f998
remove deferredLock from iterativeFind 
-fire the first iteration right away
 2018-06-07 15:02:47 -04:00
Jack Robison 4fbaaac3f3
default new contacts to protocol version 0 2018-06-07 15:02:47 -04:00
Jack Robison db06191c33
reduce default concurrent announcers to 10 
-lower rpc timeout to what it originally was
 2018-06-07 15:02:47 -04:00
Jack Robison b0a741b1f4
fix hash announcer semaphore 2018-06-07 15:02:47 -04:00
Jack Robison 537df6c8ad
log socket errors 2018-06-07 15:02:47 -04:00
Jack Robison 42eb172638
refactor announceHaveBlob 
-add cached `token` to Contact objects to minimize findValue requests
-remove self_store, always store to remote contacts even if we're the closest known node to the hash
-move the store call and error handling from announceHaveBlob to a smaller function of its own
 2018-06-07 15:02:47 -04:00
Jack Robison 9a63db4ec6
add protocol version to the dht and migrate old arg format for store 2018-06-07 15:02:47 -04:00
Jack Robison 7d21cc5822
pylint and more tests 2018-06-07 15:02:47 -04:00
Jack Robison cce3c8c7b5
increase kademlia rpc timeout to 8 seconds 2018-06-07 15:02:47 -04:00
Jack Robison 0e80123615
use 12 minutes instead of 15 as delay in contact_is_good 2018-06-07 15:02:47 -04:00
Jack Robison 659632b66c
fix and update tests 2018-06-07 15:02:47 -04:00
Jack Robison 9582b7fcf5
use maybe_ping for refreshing stale buckets and storing peers 
-move store refresh to its own looping call
 2018-06-07 15:02:47 -04:00
Jack Robison 945da5985e
fix age used in datastore to determine if a value is expired 2018-06-07 15:02:47 -04:00
Jack Robison 921ee3c4c1
use refreshTimeout in getRefreshList 2018-06-07 15:02:47 -04:00
Jack Robison 8efede6ad6
maybe_ping bad and unknown contacts instead of only unknown 2018-06-07 15:02:47 -04:00
Jack Robison fb3aac15fa
add optional delay argument to enqueue_maybe_ping, fix default value 2018-06-07 15:02:47 -04:00
Jack Robison 73e813f9ec
verify key size in Distance 2018-06-07 15:02:47 -04:00
Jack Robison 0d23c68797
raise attribute error for non-rpc functions in Contact 2018-06-07 15:02:47 -04:00
Jack Robison ec1b6b2387
comments, cleaner key_bits constant 2018-06-07 15:02:47 -04:00
Jack Robison 29d5750371
pylint 2018-06-07 15:02:47 -04:00
Jack Robison 07f92014d7
omit bad contacts from getPeersForBlob 
-refresh stored contacts to detect when they go offline or come back
 2018-06-07 15:02:47 -04:00
Jack Robison c521120b17
update and fix hash announcer test 2018-06-07 15:02:47 -04:00
Jack Robison d02ed29e50
add kademlia store and expiration test 2018-06-07 15:02:47 -04:00
Jack Robison 877da78505
ping contacts right away during refresh instead of using PingQueue 
-fixes contact expiration test
 2018-06-07 15:02:47 -04:00
Jack Robison a952d2d2c8
reset _listeningPort and _listening Deferred on teardown 2018-06-07 15:02:47 -04:00
Jack Robison 3dfc6bd2cc
update CallLaterManager to be an object 2018-06-07 15:02:47 -04:00
Jack Robison 760417ff3a
pylint 2018-06-07 15:02:47 -04:00
Jack Robison 4f72098cad
use PingQueue to try refresh all contacts 2018-06-07 15:02:47 -04:00
Jack Robison 9920ff59d4
force KBucket refreshes 2018-06-07 15:02:47 -04:00
Jack Robison aee7a3aa38
simplify announceHaveBlob, remove unused getPeersForBlob 2018-06-07 15:02:47 -04:00
Jack Robison 372fb45e06
refactor dht bootstrap 
after finding the closest nodes try to populate the buckets out by looking up random ids in their key ranges
 2018-06-07 15:02:47 -04:00
Jack Robison c65274e9e5
add PingQueue to KademliaProtocol 2018-06-07 15:02:47 -04:00
Jack Robison 1adf4f7818
fix constant used to check if a bucket is fresh 2018-06-07 15:02:47 -04:00
Jack Robison 05241012a2
update contact replacement in KBucket to follow BEP0005 
http://www.bittorrent.org/beps/bep_0005.html
 2018-06-07 15:02:47 -04:00
Jack Robison cf3359044d
fix conditions for when a kbucket should be split 
https://stackoverflow.com/a/32187456
 2018-06-07 15:02:47 -04:00
Jack Robison 5631a24881
improve findCloseNodes, choose closest contacts from higher and lower buckets 2018-06-07 15:02:47 -04:00
Jack Robison c654bfe296
use reactor clock in TreeRoutingTable instead of time module 2018-06-07 15:02:47 -04:00
Jack Robison d4e28216a0
sort KBucket.getContacts 2018-06-07 15:02:47 -04:00
Jack Robison 95ed1e044b
raise TransportNotConnected instead of logging a warning 
-add a _listening Deferred to KademliaProtocol which is called when the protocol is started
 2018-06-07 15:02:47 -04:00
Jack Robison f1e0a784d9
refactor iterativeFind, move to own file 2018-06-07 15:02:47 -04:00
Jack Robison e5703833cf
prevent duplicate entries in the datastore 2018-06-07 15:02:47 -04:00
Jack Robison 23c202b5e4
refactor Contact class, DHT RPCs, and Contact addition/removal 
-track contact failures, last replied, and last requested. use this to provide a 'contact_is_good' property on Contact objects
-ensure no duplicate contact objects are created
-remove confusing conflation of node id strings with Contact objects, update docstrings
-move RPC failure tracking to a callback/errback pair in sendRPC (so the contact is only updated once)
-handle seed nodes during the join sequence by setting their node ids after they initially reply to our ping
-name all of the kademlia RPC keyword args, remove confusing **kwargs and dictionary parsing
-add host ip/port to DHT send/receive logging to make the results comprehensible when running many nodes at once
 2018-06-07 15:02:47 -04:00
Jack Robison ad2dcf0893
add the parent node id to KBucket 2018-06-07 15:02:47 -04:00
Jack Robison 406ddaa4ef
use base class to simplify Node init 
-add looping call helpers which use the same clock as the Node
 2018-06-07 15:02:47 -04:00
Jack Robison e52689a33d
remove OptimizedTreeRoutingTable for now, use TreeRoutingTable 2018-06-07 15:02:47 -04:00
Jack Robison d65dc0aec3
disable loading DictDataStore in Node.__init__ 
-to be re-done when the datastore uses sqlite
 2018-06-07 15:02:47 -04:00
Jack Robison 159e153393
make DataStore clock mockable 2018-06-07 15:02:47 -04:00
Jack Robison 9ed08f8fc9
remove unused constant 2018-06-07 15:02:47 -04:00
Jack Robison 4bd9f3bd68
remove popular hash tracking, simplify DHTPeerFinder 2018-06-07 15:02:47 -04:00
Jack Robison e6ffd7caf8
remove hashwatcher 2018-06-07 15:02:47 -04:00
Jack Robison 11aad04c31
datagram decode error logging 2018-04-19 15:47:42 -04:00
Jack Robison b89823c968
fix blob announce locking up the daemon process 2018-04-19 15:47:19 -04:00
Jack Robison 5cea031f38
return list of dictionaries from peer_list, include peer node ids 2018-04-03 13:10:30 -04:00
Jack Robison 51662b1e5b
re-add hash_queue_size function 2018-03-29 15:06:08 -04:00
Jack Robison 492858596e
add single_announce column to blob table 
-remove deprecated blob_announce_all function
-remove announce_all parameter to blob_announce
-change blob_announce to be asynchronous
 2018-03-28 18:50:24 -04:00
Jack Robison dff1fd3fe9
logging, raise default concurrent announcers 2018-03-28 18:50:24 -04:00
Jack Robison eabf4a0e40
remove delay from udp write 2018-03-28 18:50:24 -04:00
Jack Robison 5bab6f7d39
remove bandwidth stats 2018-03-28 18:50:24 -04:00
Jack Robison a41bbd5e27
pylint and tests 2018-03-28 18:50:18 -04:00
Jack Robison 43d3f7c087
add concurrent_announcers to config 2018-03-28 15:53:51 -04:00
Jack Robison c5bf64cf0a
refactor DHTHashAnnouncer 
-remove hash_announcer from Node and DiskBlobManager
-remove announcement related functions from DiskBlobManager
-update SQLiteStorage to store announcement times and provide blob hashes needing to be announced
-use dataExpireTimeout from lbrynet.dht.constants for re-announce timing
-use DeferredSemaphore for concurrent blob announcement
 2018-03-28 15:53:51 -04:00
Jack Robison ea0ea704a2
refactor iterativeAnnounceHaveBlob 
-change to only self_store if the number of contacts to store to is less than k and we are the closest node to the hash
 2018-03-28 15:53:51 -04:00
Jack Robison 14f9bb7b82
log EWOULDBLOCK 2018-03-28 15:53:51 -04:00
Jack Robison a8025b02c6
log invalid vs missing token 2018-03-28 15:53:51 -04:00
Jack Robison a96d827c0f
use reactor time in Delay 2018-03-28 15:53:50 -04:00
Jack Robison 4eab77fa10
safe start_listening 2018-03-28 15:53:50 -04:00
Jack Robison 5013426e02
logging and docstring 2018-03-28 15:53:50 -04:00
Jack Robison cb09be5336
remove unused stuff 2018-03-28 15:53:50 -04:00
Jack Robison 5628d0825b
add CallLaterManager 2018-03-28 15:53:49 -04:00
Jack Robison 88970cb0a8
move peer manager class to lbrynet.core 2018-03-28 15:53:49 -04:00
Jack Robison 87c69742cd
log packet encoding errors and warn if the transport is not connected 2018-03-28 15:53:48 -04:00
Jack Robison b4bc5e2110
cancel callLater on error or timeout 2018-03-28 15:53:48 -04:00
Jack Robison df78f7ff9f
add response assertion to announce_to_peer 2018-03-28 15:53:48 -04:00
Jack Robison bdba263224
catch TimeoutError in _IterativeFindHelper 2018-03-28 15:53:48 -04:00
Jack Robison 16fcc3f5c1
findValue inlinecallbacks refactor 2018-03-28 15:53:48 -04:00
Jack Robison 43896c8d17
refactor joinNetwork into smaller functions 
-try to re-join network if no contacts are known
 2018-03-28 15:53:47 -04:00
Jack Robison e6caedac91
remove DHTHashSupplier class, move former functions into DHTHashAnnouncer 2018-03-28 15:53:47 -04:00
Jack Robison 6666468640
add reactor arguments to Node 
-adds reactor (clock) and reactor functions listenUDP, callLater, and resolve as arguments to Node.__init__
-set the reactor clock on LoopingCalls to make them easily testable
-convert callLater manage loops to LoopingCalls
 2018-03-28 15:53:47 -04:00
Jack Robison efaa97216f
move dht node setup back into node class 2018-03-28 15:53:47 -04:00
Jack Robison e30ea50ef4
more 2018-03-28 15:53:47 -04:00
Jack Robison 3296c0fb3d
move dht related classes to lbrynet.dht 2018-03-28 15:53:47 -04:00
Jack Robison ad6a2bef7f
handle error from old clients with a broken ping command 2018-03-28 15:53:45 -04:00
Jack Robison 446c3a88dc
refactor DHTHashAnnouncer and iterativeAnnounceHaveBlob 
-use looping call for running manage function rather than a scheduled
callLater

-track announce speed

-retry store requests that failed up to 3 times

-return a dict of {blob_hash: [storing_node_id]} results from
_announce_hashes

_refreshRoutingTable inline cb refactor

-add and use DeferredLockContextManager

-don't trap errback from iterativeFindNode in iterativeAnnounceHaveBlob
 2018-03-28 15:53:45 -04:00
Jack Robison 67ef8be7b7
convert node manage function to a looping call 2018-03-28 15:53:45 -04:00
Jack Robison ecbe4113ce
move Distance to own file 2018-03-28 15:53:45 -04:00
Alex Grintsvayg 9ee5a3511d missed this in dht-compat merge 2018-03-23 12:50:18 -04:00
Alex Grintsvayg d0aa0a7d0c make dht forwards-compatible for upcoming switch to standardized bencode dict keys 2018-03-13 09:43:32 -04:00
Antonio Quartulli 4f7885e499
dht_node: add hasContacts method 
This method can be used by other components to check
if in the Node routing table there is at least one peer.

Signed-off-by: Antonio Quartulli <antonio@mandelbit.com>
 2018-02-06 12:33:10 +08:00
Antonio Quartulli ce0af77aa9
dht_node: split network start and join procedures 
In order to attempt to join the DHT several times
(i.e. when the first attempt has failed) we need to
split the components initialization from the real
joining operation.

Create node.startNetwork() to initialize the node
and keep the rest in node.joinNetwork()

Signed-off-by: Antonio Quartulli <antonio@mandelbit.com>
 2018-02-06 12:19:33 +08:00
Kay Kurokawa b380f5d344 completely remove unused from_peer field 2017-11-07 09:56:40 -05:00
Kay Kurokawa 9776655c3c remove unneeded if statement in getPeersForBlob 2017-11-07 09:56:40 -05:00
Kay Kurokawa 7e8f3254b1 Add to DHT Node class initialization argument peerPort where it serves blobs, instead of specifying it in announceHaveBlob 2017-11-07 09:56:40 -05:00
Antonio Quartulli 965969b856 Don't filter out local node ID when returning peer list 
If a node is returning a peer list for a given blob hash
(being this been requested via CLI or via DHT) and it is
part of the resulting peer list, it will filter itself out
before returning the list.

This makes the results across the DHT inconsistent as
different nodes won't include themselves when
responding a findValue/findNode query.

Remove such filtering so that the local node ID is always
included when needed.

Signed-off-by: Antonio Quartulli <antonio@mandelbit.com>
 2017-11-07 09:56:40 -05:00
Jack Robison fdaaa0e678
changelog 2017-10-30 13:09:02 -04:00
Jack Robison e89f3db31b
fix remaining length of close nodes list 2017-10-27 21:46:17 -04:00
Antonio Quartulli 673d1c4d43
Extend close nodes list by right amount 
When a k-bucket does not contain enough close nodes, the
DHT will look into neighbouring k-buckets in order to entend
the list of returned nodes.

However, the list should not be extended beyond its maximum size.

Signed-off-by: Antonio Quartulli <antonio@mandelbit.com>
 2017-10-26 02:21:24 +08:00
Jack Robison 628c46e5d9
catch id validation errors 2017-10-24 20:26:03 -04:00