Madiator2011/lbry-sdk
1
0
Fork 0
forked from LBRYCommunity/lbry-sdk
Code Pull requests Activity
4538 commits 111 branches 641 tags 30 MiB
Commit graph

239 commits

Author SHA1 Message Date
Victor Shyba 9967857a57
port dht.encoding ~> py3 2018-08-24 11:37:53 -04:00
Victor Shyba 07a12b66e8
port dht.contacts ~> py3 2018-08-24 11:37:53 -04:00
Lex Berezhny 69446491b8
pylint and unit test fixes 2018-08-24 11:37:50 -04:00
Lex Berezhny 5520d518b5
DHT py3 compatibility, mostly commenting out implements() and fixing imports 
cryptstream py3 support, mostly commenting out implements()
lbry_file py3 support, mostly commenting out implements()
file_manager py3 support, mostly commenting out implements()
core py3 support, mostly commenting out implements() and fixing imports
 2018-08-24 11:37:25 -04:00
Jack Robison 37ddaeca12
fix ping queue bug 2018-08-23 11:18:28 -04:00
Jack Robison 2804fa4448
fix disabling the hash announcer 2018-08-09 12:04:14 -04:00
Jack Robison 917bae568d
fix disabling blob announcement by setting concurrent announcers to 0 2018-08-09 09:22:55 -04:00
Jack Robison 5fe8022b15
don't attempt to announce blobs until we we have contacts in routing 2018-08-09 09:00:36 -04:00
Jack Robison 26e32472d5
fix upnp and peer port setup 
-bump txupnp requirement
 2018-08-07 19:04:54 -04:00
Jack Robison f1b19f10cc
Merge branch 'master' into greedy_search 2018-08-03 13:08:31 -04:00
Jack Robison b06dcf0a0d
cancel starting components if the reactor is stopped before startup has finished 
-don't block starting the dht component on having found enough peers, only on setting up the protocol
 2018-07-24 20:01:38 -04:00
Jack Robison c506fc5c75
lower peer finder timeout message to debug from warning 2018-07-18 15:32:45 -04:00
Jack Robison d29f103910
pylint 2018-07-17 17:38:19 -04:00
Jack Robison ec51333b56
allow node udp interface to be configured 2018-07-17 17:13:33 -04:00
Victor Shyba 562d23df11 remove nodes that went ignored during iteration 2018-07-16 17:22:46 -03:00
Victor Shyba 5c7d279104 prune failures during is_ignored calls 2018-07-12 18:24:59 -03:00
Victor Shyba cfe8e17223 filter out the bad node in results instead of punishing the node we trusted to contact 2018-07-12 18:24:18 -03:00
Victor Shyba c589e89bf0 greedy search with exclude filtering 2018-07-05 22:03:52 -03:00
Victor Shyba 1cc7ce69ab set the peer search timeout as configured and raise it to 30s 2018-07-03 13:47:00 -03:00
Victor Shyba 7e0784d8e0 log a warning when peer search times out for real 2018-07-03 12:20:33 -03:00
Victor Shyba 56613a71d5 let find value run until it finds a value 2018-07-03 11:51:39 -03:00
Jack Robison 929a0cc5af
exclude self and querying node from closest contacts 2018-07-02 14:54:03 -04:00
Jack Robison b6289d101d
fix improper sorting when getting the closest peers to a hash 2018-07-02 14:54:03 -04:00
Jack Robison 7fe92d2df0
load all finished blobs into the node datastore 2018-07-02 14:31:09 -04:00
Jack Robison 03769b94b8
accept invalid store tokens for the first expire-time after startup 
this is to better handle nodes storing to us after we've restarted our node
 2018-07-02 13:26:38 -04:00
Jack Robison e6df378e9e
fix token validation error when the dht node has just been started (https://github.com/lbryio/lbry/issues/1248) 2018-06-14 15:07:10 -04:00
Jack Robison 7f3ead67bf
disable forced bucket refresh during join 2018-06-07 15:02:47 -04:00
Jack Robison b0e4fc4faa
fix iterative find lockup 2018-06-07 15:02:47 -04:00
Jack Robison ae631f05c3
fix teardown error 2018-06-07 15:02:47 -04:00
Jack Robison bc0da5e2d1
only use seeds in iterative bootstrap if no contacts are known yet 2018-06-07 15:02:47 -04:00
Jack Robison 44644673d7
add profiler 2018-06-07 15:02:47 -04:00
Jack Robison e8b402f998
remove deferredLock from iterativeFind 
-fire the first iteration right away
 2018-06-07 15:02:47 -04:00
Jack Robison 4fbaaac3f3
default new contacts to protocol version 0 2018-06-07 15:02:47 -04:00
Jack Robison db06191c33
reduce default concurrent announcers to 10 
-lower rpc timeout to what it originally was
 2018-06-07 15:02:47 -04:00
Jack Robison b0a741b1f4
fix hash announcer semaphore 2018-06-07 15:02:47 -04:00
Jack Robison 537df6c8ad
log socket errors 2018-06-07 15:02:47 -04:00
Jack Robison 42eb172638
refactor announceHaveBlob 
-add cached `token` to Contact objects to minimize findValue requests
-remove self_store, always store to remote contacts even if we're the closest known node to the hash
-move the store call and error handling from announceHaveBlob to a smaller function of its own
 2018-06-07 15:02:47 -04:00
Jack Robison 9a63db4ec6
add protocol version to the dht and migrate old arg format for store 2018-06-07 15:02:47 -04:00
Jack Robison 7d21cc5822
pylint and more tests 2018-06-07 15:02:47 -04:00
Jack Robison cce3c8c7b5
increase kademlia rpc timeout to 8 seconds 2018-06-07 15:02:47 -04:00
Jack Robison 0e80123615
use 12 minutes instead of 15 as delay in contact_is_good 2018-06-07 15:02:47 -04:00
Jack Robison 659632b66c
fix and update tests 2018-06-07 15:02:47 -04:00
Jack Robison 9582b7fcf5
use maybe_ping for refreshing stale buckets and storing peers 
-move store refresh to its own looping call
 2018-06-07 15:02:47 -04:00
Jack Robison 945da5985e
fix age used in datastore to determine if a value is expired 2018-06-07 15:02:47 -04:00
Jack Robison 921ee3c4c1
use refreshTimeout in getRefreshList 2018-06-07 15:02:47 -04:00
Jack Robison 8efede6ad6
maybe_ping bad and unknown contacts instead of only unknown 2018-06-07 15:02:47 -04:00
Jack Robison fb3aac15fa
add optional delay argument to enqueue_maybe_ping, fix default value 2018-06-07 15:02:47 -04:00
Jack Robison 73e813f9ec
verify key size in Distance 2018-06-07 15:02:47 -04:00
Jack Robison 0d23c68797
raise attribute error for non-rpc functions in Contact 2018-06-07 15:02:47 -04:00
Jack Robison ec1b6b2387
comments, cleaner key_bits constant 2018-06-07 15:02:47 -04:00
Jack Robison 29d5750371
pylint 2018-06-07 15:02:47 -04:00
Jack Robison 07f92014d7
omit bad contacts from getPeersForBlob 
-refresh stored contacts to detect when they go offline or come back
 2018-06-07 15:02:47 -04:00
Jack Robison c521120b17
update and fix hash announcer test 2018-06-07 15:02:47 -04:00
Jack Robison d02ed29e50
add kademlia store and expiration test 2018-06-07 15:02:47 -04:00
Jack Robison 877da78505
ping contacts right away during refresh instead of using PingQueue 
-fixes contact expiration test
 2018-06-07 15:02:47 -04:00
Jack Robison a952d2d2c8
reset _listeningPort and _listening Deferred on teardown 2018-06-07 15:02:47 -04:00
Jack Robison 3dfc6bd2cc
update CallLaterManager to be an object 2018-06-07 15:02:47 -04:00
Jack Robison 760417ff3a
pylint 2018-06-07 15:02:47 -04:00
Jack Robison 4f72098cad
use PingQueue to try refresh all contacts 2018-06-07 15:02:47 -04:00
Jack Robison 9920ff59d4
force KBucket refreshes 2018-06-07 15:02:47 -04:00
Jack Robison aee7a3aa38
simplify announceHaveBlob, remove unused getPeersForBlob 2018-06-07 15:02:47 -04:00
Jack Robison 372fb45e06
refactor dht bootstrap 
after finding the closest nodes try to populate the buckets out by looking up random ids in their key ranges
 2018-06-07 15:02:47 -04:00
Jack Robison c65274e9e5
add PingQueue to KademliaProtocol 2018-06-07 15:02:47 -04:00
Jack Robison 1adf4f7818
fix constant used to check if a bucket is fresh 2018-06-07 15:02:47 -04:00
Jack Robison 05241012a2
update contact replacement in KBucket to follow BEP0005 
http://www.bittorrent.org/beps/bep_0005.html
 2018-06-07 15:02:47 -04:00
Jack Robison cf3359044d
fix conditions for when a kbucket should be split 
https://stackoverflow.com/a/32187456
 2018-06-07 15:02:47 -04:00
Jack Robison 5631a24881
improve findCloseNodes, choose closest contacts from higher and lower buckets 2018-06-07 15:02:47 -04:00
Jack Robison c654bfe296
use reactor clock in TreeRoutingTable instead of time module 2018-06-07 15:02:47 -04:00
Jack Robison d4e28216a0
sort KBucket.getContacts 2018-06-07 15:02:47 -04:00
Jack Robison 95ed1e044b
raise TransportNotConnected instead of logging a warning 
-add a _listening Deferred to KademliaProtocol which is called when the protocol is started
 2018-06-07 15:02:47 -04:00
Jack Robison f1e0a784d9
refactor iterativeFind, move to own file 2018-06-07 15:02:47 -04:00
Jack Robison e5703833cf
prevent duplicate entries in the datastore 2018-06-07 15:02:47 -04:00
Jack Robison 23c202b5e4
refactor Contact class, DHT RPCs, and Contact addition/removal 
-track contact failures, last replied, and last requested. use this to provide a 'contact_is_good' property on Contact objects
-ensure no duplicate contact objects are created
-remove confusing conflation of node id strings with Contact objects, update docstrings
-move RPC failure tracking to a callback/errback pair in sendRPC (so the contact is only updated once)
-handle seed nodes during the join sequence by setting their node ids after they initially reply to our ping
-name all of the kademlia RPC keyword args, remove confusing **kwargs and dictionary parsing
-add host ip/port to DHT send/receive logging to make the results comprehensible when running many nodes at once
 2018-06-07 15:02:47 -04:00
Jack Robison ad2dcf0893
add the parent node id to KBucket 2018-06-07 15:02:47 -04:00
Jack Robison 406ddaa4ef
use base class to simplify Node init 
-add looping call helpers which use the same clock as the Node
 2018-06-07 15:02:47 -04:00
Jack Robison e52689a33d
remove OptimizedTreeRoutingTable for now, use TreeRoutingTable 2018-06-07 15:02:47 -04:00
Jack Robison d65dc0aec3
disable loading DictDataStore in Node.__init__ 
-to be re-done when the datastore uses sqlite
 2018-06-07 15:02:47 -04:00
Jack Robison 159e153393
make DataStore clock mockable 2018-06-07 15:02:47 -04:00
Jack Robison 9ed08f8fc9
remove unused constant 2018-06-07 15:02:47 -04:00
Jack Robison 4bd9f3bd68
remove popular hash tracking, simplify DHTPeerFinder 2018-06-07 15:02:47 -04:00
Jack Robison e6ffd7caf8
remove hashwatcher 2018-06-07 15:02:47 -04:00
Jack Robison 11aad04c31
datagram decode error logging 2018-04-19 15:47:42 -04:00
Jack Robison b89823c968
fix blob announce locking up the daemon process 2018-04-19 15:47:19 -04:00
Jack Robison 5cea031f38
return list of dictionaries from peer_list, include peer node ids 2018-04-03 13:10:30 -04:00
Jack Robison 51662b1e5b
re-add hash_queue_size function 2018-03-29 15:06:08 -04:00
Jack Robison 492858596e
add single_announce column to blob table 
-remove deprecated blob_announce_all function
-remove announce_all parameter to blob_announce
-change blob_announce to be asynchronous
 2018-03-28 18:50:24 -04:00
Jack Robison dff1fd3fe9
logging, raise default concurrent announcers 2018-03-28 18:50:24 -04:00
Jack Robison eabf4a0e40
remove delay from udp write 2018-03-28 18:50:24 -04:00
Jack Robison 5bab6f7d39
remove bandwidth stats 2018-03-28 18:50:24 -04:00
Jack Robison a41bbd5e27
pylint and tests 2018-03-28 18:50:18 -04:00
Jack Robison 43d3f7c087
add concurrent_announcers to config 2018-03-28 15:53:51 -04:00
Jack Robison c5bf64cf0a
refactor DHTHashAnnouncer 
-remove hash_announcer from Node and DiskBlobManager
-remove announcement related functions from DiskBlobManager
-update SQLiteStorage to store announcement times and provide blob hashes needing to be announced
-use dataExpireTimeout from lbrynet.dht.constants for re-announce timing
-use DeferredSemaphore for concurrent blob announcement
 2018-03-28 15:53:51 -04:00
Jack Robison ea0ea704a2
refactor iterativeAnnounceHaveBlob 
-change to only self_store if the number of contacts to store to is less than k and we are the closest node to the hash
 2018-03-28 15:53:51 -04:00
Jack Robison 14f9bb7b82
log EWOULDBLOCK 2018-03-28 15:53:51 -04:00
Jack Robison a8025b02c6
log invalid vs missing token 2018-03-28 15:53:51 -04:00
Jack Robison a96d827c0f
use reactor time in Delay 2018-03-28 15:53:50 -04:00
Jack Robison 4eab77fa10
safe start_listening 2018-03-28 15:53:50 -04:00
Jack Robison 5013426e02
logging and docstring 2018-03-28 15:53:50 -04:00
Jack Robison cb09be5336
remove unused stuff 2018-03-28 15:53:50 -04:00
Jack Robison 5628d0825b
add CallLaterManager 2018-03-28 15:53:49 -04:00