LBRYCommunity/lbry-sdk
1
0
Fork 1
Code Issues Pull requests Projects Releases 282 Packages Wiki Activity
5072 commits 110 branches 641 tags 31 MiB
Commit graph

262 commits

Author SHA1 Message Date
Victor Shyba
9178ca701c nullify and refresh tokens 2018-09-28 14:19:51 -03:00
Victor Shyba
518c447fef
standardize distance.py to int.from_bytes 2018-08-24 11:42:35 -04:00
shyba
eab95a6246
DHT fixes from review and an attempt at removing hashing and equals (#1370) 
* use int to_bytes/from_bytes instead of struct
* fix ping queue bug and dht functional tests
* run functional tests on travis
* re-add contact comparison unit test
* dont need __ne__ if its just inverting __eq__ result
 2018-08-24 11:42:35 -04:00
Victor Shyba
3b88d2465d
fixes and refactors from review 2018-08-24 11:42:08 -04:00
Victor Shyba
3f6e928cc2
fix findValue result parsing 2018-08-24 11:40:43 -04:00
Lex Berezhny
10b34d6b33
unnecessary list() added during py3 port 
instead of recursive bytes2unicode use a proper JSONEncoder to conver bytes->unicode for json.dumps()
removing excessive isinstance(data, bytes) checks
py3: / -> // and list() around .items() that gets modified in loop
moved lbrynet.undecorated to where its actually used and hopefully we can delete it eventually
removed build/upload_assets.py, travis can do all this now
 2018-08-24 11:40:43 -04:00
Lex Berezhny
bc24dbea29
refactoring of DHT tests and fixed encoding bug when dealing with bytearray 2018-08-24 11:40:43 -04:00
Victor Shyba
31630a84be
reduce entropy on DHT test suite 2018-08-24 11:40:43 -04:00
Victor Shyba
b25d592d99
more dht fixes and most of functional tests 2018-08-24 11:37:53 -04:00
Lex Berezhny
a937aff80f
pylint fixes 2018-08-24 11:37:53 -04:00
hackrush
473d2eabfa
curl works with python3 now 2018-08-24 11:37:53 -04:00
Victor Shyba
78c8c8e64d
more porting, plus some functional tests working 2018-08-24 11:37:53 -04:00
Lex Berezhny
e718caca77
final pylint fixes (hopefuly) 2018-08-24 11:37:53 -04:00
Lex Berezhny
693a3346d2
pylint fixes 2018-08-24 11:37:53 -04:00
Lex Berezhny
4ece422f48
No longer inheriting from object and added proper use of super(). 2018-08-24 11:37:53 -04:00
Victor Shyba
e1e7be63b8
more fixes on dht functionals 2018-08-24 11:37:53 -04:00
Victor Shyba
e1314a9d1e
working functional test_contact_rpc + more string bans 2018-08-24 11:37:53 -04:00
Victor Shyba
1ee682f06f
make bencoding right for both py3+py2 2018-08-24 11:37:53 -04:00
Victor Shyba
cea3b7630c
partial dht functional tests porting to py3 2018-08-24 11:37:53 -04:00
Victor Shyba
19211d4417
make sure bencoding works for bytes, not strings 2018-08-24 11:37:53 -04:00
Victor Shyba
c312d1b3a6
all unit/dht green on py3 2018-08-24 11:37:53 -04:00
Victor Shyba
e4ea1ccbfb
test_node green on py3 2018-08-24 11:37:53 -04:00
Victor Shyba
5b35c4e8f0
port dht.distance ~> py3 2018-08-24 11:37:53 -04:00
Victor Shyba
9967857a57
port dht.encoding ~> py3 2018-08-24 11:37:53 -04:00
Victor Shyba
07a12b66e8
port dht.contacts ~> py3 2018-08-24 11:37:53 -04:00
Lex Berezhny
69446491b8
pylint and unit test fixes 2018-08-24 11:37:50 -04:00
Lex Berezhny
5520d518b5
DHT py3 compatibility, mostly commenting out implements() and fixing imports 
cryptstream py3 support, mostly commenting out implements()
lbry_file py3 support, mostly commenting out implements()
file_manager py3 support, mostly commenting out implements()
core py3 support, mostly commenting out implements() and fixing imports
 2018-08-24 11:37:25 -04:00
Jack Robison
37ddaeca12
fix ping queue bug 2018-08-23 11:18:28 -04:00
Jack Robison
2804fa4448
fix disabling the hash announcer 2018-08-09 12:04:14 -04:00
Jack Robison
917bae568d
fix disabling blob announcement by setting concurrent announcers to 0 2018-08-09 09:22:55 -04:00
Jack Robison
5fe8022b15
don't attempt to announce blobs until we we have contacts in routing 2018-08-09 09:00:36 -04:00
Jack Robison
26e32472d5
fix upnp and peer port setup 
-bump txupnp requirement
 2018-08-07 19:04:54 -04:00
Jack Robison
f1b19f10cc
Merge branch 'master' into greedy_search 2018-08-03 13:08:31 -04:00
Jack Robison
b06dcf0a0d
cancel starting components if the reactor is stopped before startup has finished 
-don't block starting the dht component on having found enough peers, only on setting up the protocol
 2018-07-24 20:01:38 -04:00
Jack Robison
c506fc5c75
lower peer finder timeout message to debug from warning 2018-07-18 15:32:45 -04:00
Jack Robison
d29f103910
pylint 2018-07-17 17:38:19 -04:00
Jack Robison
ec51333b56
allow node udp interface to be configured 2018-07-17 17:13:33 -04:00
Victor Shyba
562d23df11 remove nodes that went ignored during iteration 2018-07-16 17:22:46 -03:00
Victor Shyba
5c7d279104 prune failures during is_ignored calls 2018-07-12 18:24:59 -03:00
Victor Shyba
cfe8e17223 filter out the bad node in results instead of punishing the node we trusted to contact 2018-07-12 18:24:18 -03:00
Victor Shyba
c589e89bf0 greedy search with exclude filtering 2018-07-05 22:03:52 -03:00
Victor Shyba
1cc7ce69ab set the peer search timeout as configured and raise it to 30s 2018-07-03 13:47:00 -03:00
Victor Shyba
7e0784d8e0 log a warning when peer search times out for real 2018-07-03 12:20:33 -03:00
Victor Shyba
56613a71d5 let find value run until it finds a value 2018-07-03 11:51:39 -03:00
Jack Robison
929a0cc5af
exclude self and querying node from closest contacts 2018-07-02 14:54:03 -04:00
Jack Robison
b6289d101d
fix improper sorting when getting the closest peers to a hash 2018-07-02 14:54:03 -04:00
Jack Robison
7fe92d2df0
load all finished blobs into the node datastore 2018-07-02 14:31:09 -04:00
Jack Robison
03769b94b8
accept invalid store tokens for the first expire-time after startup 
this is to better handle nodes storing to us after we've restarted our node
 2018-07-02 13:26:38 -04:00
Jack Robison
e6df378e9e
fix token validation error when the dht node has just been started (https://github.com/lbryio/lbry/issues/1248) 2018-06-14 15:07:10 -04:00
Jack Robison
7f3ead67bf
disable forced bucket refresh during join 2018-06-07 15:02:47 -04:00
Jack Robison
b0e4fc4faa
fix iterative find lockup 2018-06-07 15:02:47 -04:00
Jack Robison
ae631f05c3
fix teardown error 2018-06-07 15:02:47 -04:00
Jack Robison
bc0da5e2d1
only use seeds in iterative bootstrap if no contacts are known yet 2018-06-07 15:02:47 -04:00
Jack Robison
44644673d7
add profiler 2018-06-07 15:02:47 -04:00
Jack Robison
e8b402f998
remove deferredLock from iterativeFind 
-fire the first iteration right away
 2018-06-07 15:02:47 -04:00
Jack Robison
4fbaaac3f3
default new contacts to protocol version 0 2018-06-07 15:02:47 -04:00
Jack Robison
db06191c33
reduce default concurrent announcers to 10 
-lower rpc timeout to what it originally was
 2018-06-07 15:02:47 -04:00
Jack Robison
b0a741b1f4
fix hash announcer semaphore 2018-06-07 15:02:47 -04:00
Jack Robison
537df6c8ad
log socket errors 2018-06-07 15:02:47 -04:00
Jack Robison
42eb172638
refactor announceHaveBlob 
-add cached `token` to Contact objects to minimize findValue requests
-remove self_store, always store to remote contacts even if we're the closest known node to the hash
-move the store call and error handling from announceHaveBlob to a smaller function of its own
 2018-06-07 15:02:47 -04:00
Jack Robison
9a63db4ec6
add protocol version to the dht and migrate old arg format for store 2018-06-07 15:02:47 -04:00
Jack Robison
7d21cc5822
pylint and more tests 2018-06-07 15:02:47 -04:00
Jack Robison
cce3c8c7b5
increase kademlia rpc timeout to 8 seconds 2018-06-07 15:02:47 -04:00
Jack Robison
0e80123615
use 12 minutes instead of 15 as delay in contact_is_good 2018-06-07 15:02:47 -04:00
Jack Robison
659632b66c
fix and update tests 2018-06-07 15:02:47 -04:00
Jack Robison
9582b7fcf5
use maybe_ping for refreshing stale buckets and storing peers 
-move store refresh to its own looping call
 2018-06-07 15:02:47 -04:00
Jack Robison
945da5985e
fix age used in datastore to determine if a value is expired 2018-06-07 15:02:47 -04:00
Jack Robison
921ee3c4c1
use refreshTimeout in getRefreshList 2018-06-07 15:02:47 -04:00
Jack Robison
8efede6ad6
maybe_ping bad and unknown contacts instead of only unknown 2018-06-07 15:02:47 -04:00
Jack Robison
fb3aac15fa
add optional delay argument to enqueue_maybe_ping, fix default value 2018-06-07 15:02:47 -04:00
Jack Robison
73e813f9ec
verify key size in Distance 2018-06-07 15:02:47 -04:00
Jack Robison
0d23c68797
raise attribute error for non-rpc functions in Contact 2018-06-07 15:02:47 -04:00
Jack Robison
ec1b6b2387
comments, cleaner key_bits constant 2018-06-07 15:02:47 -04:00
Jack Robison
29d5750371
pylint 2018-06-07 15:02:47 -04:00
Jack Robison
07f92014d7
omit bad contacts from getPeersForBlob 
-refresh stored contacts to detect when they go offline or come back
 2018-06-07 15:02:47 -04:00
Jack Robison
c521120b17
update and fix hash announcer test 2018-06-07 15:02:47 -04:00
Jack Robison
d02ed29e50
add kademlia store and expiration test 2018-06-07 15:02:47 -04:00
Jack Robison
877da78505
ping contacts right away during refresh instead of using PingQueue 
-fixes contact expiration test
 2018-06-07 15:02:47 -04:00
Jack Robison
a952d2d2c8
reset _listeningPort and _listening Deferred on teardown 2018-06-07 15:02:47 -04:00
Jack Robison
3dfc6bd2cc
update CallLaterManager to be an object 2018-06-07 15:02:47 -04:00
Jack Robison
760417ff3a
pylint 2018-06-07 15:02:47 -04:00
Jack Robison
4f72098cad
use PingQueue to try refresh all contacts 2018-06-07 15:02:47 -04:00
Jack Robison
9920ff59d4
force KBucket refreshes 2018-06-07 15:02:47 -04:00
Jack Robison
aee7a3aa38
simplify announceHaveBlob, remove unused getPeersForBlob 2018-06-07 15:02:47 -04:00
Jack Robison
372fb45e06
refactor dht bootstrap 
after finding the closest nodes try to populate the buckets out by looking up random ids in their key ranges
 2018-06-07 15:02:47 -04:00
Jack Robison
c65274e9e5
add PingQueue to KademliaProtocol 2018-06-07 15:02:47 -04:00
Jack Robison
1adf4f7818
fix constant used to check if a bucket is fresh 2018-06-07 15:02:47 -04:00
Jack Robison
05241012a2
update contact replacement in KBucket to follow BEP0005 
http://www.bittorrent.org/beps/bep_0005.html
 2018-06-07 15:02:47 -04:00
Jack Robison
cf3359044d
fix conditions for when a kbucket should be split 
https://stackoverflow.com/a/32187456
 2018-06-07 15:02:47 -04:00
Jack Robison
5631a24881
improve findCloseNodes, choose closest contacts from higher and lower buckets 2018-06-07 15:02:47 -04:00
Jack Robison
c654bfe296
use reactor clock in TreeRoutingTable instead of time module 2018-06-07 15:02:47 -04:00
Jack Robison
d4e28216a0
sort KBucket.getContacts 2018-06-07 15:02:47 -04:00
Jack Robison
95ed1e044b
raise TransportNotConnected instead of logging a warning 
-add a _listening Deferred to KademliaProtocol which is called when the protocol is started
 2018-06-07 15:02:47 -04:00
Jack Robison
f1e0a784d9
refactor iterativeFind, move to own file 2018-06-07 15:02:47 -04:00
Jack Robison
e5703833cf
prevent duplicate entries in the datastore 2018-06-07 15:02:47 -04:00
Jack Robison
23c202b5e4
refactor Contact class, DHT RPCs, and Contact addition/removal 
-track contact failures, last replied, and last requested. use this to provide a 'contact_is_good' property on Contact objects
-ensure no duplicate contact objects are created
-remove confusing conflation of node id strings with Contact objects, update docstrings
-move RPC failure tracking to a callback/errback pair in sendRPC (so the contact is only updated once)
-handle seed nodes during the join sequence by setting their node ids after they initially reply to our ping
-name all of the kademlia RPC keyword args, remove confusing **kwargs and dictionary parsing
-add host ip/port to DHT send/receive logging to make the results comprehensible when running many nodes at once
 2018-06-07 15:02:47 -04:00
Jack Robison
ad2dcf0893
add the parent node id to KBucket 2018-06-07 15:02:47 -04:00
Jack Robison
406ddaa4ef
use base class to simplify Node init 
-add looping call helpers which use the same clock as the Node
 2018-06-07 15:02:47 -04:00
Jack Robison
e52689a33d
remove OptimizedTreeRoutingTable for now, use TreeRoutingTable 2018-06-07 15:02:47 -04:00
Jack Robison
d65dc0aec3
disable loading DictDataStore in Node.__init__ 
-to be re-done when the datastore uses sqlite
 2018-06-07 15:02:47 -04:00