Lex Berezhny
693a3346d2
pylint fixes
2018-08-24 11:37:53 -04:00
Lex Berezhny
4ece422f48
No longer inheriting from object and added proper use of super().
2018-08-24 11:37:53 -04:00
Victor Shyba
e1e7be63b8
more fixes on dht functionals
2018-08-24 11:37:53 -04:00
Victor Shyba
e1314a9d1e
working functional test_contact_rpc + more string bans
2018-08-24 11:37:53 -04:00
Victor Shyba
1ee682f06f
make bencoding right for both py3+py2
2018-08-24 11:37:53 -04:00
Victor Shyba
cea3b7630c
partial dht functional tests porting to py3
2018-08-24 11:37:53 -04:00
Victor Shyba
19211d4417
make sure bencoding works for bytes, not strings
2018-08-24 11:37:53 -04:00
Victor Shyba
c312d1b3a6
all unit/dht green on py3
2018-08-24 11:37:53 -04:00
Victor Shyba
e4ea1ccbfb
test_node green on py3
2018-08-24 11:37:53 -04:00
Victor Shyba
5b35c4e8f0
port dht.distance ~> py3
2018-08-24 11:37:53 -04:00
Victor Shyba
9967857a57
port dht.encoding ~> py3
2018-08-24 11:37:53 -04:00
Victor Shyba
07a12b66e8
port dht.contacts ~> py3
2018-08-24 11:37:53 -04:00
Lex Berezhny
69446491b8
pylint and unit test fixes
2018-08-24 11:37:50 -04:00
Lex Berezhny
5520d518b5
DHT py3 compatibility, mostly commenting out implements() and fixing imports
...
cryptstream py3 support, mostly commenting out implements()
lbry_file py3 support, mostly commenting out implements()
file_manager py3 support, mostly commenting out implements()
core py3 support, mostly commenting out implements() and fixing imports
2018-08-24 11:37:25 -04:00
Jack Robison
37ddaeca12
fix ping queue bug
2018-08-23 11:18:28 -04:00
Jack Robison
2804fa4448
fix disabling the hash announcer
2018-08-09 12:04:14 -04:00
Jack Robison
917bae568d
fix disabling blob announcement by setting concurrent announcers to 0
2018-08-09 09:22:55 -04:00
Jack Robison
5fe8022b15
don't attempt to announce blobs until we we have contacts in routing
2018-08-09 09:00:36 -04:00
Jack Robison
26e32472d5
fix upnp and peer port setup
...
-bump txupnp requirement
2018-08-07 19:04:54 -04:00
Jack Robison
f1b19f10cc
Merge branch 'master' into greedy_search
2018-08-03 13:08:31 -04:00
Jack Robison
b06dcf0a0d
cancel starting components if the reactor is stopped before startup has finished
...
-don't block starting the dht component on having found enough peers, only on setting up the protocol
2018-07-24 20:01:38 -04:00
Jack Robison
c506fc5c75
lower peer finder timeout message to debug from warning
2018-07-18 15:32:45 -04:00
Jack Robison
d29f103910
pylint
2018-07-17 17:38:19 -04:00
Jack Robison
ec51333b56
allow node udp interface to be configured
2018-07-17 17:13:33 -04:00
Victor Shyba
562d23df11
remove nodes that went ignored during iteration
2018-07-16 17:22:46 -03:00
Victor Shyba
5c7d279104
prune failures during is_ignored calls
2018-07-12 18:24:59 -03:00
Victor Shyba
cfe8e17223
filter out the bad node in results instead of punishing the node we trusted to contact
2018-07-12 18:24:18 -03:00
Victor Shyba
c589e89bf0
greedy search with exclude filtering
2018-07-05 22:03:52 -03:00
Victor Shyba
1cc7ce69ab
set the peer search timeout as configured and raise it to 30s
2018-07-03 13:47:00 -03:00
Victor Shyba
7e0784d8e0
log a warning when peer search times out for real
2018-07-03 12:20:33 -03:00
Victor Shyba
56613a71d5
let find value run until it finds a value
2018-07-03 11:51:39 -03:00
Jack Robison
929a0cc5af
exclude self and querying node from closest contacts
2018-07-02 14:54:03 -04:00
Jack Robison
b6289d101d
fix improper sorting when getting the closest peers to a hash
2018-07-02 14:54:03 -04:00
Jack Robison
7fe92d2df0
load all finished blobs into the node datastore
2018-07-02 14:31:09 -04:00
Jack Robison
03769b94b8
accept invalid store tokens for the first expire-time after startup
...
this is to better handle nodes storing to us after we've restarted our node
2018-07-02 13:26:38 -04:00
Jack Robison
e6df378e9e
fix token validation error when the dht node has just been started ( https://github.com/lbryio/lbry/issues/1248 )
2018-06-14 15:07:10 -04:00
Jack Robison
7f3ead67bf
disable forced bucket refresh during join
2018-06-07 15:02:47 -04:00
Jack Robison
b0e4fc4faa
fix iterative find lockup
2018-06-07 15:02:47 -04:00
Jack Robison
ae631f05c3
fix teardown error
2018-06-07 15:02:47 -04:00
Jack Robison
bc0da5e2d1
only use seeds in iterative bootstrap if no contacts are known yet
2018-06-07 15:02:47 -04:00
Jack Robison
44644673d7
add profiler
2018-06-07 15:02:47 -04:00
Jack Robison
e8b402f998
remove deferredLock from iterativeFind
...
-fire the first iteration right away
2018-06-07 15:02:47 -04:00
Jack Robison
4fbaaac3f3
default new contacts to protocol version 0
2018-06-07 15:02:47 -04:00
Jack Robison
db06191c33
reduce default concurrent announcers to 10
...
-lower rpc timeout to what it originally was
2018-06-07 15:02:47 -04:00
Jack Robison
b0a741b1f4
fix hash announcer semaphore
2018-06-07 15:02:47 -04:00
Jack Robison
537df6c8ad
log socket errors
2018-06-07 15:02:47 -04:00
Jack Robison
42eb172638
refactor announceHaveBlob
...
-add cached `token` to Contact objects to minimize findValue requests
-remove self_store, always store to remote contacts even if we're the closest known node to the hash
-move the store call and error handling from announceHaveBlob to a smaller function of its own
2018-06-07 15:02:47 -04:00
Jack Robison
9a63db4ec6
add protocol version to the dht and migrate old arg format for store
2018-06-07 15:02:47 -04:00
Jack Robison
7d21cc5822
pylint and more tests
2018-06-07 15:02:47 -04:00
Jack Robison
cce3c8c7b5
increase kademlia rpc timeout to 8 seconds
2018-06-07 15:02:47 -04:00
Jack Robison
0e80123615
use 12 minutes instead of 15 as delay in contact_is_good
2018-06-07 15:02:47 -04:00
Jack Robison
659632b66c
fix and update tests
2018-06-07 15:02:47 -04:00
Jack Robison
9582b7fcf5
use maybe_ping for refreshing stale buckets and storing peers
...
-move store refresh to its own looping call
2018-06-07 15:02:47 -04:00
Jack Robison
945da5985e
fix age used in datastore to determine if a value is expired
2018-06-07 15:02:47 -04:00
Jack Robison
921ee3c4c1
use refreshTimeout in getRefreshList
2018-06-07 15:02:47 -04:00
Jack Robison
8efede6ad6
maybe_ping bad and unknown contacts instead of only unknown
2018-06-07 15:02:47 -04:00
Jack Robison
fb3aac15fa
add optional delay argument to enqueue_maybe_ping, fix default value
2018-06-07 15:02:47 -04:00
Jack Robison
73e813f9ec
verify key size in Distance
2018-06-07 15:02:47 -04:00
Jack Robison
0d23c68797
raise attribute error for non-rpc functions in Contact
2018-06-07 15:02:47 -04:00
Jack Robison
ec1b6b2387
comments, cleaner key_bits constant
2018-06-07 15:02:47 -04:00
Jack Robison
29d5750371
pylint
2018-06-07 15:02:47 -04:00
Jack Robison
07f92014d7
omit bad contacts from getPeersForBlob
...
-refresh stored contacts to detect when they go offline or come back
2018-06-07 15:02:47 -04:00
Jack Robison
c521120b17
update and fix hash announcer test
2018-06-07 15:02:47 -04:00
Jack Robison
d02ed29e50
add kademlia store and expiration test
2018-06-07 15:02:47 -04:00
Jack Robison
877da78505
ping contacts right away during refresh instead of using PingQueue
...
-fixes contact expiration test
2018-06-07 15:02:47 -04:00
Jack Robison
a952d2d2c8
reset _listeningPort and _listening Deferred on teardown
2018-06-07 15:02:47 -04:00
Jack Robison
3dfc6bd2cc
update CallLaterManager to be an object
2018-06-07 15:02:47 -04:00
Jack Robison
760417ff3a
pylint
2018-06-07 15:02:47 -04:00
Jack Robison
4f72098cad
use PingQueue to try refresh all contacts
2018-06-07 15:02:47 -04:00
Jack Robison
9920ff59d4
force KBucket refreshes
2018-06-07 15:02:47 -04:00
Jack Robison
aee7a3aa38
simplify announceHaveBlob, remove unused getPeersForBlob
2018-06-07 15:02:47 -04:00
Jack Robison
372fb45e06
refactor dht bootstrap
...
after finding the closest nodes try to populate the buckets out by looking up random ids in their key ranges
2018-06-07 15:02:47 -04:00
Jack Robison
c65274e9e5
add PingQueue to KademliaProtocol
2018-06-07 15:02:47 -04:00
Jack Robison
1adf4f7818
fix constant used to check if a bucket is fresh
2018-06-07 15:02:47 -04:00
Jack Robison
05241012a2
update contact replacement in KBucket to follow BEP0005
...
http://www.bittorrent.org/beps/bep_0005.html
2018-06-07 15:02:47 -04:00
Jack Robison
cf3359044d
fix conditions for when a kbucket should be split
...
https://stackoverflow.com/a/32187456
2018-06-07 15:02:47 -04:00
Jack Robison
5631a24881
improve findCloseNodes, choose closest contacts from higher and lower buckets
2018-06-07 15:02:47 -04:00
Jack Robison
c654bfe296
use reactor clock in TreeRoutingTable instead of time module
2018-06-07 15:02:47 -04:00
Jack Robison
d4e28216a0
sort KBucket.getContacts
2018-06-07 15:02:47 -04:00
Jack Robison
95ed1e044b
raise TransportNotConnected instead of logging a warning
...
-add a _listening Deferred to KademliaProtocol which is called when the protocol is started
2018-06-07 15:02:47 -04:00
Jack Robison
f1e0a784d9
refactor iterativeFind, move to own file
2018-06-07 15:02:47 -04:00
Jack Robison
e5703833cf
prevent duplicate entries in the datastore
2018-06-07 15:02:47 -04:00
Jack Robison
23c202b5e4
refactor Contact class, DHT RPCs, and Contact addition/removal
...
-track contact failures, last replied, and last requested. use this to provide a 'contact_is_good' property on Contact objects
-ensure no duplicate contact objects are created
-remove confusing conflation of node id strings with Contact objects, update docstrings
-move RPC failure tracking to a callback/errback pair in sendRPC (so the contact is only updated once)
-handle seed nodes during the join sequence by setting their node ids after they initially reply to our ping
-name all of the kademlia RPC keyword args, remove confusing **kwargs and dictionary parsing
-add host ip/port to DHT send/receive logging to make the results comprehensible when running many nodes at once
2018-06-07 15:02:47 -04:00
Jack Robison
ad2dcf0893
add the parent node id to KBucket
2018-06-07 15:02:47 -04:00
Jack Robison
406ddaa4ef
use base class to simplify Node init
...
-add looping call helpers which use the same clock as the Node
2018-06-07 15:02:47 -04:00
Jack Robison
e52689a33d
remove OptimizedTreeRoutingTable for now, use TreeRoutingTable
2018-06-07 15:02:47 -04:00
Jack Robison
d65dc0aec3
disable loading DictDataStore in Node.__init__
...
-to be re-done when the datastore uses sqlite
2018-06-07 15:02:47 -04:00
Jack Robison
159e153393
make DataStore clock mockable
2018-06-07 15:02:47 -04:00
Jack Robison
9ed08f8fc9
remove unused constant
2018-06-07 15:02:47 -04:00
Jack Robison
4bd9f3bd68
remove popular hash tracking, simplify DHTPeerFinder
2018-06-07 15:02:47 -04:00
Jack Robison
e6ffd7caf8
remove hashwatcher
2018-06-07 15:02:47 -04:00
Jack Robison
11aad04c31
datagram decode error logging
2018-04-19 15:47:42 -04:00
Jack Robison
b89823c968
fix blob announce locking up the daemon process
2018-04-19 15:47:19 -04:00
Jack Robison
5cea031f38
return list of dictionaries from peer_list, include peer node ids
2018-04-03 13:10:30 -04:00
Jack Robison
51662b1e5b
re-add hash_queue_size function
2018-03-29 15:06:08 -04:00
Jack Robison
492858596e
add single_announce column to blob table
...
-remove deprecated blob_announce_all function
-remove announce_all parameter to blob_announce
-change blob_announce to be asynchronous
2018-03-28 18:50:24 -04:00
Jack Robison
dff1fd3fe9
logging, raise default concurrent announcers
2018-03-28 18:50:24 -04:00
Jack Robison
eabf4a0e40
remove delay from udp write
2018-03-28 18:50:24 -04:00
Jack Robison
5bab6f7d39
remove bandwidth stats
2018-03-28 18:50:24 -04:00
Jack Robison
a41bbd5e27
pylint and tests
2018-03-28 18:50:18 -04:00
Jack Robison
43d3f7c087
add concurrent_announcers to config
2018-03-28 15:53:51 -04:00
Jack Robison
c5bf64cf0a
refactor DHTHashAnnouncer
...
-remove hash_announcer from Node and DiskBlobManager
-remove announcement related functions from DiskBlobManager
-update SQLiteStorage to store announcement times and provide blob hashes needing to be announced
-use dataExpireTimeout from lbrynet.dht.constants for re-announce timing
-use DeferredSemaphore for concurrent blob announcement
2018-03-28 15:53:51 -04:00
Jack Robison
ea0ea704a2
refactor iterativeAnnounceHaveBlob
...
-change to only self_store if the number of contacts to store to is less than k and we are the closest node to the hash
2018-03-28 15:53:51 -04:00
Jack Robison
14f9bb7b82
log EWOULDBLOCK
2018-03-28 15:53:51 -04:00
Jack Robison
a8025b02c6
log invalid vs missing token
2018-03-28 15:53:51 -04:00
Jack Robison
a96d827c0f
use reactor time in Delay
2018-03-28 15:53:50 -04:00
Jack Robison
4eab77fa10
safe start_listening
2018-03-28 15:53:50 -04:00
Jack Robison
5013426e02
logging and docstring
2018-03-28 15:53:50 -04:00
Jack Robison
cb09be5336
remove unused stuff
2018-03-28 15:53:50 -04:00
Jack Robison
5628d0825b
add CallLaterManager
2018-03-28 15:53:49 -04:00
Jack Robison
88970cb0a8
move peer manager class to lbrynet.core
2018-03-28 15:53:49 -04:00
Jack Robison
87c69742cd
log packet encoding errors and warn if the transport is not connected
2018-03-28 15:53:48 -04:00
Jack Robison
b4bc5e2110
cancel callLater on error or timeout
2018-03-28 15:53:48 -04:00
Jack Robison
df78f7ff9f
add response assertion to announce_to_peer
2018-03-28 15:53:48 -04:00
Jack Robison
bdba263224
catch TimeoutError in _IterativeFindHelper
2018-03-28 15:53:48 -04:00
Jack Robison
16fcc3f5c1
findValue inlinecallbacks refactor
2018-03-28 15:53:48 -04:00
Jack Robison
43896c8d17
refactor joinNetwork into smaller functions
...
-try to re-join network if no contacts are known
2018-03-28 15:53:47 -04:00
Jack Robison
e6caedac91
remove DHTHashSupplier class, move former functions into DHTHashAnnouncer
2018-03-28 15:53:47 -04:00
Jack Robison
6666468640
add reactor arguments to Node
...
-adds reactor (clock) and reactor functions listenUDP, callLater, and resolve as arguments to Node.__init__
-set the reactor clock on LoopingCalls to make them easily testable
-convert callLater manage loops to LoopingCalls
2018-03-28 15:53:47 -04:00
Jack Robison
efaa97216f
move dht node setup back into node class
2018-03-28 15:53:47 -04:00
Jack Robison
e30ea50ef4
more
2018-03-28 15:53:47 -04:00
Jack Robison
3296c0fb3d
move dht related classes to lbrynet.dht
2018-03-28 15:53:47 -04:00
Jack Robison
ad6a2bef7f
handle error from old clients with a broken ping command
2018-03-28 15:53:45 -04:00
Jack Robison
446c3a88dc
refactor DHTHashAnnouncer and iterativeAnnounceHaveBlob
...
-use looping call for running manage function rather than a scheduled
callLater
-track announce speed
-retry store requests that failed up to 3 times
-return a dict of {blob_hash: [storing_node_id]} results from
_announce_hashes
_refreshRoutingTable inline cb refactor
-add and use DeferredLockContextManager
-don't trap errback from iterativeFindNode in iterativeAnnounceHaveBlob
2018-03-28 15:53:45 -04:00
Jack Robison
67ef8be7b7
convert node manage function to a looping call
2018-03-28 15:53:45 -04:00
Jack Robison
ecbe4113ce
move Distance to own file
2018-03-28 15:53:45 -04:00
Alex Grintsvayg
9ee5a3511d
missed this in dht-compat merge
2018-03-23 12:50:18 -04:00
Alex Grintsvayg
d0aa0a7d0c
make dht forwards-compatible for upcoming switch to standardized bencode dict keys
2018-03-13 09:43:32 -04:00
Antonio Quartulli
4f7885e499
dht_node: add hasContacts method
...
This method can be used by other components to check
if in the Node routing table there is at least one peer.
Signed-off-by: Antonio Quartulli <antonio@mandelbit.com>
2018-02-06 12:33:10 +08:00
Antonio Quartulli
ce0af77aa9
dht_node: split network start and join procedures
...
In order to attempt to join the DHT several times
(i.e. when the first attempt has failed) we need to
split the components initialization from the real
joining operation.
Create node.startNetwork() to initialize the node
and keep the rest in node.joinNetwork()
Signed-off-by: Antonio Quartulli <antonio@mandelbit.com>
2018-02-06 12:19:33 +08:00
Kay Kurokawa
b380f5d344
completely remove unused from_peer field
2017-11-07 09:56:40 -05:00
Kay Kurokawa
9776655c3c
remove unneeded if statement in getPeersForBlob
2017-11-07 09:56:40 -05:00
Kay Kurokawa
7e8f3254b1
Add to DHT Node class initialization argument peerPort where it serves blobs, instead of specifying it in announceHaveBlob
2017-11-07 09:56:40 -05:00
Antonio Quartulli
965969b856
Don't filter out local node ID when returning peer list
...
If a node is returning a peer list for a given blob hash
(being this been requested via CLI or via DHT) and it is
part of the resulting peer list, it will filter itself out
before returning the list.
This makes the results across the DHT inconsistent as
different nodes won't include themselves when
responding a findValue/findNode query.
Remove such filtering so that the local node ID is always
included when needed.
Signed-off-by: Antonio Quartulli <antonio@mandelbit.com>
2017-11-07 09:56:40 -05:00
Jack Robison
fdaaa0e678
changelog
2017-10-30 13:09:02 -04:00
Jack Robison
e89f3db31b
fix remaining length of close nodes list
2017-10-27 21:46:17 -04:00
Antonio Quartulli
673d1c4d43
Extend close nodes list by right amount
...
When a k-bucket does not contain enough close nodes, the
DHT will look into neighbouring k-buckets in order to entend
the list of returned nodes.
However, the list should not be extended beyond its maximum size.
Signed-off-by: Antonio Quartulli <antonio@mandelbit.com>
2017-10-26 02:21:24 +08:00
Jack Robison
628c46e5d9
catch id validation errors
2017-10-24 20:26:03 -04:00
Jack Robison
e2259fd8f7
fix dht rpc id length
2017-10-24 20:25:54 -04:00
Jack Robison
9479179259
avoid future contact id parsing bugs by passing the contact id as an arg
2017-10-24 19:17:17 -04:00
Jack Robison
0b771c16ba
fix lbry id for dead contact to replace in _IterativeFindHelper
2017-10-24 19:13:45 -04:00
Jack Robison
45adfc533b
fix id of new contact replacing a dead contact
2017-10-23 01:34:34 -04:00
Jack Robison
d23d4c1d73
fix lbryid length comparison
2017-10-23 01:23:04 -04:00
Jack Robison
e9fd8eb096
update tests and scripts
2017-10-10 13:55:31 -04:00
Jack Robison
9919fd06c6
uncomment error for store request where contact is not known
2017-10-10 13:29:54 -04:00
Jack Robison
c9515f8fb6
logging and whitespace
2017-10-10 13:29:29 -04:00
Jack Robison
5937ead17c
add Node.contacts helper property
2017-10-10 13:28:57 -04:00
Jack Robison
23ce278f8a
add removePeer to interface
2017-10-10 13:27:44 -04:00
Jack Robison
de1dc507ac
fix findCloseNodes when buckets aren't fully populated
2017-10-10 13:21:06 -04:00
Jack Robison
ab956d4a8e
use count parameter in findCloseNodes
2017-10-10 13:20:19 -04:00
