LBRYCommunity/lbry-sdk
1
0
Fork 1
Code Issues Pull requests Packages Projects Releases 282 Wiki Activity
4135 commits 110 branches 641 tags 31 MiB
Commit graph

209 commits

Author SHA1 Message Date
Jack Robison e6ffd7caf8
remove hashwatcher 2018-06-07 15:02:47 -04:00
Jack Robison 11aad04c31
datagram decode error logging 2018-04-19 15:47:42 -04:00
Jack Robison b89823c968
fix blob announce locking up the daemon process 2018-04-19 15:47:19 -04:00
Jack Robison 5cea031f38
return list of dictionaries from peer_list, include peer node ids 2018-04-03 13:10:30 -04:00
Jack Robison 51662b1e5b
re-add hash_queue_size function 2018-03-29 15:06:08 -04:00
Jack Robison 492858596e
add single_announce column to blob table 
-remove deprecated blob_announce_all function
-remove announce_all parameter to blob_announce
-change blob_announce to be asynchronous
 2018-03-28 18:50:24 -04:00
Jack Robison dff1fd3fe9
logging, raise default concurrent announcers 2018-03-28 18:50:24 -04:00
Jack Robison eabf4a0e40
remove delay from udp write 2018-03-28 18:50:24 -04:00
Jack Robison 5bab6f7d39
remove bandwidth stats 2018-03-28 18:50:24 -04:00
Jack Robison a41bbd5e27
pylint and tests 2018-03-28 18:50:18 -04:00
Jack Robison 43d3f7c087
add concurrent_announcers to config 2018-03-28 15:53:51 -04:00
Jack Robison c5bf64cf0a
refactor DHTHashAnnouncer 
-remove hash_announcer from Node and DiskBlobManager
-remove announcement related functions from DiskBlobManager
-update SQLiteStorage to store announcement times and provide blob hashes needing to be announced
-use dataExpireTimeout from lbrynet.dht.constants for re-announce timing
-use DeferredSemaphore for concurrent blob announcement
 2018-03-28 15:53:51 -04:00
Jack Robison ea0ea704a2
refactor iterativeAnnounceHaveBlob 
-change to only self_store if the number of contacts to store to is less than k and we are the closest node to the hash
 2018-03-28 15:53:51 -04:00
Jack Robison 14f9bb7b82
log EWOULDBLOCK 2018-03-28 15:53:51 -04:00
Jack Robison a8025b02c6
log invalid vs missing token 2018-03-28 15:53:51 -04:00
Jack Robison a96d827c0f
use reactor time in Delay 2018-03-28 15:53:50 -04:00
Jack Robison 4eab77fa10
safe start_listening 2018-03-28 15:53:50 -04:00
Jack Robison 5013426e02
logging and docstring 2018-03-28 15:53:50 -04:00
Jack Robison cb09be5336
remove unused stuff 2018-03-28 15:53:50 -04:00
Jack Robison 5628d0825b
add CallLaterManager 2018-03-28 15:53:49 -04:00
Jack Robison 88970cb0a8
move peer manager class to lbrynet.core 2018-03-28 15:53:49 -04:00
Jack Robison 87c69742cd
log packet encoding errors and warn if the transport is not connected 2018-03-28 15:53:48 -04:00
Jack Robison b4bc5e2110
cancel callLater on error or timeout 2018-03-28 15:53:48 -04:00
Jack Robison df78f7ff9f
add response assertion to announce_to_peer 2018-03-28 15:53:48 -04:00
Jack Robison bdba263224
catch TimeoutError in _IterativeFindHelper 2018-03-28 15:53:48 -04:00
Jack Robison 16fcc3f5c1
findValue inlinecallbacks refactor 2018-03-28 15:53:48 -04:00
Jack Robison 43896c8d17
refactor joinNetwork into smaller functions 
-try to re-join network if no contacts are known
 2018-03-28 15:53:47 -04:00
Jack Robison e6caedac91
remove DHTHashSupplier class, move former functions into DHTHashAnnouncer 2018-03-28 15:53:47 -04:00
Jack Robison 6666468640
add reactor arguments to Node 
-adds reactor (clock) and reactor functions listenUDP, callLater, and resolve as arguments to Node.__init__
-set the reactor clock on LoopingCalls to make them easily testable
-convert callLater manage loops to LoopingCalls
 2018-03-28 15:53:47 -04:00
Jack Robison efaa97216f
move dht node setup back into node class 2018-03-28 15:53:47 -04:00
Jack Robison e30ea50ef4
more 2018-03-28 15:53:47 -04:00
Jack Robison 3296c0fb3d
move dht related classes to lbrynet.dht 2018-03-28 15:53:47 -04:00
Jack Robison ad6a2bef7f
handle error from old clients with a broken ping command 2018-03-28 15:53:45 -04:00
Jack Robison 446c3a88dc
refactor DHTHashAnnouncer and iterativeAnnounceHaveBlob 
-use looping call for running manage function rather than a scheduled
callLater

-track announce speed

-retry store requests that failed up to 3 times

-return a dict of {blob_hash: [storing_node_id]} results from
_announce_hashes

_refreshRoutingTable inline cb refactor

-add and use DeferredLockContextManager

-don't trap errback from iterativeFindNode in iterativeAnnounceHaveBlob
 2018-03-28 15:53:45 -04:00
Jack Robison 67ef8be7b7
convert node manage function to a looping call 2018-03-28 15:53:45 -04:00
Jack Robison ecbe4113ce
move Distance to own file 2018-03-28 15:53:45 -04:00
Alex Grintsvayg 9ee5a3511d missed this in dht-compat merge 2018-03-23 12:50:18 -04:00
Alex Grintsvayg d0aa0a7d0c make dht forwards-compatible for upcoming switch to standardized bencode dict keys 2018-03-13 09:43:32 -04:00
Antonio Quartulli 4f7885e499
dht_node: add hasContacts method 
This method can be used by other components to check
if in the Node routing table there is at least one peer.

Signed-off-by: Antonio Quartulli <antonio@mandelbit.com>
 2018-02-06 12:33:10 +08:00
Antonio Quartulli ce0af77aa9
dht_node: split network start and join procedures 
In order to attempt to join the DHT several times
(i.e. when the first attempt has failed) we need to
split the components initialization from the real
joining operation.

Create node.startNetwork() to initialize the node
and keep the rest in node.joinNetwork()

Signed-off-by: Antonio Quartulli <antonio@mandelbit.com>
 2018-02-06 12:19:33 +08:00
Kay Kurokawa b380f5d344 completely remove unused from_peer field 2017-11-07 09:56:40 -05:00
Kay Kurokawa 9776655c3c remove unneeded if statement in getPeersForBlob 2017-11-07 09:56:40 -05:00
Kay Kurokawa 7e8f3254b1 Add to DHT Node class initialization argument peerPort where it serves blobs, instead of specifying it in announceHaveBlob 2017-11-07 09:56:40 -05:00
Antonio Quartulli 965969b856 Don't filter out local node ID when returning peer list 
If a node is returning a peer list for a given blob hash
(being this been requested via CLI or via DHT) and it is
part of the resulting peer list, it will filter itself out
before returning the list.

This makes the results across the DHT inconsistent as
different nodes won't include themselves when
responding a findValue/findNode query.

Remove such filtering so that the local node ID is always
included when needed.

Signed-off-by: Antonio Quartulli <antonio@mandelbit.com>
 2017-11-07 09:56:40 -05:00
Jack Robison fdaaa0e678
changelog 2017-10-30 13:09:02 -04:00
Jack Robison e89f3db31b
fix remaining length of close nodes list 2017-10-27 21:46:17 -04:00
Antonio Quartulli 673d1c4d43
Extend close nodes list by right amount 
When a k-bucket does not contain enough close nodes, the
DHT will look into neighbouring k-buckets in order to entend
the list of returned nodes.

However, the list should not be extended beyond its maximum size.

Signed-off-by: Antonio Quartulli <antonio@mandelbit.com>
 2017-10-26 02:21:24 +08:00
Jack Robison 628c46e5d9
catch id validation errors 2017-10-24 20:26:03 -04:00
Jack Robison e2259fd8f7
fix dht rpc id length 2017-10-24 20:25:54 -04:00
Jack Robison 9479179259
avoid future contact id parsing bugs by passing the contact id as an arg 2017-10-24 19:17:17 -04:00
Jack Robison 0b771c16ba
fix lbry id for dead contact to replace in _IterativeFindHelper 2017-10-24 19:13:45 -04:00
Jack Robison 45adfc533b
fix id of new contact replacing a dead contact 2017-10-23 01:34:34 -04:00
Jack Robison d23d4c1d73
fix lbryid length comparison 2017-10-23 01:23:04 -04:00
Jack Robison e9fd8eb096
update tests and scripts 2017-10-10 13:55:31 -04:00
Jack Robison 9919fd06c6
uncomment error for store request where contact is not known 2017-10-10 13:29:54 -04:00
Jack Robison c9515f8fb6
logging and whitespace 2017-10-10 13:29:29 -04:00
Jack Robison 5937ead17c
add Node.contacts helper property 2017-10-10 13:28:57 -04:00
Jack Robison 23ce278f8a
add removePeer to interface 2017-10-10 13:27:44 -04:00
Jack Robison de1dc507ac
fix findCloseNodes when buckets aren't fully populated 2017-10-10 13:21:06 -04:00
Jack Robison ab956d4a8e
use count parameter in findCloseNodes 2017-10-10 13:20:19 -04:00
Jack Robison 7c50e26bd7
include method and args in _sentMessages 2017-10-10 13:19:03 -04:00
Jack Robison a942e6f3eb
fix args for ping() 2017-10-10 13:18:38 -04:00
Jack Robison fe2d6bad1b
fix logging error for dht rpc methods with no args (ping) 2017-10-10 13:18:00 -04:00
Jack Robison 3096c89e37
don't wrap exceptions with Failure 2017-10-10 13:17:36 -04:00
Jack Robison 2a47385b62
add rpcAttempts and replacementCacheSize constants 2017-10-10 13:16:26 -04:00
Jack Robison fcaca05a81
fix redundant lbryid 2017-10-10 13:15:25 -04:00
Jack Robison 46e31d5b45
getPeersForBlob inlinecallbacks refactor 2017-10-10 13:12:47 -04:00
Jack Robison 8e9f3c90a1
use looping call for Node.change_token to avoid hanging delayedCalls 2017-10-10 13:10:47 -04:00
Jack Robison f1980f524e
fix raising remote exceptions 2017-10-10 13:09:25 -04:00
Jack Robison 4a567f7ab1
organize dht errors and interfaces 2017-10-10 13:08:22 -04:00
Jack Robison 422dbb549b
catch network-unreachable error in dht protocol 2017-07-14 14:29:12 -04:00
Jack Robison 9f15573742 add unique_contacts 2017-06-14 15:46:38 -04:00
Jack Robison 02475ff985 add dht bandwidth tracking 2017-05-30 11:41:54 -04:00
Jack Robison 8243349b20 use reactor.fireSystemEvent instead of reactor.stop 2017-05-01 21:09:24 -04:00
Alex Grintsvayg cdf67de46c pylint type checking 2017-04-25 14:36:14 -04:00
Alex Grintsvayg fd51d8ddf0 pylint 2017-04-25 14:36:14 -04:00
Jack Robison d2f01a214d reduce logging level for noisy dht warning 2017-04-19 15:59:50 -04:00
Jack Robison 105fa0b947 pylint 2017-04-10 13:59:31 -04:00
Alex Grintsvayg 8db7c37fa7 exploring dht 2017-04-10 13:45:43 -04:00
Alex Grintsvayg 99e4f9b00b we dont always want to encode key here. bucket.keyInRange() already does it when necessary 2017-04-04 15:01:59 -04:00
Alex Grintsvayg 502e2227b5 fix KeyError bug in dht 2017-03-31 14:26:19 -04:00
Alex Grintsvayg 850f51140e formatting 2017-03-31 13:32:43 -04:00
jobevers 71123c0de1 add warning log on EWOULDBLOCK 2017-02-11 13:04:36 -06:00
Alex Grintsvayg 35b2e56bd4 silence "[Errno 11] Resource temporarily unavailable" error 2017-02-11 13:04:35 -06:00
Job Evers-Meltzer 8fe15f507b Better dht shutdown. 
The old code relied on the timing of DelayedCalls
to see what had been called or not, but unfortunately
we don't have a real-time OS so the timing on DelayedCalls
can only be approximate. Changed to explicitly keep
track of which calls had been made.

This simplifies the shutdown logic drastically, and I believe
we can take out the reactor.iterate() now
 2016-12-15 23:44:35 -06:00
Job Evers-Meltzer 84baa5e065 dht: refactor _msgTimeout 2016-12-15 18:58:16 -06:00
Job Evers-Meltzer 4f3b5cd802 Better logging on DHT errors 
Timeout errors are common on the dht so log those
at debug, but other errors need to (potentially) recieve
more attention
 2016-12-15 18:58:16 -06:00
Job Evers-Meltzer 250831a86a remove unused distance function from routingtable 2016-12-15 18:58:15 -06:00
Job Evers-Meltzer 323bccb0ae another distance optimization 2016-12-15 18:58:15 -06:00
Job Evers-Meltzer 0084d4684f Add distance optimization 2016-12-15 18:58:15 -06:00
Job Evers-Meltzer 740fad5cbe Refactor dht.node.iterativeFind 
Move nested functions into a helper class.
Add new, smaller functions to increase readability
 2016-12-15 18:58:15 -06:00
Job Evers-Meltzer c30ea04959 Remove commented code 
This is the result of running eradicate (https://github.com/myint/eradicate)
on the code and double-checking the changes.
 2016-12-15 18:58:06 -06:00
Job Evers-Meltzer 7bce37a720 pylint: add check for missing-final-newline 2016-12-14 19:21:59 -06:00
Job Evers-Meltzer 14ac2bac39 pylint: fix trailing-whitespace 2016-12-14 19:21:59 -06:00
Job Evers-Meltzer b64fa51567 fixup long lines 2016-12-07 09:38:34 -05:00
Jack 5ae3485411 Merge branch 'master' into error-messages 
# Conflicts:
#	lbrynet/dht/protocol.py
 2016-11-14 14:11:02 -05:00
Jack f9fef3d586 reactor.iterate() in dht shutdown 
-fixes enigmatic  `’Port' object has no attribute 'socket’` error
 2016-11-11 13:41:36 -05:00
Job Evers-Meltzer d41881596d Restore reactor.iterate in dht shutdown 
This reverts the change made in 693fef1964

Not sure why this is needed, but taking this out sometimes causes
`exceptions.AttributeError: 'Port' object has no attribute 'socket'`
to happen on shutdown.
 2016-11-11 10:21:50 -06:00
Job Evers-Meltzer 61a354fa60 add bad-whitespace check to pylint 2016-11-09 11:27:07 -06:00
Job Evers-Meltzer 693fef1964 remove reactor.iterate from dht shutdown 2016-11-03 10:43:24 -05:00
Jack fe7ea7c679 refactor resources and file streamer into their own files, use NoCacheStaticFile instead of static.File 2016-10-07 14:01:59 -04:00
Jack 6b4519e959 squelch very verbose dht error log 2016-08-12 10:46:05 -05:00
Job Evers-Meltzer fccdf32c9f Remove all unused imports; add check to pylint 2016-08-08 15:42:35 -05:00
Job Evers a2eab1577d Have TimeoutError hexlify blobs 2016-07-26 13:54:46 -05:00
Jimmy Kiselak 45f9a10bfb hide exceptions that show up in the dht and due to bad metadata in the blockchain 2016-03-12 14:08:15 -05:00
Jack 49ae029572 Don't try to process blank IPs 2016-01-26 16:07:33 -05:00
Jimmy Kiselak 7df6e99e28 fix typo and formatting errors, stop console's shutdown from running twice on exit, give more informative message if ports can't be bound, only set upnp if there isn't already a redirect on that port/protocol, and only unset upnp if the redirect was set by the program 2015-10-19 15:05:17 -04:00
Jimmy Kiselak bb5cd49327 Use named loggers, fix some pep8 formatting mistakes 2015-09-08 15:42:56 -04:00
Jimmy Kiselak 7240ff6b1c initial commit 2015-08-20 11:27:15 -04:00