223 commits

Author	SHA1	Message	Date
Jack Robison	5631a24881	improve findCloseNodes, choose closest contacts from higher and lower buckets	2018-06-07 15:02:47 -04:00
Jack Robison	c654bfe296	use reactor clock in TreeRoutingTable instead of time module	2018-06-07 15:02:47 -04:00
Jack Robison	d4e28216a0	sort KBucket.getContacts	2018-06-07 15:02:47 -04:00
Jack Robison	95ed1e044b	raise TransportNotConnected instead of logging a warning ... -add a _listening Deferred to KademliaProtocol which is called when the protocol is started	2018-06-07 15:02:47 -04:00
Jack Robison	f1e0a784d9	refactor iterativeFind, move to own file	2018-06-07 15:02:47 -04:00
Jack Robison	e5703833cf	prevent duplicate entries in the datastore	2018-06-07 15:02:47 -04:00
Jack Robison	23c202b5e4	refactor Contact class, DHT RPCs, and Contact addition/removal ... -track contact failures, last replied, and last requested. use this to provide a 'contact_is_good' property on Contact objects -ensure no duplicate contact objects are created -remove confusing conflation of node id strings with Contact objects, update docstrings -move RPC failure tracking to a callback/errback pair in sendRPC (so the contact is only updated once) -handle seed nodes during the join sequence by setting their node ids after they initially reply to our ping -name all of the kademlia RPC keyword args, remove confusing **kwargs and dictionary parsing -add host ip/port to DHT send/receive logging to make the results comprehensible when running many nodes at once	2018-06-07 15:02:47 -04:00
Jack Robison	ad2dcf0893	add the parent node id to KBucket	2018-06-07 15:02:47 -04:00
Jack Robison	406ddaa4ef	use base class to simplify Node init ... -add looping call helpers which use the same clock as the Node	2018-06-07 15:02:47 -04:00
Jack Robison	e52689a33d	remove OptimizedTreeRoutingTable for now, use TreeRoutingTable	2018-06-07 15:02:47 -04:00
Jack Robison	d65dc0aec3	disable loading DictDataStore in Node.__init__ ... -to be re-done when the datastore uses sqlite	2018-06-07 15:02:47 -04:00
Jack Robison	159e153393	make DataStore clock mockable	2018-06-07 15:02:47 -04:00
Jack Robison	9ed08f8fc9	remove unused constant	2018-06-07 15:02:47 -04:00
Jack Robison	4bd9f3bd68	remove popular hash tracking, simplify DHTPeerFinder	2018-06-07 15:02:47 -04:00
Jack Robison	e6ffd7caf8	remove hashwatcher	2018-06-07 15:02:47 -04:00
Jack Robison	11aad04c31	datagram decode error logging	2018-04-19 15:47:42 -04:00
Jack Robison	b89823c968	fix blob announce locking up the daemon process	2018-04-19 15:47:19 -04:00
Jack Robison	5cea031f38	return list of dictionaries from peer_list, include peer node ids	2018-04-03 13:10:30 -04:00
Jack Robison	51662b1e5b	re-add hash_queue_size function	2018-03-29 15:06:08 -04:00
Jack Robison	492858596e	add single_announce column to blob table ... -remove deprecated blob_announce_all function -remove announce_all parameter to blob_announce -change blob_announce to be asynchronous	2018-03-28 18:50:24 -04:00
Jack Robison	dff1fd3fe9	logging, raise default concurrent announcers	2018-03-28 18:50:24 -04:00
Jack Robison	eabf4a0e40	remove delay from udp write	2018-03-28 18:50:24 -04:00
Jack Robison	5bab6f7d39	remove bandwidth stats	2018-03-28 18:50:24 -04:00
Jack Robison	a41bbd5e27	pylint and tests	2018-03-28 18:50:18 -04:00
Jack Robison	43d3f7c087	add concurrent_announcers to config	2018-03-28 15:53:51 -04:00
Jack Robison	c5bf64cf0a	refactor DHTHashAnnouncer ... -remove hash_announcer from Node and DiskBlobManager -remove announcement related functions from DiskBlobManager -update SQLiteStorage to store announcement times and provide blob hashes needing to be announced -use dataExpireTimeout from lbrynet.dht.constants for re-announce timing -use DeferredSemaphore for concurrent blob announcement	2018-03-28 15:53:51 -04:00
Jack Robison	ea0ea704a2	refactor iterativeAnnounceHaveBlob ... -change to only self_store if the number of contacts to store to is less than k and we are the closest node to the hash	2018-03-28 15:53:51 -04:00
Jack Robison	14f9bb7b82	log EWOULDBLOCK	2018-03-28 15:53:51 -04:00
Jack Robison	a8025b02c6	log invalid vs missing token	2018-03-28 15:53:51 -04:00
Jack Robison	a96d827c0f	use reactor time in Delay	2018-03-28 15:53:50 -04:00
Jack Robison	4eab77fa10	safe start_listening	2018-03-28 15:53:50 -04:00
Jack Robison	5013426e02	logging and docstring	2018-03-28 15:53:50 -04:00
Jack Robison	cb09be5336	remove unused stuff	2018-03-28 15:53:50 -04:00
Jack Robison	5628d0825b	add CallLaterManager	2018-03-28 15:53:49 -04:00
Jack Robison	88970cb0a8	move peer manager class to lbrynet.core	2018-03-28 15:53:49 -04:00
Jack Robison	87c69742cd	log packet encoding errors and warn if the transport is not connected	2018-03-28 15:53:48 -04:00
Jack Robison	b4bc5e2110	cancel callLater on error or timeout	2018-03-28 15:53:48 -04:00
Jack Robison	df78f7ff9f	add response assertion to announce_to_peer	2018-03-28 15:53:48 -04:00
Jack Robison	bdba263224	catch TimeoutError in _IterativeFindHelper	2018-03-28 15:53:48 -04:00
Jack Robison	16fcc3f5c1	findValue inlinecallbacks refactor	2018-03-28 15:53:48 -04:00
Jack Robison	43896c8d17	refactor joinNetwork into smaller functions ... -try to re-join network if no contacts are known	2018-03-28 15:53:47 -04:00
Jack Robison	e6caedac91	remove DHTHashSupplier class, move former functions into DHTHashAnnouncer	2018-03-28 15:53:47 -04:00
Jack Robison	6666468640	add reactor arguments to Node ... -adds reactor (clock) and reactor functions listenUDP, callLater, and resolve as arguments to Node.__init__ -set the reactor clock on LoopingCalls to make them easily testable -convert callLater manage loops to LoopingCalls	2018-03-28 15:53:47 -04:00
Jack Robison	efaa97216f	move dht node setup back into node class	2018-03-28 15:53:47 -04:00
Jack Robison	e30ea50ef4	more	2018-03-28 15:53:47 -04:00
Jack Robison	3296c0fb3d	move dht related classes to lbrynet.dht	2018-03-28 15:53:47 -04:00
Jack Robison	ad6a2bef7f	handle error from old clients with a broken ping command	2018-03-28 15:53:45 -04:00
Jack Robison	446c3a88dc	refactor DHTHashAnnouncer and iterativeAnnounceHaveBlob ... -use looping call for running manage function rather than a scheduled callLater -track announce speed -retry store requests that failed up to 3 times -return a dict of {blob_hash: [storing_node_id]} results from _announce_hashes _refreshRoutingTable inline cb refactor -add and use DeferredLockContextManager -don't trap errback from iterativeFindNode in iterativeAnnounceHaveBlob	2018-03-28 15:53:45 -04:00
Jack Robison	67ef8be7b7	convert node manage function to a looping call	2018-03-28 15:53:45 -04:00
Jack Robison	ecbe4113ce	move Distance to own file	2018-03-28 15:53:45 -04:00
Alex Grintsvayg	9ee5a3511d	missed this in dht-compat merge	2018-03-23 12:50:18 -04:00
Alex Grintsvayg	d0aa0a7d0c	make dht forwards-compatible for upcoming switch to standardized bencode dict keys	2018-03-13 09:43:32 -04:00
Antonio Quartulli	4f7885e499	dht_node: add hasContacts method ... This method can be used by other components to check if in the Node routing table there is at least one peer. Signed-off-by: Antonio Quartulli <antonio@mandelbit.com>	2018-02-06 12:33:10 +08:00
Antonio Quartulli	ce0af77aa9	dht_node: split network start and join procedures ... In order to attempt to join the DHT several times (i.e. when the first attempt has failed) we need to split the components initialization from the real joining operation. Create node.startNetwork() to initialize the node and keep the rest in node.joinNetwork() Signed-off-by: Antonio Quartulli <antonio@mandelbit.com>	2018-02-06 12:19:33 +08:00
Kay Kurokawa	b380f5d344	completely remove unused from_peer field	2017-11-07 09:56:40 -05:00
Kay Kurokawa	9776655c3c	remove unneeded if statement in getPeersForBlob	2017-11-07 09:56:40 -05:00
Kay Kurokawa	7e8f3254b1	Add to DHT Node class initialization argument peerPort where it serves blobs, instead of specifying it in announceHaveBlob	2017-11-07 09:56:40 -05:00
Antonio Quartulli	965969b856	Don't filter out local node ID when returning peer list ... If a node is returning a peer list for a given blob hash (being this been requested via CLI or via DHT) and it is part of the resulting peer list, it will filter itself out before returning the list. This makes the results across the DHT inconsistent as different nodes won't include themselves when responding a findValue/findNode query. Remove such filtering so that the local node ID is always included when needed. Signed-off-by: Antonio Quartulli <antonio@mandelbit.com>	2017-11-07 09:56:40 -05:00
Jack Robison	fdaaa0e678	changelog	2017-10-30 13:09:02 -04:00
Jack Robison	e89f3db31b	fix remaining length of close nodes list	2017-10-27 21:46:17 -04:00
Antonio Quartulli	673d1c4d43	Extend close nodes list by right amount ... When a k-bucket does not contain enough close nodes, the DHT will look into neighbouring k-buckets in order to entend the list of returned nodes. However, the list should not be extended beyond its maximum size. Signed-off-by: Antonio Quartulli <antonio@mandelbit.com>	2017-10-26 02:21:24 +08:00
Jack Robison	628c46e5d9	catch id validation errors	2017-10-24 20:26:03 -04:00
Jack Robison	e2259fd8f7	fix dht rpc id length	2017-10-24 20:25:54 -04:00
Jack Robison	9479179259	avoid future contact id parsing bugs by passing the contact id as an arg	2017-10-24 19:17:17 -04:00
Jack Robison	0b771c16ba	fix lbry id for dead contact to replace in _IterativeFindHelper	2017-10-24 19:13:45 -04:00
Jack Robison	45adfc533b	fix id of new contact replacing a dead contact	2017-10-23 01:34:34 -04:00
Jack Robison	d23d4c1d73	fix lbryid length comparison	2017-10-23 01:23:04 -04:00
Jack Robison	e9fd8eb096	update tests and scripts	2017-10-10 13:55:31 -04:00
Jack Robison	9919fd06c6	uncomment error for store request where contact is not known	2017-10-10 13:29:54 -04:00
Jack Robison	c9515f8fb6	logging and whitespace	2017-10-10 13:29:29 -04:00
Jack Robison	5937ead17c	add Node.contacts helper property	2017-10-10 13:28:57 -04:00
Jack Robison	23ce278f8a	add removePeer to interface	2017-10-10 13:27:44 -04:00
Jack Robison	de1dc507ac	fix findCloseNodes when buckets aren't fully populated	2017-10-10 13:21:06 -04:00
Jack Robison	ab956d4a8e	use count parameter in findCloseNodes	2017-10-10 13:20:19 -04:00
Jack Robison	7c50e26bd7	include method and args in _sentMessages	2017-10-10 13:19:03 -04:00
Jack Robison	a942e6f3eb	fix args for ping()	2017-10-10 13:18:38 -04:00
Jack Robison	fe2d6bad1b	fix logging error for dht rpc methods with no args (ping)	2017-10-10 13:18:00 -04:00
Jack Robison	3096c89e37	don't wrap exceptions with Failure	2017-10-10 13:17:36 -04:00
Jack Robison	2a47385b62	add rpcAttempts and replacementCacheSize constants	2017-10-10 13:16:26 -04:00
Jack Robison	fcaca05a81	fix redundant lbryid	2017-10-10 13:15:25 -04:00
Jack Robison	46e31d5b45	getPeersForBlob inlinecallbacks refactor	2017-10-10 13:12:47 -04:00
Jack Robison	8e9f3c90a1	use looping call for Node.change_token to avoid hanging delayedCalls	2017-10-10 13:10:47 -04:00
Jack Robison	f1980f524e	fix raising remote exceptions	2017-10-10 13:09:25 -04:00
Jack Robison	4a567f7ab1	organize dht errors and interfaces	2017-10-10 13:08:22 -04:00
Jack Robison	422dbb549b	catch network-unreachable error in dht protocol	2017-07-14 14:29:12 -04:00
Jack Robison	9f15573742	add unique_contacts	2017-06-14 15:46:38 -04:00
Jack Robison	02475ff985	add dht bandwidth tracking	2017-05-30 11:41:54 -04:00
Jack Robison	8243349b20	use reactor.fireSystemEvent instead of reactor.stop	2017-05-01 21:09:24 -04:00
Alex Grintsvayg	cdf67de46c	pylint type checking	2017-04-25 14:36:14 -04:00
Alex Grintsvayg	fd51d8ddf0	pylint	2017-04-25 14:36:14 -04:00
Jack Robison	d2f01a214d	reduce logging level for noisy dht warning	2017-04-19 15:59:50 -04:00
Jack Robison	105fa0b947	pylint	2017-04-10 13:59:31 -04:00
Alex Grintsvayg	8db7c37fa7	exploring dht	2017-04-10 13:45:43 -04:00
Alex Grintsvayg	99e4f9b00b	we dont always want to encode key here. bucket.keyInRange() already does it when necessary	2017-04-04 15:01:59 -04:00
Alex Grintsvayg	502e2227b5	fix KeyError bug in dht	2017-03-31 14:26:19 -04:00
Alex Grintsvayg	850f51140e	formatting	2017-03-31 13:32:43 -04:00
jobevers	71123c0de1	add warning log on EWOULDBLOCK	2017-02-11 13:04:36 -06:00
Alex Grintsvayg	35b2e56bd4	silence "[Errno 11] Resource temporarily unavailable" error	2017-02-11 13:04:35 -06:00
Job Evers-Meltzer	8fe15f507b	Better dht shutdown. ... The old code relied on the timing of DelayedCalls to see what had been called or not, but unfortunately we don't have a real-time OS so the timing on DelayedCalls can only be approximate. Changed to explicitly keep track of which calls had been made. This simplifies the shutdown logic drastically, and I believe we can take out the reactor.iterate() now	2016-12-15 23:44:35 -06:00
Job Evers-Meltzer	84baa5e065	dht: refactor _msgTimeout	2016-12-15 18:58:16 -06:00
Job Evers-Meltzer	4f3b5cd802	Better logging on DHT errors ... Timeout errors are common on the dht so log those at debug, but other errors need to (potentially) recieve more attention	2016-12-15 18:58:16 -06:00
Job Evers-Meltzer	250831a86a	remove unused distance function from routingtable	2016-12-15 18:58:15 -06:00
Job Evers-Meltzer	323bccb0ae	another distance optimization	2016-12-15 18:58:15 -06:00
Job Evers-Meltzer	0084d4684f	Add distance optimization	2016-12-15 18:58:15 -06:00
Job Evers-Meltzer	740fad5cbe	Refactor dht.node.iterativeFind ... Move nested functions into a helper class. Add new, smaller functions to increase readability	2016-12-15 18:58:15 -06:00
Job Evers-Meltzer	c30ea04959	Remove commented code ... This is the result of running eradicate (https://github.com/myint/eradicate) on the code and double-checking the changes.	2016-12-15 18:58:06 -06:00
Job Evers-Meltzer	7bce37a720	pylint: add check for missing-final-newline	2016-12-14 19:21:59 -06:00
Job Evers-Meltzer	14ac2bac39	pylint: fix trailing-whitespace	2016-12-14 19:21:59 -06:00
Job Evers-Meltzer	b64fa51567	fixup long lines	2016-12-07 09:38:34 -05:00
Jack	5ae3485411	Merge branch 'master' into error-messages ... # Conflicts: # lbrynet/dht/protocol.py	2016-11-14 14:11:02 -05:00
Jack	f9fef3d586	reactor.iterate() in dht shutdown ... -fixes enigmatic `’Port' object has no attribute 'socket’` error	2016-11-11 13:41:36 -05:00
Job Evers-Meltzer	d41881596d	Restore reactor.iterate in dht shutdown ... This reverts the change made in 693fef1964 Not sure why this is needed, but taking this out sometimes causes `exceptions.AttributeError: 'Port' object has no attribute 'socket'` to happen on shutdown.	2016-11-11 10:21:50 -06:00
Job Evers-Meltzer	61a354fa60	add bad-whitespace check to pylint	2016-11-09 11:27:07 -06:00
Job Evers-Meltzer	693fef1964	remove reactor.iterate from dht shutdown	2016-11-03 10:43:24 -05:00
Jack	fe7ea7c679	refactor resources and file streamer into their own files, use NoCacheStaticFile instead of static.File	2016-10-07 14:01:59 -04:00
Jack	6b4519e959	squelch very verbose dht error log	2016-08-12 10:46:05 -05:00
Job Evers-Meltzer	fccdf32c9f	Remove all unused imports; add check to pylint	2016-08-08 15:42:35 -05:00
Job Evers	a2eab1577d	Have TimeoutError hexlify blobs	2016-07-26 13:54:46 -05:00
Jimmy Kiselak	45f9a10bfb	hide exceptions that show up in the dht and due to bad metadata in the blockchain	2016-03-12 14:08:15 -05:00
Jack	49ae029572	Don't try to process blank IPs	2016-01-26 16:07:33 -05:00
Jimmy Kiselak	7df6e99e28	fix typo and formatting errors, stop console's shutdown from running twice on exit, give more informative message if ports can't be bound, only set upnp if there isn't already a redirect on that port/protocol, and only unset upnp if the redirect was set by the program	2015-10-19 15:05:17 -04:00
Jimmy Kiselak	bb5cd49327	Use named loggers, fix some pep8 formatting mistakes	2015-09-08 15:42:56 -04:00
Jimmy Kiselak	7240ff6b1c	initial commit	2015-08-20 11:27:15 -04:00