3516 commits

Author	SHA1	Message	Date
Jack Robison	da9b4e317a	limit blobs to announce	2019-02-08 16:29:55 -05:00
Jack Robison	e414cc5c48	whitespace	2019-02-08 16:11:56 -05:00
Jack Robison	939edb2f18	logging ... -call loop.set_debug when using --verbose	2019-02-08 16:06:38 -05:00
Jack Robison	868110a6f2	populate stream manager with a single batch query	2019-02-08 16:06:35 -05:00
Victor Shyba	9ecec52412	check lock acquired to prevent races	2019-02-08 04:39:28 -03:00
Victor Shyba	8b25acff6e	do not wait blob to be written, log errors from races	2019-02-08 04:32:53 -03:00
Victor Shyba	e2b06677b5	defer db insert during downloads	2019-02-08 03:45:23 -03:00
Victor Shyba	af1619ebfb	disable race limiting for now	2019-02-08 03:32:38 -03:00
Victor Shyba	3352e0e4f4	score peers by speed	2019-02-08 03:08:08 -03:00
Victor Shyba	141d68a2cd	fix write race condition	2019-02-08 02:39:29 -03:00
Victor Shyba	c06ec6cd69	manage connections, reusing them when possible	2019-02-08 02:27:58 -03:00
Victor Shyba	1be5dce30e	stop racing during long streams	2019-02-08 01:28:03 -03:00
Victor Shyba	73da223ae1	conf: give up first contact earlier, but let the transfer go longer	2019-02-08 01:05:58 -03:00
Victor Shyba	2d7eb83518	change score calculation, wait for active peers too, simplify peer sorting/keeping	2019-02-08 01:04:38 -03:00
Victor Shyba	ad03f91d24	let cancelled errors flow	2019-02-08 01:03:13 -03:00
Victor Shyba	5586a226c2	bypass parser during download	2019-02-07 21:19:31 -03:00
Victor Shyba	a616582733	wait on connection tasks	2019-02-07 20:11:57 -03:00
Victor Shyba	43ac928f0b	remove dht locks	2019-02-07 20:11:57 -03:00
Lex Berezhny	a121110743	v0.31.0	2019-02-06 14:28:40 -05:00
Victor Shyba	3a15ff4bcd	test incomplete streams, respect real length, delete on incomplete assembly	2019-02-06 15:56:06 -03:00
Jack Robison	dfdc8eda00	improve test, fix stream_finished_event	2019-02-06 13:32:50 -05:00
Jack Robison	58f6cb71c6	test stream manager	2019-02-06 13:08:57 -05:00
Jack Robison	ae11c5bb4b	fix setting finished event	2019-02-06 12:18:24 -05:00
Jack Robison	f22b5da170	fix raised error for a peer address mismatch	2019-02-06 12:11:22 -05:00
Jack Robison	c1c6d5bc99	fix deleting partial downloads when stopped and previous streams when updating a publish	2019-02-06 11:43:05 -05:00
Jack Robison	0e972ec2ae	refactor BlobFile.close to be non async	2019-02-06 11:00:57 -05:00
Victor Shyba	8b5803c945	format error dict on get	2019-02-06 11:55:12 -03:00
Jack Robison	76c8447a56	add stream descriptor tests	2019-02-05 21:25:31 -05:00
Jack Robison	7c8ad0fe30	fix deleting incomplete downloads upon stop ... fix deleting previous streams (not assembled/published files) for a claim being updated	2019-02-05 15:33:08 -05:00
Jack Robison	073ffdd049	proto and type args	2019-02-05 13:36:25 -05:00
Jack Robison	7fe211d134	fix resolve_host	2019-02-05 13:31:57 -05:00
Victor Shyba	a4ced4dbd2	set verified after callback	2019-02-05 13:17:00 -03:00
Victor Shyba	c6b910a5f3	use time_getter on storage	2019-02-05 13:14:59 -03:00
Victor Shyba	63d1ec3c28	fixes from jack comments	2019-02-05 13:01:12 -03:00
Victor Shyba	7b8f42e6d0	remove dead set_should_announce code	2019-02-05 13:01:12 -03:00
Victor Shyba	2c275efa48	remove redundant calls to should announce	2019-02-05 13:01:12 -03:00
Victor Shyba	6ead932ccb	announces naturally, removes redundant calls	2019-02-05 13:01:12 -03:00
Victor Shyba	f766dbefe0	announce and set completed after inserts	2019-02-05 13:01:12 -03:00
Jack Robison	0685bc5689	include torba in log file	2019-02-04 18:25:47 -05:00
Jack Robison	8cda3143f2	fix setting should_announce and blob status	2019-02-04 18:22:44 -05:00
Jack Robison	5af351a234	reflector logging	2019-02-04 17:33:25 -05:00
Victor Shyba	b82d2c3120	handle and log if downloading from bad sendfile implementations	2019-02-04 19:13:31 -03:00
Jack Robison	2bce7cecfa	fix sendfile on windows	2019-02-04 16:48:48 -05:00
Jack Robison	8883587cc7	logging	2019-02-04 16:19:27 -05:00
Jack Robison	c85991704d	catch invalid data error	2019-02-04 16:19:27 -05:00
Jack Robison	377442d9f0	fix request_blob cancellation	2019-02-04 16:19:24 -05:00
Jack Robison	f0b49fc153	always return close peers from findValue	2019-02-04 15:07:42 -05:00
Jack Robison	ae1162f5b0	don't try downloading from oneself ... -don't return the querying peer in results for findValue	2019-02-04 14:50:53 -05:00
Victor Shyba	ad71a9a89d	refactor reflect to close protocol	2019-02-04 13:34:18 -03:00
Jack Robison	13b3e4ba8e	v0.31.0rc5	2019-02-04 10:22:43 -05:00
Victor Shyba	736a949cb5	resume streams on startup	2019-02-03 20:49:09 -03:00
Victor Shyba	11bb1cc7ee	use with clause to avoid leaking fds	2019-02-03 20:41:47 -03:00
Jack Robison	dcd78bd86e	always delete file from downloads if download was not complete	2019-02-03 16:43:40 -05:00
Jack Robison	e5af041627	fix connection status message	2019-02-03 16:20:52 -05:00
Jack Robison	0a027cf2de	logging	2019-02-03 15:47:19 -05:00
Jack Robison	3f98b16800	fix blob announce time, use time.time instead of loop.time	2019-02-03 15:47:15 -05:00
Jack Robison	8f5a8ab8ba	fix peer_ping	2019-02-02 02:02:19 -05:00
Jack Robison	91e581422c	v0.31.0rc4	2019-02-02 00:30:57 -05:00
Jack Robison	f2968aab22	add re-reflect task to stream manager, add concurrent_reflector_uploads to config	2019-02-01 22:59:41 -05:00
Jack Robison	ccbe8aa0eb	create data. download, and wallet directories if they don't exist on startup	2019-02-01 22:45:28 -05:00
Jack Robison	478f79353a	fix bytes error	2019-02-01 22:20:11 -05:00
Jack Robison	c3b0151943	fix dht status error during startup	2019-02-01 22:12:07 -05:00
Jack Robison	e96b75a0d0	fix file/download paths ... move download-from-uri logic into stream manager	2019-02-01 21:40:35 -05:00
Jack Robison	efe4afd09e	fix blob announce time	2019-02-01 21:28:59 -05:00
Jack Robison	c7cb6822cc	logging	2019-02-01 21:28:59 -05:00
Jack Robison	c75665d3f0	update file_list docs, add blobs_remaining to file list args and to file dict result	2019-02-01 21:28:59 -05:00
Jack Robison	744375b2c0	re-assemble file / resume downloads	2019-02-01 21:28:59 -05:00
Victor Shyba	40da5b6cb6	rework data parser	2019-02-01 20:09:14 -03:00
Victor Shyba	b36cf3e36c	remove closing peers from active connections	2019-02-01 20:09:14 -03:00
Jack Robison	0a08f445a4	close blob server transport on sendfile failing	2019-02-01 15:11:37 -05:00
Victor Shyba	f5bf1463af	fixes from review	2019-02-01 17:09:37 -03:00
Victor Shyba	ca4a55ef28	set verified event earlier and remove stop awaits	2019-02-01 16:05:14 -03:00
Victor Shyba	832be0758b	let cancel propagate	2019-02-01 16:05:14 -03:00
Victor Shyba	25050fdeeb	fixes for writers (check inside lock and dont hold verified bytes)	2019-02-01 16:05:14 -03:00
Jack Robison	dabd9cbe1e	faster stream manager startup	2019-02-01 13:53:03 -05:00
Jack Robison	2336015f9a	fix publish ... fixes https://github.com/lbryio/lbry/issues/1826 - remove `sources` argument from `publish` - only add/update files we have - delete existing stream if updating with a new one	2019-02-01 13:35:18 -05:00
Jack Robison	31445c7797	remove lock from PeerManager	2019-01-31 20:57:48 -05:00
Jack Robison	87f5d1bd4a	fix updating claim with existing stream	2019-01-31 20:49:19 -05:00
Jack Robison	2e9acd1baa	remove blob request asyncio.Lock	2019-01-31 20:19:50 -05:00
Jack Robison	df86937f4e	fix deleting a file mid download	2019-01-31 17:59:42 -05:00
Jack Robison	957d0290df	add torba logging	2019-01-31 16:53:54 -05:00
Jack Robison	40b90ce79b	fix hanging download	2019-01-31 16:53:27 -05:00
Jack Robison	aa7f0845cc	catch sendfile errors	2019-01-31 16:10:27 -05:00
Jack Robison	6e69d306de	tests	2019-01-31 15:39:23 -05:00
Jack Robison	d85d3dfaa4	increase default max_connections_per_download	2019-01-31 15:39:20 -05:00
Jack Robison	2b035009ef	handle both urls and ip addresses for fixed nodes and reflector servers	2019-01-31 15:39:20 -05:00
Jack Robison	f9fd62c214	add fixed peers immediately if the dht has no peers	2019-01-31 15:39:20 -05:00
Jack Robison	e8a1953815	add fixed peers immediately if dht is skipped	2019-01-31 15:39:20 -05:00
Jack Robison	34776e7eb9	fix test	2019-01-31 15:39:20 -05:00
Jack Robison	a4419f562f	pylint	2019-01-31 15:39:20 -05:00
Jack Robison	e60e5b3919	fix updating content claim for a file ... -fix file name and download dir not being hex encoded during publish, add repair script	2019-01-31 15:39:20 -05:00
Jack Robison	3589cc9977	call update_reflected_stream upon reflector finishing	2019-01-31 15:39:20 -05:00
Jack Robison	2e978c00b2	prioritize peers to download from	2019-01-31 15:39:20 -05:00
Jack Robison	2335243de9	pylint	2019-01-31 15:39:20 -05:00
Jack Robison	6ad68eabd9	tests	2019-01-31 15:39:20 -05:00
Jack Robison	262b9a624b	update file_reflect	2019-01-31 15:39:20 -05:00
Jack Robison	05915c540a	set fully reflected if the server indicates it had all of the stream	2019-01-31 15:39:20 -05:00
Jack Robison	6aef6a80b7	refactor downloader ... split peer accumulation out, use Queues instead of locks	2019-01-31 15:39:20 -05:00
Jack Robison	16efe9ba95	add fixed_peer_delay setting	2019-01-31 15:39:20 -05:00
Jack Robison	fcda9500fe	add accumulate_peers to Node ... simplifies collecting peers during a download by giving a queue of hashes to search for and a queue or peers as they are found	2019-01-31 15:39:20 -05:00