9130 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Jonathan Moody	45a255e7a2	Reuse generate() logic to wait on hub ... instead of half-baked reorg() logic.	2022-09-08 18:13:26 -04:00
Jonathan Moody	d2738c2e72	Add more failure message details for debugging.	2022-09-08 18:13:26 -04:00
Jonathan Moody	a7c7ab7f7b	Correct the terminal height we wait for in generate().	2022-09-08 18:13:26 -04:00
Jonathan Moody	988f288715	Lint fix for _es_height checks.	2022-09-08 18:13:26 -04:00
Jonathan Moody	38e9b5b432	Wait for _es_height in addition to db_height.	2022-09-08 18:13:26 -04:00
Victor Shyba	f7455600cc	Merge pull request #3625 from lbryio/dht_crawler ... Add script to collect DHT metrics	2022-09-07 12:56:41 -03:00
Victor Shyba	c7c2d6fe5a	collect connections reachability	2022-09-07 12:03:11 -03:00
Victor Shyba	c6c0228970	fix crawler startup query	2022-09-07 12:03:11 -03:00
Victor Shyba	8d9d2c76ae	routing table sizes as histogram	2022-09-07 12:03:11 -03:00
Victor Shyba	0b059a5445	use a histogram for latency, remove labels	2022-09-07 12:03:11 -03:00
Victor Shyba	ab67f417ee	dht_crawler: wait and retry during port switch	2022-09-07 12:03:11 -03:00
Victor Shyba	0e7a1aee0a	dht_crawler: clean in memory set for expired peers	2022-09-07 12:03:11 -03:00
Victor Shyba	d0497cf6b5	dht_crawler: skip saving connections for now	2022-09-07 12:03:11 -03:00
Victor Shyba	c38573d5de	dht_crawler: gather both loops, avoid task exceptions being hidden	2022-09-07 12:03:11 -03:00
Victor Shyba	f077e56cec	dht_crawler:only count latency during findNode	2022-09-07 12:03:11 -03:00
Victor Shyba	5e58c2f224	fix hosting metrics, improve logging	2022-09-07 12:03:11 -03:00
Victor Shyba	cc64789e96	dht_crawler: fix logging for missing ports	2022-09-07 12:03:11 -03:00
Victor Shyba	b5c390ca04	docker: add volume declaration	2022-09-07 12:03:11 -03:00
Victor Shyba	da2ffb000e	skip peers with bad ports without raising	2022-09-07 12:03:11 -03:00
Victor Shyba	df77392fe0	dht crawler:improve logging, metrics, make startup concurrent	2022-09-07 12:03:11 -03:00
Victor Shyba	9aa9ecdc0a	add arg for db path	2022-09-07 12:03:11 -03:00
Victor Shyba	43b45a939b	format logging	2022-09-07 12:03:11 -03:00
Victor Shyba	e2922a434f	add script to generate probe dataset	2022-09-07 12:03:11 -03:00
Victor Shyba	0d6125de0b	add sd_hash prober	2022-09-07 12:03:11 -03:00
Victor Shyba	13af7800c2	refactor script, remove dep	2022-09-07 12:03:11 -03:00
Victor Shyba	47a5d37d7c	change default metric port, add sqlalchemy to dockerfile	2022-09-07 12:03:11 -03:00
Victor Shyba	4a3a7e318d	update pip and setuptools on dht dockerfile	2022-09-07 12:03:11 -03:00
Victor Shyba	85ff487af5	dht_crawler: randomize port when idle	2022-09-07 12:03:11 -03:00
Victor Shyba	62eb9d5c75	dht_crawler: only count non zero connections	2022-09-07 12:03:11 -03:00
Victor Shyba	cfe5c8de8a	dht_crawler: serve prometheus metrics at 7070	2022-09-07 12:03:11 -03:00
Victor Shyba	0497698c5b	dht_crawler: skip ping if known node_id	2022-09-07 12:03:11 -03:00
Victor Shyba	508bdb8e94	dht_crawler: keep working set in memory, flush to db on intervals	2022-09-07 12:03:11 -03:00
Victor Shyba	cd42f0d726	dht_crawler: fix node id store	2022-09-07 12:03:11 -03:00
Victor Shyba	2706b66a92	dht_crawler: dont re-bootstrap. try known reachable even when they expire	2022-09-07 12:03:11 -03:00
Victor Shyba	29c2d5715d	dht_crawler: fix last_seen update	2022-09-07 12:03:11 -03:00
Victor Shyba	965389b759	dht_crawler: process older first, avoid discarding	2022-09-07 12:03:11 -03:00
Victor Shyba	174439f517	dht_crawler: cleanup, try not to reset key	2022-09-07 12:03:11 -03:00
Victor Shyba	baf422fc03	dht_crawler: extract refresh_limit, bump to 1h	2022-09-07 12:03:11 -03:00
Victor Shyba	61f7fbe230	dht_crawler: avoid reads	2022-09-07 12:03:11 -03:00
Victor Shyba	c6c27925b7	dht_crawler: flush/commit only when finished	2022-09-07 12:03:11 -03:00
Victor Shyba	be4c62cf32	check membership instead of one update per peer	2022-09-07 12:03:11 -03:00
Victor Shyba	443a1c32fa	dht_crawler: save a set of connections to avoid dupes, enable initial crawl	2022-09-07 12:03:11 -03:00
Victor Shyba	90c2a58470	dht_crawler: dont gather empty, fix crash	2022-09-07 12:03:11 -03:00
Victor Shyba	adc79ec404	dht_crawler: only warn for missing key if it replied	2022-09-07 12:03:11 -03:00
Victor Shyba	137d8ca4ac	dht_crawler: enable WAL	2022-09-07 12:03:11 -03:00
Victor Shyba	abf4d888af	dht_crawler: warn if we cannot get node id	2022-09-07 12:03:11 -03:00
Victor Shyba	6c350e57dd	dht_crawler: query recently checked as stats	2022-09-07 12:03:11 -03:00
Victor Shyba	fb7a93096e	only count checked unreachable	2022-09-07 12:03:11 -03:00
Victor Shyba	7ea88e7b31	dht_crawler: store data	2022-09-07 12:03:11 -03:00
Victor Shyba	2361e34541	dht crawler, initial version	2022-09-07 12:03:11 -03:00