19 commits

Author	SHA1	Message	Date
Victor Shyba	508bdb8e94	dht_crawler: keep working set in memory, flush to db on intervals	2022-09-07 12:03:11 -03:00
Victor Shyba	cd42f0d726	dht_crawler: fix node id store	2022-09-07 12:03:11 -03:00
Victor Shyba	2706b66a92	dht_crawler: dont re-bootstrap. try known reachable even when they expire	2022-09-07 12:03:11 -03:00
Victor Shyba	29c2d5715d	dht_crawler: fix last_seen update	2022-09-07 12:03:11 -03:00
Victor Shyba	965389b759	dht_crawler: process older first, avoid discarding	2022-09-07 12:03:11 -03:00
Victor Shyba	174439f517	dht_crawler: cleanup, try not to reset key	2022-09-07 12:03:11 -03:00
Victor Shyba	baf422fc03	dht_crawler: extract refresh_limit, bump to 1h	2022-09-07 12:03:11 -03:00
Victor Shyba	61f7fbe230	dht_crawler: avoid reads	2022-09-07 12:03:11 -03:00
Victor Shyba	c6c27925b7	dht_crawler: flush/commit only when finished	2022-09-07 12:03:11 -03:00
Victor Shyba	be4c62cf32	check membership instead of one update per peer	2022-09-07 12:03:11 -03:00
Victor Shyba	443a1c32fa	dht_crawler: save a set of connections to avoid dupes, enable initial crawl	2022-09-07 12:03:11 -03:00
Victor Shyba	90c2a58470	dht_crawler: dont gather empty, fix crash	2022-09-07 12:03:11 -03:00
Victor Shyba	adc79ec404	dht_crawler: only warn for missing key if it replied	2022-09-07 12:03:11 -03:00
Victor Shyba	137d8ca4ac	dht_crawler: enable WAL	2022-09-07 12:03:11 -03:00
Victor Shyba	abf4d888af	dht_crawler: warn if we cannot get node id	2022-09-07 12:03:11 -03:00
Victor Shyba	6c350e57dd	dht_crawler: query recently checked as stats	2022-09-07 12:03:11 -03:00
Victor Shyba	fb7a93096e	only count checked unreachable	2022-09-07 12:03:11 -03:00
Victor Shyba	7ea88e7b31	dht_crawler: store data	2022-09-07 12:03:11 -03:00
Victor Shyba	2361e34541	dht crawler, initial version	2022-09-07 12:03:11 -03:00