245 commits

Author	SHA1	Message	Date
Victor Shyba	f077e56cec	dht_crawler:only count latency during findNode	2022-09-07 12:03:11 -03:00
Victor Shyba	5e58c2f224	fix hosting metrics, improve logging	2022-09-07 12:03:11 -03:00
Victor Shyba	cc64789e96	dht_crawler: fix logging for missing ports	2022-09-07 12:03:11 -03:00
Victor Shyba	da2ffb000e	skip peers with bad ports without raising	2022-09-07 12:03:11 -03:00
Victor Shyba	df77392fe0	dht crawler:improve logging, metrics, make startup concurrent	2022-09-07 12:03:11 -03:00
Victor Shyba	9aa9ecdc0a	add arg for db path	2022-09-07 12:03:11 -03:00
Victor Shyba	43b45a939b	format logging	2022-09-07 12:03:11 -03:00
Victor Shyba	e2922a434f	add script to generate probe dataset	2022-09-07 12:03:11 -03:00
Victor Shyba	0d6125de0b	add sd_hash prober	2022-09-07 12:03:11 -03:00
Victor Shyba	13af7800c2	refactor script, remove dep	2022-09-07 12:03:11 -03:00
Victor Shyba	47a5d37d7c	change default metric port, add sqlalchemy to dockerfile	2022-09-07 12:03:11 -03:00
Victor Shyba	85ff487af5	dht_crawler: randomize port when idle	2022-09-07 12:03:11 -03:00
Victor Shyba	62eb9d5c75	dht_crawler: only count non zero connections	2022-09-07 12:03:11 -03:00
Victor Shyba	cfe5c8de8a	dht_crawler: serve prometheus metrics at 7070	2022-09-07 12:03:11 -03:00
Victor Shyba	0497698c5b	dht_crawler: skip ping if known node_id	2022-09-07 12:03:11 -03:00
Victor Shyba	508bdb8e94	dht_crawler: keep working set in memory, flush to db on intervals	2022-09-07 12:03:11 -03:00
Victor Shyba	cd42f0d726	dht_crawler: fix node id store	2022-09-07 12:03:11 -03:00
Victor Shyba	2706b66a92	dht_crawler: dont re-bootstrap. try known reachable even when they expire	2022-09-07 12:03:11 -03:00
Victor Shyba	29c2d5715d	dht_crawler: fix last_seen update	2022-09-07 12:03:11 -03:00
Victor Shyba	965389b759	dht_crawler: process older first, avoid discarding	2022-09-07 12:03:11 -03:00
Victor Shyba	174439f517	dht_crawler: cleanup, try not to reset key	2022-09-07 12:03:11 -03:00
Victor Shyba	baf422fc03	dht_crawler: extract refresh_limit, bump to 1h	2022-09-07 12:03:11 -03:00
Victor Shyba	61f7fbe230	dht_crawler: avoid reads	2022-09-07 12:03:11 -03:00
Victor Shyba	c6c27925b7	dht_crawler: flush/commit only when finished	2022-09-07 12:03:11 -03:00
Victor Shyba	be4c62cf32	check membership instead of one update per peer	2022-09-07 12:03:11 -03:00
Victor Shyba	443a1c32fa	dht_crawler: save a set of connections to avoid dupes, enable initial crawl	2022-09-07 12:03:11 -03:00
Victor Shyba	90c2a58470	dht_crawler: dont gather empty, fix crash	2022-09-07 12:03:11 -03:00
Victor Shyba	adc79ec404	dht_crawler: only warn for missing key if it replied	2022-09-07 12:03:11 -03:00
Victor Shyba	137d8ca4ac	dht_crawler: enable WAL	2022-09-07 12:03:11 -03:00
Victor Shyba	abf4d888af	dht_crawler: warn if we cannot get node id	2022-09-07 12:03:11 -03:00
Victor Shyba	6c350e57dd	dht_crawler: query recently checked as stats	2022-09-07 12:03:11 -03:00
Victor Shyba	fb7a93096e	only count checked unreachable	2022-09-07 12:03:11 -03:00
Victor Shyba	7ea88e7b31	dht_crawler: store data	2022-09-07 12:03:11 -03:00
Victor Shyba	2361e34541	dht crawler, initial version	2022-09-07 12:03:11 -03:00
Victor Shyba	318728aebd	add bootstrap flag to routing table	2022-08-11 20:38:42 -03:00
Lex Berezhny	ec82486e15	removed go hub dependency	2022-08-08 14:55:44 -04:00
Lex Berezhny	b8867cd18c	release.py script changed to use gh auth login for authentication	2022-04-10 23:28:16 -04:00
Jack Robison	9faf6e46ca	move lbry.wallet.server to new project called scribe ... switch from using lbrycrd to lbcd	2022-03-27 23:33:26 -04:00
Victor Shyba	8e6fa3490c	disable CSV endpoints by default	2022-01-12 12:39:23 -03:00
Victor Shyba	8a1a1a4000	remove estimation endpoints as that is done over prometheus metrics now	2022-01-12 12:39:23 -03:00
Victor Shyba	2d9130b4e0	prometheus: move blobs_stored and peers to SDK. add buckets_in_routing_table	2022-01-12 12:39:23 -03:00
Victor Shyba	470ee72462	add passive estimation to prometheus	2022-01-12 12:39:23 -03:00
Victor Shyba	add147b409	fix missing async	2022-01-12 12:39:23 -03:00
Victor Shyba	371df6e6c2	keep same node id between runs	2022-01-12 12:39:23 -03:00
Victor Shyba	7ed5fe8f66	add semaphore on active estimation to avoid abuse	2022-01-12 12:39:23 -03:00
Victor Shyba	a6ca7a6f38	same api across different estimation methods	2022-01-12 12:39:23 -03:00
Victor Shyba	1c857b8dd8	be explicit about ignoring params	2022-01-12 12:39:23 -03:00
Victor Shyba	87ff3f95ff	better endpoint names, small docs	2022-01-12 12:39:23 -03:00
Jack Robison	e7d9079389	improve script	2022-01-12 12:39:23 -03:00
Victor Shyba	9cdcff0e1e	first attempt at crawling	2022-01-12 12:39:23 -03:00