Commit graph

9171 commits

Author SHA1 Message Date
Victor Shyba
0b059a5445 use a histogram for latency, remove labels 2022-09-07 12:03:11 -03:00
Victor Shyba
ab67f417ee dht_crawler: wait and retry during port switch 2022-09-07 12:03:11 -03:00
Victor Shyba
0e7a1aee0a dht_crawler: clean in memory set for expired peers 2022-09-07 12:03:11 -03:00
Victor Shyba
d0497cf6b5 dht_crawler: skip saving connections for now 2022-09-07 12:03:11 -03:00
Victor Shyba
c38573d5de dht_crawler: gather both loops, avoid task exceptions being hidden 2022-09-07 12:03:11 -03:00
Victor Shyba
f077e56cec dht_crawler:only count latency during findNode 2022-09-07 12:03:11 -03:00
Victor Shyba
5e58c2f224 fix hosting metrics, improve logging 2022-09-07 12:03:11 -03:00
Victor Shyba
cc64789e96 dht_crawler: fix logging for missing ports 2022-09-07 12:03:11 -03:00
Victor Shyba
b5c390ca04 docker: add volume declaration 2022-09-07 12:03:11 -03:00
Victor Shyba
da2ffb000e skip peers with bad ports without raising 2022-09-07 12:03:11 -03:00
Victor Shyba
df77392fe0 dht crawler:improve logging, metrics, make startup concurrent 2022-09-07 12:03:11 -03:00
Victor Shyba
9aa9ecdc0a add arg for db path 2022-09-07 12:03:11 -03:00
Victor Shyba
43b45a939b format logging 2022-09-07 12:03:11 -03:00
Victor Shyba
e2922a434f add script to generate probe dataset 2022-09-07 12:03:11 -03:00
Victor Shyba
0d6125de0b add sd_hash prober 2022-09-07 12:03:11 -03:00
Victor Shyba
13af7800c2 refactor script, remove dep 2022-09-07 12:03:11 -03:00
Victor Shyba
47a5d37d7c change default metric port, add sqlalchemy to dockerfile 2022-09-07 12:03:11 -03:00
Victor Shyba
4a3a7e318d update pip and setuptools on dht dockerfile 2022-09-07 12:03:11 -03:00
Victor Shyba
85ff487af5 dht_crawler: randomize port when idle 2022-09-07 12:03:11 -03:00
Victor Shyba
62eb9d5c75 dht_crawler: only count non zero connections 2022-09-07 12:03:11 -03:00
Victor Shyba
cfe5c8de8a dht_crawler: serve prometheus metrics at 7070 2022-09-07 12:03:11 -03:00
Victor Shyba
0497698c5b dht_crawler: skip ping if known node_id 2022-09-07 12:03:11 -03:00
Victor Shyba
508bdb8e94 dht_crawler: keep working set in memory, flush to db on intervals 2022-09-07 12:03:11 -03:00
Victor Shyba
cd42f0d726 dht_crawler: fix node id store 2022-09-07 12:03:11 -03:00
Victor Shyba
2706b66a92 dht_crawler: dont re-bootstrap. try known reachable even when they expire 2022-09-07 12:03:11 -03:00
Victor Shyba
29c2d5715d dht_crawler: fix last_seen update 2022-09-07 12:03:11 -03:00
Victor Shyba
965389b759 dht_crawler: process older first, avoid discarding 2022-09-07 12:03:11 -03:00
Victor Shyba
174439f517 dht_crawler: cleanup, try not to reset key 2022-09-07 12:03:11 -03:00
Victor Shyba
baf422fc03 dht_crawler: extract refresh_limit, bump to 1h 2022-09-07 12:03:11 -03:00
Victor Shyba
61f7fbe230 dht_crawler: avoid reads 2022-09-07 12:03:11 -03:00
Victor Shyba
c6c27925b7 dht_crawler: flush/commit only when finished 2022-09-07 12:03:11 -03:00
Victor Shyba
be4c62cf32 check membership instead of one update per peer 2022-09-07 12:03:11 -03:00
Victor Shyba
443a1c32fa dht_crawler: save a set of connections to avoid dupes, enable initial crawl 2022-09-07 12:03:11 -03:00
Victor Shyba
90c2a58470 dht_crawler: dont gather empty, fix crash 2022-09-07 12:03:11 -03:00
Victor Shyba
adc79ec404 dht_crawler: only warn for missing key if it replied 2022-09-07 12:03:11 -03:00
Victor Shyba
137d8ca4ac dht_crawler: enable WAL 2022-09-07 12:03:11 -03:00
Victor Shyba
abf4d888af dht_crawler: warn if we cannot get node id 2022-09-07 12:03:11 -03:00
Victor Shyba
6c350e57dd dht_crawler: query recently checked as stats 2022-09-07 12:03:11 -03:00
Victor Shyba
fb7a93096e only count checked unreachable 2022-09-07 12:03:11 -03:00
Victor Shyba
7ea88e7b31 dht_crawler: store data 2022-09-07 12:03:11 -03:00
Victor Shyba
2361e34541 dht crawler, initial version 2022-09-07 12:03:11 -03:00
Victor Shyba
be06378437 add method for getting the node_id from a known peer on peer manager 2022-09-07 12:03:11 -03:00
Victor Shyba
a334a93757
Merge pull request #3631 from lbryio/bootstrap_node
Add all peers when running as a bootstrap node
2022-08-29 11:07:29 -03:00
Victor Shyba
e3ee3892b2 better test name 2022-08-22 18:45:18 -03:00
Victor Shyba
d61accea1a simplify bucket refresh loop 2022-08-11 21:14:56 -03:00
Victor Shyba
e887453aa5 remove unused last_accessed 2022-08-11 20:39:51 -03:00
Victor Shyba
c3e4f0b988 add 'is_bootstrap_node' conf 2022-08-11 20:38:42 -03:00
Victor Shyba
318728aebd add bootstrap flag to routing table 2022-08-11 20:38:42 -03:00
Victor Shyba
d8c1aaebc2 routing table: mark private methods 2022-08-11 20:38:42 -03:00
Victor Shyba
d7b65c15d2 return none instead of raising 2022-08-11 20:38:42 -03:00