Victor Shyba
|
f7455600cc
|
Merge pull request #3625 from lbryio/dht_crawler
Add script to collect DHT metrics
|
2022-09-07 12:56:41 -03:00 |
|
Victor Shyba
|
c7c2d6fe5a
|
collect connections reachability
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
c6c0228970
|
fix crawler startup query
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
8d9d2c76ae
|
routing table sizes as histogram
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
0b059a5445
|
use a histogram for latency, remove labels
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
ab67f417ee
|
dht_crawler: wait and retry during port switch
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
0e7a1aee0a
|
dht_crawler: clean in memory set for expired peers
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
d0497cf6b5
|
dht_crawler: skip saving connections for now
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
c38573d5de
|
dht_crawler: gather both loops, avoid task exceptions being hidden
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
f077e56cec
|
dht_crawler:only count latency during findNode
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
5e58c2f224
|
fix hosting metrics, improve logging
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
cc64789e96
|
dht_crawler: fix logging for missing ports
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
b5c390ca04
|
docker: add volume declaration
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
da2ffb000e
|
skip peers with bad ports without raising
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
df77392fe0
|
dht crawler:improve logging, metrics, make startup concurrent
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
9aa9ecdc0a
|
add arg for db path
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
43b45a939b
|
format logging
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
e2922a434f
|
add script to generate probe dataset
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
0d6125de0b
|
add sd_hash prober
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
13af7800c2
|
refactor script, remove dep
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
47a5d37d7c
|
change default metric port, add sqlalchemy to dockerfile
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
4a3a7e318d
|
update pip and setuptools on dht dockerfile
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
85ff487af5
|
dht_crawler: randomize port when idle
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
62eb9d5c75
|
dht_crawler: only count non zero connections
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
cfe5c8de8a
|
dht_crawler: serve prometheus metrics at 7070
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
0497698c5b
|
dht_crawler: skip ping if known node_id
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
508bdb8e94
|
dht_crawler: keep working set in memory, flush to db on intervals
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
cd42f0d726
|
dht_crawler: fix node id store
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
2706b66a92
|
dht_crawler: dont re-bootstrap. try known reachable even when they expire
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
29c2d5715d
|
dht_crawler: fix last_seen update
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
965389b759
|
dht_crawler: process older first, avoid discarding
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
174439f517
|
dht_crawler: cleanup, try not to reset key
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
baf422fc03
|
dht_crawler: extract refresh_limit, bump to 1h
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
61f7fbe230
|
dht_crawler: avoid reads
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
c6c27925b7
|
dht_crawler: flush/commit only when finished
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
be4c62cf32
|
check membership instead of one update per peer
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
443a1c32fa
|
dht_crawler: save a set of connections to avoid dupes, enable initial crawl
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
90c2a58470
|
dht_crawler: dont gather empty, fix crash
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
adc79ec404
|
dht_crawler: only warn for missing key if it replied
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
137d8ca4ac
|
dht_crawler: enable WAL
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
abf4d888af
|
dht_crawler: warn if we cannot get node id
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
6c350e57dd
|
dht_crawler: query recently checked as stats
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
fb7a93096e
|
only count checked unreachable
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
7ea88e7b31
|
dht_crawler: store data
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
2361e34541
|
dht crawler, initial version
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
be06378437
|
add method for getting the node_id from a known peer on peer manager
|
2022-09-07 12:03:11 -03:00 |
|
Victor Shyba
|
a334a93757
|
Merge pull request #3631 from lbryio/bootstrap_node
Add all peers when running as a bootstrap node
|
2022-08-29 11:07:29 -03:00 |
|
Victor Shyba
|
e3ee3892b2
|
better test name
|
2022-08-22 18:45:18 -03:00 |
|
Victor Shyba
|
d61accea1a
|
simplify bucket refresh loop
|
2022-08-11 21:14:56 -03:00 |
|
Victor Shyba
|
e887453aa5
|
remove unused last_accessed
|
2022-08-11 20:39:51 -03:00 |
|