LBRYCommunity/reflector.go
1
0
Fork 0
Code Issues 7 Pull requests Packages Projects Releases 10 Wiki Activity
375 commits 6 branches 14 tags 1.3 MiB
Commit graph

375 commits

This branch
This branch
All branches
Author SHA1 Message Date
Mark Beamer Jr 724ee47c8b add metric calls for other packages 2021-07-24 01:03:51 +02:00
Mark Beamer Jr caaec6fcb1 add guage metrics for go routines in reflector package 2021-07-24 01:03:51 +02:00
Mark Beamer Jr 15984b8fd9 add gops to reflector server 2021-07-24 01:03:51 +02:00
Niko Storni 2be913b077 request queue size param 2021-07-24 01:03:51 +02:00
Niko Storni 34c11b0a0e increase window size 2021-07-24 01:03:51 +02:00
Niko Storni 64acdc29c3 improve disk cleanup 
add index to is_stored
fix test
replace LRU cache
 2021-07-24 01:03:51 +02:00
Niko Storni 598773c90d fix mess with lbry.go 2021-07-24 01:03:47 +02:00
Niko Storni 766238fd7e add if this than that store 
switch to wasabi for uploads
 2021-07-24 01:03:47 +02:00
Niko Storni ac5242f173 add integrity check cmd 
throttle live integrity checks
bug fixes
 2021-07-24 01:03:47 +02:00
Mark Beamer Jr 215103cb33 use wait group not stopper 2021-07-24 01:03:47 +02:00
Mark Beamer Jr ed3622d0a6 Wait for request to be handled before returning 2021-07-24 01:03:47 +02:00
Mark Beamer Jr 848fce5afa Add request queue for blob cache 2021-07-24 01:03:47 +02:00
Niko Storni e37eeba0c9 check blobs when reading them 2021-07-24 01:03:47 +02:00
Niko Storni 7da49a4ccb upgrade quic-go 
add cache for blobs not found
 2021-07-24 01:03:47 +02:00
Niko Storni 7b02ace5e2 fix issues caused by beamer's renaming 2021-07-24 01:03:47 +02:00
Niko Storni 5fb67b32db run go mod tidy 2021-07-24 01:03:47 +02:00
Niko Storni a0c9ed2ace make it simpler 2021-07-24 01:03:47 +02:00
Niko Storni 998b082a06 remove panics 2021-07-24 01:03:47 +02:00
Niko Storni 36d4156e2a add tracing to blobs 2021-07-24 01:03:47 +02:00
Niko Storni 74925ebba2 optimize batch insertions 
reduce touch time to every 6 hours
 2021-07-24 01:03:47 +02:00
Alex Grintsvayg 6f95b3395f avoid heavy interpolateparams call 2021-07-24 01:03:47 +02:00
Alex Grintsvayg dff00e2317 fix long query 2021-07-24 01:03:47 +02:00
Alex Grintsvayg 9a5d9d7ff5 only touch blobs when you get them 2021-07-24 01:03:47 +02:00
Niko Storni 5794c57898 save uploaded blobs and work around the blocklist issue 2021-07-24 01:03:47 +02:00
Niko Storni 35c713a26e add cmd to populate db 
fix store init
try fixing unreasonable db bottleneck
 2021-07-24 01:03:47 +02:00
Alex Grintsvayg 6fb0620091 something like this 2021-07-24 01:03:38 +02:00
Niko Storni 03df751bc7 add PoC for litedb to avoid all the overhead 2021-07-24 01:03:16 +02:00
Niko Storni c902858958 address some review comments 2021-07-24 01:03:16 +02:00
Niko Storni 84fabdd5f4 add option to run with RO-CF only as upstream 
increase idle timeout to avoid errors downstream
add option to delete blobs from DB if storage doesn't have it (for future local tracking)
 2021-07-24 01:03:16 +02:00
Niko Storni f5cad15f84 upgrade quic 2021-07-24 01:03:10 +02:00
Niko Storni dd3d0ae42c update lfuda library 2021-07-24 01:03:10 +02:00
Niko Storni 0b565852b8 only store the blobs in the underlying storage if LFUDA accepted them 2021-07-24 01:03:10 +02:00
Niko Storni ff13d7b2f7 fix cache size mess 2021-07-24 01:03:10 +02:00
Niko Storni 7f5a89fa5a fix buffer cache running out of space 2021-07-24 01:03:10 +02:00
Niko Storni 704e15f8c1 use LFUDA store 
swap size to bytes
 2021-07-24 01:03:10 +02:00
Niko Storni 5eb1f13b54 add LFUDA store 
update quic
fix tests
 2021-07-24 01:03:10 +02:00
Alex Grintsvayg 176e05714e rename cahces 2021-07-24 01:03:10 +02:00
Niko Storni eefd84b02d add buffer cache for nvme drive 2021-07-24 01:03:10 +02:00
Niko Storni af2742c34f update quic 
don't wait for a blob to be written to disk before sending it downstream
don't wait for the disk store to be walked before starting everything up
 2021-07-24 01:03:10 +02:00
Alex Grintsvayg 2cf4acdb59
add 'send' command to efficiently send a file to reflector 2021-04-02 14:30:36 -04:00
Alex Grintsvayg 044e2fe5d7
Merge branch 'fix_mem_leak' 
* fix_mem_leak:
  dont over-allocate ram when reading blobs via QUIC
  dont overallocate ram when reading blobs from disk
 2020-11-27 16:18:44 -05:00
Alex Grintsvayg fb77bf621e
dont over-allocate ram when reading blobs via QUIC 2020-11-27 16:18:26 -05:00
Alex Grintsvayg e70b9af3e4
dont overallocate ram when reading blobs from disk 
ReadFile checks the file size and allocates a bit more space than we
expect we'll need. ReadAll uses Go's standard resizing algo, which
doubles the underlying array each time you hit the end. So ReadAll
ends up allocating 4MB for a full blob, while ReadFile allocates
slightly over 2MB.
 2020-11-27 16:18:26 -05:00
Alex Grintsvayg 5df05cf46f
ensure autodeploy is off 2020-11-23 12:41:09 -05:00
Alex Grintsvayg 7bddcf01b8
Merge branch 'smarter_caches' 
* smarter_caches:
  use speedwalk for faster file listing
  remove afero fs abstraction in prep for using speedwalk
  add faster file.Walk fn. meant for DiskStore.list()
  clarify Get() error requirement
  separate singleflight cache wrapper, component names for cache metrics
  apparently the normal check doesn't work
  split cloudfront into RO and RW stores
  add noop store that does nothing
  add test for DiskStore.list()
  add lru cache eviction metric
  rename the stores, add caching to reflector cmd
  separate disk and lru behavior
  LRU cache for disk store, abstract fs in disk store for testing
 2020-11-04 16:04:16 -05:00
Alex Grintsvayg 659a6e73cc
use speedwalk for faster file listing 2020-11-04 16:03:47 -05:00
Alex Grintsvayg aaae3ffa5b
remove afero fs abstraction in prep for using speedwalk 2020-11-04 16:03:47 -05:00
Alex Grintsvayg 131fed28d2
add faster file.Walk fn. meant for DiskStore.list() 2020-11-04 16:03:47 -05:00
Alex Grintsvayg 72571236ab
clarify Get() error requirement 2020-11-04 16:03:46 -05:00
Alex Grintsvayg 560e180e36
separate singleflight cache wrapper, component names for cache metrics 2020-11-04 16:03:46 -05:00