LBRYCommunity/reflector.go
1
0
Fork 0
Code Issues 7 Pull requests Projects Releases 10 Packages Wiki Activity
360 commits 6 branches 14 tags 1.3 MiB
Commit graph

360 commits

This branch
This branch
All branches
Author SHA1 Message Date
Niko Storni
ebb62d0a24 run go mod tidy 2021-03-29 19:44:27 +02:00
Niko Storni
8cb7389619 make it simpler 2021-02-23 15:23:46 +01:00
Niko Storni
7b49dd115b remove panics 2021-02-23 15:08:32 +01:00
Niko Storni
6291e33ee1 add tracing to blobs 2021-01-14 20:38:04 +01:00
Niko Storni
3e475e537b optimize batch insertions 
reduce touch time to every 6 hours
 2021-01-07 01:28:34 +01:00
Alex Grintsvayg
c4504631bc
avoid heavy interpolateparams call 2021-01-06 10:43:35 -05:00
Alex Grintsvayg
cc504e6c44
fix long query 2021-01-05 12:16:44 -05:00
Alex Grintsvayg
49714c02a6
only touch blobs when you get them 2021-01-05 11:36:33 -05:00
Niko Storni
b33651ae26 save uploaded blobs and work around the blocklist issue 2021-01-05 05:09:55 +01:00
Niko Storni
0d5004a83b add cmd to populate db 
fix store init
try fixing unreasonable db bottleneck
 2020-12-30 04:24:11 +01:00
Niko Storni
04f6859c74 Merge branch 'grin' into litedb 2020-12-24 23:13:31 +01:00
Alex Grintsvayg
3a1d9d3304
something like this 2020-12-23 17:08:13 -05:00
Niko Storni
03304312e8 add PoC for litedb to avoid all the overhead 2020-12-23 06:04:42 +01:00
Niko Storni
869030fc58 address some review comments 2020-12-22 21:19:48 +01:00
Niko Storni
def551cc89 add option to run with RO-CF only as upstream 
increase idle timeout to avoid errors downstream
add option to delete blobs from DB if storage doesn't have it (for future local tracking)
 2020-12-22 20:53:48 +01:00
Niko Storni
74b76a11e4 upgrade quic 2020-12-17 23:49:37 +01:00
Niko Storni
2c0df2ca8a
update lfuda library 2020-11-27 16:20:50 -05:00
Niko Storni
9fc96ac01b
only store the blobs in the underlying storage if LFUDA accepted them 2020-11-27 16:20:50 -05:00
Niko Storni
ff9b61b034
fix cache size mess 2020-11-27 16:20:49 -05:00
Niko Storni
7b80b2d4d2
fix buffer cache running out of space 2020-11-27 16:20:49 -05:00
Niko Storni
d45abdbdb0
use LFUDA store 
swap size to bytes
 2020-11-27 16:19:46 -05:00
Niko Storni
bc54601dde
add LFUDA store 
update quic
fix tests
 2020-11-27 16:19:46 -05:00
Alex Grintsvayg
bb41a84bb7
rename cahces 2020-11-27 16:19:45 -05:00
Niko Storni
a574fecf4e
add buffer cache for nvme drive 2020-11-27 16:19:45 -05:00
Niko Storni
9146c8b084
update quic 
don't wait for a blob to be written to disk before sending it downstream
don't wait for the disk store to be walked before starting everything up
 2020-11-27 16:19:45 -05:00
Alex Grintsvayg
044e2fe5d7
Merge branch 'fix_mem_leak' 
* fix_mem_leak:
  dont over-allocate ram when reading blobs via QUIC
  dont overallocate ram when reading blobs from disk
 2020-11-27 16:18:44 -05:00
Alex Grintsvayg
fb77bf621e
dont over-allocate ram when reading blobs via QUIC 2020-11-27 16:18:26 -05:00
Alex Grintsvayg
e70b9af3e4
dont overallocate ram when reading blobs from disk 
ReadFile checks the file size and allocates a bit more space than we
expect we'll need. ReadAll uses Go's standard resizing algo, which
doubles the underlying array each time you hit the end. So ReadAll
ends up allocating 4MB for a full blob, while ReadFile allocates
slightly over 2MB.
 2020-11-27 16:18:26 -05:00
Alex Grintsvayg
5df05cf46f
ensure autodeploy is off 2020-11-23 12:41:09 -05:00
Alex Grintsvayg
7bddcf01b8
Merge branch 'smarter_caches' 
* smarter_caches:
  use speedwalk for faster file listing
  remove afero fs abstraction in prep for using speedwalk
  add faster file.Walk fn. meant for DiskStore.list()
  clarify Get() error requirement
  separate singleflight cache wrapper, component names for cache metrics
  apparently the normal check doesn't work
  split cloudfront into RO and RW stores
  add noop store that does nothing
  add test for DiskStore.list()
  add lru cache eviction metric
  rename the stores, add caching to reflector cmd
  separate disk and lru behavior
  LRU cache for disk store, abstract fs in disk store for testing
 2020-11-04 16:04:16 -05:00
Alex Grintsvayg
659a6e73cc
use speedwalk for faster file listing 2020-11-04 16:03:47 -05:00
Alex Grintsvayg
aaae3ffa5b
remove afero fs abstraction in prep for using speedwalk 2020-11-04 16:03:47 -05:00
Alex Grintsvayg
131fed28d2
add faster file.Walk fn. meant for DiskStore.list() 2020-11-04 16:03:47 -05:00
Alex Grintsvayg
72571236ab
clarify Get() error requirement 2020-11-04 16:03:46 -05:00
Alex Grintsvayg
560e180e36
separate singleflight cache wrapper, component names for cache metrics 2020-11-04 16:03:46 -05:00
Alex Grintsvayg
070c378dfd
apparently the normal check doesn't work 2020-11-04 16:03:46 -05:00
Alex Grintsvayg
124d4065c2
split cloudfront into RO and RW stores 2020-11-04 16:03:45 -05:00
Alex Grintsvayg
f131c1f35b
add noop store that does nothing 2020-11-04 16:03:45 -05:00
Alex Grintsvayg
7a3225434e
add test for DiskStore.list() 2020-11-04 16:03:45 -05:00
Alex Grintsvayg
3608971f0b
add lru cache eviction metric 2020-11-04 16:03:44 -05:00
Alex Grintsvayg
c9fa04043c
rename the stores, add caching to reflector cmd 2020-11-04 16:03:44 -05:00
Alex Grintsvayg
c6b53792c8
separate disk and lru behavior 2020-11-04 16:03:44 -05:00
Alex Grintsvayg
69fa06420b
LRU cache for disk store, abstract fs in disk store for testing 2020-11-04 16:03:43 -05:00
Niko Storni
5cb1365903 increase idle timeout 2020-10-19 23:42:40 +02:00
Niko Storni
e430c2fd40 bump go version 2020-10-15 03:43:03 +02:00
Niko
66024716ac
Merge pull request #45 from lbryio/thundering_herd 
ensure only single origin Get request is in flight per hash. protects against thundering herd.
 2020-10-15 03:16:34 +02:00
Niko Storni
f043516a14 revert upload changes 2020-10-15 03:08:27 +02:00
Niko Storni
c3db95a6c1 add more metrics 
increase handshake timeout by 1 second
 2020-10-15 02:59:12 +02:00
Niko Storni
08c93d44fd update quic lib 2020-10-15 00:13:16 +02:00
Alex Grintsvayg
b02e80d472 ensure only single origin Get request is in flight per hash. protects against thundering herd. 2020-10-15 00:12:31 +02:00