Mark Beamer Jr
45130499cd
Add single flight for cache not just origin
2021-05-20 19:05:48 -04:00
Mark Beamer Jr
4ecce75e23
add metric calls for other packages
2021-05-20 18:12:30 -04:00
Mark Beamer Jr
0152300d8d
add guage metrics for go routines in reflector package
2021-05-20 18:01:13 -04:00
Mark Beamer Jr
eafc62f2a6
add gops to reflector server
2021-05-20 17:21:35 -04:00
Niko Storni
50c077a9cb
request queue size param
2021-05-20 23:17:18 +02:00
Niko Storni
070938e12a
increase window size
2021-05-06 22:53:18 +02:00
Niko Storni
c4084eeb68
improve disk cleanup
...
add index to is_stored
fix test
replace LRU cache
2021-04-29 03:41:18 +02:00
Niko Storni
4392c97242
fix mess with lbry.go
2021-04-13 00:52:56 +02:00
Niko Storni
ec3aae33ba
add if this than that store
...
switch to wasabi for uploads
2021-04-12 23:05:50 +02:00
Niko Storni
dc95351cf3
add integrity check cmd
...
throttle live integrity checks
bug fixes
2021-04-07 04:46:18 +02:00
Mark Beamer Jr
25a7fac4f0
use wait group not stopper
2021-04-06 14:28:29 -04:00
Mark Beamer Jr
b97595311f
Wait for request to be handled before returning
2021-04-06 14:21:05 -04:00
Mark Beamer Jr
bd13836897
Add request queue for blob cache
2021-04-06 14:00:36 -04:00
Niko Storni
38b44218f2
check blobs when reading them
2021-04-05 23:34:45 +02:00
Niko Storni
90c36fbe24
upgrade quic-go
...
add cache for blobs not found
2021-03-31 04:53:27 +02:00
Niko Storni
3a441aed3a
fix issues caused by beamer's renaming
2021-03-29 19:56:18 +02:00
Niko Storni
ebb62d0a24
run go mod tidy
2021-03-29 19:44:27 +02:00
Niko Storni
8cb7389619
make it simpler
2021-02-23 15:23:46 +01:00
Niko Storni
7b49dd115b
remove panics
2021-02-23 15:08:32 +01:00
Niko Storni
6291e33ee1
add tracing to blobs
2021-01-14 20:38:04 +01:00
Niko Storni
3e475e537b
optimize batch insertions
...
reduce touch time to every 6 hours
2021-01-07 01:28:34 +01:00
Alex Grintsvayg
c4504631bc
avoid heavy interpolateparams call
2021-01-06 10:43:35 -05:00
Alex Grintsvayg
cc504e6c44
fix long query
2021-01-05 12:16:44 -05:00
Alex Grintsvayg
49714c02a6
only touch blobs when you get them
2021-01-05 11:36:33 -05:00
Niko Storni
b33651ae26
save uploaded blobs and work around the blocklist issue
2021-01-05 05:09:55 +01:00
Niko Storni
0d5004a83b
add cmd to populate db
...
fix store init
try fixing unreasonable db bottleneck
2020-12-30 04:24:11 +01:00
Niko Storni
04f6859c74
Merge branch 'grin' into litedb
2020-12-24 23:13:31 +01:00
Alex Grintsvayg
3a1d9d3304
something like this
2020-12-23 17:08:13 -05:00
Niko Storni
03304312e8
add PoC for litedb to avoid all the overhead
2020-12-23 06:04:42 +01:00
Niko Storni
869030fc58
address some review comments
2020-12-22 21:19:48 +01:00
Niko Storni
def551cc89
add option to run with RO-CF only as upstream
...
increase idle timeout to avoid errors downstream
add option to delete blobs from DB if storage doesn't have it (for future local tracking)
2020-12-22 20:53:48 +01:00
Niko Storni
74b76a11e4
upgrade quic
2020-12-17 23:49:37 +01:00
Niko Storni
2c0df2ca8a
update lfuda library
2020-11-27 16:20:50 -05:00
Niko Storni
9fc96ac01b
only store the blobs in the underlying storage if LFUDA accepted them
2020-11-27 16:20:50 -05:00
Niko Storni
ff9b61b034
fix cache size mess
2020-11-27 16:20:49 -05:00
Niko Storni
7b80b2d4d2
fix buffer cache running out of space
2020-11-27 16:20:49 -05:00
Niko Storni
d45abdbdb0
use LFUDA store
...
swap size to bytes
2020-11-27 16:19:46 -05:00
Niko Storni
bc54601dde
add LFUDA store
...
update quic
fix tests
2020-11-27 16:19:46 -05:00
Alex Grintsvayg
bb41a84bb7
rename cahces
2020-11-27 16:19:45 -05:00
Niko Storni
a574fecf4e
add buffer cache for nvme drive
2020-11-27 16:19:45 -05:00
Niko Storni
9146c8b084
update quic
...
don't wait for a blob to be written to disk before sending it downstream
don't wait for the disk store to be walked before starting everything up
2020-11-27 16:19:45 -05:00
Alex Grintsvayg
044e2fe5d7
Merge branch 'fix_mem_leak'
...
* fix_mem_leak:
dont over-allocate ram when reading blobs via QUIC
dont overallocate ram when reading blobs from disk
2020-11-27 16:18:44 -05:00
Alex Grintsvayg
fb77bf621e
dont over-allocate ram when reading blobs via QUIC
2020-11-27 16:18:26 -05:00
Alex Grintsvayg
e70b9af3e4
dont overallocate ram when reading blobs from disk
...
ReadFile checks the file size and allocates a bit more space than we
expect we'll need. ReadAll uses Go's standard resizing algo, which
doubles the underlying array each time you hit the end. So ReadAll
ends up allocating 4MB for a full blob, while ReadFile allocates
slightly over 2MB.
2020-11-27 16:18:26 -05:00
Alex Grintsvayg
5df05cf46f
ensure autodeploy is off
2020-11-23 12:41:09 -05:00
Alex Grintsvayg
7bddcf01b8
Merge branch 'smarter_caches'
...
* smarter_caches:
use speedwalk for faster file listing
remove afero fs abstraction in prep for using speedwalk
add faster file.Walk fn. meant for DiskStore.list()
clarify Get() error requirement
separate singleflight cache wrapper, component names for cache metrics
apparently the normal check doesn't work
split cloudfront into RO and RW stores
add noop store that does nothing
add test for DiskStore.list()
add lru cache eviction metric
rename the stores, add caching to reflector cmd
separate disk and lru behavior
LRU cache for disk store, abstract fs in disk store for testing
2020-11-04 16:04:16 -05:00
Alex Grintsvayg
659a6e73cc
use speedwalk for faster file listing
2020-11-04 16:03:47 -05:00
Alex Grintsvayg
aaae3ffa5b
remove afero fs abstraction in prep for using speedwalk
2020-11-04 16:03:47 -05:00
Alex Grintsvayg
131fed28d2
add faster file.Walk fn. meant for DiskStore.list()
2020-11-04 16:03:47 -05:00
Alex Grintsvayg
72571236ab
clarify Get() error requirement
2020-11-04 16:03:46 -05:00
