Alex Grintsvayg
6fb0620091
something like this
2021-07-24 01:03:38 +02:00
Niko Storni
03df751bc7
add PoC for litedb to avoid all the overhead
2021-07-24 01:03:16 +02:00
Niko Storni
c902858958
address some review comments
2021-07-24 01:03:16 +02:00
Niko Storni
84fabdd5f4
add option to run with RO-CF only as upstream
...
increase idle timeout to avoid errors downstream
add option to delete blobs from DB if storage doesn't have it (for future local tracking)
2021-07-24 01:03:16 +02:00
Niko Storni
f5cad15f84
upgrade quic
2021-07-24 01:03:10 +02:00
Niko Storni
dd3d0ae42c
update lfuda library
2021-07-24 01:03:10 +02:00
Niko Storni
0b565852b8
only store the blobs in the underlying storage if LFUDA accepted them
2021-07-24 01:03:10 +02:00
Niko Storni
ff13d7b2f7
fix cache size mess
2021-07-24 01:03:10 +02:00
Niko Storni
7f5a89fa5a
fix buffer cache running out of space
2021-07-24 01:03:10 +02:00
Niko Storni
704e15f8c1
use LFUDA store
...
swap size to bytes
2021-07-24 01:03:10 +02:00
Niko Storni
5eb1f13b54
add LFUDA store
...
update quic
fix tests
2021-07-24 01:03:10 +02:00
Alex Grintsvayg
176e05714e
rename cahces
2021-07-24 01:03:10 +02:00
Niko Storni
eefd84b02d
add buffer cache for nvme drive
2021-07-24 01:03:10 +02:00
Niko Storni
af2742c34f
update quic
...
don't wait for a blob to be written to disk before sending it downstream
don't wait for the disk store to be walked before starting everything up
2021-07-24 01:03:10 +02:00
Alex Grintsvayg
2cf4acdb59
add 'send' command to efficiently send a file to reflector
2021-04-02 14:30:36 -04:00
Alex Grintsvayg
044e2fe5d7
Merge branch 'fix_mem_leak'
...
* fix_mem_leak:
dont over-allocate ram when reading blobs via QUIC
dont overallocate ram when reading blobs from disk
2020-11-27 16:18:44 -05:00
Alex Grintsvayg
fb77bf621e
dont over-allocate ram when reading blobs via QUIC
2020-11-27 16:18:26 -05:00
Alex Grintsvayg
e70b9af3e4
dont overallocate ram when reading blobs from disk
...
ReadFile checks the file size and allocates a bit more space than we
expect we'll need. ReadAll uses Go's standard resizing algo, which
doubles the underlying array each time you hit the end. So ReadAll
ends up allocating 4MB for a full blob, while ReadFile allocates
slightly over 2MB.
2020-11-27 16:18:26 -05:00
Alex Grintsvayg
5df05cf46f
ensure autodeploy is off
2020-11-23 12:41:09 -05:00
Alex Grintsvayg
7bddcf01b8
Merge branch 'smarter_caches'
...
* smarter_caches:
use speedwalk for faster file listing
remove afero fs abstraction in prep for using speedwalk
add faster file.Walk fn. meant for DiskStore.list()
clarify Get() error requirement
separate singleflight cache wrapper, component names for cache metrics
apparently the normal check doesn't work
split cloudfront into RO and RW stores
add noop store that does nothing
add test for DiskStore.list()
add lru cache eviction metric
rename the stores, add caching to reflector cmd
separate disk and lru behavior
LRU cache for disk store, abstract fs in disk store for testing
2020-11-04 16:04:16 -05:00
Alex Grintsvayg
659a6e73cc
use speedwalk for faster file listing
2020-11-04 16:03:47 -05:00
Alex Grintsvayg
aaae3ffa5b
remove afero fs abstraction in prep for using speedwalk
2020-11-04 16:03:47 -05:00
Alex Grintsvayg
131fed28d2
add faster file.Walk fn. meant for DiskStore.list()
2020-11-04 16:03:47 -05:00
Alex Grintsvayg
72571236ab
clarify Get() error requirement
2020-11-04 16:03:46 -05:00
Alex Grintsvayg
560e180e36
separate singleflight cache wrapper, component names for cache metrics
2020-11-04 16:03:46 -05:00
Alex Grintsvayg
070c378dfd
apparently the normal check doesn't work
2020-11-04 16:03:46 -05:00
Alex Grintsvayg
124d4065c2
split cloudfront into RO and RW stores
2020-11-04 16:03:45 -05:00
Alex Grintsvayg
f131c1f35b
add noop store that does nothing
2020-11-04 16:03:45 -05:00
Alex Grintsvayg
7a3225434e
add test for DiskStore.list()
2020-11-04 16:03:45 -05:00
Alex Grintsvayg
3608971f0b
add lru cache eviction metric
2020-11-04 16:03:44 -05:00
Alex Grintsvayg
c9fa04043c
rename the stores, add caching to reflector cmd
2020-11-04 16:03:44 -05:00
Alex Grintsvayg
c6b53792c8
separate disk and lru behavior
2020-11-04 16:03:44 -05:00
Alex Grintsvayg
69fa06420b
LRU cache for disk store, abstract fs in disk store for testing
2020-11-04 16:03:43 -05:00
Niko Storni
5cb1365903
increase idle timeout
2020-10-19 23:42:40 +02:00
Niko Storni
e430c2fd40
bump go version
2020-10-15 03:43:03 +02:00
Niko
66024716ac
Merge pull request #45 from lbryio/thundering_herd
...
ensure only single origin Get request is in flight per hash. protects against thundering herd.
2020-10-15 03:16:34 +02:00
Niko Storni
f043516a14
revert upload changes
2020-10-15 03:08:27 +02:00
Niko Storni
c3db95a6c1
add more metrics
...
increase handshake timeout by 1 second
2020-10-15 02:59:12 +02:00
Niko Storni
08c93d44fd
update quic lib
2020-10-15 00:13:16 +02:00
Alex Grintsvayg
b02e80d472
ensure only single origin Get request is in flight per hash. protects against thundering herd.
2020-10-15 00:12:31 +02:00
Niko Storni
f1875454cc
store uploaded blobs
2020-10-15 00:12:18 +02:00
Alex Grintsvayg
a80e0f5b0d
change error msg
2020-10-06 10:17:50 -04:00
Alex Grintsvayg
2b3581a692
silence all those quic errors
2020-10-06 10:11:36 -04:00
Alex Grintsvayg
a0f78028cc
handle the case where last_accessed_at is null
2020-10-06 09:39:43 -04:00
Alex Grintsvayg
a084330055
drop mediainfo requirement since travis cant install it
2020-10-05 18:28:41 -04:00
Alex Grintsvayg
ea80ed6506
install mediainfo during build
2020-10-05 18:22:33 -04:00
Alex Grintsvayg
4b335ed692
bumping to 1.15 for the quic-go dependency
2020-10-05 18:15:02 -04:00
Alex Grintsvayg
6118dde36c
Merge branch 'track_access'
...
* track_access:
small changes for easier testing
track approximate access time for blobs
2020-10-05 18:10:49 -04:00
Alex Grintsvayg
8364d3fc54
small changes for easier testing
2020-10-05 18:08:54 -04:00
Alex Grintsvayg
de0ccd4da7
track approximate access time for blobs
2020-10-05 18:08:53 -04:00
