Niko Storni
dc95351cf3
add integrity check cmd
...
throttle live integrity checks
bug fixes
2021-04-07 04:46:18 +02:00
Niko Storni
38b44218f2
check blobs when reading them
2021-04-05 23:34:45 +02:00
Niko Storni
8cb7389619
make it simpler
2021-02-23 15:23:46 +01:00
Niko Storni
7b49dd115b
remove panics
2021-02-23 15:08:32 +01:00
Niko Storni
6291e33ee1
add tracing to blobs
2021-01-14 20:38:04 +01:00
Alex Grintsvayg
49714c02a6
only touch blobs when you get them
2021-01-05 11:36:33 -05:00
Niko Storni
0d5004a83b
add cmd to populate db
...
fix store init
try fixing unreasonable db bottleneck
2020-12-30 04:24:11 +01:00
Niko Storni
04f6859c74
Merge branch 'grin' into litedb
2020-12-24 23:13:31 +01:00
Alex Grintsvayg
3a1d9d3304
something like this
2020-12-23 17:08:13 -05:00
Niko Storni
03304312e8
add PoC for litedb to avoid all the overhead
2020-12-23 06:04:42 +01:00
Niko Storni
869030fc58
address some review comments
2020-12-22 21:19:48 +01:00
Niko Storni
def551cc89
add option to run with RO-CF only as upstream
...
increase idle timeout to avoid errors downstream
add option to delete blobs from DB if storage doesn't have it (for future local tracking)
2020-12-22 20:53:48 +01:00
Niko Storni
9fc96ac01b
only store the blobs in the underlying storage if LFUDA accepted them
2020-11-27 16:20:50 -05:00
Niko Storni
ff9b61b034
fix cache size mess
2020-11-27 16:20:49 -05:00
Niko Storni
bc54601dde
add LFUDA store
...
update quic
fix tests
2020-11-27 16:19:46 -05:00
Alex Grintsvayg
bb41a84bb7
rename cahces
2020-11-27 16:19:45 -05:00
Niko Storni
9146c8b084
update quic
...
don't wait for a blob to be written to disk before sending it downstream
don't wait for the disk store to be walked before starting everything up
2020-11-27 16:19:45 -05:00
Alex Grintsvayg
e70b9af3e4
dont overallocate ram when reading blobs from disk
...
ReadFile checks the file size and allocates a bit more space than we
expect we'll need. ReadAll uses Go's standard resizing algo, which
doubles the underlying array each time you hit the end. So ReadAll
ends up allocating 4MB for a full blob, while ReadFile allocates
slightly over 2MB.
2020-11-27 16:18:26 -05:00
Alex Grintsvayg
659a6e73cc
use speedwalk for faster file listing
2020-11-04 16:03:47 -05:00
Alex Grintsvayg
aaae3ffa5b
remove afero fs abstraction in prep for using speedwalk
2020-11-04 16:03:47 -05:00
Alex Grintsvayg
131fed28d2
add faster file.Walk fn. meant for DiskStore.list()
2020-11-04 16:03:47 -05:00
Alex Grintsvayg
72571236ab
clarify Get() error requirement
2020-11-04 16:03:46 -05:00
Alex Grintsvayg
560e180e36
separate singleflight cache wrapper, component names for cache metrics
2020-11-04 16:03:46 -05:00
Alex Grintsvayg
124d4065c2
split cloudfront into RO and RW stores
2020-11-04 16:03:45 -05:00
Alex Grintsvayg
f131c1f35b
add noop store that does nothing
2020-11-04 16:03:45 -05:00
Alex Grintsvayg
7a3225434e
add test for DiskStore.list()
2020-11-04 16:03:45 -05:00
Alex Grintsvayg
3608971f0b
add lru cache eviction metric
2020-11-04 16:03:44 -05:00
Alex Grintsvayg
c9fa04043c
rename the stores, add caching to reflector cmd
2020-11-04 16:03:44 -05:00
Alex Grintsvayg
c6b53792c8
separate disk and lru behavior
2020-11-04 16:03:44 -05:00
Alex Grintsvayg
69fa06420b
LRU cache for disk store, abstract fs in disk store for testing
2020-11-04 16:03:43 -05:00
Niko Storni
c3db95a6c1
add more metrics
...
increase handshake timeout by 1 second
2020-10-15 02:59:12 +02:00
Alex Grintsvayg
b02e80d472
ensure only single origin Get request is in flight per hash. protects against thundering herd.
2020-10-15 00:12:31 +02:00
Alex Grintsvayg
de0ccd4da7
track approximate access time for blobs
2020-10-05 18:08:53 -04:00
Niko Storni
fc5f5ff7d3
add cloudfront support
2020-09-09 00:18:07 +02:00
Niko Storni
47f28002ff
close unclosed handle
...
reduce idle timeout for http3
update QUIC library
2020-07-10 15:19:57 +02:00
Niko Storni
694bda105c
add metrics
2020-07-09 15:02:32 +02:00
Niko Storni
5c91051b78
disable disk cleanup routine
2020-07-09 15:02:32 +02:00
Niko Storni
3ffe7a10c7
add other reflector store
...
add flags
improve disk cleanup
2020-07-09 15:02:32 +02:00
Andrey Beletsky
1bf3cb81b3
Use ModTime on systems that don't provide Atim file stat field
2020-07-09 15:02:32 +02:00
Niko Storni
4a5a148843
implement disk cleanup
2020-02-27 14:53:33 -05:00
Alex Grintsvayg
d291c063ec
add traces to unmarshall errors
2020-02-25 15:49:51 -05:00
Mark Beamer Jr
86a553b876
upgrade to lbry.go v2.4.0
2019-11-13 19:11:35 -05:00
Alex Grintsvayg
661c20a21d
make db-backed store more generic (not specific to s3)
2019-10-03 16:58:17 -04:00
Alex Grintsvayg
69f1e0f4ca
make MemoryStore consistent with the New...() pattern
2019-10-03 16:49:28 -04:00
Alex Grintsvayg
2ca83139df
use stream.Blob for BlobStore interface
2019-10-03 16:34:57 -04:00
Alex Grintsvayg
0af6d65d40
rename FileStore -> DiskStore
2019-10-03 16:24:59 -04:00
Alex Grintsvayg
24f885e268
store blobs on disk in prefix-based subdirectories to avoid too many files in one dir
2019-10-03 16:12:49 -04:00
Alex Grintsvayg
36ee7e8d1f
add caching blob store
2019-10-03 13:36:35 -04:00
Alex Grintsvayg
c7643fb5da
check blocklist first, its cheaper
2019-07-02 09:21:26 -04:00
Alex Grintsvayg
05e0a506b4
better debug output
2019-01-29 14:42:45 -05:00
