Commit graph

95 commits

Author SHA1 Message Date
Niko Storni
e93c097fd9 upgrade dependencies
replace deprecated function calls
refactor build process
2023-03-09 18:41:41 +01:00
Niko Storni
0177dd4ce0 fix query param 2022-07-29 05:40:50 +02:00
Niko Storni
5693529216 protect protected content 2022-07-29 04:59:15 +02:00
Andrey Beletsky
9c0554ef05 Do not use direction package on macos 2022-05-08 15:39:34 +07:00
Niko Storni
6dde793745 do not delete blobs for blocked content 2021-09-21 18:00:55 +02:00
Victor Shyba
90d6d29452 remove all hashing on the download path 2021-09-21 16:19:49 +02:00
Niko Storni
b2272fef3a delete overflowing blobs from underlying cache 2021-08-18 19:36:47 +02:00
Niko Storni
b4913ecedf cleanup 2021-07-24 01:03:51 +02:00
Niko Storni
63a574ec2f unify caches
fix tests
2021-07-24 01:03:51 +02:00
Niko Storni
170dfef3a8 fix copy pasta mistake 2021-07-24 01:03:51 +02:00
Niko Storni
febfc51cb0 refactor refactor refactor 2021-07-24 01:03:51 +02:00
Andrey Beletsky
72be487262 Fix broken import 2021-07-24 01:03:51 +02:00
Andrey Beletsky
94e7d81bd3 Fix OpenFile call flags for macos 2021-07-24 01:03:51 +02:00
Niko Storni
2e101083e6 write blobs to tmp dir to avoid corruption 2021-07-24 01:03:51 +02:00
Niko Storni
63aacd8a69 use O_DIRECT to write to disk (fixes everything)
add queue back to serving blobs
improve a lot of things
upgrade modules
2021-07-24 01:03:51 +02:00
Niko Storni
c03ae6487d fix unsafe dereference 2021-07-24 01:03:51 +02:00
Niko Storni
0c4f455f0c add metrics 2021-07-24 01:03:51 +02:00
Niko Storni
975bfe7fac upgrade singleflight
http store fix
2021-07-24 01:03:51 +02:00
Niko Storni
b075d948bb remove locks causing deadlocks 2021-07-24 01:03:51 +02:00
Niko Storni
2651a64dbb add http server/client 2021-07-24 01:03:51 +02:00
Mark Beamer Jr
fa7150cf2b Add queue to prevent writing too many files at once. 2021-07-24 01:03:51 +02:00
Mark Beamer Jr
6c4db980c9 Add queue to prevent writing too many files at once. 2021-07-24 01:03:51 +02:00
Mark Beamer Jr
7adaa510fd Add locks to disk store. 2021-07-24 01:03:51 +02:00
Niko Storni
64ed7304f6 add a lot of extra heavy debugging 2021-07-24 01:03:51 +02:00
Mark Beamer Jr
5aefaf061e Add single flight for cache not just origin 2021-07-24 01:03:51 +02:00
Mark Beamer Jr
724ee47c8b add metric calls for other packages 2021-07-24 01:03:51 +02:00
Niko Storni
64acdc29c3 improve disk cleanup
add index to is_stored
fix test
replace LRU cache
2021-07-24 01:03:51 +02:00
Niko Storni
766238fd7e add if this than that store
switch to wasabi for uploads
2021-07-24 01:03:47 +02:00
Niko Storni
ac5242f173 add integrity check cmd
throttle live integrity checks
bug fixes
2021-07-24 01:03:47 +02:00
Niko Storni
e37eeba0c9 check blobs when reading them 2021-07-24 01:03:47 +02:00
Niko Storni
a0c9ed2ace make it simpler 2021-07-24 01:03:47 +02:00
Niko Storni
998b082a06 remove panics 2021-07-24 01:03:47 +02:00
Niko Storni
36d4156e2a add tracing to blobs 2021-07-24 01:03:47 +02:00
Alex Grintsvayg
9a5d9d7ff5 only touch blobs when you get them 2021-07-24 01:03:47 +02:00
Niko Storni
35c713a26e add cmd to populate db
fix store init
try fixing unreasonable db bottleneck
2021-07-24 01:03:47 +02:00
Alex Grintsvayg
6fb0620091 something like this 2021-07-24 01:03:38 +02:00
Niko Storni
03df751bc7 add PoC for litedb to avoid all the overhead 2021-07-24 01:03:16 +02:00
Niko Storni
c902858958 address some review comments 2021-07-24 01:03:16 +02:00
Niko Storni
84fabdd5f4 add option to run with RO-CF only as upstream
increase idle timeout to avoid errors downstream
add option to delete blobs from DB if storage doesn't have it (for future local tracking)
2021-07-24 01:03:16 +02:00
Niko Storni
0b565852b8 only store the blobs in the underlying storage if LFUDA accepted them 2021-07-24 01:03:10 +02:00
Niko Storni
ff13d7b2f7 fix cache size mess 2021-07-24 01:03:10 +02:00
Niko Storni
5eb1f13b54 add LFUDA store
update quic
fix tests
2021-07-24 01:03:10 +02:00
Alex Grintsvayg
176e05714e rename cahces 2021-07-24 01:03:10 +02:00
Niko Storni
af2742c34f update quic
don't wait for a blob to be written to disk before sending it downstream
don't wait for the disk store to be walked before starting everything up
2021-07-24 01:03:10 +02:00
Alex Grintsvayg
e70b9af3e4
dont overallocate ram when reading blobs from disk
ReadFile checks the file size and allocates a bit more space than we
expect we'll need. ReadAll uses Go's standard resizing algo, which
doubles the underlying array each time you hit the end. So ReadAll
ends up allocating 4MB for a full blob, while ReadFile allocates
slightly over 2MB.
2020-11-27 16:18:26 -05:00
Alex Grintsvayg
659a6e73cc
use speedwalk for faster file listing 2020-11-04 16:03:47 -05:00
Alex Grintsvayg
aaae3ffa5b
remove afero fs abstraction in prep for using speedwalk 2020-11-04 16:03:47 -05:00
Alex Grintsvayg
131fed28d2
add faster file.Walk fn. meant for DiskStore.list() 2020-11-04 16:03:47 -05:00
Alex Grintsvayg
72571236ab
clarify Get() error requirement 2020-11-04 16:03:46 -05:00
Alex Grintsvayg
560e180e36
separate singleflight cache wrapper, component names for cache metrics 2020-11-04 16:03:46 -05:00