Niko Storni
0177dd4ce0
fix query param
2022-07-29 05:40:50 +02:00
Niko Storni
5693529216
protect protected content
2022-07-29 04:59:15 +02:00
Andrey Beletsky
9c0554ef05
Do not use direction package on macos
2022-05-08 15:39:34 +07:00
Niko Storni
6dde793745
do not delete blobs for blocked content
2021-09-21 18:00:55 +02:00
Victor Shyba
90d6d29452
remove all hashing on the download path
2021-09-21 16:19:49 +02:00
Niko Storni
b2272fef3a
delete overflowing blobs from underlying cache
2021-08-18 19:36:47 +02:00
Niko Storni
b4913ecedf
cleanup
2021-07-24 01:03:51 +02:00
Niko Storni
63a574ec2f
unify caches
...
fix tests
2021-07-24 01:03:51 +02:00
Niko Storni
170dfef3a8
fix copy pasta mistake
2021-07-24 01:03:51 +02:00
Niko Storni
febfc51cb0
refactor refactor refactor
2021-07-24 01:03:51 +02:00
Andrey Beletsky
72be487262
Fix broken import
2021-07-24 01:03:51 +02:00
Andrey Beletsky
94e7d81bd3
Fix OpenFile call flags for macos
2021-07-24 01:03:51 +02:00
Niko Storni
2e101083e6
write blobs to tmp dir to avoid corruption
2021-07-24 01:03:51 +02:00
Niko Storni
63aacd8a69
use O_DIRECT to write to disk (fixes everything)
...
add queue back to serving blobs
improve a lot of things
upgrade modules
2021-07-24 01:03:51 +02:00
Niko Storni
c03ae6487d
fix unsafe dereference
2021-07-24 01:03:51 +02:00
Niko Storni
0c4f455f0c
add metrics
2021-07-24 01:03:51 +02:00
Niko Storni
975bfe7fac
upgrade singleflight
...
http store fix
2021-07-24 01:03:51 +02:00
Niko Storni
b075d948bb
remove locks causing deadlocks
2021-07-24 01:03:51 +02:00
Niko Storni
2651a64dbb
add http server/client
2021-07-24 01:03:51 +02:00
Mark Beamer Jr
fa7150cf2b
Add queue to prevent writing too many files at once.
2021-07-24 01:03:51 +02:00
Mark Beamer Jr
6c4db980c9
Add queue to prevent writing too many files at once.
2021-07-24 01:03:51 +02:00
Mark Beamer Jr
7adaa510fd
Add locks to disk store.
2021-07-24 01:03:51 +02:00
Niko Storni
64ed7304f6
add a lot of extra heavy debugging
2021-07-24 01:03:51 +02:00
Mark Beamer Jr
5aefaf061e
Add single flight for cache not just origin
2021-07-24 01:03:51 +02:00
Mark Beamer Jr
724ee47c8b
add metric calls for other packages
2021-07-24 01:03:51 +02:00
Niko Storni
64acdc29c3
improve disk cleanup
...
add index to is_stored
fix test
replace LRU cache
2021-07-24 01:03:51 +02:00
Niko Storni
766238fd7e
add if this than that store
...
switch to wasabi for uploads
2021-07-24 01:03:47 +02:00
Niko Storni
ac5242f173
add integrity check cmd
...
throttle live integrity checks
bug fixes
2021-07-24 01:03:47 +02:00
Niko Storni
e37eeba0c9
check blobs when reading them
2021-07-24 01:03:47 +02:00
Niko Storni
a0c9ed2ace
make it simpler
2021-07-24 01:03:47 +02:00
Niko Storni
998b082a06
remove panics
2021-07-24 01:03:47 +02:00
Niko Storni
36d4156e2a
add tracing to blobs
2021-07-24 01:03:47 +02:00
Alex Grintsvayg
9a5d9d7ff5
only touch blobs when you get them
2021-07-24 01:03:47 +02:00
Niko Storni
35c713a26e
add cmd to populate db
...
fix store init
try fixing unreasonable db bottleneck
2021-07-24 01:03:47 +02:00
Alex Grintsvayg
6fb0620091
something like this
2021-07-24 01:03:38 +02:00
Niko Storni
03df751bc7
add PoC for litedb to avoid all the overhead
2021-07-24 01:03:16 +02:00
Niko Storni
c902858958
address some review comments
2021-07-24 01:03:16 +02:00
Niko Storni
84fabdd5f4
add option to run with RO-CF only as upstream
...
increase idle timeout to avoid errors downstream
add option to delete blobs from DB if storage doesn't have it (for future local tracking)
2021-07-24 01:03:16 +02:00
Niko Storni
0b565852b8
only store the blobs in the underlying storage if LFUDA accepted them
2021-07-24 01:03:10 +02:00
Niko Storni
ff13d7b2f7
fix cache size mess
2021-07-24 01:03:10 +02:00
Niko Storni
5eb1f13b54
add LFUDA store
...
update quic
fix tests
2021-07-24 01:03:10 +02:00
Alex Grintsvayg
176e05714e
rename cahces
2021-07-24 01:03:10 +02:00
Niko Storni
af2742c34f
update quic
...
don't wait for a blob to be written to disk before sending it downstream
don't wait for the disk store to be walked before starting everything up
2021-07-24 01:03:10 +02:00
Alex Grintsvayg
e70b9af3e4
dont overallocate ram when reading blobs from disk
...
ReadFile checks the file size and allocates a bit more space than we
expect we'll need. ReadAll uses Go's standard resizing algo, which
doubles the underlying array each time you hit the end. So ReadAll
ends up allocating 4MB for a full blob, while ReadFile allocates
slightly over 2MB.
2020-11-27 16:18:26 -05:00
Alex Grintsvayg
659a6e73cc
use speedwalk for faster file listing
2020-11-04 16:03:47 -05:00
Alex Grintsvayg
aaae3ffa5b
remove afero fs abstraction in prep for using speedwalk
2020-11-04 16:03:47 -05:00
Alex Grintsvayg
131fed28d2
add faster file.Walk fn. meant for DiskStore.list()
2020-11-04 16:03:47 -05:00
Alex Grintsvayg
72571236ab
clarify Get() error requirement
2020-11-04 16:03:46 -05:00
Alex Grintsvayg
560e180e36
separate singleflight cache wrapper, component names for cache metrics
2020-11-04 16:03:46 -05:00
Alex Grintsvayg
124d4065c2
split cloudfront into RO and RW stores
2020-11-04 16:03:45 -05:00
