Commit graph

145 commits

Author SHA1 Message Date
Niko Storni 86f3e62aa8 fix a panic error
update gin-go
2021-08-05 17:47:20 +02:00
Niko Storni b4913ecedf cleanup 2021-07-24 01:03:51 +02:00
Niko Storni 63a574ec2f unify caches
fix tests
2021-07-24 01:03:51 +02:00
Niko Storni b8af3408e0 move server packages 2021-07-24 01:03:51 +02:00
Niko Storni 2b458a6bd0 fix params
more cleanups
2021-07-24 01:03:51 +02:00
Niko Storni febfc51cb0 refactor refactor refactor 2021-07-24 01:03:51 +02:00
Niko Storni 63aacd8a69 use O_DIRECT to write to disk (fixes everything)
add queue back to serving blobs
improve a lot of things
upgrade modules
2021-07-24 01:03:51 +02:00
Niko Storni 2651a64dbb add http server/client 2021-07-24 01:03:51 +02:00
Niko Storni 2be913b077 request queue size param 2021-07-24 01:03:51 +02:00
Niko Storni 64acdc29c3 improve disk cleanup
add index to is_stored
fix test
replace LRU cache
2021-07-24 01:03:51 +02:00
Niko Storni 766238fd7e add if this than that store
switch to wasabi for uploads
2021-07-24 01:03:47 +02:00
Niko Storni ac5242f173 add integrity check cmd
throttle live integrity checks
bug fixes
2021-07-24 01:03:47 +02:00
Niko Storni 36d4156e2a add tracing to blobs 2021-07-24 01:03:47 +02:00
Alex Grintsvayg 6f95b3395f avoid heavy interpolateparams call 2021-07-24 01:03:47 +02:00
Niko Storni 5794c57898 save uploaded blobs and work around the blocklist issue 2021-07-24 01:03:47 +02:00
Niko Storni 35c713a26e add cmd to populate db
fix store init
try fixing unreasonable db bottleneck
2021-07-24 01:03:47 +02:00
Alex Grintsvayg 6fb0620091 something like this 2021-07-24 01:03:38 +02:00
Niko Storni 03df751bc7 add PoC for litedb to avoid all the overhead 2021-07-24 01:03:16 +02:00
Niko Storni 84fabdd5f4 add option to run with RO-CF only as upstream
increase idle timeout to avoid errors downstream
add option to delete blobs from DB if storage doesn't have it (for future local tracking)
2021-07-24 01:03:16 +02:00
Niko Storni ff13d7b2f7 fix cache size mess 2021-07-24 01:03:10 +02:00
Niko Storni 7f5a89fa5a fix buffer cache running out of space 2021-07-24 01:03:10 +02:00
Niko Storni 704e15f8c1 use LFUDA store
swap size to bytes
2021-07-24 01:03:10 +02:00
Alex Grintsvayg 176e05714e rename cahces 2021-07-24 01:03:10 +02:00
Niko Storni eefd84b02d add buffer cache for nvme drive 2021-07-24 01:03:10 +02:00
Alex Grintsvayg 2cf4acdb59
add 'send' command to efficiently send a file to reflector 2021-04-02 14:30:36 -04:00
Alex Grintsvayg 560e180e36
separate singleflight cache wrapper, component names for cache metrics 2020-11-04 16:03:46 -05:00
Alex Grintsvayg 124d4065c2
split cloudfront into RO and RW stores 2020-11-04 16:03:45 -05:00
Alex Grintsvayg 3608971f0b
add lru cache eviction metric 2020-11-04 16:03:44 -05:00
Alex Grintsvayg c9fa04043c
rename the stores, add caching to reflector cmd 2020-11-04 16:03:44 -05:00
Alex Grintsvayg 69fa06420b
LRU cache for disk store, abstract fs in disk store for testing 2020-11-04 16:03:43 -05:00
Niko Storni f043516a14 revert upload changes 2020-10-15 03:08:27 +02:00
Niko Storni f1875454cc store uploaded blobs 2020-10-15 00:12:18 +02:00
Alex Grintsvayg 8364d3fc54
small changes for easier testing 2020-10-05 18:08:54 -04:00
Alex Grintsvayg de0ccd4da7
track approximate access time for blobs 2020-10-05 18:08:53 -04:00
Alex Grintsvayg 5658fe4607
add publish command 2020-10-05 16:30:57 -04:00
Niko Storni fc5f5ff7d3 add cloudfront support 2020-09-09 00:18:07 +02:00
Niko Storni e8b98bc862 update dependencies 2020-09-01 20:15:34 +02:00
Niko Storni 09c7718f30 refactor code 2020-07-09 15:02:32 +02:00
Niko Storni 8a5f57b14f remove QUIC protocol
add HTTP3/QUIC protocol
2020-07-09 15:02:32 +02:00
Niko Storni 90997b9918 use new connections for each blob 2020-07-09 15:02:32 +02:00
Niko Storni e0da2674a1 use new connections for each action
reduce timeout
2020-07-09 15:02:32 +02:00
Niko Storni a80599413c improve params description 2020-07-09 15:02:32 +02:00
Niko Storni 3ffe7a10c7 add other reflector store
add flags
improve disk cleanup
2020-07-09 15:02:32 +02:00
Niko Storni fb0004bac4 increase packet size
add retro-compatibility
remove unused SPV servers
remove travis test failure
2020-07-09 15:02:32 +02:00
Alex Grintsvayg be69c2f05c
add flag to delete blobs after upload 2020-05-02 14:31:10 -04:00
Alex Grintsvayg dde93a1fe6
make cache dir a cli flag 2020-02-27 14:53:33 -05:00
Niko Storni 4a5a148843
implement disk cleanup 2020-02-27 14:53:33 -05:00
Alex Grintsvayg d291c063ec
add traces to unmarshall errors 2020-02-25 15:49:51 -05:00
Alex Grintsvayg 50089481fb
switch to prometheus for metrics 2020-01-02 13:12:33 -05:00
Alex Grintsvayg dc6dd8d12b
fix blocklist, log future blocklist errors, add resolve wallet server method 2019-12-28 19:17:52 -05:00