Commit graph

405 commits

Author SHA1 Message Date
Alex Grin
def0a97f49
Update readme.md 2021-09-28 10:15:24 -04:00
Niko Storni
6dde793745 do not delete blobs for blocked content 2021-09-21 18:00:55 +02:00
Niko Storni
654cc44935 Merge branch 'hash_twice' 2021-09-21 16:21:25 +02:00
Victor Shyba
90d6d29452 remove all hashing on the download path 2021-09-21 16:19:49 +02:00
Niko Storni
b2272fef3a delete overflowing blobs from underlying cache 2021-08-18 19:36:47 +02:00
Niko Storni
86f3e62aa8 fix a panic error
update gin-go
2021-08-05 17:47:20 +02:00
Niko
e1b4f21e00
Merge pull request #52 from lbryio/ittt
Merge months of work including all sort of caches and O_DIRECT optimizations
2021-07-23 20:35:22 -04:00
Niko Storni
b4913ecedf cleanup 2021-07-24 01:03:51 +02:00
Niko Storni
63a574ec2f unify caches
fix tests
2021-07-24 01:03:51 +02:00
Niko Storni
b8af3408e0 move server packages 2021-07-24 01:03:51 +02:00
Niko Storni
847089d0d6 fix error propagation
update readme
2021-07-24 01:03:51 +02:00
Niko Storni
170dfef3a8 fix copy pasta mistake 2021-07-24 01:03:51 +02:00
Niko Storni
2b458a6bd0 fix params
more cleanups
2021-07-24 01:03:51 +02:00
Niko Storni
febfc51cb0 refactor refactor refactor 2021-07-24 01:03:51 +02:00
Andrey Beletsky
72be487262 Fix broken import 2021-07-24 01:03:51 +02:00
Andrey Beletsky
94e7d81bd3 Fix OpenFile call flags for macos 2021-07-24 01:03:51 +02:00
Niko Storni
c6c779da39 fix panic
fix counter leak
2021-07-24 01:03:51 +02:00
Niko Storni
2e101083e6 write blobs to tmp dir to avoid corruption 2021-07-24 01:03:51 +02:00
Niko Storni
63aacd8a69 use O_DIRECT to write to disk (fixes everything)
add queue back to serving blobs
improve a lot of things
upgrade modules
2021-07-24 01:03:51 +02:00
Niko Storni
c03ae6487d fix unsafe dereference 2021-07-24 01:03:51 +02:00
Niko Storni
0c4f455f0c add metrics 2021-07-24 01:03:51 +02:00
Niko Storni
af3e08c446 update lbry.go dep 2021-07-24 01:03:51 +02:00
Niko Storni
975bfe7fac upgrade singleflight
http store fix
2021-07-24 01:03:51 +02:00
Niko Storni
b075d948bb remove locks causing deadlocks 2021-07-24 01:03:51 +02:00
Niko Storni
2651a64dbb add http server/client 2021-07-24 01:03:51 +02:00
Mark Beamer Jr
fa7150cf2b Add queue to prevent writing too many files at once. 2021-07-24 01:03:51 +02:00
Mark Beamer Jr
6c4db980c9 Add queue to prevent writing too many files at once. 2021-07-24 01:03:51 +02:00
Mark Beamer Jr
7adaa510fd Add locks to disk store. 2021-07-24 01:03:51 +02:00
Niko Storni
64ed7304f6 add a lot of extra heavy debugging 2021-07-24 01:03:51 +02:00
Mark Beamer Jr
5aefaf061e Add single flight for cache not just origin 2021-07-24 01:03:51 +02:00
Mark Beamer Jr
724ee47c8b add metric calls for other packages 2021-07-24 01:03:51 +02:00
Mark Beamer Jr
caaec6fcb1 add guage metrics for go routines in reflector package 2021-07-24 01:03:51 +02:00
Mark Beamer Jr
15984b8fd9 add gops to reflector server 2021-07-24 01:03:51 +02:00
Niko Storni
2be913b077 request queue size param 2021-07-24 01:03:51 +02:00
Niko Storni
34c11b0a0e increase window size 2021-07-24 01:03:51 +02:00
Niko Storni
64acdc29c3 improve disk cleanup
add index to is_stored
fix test
replace LRU cache
2021-07-24 01:03:51 +02:00
Niko Storni
598773c90d fix mess with lbry.go 2021-07-24 01:03:47 +02:00
Niko Storni
766238fd7e add if this than that store
switch to wasabi for uploads
2021-07-24 01:03:47 +02:00
Niko Storni
ac5242f173 add integrity check cmd
throttle live integrity checks
bug fixes
2021-07-24 01:03:47 +02:00
Mark Beamer Jr
215103cb33 use wait group not stopper 2021-07-24 01:03:47 +02:00
Mark Beamer Jr
ed3622d0a6 Wait for request to be handled before returning 2021-07-24 01:03:47 +02:00
Mark Beamer Jr
848fce5afa Add request queue for blob cache 2021-07-24 01:03:47 +02:00
Niko Storni
e37eeba0c9 check blobs when reading them 2021-07-24 01:03:47 +02:00
Niko Storni
7da49a4ccb upgrade quic-go
add cache for blobs not found
2021-07-24 01:03:47 +02:00
Niko Storni
7b02ace5e2 fix issues caused by beamer's renaming 2021-07-24 01:03:47 +02:00
Niko Storni
5fb67b32db run go mod tidy 2021-07-24 01:03:47 +02:00
Niko Storni
a0c9ed2ace make it simpler 2021-07-24 01:03:47 +02:00
Niko Storni
998b082a06 remove panics 2021-07-24 01:03:47 +02:00
Niko Storni
36d4156e2a add tracing to blobs 2021-07-24 01:03:47 +02:00
Niko Storni
74925ebba2 optimize batch insertions
reduce touch time to every 6 hours
2021-07-24 01:03:47 +02:00