Alex Grintsvayg
2a1557845d
wrap blob insertion in tx. fixes lbryio/lbry-sdk#3296
...
The problem is that inserting an sd blob with ~5k
blobs takes longer than 30 seconds. So the client
times out and retries the request. At that point,
reflector is not done inserting so it replies with
a smaller number of blobs than it should. The client
uploads that many blobs and marks the stream as
reflected. The remaining blobs never get uploaded.
Doing the insert inside a transaction should be
faster than doing 10k (2 per blob) inserts
independently.
2021-05-24 10:40:31 -04:00
Niko Storni
9670bc14f8
fix unsafe dereference
2021-05-21 21:06:59 +02:00
Niko Storni
df881e16b5
add metrics
2021-05-21 19:09:02 +02:00
Niko Storni
bc889001bb
update lbry.go dep
2021-05-21 18:05:37 +02:00
Niko Storni
1ec2184833
upgrade singleflight
...
http store fix
2021-05-21 17:58:33 +02:00
Niko Storni
5cc1e84adb
remove locks causing deadlocks
2021-05-21 05:53:13 +02:00
Niko Storni
a7086a00f3
add http server/client
2021-05-21 05:49:02 +02:00
Mark Beamer Jr
76ece1e117
Add queue to prevent writing too many files at once.
2021-05-20 20:43:01 -04:00
Mark Beamer Jr
c1caf1938c
Add queue to prevent writing too many files at once.
2021-05-20 20:41:47 -04:00
Mark Beamer Jr
213d21b021
Add locks to disk store.
2021-05-20 19:59:50 -04:00
Niko Storni
006b04f6e9
add a lot of extra heavy debugging
2021-05-21 01:48:46 +02:00
Mark Beamer Jr
45130499cd
Add single flight for cache not just origin
2021-05-20 19:05:48 -04:00
Mark Beamer Jr
4ecce75e23
add metric calls for other packages
2021-05-20 18:12:30 -04:00
Mark Beamer Jr
0152300d8d
add guage metrics for go routines in reflector package
2021-05-20 18:01:13 -04:00
Mark Beamer Jr
eafc62f2a6
add gops to reflector server
2021-05-20 17:21:35 -04:00
Niko Storni
50c077a9cb
request queue size param
2021-05-20 23:17:18 +02:00
Niko Storni
070938e12a
increase window size
2021-05-06 22:53:18 +02:00
Niko Storni
c4084eeb68
improve disk cleanup
...
add index to is_stored
fix test
replace LRU cache
2021-04-29 03:41:18 +02:00
Niko Storni
4392c97242
fix mess with lbry.go
2021-04-13 00:52:56 +02:00
Niko Storni
ec3aae33ba
add if this than that store
...
switch to wasabi for uploads
2021-04-12 23:05:50 +02:00
Niko Storni
dc95351cf3
add integrity check cmd
...
throttle live integrity checks
bug fixes
2021-04-07 04:46:18 +02:00
Mark Beamer Jr
25a7fac4f0
use wait group not stopper
2021-04-06 14:28:29 -04:00
Mark Beamer Jr
b97595311f
Wait for request to be handled before returning
2021-04-06 14:21:05 -04:00
Mark Beamer Jr
bd13836897
Add request queue for blob cache
2021-04-06 14:00:36 -04:00
Niko Storni
38b44218f2
check blobs when reading them
2021-04-05 23:34:45 +02:00
Niko Storni
90c36fbe24
upgrade quic-go
...
add cache for blobs not found
2021-03-31 04:53:27 +02:00
Niko Storni
3a441aed3a
fix issues caused by beamer's renaming
2021-03-29 19:56:18 +02:00
Niko Storni
ebb62d0a24
run go mod tidy
2021-03-29 19:44:27 +02:00
Niko Storni
8cb7389619
make it simpler
2021-02-23 15:23:46 +01:00
Niko Storni
7b49dd115b
remove panics
2021-02-23 15:08:32 +01:00
Niko Storni
6291e33ee1
add tracing to blobs
2021-01-14 20:38:04 +01:00
Niko Storni
3e475e537b
optimize batch insertions
...
reduce touch time to every 6 hours
2021-01-07 01:28:34 +01:00
Alex Grintsvayg
c4504631bc
avoid heavy interpolateparams call
2021-01-06 10:43:35 -05:00
Alex Grintsvayg
cc504e6c44
fix long query
2021-01-05 12:16:44 -05:00
Alex Grintsvayg
49714c02a6
only touch blobs when you get them
2021-01-05 11:36:33 -05:00
Niko Storni
b33651ae26
save uploaded blobs and work around the blocklist issue
2021-01-05 05:09:55 +01:00
Niko Storni
0d5004a83b
add cmd to populate db
...
fix store init
try fixing unreasonable db bottleneck
2020-12-30 04:24:11 +01:00
Niko Storni
04f6859c74
Merge branch 'grin' into litedb
2020-12-24 23:13:31 +01:00
Alex Grintsvayg
3a1d9d3304
something like this
2020-12-23 17:08:13 -05:00
Niko Storni
03304312e8
add PoC for litedb to avoid all the overhead
2020-12-23 06:04:42 +01:00
Niko Storni
869030fc58
address some review comments
2020-12-22 21:19:48 +01:00
Niko Storni
def551cc89
add option to run with RO-CF only as upstream
...
increase idle timeout to avoid errors downstream
add option to delete blobs from DB if storage doesn't have it (for future local tracking)
2020-12-22 20:53:48 +01:00
Niko Storni
74b76a11e4
upgrade quic
2020-12-17 23:49:37 +01:00
Niko Storni
2c0df2ca8a
update lfuda library
2020-11-27 16:20:50 -05:00
Niko Storni
9fc96ac01b
only store the blobs in the underlying storage if LFUDA accepted them
2020-11-27 16:20:50 -05:00
Niko Storni
ff9b61b034
fix cache size mess
2020-11-27 16:20:49 -05:00
Niko Storni
7b80b2d4d2
fix buffer cache running out of space
2020-11-27 16:20:49 -05:00
Niko Storni
d45abdbdb0
use LFUDA store
...
swap size to bytes
2020-11-27 16:19:46 -05:00
Niko Storni
bc54601dde
add LFUDA store
...
update quic
fix tests
2020-11-27 16:19:46 -05:00
Alex Grintsvayg
bb41a84bb7
rename cahces
2020-11-27 16:19:45 -05:00
Niko Storni
a574fecf4e
add buffer cache for nvme drive
2020-11-27 16:19:45 -05:00
Niko Storni
9146c8b084
update quic
...
don't wait for a blob to be written to disk before sending it downstream
don't wait for the disk store to be walked before starting everything up
2020-11-27 16:19:45 -05:00
