LBRYCommunity/reflector.go
1
0
Fork 0
Code Issues 7 Pull requests Projects Releases 10 Packages Wiki Activity
347 commits 6 branches 14 tags 1.3 MiB
Commit graph

57 commits

Author SHA1 Message Date
Niko Storni
84fabdd5f4 add option to run with RO-CF only as upstream 
increase idle timeout to avoid errors downstream
add option to delete blobs from DB if storage doesn't have it (for future local tracking)
 2021-07-24 01:03:16 +02:00
Niko Storni
0b565852b8 only store the blobs in the underlying storage if LFUDA accepted them 2021-07-24 01:03:10 +02:00
Niko Storni
ff13d7b2f7 fix cache size mess 2021-07-24 01:03:10 +02:00
Niko Storni
5eb1f13b54 add LFUDA store 
update quic
fix tests
 2021-07-24 01:03:10 +02:00
Alex Grintsvayg
176e05714e rename cahces 2021-07-24 01:03:10 +02:00
Niko Storni
af2742c34f update quic 
don't wait for a blob to be written to disk before sending it downstream
don't wait for the disk store to be walked before starting everything up
 2021-07-24 01:03:10 +02:00
Alex Grintsvayg
e70b9af3e4
dont overallocate ram when reading blobs from disk 
ReadFile checks the file size and allocates a bit more space than we
expect we'll need. ReadAll uses Go's standard resizing algo, which
doubles the underlying array each time you hit the end. So ReadAll
ends up allocating 4MB for a full blob, while ReadFile allocates
slightly over 2MB.
 2020-11-27 16:18:26 -05:00
Alex Grintsvayg
659a6e73cc
use speedwalk for faster file listing 2020-11-04 16:03:47 -05:00
Alex Grintsvayg
aaae3ffa5b
remove afero fs abstraction in prep for using speedwalk 2020-11-04 16:03:47 -05:00
Alex Grintsvayg
131fed28d2
add faster file.Walk fn. meant for DiskStore.list() 2020-11-04 16:03:47 -05:00
Alex Grintsvayg
72571236ab
clarify Get() error requirement 2020-11-04 16:03:46 -05:00
Alex Grintsvayg
560e180e36
separate singleflight cache wrapper, component names for cache metrics 2020-11-04 16:03:46 -05:00
Alex Grintsvayg
124d4065c2
split cloudfront into RO and RW stores 2020-11-04 16:03:45 -05:00
Alex Grintsvayg
f131c1f35b
add noop store that does nothing 2020-11-04 16:03:45 -05:00
Alex Grintsvayg
7a3225434e
add test for DiskStore.list() 2020-11-04 16:03:45 -05:00
Alex Grintsvayg
3608971f0b
add lru cache eviction metric 2020-11-04 16:03:44 -05:00
Alex Grintsvayg
c9fa04043c
rename the stores, add caching to reflector cmd 2020-11-04 16:03:44 -05:00
Alex Grintsvayg
c6b53792c8
separate disk and lru behavior 2020-11-04 16:03:44 -05:00
Alex Grintsvayg
69fa06420b
LRU cache for disk store, abstract fs in disk store for testing 2020-11-04 16:03:43 -05:00
Niko Storni
c3db95a6c1 add more metrics 
increase handshake timeout by 1 second
 2020-10-15 02:59:12 +02:00
Alex Grintsvayg
b02e80d472 ensure only single origin Get request is in flight per hash. protects against thundering herd. 2020-10-15 00:12:31 +02:00
Alex Grintsvayg
de0ccd4da7
track approximate access time for blobs 2020-10-05 18:08:53 -04:00
Niko Storni
fc5f5ff7d3 add cloudfront support 2020-09-09 00:18:07 +02:00
Niko Storni
47f28002ff close unclosed handle 
reduce idle timeout for http3
update QUIC library
 2020-07-10 15:19:57 +02:00
Niko Storni
694bda105c add metrics 2020-07-09 15:02:32 +02:00
Niko Storni
5c91051b78 disable disk cleanup routine 2020-07-09 15:02:32 +02:00
Niko Storni
3ffe7a10c7 add other reflector store 
add flags
improve disk cleanup
 2020-07-09 15:02:32 +02:00
Andrey Beletsky
1bf3cb81b3 Use ModTime on systems that don't provide Atim file stat field 2020-07-09 15:02:32 +02:00
Niko Storni
4a5a148843
implement disk cleanup 2020-02-27 14:53:33 -05:00
Alex Grintsvayg
d291c063ec
add traces to unmarshall errors 2020-02-25 15:49:51 -05:00
Mark Beamer Jr
86a553b876
upgrade to lbry.go v2.4.0 2019-11-13 19:11:35 -05:00
Alex Grintsvayg
661c20a21d
make db-backed store more generic (not specific to s3) 2019-10-03 16:58:17 -04:00
Alex Grintsvayg
69f1e0f4ca
make MemoryStore consistent with the New...() pattern 2019-10-03 16:49:28 -04:00
Alex Grintsvayg
2ca83139df
use stream.Blob for BlobStore interface 2019-10-03 16:34:57 -04:00
Alex Grintsvayg
0af6d65d40
rename FileStore -> DiskStore 2019-10-03 16:24:59 -04:00
Alex Grintsvayg
24f885e268
store blobs on disk in prefix-based subdirectories to avoid too many files in one dir 2019-10-03 16:12:49 -04:00
Alex Grintsvayg
36ee7e8d1f
add caching blob store 2019-10-03 13:36:35 -04:00
Alex Grintsvayg
c7643fb5da
check blocklist first, its cheaper 2019-07-02 09:21:26 -04:00
Alex Grintsvayg
05e0a506b4
better debug output 2019-01-29 14:42:45 -05:00
Alex Grintsvayg
14d6d32a41 move dht to lbry.go repo 2019-01-09 17:52:30 -05:00
Alex Grintsvayg
db791e26ef store blobs in s3 intelligent-tiering storage class 2018-12-26 15:05:32 -05:00
Alex Grintsvayg
e67e3dbf85 typo 2018-10-08 13:29:05 -04:00
Alex Grintsvayg
11ebfb822b started work on go blob primitives. successfully matched python's blob crypto (excluding canonical JSON) 2018-10-04 15:05:19 -04:00
Alex Grintsvayg
61e83d86de actually done now 2018-09-20 11:29:35 -04:00
Alex Grintsvayg
9fb824790b done, but NEEDS MIGRATION AND TESTING 2018-09-20 11:29:35 -04:00
Alex Grintsvayg
75886211b1 fix partial stream upload 2018-08-15 20:18:54 -04:00
Alex Grintsvayg
0e0b2aaea3 drop DB interface, attempt to fix max conn issues using interpolateParams 2018-08-07 16:51:36 -04:00
Alex Grintsvayg
8f395d8743 fix reflector responding correctly when we have the full stream. fixes lbryio/reflector-cluster#60 2018-07-26 10:25:47 -04:00
Alex Grintsvayg
4535122a06 starting to put together the pieces 
- prism start command
- more configs for prism when assembling the pieces
- cluster notifies on membership change, determines hash range, announces hashes
 2018-06-14 22:30:38 -04:00
Mark Beamer Jr
35f98ce162 code cleanup 
-Added travis support
-updated travis to analyze code beneath the root.
-refactored upload.go to fix travis errors.
-gocyclo should ignore test files. $GOFILES needed to be adjusted.
-fix rows.Close() ignoring error. Created func to handle so defer can be used when needed also.
-fixed ignored errors.
-fixed unit test that was not passing correctly to anonymous function.
-fixed govet error for passing param inside go func.
-removed returned error, in favor of logging instead.
-added error logging for ignored error.
-fixed potential race conditions.
-removed unused append
-fixed time usage to align with go standards.
-removed unused variables
-made changes for code review.
-code comments for exported functions.
-Documented bitmap.go and insert into contact list.
-Documented dht, message, bootstrap
-Fixed comment typos
-Documented message,node, routing_table, testing in DHT package.
-Documented server, client, prism, server and shared in peer and reflector packages.
-Documented the stores in Store package.
-made defer adjustments inline and deleted the separate function.
-adjusted method in upload to take the only parameter it requires.
 2018-06-13 09:29:13 -04:00