reflector.go should implement a server-side cache to reduce S3 traffic #28

New issue

Closed

opened 2019-10-16 15:14:20 +02:00 by kauffj · 5 comments

kauffj commented

2019-10-16 15:14:20 +02:00

(Migrated from github.com)

buy ovh server and set it up with ansible
write code to cache blobs on disk between S3 download and serving to client
blobs should be split into subdirectories so we don't have too many files in one dir
configure relatime so we can efficiently track which blobs were accessed least recently
write code to clear cache to keep it under X (45TB?) in size
file issue in SDK to separate reflector upload and download servers
look into benefits/drawbacks of using another filesystem than ext4
set up monitoring for this server

- [x] buy ovh server and set it up with ansible - [x] write code to cache blobs on disk between S3 download and serving to client - [x] blobs should be split into subdirectories so we don't have too many files in one dir - [x] configure relatime so we can efficiently track which blobs were accessed least recently - [x] write code to clear cache to keep it under X (45TB?) in size - [x] file issue in SDK to separate reflector upload and download servers - [ ] look into benefits/drawbacks of using another filesystem than ext4 - [x] set up monitoring for this server

chron0 commented

2020-06-06 14:39:19 +02:00

(Migrated from github.com)

I would also really love to see it being able to work independently of S3, so that all storage would be on the server running the reflector instead of having to rely on AWS (at least as an option).

lyoshenka commented

2020-06-08 19:45:09 +02:00

(Migrated from github.com)

@chron0 this issue talks about s3 because we happen to use s3. however you can use any data store. look at the store package. the BlobStore interface is used throughout the rest of the code, so you can swap in any store you want. we even have a disk store already written.

@chron0 this issue talks about s3 because we happen to use s3. however you can use any data store. look at the [store](https://github.com/lbryio/reflector.go/tree/master@%7B2020-06-08T17:45:09Z%7D/store) package. the `BlobStore` interface is used throughout the rest of the code, so you can swap in any store you want. we even have a disk store already written.

chron0 commented

2020-06-08 20:16:40 +02:00

(Migrated from github.com)

Hmmm, gotta look deeper into it. I like the idea of running a "public reflector/storage node", especially if there is also some kind of reward system to do so. In the meantime my focus is on not relying on my local client to keep my content online. Tom already gave me the franklin package to hack around a while back but it's php and this is go and I'd love to have like my "personal" reflector storing my content so it will be available via p2p, even if my client is offline. Would you say this is already within the scope of the code so that I can dig deeper into it?

lyoshenka commented

2020-06-08 21:23:39 +02:00

(Migrated from github.com)

A lot of the pieces are still missing. The DHT doesn't work quite right, and there's no federation or incentive to host files. I'm glad to hear that you want to play with this even though its not finished. Some of this work is on our roadmap for 2020 but we haven't gotten to it yet. If you want to get your hands dirty, I'm happy to hash out concrete next steps with you sooner.

Our vision with reflector was to have a federated network of hosts that store data for users and get paid for providing that data when users request it. This is the start of that work, and it sounds like that's what you're looking for. A lot of the pieces are still missing. The DHT doesn't work quite right, and there's no federation or incentive to host files. I'm glad to hear that you want to play with this even though its not finished. Some of this work is on our roadmap for 2020 but we haven't gotten to it yet. If you want to get your hands dirty, I'm happy to hash out concrete next steps with you sooner.

chron0 commented

2020-06-09 19:21:40 +02:00

(Migrated from github.com)

I think both use cases would be great to solve in this:

Have federated general nodes with a store/credit system as an incentive for people to run
Have headless non-local storage for creators to keep their content online without lbry-desktop open

IC. I dont mind getting my hands dirty, but I'm not gods greatest gift as a programmer. As for the "vision", ye, I've read that between the lines or was at least kinda hoping for something of a decentralized nature to have headless storage in the network. We can try this "you give me concrete small bits" to solve and I give it a shot. I think both use cases would be great to solve in this: * Have federated general nodes with a store/credit system as an incentive for people to run * Have headless non-local storage for creators to keep their content online without lbry-desktop open

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: LBRYCommunity/reflector.go#28

No description provided.