LBRYCommunity/lbry-docker
1
0
Fork 0
Code Issues 2 Pull requests 2 Projects Releases 1 Packages Wiki Activity

Adds cloud-init systemd HOWTO for lbrycrd on DigitalOcean #45

Closed
EnigmaCurry wants to merge 1 commit from cloud-init into master
pull from: cloud-init
merge into: LBRYCommunity:master
Conversation 11 Commits 1 Files changed 2 +187
EnigmaCurry commented 2019-04-16 01:23:13 +02:00 (Migrated from github.com)
Copy link

Hi there!

I wanted to have an automated and self-contained script for deploying lbrycrd to docker on DigitalOcean, so I created this little howto.

I utilized the main Dockerfile in this repo as a base, but I have made several changes (mostly removed all of the TODOs as they were not necessary for this use-case.) Instead of replacing the original Dockerfile I chose to just create another directory for mine, as it's a different strategy for deployment.

If this gets merged, there is some cleanup that needs to occur afterward. The README links to my own docker-hub image, but this can be setup under an lbry account so it has a more official home. The Dockerfile included is the exact same one used to create the current image on docker-hub.

Hi there! I wanted to have an automated and self-contained script for deploying lbrycrd to docker on DigitalOcean, so I created this little howto. I utilized the main Dockerfile in this repo as a base, but I have made several changes (mostly removed all of the TODOs as they were not necessary for this use-case.) Instead of replacing the original Dockerfile I chose to just create another directory for mine, as it's a different strategy for deployment. If this gets merged, there is some cleanup that needs to occur afterward. The README links to my own docker-hub image, but this can be setup under an lbry account so it has a more official home. The Dockerfile included is the exact same one used to create the current image on docker-hub.
Leopere commented 2019-04-16 01:47:27 +02:00 (Migrated from github.com)
Copy link

Without having reviewed the container yet can you explain the reason you can't just go with a docker library official image as your base? We really try to avoid adding technical debt/complexity as much as possible which keeps maintainability high.

Without having reviewed the container yet can you explain the reason you can't just go with a docker library official image as your base? We really try to avoid adding technical debt/complexity as much as possible which keeps maintainability high.
Leopere commented 2019-04-16 02:01:30 +02:00 (Migrated from github.com)
Copy link

Okay after reviewing your commit, I like the direction that you're taking with the addition of the README.md for a recipe for deploying this on DigitalOcean but I disagree with how the Dockerfile seems to regress from the intended functionality of the main Dockerfile and would want to know the benefits of switching to a less stateless form factor.

I'm looking forward to more commits and replies, but I'm not sure we need another Dockerfile for the same function. I think that the TODO's need removed and converted into real GitHub issues that can get organized into the project management board if I haven't done so already.

To give some perspective on some of the design choices that I'd made at the time, the start.sh's goal is intended to offer the user the ability to set defaults with an externally injected config if they desire as well as allow them just to use environment variables and volumes. Ultimately this thing should function by default with zero user interaction aside from just copying and pasting commands because we want a liquid smooth deployment experience so that people can get testing their ideas immediately.

Okay after reviewing your commit, I like the direction that you're taking with the addition of the README.md for a recipe for deploying this on DigitalOcean but I disagree with how the Dockerfile seems to regress from the intended functionality of the main Dockerfile and would want to know the benefits of switching to a less stateless form factor. I'm looking forward to more commits and replies, but I'm not sure we need another Dockerfile for the same function. I think that the TODO's need removed and converted into real GitHub issues that can get organized into the project management board if I haven't done so already. To give some perspective on some of the design choices that I'd made at the time, the start.sh's goal is intended to offer the user the ability to set defaults with an externally injected config if they desire as well as allow them just to use environment variables and volumes. Ultimately this thing should function by default with zero user interaction aside from just copying and pasting commands because we want a liquid smooth deployment experience so that people can get testing their ideas immediately.
EnigmaCurry commented 2019-04-16 02:31:08 +02:00 (Migrated from github.com)
Copy link

Thanks for the quick detailed response.

  • I'm open to changing the "FROM ubuntu:18.04" I just copied that from the other Dockerfile.
  • I removed the start.sh because it was getting in the way of just using the arguments I needed. Instead, I chose to wrap it with a systemd service and a bash_alias to encapsulate lbrycrd-cli into a command. The upside is that now it's just a README and a Dockerfile.
  • We could add the start.sh back in, since you can still pass whatever command you want to a container that has a CMD. I would be opposed to making it an ENTRYPOINT though, because (IIRC) that would limit the interaction only to start.sh. (There's no good reason that the docker container for lbrycrd should be less flexible than the binary itself.)

I'll see if I can rework what I have with your existing container. Does an image exist on the docker hub? I couldn't find it, if it does.

Thanks for the quick detailed response. * I'm open to changing the "FROM ubuntu:18.04" I just copied that from the other Dockerfile. * I removed the start.sh because it was getting in the way of just using the arguments I needed. Instead, I chose to wrap it with a systemd service and a bash_alias to encapsulate lbrycrd-cli into a command. The upside is that now it's just a README and a Dockerfile. * We could add the start.sh back in, since you can still pass whatever command you want to a container that has a CMD. I would be opposed to making it an ENTRYPOINT though, because (IIRC) that would limit the interaction only to start.sh. (There's no good reason that the docker container for lbrycrd should be less flexible than the binary itself.) I'll see if I can rework what I have with your existing container. Does an image exist on the docker hub? I couldn't find it, if it does.
EnigmaCurry commented 2019-04-16 03:09:53 +02:00 (Migrated from github.com)
Copy link

Some of the design changes to my Dockerfile were:

  • If you make all of your RUN as one single RUN, it will mean that the image tree size is smaller. (Each RUN and COPY creates its own successive docker image.)
  • It would be good to bring the version up to the top as an ARG.
  • Treat the config file location as seperate from the data volume (more configurable to end users)
Some of the design changes to my Dockerfile were: * If you make all of your RUN as one single RUN, it will mean that the image tree size is smaller. (Each RUN and COPY creates its own successive docker image.) * It would be good to bring the version up to the top as an ARG. * Treat the config file location as seperate from the data volume (more configurable to end users)
Leopere commented 2019-04-16 03:28:23 +02:00 (Migrated from github.com)
Copy link

I'll see if I can rework what I have with your current container. Does an image exist on the docker hub? I couldn't find it if it does.

It's not been added to Docker hub, and I do however have access to put it there. Unfortunately, we ended up at a few blocking points the last time I had a moment to post the containers mainly that they won't build automatically without having to modify the Travis-CI.yml files for the upstream repositories. I was attempting to get the Dockerfiles accepted into the upstream repositories themselves so that when a new version is pushed it would build in the Travis-CI

I am aware of the number of layers being a bit high this was the initial 1.0 we're eventually going to optimize significantly. My priority was stateless and maintainability. Then I was going to re-double down on making certain security is top notch etc and expand on examples.

There's a good number of priorities laid out in the boards but I'm flexible and 100% interested in any other contributors its great having external interest.

```I'll see if I can rework what I have with your current container. Does an image exist on the docker hub? I couldn't find it if it does.``` It's not been added to Docker hub, and I do however have access to put it there. Unfortunately, we ended up at a few blocking points the last time I had a moment to post the containers mainly that they won't build automatically without having to modify the Travis-CI.yml files for the upstream repositories. I was attempting to get the Dockerfiles accepted into the upstream repositories themselves so that when a new version is pushed it would build in the Travis-CI I am aware of the number of layers being a bit high this was the initial 1.0 we're eventually going to optimize significantly. My priority was stateless and maintainability. Then I was going to re-double down on making certain security is top notch etc and expand on examples. There's a good number of priorities laid out in the boards but I'm flexible and 100% interested in any other contributors its great having external interest.
❤️ 1
EnigmaCurry commented 2019-04-16 04:31:26 +02:00 (Migrated from github.com)
Copy link

I realize now your first question was in regards to the docker host image, not the container image. So I'll explain a bit my thoughts.

I think that the "container distros" like CoreOS, Fedora Atomic, Rancher are great, and if you're using Kubernetes, even better. However, they are quite difficult to work with if you are a novice user. Ubuntu is much more well understood. If a user needs to edit the config file, they need a real OS, that has a package manager, so they can install Emacs. It needs state. This is the exact opposite of what you want in the enterprise worlds, where you treat every thing as "cattle". But in this case I think we want to grow a decentralized network of users running lbrycrd. That means a lot of users running single nodes. The extra hurdles needed to learn about container OSes is an extra burden to users.

Systemd is great at configuring single nodes. It's baked into every linux distro out there (pretty much). It has a really consistent interface and good automation support. One of the nice things about containers is it frees people up to deploy them however they want. So systemd isn't for everyone. That's why I seperated this from the other directory, as I didn't care to use docker-compose. Perhaps this README should just live on a wiki? That way we can focus this repository just on the image itself.

I realize now your first question was in regards to the docker host image, not the container image. So I'll explain a bit my thoughts. I think that the "container distros" like CoreOS, Fedora Atomic, Rancher are great, and if you're using Kubernetes, even better. However, they are quite difficult to work with if you are a novice user. Ubuntu is much more well understood. If a user needs to edit the config file, they need a real OS, that has a package manager, so they can install Emacs. It needs state. This is the exact opposite of what you want in the enterprise worlds, where you treat every thing as "cattle". But in this case I think we want to grow a decentralized network of users running lbrycrd. That means a lot of users running single nodes. The extra hurdles needed to learn about container OSes is an extra burden to users. Systemd is great at configuring single nodes. It's baked into every linux distro out there (pretty much). It has a really consistent interface and good automation support. One of the nice things about containers is it frees people up to deploy them however they want. So systemd isn't for everyone. That's why I seperated this from the other directory, as I didn't care to use docker-compose. Perhaps this README should just live on a wiki? That way we can focus this repository just on the image itself.
Leopere (Migrated from github.com) reviewed 2019-04-16 04:52:33 +02:00
Leopere (Migrated from github.com) left a comment
Copy link

Great first PR and welcoming more in the future but ideally use the mainline container as there's no sense in maintaining multiple containers that attempt to achieve the same objectives.

Great first PR and welcoming more in the future but ideally use the mainline container as there's no sense in maintaining multiple containers that attempt to achieve the same objectives.
lbrycrd-systemd/README.md
@ -0,0 +57,4 @@
--mount type=volume,source=lbrycrd-data,target=/data \
--mount type=bind,source=/etc/lbry/lbrycrd.conf,target=/etc/lbry/lbrycrd.conf \
--hostname lbrycrd \
-e RUN_MODE=default \
Leopere (Migrated from github.com) commented 2019-04-16 02:08:21 +02:00
Copy link

This won't work if you don't include the start.sh which was included in the main Dockerfile

This won't work if you don't include the start.sh which was included in the main Dockerfile
Leopere commented 2019-04-16 04:53:41 +02:00 (Migrated from github.com)
Copy link

I'll leave it up to @EnigmaCurry as to whether or not you wish to modify this PR or submit a new one.

I'll leave it up to @EnigmaCurry as to whether or not you wish to modify this PR or submit a new one.
EnigmaCurry commented 2019-04-16 05:05:17 +02:00 (Migrated from github.com)
Copy link

I would need to modify start.sh to not create the config file, in order to be compatible with what I have. I think it is more configurable to allow the user to specify the whole config file rather than use individial environment variables for each of the items. So I can't use your container as is. Also it's weird that the config file lives in the data volume, but is overwritten each time. So it may appear that you can edit the config in the data directory, but if you do, it will just be overwritten the next time start.sh runs.

I would need to modify start.sh to not create the config file, in order to be compatible with what I have. I think it is more configurable to allow the user to specify the whole config file rather than use individial environment variables for each of the items. So I can't use your container as is. Also it's weird that the config file lives in the data volume, but is overwritten each time. So it may appear that you can edit the config in the data directory, but if you do, it will just be overwritten the next time start.sh runs.
EnigmaCurry (Migrated from github.com) reviewed 2019-04-16 05:18:10 +02:00
lbrycrd-systemd/README.md
@ -0,0 +57,4 @@
--mount type=volume,source=lbrycrd-data,target=/data \
--mount type=bind,source=/etc/lbry/lbrycrd.conf,target=/etc/lbry/lbrycrd.conf \
--hostname lbrycrd \
-e RUN_MODE=default \
EnigmaCurry (Migrated from github.com) commented 2019-04-16 05:18:10 +02:00
Copy link

oh yes I mistakenly left the RUN_MODE environment variable there, from before I ripped out start.sh. It does work though, just the variable is not used.

oh yes I mistakenly left the RUN_MODE environment variable there, from before I ripped out start.sh. It does work though, just the variable is not used.
Leopere commented 2019-04-16 20:34:04 +02:00 (Migrated from github.com)
Copy link

So to follow up on this last comment @EnigmaCurry the start.sh file should only edit if there's an alternative setting. You could use sed, grep, and awk to achieve these goals. It's okay albeit not ideal to miss a few config settings; however ideally you want to set some defaults that work.

My original goal was to abstract a few default settings to a RUN_MODE env-var which would pre-set a few things to function in various modes like Regtest and so on.

So to follow up on this last comment @EnigmaCurry the `start.sh` file should only edit if there's an alternative setting. You could use sed, grep, and awk to achieve these goals. It's okay albeit not ideal to miss a few config settings; however ideally you want to set some defaults that work. My original goal was to abstract a few default settings to a `RUN_MODE` env-var which would pre-set a few things to function in various modes like Regtest and so on.

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
Accepted

Just for marking accepted PR's

  
area: devops

   
area: discovery

   
area: docs

   
area: livestream

   
area: proposal

Work is part of a proposal

  
consider soon

Discuss this issue at the next planning meeting, then remove this label

  
Epic

   
good first issue

Good for newcomers

  
hacktoberfest

Welcome to Hacktoberfest

  
help wanted

Extra attention is needed

  
icebox

Long-term storage

  
In Review

Pull Requests that are currently in review and potentially waiting tweaks or feedback for clarity.

  
level: 1

No knowledge of the existing code required

  
level: 2

Some knowledge of the existing code is recommended

  
level: 3

Significant knowledge of the existing code is recommended

  
level: 4

Intimate knowledge of the existing code is recommended

  
needs: exploration

Solution unclear, needs research

  
needs: grooming

Issue needs to be groomed before work can start

  
needs: priority

Priority level needs to be defined

  
needs: repro

Issue needs step-by-step instructions on how to reproduce in latest code

  
needs: tech design

Needs technical design signoff before implementation can begin

  
on hold

Temporarily paused

  
priority: blocker

Issue is blocking release, do ASAP

  
priority: high

Work needs to be moved into sprint ASAP

  
priority: low

Work should be done but can stay in the backlog for now

  
priority: medium

Work needs to be done within 2-3 sprints

  
resilience

   
Tom's Wishlist

   
type: bug

Existing functionality is wrong or broken

  
type: discussion

A conversation. No specific changes requested

  
type: improvement

Existing (or partially existing) functionality needs to be changed

  
type: new feature

New functionality that does not exist yet

  
type: refactor

Minimal user-visible changes, but significant internal work

  
type: task

Either work that's not related to the code, or a small chore that does not fit into other categories

  
type: testing

Solution needs additional user testing

  
unplanned

Work that was not planned into the spirnt, took priority over planned work

No labels
Accepted
area: devops
area: discovery
area: docs
area: livestream
area: proposal
consider soon
Epic
good first issue
hacktoberfest
help wanted
icebox
In Review
level: 1
level: 2
level: 3
level: 4
needs: exploration
needs: grooming
needs: priority
needs: repro
needs: tech design
on hold
priority: blocker
priority: high
priority: low
priority: medium
resilience
Tom's Wishlist
type: bug
type: discussion
type: improvement
type: new feature
type: refactor
type: task
type: testing
unplanned
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: LBRYCommunity/lbry-docker#45
No description provided.
Delete branch "cloud-init"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?