LBRYCommunity/lbry-docker
1
0
Fork 0
Code Issues 2 Pull requests 2 Projects Releases 1 Packages Wiki Activity

LBRY on Kubernetes with Helm #71

Merged
EnigmaCurry merged 7 commits from k8s-lbry into master 2019-06-15 17:55:53 +02:00
Conversation 25 Commits 7 Files changed 56 +22638
EnigmaCurry commented 2019-05-01 01:17:02 +02:00 (Migrated from github.com)
Copy link

This is a working Helm chart and HOWTO for running lbrycrd, chainquery, lbrynet, and spee.ch on Kubernetes, using the containers from this repo.

README

This is a working Helm chart and HOWTO for running lbrycrd, chainquery, lbrynet, and spee.ch on Kubernetes, using the containers from this repo. [README](https://github.com/EnigmaCurry/lbry-docker/tree/k8s-lbry/contrib/k8s-lbry)
EnigmaCurry commented 2019-05-01 01:19:00 +02:00 (Migrated from github.com)
Copy link

This requires changes to the lbrycrd image outlined in #70

This requires changes to the lbrycrd image outlined in #70
Leopere commented 2019-05-01 16:05:02 +02:00 (Migrated from github.com)
Copy link

Would you be interested in showing a reasonable platform agnostic playthrough of this guide via a video similar to your Cloud-init?

Would you be interested in showing a reasonable platform agnostic playthrough of this guide via a video similar to your Cloud-init?
👍 1
Leopere commented 2019-05-03 13:58:43 +02:00 (Migrated from github.com)
Copy link

There's quite a lot to review here so I would like to sit down with you in slack today and look over its functionality and give it a run through test. Maybe record a video for embedding into the tutorial.

There's quite a lot to review here so I would like to sit down with you in slack today and look over its functionality and give it a run through test. Maybe record a video for embedding into the tutorial.
EnigmaCurry commented 2019-05-08 02:51:45 +02:00 (Migrated from github.com)
Copy link

I want to refine this process a bit with a few more goals:

  • Don't install tiller on the cluster - https://rimusz.net/tillerless-helm
  • Use namespaces by default
  • Use a Makefile or frontend code to encapsulate helm and kubectl commands to make this simpler.
  • Solve the data sharing problem between spee.ch and lbrynet. (share a single volume between containers? Stream results from lbrynet into spee.ch, for spee.ch to store and cache in it's own volume?)
I want to refine this process a bit with a few more goals: * Don't install tiller on the cluster - https://rimusz.net/tillerless-helm * Use namespaces by default * Use a Makefile or frontend code to encapsulate helm and kubectl commands to make this simpler. * Solve the data sharing problem between spee.ch and lbrynet. (share a single volume between containers? Stream results from lbrynet into spee.ch, for spee.ch to store and cache in it's own volume?)
Leopere commented 2019-05-08 04:05:55 +02:00 (Migrated from github.com)
Copy link

@EnigmaCurry I would say a shared volume is the solution for shared data between spee.ch and lbrynet.

The makefile could be a really good option however BASH is generally more available than Make is it possible to just use a bash script?

@EnigmaCurry I would say a shared volume is the solution for shared data between spee.ch and lbrynet. The makefile could be a really good option however BASH is generally more available than Make is it possible to just use a bash script?
EnigmaCurry commented 2019-05-17 15:24:48 +02:00 (Migrated from github.com)
Copy link

I've updated my k8s-lbry branch with the new additions outlined above.

  • There is a new run.sh wrapper script
  • Tiller is run locally from run.sh and no longer installed on the cluster
  • Everything installed to the k8s-lbry namespace
  • Speech includes its own lbrynet sidecar container (and mounts a shared data dir between the two)

Right now I'm building my own spee.ch container from https://gist.github.com/EnigmaCurry/57e9773d9b3748b248a52fcebd447fba

Things to do before merge:

  • Finish production spee.ch container
  • Update image references in values.yaml
  • Update HELM_REPO in run.sh to lbryio official repository.
  • Publish production lbrynet v0.33.0 as this is the version that works with spee.ch right now
  • Figure out why uploads don't work in spee.ch
I've updated my k8s-lbry branch with the new additions outlined above. * There is a new run.sh wrapper script * Tiller is run locally from run.sh and no longer installed on the cluster * Everything installed to the k8s-lbry namespace * Speech includes its own lbrynet sidecar container (and mounts a shared data dir between the two) Right now I'm building my own spee.ch container from https://gist.github.com/EnigmaCurry/57e9773d9b3748b248a52fcebd447fba Things to do before merge: * Finish production spee.ch container * Update image references in values.yaml * Update HELM_REPO in run.sh to lbryio official repository. * Publish production lbrynet v0.33.0 as this is the version that works with spee.ch right now * Figure out why uploads don't work in spee.ch
EnigmaCurry commented 2019-05-31 01:25:18 +02:00 (Migrated from github.com)
Copy link
  • Spee.ch upstream has been upgraded to SDK 0.37.2, speech-lbrynet container updated to new version.
  • Made PR #74 for compatible spee.ch dockerfile
  • Made PR #73 for building new lbrynet container. This needs to run on 0.37.2.
  • Anonymous publishing is working, there is an error in the logs though:
2019-05-30T21:16:20.981Z - error: Error on /api/file/availability/479364455881011fc6306o2600/86e53ddc594fcc13e558dc0c1f94b31f4ee1fb99 
{ stack: 'TypeError: Cannot read property \'dataValues\' of null\n    at /spee.ch/server/bundle/server.js:1:638181\n    at <anonymous>',
  message: 'Cannot read property \'dataValues\' of null' }
  • Once the claim is accepted onto the blockchain, the image resolves correctly.
  • Test page (temporarily online only) : https://speech.rymcg.tech/8/479364455881011fc6306o2600
  • Publishing within a channel is not working, with an error message of no channel password provided, although logging into the channel appears to work, and I do see the password in the log when the account was created.
  • run.sh still points to my forked helm chart values, this still needs to be switched before merged.
  • values.yaml contains docker images for my own repository that needs to be switched once we have spee.ch and lbrynet (v0.37.2) production images.
  • I added the asciicast directly to the contrib/k8s-lbry/kick-ascii directory. Turn on github pages in the repository settings for the master branch.
  • There is a helm repository created on S3 that is being used currently, this could be migrated to github pages as well. That would mean mainting a full history of tarballed helm chart releases in contrib/k8s-lbry/repository. Alternatively, we could just keep it as S3, and host it elsewhere. I think the latter option is a bit cleaner.
* Spee.ch upstream has been upgraded to SDK 0.37.2, speech-lbrynet container updated to new version. * Made PR #74 for compatible spee.ch dockerfile * Made PR #73 for building new lbrynet container. This needs to run on 0.37.2. * Anonymous publishing is working, there is an error in the logs though: ``` 2019-05-30T21:16:20.981Z - error: Error on /api/file/availability/479364455881011fc6306o2600/86e53ddc594fcc13e558dc0c1f94b31f4ee1fb99 { stack: 'TypeError: Cannot read property \'dataValues\' of null\n at /spee.ch/server/bundle/server.js:1:638181\n at <anonymous>', message: 'Cannot read property \'dataValues\' of null' } ``` * Once the claim is accepted onto the blockchain, the image resolves correctly. * Test page (temporarily online only) : https://speech.rymcg.tech/8/479364455881011fc6306o2600 * Publishing within a channel is not working, with an error message of `no channel password provided`, although logging into the channel appears to work, and I do see the password in the log when the account was created. * run.sh still points to my forked helm chart values, this still needs to be switched before merged. * values.yaml contains docker images for my own repository that needs to be switched once we have spee.ch and lbrynet (v0.37.2) production images. * I added the asciicast directly to the `contrib/k8s-lbry/kick-ascii` directory. Turn on github pages in the repository settings for the `master` branch. * There is a helm repository created on S3 that is being used currently, this could be migrated to github pages as well. That would mean mainting a full history of tarballed helm chart releases in `contrib/k8s-lbry/repository`. Alternatively, we could just keep it as S3, and host it elsewhere. I think the latter option is a bit cleaner.
Leopere (Migrated from github.com) requested changes 2019-06-14 20:53:47 +02:00
Leopere (Migrated from github.com) left a comment
Copy link

Looks like the Readme is 97% of the way there let us get it the last 3% and I'll push this into the repository!

Looks like the Readme is 97% of the way there let us get it the last 3% and I'll push this into the repository!
contrib/k8s-lbry/README.md
Leopere (Migrated from github.com) commented 2019-06-14 19:54:38 +02:00
Copy link

Ideally, this should be hosted on this repository completely.

Ideally, this should be hosted on this repository completely.
Leopere (Migrated from github.com) commented 2019-06-14 20:42:29 +02:00
Copy link

Is this a whole env-var? Looks like a combination between a variable and a command. This should be clearer.

Is this a whole env-var? Looks like a combination between a variable and a command. This should be clearer.
contrib/k8s-lbry/README.md
@ -0,0 +1,876 @@
# LBRY on Kubernetes with Helm
Leopere (Migrated from github.com) commented 2019-06-14 20:24:33 +02:00
Copy link

One of the things that I would love for the documentation is to avoid making assumptions of knowledge in general. So when a user jumps into wanting to host their own K8's based instance of this we should assume that the user will want links back to the official documentation for various processes.

One of the things that I would love for the documentation is to avoid making assumptions of knowledge in general. So when a user jumps into wanting to host their own K8's based instance of this we should assume that the user will want links back to the official documentation for various processes.
contrib/k8s-lbry/README.md
@ -0,0 +56,4 @@
<!-- tocstop -->
## Requirements
Leopere (Migrated from github.com) commented 2019-06-14 19:59:31 +02:00
Copy link

Just for the sake of ramping up adoption of this method, we should definitely attempt to have a proper screencast captured and hosted on LBRY for this.

Nothing worse than making the wrong assumptions we should really aim to have a full zero to hero guide for everything.

Just for the sake of ramping up adoption of this method, we should definitely attempt to have a proper screencast captured and hosted on LBRY for this. Nothing worse than making the wrong assumptions we should really aim to have a full zero to hero guide for everything.
contrib/k8s-lbry/README.md
@ -0,0 +93,4 @@
If everything is working, you should see a list of one or more nodes running and
showing `STATUS=Ready`
## Security Notice
Leopere (Migrated from github.com) commented 2019-06-14 20:01:02 +02:00
Copy link

We need to also refer to vulnerability reporting and any responsible disclosure stuff in a security notice.

We need to also refer to vulnerability reporting and any responsible disclosure stuff in a security notice.
contrib/k8s-lbry/README.md
@ -0,0 +147,4 @@
```
curl -Lo run.sh https://raw.githubusercontent.com/EnigmaCurry/lbry-docker/k8s-lbry/contrib/k8s-lbry/run.sh
Leopere (Migrated from github.com) commented 2019-06-14 20:04:55 +02:00
Copy link

Possibly put a comment here mentioning that people should always review the contents of a curl'd script for secure operations involving money.

Possibly put a comment here mentioning that people should always review the contents of a curl'd script for secure operations involving money.
contrib/k8s-lbry/README.md
@ -0,0 +225,4 @@
k8s-lbry kubectl get svc nginx-ingress-controller -o wide
```
If you find a hostname instead of an IP address, this means your load balancer
Leopere (Migrated from github.com) commented 2019-06-14 20:11:11 +02:00
Copy link

Could we not replace part of this with a script that would just use the OS's included dig installation to do this? If that sounds like a proper pain in the ass nevermind but just sounds like a step we can do for them potentially.

Could we not replace part of this with a script that would just use the OS's included `dig` installation to do this? If that sounds like a proper pain in the ass nevermind but just sounds like a step we can do for them potentially.
contrib/k8s-lbry/README.md
@ -0,0 +468,4 @@
* mysql for storing the speech database.
Edit `values-dev.yaml`.
Leopere (Migrated from github.com) commented 2019-06-14 20:19:22 +02:00
Copy link

Can you update references to these files with URL's referencing their location in the repository? Just might help people know where the defaults are.

Can you update references to these files with URL's referencing their location in the repository? Just might help people know where the defaults are.
contrib/k8s-lbry/README.md
@ -0,0 +492,4 @@
Copy the wallet in case the volume gets destroyed:
```
k8s-lbry speech-lbrynet-copy-wallet /tmp/k8s-lbry-speech-lbrynet-wallet-backup.json
Leopere (Migrated from github.com) commented 2019-06-14 20:22:03 +02:00
Copy link

Could we potentially cat and pipe this into apt-get install qrencode on the host somehow so the user could scan it into a wallet app or something for backup?

Could we potentially `cat` and pipe this into `apt-get install qrencode` on the host somehow so the user could scan it into a wallet app or something for backup?
contrib/k8s-lbry/kick-ascii/README.md
Leopere (Migrated from github.com) commented 2019-06-14 19:55:21 +02:00
Copy link

Can these things be moved into a directory within this repository?

Can these things be moved into a directory within this repository?
contrib/k8s-lbry/run.sh
@ -0,0 +541,4 @@
fi
}
setup-alias() {
Leopere (Migrated from github.com) commented 2019-06-14 20:06:30 +02:00
Copy link

This should verify that the user's shell is BASH if you're planning on adding this.

This should verify that the user's shell is BASH if you're planning on adding this.
EnigmaCurry (Migrated from github.com) reviewed 2019-06-14 21:05:28 +02:00
contrib/k8s-lbry/kick-ascii/README.md
EnigmaCurry (Migrated from github.com) commented 2019-06-14 21:05:28 +02:00
Copy link

I copied everything from https://github.com/enigmacurry/kick-ascii into this directory. This is just the upstream README. I can actually just delete the file, its not needed..

I copied everything from https://github.com/enigmacurry/kick-ascii into this directory. This is just the upstream README. I can actually just delete the file, its not needed..
EnigmaCurry (Migrated from github.com) reviewed 2019-06-14 21:19:58 +02:00
contrib/k8s-lbry/README.md
@ -0,0 +225,4 @@
k8s-lbry kubectl get svc nginx-ingress-controller -o wide
```
If you find a hostname instead of an IP address, this means your load balancer
EnigmaCurry (Migrated from github.com) commented 2019-06-14 21:19:58 +02:00
Copy link

This whole paragraph is an implementation detail that only affects AWS users (I'm not sure if it affectrs all AWS users, or if it was just my config). On DigitalOcean its a non-issue. I will add a comment they can use dig themselves. Scripting this away might make this harder to debug I think, plus end-users might have a better idea of how to route via two ip address than me?

This whole paragraph is an implementation detail that only affects AWS users (I'm not sure if it affectrs *all* AWS users, or if it was just my config). On DigitalOcean its a non-issue. I will add a comment they can use `dig` themselves. Scripting this away might make this harder to debug I think, plus end-users might have a better idea of how to route via two ip address than me?
EnigmaCurry (Migrated from github.com) reviewed 2019-06-14 21:22:32 +02:00
contrib/k8s-lbry/README.md
@ -0,0 +468,4 @@
* mysql for storing the speech database.
Edit `values-dev.yaml`.
EnigmaCurry (Migrated from github.com) commented 2019-06-14 21:22:32 +02:00
Copy link

values-dev.yaml does not exist in the repository, only values.yaml does. values-dev is created by run.sh when the user runs setup.

values-dev.yaml does not exist in the repository, only values.yaml does. values-dev is created by run.sh when the user runs setup.
EnigmaCurry (Migrated from github.com) reviewed 2019-06-14 21:24:04 +02:00
contrib/k8s-lbry/README.md
@ -0,0 +492,4 @@
Copy the wallet in case the volume gets destroyed:
```
k8s-lbry speech-lbrynet-copy-wallet /tmp/k8s-lbry-speech-lbrynet-wallet-backup.json
EnigmaCurry (Migrated from github.com) commented 2019-06-14 21:24:04 +02:00
Copy link

I don't think I want to introduce a new dependency, but adding instructions to do this is a great idea!

I don't think I want to introduce a new dependency, but adding instructions to do this is a great idea!
EnigmaCurry (Migrated from github.com) reviewed 2019-06-14 21:24:55 +02:00
contrib/k8s-lbry/README.md
@ -0,0 +492,4 @@
Copy the wallet in case the volume gets destroyed:
```
k8s-lbry speech-lbrynet-copy-wallet /tmp/k8s-lbry-speech-lbrynet-wallet-backup.json
EnigmaCurry (Migrated from github.com) commented 2019-06-14 21:24:55 +02:00
Copy link

actually if the dependency is only for one subcommand, maybe its ok. Lemme think about this. Its a grand idea.

actually if the dependency is only for one subcommand, maybe its ok. Lemme think about this. Its a grand idea.
EnigmaCurry (Migrated from github.com) reviewed 2019-06-14 21:31:25 +02:00
contrib/k8s-lbry/README.md
EnigmaCurry (Migrated from github.com) commented 2019-06-14 21:31:25 +02:00
Copy link

how about this?

(
  NGINX_ENABLE_HTTPS=true 
  k8s-lbry upgrade-nginx-ingress
)

Only problem with that though is if the user thinks they don't need the parentheses. The env var is only supposed to be temporary for that one command.

how about this? ``` ( NGINX_ENABLE_HTTPS=true k8s-lbry upgrade-nginx-ingress ) ``` Only problem with that though is if the user thinks they don't need the parentheses. The env var is only supposed to be temporary for that one command.
EnigmaCurry (Migrated from github.com) reviewed 2019-06-14 21:31:55 +02:00
contrib/k8s-lbry/README.md
EnigmaCurry (Migrated from github.com) commented 2019-06-14 21:31:55 +02:00
Copy link

I'll just leave a note to run it with the parens.

I'll just leave a note to run it with the parens.
EnigmaCurry (Migrated from github.com) reviewed 2019-06-14 21:35:59 +02:00
contrib/k8s-lbry/README.md
@ -0,0 +93,4 @@
If everything is working, you should see a list of one or more nodes running and
showing `STATUS=Ready`
## Security Notice
EnigmaCurry (Migrated from github.com) commented 2019-06-14 21:35:59 +02:00
Copy link

added!

added!
EnigmaCurry (Migrated from github.com) reviewed 2019-06-14 21:36:11 +02:00
contrib/k8s-lbry/README.md
@ -0,0 +147,4 @@
```
curl -Lo run.sh https://raw.githubusercontent.com/EnigmaCurry/lbry-docker/k8s-lbry/contrib/k8s-lbry/run.sh
EnigmaCurry (Migrated from github.com) commented 2019-06-14 21:36:11 +02:00
Copy link

added

added
EnigmaCurry (Migrated from github.com) reviewed 2019-06-14 21:36:50 +02:00
contrib/k8s-lbry/README.md
@ -0,0 +225,4 @@
k8s-lbry kubectl get svc nginx-ingress-controller -o wide
```
If you find a hostname instead of an IP address, this means your load balancer
EnigmaCurry (Migrated from github.com) commented 2019-06-14 21:36:50 +02:00
Copy link

added note to use dig if available, but otherwise leaving this as-is.

added note to use dig if available, but otherwise leaving this as-is.
EnigmaCurry (Migrated from github.com) reviewed 2019-06-14 21:37:11 +02:00
contrib/k8s-lbry/README.md
@ -0,0 +1,876 @@
# LBRY on Kubernetes with Helm
EnigmaCurry (Migrated from github.com) commented 2019-06-14 21:37:11 +02:00
Copy link

added

added
EnigmaCurry (Migrated from github.com) reviewed 2019-06-14 21:37:47 +02:00
contrib/k8s-lbry/README.md
@ -0,0 +56,4 @@
<!-- tocstop -->
## Requirements
EnigmaCurry (Migrated from github.com) commented 2019-06-14 21:37:47 +02:00
Copy link

asciicast links should resolve to lbryio.github.io now.

asciicast links should resolve to lbryio.github.io now.
EnigmaCurry (Migrated from github.com) reviewed 2019-06-14 21:49:03 +02:00
contrib/k8s-lbry/README.md
@ -0,0 +492,4 @@
Copy the wallet in case the volume gets destroyed:
```
k8s-lbry speech-lbrynet-copy-wallet /tmp/k8s-lbry-speech-lbrynet-wallet-backup.json
EnigmaCurry (Migrated from github.com) commented 2019-06-14 21:49:03 +02:00
Copy link

OK after thinking about it, I think this is the wrong place to implement this. I feel like lbrynet itself should have this feature. Otherwise someone might look at the lbrynet docs, and not know what to do with this qrcode.

OK after thinking about it, I think this is the wrong place to implement this. I feel like lbrynet itself should have this feature. Otherwise someone might look at the lbrynet docs, and not know what to do with this qrcode.
EnigmaCurry (Migrated from github.com) reviewed 2019-06-14 22:47:49 +02:00
contrib/k8s-lbry/run.sh
@ -0,0 +541,4 @@
fi
}
setup-alias() {
EnigmaCurry (Migrated from github.com) commented 2019-06-14 22:47:49 +02:00
Copy link

Now it does. thank you.

Now it does. thank you.
Leopere commented 2019-06-14 23:15:29 +02:00 (Migrated from github.com)
Copy link

Just waiting on @tiger5226 to enable the github pages for this repo so I can rebase and merge this at this point.

Just waiting on @tiger5226 to enable the github pages for this repo so I can rebase and merge this at this point.
Leopere (Migrated from github.com) approved these changes 2019-06-14 23:15:50 +02:00
tiger5226 commented 2019-06-15 17:12:33 +02:00 (Migrated from github.com)
Copy link

its enabled

its enabled
👍 1
Sign in to join this conversation.
Reviewers
No reviewers
Leopere
Labels
Clear labels   
Accepted

Just for marking accepted PR's

  
area: devops

   
area: discovery

   
area: docs

   
area: livestream

   
area: proposal

Work is part of a proposal

  
consider soon

Discuss this issue at the next planning meeting, then remove this label

  
Epic

   
good first issue

Good for newcomers

  
hacktoberfest

Welcome to Hacktoberfest

  
help wanted

Extra attention is needed

  
icebox

Long-term storage

  
In Review

Pull Requests that are currently in review and potentially waiting tweaks or feedback for clarity.

  
level: 1

No knowledge of the existing code required

  
level: 2

Some knowledge of the existing code is recommended

  
level: 3

Significant knowledge of the existing code is recommended

  
level: 4

Intimate knowledge of the existing code is recommended

  
needs: exploration

Solution unclear, needs research

  
needs: grooming

Issue needs to be groomed before work can start

  
needs: priority

Priority level needs to be defined

  
needs: repro

Issue needs step-by-step instructions on how to reproduce in latest code

  
needs: tech design

Needs technical design signoff before implementation can begin

  
on hold

Temporarily paused

  
priority: blocker

Issue is blocking release, do ASAP

  
priority: high

Work needs to be moved into sprint ASAP

  
priority: low

Work should be done but can stay in the backlog for now

  
priority: medium

Work needs to be done within 2-3 sprints

  
resilience

   
Tom's Wishlist

   
type: bug

Existing functionality is wrong or broken

  
type: discussion

A conversation. No specific changes requested

  
type: improvement

Existing (or partially existing) functionality needs to be changed

  
type: new feature

New functionality that does not exist yet

  
type: refactor

Minimal user-visible changes, but significant internal work

  
type: task

Either work that's not related to the code, or a small chore that does not fit into other categories

  
type: testing

Solution needs additional user testing

  
unplanned

Work that was not planned into the spirnt, took priority over planned work

No labels
Accepted
area: devops
area: discovery
area: docs
area: livestream
area: proposal
consider soon
Epic
good first issue
hacktoberfest
help wanted
icebox
In Review
level: 1
level: 2
level: 3
level: 4
needs: exploration
needs: grooming
needs: priority
needs: repro
needs: tech design
on hold
priority: blocker
priority: high
priority: low
priority: medium
resilience
Tom's Wishlist
type: bug
type: discussion
type: improvement
type: new feature
type: refactor
type: task
type: testing
unplanned
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: LBRYCommunity/lbry-docker#71
No description provided.
Delete branch "k8s-lbry"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?