# LBRY on Kubernetes with Helm Contributing Author: [EnigmaCurry](https://www.enigmacurry.com) Last Update: June 14 2019 Deploy [lbrycrd](https://github.com/lbryio/lbrycrd), [lbrynet](https://github.com/lbryio/lbry), [chainquery](https://github.com/lbryio/chainquery), [mysql](https://www.mysql.com), and [spee.ch](https://github.com/lbryio/spee.ch) on your [Kubernetes](https://kubernetes.io/) cluster. [![asciicast](https://lbryio.github.io/lbry-docker/contrib/k8s-lbry/kick-ascii/cast/k8s-lbry.png)](https://lbryio.github.io/lbry-docker/contrib/k8s-lbry/kick-ascii/?cast=k8s-lbry&bg=lbry.png) - [Requirements](#requirements) - [Security Notice](#security-notice) - [Installation](#installation) * [Create a project directory](#create-a-project-directory) * [Setup alias and tab completion](#setup-alias-and-tab-completion) * [k8s-lbry setup](#k8s-lbry-setup) * [k8s-lbry install-nginx-ingress](#k8s-lbry-install-nginx-ingress) * [k8s-lbry install-cert-manager](#k8s-lbry-install-cert-manager) * [k8s-lbry install](#k8s-lbry-install) * [k8s-lbry upgrade](#k8s-lbry-upgrade) - [Services](#services) * [lbrycrd](#lbrycrd) * [chainquery](#chainquery) + [MySQL for chainquery](#mysql-for-chainquery) + [Start chainquery](#start-chainquery) + [Startup chainquery with a database snapshot](#startup-chainquery-with-a-database-snapshot) * [lbrynet API service (not for spee.ch)](#lbrynet-api-service-not-for-speech) + [IMPORTANT - Backup your cluster wallet](#important---backup-your-cluster-wallet) * [spee.ch (and lbrynet sidecar and mysql)](#speech-and-lbrynet-sidecar-and-mysql) + [IMPORTANT - Backup your speech wallet](#important---backup-your-speech-wallet) + [Fund your speech wallet](#fund-your-speech-wallet) + [Create a thumbnail channel](#create-a-thumbnail-channel) + [Finish speech setup](#finish-speech-setup) - [Extra commands that k8s-lbry (run.sh) provides](#extra-commands-that-k8s-lbry-runsh-provides) * [k8s-lbry helm](#k8s-lbry-helm) * [k8s-lbry kubectl](#k8s-lbry-kubectl) * [k8s-lbry logs](#k8s-lbry-logs) * [k8s-lbry shell](#k8s-lbry-shell) * [k8s-lbry shell-pvc](#k8s-lbry-shell-pvc) * [k8s-lbry restart](#k8s-lbry-restart) * [k8s-lbry lbrynet](#k8s-lbry-lbrynet) * [k8s-lbry chainquery-mysql-client](#k8s-lbry-chainquery-mysql-client) * [k8s-lbry speech-mysql-client](#k8s-lbry-speech-mysql-client) * [k8s-lbry lbrynet-copy-wallet](#k8s-lbry-lbrynet-copy-wallet-) * [k8s-lbry package](#k8s-lbry-package) - [TLS / SSL / HTTPS](#tls--ssl--https) - [Cloud specific notes](#cloud-specific-notes) * [AWS](#aws) * [minikube](#minikube) - [Uninstall](#uninstall) ## Requirements * A Kubernetes cluster. * Tested on DigitalOcean managed Kubernetes cluster on nodes with 8GB of RAM, on kubernetes 1.14.1. * Tested on AWS with [Charmed Kubenetes Distribution](https://www.ubuntu.com/kubernetes/docs/quickstart) - See [AWS specific notes](#aws). * Tested on [minikube](https://kubernetes.io/docs/tasks/tools/install-minikube/) for a self-contained virtual machine running kubernetes in VirtualBox - - See [minikube specific notes](#minikube). * Local development machine dependencies: * [GNU Bash](https://www.gnu.org/software/bash/) and friends. If you're on Linux or Mac, you should be good to go. * [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) * Tested with kubectl v1.14.0 * [helm](https://github.com/helm/helm/releases) * Tested with helm v2.13.1 * Optional: for TLS / HTTPs support, you will also need an internet domain name, and the ability to update its DNS. Your cloud provider should have instructions for setting up `kubectl` to talk to your cluster. This usually involves downloading a config file and putting it in `$HOME/.kube/config`. (The file has to be renamed `config` and put in the `$HOME/.kube` directory.) Test that your `kubectl` can talk to your cluster, by querying for a list of running nodes: ``` kubectl get nodes ``` If everything is working, you should see a list of one or more nodes running and showing `STATUS=Ready` ## Security Notice Any cryptocurrency wallet that is online is a security concern. For any real-world production deployment, you will need to review this architecture closely to see if it fits with your chosen platform and network environment. This system is currently designed for a kubernetes cluster that has a single administrator (or possibly a small team of trusted users). It will not support untrusted multi-tenancy out of the box. All of the services are created in their own namespace, but no [Security Policies](https://kubernetes.io/docs/concepts/policy/pod-security-policy/) have been applied to the pods. The Helm configuration file contains *all* of the configuration for the system, *including passwords* in plain text. The lbrynet SDK wallets are individually stored unencrypted in their own persistent volumes. “We take security seriously. Please contact [security@lbry.io](mailto:security@lbry.io) regarding any security issues. Our PGP key is [here](https://keybase.io/lbry/key.asc) if you need it.” If you find vulnerabilites, especially any that might increase the risk of someone losing their crypto currency [Responsible Disclosure](https://en.wikipedia.org/wiki/Responsible_disclosure) is appreciated. ## Installation This system is installed via [Helm](https://helm.sh/docs/), the package manager for Kubernetes. [Helm Charts](https://helm.sh/docs/developing_charts/#charts) are the basis for packages in Helm. This directory is a Helm chart itself. All of the helm and kubectl commands necessary to install, upgrade, and maintain your deployments, are wrapped in the included [`run.sh`](run.sh) script. For debugging purposes, this wrapper also prints to stdout the full underlying commands (helm, kubectl, etc) as they are run. ### Create a project directory Create a new directory someplace to store your deployment configuration. For the rest of this tutorial, you will work from this directory: ``` mkdir $HOME/k8s-lbry-test cd $HOME/k8s-lbry-test ``` Download `run.sh` to this same directory: ``` curl -Lo run.sh https://raw.githubusercontent.com/lbryio/lbry-docker/master/contrib/k8s-lbry/run.sh chmod a+x run.sh ``` It's a good idea to always read any script you download from the internet, before running it. ### Setup alias and tab completion `run.sh` can be run directly without any setup. However, without adding it to your `PATH`, you need to specify the full path to the script each time. Setting a bash alias for `run.sh` is the quickest way of setting up to run from anywhere, as well as activating support for bash tab completion. One time setup to install alias to `$HOME/.bashrc`: ``` ./run.sh setup-alias ``` It should prompt you if it is OK for the script to edit `$HOME/.bashrc`. Once you confirm, close your terminal session, then reopen it. Verify the new `k8s-lbry` alias to `run.sh` is working: ``` k8s-lbry kubectl get nodes ``` Notice that tab completion should work throughout typing the above command. ### k8s-lbry setup Setup will check for dependencies, update helm repositories, and create an initial config file (`values-dev.yaml`). ``` k8s-lbry setup ``` ### k8s-lbry install-nginx-ingress An Ingress Controller ([nginx-ingress](https://github.com/helm/charts/tree/master/stable/nginx-ingress)) will help you to route outside internet traffic into your cluster. nginx-ingress will also help terminate TLS connections (SSL) so that your containers don't need to worry about encryption of traffic. Install nginx-ingress into the `k8s-lbry` namespace: ``` k8s-lbry install-nginx-ingress ``` ### k8s-lbry install-cert-manager [cert-manager](https://docs.cert-manager.io/en/latest/index.html) will provide TLS certificates (SSL) for your cluster, using [Let's Encrypt](https://letsencrypt.org/). Install cert-manager into the `cert-manager` namespace: ``` k8s-lbry install-cert-manager ``` ### k8s-lbry install Once nginx-ingress and cert-manager are installed, the main helm chart can be installed. This installs lbrycrd, chainquery, lbrynet, spee.ch, and mysql, depending on what you enable in `values-dev.yaml`. Find the External IP address for your load balancer: ``` k8s-lbry kubectl get svc nginx-ingress-controller -o wide ``` If you find a hostname instead of an IP address, this means your load balancer has multiple IP addresses. In this case, you will need to resolve the domain name to find the IP addresses. If this affects you, run `dig` with the hostname as the second argument. If your system does not have `dig` installed, you can [paste the hostname into this tool](https://toolbox.googleapps.com/apps/dig/). Look for the `;ANSWER` section and you should see two or more IP addresses listed. Since lbrycrd will only advertise one IP address, pick just one of the IP addresses to use for the purposes of this tutorial. You must edit your own `values-dev.yaml`. (The setup procedure created an initial configuration in the same directory as `run.sh`.) To use a different config file, export the `VALUES` environment variable before subsequent commands, specifying the full path to your values file. Edit `values-dev.yaml`. You only need to change one thing right now: * Change `lbrycrd.configurationFile.lbrycrd.conf` at the bottom of this section find `externalip=` and set it equal to the External IP address of the Load Balancer obtained above. (Example: `externalip=123.123.123.123`) Save `values-dev.yaml`. Now run the install script to create the new release: ``` k8s-lbry install ``` ### k8s-lbry upgrade For helm, `upgrade` does not necessarily mean you are upgrading to a new version of any particular software, `upgrade` just means to apply your configuration file to the cluster. If you edit `values-dev.yaml`, you then need to apply your changes with `k8s-lbry upgrade`. You can make changes to `values-dev.yaml` at any time. You can apply your configuration to your cluster by upgrading the release: ``` k8s-lbry upgrade ``` You can upgrade as often as you want. Each time you upgrade the release, helm increases the `REVISION` number: ``` k8s-lbry helm ls ``` ## Services ### lbrycrd After running the installation above, you should now have a running lbrycrd pod. Verify this by listing the pods for the `k8s-lbry` namespace: ``` k8s-lbry kubectl get pods ``` You should see a pod listed with a name that starts with `lbrycrd`. Check the lbrycrd logs: ``` k8s-lbry logs lbrycrd ``` Press Ctrl-C to stop viewing the log. It is advisable to wait for lbrycrd to synchronize with the full blockchain before starting other services, so watch the logs until synchronization completes (`progress=1.0`). You can utilize `lbrycrd-cli` as well: ``` k8s-lbry lbrycrd-cli --help ``` ### chainquery #### MySQL for chainquery [MySQL](https://github.com/helm/charts/tree/master/stable/mysql) is used as the database chainquery talks to. Edit `values-dev.yaml` and set `chainquery-mysql.enabled` to `true`. Upgrade the release to turn on mysql for chainquery: ``` k8s-lbry upgrade ``` You can try logging into the mysql shell if you like: ``` k8s-lbry chainquery-mysql-client ``` You can view the mysql logs: ``` k8s-lbry logs chainquery-mysql ``` Press Ctrl-C to stop viewing the log. #### Start chainquery Edit `values-dev.yaml` and set `chainquery.enabled` to `true`. Upgrade the release to turn on chainquery: ``` k8s-lbry upgrade ``` You can view the chainquery logs: ``` k8s-lbry logs chainquery ``` #### Startup chainquery with a database snapshot If chainquery is starting with a blank MySQL database, it will take several days to synchronize with the full lbrycrd blockchain. If this is OK, you can just watch the chainquery logs and wait for it to get to the [current block height](https://explorer.lbry.io/). If you cannot wait that long, you can scrap your existing chainquery database and restart from a more recent database snapshot: ``` k8s-lbry chainquery-override-snapshot ``` This will prompt if you really wish to destroy the current chainquery database. If you confirm, the existing chainquery and chainquery-mysql deployments will be deleted, and pods will be terminated, ***and the contents of the Persistent Volume Claim (PVC) for chainquery-mysql will be deleted.*** The snapshot will be downloaded and restored in its place. Once the snapshot is restored, upgrade the release to restore the chainquery and chainquery-mysql deployments, and restart pods: ``` k8s-lbry upgrade ``` You can verify that the database now has data up to the height of the database snapshot. Login to the mysql shell: ``` k8s-lbry chainquery-mysql-client ``` Then query for the latest block height: ``` mysql> select height from chainquery.block order by id desc limit 1; +--------+ | height | +--------+ | 561080 | +--------+ 1 row in set (0.00 sec) ``` Also verify that chainquery is again happy. View the chainquery logs: ``` k8s-lbry logs chainquery ``` Press Ctrl-C to quit viewing the logs. ### lbrynet API service (not for spee.ch) This is for a standalone lbrynet API service inside your cluster. Blob storage goes to its own persistent volume, but is configured with `save_files=false`. There is no outside access to the Downloads directory provided. You can stream blobs from lbrynet via `http://lbrynet:5279/get/CLAIM_NAME/CLAIM_ID`. This particular lbrynet configuration won't work for spee.ch (v0.5.12). spee.ch needs direct access to the Downloads directory of lbrynet. **If you are wanting lbrynet for spee.ch, skip this section, and head directly to the [spee.ch section](#speech-and-lbrynet-sidecar-and-mysql), which implements its own lbrynet sidecar.** Edit `values-dev.yaml` and set `lbrynet.enabled` to `true`. Upgrade the release to turn on lbrynet: ``` k8s-lbry upgrade ``` You can view the lbrynet logs: ``` k8s-lbry logs lbrynet ``` #### IMPORTANT - Backup your cluster wallet The wallet is created inside the `lbrynet` persistent volume. Copy the wallet in case the volume gets destroyed: ``` k8s-lbry lbrynet-copy-wallet /tmp/k8s-lbry-lbrynet-wallet-backup.json ``` Check the contents of `/tmp/k8s-lbry-lbrynet-wallet-backup.json` and move the file to a safe place for backup (make sure to delete the temporary file.) Once your wallet is backed up, you can generate a receiving address in order to deposit LBC: ``` k8s-lbry lbrynet address unused ``` ### spee.ch (and lbrynet sidecar and mysql) *Note: Throughout this deployment, the unstylized name `speech` is used.* Speech needs three containers, running in two pods: * `speech` pod: * speech, the nodejs server container. * lbrynet, running in the same pod as speech, so as to share one downloads directory. (This is called a 'sidecar' container, which is guaranteed to run on the same kubernetes node as the spee.ch container.) * `speech-mysql` pod: * mysql for storing the speech database. Edit `values-dev.yaml`. * Set `speech-mysql.enabled` to `true`. * Set `speech.enabled` to `true`. * Set `speech.service.hostname` to your subdomain name for speech. * Set `speech.site.details.host` to your subdomain name for speech. * Set `speech.site.details.ipAddress` to your Load Balancer external IP address. * Set `speech.site.details.title` Upgrade the release to turn on `speech`, `speech-lbrynet`, and `speech-mysql`: ``` k8s-lbry upgrade ``` Speech will not work yet! Continue on through the next sections. #### IMPORTANT - Backup your speech wallet The wallet for speech is created inside the `speech-lbrynet` persistent volume. Copy the wallet in case the volume gets destroyed: ``` k8s-lbry speech-lbrynet-copy-wallet /tmp/k8s-lbry-speech-lbrynet-wallet-backup.json ``` Check the contents of `/tmp/k8s-lbry-speech-lbrynet-wallet-backup.json` and move the file to a safe place for backup (make sure to delete the temporary file.) #### Fund your speech wallet Once your wallet is backed up, you can generate a receiving address in order to deposit LBC: ``` k8s-lbry speech-lbrynet address unused ``` Now send at least 5 LBC to your new speech wallet address. Verify your speech wallet balance: ``` k8s-lbry speech-lbrynet account balance ``` #### Create a thumbnail channel Create the LBRY channel for hosting speech thumbnails. Replace `@YOUR_NAME_HERE` with your chosen (unique) channel name to create. Amount is how much LBC to reserve for the claim: ``` k8s-lbry speech-lbrynet channel new @YOUR_NAME_HERE --amount=1.0 ``` Get the claim id for the channel: ``` k8s-lbry speech-lbrynet channel list ``` The `claim_id` field is your `thumbnailChannelId` used in the next section. #### Finish speech setup Edit `values-dev.yaml` again: * Set `speech.site.publishing.primaryClaimAddress` The fresh wallet address generated above. * Set `speech.site.publishing.thumbnailChannel` The name of the channel to publish thumbnails * Set `speech.site.publishing.thumbnailChannelId` The claim id of the channel to publish thumbnails. (see `k8s-lbry speech-lbrynet channel list`) * Set `speech.site.publishing.serviceOnlyApproved` if you want to limit the channels served. * Set `speech.site.publishing.approvedChannels` if you want to limit the channels served. * Set `speech.site.analytics.googleId` See the [speech settings docs for more info](https://github.com/lbryio/spee.ch/blob/master/docs/settings.md) Upgrade the release to apply the new settings: ``` k8s-lbry upgrade ``` Restart the speech pod: ``` k8s-lbry restart speech ``` ## Extra commands that k8s-lbry (run.sh) provides You can run `k8s-lbry` without any arguments, and it will provide you some help. ### k8s-lbry helm This script encapsulates helm so that it can run it's own local instance of tiller through [helm-tiller](https://github.com/rimusz/helm-tiller). As a convenience function, run.sh can start tiller locally, pass through any helm commands to your cluster, and then shutdown tiller: Example: ``` k8s-lbry helm ls ``` If you try to run `helm` without the `run.sh helm` wrapper, you should expect to see this error: ``` Error: could not find tiller ``` By design, [tiller is not running on your cluster](https://rimusz.net/tillerless-helm), it just runs locally for the duration that `run.sh` needs it, then shuts down. ### k8s-lbry kubectl This script encapsulates kubectl so that you do not have to keep typing `--namespace k8s-lbry` all the time. All kubectl commands will default to `k8s-lbry` or the `NAMESPACE` environment variable if set. Example: ``` k8s-lbry kubectl get pods ``` ### k8s-lbry logs Stream the logs for a pod into your terminal, given the helm app name. If the pod contains more than one container you must specify it as the third argument. Examples: ``` k8s-lbry logs lbrycrd k8s-lbry logs speech speech-lbrynet ``` Press Ctrl-C to stop streaming the logs. If the logs seemingly hang forever, press Ctrl-C and try the command again. ### k8s-lbry shell When doing maintainance tasks, it is often useful to be able to attach a shell to a running pod. This is a convenience wrapper that uses the helm app name to connect to the correct pod. This will connect to the pod running the `lbrynet` service. Example: ``` k8s-lbry shell lbrynet ``` Once in the shell, do whatever maintaince is necessary, and press `Ctrl-D` or type `exit` when done. ### k8s-lbry shell-pvc When doing maintainance tasks, it is often useful to be able to run a utility container that mounts the persistent volume ([PVC](https://kubernetes.io/docs/concepts/storage/persistent-volumes/)) of another container. This is especially useful in scenarios where the pod will not start, and therefore cannot use the `run.sh shell` command in the previous section. This will run a shell in a new utility container, mounting the lbrynet PVC to `/pvcs/lbrynet`. Example: ``` k8s-lbry shell-pvc lbrynet ``` Once in the shell, do whatever maintaince is necessary, and press `Ctrl-D` or type `exit` when done. ### k8s-lbry restart Delete a pod for a given app name. The existing deployment will immediately restart a new pod. Example: ``` k8s-lbry restart speech ``` ### k8s-lbry lbrynet You can run the `lbrynet` client from within the running pod, redirecting output to your local console. Example: ``` k8s-lbry lbrynet --help ``` ### k8s-lbry chainquery-mysql-client Run the mysql shell for the chainquery database. ### k8s-lbry speech-mysql-client Run the mysql shell for the speech database. ### k8s-lbry lbrynet-copy-wallet Backup the lbrynet wallet to a local path. Example: ``` k8s-lbry lbrynet-copy-wallet /tmp/k8s-lbry-lbrynet-wallet-backup.json ``` ### k8s-lbry package This is for the developer of this package to build and maintain the helm package releases, and upload to the S3 package repository. Requires `s3cmd` installed. Example: ``` k8s-lbry package 0.1.1 ``` ## TLS / SSL / HTTPS You have already installed cert-manager for your cluster, but HTTPs is not turned on out of the box. Setup is easy: * You need to create a DNS A record for your domain pointing to the External IP address of the nginx-ingress Load Balancer. (Preferably create a wildcard record for an entire subdomain [`*.example.com` or `*.lbry.example.com`], that way you only have to set this up once, no matter how many sub-domains you end up needing.) Refer to the [install section](https://github.com/EnigmaCurry/lbry-docker/tree/k8s-lbry/contrib/k8s-lbry#k8s-lbry-install) for how to retrieve the IP address. * Edit `values-dev.yaml` * Change `cert-manager-issuer.email` from the example email address to your own. [Let's Encrypt](https://letsencrypt.org/) is a free TLS certificate issuer, and they will send you important emails about your domain and certificate expirations. * You can turn on the echo service to test with: * Change `echo-http-server.enabled` to `true` * Change `echo-http-server.hostname` to a hostname you've configured the DNS for. Upgrade nginx-ingress, turning on HTTPs support (copy the command **including the wrapping parentheses**): ``` ( NGINX_ENABLE_HTTPS=true k8s-lbry upgrade-nginx-ingress ) ``` And Upgrade `k8s-lbry`: ``` k8s-lbry upgrade ``` If you turned on the echo service, try it out with curl: ``` curl -L https://echo.example.com ``` It should return the name of the service: `echo-http-server`. If you get any certificate validation errors, then you may need to wait for up to 20 minutes for the certificate to be issued, and then retry. If you run into problems with certificates, check out the cert-manager logs: ``` kubectl -n cert-manager logs -l app=cert-manager -f ``` Also check the certificate resources: ``` k8s-lbry kubectl get certificates ``` You should see the `echo-http-server-tls` certificate resource listed. The `READY` status is the indicator as to whether the certificate has been issued yet or not. ## Cloud specific notes ### AWS Deployment on AWS requires these modifications: Following the [CDK on AWS](https://www.ubuntu.com/kubernetes/docs/aws-integration) docs, install the StorageClass for EBS: ``` kubectl create -f - <