lbry-docker/contrib/k8s-lbry

LBRY on Kubernetes with Helm

Contributing Author: EnigmaCurry

Last Update: May 6 2019

Deploy lbrycrd, lbrynet, chainquery, mysql, and spee.ch on your Kubernetes cluster.

• Requirements
• Helm Charts
  • Tiller
  • nginx-ingress
  • cert-manager
  • k8s-lbry
  • lbrycrd
  • chainquery
    • MySQL for chainquery
    • Start chainquery
    • Startup chainquery with a database snapshot
  • lbrynet
    • IMPORTANT - Backup your cluster wallet
  • spee.ch
    • MySQL for speech
    • Configure Speech
• TLS Support
  • Assign DNS name(s) to your Load Balancer
  • Enable TLS
• Improvements

Requirements

• A Kubernetes cluster with role-based access control (RBAC) enabled.
  • This tutorial was tested on a fresh DigitalOcean managed cluster on nodes with 8GB of RAM, on kubernetes 1.13.5.
• kubectl command line tool installed on your local development machine.
  • Tested with kubectl v1.14.0
• Helm command line tool installed on your local development machine.
  • Tested with helm v2.13.1

Your cloud provider should have instructions for setting up kubectl to talk to your cluster. This usually involves downloading a config file and putting it in $HOME/.kube/config. (The file has to be renamed config and put in the $HOME/.kube directory.)

Note: If you want to download the cluster config to a location other than $HOME/.kube/config, you can set the KUBECONFIG environment variable to the full path of your config file, or create a symlink from your config file to $HOME/.kube/config, or you can use the --kubeconfig parameter to both kubectl and helm commands every time you use them.

Test that your kubectl can talk to your cluster, by querying for a list of running nodes:

kubectl get nodes

If everything is working, you should see a list of one or more nodes running and showing STATUS=Ready

Helm Charts

This system is installed via Helm, the package manager for Kubernetes. Helm Charts are the basis for packages in Helm. This directory is a Helm chart itself.

Tiller

Tiller is the cluster-side component of helm, and needs to be installed before you can use helm with your cluster. Run the following to install tiller to your cluster:

kubectl -n kube-system create serviceaccount tiller

kubectl create clusterrolebinding tiller --clusterrole cluster-admin \
      --serviceaccount=kube-system:tiller

helm init --service-account tiller
helm repo update

Now you can use helm locally to install things to your remote cluster.

nginx-ingress

An Ingress Controller (nginx-ingress) will help you to route outside internet traffic into your cluster. nginx-ingress will also help terminate TLS connections (SSL) so that your containers don't need to worry about encryption.

Install nginx-ingress, with HTTPs turned off initially:

helm install stable/nginx-ingress --name nginx-ingress \
  --set nginx-ingress.controller.service.enableHttps=false

cert-manager

cert-manager will provide TLS certificates (SSL) for your cluster, using Let's Encrypt.

Install cert-manager:

kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/00-crds.yaml

helm repo add jetstack https://charts.jetstack.io
helm repo update

helm install --name cert-manager --namespace cert-manager jetstack/cert-manager --version v0.7.1

kubectl label namespace cert-manager certmanager.k8s.io/disable-validation="true"

k8s-lbry

The k8s-lbry helm chart installs lbrycrd, chainquery, lbrynet, and mysql.

Wait for the Load Balancer to show an External IP:

kubectl get svc -l app=nginx-ingress,component=controller -w

Press Ctrl-C to quit once you see the External IP listed (and not <pending>).

Add the k8s-lbry helm repository:

helm repo add k8s-lbry https://k8s-lbry.sfo2.digitaloceanspaces.com

Create a directory to store your configuration file for k8s-lbry. You can download the default configuration file for the helm chart (values.yaml):

VALUES=https://raw.githubusercontent.com/lbryio/lbry-docker/master/contrib/k8s-lbry/values.yaml

curl -Lo values.yaml $VALUES

values.yaml is your own configuration file for k8s-lbry. You will need it everytime you need to update your deployment. Commit the file to a git repository, or save it someplace safe.

Edit values.yaml, change the following things:

• Change lbrycrd.configurationFile.lbrycrd.conf at the bottom find externalip= and set it equal to the External IP address of the Load Balancer obtained above.
• Change cert-manager-issuer.email to your email address to receive notices from Let's Encrypt. (Only used if you choose to enable TLS.)
• Change echo-http-server.hostname to any domain name you choose. (It must be a real internet domain that you control, if you choose to enable TLS.)

Save values.yaml.

Now install k8s-lbry:

helm install -n k8s-lbry k8s-lbry/k8s-lbry -f values.yaml

This will create a new helm release for your cluster called k8s-lbry, from the helm repository called k8s-lbry, using the package named k8s-lbry, using the local configuration file called values.yaml.

lbrycrd

Find the lbrycrd pod to ensure it has started correctly:

kubectl get pods -l app=lbrycrd

Tail the logs (Press Ctrl-C to quit):

kubectl logs -f -l app=lbrycrd

You can use lbrycrd-cli from the running pod:

POD=`kubectl get pod -l app=lbrycrd -o name | sed s+pod/++` && \
  kubectl exec $POD -- lbrycrd-cli -rpcuser=lbry -rpcpassword=lbry getinfo

Upgrade the nginx-ingress release to allow forwarding port 9246 to lbrycrd:

helm upgrade nginx-ingress stable/nginx-ingress \
  --set tcp.9246="default/k8s-lbry-lbrycrd:9246"

Verify the port is now open (9246 listed under PORTS):

kubectl get svc nginx-ingress-controller

After your lbrycrd service has been online for awhile, check back with the lbrcrd-cli getinfo command from above. You will know that nginx-ingress is properly connected to lbrycrd if you see that the number of connections listed is a number greater than 8.

chainquery

MySQL for chainquery

MySQL is used as the database chainquery talks to.

Edit values.yaml and set chainquery-mysql.enabled to true.

Upgrade the release to turn on mysql for chainquery:

helm upgrade k8s-lbry k8s-lbry/k8s-lbry -f values.yaml

You can try logging into the mysql shell if you like (default password is chainquery):

POD=`kubectl get pod -l app=k8s-lbry-chainquery-mysql -o name | sed s+pod/++` && \
  kubectl exec -it $POD -- mysql -u chainquery -p

You can view the mysql logs:

kubectl logs -l app=k8s-lbry-chainquery-mysql -f

Start chainquery

Edit values.yaml and set chainquery.enabled to true.

Upgrade the release to turn on chainquery:

helm upgrade k8s-lbry k8s-lbry/k8s-lbry -f values.yaml

You can view the chainquery logs:

kubectl logs -l app=chainquery -f

Startup chainquery with a database snapshot

If chainquery is starting with a blank MySQL database, it will take several days to synchronize with the full lbycrd blockchain. If this is OK, you can just watch the chainquery logs and wait for it to get to the current block height.

If you cannot wait that long, you may start from a database snapshot to speed up this process.

Delete the chainquery and mysql deployments:

kubectl delete deployments k8s-lbry-chainquery k8s-lbry-chainquery-mysql

The pods will automatically terminate.

The mysql data still exists in a PersistentVolumeClaim, k8s-lbry-chainquery-mysql. Check that it still exists:

kubectl get pvc

There's an included script to start a utility container with a PersistentVolume attached. Download the script:

SCRIPT=https://raw.githubusercontent.com/lbryio/lbry-docker/master/contrib/k8s-lbry/scripts/kubectl-run-with-pvc.sh

curl -Lo kubectl-run-with-pvc.sh $SCRIPT && chmod a+x kubectl-run-with-pvc.sh

Run the kubectl-run-with-pvc script, attaching the mysql PVC:

./kubectl-run-with-pvc.sh k8s-lbry-chainquery-mysql

Wait a second for the container to start, and you should then be placed into a container shell, indicated by the shell prompt changing to the container's prompt.

In the container shell, delete any existing mysql data from the volume:

rm /pvcs/k8s-lbry-chainquery-mysql/* -rf

Still in the container shell, download the backup and extract it to the volume:

apt update && apt install -y curl

BACKUP_URL=https://lbry-chainquery-mysql-dump.sfo2.digitaloceanspaces.com/chainquery_height_560900.mysql-backup.tar.gz
curl $BACKUP_URL | tar xvz -C /pvcs/k8s-lbry-chainquery-mysql/

Once the download and extraction completes, exit the container (or just press Ctrl-D):

exit

Now back on your local shell, upgrade the release to re-create the mysql and chainquery deployments:

helm upgrade k8s-lbry k8s-lbry/k8s-lbry -f values.yaml

You can verify that the database now has data up to the height of the database snapshot. Login to the mysql shell (password: chainquery):

POD=`kubectl get pod -l app=k8s-lbry-chainquery-mysql -o name | sed s+pod/++` && \
  kubectl exec -it $POD -- mysql -u chainquery -p

Then query for the number of blocks:

mysql> select count(*) from chainquery.block;
+----------+
| count(*) |
+----------+
|   561034 |
+----------+
1 row in set (15.00 sec)

Also verify that chainquery is again happy. View the chainquery logs:

kubectl logs -l app=chainquery -f

lbrynet

Edit values.yaml and set lbrynet.enabled to true.

Update the release to turn on lbrynet:

helm upgrade k8s-lbry k8s-lbry/k8s-lbry -f values.yaml

You can view the lbrynet logs:

kubectl logs -l app=lbrynet -f

IMPORTANT - Backup your cluster wallet

The wallet is stored inside the k8s-lbry-lbrynet persistent volume.

Copy the wallet in case the volume gets destroyed:

WALLET=/home/lbrynet/.local/share/lbry/lbryum/wallets/default_wallet \
POD=`kubectl get pod -l app=lbrynet -o name | sed s+pod/++` && \
  kubectl cp $POD:$WALLET /tmp/k8s-lbry-lbrynet-wallet-backup.json

Check the contents of /tmp/k8s-lbry-lbrynet-wallet-backup.json and move the file to a safe place for backup (and delete this temporary file.)

Once your wallet is backed up, you can generate a receiving address in order to deposit LBC:

POD=`kubectl get pod -l app=lbrynet -o name | sed s+pod/++` && \
  kubectl exec $POD -- lbrynet address unused

spee.ch

Note: Throughout this deployment, the unstylized name speech is used.

MySQL for speech

MySQL is used as the database speech talks to.

Edit values.yaml and set speech-mysql.enabled to true.

Upgrade the release to turn on mysql for speech:

helm upgrade k8s-lbry k8s-lbry/k8s-lbry -f values.yaml

You can try logging into the mysql shell if you like (default password is speech):

POD=`kubectl get pod -l app=k8s-lbry-speech-mysql -o name | sed s+pod/++` && \
  kubectl exec -it $POD -- mysql -u speech -p

You can view the mysql logs:

kubectl logs -l app=k8s-lbry-speech-mysql -f

Configure Speech

Before you can fully configure speech, you must fund your lbrynet wallet in the k8s-lbry-lbrynet deployment. Check the lbrynet section for details on generating a receiving address for your wallet, as well as backing up your wallet.

Speech has a large configuration, all of which is found in values.yaml. The most important settings to configure yourself are:

• speech.enabled - turns on/off the the speech deployment.

• speech.service.hostname - The external hostname for speech.

• speech.persistence.size - How large of a data directory for speech.

• speech.auth.masterPassword

• speech.details

• speech.publishing.primaryClaimAddress

  • This can be retrieved from the lbrynet pod:

POD=`kubectl get pod -l app=lbrynet -o name | sed s+pod/++` && \
  kubectl exec $POD -- lbrynet address list

• Copy the first address from the list. This is your primaryClaimAddress.

• speech.publishing.publishOnlyApproved

• speech.publishing.approvedChannels

• speech.publishing.thumbnailChannel

  • In order to publish thumbnails, you must create a channel. There are many options in creation. See the help from the lbrynet command to list them all:

POD=`kubectl get pod -l app=lbrynet -o name | sed s+pod/++` && \
  kubectl exec $POD -- lbrynet channel create --help

• For example, this will create the channel named YourChannel, bidding 1 LBC for the name:

POD=`kubectl get pod -l app=lbrynet -o name | sed s+pod/++` && \
  kubectl exec $POD -- lbrynet channel create --name @YourChannel --bid 1.0

• Make sure that when you copy the channel name to values.yaml that you use double quotes surrounding the value for thumbnailChannel. This is because in YAML, the @ symbol cannot be used without quotes. ie: thumbnailChannel: "@YourChannel"

• speech.publishing.thumbnailChannelId

  • When you create the channel, listed in the outputs section, you will find claim_id; this is the thumbnailChannelId. You can also retrieve this information again by running channel list:

POD=`kubectl get pod -l app=lbrynet -o name | sed s+pod/++` && \ 
  kubectl exec $POD -- lbrynet channel list

Once you've configured speech in values.yaml, upgrade the helm release to apply the changes:

helm upgrade k8s-lbry k8s-lbry/k8s-lbry -f values.yaml

Open your browser to the hostname specified in speech.service.hostname and demo the site.

TLS Support

Enabling TLS (SSL) for your cluster is optional, but it is useful if you are going to expose any HTTP services externally.

Assign DNS name(s) to your Load Balancer

The k8s-lbry chart started a Load Balancer as part of the Ingress Controller. You can assign a DNS name to the Load Balancer External IP address.

Get the External IP of the Load Balancer:

kubectl get svc -l app=nginx-ingress,component=controller

Copy the External IP address shown. Update your DNS provider for your domain accordingly, with a subdomain of your choice to point to the External IP address.

Edit values.yaml and set echo-service.enabled to true. Set echo-service.hostname to the domain name you configued in your DNS.

Upgrade the release to turn on the echo-http-server:

helm upgrade k8s-lbry k8s-lbry/k8s-lbry -f values.yaml

Verify that the DNS is setup correctly by using curl to the echo-http-server on port 80:

curl http://echo.example.com

(Replace echo.example.com with the domain you used in values.yaml.)

You should see the word echo returned.

Enable TLS

Once you've verified that DNS for your domain correctly routes to the echo-http-server, upgrade the nginx-ingress release with HTTPs now turned on:

helm upgrade nginx-ingress stable/nginx-ingress \
  --set nginx-ingress.controller.service.enableHttps=true

Upgrade the k8s-lbry release, turning on HTTPs for the echo-http-server:

helm upgrade k8s-lbry k8s-lbry/k8s-lbry -f values.yaml --set echo-http-server.enableHttps=true

Check that HTTPs connection to the echo service is working:

curl https://echo.example.com

(Replace echo.example.com with the domain you used in values.yaml.)

You should see the word echo returned. However, it may take up to 5 minutes for it to start to work.

Watch the cert-manager log:

kubectl logs --namespace cert-manager -l app=cert-manager -f

A successful certificate message would look like:

Certificate "echo-tls" for ingress "echo" is up to date

Retry the curl command until you get an echo response.

Improvements

Beyond this point, there are several things one could do to improve this configuration and harden for production.

• Secrets

  • At this stage, all your configuration resides in values.yaml, including passwords. You can seperate these secrets out of your config and put them into a Kubernetes Secret.

  • Sealed Secrets

  • Helm Secrets

• Namespaces

  • If you are using the cluster for things other than lbry, you should install k8s-lbry into its own namespace. This will allow pods within the same namespace to talk to eachother, but not to pods in other namespaces.

  • Using a namespace in the introductory docs above, would have complicated the (already complex) helm and kubectl commands, so they were omitted.

  • Both helm and kubectl support the --namespace argument. You can translate all the commands above, adding the --namespace argument.

    For example, to install the k8s-lbry chart in its own k8s-lbry namespace:

    ## helm install RELEASE REPO/CHART --namespace NAMESPACE -f VALUES
    helm install k8s-lbry k8s-lbry/k8s-lbry --namespace k8s-lbry -f values.yaml
    

    And to look at pods in the k8s-lbry namespace:

    kubectl get pods --namespace k8s-lbry