Merge pull request #31 from lbryio/leopere3

finalized for 11/11 release

chainquery/.env

@@ -3,8 +3,10 @@ COMPOSE_PROJECT_NAME=chainquery
 #########################
 ## Chainquery Settings ##
 #########################
-RPC_USER=lbryrpc                ## Not super necessery to change this.
+## TODO: Test to ensure these vars can be pulled from the lbrycrd .env
-RPC_PASSWORD=changeme           ## Please replace changeme.
+#RPC_USER=lbryrpc               ## Not super necessery to change this.
+#RPC_PASSWORD=changeme          ## Please replace changeme.
+
 RPC_ALLOW_IP=10.5.1.3           ## You're better off not changing this.
 
 #################

chainquery/Dockerfile

@@ -30,14 +30,14 @@ RUN adduser chainquery --gecos GECOS --shell /bin/bash --disabled-password --hom
 COPY --from=0 /download/chainquery /usr/bin
 
 # ADD --chown=1000:1000 https://github.com/lbryio/chainquery/releases/download/latest/chainquery_latest_Linux_x86_64.zip /usr/bin
-COPY chainqueryconfig.toml /etc/chainquery/
+COPY stuff/chainqueryconfig.toml /etc/chainquery/
 
 ## Get latest chainqueryconfig.toml from master on github as template.
 # TODO: add magic which pattern matches for chainqueryconfig.toml DEFAULT strings and replaces them with configured settings on container start.
 # ADD --chown=1000:1000 https://raw.githubusercontent.com/lbryio/chainquery/master/config/default/chainqueryconfig.toml /etc/chainquery/chainqueryconfig.toml.orig
 
 ## Install start.sh as executable script in container $PATH
-COPY start.sh /usr/local/bin/start
+COPY stuff/start.sh /usr/local/bin/start
 
 # TODO: Implement docker-entrypoint if later required this might be handy if we need certain maintainence tasks to be executed in the live container.
 # COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint

chainquery/README.md

@@ -1,3 +1,41 @@
+# Installation instructions for Chainquery and dependencies
+## Preface
+I have set up this tutorial so that you should be able to simply copy and paste the commands into your linux console.  However if you run into issues please be sure to read the detailed expansions on the current and prior steps to ensure you didn't miss a detail.
+
+## Goals of this README.md
+This readme is targeted towards people who need a copy pasta recipe to get Chainquery online and running asap and with some degree of repeatability.  Once this is up and running on your machine you should be able to see what a running instance of Chainquery looks like and start writing apps against it's API calls etc.
+
+## Dependencies
+My goal is to avoid as many dependencies as possible so for now the only pre-requisites you are required to have will be `git`, `docker`, and `docker-compose` for the most part your understanding of the technologies can be superficial so long as you can follow commands and are open to reading a bit you should be fine.
+
+### Docker
+Docker
+Docker is effectively the cross platform software package repository it allows you to ship an entire environment in what's referred to as a container.  Implying that containers hold everything that is needed to ship what's inside effectively from one place to another with a degree of standard that makes it easy for everyone along the way to replicate what is needed for their step in the chain.  
+
+The other side of docker is it brings everything that we would normally have to teach you about the individual components of your soon to be installed Chainquery, Lbrycrd, and MySQL stack and wraps it up nicely in a handfull of easy to follow steps that should result in the same environment on everyones host.
+
+The installation method we recommend is via the `docker.com` website however if your specific operating system's repository versions are at the latest along with docker you should be good to launch with you using whatever instructions you wish.  The version of Docker we used in our deployment for testing this process was `Docker version 17.05.0-ce, build 89658be` however any versions later than this will suffice.  At the writing of this tutorial this was not the latest version of Docker.
+
+#### Just the link to installation instructions please
+Instructions for installing on Ubuntu are at the link that follows:
+https://docs.docker.com/install/linux/docker-ce/ubuntu/
+
+If you plan on using other versions of OS you should at least aim to be a linux base with a x64 CPU and the appropriate minimum version of the dependencies.
+
+### Docker-compose
+Docker Compose's role in this deployment is to get you a fully working cluster of microservices in containers that will deploy Chainquery, MySQL, and LBRYCrd exactly as you would need it to have a reasonable instance for testing / developing on.  You are encouraged to learn how to use the Docker-Compose format but it's not a required prerequisite for getting this running you just need to have it installed successfully.
+
+Install Docker Compose via this guide below, its important if you're using an older version of linux to use the official documentation from Docker.com because you will need the more recent version of docker-compose at least version 3.4 aka 1.22.0 or newer.
+
+#### Just the link to installation instructions please
+https://docs.docker.com/compose/install/
+
+### Git
+For now the recommended install includes grabbing the latest git repository from https://github.com/lbryio/lbry-docker for this you're going to need to install git with the following command.  The amount of git knowledge required for this is ideally fairly minimal.
+
+#### Just the instructions please
+`apt-get install git -y`
+
 ## Get the lbry-docker repository
 
 `sudo git clone https://github.com/lbryio/lbry-docker.git`
@@ -18,11 +56,13 @@ This is only required on host mounted volumes. (the default settings)
 
 `sudo chmod -R 755 ./lbry-docker/lbrycrd/data`
 
+`sudo chown -R 1000:1000 ./lbry-docker/lbrycrd/data`
+
 ## Setup lbrycrd
 
 `cd lbry-docker/lbrycrd`
 
-`docker-compose up -d && docker-compose logs -f`
+`sudo docker-compose up -d && docker-compose logs -f`
 
 Wait for lbrycrd to reach the top of the blockchain (console output should noticeably slow down)
 
@@ -43,8 +83,6 @@ Now that you're done syncing your own copy of the lbry blockchain into the `lbry
 
 ## docker-compose.override.yml
 
-Add a PORTS directive for binding chainquery to the host and then run.
-
 `./quick-bootstrap.sh start`
 
 *OR*

chainquery/db-seed.sha

@@ -1 +0,0 @@
-CHECKSUM=6c456f3c3687fab98b1bb3869578ccae1e5ccfd351f8907b3870d204a21ea82e

chainquery/docker-compose.yml

@@ -22,11 +22,13 @@ services:
       - .env
     expose:
       - 3306
+    ## TODO: I want to find a way that is acceptable to everyone to lock this up
+    ## and not share it with everyone at least eventually.
     ports:
       - 3306:3306
     volumes:
       - ./data/db:/var/lib/mysql
-      - ./my.cnf:/etc/mysql/conf.d/chainquery-optimizations.cnf
+      - ./stuff/my.cnf:/etc/mysql/conf.d/chainquery-optimizations.cnf
 
 ################
 ## Chainquery ##
@@ -40,7 +42,8 @@ services:
       lbrynet:
         ipv4_address: 10.5.1.3
     env_file:
-      - ./.env
+      - .env
+      - ../lbrycrd/.env
     labels:
       - "traefik.expose=false"
     expose:

chainquery/quick-bootstrap.sh

@@ -41,18 +41,37 @@ for i in "${!DEPENDENCIES[@]}"; do
 done
 }
 
-## Add ways to get into and out of a bind here.
+function get_checkpoint() {
+  ## Get DB Checkpoint data.
+  echo Asked to get the latest checkpoint data, downloading latest checkpoint.
+  echo This data is fairly large so this saves you a few days of parsing the LBRY blockchain.
+  docker run -v $(pwd)/:/download --rm leopere/axel-docker http://chainquery-data.s3.amazonaws.com/chainquery-data.zip -o ./chainquery.zip
+}
+
+#################################
+## The real action begins here ##
+#################################
+## TODO: Add ways to get into and out of a bind here.
 case $1 in
   getdata )
-    ## Get DB Checkpoint data.
+    if [[ -f ./chainquery.zip ]]; then
-    echo Asked to get the latest checkpoint data from
+      echo "Found a copy of ./chainquery.zip already in your system."
-    docker run -v $(pwd)/:/download --rm leopere/axel-docker http://chainquery-data.s3.amazonaws.com/chainquery-data.zip -o ./chainquery.zip
+      echo "We recommend that you delete this data before proceeding and grab a fresh copy."
+      QandA "rm -f ./chainquery.zip"
+      get_checkpoint
+    else
+      get_checkpoint
+    fi
     ;;
   extract )
     ## Unpack the data again if need be.
     echo Asked to unpack chainquery.zip if downloaded.
-    # TODO: add some magic here which will check for the presence of chainquery.zip and notify if its already gone.
+    if [[ -f ./chainquery.zip ]]; then
-    docker run -v $(pwd)/:/data --rm leopere/unzip-docker ./chainquery.zip
+      docker run -v $(pwd)/:/data --rm leopere/unzip-docker ./chainquery.zip
+    else
+      echo "Could not extractas chainquery.zip did not exist."
+      echo "Feel free to execute './quick-bootstrap.sh getdata' first next time."
+    fi
     ;;
   cleanup )
     ## Remove any junk here.
@@ -62,25 +81,31 @@ case $1 in
   reset )
     ## Give up on everything and try again.
     ## TODO: Make it very obvious with a nice little Y/N prompt that you're about to trash your settings and start over.
+    echo "Agressively Killing all chainquery and dependency containers."
+    echo "executing: docker-compose kill"
     docker-compose kill
+    echo "Cleaning up stopped containers."
+    echo "executing: docker-compose rm -f"
     docker-compose rm -f
     rm -Rf ./data
     rm -f ./chainquery.zip
-    ## TODO: Consider moving this somewhere as a function.
-    # docker-compose up -d mysql
-    # sleep 30
-    # docker-compose up -d chainquery
     ;;
   start )
     ## Unsupported start command to start containers.
     ## You can use this if you want to start this thing gracefully.
     ## Ideally you would not use this in production.
     echo "Asked to start chainquery gracefully for you."
+    echo "executing: docker-compose up -d mysql"
     docker-compose up -d mysql
     echo "giving mysql some time to establish schema, crypto, users, permissions, and tables"
     sleep 30
     echo "Starting Chainquery"
+    echo "executing: docker-compose up -d chainquery"
     docker-compose up -d chainquery
+    ## TODO: verify chainquery instance is up and healthy, this requires a functional HEALTHCHECK
+    echo "This should have chainquery up and running, currently theres no checks in this function to verify this however."
+    echo "Do feel free to execute 'docker-compose ps' to verify if its running and not restarting or exited."
+    echo "Final Note: You should try to use the docker-compose commands in the tutorial at https://github.com/lbryio/lbry-docker/blob/master/chainquery/README.md"
     ;;
   compress-latest-checkpoint-data )
     ## This is not intended for public use.

chainquery/stuff/chainqueryconfig.toml

@@ -1,3 +1,5 @@
+## TODO: Don't hardcode this stuff for production
+
 #Debug mode outputs specific information to the console
 debugmode=false
 

chainquery/stuff/env-example

@@ -0,0 +1,19 @@
+COMPOSE_PROJECT_NAME=chainquery
+
+#########################
+## Chainquery Settings ##
+#########################
+## TODO: Test to ensure these vars can be pulled from the lbrycrd .env
+#RPC_USER=lbryrpc               ## Not super necessery to change this.
+#RPC_PASSWORD=changeme          ## Please replace changeme.
+
+RPC_ALLOW_IP=10.5.1.3           ## You're better off not changing this.
+
+#################
+## Mysql Creds ##
+#################
+MYSQL_SERVER=10.5.1.10          ## You're better off not changing this.
+MYSQL_USER=changeme             ## This could be changed.
+MYSQL_PASSWORD=changeme         ## This could be set to something random it sets this string for both Mysql's main user and Chainquery's MysqlDSN.
+MYSQL_DATABASE=chainquery       ## This can stay the same.
+MYSQL_ROOT_PASSWORD=changeme    ## Set this to something random and obnoxious we're not using it.

chainquery/stuff/healthcheck.sh

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+## TODO: Implement this at some point it requires CURL in the container.
+curl http://localhost:6300/api/status

lbrycrd/.env

@@ -1,5 +1,14 @@
 COMPOSE_PROJECT_NAME=lbrycrd
+
+#############
+## Lbrycrd ##
+#############
+## TODO: The credentials are a formality but we should randomize these with magic.
 RPC_USER=${RPC_USER=lbryrpc}
 RPC_PASSWORD=${RPC_PASSWORD:-changeme}
+UID=$UID
+GID=$UID
+
 ## This should be the internal container IP from which you'll be calling the RPC for Lbrycrdd from.
+## TODO: make this more dynamic before we move to scalability
 RPC_ALLOW_IP=${RPC_ALLOW_IP:-10.5.1.3}

lbrycrd/Dockerfile

@@ -3,13 +3,12 @@
 FROM ubuntu:18.04
 LABEL MAINTAINER="leopere [at] nixc [dot] us"
 
-RUN adduser lbrycrdd --gecos GECOS --shell /bin/bash --disabled-password --home /data/ && \
+RUN addgroup --gid 1000 lbrycrd && \
+    adduser lbrycrd --uid 1000 --gid 1000 --gecos GECOS --shell /bin/bash --disabled-password --home /data && \
     apt-get update && \
     apt-get -y install unzip wget curl && \
     apt-get autoclean -y && \
-    rm -rf /var/lib/apt/lists/* && \
+    rm -rf /var/lib/apt/lists/*
-    mkdir -p /etc/lbrycrdd && \
-    chown -R 1000:1000 /etc/lbrycrdd
 
 ## TODO: Consider adding debugpaste or variant
 # RUN wget -O /usr/bin/debugpaste https://github.com/nixc-us/debugpaste-it/raw/master/bin/debugpaste_64 && \
@@ -21,17 +20,15 @@ RUN wget -O /usr/bin/lbrycrd-linux.zip https://github.com/lbryio/lbrycrd/release
     rm lbrycrd-linux.zip && \
     chmod +x lbrycrdd lbrycrd-cli lbrycrd-tx
 
-COPY debugpaste-it.sh /usr/local/bin/debugpaste-it
+COPY stuff/debugpaste-it.sh /usr/local/bin/debugpaste-it
-COPY start.sh /usr/local/bin/start
+COPY stuff/start.sh /usr/local/bin/start
-COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint
+COPY stuff/docker-entrypoint.sh /usr/local/bin/docker-entrypoint
-COPY healthcheck.sh /usr/local/bin/healthcheck
+COPY stuff/healthcheck.sh /usr/local/bin/healthcheck
 
-USER lbrycrdd
+# USER lbrycrd
-
+# RUN mkdir /data
-RUN mkdir -p /data/
+VOLUME ["/data"]
-
+WORKDIR /data
-VOLUME /data/
-VOLUME /etc/lbrycrdd/
 
 ## TODO: Implement healthcheck.
 # HEALTHCHECK ["healthcheck"]
@@ -39,6 +36,9 @@ VOLUME /etc/lbrycrdd/
 ## Exposing daemon port and RPC port
 EXPOSE 9245 9246
 
+## TODO: Decide what's important for lbrycrd and possibly add an entrypoint.
+## Maybe catch things that might match things that can be easily executed in the
+## lbrycrd cli and if nothing is entered just default to the containers shell.
 ## For now this is a placeholder that executes /bin/bash on `docker exec`
 # ENTRYPOINT ["docker-entrypoint"]
 

lbrycrd/compile/Dockerfile

@@ -2,7 +2,8 @@
 ## Compiler/Builder Step ##
 ###########################
 FROM ubuntu:18.04 as builder
-LABEL MAINTAINER="@Brannon Update Me Or Put Mine Here"
+## TODO: Decide who gets the MAINTAINER label
+## LABEL MAINTAINER="@Brannon Update Me Or Put Mine Here"
 
 ## Added apt work as single container layer then remove cruft before layer is committed.
 RUN apt-get update && \

lbrycrd/compile/start.sh

@@ -8,17 +8,17 @@
 # ## This is specific to chainquery.
 
 ## Ensure perms are correct prior to running main binary
+mkdir -p /data/.lbrycrd
 chown -R 1000:1000 /data
 chmod -R 755 /data
-chown -R 1000:1000 /etc/lbrycrdd
+# chown -R 1000:1000 /etc/lbrycrdd
-chmod -R 755 /etc/lbrycrdd
+# chmod -R 755 /etc/lbrycrdd
 rm -f /var/run/lbrycrdd.pid
-mkdir -p ~/.lbrycrd
 
 ## Set config params
-echo -e "rpcuser=lbryrpc\nrpcpassword=${RPC_PASSWORD:-changeme}" > ~/.lbrycrd/lbrycrd.conf
+echo 'rpcuser=lbryrpc\nrpcpassword='${RPC_PASSWORD:-changeme} > ~/.lbrycrd/lbrycrd.conf
-echo -e "rpcallowip=${RPC_ALLOW_IP:-10.5.1.3}" >> ~/.lbrycrd/lbrycrd.conf
+echo "rpcallowip=${RPC_ALLOW_IP:-10.5.1.3}" >> ~/.lbrycrd/lbrycrd.conf
-echo -e "rpcuser=${RPC_USER:-lbryrpc}" >> ~/.lbrycrd/lbrycrd.conf
+echo "rpcuser=${RPC_USER:-lbryrpc}" >> ~/.lbrycrd/lbrycrd.conf
 
 ## Control this invocation through envvar.
 case ${RUN_MODE:-default} in

lbrycrd/docker-compose.yml

@@ -16,7 +16,6 @@ services:
         ipv4_address: 10.5.1.2
     labels:
       - "traefik.expose=false"
-    user: "${UID:-1000}:${GID:-1000}"
     environment:
       RUN_MODE: chainquery
     expose:
@@ -24,4 +23,4 @@ services:
       - 9246
     ## host volumes for persistent data such as wallet private keys.
     volumes:
-      - ./data:/data
+      - "./data:/data"

lbrycrd/stuff/env-example

@@ -0,0 +1,12 @@
+COMPOSE_PROJECT_NAME=lbrycrd
+
+#############
+## Lbrycrd ##
+#############
+## TODO: The credentials are a formality but we should randomize these with magic.
+RPC_USER=${RPC_USER=lbryrpc}
+RPC_PASSWORD=${RPC_PASSWORD:-changeme}
+
+## This should be the internal container IP from which you'll be calling the RPC for Lbrycrdd from.
+## TODO: make this more dynamic before we move to scalability
+RPC_ALLOW_IP=${RPC_ALLOW_IP:-10.5.1.3}

lbrycrd/stuff/start.sh

@@ -7,43 +7,32 @@
 # ## This is specific to chainquery.
 
 ## Ensure perms are correct prior to running main binary
-chown -R 1000:1000 /data
+mkdir -p /data/.lbrycrd
-chmod -R 755 /data
+chown -R lbrycrd:lbrycrd /data
+chmod -R 755 /data/
 
 ## TODO: Consider a config directory for future magic.
 # chown -R 1000:1000 /etc/lbrycrd
 # chmod -R 755 /etc/lbrycrd
 rm -f /var/run/lbrycrd.pid
-mkdir -p ~/.lbrycrd
+
 
 ## Set config params
 ## TODO: Make this more automagic in the future.
-echo -e "rpcuser=lbryrpc\nrpcpassword=${RPC_PASSWORD:-changeme}" > $HOME/.lbrycrd/lbrycrd.conf
+echo "rpcuser=lbryrpc\nrpcpassword=${RPC_PASSWORD:-changeme}" > /data/.lbrycrd/lbrycrd.conf
-echo -e "rpcallowip=${RPC_ALLOW_IP:-10.5.1.3}" >> $HOME/.lbrycrd/lbrycrd.conf
+echo "rpcallowip=${RPC_ALLOW_IP:-10.5.1.3}" >> /data/.lbrycrd/lbrycrd.conf
-echo -e "rpcuser=${RPC_USER:-lbryrpc}" >> $HOME/.lbrycrd/lbrycrd.conf
+echo "rpcuser=${RPC_USER:-lbryrpc}" >> /data/.lbrycrd/lbrycrd.conf
 
 ## Control this invocation through envvar.
 case ${RUN_MODE:-default} in
   default )
-    lbrycrdd \
+    su -c "lbrycrdd -server -conf=/data/.lbrycrd/ -printtoconsole" lbrycrd
-      -server \
-      -conf=$HOME.lbrycrd/lbrycrd.conf \
-      -printtoconsole
     ;;
   reindex )
-    lbrycrdd \
+    su -c "lbrycrdd -server -txindex -reindex -conf=/data/.lbrycrd/ -printtoconsole" lbrycrd
-      -server \
-      -txindex \
-      -reindex \
-      -conf=$HOME.lbrycrd/lbrycrd.conf \
-      -printtoconsole
     ;;
   chainquery )
-    lbrycrdd \
+    su -c "lbrycrdd -server -txindex -conf=/data/.lbrycrd/ -printtoconsole" lbrycrd
-      -server \
-      -txindex \
-      -conf=$HOME.lbrycrd/lbrycrd.conf \
-      -printtoconsole
     ;;
 esac