Chainquery works mostly and so does lbrycrd

Merge branch 'master' of github.com:lbryio/lbry-docker

Starting on a .travis.yml but may be unnecessary

Renamed www.spee.ch and lbrynet-daemon to reflect their upstream repos.

Disabled Adminer
Started debug for chainquery container.

Get latest release tagged url from github API

remove zip.zip maybe?

debugging curl

explicit unzip

Adding -o for unzip

simplified, removed debug and fixed curl

reverted to original unzip function

Pull chainquery from prep container

passing alternative config path
Aligned config path location with linux spec

updated config location

maybe don't be as explicit

adding debug for chainqueryconfig.toml

Fixes magicless chainquery and starts on bootstrap
added chainquery/.gitignore to ignore large blobs.
fixed Dockerfile up to use staged prep and production
removed db-seed.sh as we have quick-bootstrap.sh
start.sh needed modification to pull config from the right location in the linux FS
Added chainquery/.dockerignore to prevent extremely long build times in the future
Removed fancy bash vars in chainquery/.env
Started work on getting quick-bootstrap.sh ready for release.

Switch to lbrynet network

Switched network name to lbrynet

Merge branch 'leopere' of github.com:lbryio/lbry-docker into leopere

Remove debugging access port bind

Expose chainquery to the internet

Merge branch 'leopere' of github.com:lbryio/lbry-docker into leopere

Removed now unnecessary run command

Undo

Cleanup and commenting on environment variables

Start getting ready for adding chainquery start.sh magic

Quick update

.travis.yml

@@ -0,0 +1,14 @@
+# os: linux
+# dist: trusty
+#
+# services:
+#  - docker
+#
+# before_install:
+#  - docker build -t leopere/POTATO .
+#  # - docker run -d -p 127.0.0.1:80:4567 carlad/sinatra /bin/sh -c "cd /root/sinatra; bundle exec foreman start;"
+#  - docker ps -a
+#  - docker run carlad/sinatra /bin/sh -c "cd /root/sinatra; bundle exec rake test"
+#
+# script:
+#  - bundle exec rake test

README.md

@@ -30,7 +30,8 @@ For now I don't recommend using this container cluster however you're welcomed t
 
 ## Running from source
 Running this stuff from source should be possible if you have both [Docker](https://docs.docker.com/install/) and [docker-compose](https://docs.docker.com/compose/install/) both installed.  If these are both installed you can proceed to run the following from within your development directory.
-``` git clone https://github.com/lbryio/lbry-docker.git
+```
+git clone https://github.com/lbryio/lbry-docker.git
 ```
 Once you have a local copy of the recent source you will want to consider what containers/applications you require in your environment.  At the moment since at the writing of this documentation this comes with an assertion of [YMMV](https://dictionary.cambridge.org/dictionary/english/ymmv) so if something isn't working feel free to make suggestions in the form of a PR for how we should do this better.  The beauty of Open Source is learning better ways to do things as well as contributing to the world so I'm always going to be welcoming to contributions.
 
@@ -38,7 +39,7 @@ Once you have a local copy of the recent source you will want to consider what c
 Running from source for contributing and Merge/Pull requests.
 My goal is to make contributing to this possible using Docker and also GitLab CI/CD time.  
 
-## [Contributing]()
+## [Contributing](CONTRIBUTING.md)
 Keep in mind [I am](https://github.com/leopere/) preferential to receiving patches over rule following as we can always nudge you in the right direction to get things more compatible with the project ethos if it's not.  Never be afraid to file a PR no one should be offended.  This said following the next two guides will greatly improve the speed at which we can integrate your improvements.
 * [Repository Standards]( https://lbry.tech/resources/repository-standards)
 * [Contribute](https://lbry.tech/contribute)

chainquery/.dockerignore

@@ -1 +1,3 @@
 data/
+data.z*
+chainquery.z*

chainquery/.env

@@ -1,10 +1,17 @@
 COMPOSE_PROJECT_NAME=chainquery
-RPC_USER=${RPC_USER=lbryrpc}
+
-RPC_PASSWORD=${RPC_PASSWORD:-changeme}
+#########################
-## This should be the internal container IP from which you'll be calling the RPC for Lbrycrdd from.
+## Chainquery Settings ##
-RPC_ALLOW_IP=${RPC_ALLOW_IP:-10.5.1.3}
+#########################
-MYSQL_SERVER=${MYSQL_SERVER:-10.5.1.10}
+RPC_USER=lbryrpc                ## Not super necessery to change this.
-MYSQL_USER=${MYSQL_USER:-changeme}
+RPC_PASSWORD=changeme           ## Please replace changeme.
-MYSQL_PASSWORD=${MYSQL_PASSWORD:-changeme}
+RPC_ALLOW_IP=10.5.1.3           ## You're better off not changing this.
-MYSQL_DATABASE=${MYSQL_DATABASE:-chainquery}
+
-MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD:-changeme}
+#################
+## Mysql Creds ##
+#################
+MYSQL_SERVER=10.5.1.10          ## You're better off not changing this.
+MYSQL_USER=changeme             ## This could be changed.
+MYSQL_PASSWORD=changeme         ## This could be set to something random it sets this string for both Mysql's main user and Chainquery's MysqlDSN.
+MYSQL_DATABASE=chainquery       ## This can stay the same.
+MYSQL_ROOT_PASSWORD=changeme    ## Set this to something random and obnoxious we're not using it.

chainquery/.gitignore

@@ -0,0 +1,3 @@
+data/
+data.z*
+chainquery.z*

chainquery/Dockerfile

@@ -3,14 +3,18 @@ FROM ubuntu:18.04 as prep
 LABEL MAINTAINER="leopere [at] nixc [dot] us"
 
 RUN apt-get update && \
-    apt-get -y install unzip && \
+    apt-get -y install unzip curl && \
     apt-get autoclean -y && \
-    rm -rf /var/lib/apt/lists/*
+    rm -rf /var/lib/apt/lists/* && \
+    mkdir /download
 
-ADD --chown=1000:1000 https://github.com/lbryio/chainquery/releases/download/v1.1.2/chainquery_1.1.2_Linux_x86_64.zip /download
+RUN curl -L -o /download/chainquery.zip $(curl -s https://api.github.com/repos/lbryio/chainquery/releases | fgrep 'Linux_x86_64.zip' | grep download | head -n 1 | cut -d'"' -f4) && \
-RUN cd /download/ && \
+    cd /download && \
-    unzip chainquery_1.1.2_Linux_x86_64.zip && \
+    ls -lAh /download && pwd && \
-    rm chainquery_1.1.2_Linux_x86_64.zip
+    unzip chainquery.zip && \
+    rm chainquery.zip
+## I really want to use this instead of all the curl stuff at one point.
+# ADD --chown=1000:1000 https://github.com/lbryio/chainquery/releases/download/latest/chainquery_latest_Linux_x86_64.zip /download
 
 
 ## Switch to Alpine eventually we might need to compile in Alpine for this.
@@ -18,7 +22,11 @@ FROM ubuntu:18.04 as app
 
 RUN adduser chainquery --gecos GECOS --shell /bin/bash --disabled-password --home /home/chainquery
 
-COPY chainqueryconfig.toml /data/chainqueryconfig.toml
+COPY --from=0 /download/chainquery /usr/bin
+
+# ADD --chown=1000:1000 https://github.com/lbryio/chainquery/releases/download/latest/chainquery_latest_Linux_x86_64.zip /usr/bin
+COPY chainqueryconfig.toml /etc/chainquery/
+ADD --chown=1000:1000 https://raw.githubusercontent.com/lbryio/chainquery/master/config/default/chainqueryconfig.toml /etc/chainquery/chainqueryconfig.toml.orig
 COPY start.sh /usr/local/bin/start
 COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint
 

chainquery/db-seed.sh

@@ -1,28 +0,0 @@
-#!/usr/bin/env bash
-source ./db-seed.sha
-echo "Checking if seeding is needed"
-if [[ -d !./data/db/chainquery ]]; then
-  echo "It appears as though you don't currently have the db created."
-  echo "Downloading the Chainquery DB checkpoint data."
-  wget -O data.zip https://s3bucketURL/here.zip
-  if [[ -f !./data.zip ]]; then
-    echo "It seems that downloading the checkpoint data failed."
-  else
-    echo "Checkpoint data received verifying download integrity."
-    if ! echo "$CHECKSUM data.zip" | sha256sum -c -; then
-      echo "Checksum failed, somehow the checkpoint data doesn't match what it's supposed to." >&2
-      exit 1
-    else
-      echo "clearing ./data directory in case it contains something strange."
-      echo "Uncompressing chainquery checkpoint data."
-      rm -Rf ./data
-      unzip ./data.zip
-      if [[ -d !./data/db/chainquery ]]; then
-        echo "Something went wrong with uncompressing checkpoint data."
-        exit 1
-      else
-        echo "Checkpoint data has been successfully obtained you can now run the Chainquery appliance."
-      fi
-    fi
-  fi
-fi

chainquery/docker-compose.yml

@@ -1,7 +1,7 @@
 version: '3.4'
 
 networks:
-  traefik:
+  lbrynet:
     external: true
 
 services:
@@ -14,49 +14,18 @@ services:
     image: mysql:5.7.23
     restart: always
     networks:
-      traefik:
+      lbrynet:
         ipv4_address: 10.5.1.10
         aliases:
           - mysql
-    environment:
+    env_file:
-      ## These variables are stored in the .env file next to this docker-compose.yml file.
+      - .env
-      ## I will include a default .env file and .gitignore the ".env" pattern so you should be able to just git pull in the future if you need to.
-      MYSQL_SERVER: ${MYSQL_SERVER:-10.5.1.10}
-      MYSQL_USER: ${MYSQL_USER:-changeme}
-      MYSQL_PASSWORD: ${MYSQL_PASSWORD:-changeme}
-      MYSQL_DATABASE: ${MYSQL_DATABASE:-chainquery}
-      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD:-changeme}
     expose:
       - 3306
     volumes:
       - ./data/db:/var/lib/mysql
       - ./my.cnf:/etc/mysql/conf.d/chainquery-optimizations.cnf
 
-#######################
-## Adminer container ##
-#######################
-  adminer:
-    image: adminer
-    # restart: always
-    links:
-      - "mysql:database"
-    depends_on:
-      - mysql
-    labels:
-      # https://docs.traefik.io/user-guide/docker-and-lets-encrypt/
-      - "traefik.backend=adminer"
-      - "traefik.docker.network=lbrynet"
-      - "traefik.frontend.rule=Host:adminer.lbry-demo.nixc.us"
-      - "traefik.expose=true"
-      - "traefik.port=8080"
-    # healthcheck:
-    #   test: ["CMD", "curl", "--fail", "http://localhost:8080/", "||", "exit", "1"]
-    networks:
-      traefik:
-        ipv4_address: 10.5.1.19
-        aliases:
-          - adminer
-
 ################
 ## Chainquery ##
 ################
@@ -66,11 +35,15 @@ services:
       target: app
     restart: always
     networks:
-      traefik:
+      lbrynet:
         ipv4_address: 10.5.1.3
+    env_file:
+      - ./.env
     labels:
       - "traefik.expose=false"
     expose:
       - 6300
+    ports:
+      - 6300:6300
     depends_on:
       - mysql

chainquery/quick-bootstrap.sh

@@ -0,0 +1,72 @@
+#!/usr/bin/env bash
+
+## Be Polite and ask for permission.
+function QandA() {
+  read -r -p "Continue with $1 [y/N] " response
+  response=${response,,}    # tolower
+  if [[ "$response" =~ ^(yes|y)$ ]]; then
+    echo "Continuing with this."
+    eval $1
+  else
+    echo "Skipping the $1."
+  fi
+}
+
+## Check your $PATH for required dependencies.
+## Stop and complain for now, later automagically install them.
+function test_for_deps() {
+  ## Test for Command
+  if ! [ -x "$(command -v $1)" ]; then
+    echo "Error: $1 is not installed." >&2
+    echo "You must have $1 installed."
+  else
+    echo "Info: $1 is installed."
+  fi
+}
+
+## Declare Linux app dependencies to check for.
+DEPENDENCIES=(
+  wget
+  unzip
+  docker
+)
+## Recommended.
+BONUS_DEPENDENCIES=(
+  docker-compose
+)
+
+function check_deps() {
+for i in "${!DEPENDENCIES[@]}"; do
+  echo ${DEPENDENCIES[$i]}"_KEY"
+  ## Indirect references http://tldp.org/LDP/abs/html/ivr.html
+  eval TESTDEP=\$"${DEPENDENCIES[$i]}"
+  test_for_deps $TESTDEP
+done
+}
+
+## Add ways to get into and out of a bind here.
+case $1 in
+  getdata )
+    ## Get DB Checkpoint data.
+    axel -a -n 6 http://chainquery-data.s3.amazonaws.com/data.zip -o ./chainquery.zip
+    ;;
+  extract )
+    ## Unpack the data again if need be.
+    unzip ./chainquery.zip
+    ;;
+  cleanup )
+    ## Remove any junk here.
+    rm chainquery.zip
+    ;;
+  reset )
+    ## Give up on everything and try again.
+    # rm -Rf ./data
+    # rm -f ./chainquery.zip
+    ;;
+  * )
+    echo "=================================================="
+    echo "You look like you need usage examples let me help."
+    echo "=================================================="
+    echo "Add documentation on script params HERE"
+    ;;
+esac

chainquery/start.sh

@@ -1,4 +1,28 @@
 #!/usr/bin/env bash
 
+#########################
+## Chainquery Settings ##
+#########################
+RPC_USER=lbryrpc                ## Not super necessery to change this.
+RPC_PASSWORD=changeme           ## Please replace changeme.
+RPC_ALLOW_IP=10.5.1.3           ## You're better off not changing this.
+
+#################
+## Mysql Creds ##
+#################
+MYSQL_SERVER=10.5.1.10          ## You're better off not changing this.
+MYSQL_USER=changeme             ## This could be changed.
+MYSQL_PASSWORD=changeme         ## This could be set to something random it sets this string for both Mysql's main user and Chainquery's MysqlDSN.
+MYSQL_DATABASE=chainquery       ## This can stay the same.
+MYSQL_ROOT_PASSWORD=changeme    ## Set this to something random and obnoxious we're not using it.
+
+sed -i '' 
+
+
+debugmode=${DEBUGMODE:-false}
+lbrycrdurl="rpc://${RPC_USER:-lbryrpc}:${RPC_PASSWORD:-changeme}@10.5.1.2:9245"
+mysqldsn="${MYSQL_USER:-changeme}:${MYSQL_PASSWORD:-changeme}@tcp(${MYSQL_SERVER:-10.5.1.10}:3306)/${MYSQL_DATABASE:-chainquery}"
+apimysqldsn="${MYSQL_USER:-changeme}:${MYSQL_PASSWORD:-changeme}@tcp(${MYSQL_SERVER:-10.5.1.10}:3306)/${MYSQL_DATABASE:-chainquery}"
+
 ## For now keeping this simple. Potentially eventually add all command args as envvars for the Dockerfile or use safe way to add args via docker-compose.yml
-chainquery serve
+chainquery serve -c "/etc/chainquery/"

lbrycrd/docker-compose.yml

@@ -1,7 +1,7 @@
 version: '3.4'
 
 networks:
-  traefik:
+  lbrynet:
     external: true
 
 services:
@@ -12,7 +12,7 @@ services:
     build: .
     restart: always
     networks:
-      traefik:
+      lbrynet:
         ipv4_address: 10.5.1.2
     labels:
       - "traefik.expose=false"

lbrynet/docker-compose.yml

@@ -1,7 +1,7 @@
 version: '3.4'
 
 networks:
-  traefik:
+  lbrynet:
     external: true
 
 services:
@@ -12,7 +12,7 @@ services:
     build: .
     restart: always
     networks:
-      traefik:
+      lbrynet:
         ipv4_address: 10.5.1.4
     labels:
       - "traefik.expose=false"

reflector.go/compile/docker-compose.yml

@@ -1,7 +1,7 @@
 version: '3.4'
 
 networks:
-  traefik:
+  lbrynet:
     external: true
 
 services:
@@ -14,7 +14,7 @@ services:
     image: mysql.5.7.23
     restart: always
     networks:
-      traefik:
+      lbrynet:
         ipv4_address: 10.5.1.11
         aliases:
           - mysql
@@ -38,7 +38,7 @@ services:
     build: .
     restart: always
     networks:
-      traefik:
+      lbrynet:
         ipv4_address: 10.5.1.20
     environment:
       AWS_ID: ${AWS_ID:-default}

reflector.go/docker-compose.yml

@@ -1,7 +1,7 @@
 version: '3.4'
 
 networks:
-  traefik:
+  lbrynet:
     external: true
 
 services:
@@ -14,7 +14,7 @@ services:
     image: mysql.5.7.23
     restart: always
     networks:
-      traefik:
+      lbrynet:
         ipv4_address: 10.5.1.11
         aliases:
           - mysql
@@ -40,7 +40,7 @@ services:
     build: .
     restart: always
     networks:
-      traefik:
+      lbrynet:
         ipv4_address: 10.5.1.20
     env_file:
       - .env

spee.ch/README.md

@@ -10,7 +10,7 @@ This is still a WIP and documentation may be wildly out of sync with reality.
 			- [Your changes take priority](#your-changes-take-priority)
 			- [Configuration](#configuration)
 - [Try it out immediately](#try-it-out-immediately)
-	- [Docker & Docker-compose](#docker-docker-compose)
+	- [Docker & Docker-compose](#docker--docker-compose)
 
 <!-- /TOC -->
 

spee.ch/advanced-docker-compose.yml

@@ -30,7 +30,7 @@ services:
 #############
 ## Lbrynet ##
 #############
-  traefik:
+  lbrynet:
     build:
       context: ../lbrynet-daemon/
       dockerfile: Dockerfile

spee.ch/docker-compose.yml

@@ -26,7 +26,7 @@ services:
 #############
 ## Lbrynet ##
 #############
-  traefik:
+  lbrynet:
     build:
       context: ../lbrynet-daemon/
       dockerfile: Dockerfile

traefik/docker-compose.yml

@@ -1,7 +1,7 @@
 version: '3.4'
 
 networks:
-  traefik:
+  lbrynet:
     external: true
 
 services:
@@ -14,7 +14,7 @@ services:
     image: traefik:alpine
     command: --api --docker --docker.domain=docker.localhost --logLevel=DEBUG
     networks:
-      traefik:
+      lbrynet:
         ipv4_address: 10.5.1.100
     ports:
       - target: 80
@@ -36,7 +36,7 @@ services:
     labels:
       # https://docs.traefik.io/user-guide/docker-and-lets-encrypt/
       - "traefik.backend=proxy"
-      - "traefik.docker.network=traefik"
+      - "traefik.docker.network=lbrynet"
       ## Make sure that you either set the .env variable included in this repository or replace the domain string below.
       ## Also your DNS record will need to exist pointing at the machine your Traefik Reverse Proxy will be hosted on before you launch this app cluster.
       - "traefik.frontend.rule=Host:monitor.${DOMAIN}"