Chainquery works mostly and so does lbrycrd

Merge branch 'master' of github.com:lbryio/lbry-docker

Starting on a .travis.yml but may be unnecessary

Renamed www.spee.ch and lbrynet-daemon to reflect their upstream repos.

Disabled Adminer
Started debug for chainquery container.

Get latest release tagged url from github API

remove zip.zip maybe?

debugging curl

explicit unzip

Adding -o for unzip

simplified, removed debug and fixed curl

reverted to original unzip function

Pull chainquery from prep container

passing alternative config path
Aligned config path location with linux spec

updated config location

maybe don't be as explicit

adding debug for chainqueryconfig.toml

Fixes magicless chainquery and starts on bootstrap
added chainquery/.gitignore to ignore large blobs.
fixed Dockerfile up to use staged prep and production
removed db-seed.sh as we have quick-bootstrap.sh
start.sh needed modification to pull config from the right location in the linux FS
Added chainquery/.dockerignore to prevent extremely long build times in the future
Removed fancy bash vars in chainquery/.env
Started work on getting quick-bootstrap.sh ready for release.

Switch to lbrynet network

Switched network name to lbrynet

Merge branch 'leopere' of github.com:lbryio/lbry-docker into leopere

Remove debugging access port bind

Expose chainquery to the internet

Merge branch 'leopere' of github.com:lbryio/lbry-docker into leopere

Removed now unnecessary run command

Undo

Cleanup and commenting on environment variables

Start getting ready for adding chainquery start.sh magic

Quick update
This commit is contained in:
Leopere 2018-11-04 22:42:35 -05:00
parent 7aab2d0ceb
commit 3c9fcf6fce
30 changed files with 175 additions and 99 deletions

14
.travis.yml Normal file
View file

@ -0,0 +1,14 @@
# os: linux
# dist: trusty
#
# services:
# - docker
#
# before_install:
# - docker build -t leopere/POTATO .
# # - docker run -d -p 127.0.0.1:80:4567 carlad/sinatra /bin/sh -c "cd /root/sinatra; bundle exec foreman start;"
# - docker ps -a
# - docker run carlad/sinatra /bin/sh -c "cd /root/sinatra; bundle exec rake test"
#
# script:
# - bundle exec rake test

View file

@ -30,7 +30,8 @@ For now I don't recommend using this container cluster however you're welcomed t
## Running from source
Running this stuff from source should be possible if you have both [Docker](https://docs.docker.com/install/) and [docker-compose](https://docs.docker.com/compose/install/) both installed. If these are both installed you can proceed to run the following from within your development directory.
``` git clone https://github.com/lbryio/lbry-docker.git
```
git clone https://github.com/lbryio/lbry-docker.git
```
Once you have a local copy of the recent source you will want to consider what containers/applications you require in your environment. At the moment since at the writing of this documentation this comes with an assertion of [YMMV](https://dictionary.cambridge.org/dictionary/english/ymmv) so if something isn't working feel free to make suggestions in the form of a PR for how we should do this better. The beauty of Open Source is learning better ways to do things as well as contributing to the world so I'm always going to be welcoming to contributions.
@ -38,7 +39,7 @@ Once you have a local copy of the recent source you will want to consider what c
Running from source for contributing and Merge/Pull requests.
My goal is to make contributing to this possible using Docker and also GitLab CI/CD time.
## [Contributing]()
## [Contributing](CONTRIBUTING.md)
Keep in mind [I am](https://github.com/leopere/) preferential to receiving patches over rule following as we can always nudge you in the right direction to get things more compatible with the project ethos if it's not. Never be afraid to file a PR no one should be offended. This said following the next two guides will greatly improve the speed at which we can integrate your improvements.
* [Repository Standards]( https://lbry.tech/resources/repository-standards)
* [Contribute](https://lbry.tech/contribute)

View file

@ -1 +1,3 @@
data/
data.z*
chainquery.z*

View file

@ -1,10 +1,17 @@
COMPOSE_PROJECT_NAME=chainquery
RPC_USER=${RPC_USER=lbryrpc}
RPC_PASSWORD=${RPC_PASSWORD:-changeme}
## This should be the internal container IP from which you'll be calling the RPC for Lbrycrdd from.
RPC_ALLOW_IP=${RPC_ALLOW_IP:-10.5.1.3}
MYSQL_SERVER=${MYSQL_SERVER:-10.5.1.10}
MYSQL_USER=${MYSQL_USER:-changeme}
MYSQL_PASSWORD=${MYSQL_PASSWORD:-changeme}
MYSQL_DATABASE=${MYSQL_DATABASE:-chainquery}
MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD:-changeme}
#########################
## Chainquery Settings ##
#########################
RPC_USER=lbryrpc ## Not super necessery to change this.
RPC_PASSWORD=changeme ## Please replace changeme.
RPC_ALLOW_IP=10.5.1.3 ## You're better off not changing this.
#################
## Mysql Creds ##
#################
MYSQL_SERVER=10.5.1.10 ## You're better off not changing this.
MYSQL_USER=changeme ## This could be changed.
MYSQL_PASSWORD=changeme ## This could be set to something random it sets this string for both Mysql's main user and Chainquery's MysqlDSN.
MYSQL_DATABASE=chainquery ## This can stay the same.
MYSQL_ROOT_PASSWORD=changeme ## Set this to something random and obnoxious we're not using it.

3
chainquery/.gitignore vendored Normal file
View file

@ -0,0 +1,3 @@
data/
data.z*
chainquery.z*

View file

@ -3,14 +3,18 @@ FROM ubuntu:18.04 as prep
LABEL MAINTAINER="leopere [at] nixc [dot] us"
RUN apt-get update && \
apt-get -y install unzip && \
apt-get -y install unzip curl && \
apt-get autoclean -y && \
rm -rf /var/lib/apt/lists/*
rm -rf /var/lib/apt/lists/* && \
mkdir /download
ADD --chown=1000:1000 https://github.com/lbryio/chainquery/releases/download/v1.1.2/chainquery_1.1.2_Linux_x86_64.zip /download
RUN cd /download/ && \
unzip chainquery_1.1.2_Linux_x86_64.zip && \
rm chainquery_1.1.2_Linux_x86_64.zip
RUN curl -L -o /download/chainquery.zip $(curl -s https://api.github.com/repos/lbryio/chainquery/releases | fgrep 'Linux_x86_64.zip' | grep download | head -n 1 | cut -d'"' -f4) && \
cd /download && \
ls -lAh /download && pwd && \
unzip chainquery.zip && \
rm chainquery.zip
## I really want to use this instead of all the curl stuff at one point.
# ADD --chown=1000:1000 https://github.com/lbryio/chainquery/releases/download/latest/chainquery_latest_Linux_x86_64.zip /download
## Switch to Alpine eventually we might need to compile in Alpine for this.
@ -18,7 +22,11 @@ FROM ubuntu:18.04 as app
RUN adduser chainquery --gecos GECOS --shell /bin/bash --disabled-password --home /home/chainquery
COPY chainqueryconfig.toml /data/chainqueryconfig.toml
COPY --from=0 /download/chainquery /usr/bin
# ADD --chown=1000:1000 https://github.com/lbryio/chainquery/releases/download/latest/chainquery_latest_Linux_x86_64.zip /usr/bin
COPY chainqueryconfig.toml /etc/chainquery/
ADD --chown=1000:1000 https://raw.githubusercontent.com/lbryio/chainquery/master/config/default/chainqueryconfig.toml /etc/chainquery/chainqueryconfig.toml.orig
COPY start.sh /usr/local/bin/start
COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint

View file

@ -1,28 +0,0 @@
#!/usr/bin/env bash
source ./db-seed.sha
echo "Checking if seeding is needed"
if [[ -d !./data/db/chainquery ]]; then
echo "It appears as though you don't currently have the db created."
echo "Downloading the Chainquery DB checkpoint data."
wget -O data.zip https://s3bucketURL/here.zip
if [[ -f !./data.zip ]]; then
echo "It seems that downloading the checkpoint data failed."
else
echo "Checkpoint data received verifying download integrity."
if ! echo "$CHECKSUM data.zip" | sha256sum -c -; then
echo "Checksum failed, somehow the checkpoint data doesn't match what it's supposed to." >&2
exit 1
else
echo "clearing ./data directory in case it contains something strange."
echo "Uncompressing chainquery checkpoint data."
rm -Rf ./data
unzip ./data.zip
if [[ -d !./data/db/chainquery ]]; then
echo "Something went wrong with uncompressing checkpoint data."
exit 1
else
echo "Checkpoint data has been successfully obtained you can now run the Chainquery appliance."
fi
fi
fi
fi

View file

@ -1,7 +1,7 @@
version: '3.4'
networks:
traefik:
lbrynet:
external: true
services:
@ -14,49 +14,18 @@ services:
image: mysql:5.7.23
restart: always
networks:
traefik:
lbrynet:
ipv4_address: 10.5.1.10
aliases:
- mysql
environment:
## These variables are stored in the .env file next to this docker-compose.yml file.
## I will include a default .env file and .gitignore the ".env" pattern so you should be able to just git pull in the future if you need to.
MYSQL_SERVER: ${MYSQL_SERVER:-10.5.1.10}
MYSQL_USER: ${MYSQL_USER:-changeme}
MYSQL_PASSWORD: ${MYSQL_PASSWORD:-changeme}
MYSQL_DATABASE: ${MYSQL_DATABASE:-chainquery}
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD:-changeme}
env_file:
- .env
expose:
- 3306
volumes:
- ./data/db:/var/lib/mysql
- ./my.cnf:/etc/mysql/conf.d/chainquery-optimizations.cnf
#######################
## Adminer container ##
#######################
adminer:
image: adminer
# restart: always
links:
- "mysql:database"
depends_on:
- mysql
labels:
# https://docs.traefik.io/user-guide/docker-and-lets-encrypt/
- "traefik.backend=adminer"
- "traefik.docker.network=lbrynet"
- "traefik.frontend.rule=Host:adminer.lbry-demo.nixc.us"
- "traefik.expose=true"
- "traefik.port=8080"
# healthcheck:
# test: ["CMD", "curl", "--fail", "http://localhost:8080/", "||", "exit", "1"]
networks:
traefik:
ipv4_address: 10.5.1.19
aliases:
- adminer
################
## Chainquery ##
################
@ -66,11 +35,15 @@ services:
target: app
restart: always
networks:
traefik:
lbrynet:
ipv4_address: 10.5.1.3
env_file:
- ./.env
labels:
- "traefik.expose=false"
expose:
- 6300
ports:
- 6300:6300
depends_on:
- mysql

72
chainquery/quick-bootstrap.sh Executable file
View file

@ -0,0 +1,72 @@
#!/usr/bin/env bash
## Be Polite and ask for permission.
function QandA() {
read -r -p "Continue with $1 [y/N] " response
response=${response,,} # tolower
if [[ "$response" =~ ^(yes|y)$ ]]; then
echo "Continuing with this."
eval $1
else
echo "Skipping the $1."
fi
}
## Check your $PATH for required dependencies.
## Stop and complain for now, later automagically install them.
function test_for_deps() {
## Test for Command
if ! [ -x "$(command -v $1)" ]; then
echo "Error: $1 is not installed." >&2
echo "You must have $1 installed."
else
echo "Info: $1 is installed."
fi
}
## Declare Linux app dependencies to check for.
DEPENDENCIES=(
wget
unzip
docker
)
## Recommended.
BONUS_DEPENDENCIES=(
docker-compose
)
function check_deps() {
for i in "${!DEPENDENCIES[@]}"; do
echo ${DEPENDENCIES[$i]}"_KEY"
## Indirect references http://tldp.org/LDP/abs/html/ivr.html
eval TESTDEP=\$"${DEPENDENCIES[$i]}"
test_for_deps $TESTDEP
done
}
## Add ways to get into and out of a bind here.
case $1 in
getdata )
## Get DB Checkpoint data.
axel -a -n 6 http://chainquery-data.s3.amazonaws.com/data.zip -o ./chainquery.zip
;;
extract )
## Unpack the data again if need be.
unzip ./chainquery.zip
;;
cleanup )
## Remove any junk here.
rm chainquery.zip
;;
reset )
## Give up on everything and try again.
# rm -Rf ./data
# rm -f ./chainquery.zip
;;
* )
echo "=================================================="
echo "You look like you need usage examples let me help."
echo "=================================================="
echo "Add documentation on script params HERE"
;;
esac

View file

@ -1,4 +1,28 @@
#!/usr/bin/env bash
#########################
## Chainquery Settings ##
#########################
RPC_USER=lbryrpc ## Not super necessery to change this.
RPC_PASSWORD=changeme ## Please replace changeme.
RPC_ALLOW_IP=10.5.1.3 ## You're better off not changing this.
#################
## Mysql Creds ##
#################
MYSQL_SERVER=10.5.1.10 ## You're better off not changing this.
MYSQL_USER=changeme ## This could be changed.
MYSQL_PASSWORD=changeme ## This could be set to something random it sets this string for both Mysql's main user and Chainquery's MysqlDSN.
MYSQL_DATABASE=chainquery ## This can stay the same.
MYSQL_ROOT_PASSWORD=changeme ## Set this to something random and obnoxious we're not using it.
sed -i ''
debugmode=${DEBUGMODE:-false}
lbrycrdurl="rpc://${RPC_USER:-lbryrpc}:${RPC_PASSWORD:-changeme}@10.5.1.2:9245"
mysqldsn="${MYSQL_USER:-changeme}:${MYSQL_PASSWORD:-changeme}@tcp(${MYSQL_SERVER:-10.5.1.10}:3306)/${MYSQL_DATABASE:-chainquery}"
apimysqldsn="${MYSQL_USER:-changeme}:${MYSQL_PASSWORD:-changeme}@tcp(${MYSQL_SERVER:-10.5.1.10}:3306)/${MYSQL_DATABASE:-chainquery}"
## For now keeping this simple. Potentially eventually add all command args as envvars for the Dockerfile or use safe way to add args via docker-compose.yml
chainquery serve
chainquery serve -c "/etc/chainquery/"

View file

@ -1,7 +1,7 @@
version: '3.4'
networks:
traefik:
lbrynet:
external: true
services:
@ -12,7 +12,7 @@ services:
build: .
restart: always
networks:
traefik:
lbrynet:
ipv4_address: 10.5.1.2
labels:
- "traefik.expose=false"

View file

@ -1,7 +1,7 @@
version: '3.4'
networks:
traefik:
lbrynet:
external: true
services:
@ -12,7 +12,7 @@ services:
build: .
restart: always
networks:
traefik:
lbrynet:
ipv4_address: 10.5.1.4
labels:
- "traefik.expose=false"

View file

@ -1,7 +1,7 @@
version: '3.4'
networks:
traefik:
lbrynet:
external: true
services:
@ -14,7 +14,7 @@ services:
image: mysql.5.7.23
restart: always
networks:
traefik:
lbrynet:
ipv4_address: 10.5.1.11
aliases:
- mysql
@ -38,7 +38,7 @@ services:
build: .
restart: always
networks:
traefik:
lbrynet:
ipv4_address: 10.5.1.20
environment:
AWS_ID: ${AWS_ID:-default}

View file

@ -1,7 +1,7 @@
version: '3.4'
networks:
traefik:
lbrynet:
external: true
services:
@ -14,7 +14,7 @@ services:
image: mysql.5.7.23
restart: always
networks:
traefik:
lbrynet:
ipv4_address: 10.5.1.11
aliases:
- mysql
@ -40,7 +40,7 @@ services:
build: .
restart: always
networks:
traefik:
lbrynet:
ipv4_address: 10.5.1.20
env_file:
- .env

View file

@ -10,7 +10,7 @@ This is still a WIP and documentation may be wildly out of sync with reality.
- [Your changes take priority](#your-changes-take-priority)
- [Configuration](#configuration)
- [Try it out immediately](#try-it-out-immediately)
- [Docker & Docker-compose](#docker-docker-compose)
- [Docker & Docker-compose](#docker--docker-compose)
<!-- /TOC -->

View file

@ -30,7 +30,7 @@ services:
#############
## Lbrynet ##
#############
traefik:
lbrynet:
build:
context: ../lbrynet-daemon/
dockerfile: Dockerfile

View file

@ -26,7 +26,7 @@ services:
#############
## Lbrynet ##
#############
traefik:
lbrynet:
build:
context: ../lbrynet-daemon/
dockerfile: Dockerfile

View file

@ -1,7 +1,7 @@
version: '3.4'
networks:
traefik:
lbrynet:
external: true
services:
@ -14,7 +14,7 @@ services:
image: traefik:alpine
command: --api --docker --docker.domain=docker.localhost --logLevel=DEBUG
networks:
traefik:
lbrynet:
ipv4_address: 10.5.1.100
ports:
- target: 80
@ -36,7 +36,7 @@ services:
labels:
# https://docs.traefik.io/user-guide/docker-and-lets-encrypt/
- "traefik.backend=proxy"
- "traefik.docker.network=traefik"
- "traefik.docker.network=lbrynet"
## Make sure that you either set the .env variable included in this repository or replace the domain string below.
## Also your DNS record will need to exist pointing at the machine your Traefik Reverse Proxy will be hosted on before you launch this app cluster.
- "traefik.frontend.rule=Host:monitor.${DOMAIN}"