Merge pull request #31 from lbryio/leopere3

finalized for 11/11 release
This commit is contained in:
Leopere 2018-11-10 01:09:53 +00:00 committed by GitHub
commit c96f058af0
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
24 changed files with 165 additions and 64 deletions

View file

@ -3,8 +3,10 @@ COMPOSE_PROJECT_NAME=chainquery
#########################
## Chainquery Settings ##
#########################
RPC_USER=lbryrpc ## Not super necessery to change this.
RPC_PASSWORD=changeme ## Please replace changeme.
## TODO: Test to ensure these vars can be pulled from the lbrycrd .env
#RPC_USER=lbryrpc ## Not super necessery to change this.
#RPC_PASSWORD=changeme ## Please replace changeme.
RPC_ALLOW_IP=10.5.1.3 ## You're better off not changing this.
#################

View file

@ -30,14 +30,14 @@ RUN adduser chainquery --gecos GECOS --shell /bin/bash --disabled-password --hom
COPY --from=0 /download/chainquery /usr/bin
# ADD --chown=1000:1000 https://github.com/lbryio/chainquery/releases/download/latest/chainquery_latest_Linux_x86_64.zip /usr/bin
COPY chainqueryconfig.toml /etc/chainquery/
COPY stuff/chainqueryconfig.toml /etc/chainquery/
## Get latest chainqueryconfig.toml from master on github as template.
# TODO: add magic which pattern matches for chainqueryconfig.toml DEFAULT strings and replaces them with configured settings on container start.
# ADD --chown=1000:1000 https://raw.githubusercontent.com/lbryio/chainquery/master/config/default/chainqueryconfig.toml /etc/chainquery/chainqueryconfig.toml.orig
## Install start.sh as executable script in container $PATH
COPY start.sh /usr/local/bin/start
COPY stuff/start.sh /usr/local/bin/start
# TODO: Implement docker-entrypoint if later required this might be handy if we need certain maintainence tasks to be executed in the live container.
# COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint

View file

@ -1,3 +1,41 @@
# Installation instructions for Chainquery and dependencies
## Preface
I have set up this tutorial so that you should be able to simply copy and paste the commands into your linux console. However if you run into issues please be sure to read the detailed expansions on the current and prior steps to ensure you didn't miss a detail.
## Goals of this README.md
This readme is targeted towards people who need a copy pasta recipe to get Chainquery online and running asap and with some degree of repeatability. Once this is up and running on your machine you should be able to see what a running instance of Chainquery looks like and start writing apps against it's API calls etc.
## Dependencies
My goal is to avoid as many dependencies as possible so for now the only pre-requisites you are required to have will be `git`, `docker`, and `docker-compose` for the most part your understanding of the technologies can be superficial so long as you can follow commands and are open to reading a bit you should be fine.
### Docker
Docker
Docker is effectively the cross platform software package repository it allows you to ship an entire environment in what's referred to as a container. Implying that containers hold everything that is needed to ship what's inside effectively from one place to another with a degree of standard that makes it easy for everyone along the way to replicate what is needed for their step in the chain.
The other side of docker is it brings everything that we would normally have to teach you about the individual components of your soon to be installed Chainquery, Lbrycrd, and MySQL stack and wraps it up nicely in a handfull of easy to follow steps that should result in the same environment on everyones host.
The installation method we recommend is via the `docker.com` website however if your specific operating system's repository versions are at the latest along with docker you should be good to launch with you using whatever instructions you wish. The version of Docker we used in our deployment for testing this process was `Docker version 17.05.0-ce, build 89658be` however any versions later than this will suffice. At the writing of this tutorial this was not the latest version of Docker.
#### Just the link to installation instructions please
Instructions for installing on Ubuntu are at the link that follows:
https://docs.docker.com/install/linux/docker-ce/ubuntu/
If you plan on using other versions of OS you should at least aim to be a linux base with a x64 CPU and the appropriate minimum version of the dependencies.
### Docker-compose
Docker Compose's role in this deployment is to get you a fully working cluster of microservices in containers that will deploy Chainquery, MySQL, and LBRYCrd exactly as you would need it to have a reasonable instance for testing / developing on. You are encouraged to learn how to use the Docker-Compose format but it's not a required prerequisite for getting this running you just need to have it installed successfully.
Install Docker Compose via this guide below, its important if you're using an older version of linux to use the official documentation from Docker.com because you will need the more recent version of docker-compose at least version 3.4 aka 1.22.0 or newer.
#### Just the link to installation instructions please
https://docs.docker.com/compose/install/
### Git
For now the recommended install includes grabbing the latest git repository from https://github.com/lbryio/lbry-docker for this you're going to need to install git with the following command. The amount of git knowledge required for this is ideally fairly minimal.
#### Just the instructions please
`apt-get install git -y`
## Get the lbry-docker repository
`sudo git clone https://github.com/lbryio/lbry-docker.git`
@ -18,11 +56,13 @@ This is only required on host mounted volumes. (the default settings)
`sudo chmod -R 755 ./lbry-docker/lbrycrd/data`
`sudo chown -R 1000:1000 ./lbry-docker/lbrycrd/data`
## Setup lbrycrd
`cd lbry-docker/lbrycrd`
`docker-compose up -d && docker-compose logs -f`
`sudo docker-compose up -d && docker-compose logs -f`
Wait for lbrycrd to reach the top of the blockchain (console output should noticeably slow down)
@ -43,8 +83,6 @@ Now that you're done syncing your own copy of the lbry blockchain into the `lbry
## docker-compose.override.yml
Add a PORTS directive for binding chainquery to the host and then run.
`./quick-bootstrap.sh start`
*OR*

View file

@ -1 +0,0 @@
CHECKSUM=6c456f3c3687fab98b1bb3869578ccae1e5ccfd351f8907b3870d204a21ea82e

View file

@ -22,11 +22,13 @@ services:
- .env
expose:
- 3306
## TODO: I want to find a way that is acceptable to everyone to lock this up
## and not share it with everyone at least eventually.
ports:
- 3306:3306
volumes:
- ./data/db:/var/lib/mysql
- ./my.cnf:/etc/mysql/conf.d/chainquery-optimizations.cnf
- ./stuff/my.cnf:/etc/mysql/conf.d/chainquery-optimizations.cnf
################
## Chainquery ##
@ -40,7 +42,8 @@ services:
lbrynet:
ipv4_address: 10.5.1.3
env_file:
- ./.env
- .env
- ../lbrycrd/.env
labels:
- "traefik.expose=false"
expose:

View file

@ -41,18 +41,37 @@ for i in "${!DEPENDENCIES[@]}"; do
done
}
## Add ways to get into and out of a bind here.
function get_checkpoint() {
## Get DB Checkpoint data.
echo Asked to get the latest checkpoint data, downloading latest checkpoint.
echo This data is fairly large so this saves you a few days of parsing the LBRY blockchain.
docker run -v $(pwd)/:/download --rm leopere/axel-docker http://chainquery-data.s3.amazonaws.com/chainquery-data.zip -o ./chainquery.zip
}
#################################
## The real action begins here ##
#################################
## TODO: Add ways to get into and out of a bind here.
case $1 in
getdata )
## Get DB Checkpoint data.
echo Asked to get the latest checkpoint data from
docker run -v $(pwd)/:/download --rm leopere/axel-docker http://chainquery-data.s3.amazonaws.com/chainquery-data.zip -o ./chainquery.zip
if [[ -f ./chainquery.zip ]]; then
echo "Found a copy of ./chainquery.zip already in your system."
echo "We recommend that you delete this data before proceeding and grab a fresh copy."
QandA "rm -f ./chainquery.zip"
get_checkpoint
else
get_checkpoint
fi
;;
extract )
## Unpack the data again if need be.
echo Asked to unpack chainquery.zip if downloaded.
# TODO: add some magic here which will check for the presence of chainquery.zip and notify if its already gone.
docker run -v $(pwd)/:/data --rm leopere/unzip-docker ./chainquery.zip
if [[ -f ./chainquery.zip ]]; then
docker run -v $(pwd)/:/data --rm leopere/unzip-docker ./chainquery.zip
else
echo "Could not extractas chainquery.zip did not exist."
echo "Feel free to execute './quick-bootstrap.sh getdata' first next time."
fi
;;
cleanup )
## Remove any junk here.
@ -62,25 +81,31 @@ case $1 in
reset )
## Give up on everything and try again.
## TODO: Make it very obvious with a nice little Y/N prompt that you're about to trash your settings and start over.
echo "Agressively Killing all chainquery and dependency containers."
echo "executing: docker-compose kill"
docker-compose kill
echo "Cleaning up stopped containers."
echo "executing: docker-compose rm -f"
docker-compose rm -f
rm -Rf ./data
rm -f ./chainquery.zip
## TODO: Consider moving this somewhere as a function.
# docker-compose up -d mysql
# sleep 30
# docker-compose up -d chainquery
;;
start )
## Unsupported start command to start containers.
## You can use this if you want to start this thing gracefully.
## Ideally you would not use this in production.
echo "Asked to start chainquery gracefully for you."
echo "executing: docker-compose up -d mysql"
docker-compose up -d mysql
echo "giving mysql some time to establish schema, crypto, users, permissions, and tables"
sleep 30
echo "Starting Chainquery"
echo "executing: docker-compose up -d chainquery"
docker-compose up -d chainquery
## TODO: verify chainquery instance is up and healthy, this requires a functional HEALTHCHECK
echo "This should have chainquery up and running, currently theres no checks in this function to verify this however."
echo "Do feel free to execute 'docker-compose ps' to verify if its running and not restarting or exited."
echo "Final Note: You should try to use the docker-compose commands in the tutorial at https://github.com/lbryio/lbry-docker/blob/master/chainquery/README.md"
;;
compress-latest-checkpoint-data )
## This is not intended for public use.

View file

@ -1,3 +1,5 @@
## TODO: Don't hardcode this stuff for production
#Debug mode outputs specific information to the console
debugmode=false

View file

@ -0,0 +1,19 @@
COMPOSE_PROJECT_NAME=chainquery
#########################
## Chainquery Settings ##
#########################
## TODO: Test to ensure these vars can be pulled from the lbrycrd .env
#RPC_USER=lbryrpc ## Not super necessery to change this.
#RPC_PASSWORD=changeme ## Please replace changeme.
RPC_ALLOW_IP=10.5.1.3 ## You're better off not changing this.
#################
## Mysql Creds ##
#################
MYSQL_SERVER=10.5.1.10 ## You're better off not changing this.
MYSQL_USER=changeme ## This could be changed.
MYSQL_PASSWORD=changeme ## This could be set to something random it sets this string for both Mysql's main user and Chainquery's MysqlDSN.
MYSQL_DATABASE=chainquery ## This can stay the same.
MYSQL_ROOT_PASSWORD=changeme ## Set this to something random and obnoxious we're not using it.

View file

@ -0,0 +1,3 @@
#!/usr/bin/env bash
## TODO: Implement this at some point it requires CURL in the container.
curl http://localhost:6300/api/status

View file

@ -1,5 +1,14 @@
COMPOSE_PROJECT_NAME=lbrycrd
#############
## Lbrycrd ##
#############
## TODO: The credentials are a formality but we should randomize these with magic.
RPC_USER=${RPC_USER=lbryrpc}
RPC_PASSWORD=${RPC_PASSWORD:-changeme}
UID=$UID
GID=$UID
## This should be the internal container IP from which you'll be calling the RPC for Lbrycrdd from.
## TODO: make this more dynamic before we move to scalability
RPC_ALLOW_IP=${RPC_ALLOW_IP:-10.5.1.3}

View file

@ -3,13 +3,12 @@
FROM ubuntu:18.04
LABEL MAINTAINER="leopere [at] nixc [dot] us"
RUN adduser lbrycrdd --gecos GECOS --shell /bin/bash --disabled-password --home /data/ && \
RUN addgroup --gid 1000 lbrycrd && \
adduser lbrycrd --uid 1000 --gid 1000 --gecos GECOS --shell /bin/bash --disabled-password --home /data && \
apt-get update && \
apt-get -y install unzip wget curl && \
apt-get autoclean -y && \
rm -rf /var/lib/apt/lists/* && \
mkdir -p /etc/lbrycrdd && \
chown -R 1000:1000 /etc/lbrycrdd
rm -rf /var/lib/apt/lists/*
## TODO: Consider adding debugpaste or variant
# RUN wget -O /usr/bin/debugpaste https://github.com/nixc-us/debugpaste-it/raw/master/bin/debugpaste_64 && \
@ -21,17 +20,15 @@ RUN wget -O /usr/bin/lbrycrd-linux.zip https://github.com/lbryio/lbrycrd/release
rm lbrycrd-linux.zip && \
chmod +x lbrycrdd lbrycrd-cli lbrycrd-tx
COPY debugpaste-it.sh /usr/local/bin/debugpaste-it
COPY start.sh /usr/local/bin/start
COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint
COPY healthcheck.sh /usr/local/bin/healthcheck
COPY stuff/debugpaste-it.sh /usr/local/bin/debugpaste-it
COPY stuff/start.sh /usr/local/bin/start
COPY stuff/docker-entrypoint.sh /usr/local/bin/docker-entrypoint
COPY stuff/healthcheck.sh /usr/local/bin/healthcheck
USER lbrycrdd
RUN mkdir -p /data/
VOLUME /data/
VOLUME /etc/lbrycrdd/
# USER lbrycrd
# RUN mkdir /data
VOLUME ["/data"]
WORKDIR /data
## TODO: Implement healthcheck.
# HEALTHCHECK ["healthcheck"]
@ -39,6 +36,9 @@ VOLUME /etc/lbrycrdd/
## Exposing daemon port and RPC port
EXPOSE 9245 9246
## TODO: Decide what's important for lbrycrd and possibly add an entrypoint.
## Maybe catch things that might match things that can be easily executed in the
## lbrycrd cli and if nothing is entered just default to the containers shell.
## For now this is a placeholder that executes /bin/bash on `docker exec`
# ENTRYPOINT ["docker-entrypoint"]

View file

@ -2,7 +2,8 @@
## Compiler/Builder Step ##
###########################
FROM ubuntu:18.04 as builder
LABEL MAINTAINER="@Brannon Update Me Or Put Mine Here"
## TODO: Decide who gets the MAINTAINER label
## LABEL MAINTAINER="@Brannon Update Me Or Put Mine Here"
## Added apt work as single container layer then remove cruft before layer is committed.
RUN apt-get update && \

View file

@ -8,17 +8,17 @@
# ## This is specific to chainquery.
## Ensure perms are correct prior to running main binary
mkdir -p /data/.lbrycrd
chown -R 1000:1000 /data
chmod -R 755 /data
chown -R 1000:1000 /etc/lbrycrdd
chmod -R 755 /etc/lbrycrdd
# chown -R 1000:1000 /etc/lbrycrdd
# chmod -R 755 /etc/lbrycrdd
rm -f /var/run/lbrycrdd.pid
mkdir -p ~/.lbrycrd
## Set config params
echo -e "rpcuser=lbryrpc\nrpcpassword=${RPC_PASSWORD:-changeme}" > ~/.lbrycrd/lbrycrd.conf
echo -e "rpcallowip=${RPC_ALLOW_IP:-10.5.1.3}" >> ~/.lbrycrd/lbrycrd.conf
echo -e "rpcuser=${RPC_USER:-lbryrpc}" >> ~/.lbrycrd/lbrycrd.conf
echo 'rpcuser=lbryrpc\nrpcpassword='${RPC_PASSWORD:-changeme} > ~/.lbrycrd/lbrycrd.conf
echo "rpcallowip=${RPC_ALLOW_IP:-10.5.1.3}" >> ~/.lbrycrd/lbrycrd.conf
echo "rpcuser=${RPC_USER:-lbryrpc}" >> ~/.lbrycrd/lbrycrd.conf
## Control this invocation through envvar.
case ${RUN_MODE:-default} in

View file

View file

@ -16,7 +16,6 @@ services:
ipv4_address: 10.5.1.2
labels:
- "traefik.expose=false"
user: "${UID:-1000}:${GID:-1000}"
environment:
RUN_MODE: chainquery
expose:
@ -24,4 +23,4 @@ services:
- 9246
## host volumes for persistent data such as wallet private keys.
volumes:
- ./data:/data
- "./data:/data"

12
lbrycrd/stuff/env-example Normal file
View file

@ -0,0 +1,12 @@
COMPOSE_PROJECT_NAME=lbrycrd
#############
## Lbrycrd ##
#############
## TODO: The credentials are a formality but we should randomize these with magic.
RPC_USER=${RPC_USER=lbryrpc}
RPC_PASSWORD=${RPC_PASSWORD:-changeme}
## This should be the internal container IP from which you'll be calling the RPC for Lbrycrdd from.
## TODO: make this more dynamic before we move to scalability
RPC_ALLOW_IP=${RPC_ALLOW_IP:-10.5.1.3}

View file

@ -7,43 +7,32 @@
# ## This is specific to chainquery.
## Ensure perms are correct prior to running main binary
chown -R 1000:1000 /data
chmod -R 755 /data
mkdir -p /data/.lbrycrd
chown -R lbrycrd:lbrycrd /data
chmod -R 755 /data/
## TODO: Consider a config directory for future magic.
# chown -R 1000:1000 /etc/lbrycrd
# chmod -R 755 /etc/lbrycrd
rm -f /var/run/lbrycrd.pid
mkdir -p ~/.lbrycrd
## Set config params
## TODO: Make this more automagic in the future.
echo -e "rpcuser=lbryrpc\nrpcpassword=${RPC_PASSWORD:-changeme}" > $HOME/.lbrycrd/lbrycrd.conf
echo -e "rpcallowip=${RPC_ALLOW_IP:-10.5.1.3}" >> $HOME/.lbrycrd/lbrycrd.conf
echo -e "rpcuser=${RPC_USER:-lbryrpc}" >> $HOME/.lbrycrd/lbrycrd.conf
echo "rpcuser=lbryrpc\nrpcpassword=${RPC_PASSWORD:-changeme}" > /data/.lbrycrd/lbrycrd.conf
echo "rpcallowip=${RPC_ALLOW_IP:-10.5.1.3}" >> /data/.lbrycrd/lbrycrd.conf
echo "rpcuser=${RPC_USER:-lbryrpc}" >> /data/.lbrycrd/lbrycrd.conf
## Control this invocation through envvar.
case ${RUN_MODE:-default} in
default )
lbrycrdd \
-server \
-conf=$HOME.lbrycrd/lbrycrd.conf \
-printtoconsole
su -c "lbrycrdd -server -conf=/data/.lbrycrd/ -printtoconsole" lbrycrd
;;
reindex )
lbrycrdd \
-server \
-txindex \
-reindex \
-conf=$HOME.lbrycrd/lbrycrd.conf \
-printtoconsole
su -c "lbrycrdd -server -txindex -reindex -conf=/data/.lbrycrd/ -printtoconsole" lbrycrd
;;
chainquery )
lbrycrdd \
-server \
-txindex \
-conf=$HOME.lbrycrd/lbrycrd.conf \
-printtoconsole
su -c "lbrycrdd -server -txindex -conf=/data/.lbrycrd/ -printtoconsole" lbrycrd
;;
esac