From 93ea84bb81d3a8b1b2fd50619a4eb623d9bb0aa0 Mon Sep 17 00:00:00 2001 From: Leopere Date: Thu, 25 Apr 2019 20:41:42 -0400 Subject: [PATCH] More removals Renamed to fit with new nomenclature Removed directory and renamed/moved Dockerfile Updated COPY path Moved docker-compose.yml for Lbrynet Cleaned up comments a bit Important notice to users in new comment Updated compose example for image instead of build updated paths Added a step for chainquery-bootstrap setup Moved and updated lbrycrd's env_file Created lbrynet env_file Moved and updated chainquery's env_file Moved and updated lbrynet's compose file More Cleanup Added TODO's for checkmount.sh Renamed compiler Renamed production container Final rename fixup fixup Updated README refs Just readability improvements Packing directories with the repo Git won't track empty directories but if you take the path of running these containers locally from within this repo you should have this directory. Updated image refs to final production locations --- README.md | 18 ++++++------- chainquery/Dockerfile-linux-x86_64-production | 2 +- chainquery/README.md | 2 ++ .../compose/docker-compose.yml-prod-example | 8 +++--- contrib/systemd-cloud-init.md | 27 +++++++++---------- chainquery/env => environment/chainquery.env | 2 +- lbrycrd/env => environment/lbrycrd.env | 3 +-- environment/lbrynet.env | 2 ++ ...ile => Dockerfile-linux-x86_64-production} | 8 +++--- lbrycrd/compose/docker-compose.yml-regtest | 2 +- lbrycrd/compose/docker-compose.yml-testnet | 2 +- ...ux => Dockerfile-linux-multiarch-compiler} | 0 ...ion => Dockerfile-linux-x86_64-production} | 0 lbrynet/README.md | 8 +++--- .../docker-compose.yml-prod-example} | 11 ++++---- lbrynet/linux-armhf-production/.gitkeep | 1 - lbrynet/stuff/checkmount.sh | 3 ++- persist/.gitkeep | 0 18 files changed, 50 insertions(+), 49 deletions(-) rename chainquery/env => environment/chainquery.env (83%) rename lbrycrd/env => environment/lbrycrd.env (86%) create mode 100644 environment/lbrynet.env rename lbrycrd/{linux-x86_64-production/Dockerfile => Dockerfile-linux-x86_64-production} (87%) rename lbrynet/{Dockerfile-compiler-linux => Dockerfile-linux-multiarch-compiler} (100%) rename lbrynet/{Dockerfile-x86_64-production => Dockerfile-linux-x86_64-production} (100%) rename lbrynet/{docker-compose.yml => compose/docker-compose.yml-prod-example} (58%) delete mode 100644 lbrynet/linux-armhf-production/.gitkeep create mode 100644 persist/.gitkeep diff --git a/README.md b/README.md index 0a6e882..c462c05 100644 --- a/README.md +++ b/README.md @@ -4,16 +4,16 @@ Document tags and link to their Dockerfiles here. ## Scope -This repository is in heavy flux as it travels towards [version 1.0](https://github.com/lbryio/lbry-docker/projects/1) however its goal is to make development for and adoption of any of the LBRY appliances trivial. You should be able to clone pull fork your way to a better LBRY without having to do much more than some light reading of a README to get started. +This repository is in flux as it travels towards [version 1.0](https://github.com/lbryio/lbry-docker/projects/1); however its goal is to make development for and adoption of any of the LBRY appliances trivial. You should be able to clone pull fork your way to a better LBRY without having to do much more than some light reading of a README to get started. #### Documentation is WIP -Currently this repository is a WIP and is under heavy construction, use at your own risk make sure you keep regular backups of your wallets. Your milage may vary as how far this will work for you be sure to file good and concise issues if you plan to and keep in mind we're allergic to regressions when filing PR's. +Currently, this repository is a WIP and is under heavy construction, use at your own risk make sure you keep regular backups of your wallets. Your mileage may vary as to how far this will work for you be sure to file good and concise issues if you plan to and be mindful of we're allergic to regressions when filing PRs. #### Goals This repository aims for [Docker Best Practices](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) minimalism, ephemerality, and statelessness. It will use well commented **`Shell scripts`**, Dockerfiles, and Docker-Compose which is a template that is a baseline for many other container management services such as [RancherOS](https://rancher.com/rancher-os/) and [Kubernetes](https://kubernetes.io/). #### Beautiful Screenshot/Gif -Since there is nothing to see here just yet I'll entertain you with the current state of affairs with this repository. +Since there is nothing to see here just yet, I'll entertain you with the current state of affairs with this repository. ![image](https://spee.ch/855d1958650b850b249b9ee592ba2f4c6fc7eeec/container-unloading-gone-wrong-151175.gif) ## Installation @@ -23,7 +23,7 @@ See [Running from source](##Running-from-source) for the current instructions on #### Currently supported platforms -**X86_64 cpu architecture** +**X86_64 CPU architecture** Lbrynet, Lbrycrd, Chainqery are currently supported. **Some arm architectures** @@ -31,25 +31,25 @@ Lbrynet, Lbrycrd, Chainqery are currently supported. **More will be added on request and over time** ## Usage -For now I don't recommend using this container cluster however you're welcomed to [contribute](#contributing) if you feel up to the task. +For now, I don't recommend using this container cluster however you're welcomed to [contribute](#contributing) if you feel up to the task. ## Running from source -Running this stuff from source should be possible if you have both [Docker](https://docs.docker.com/install/) and [docker-compose](https://docs.docker.com/compose/install/) both installed. If these are both installed you can proceed to run the following from within your development directory. +Running this stuff from source should be possible if you have both [Docker](https://docs.docker.com/install/) and [docker-compose](https://docs.docker.com/compose/install/) both installed. If these are both installed, you can proceed to run the following from within your development directory. ``` git clone https://github.com/lbryio/lbry-docker.git ``` -Once you have a local copy of the recent source you will want to consider what containers/applications you require in your environment. At the moment since at the writing of this documentation this comes with an assertion of [YMMV](https://dictionary.cambridge.org/dictionary/english/ymmv) so if something isn't working feel free to make suggestions in the form of a PR for how we should do this better. The beauty of Open Source is learning better ways to do things as well as contributing to the world so I'm always going to be welcoming to contributions. +Once you have a local copy of the recent source, you will want to consider what containers/applications you require in your environment. At the moment since at the writing of this documentation, this comes with an assertion of [YMMV](https://dictionary.cambridge.org/dictionary/english/ymmv) so if something isn't working feel free to make suggestions in the form of a PR for how we should do this better. The beauty of Open Source is learning better ways to do things as well as contributing to the world, so I'm always going to be welcoming to contributions. #### From Source for Contributions Running from source for contributing and Merge/Pull requests. My goal is to make contributing to this possible using Docker and also GitLab CI/CD time. ## [Contributing](CONTRIBUTING.md) -Keep in mind [I am](https://github.com/leopere/) preferential to receiving patches over rule following as we can always nudge you in the right direction to get things more compatible with the project ethos if it's not. Never be afraid to file a PR no one should be offended. This said following the next two guides will greatly improve the speed at which we can integrate your improvements. +Keep in mind [I am](https://github.com/leopere/) preferential to receiving patches over rule-following as we can always nudge you in the right direction to get things more compatible with the project ethos if it's not. Never be afraid to file a PR no one should be offended. Having said this following the next two guides will greatly improve the speed at which we can integrate your improvements. * [Repository Standards]( https://lbry.tech/resources/repository-standards) * [Contribute](https://lbry.tech/contribute) -* Have a LBC wallet ready as we want you to have some for the help! Hell why not post it in your Commit or Merge Request for all I care but take your tips! +* Have an LBC wallet ready as we want you to have some for the help! Hell, why not post it in your Commit or Merge Request for all I care but take your tips! ## Getting Support diff --git a/chainquery/Dockerfile-linux-x86_64-production b/chainquery/Dockerfile-linux-x86_64-production index c07b243..209ae25 100644 --- a/chainquery/Dockerfile-linux-x86_64-production +++ b/chainquery/Dockerfile-linux-x86_64-production @@ -8,7 +8,7 @@ RUN apt-get update && \ rm -rf /var/lib/apt/lists/* WORKDIR / SHELL ["/bin/bash", "-o", "pipefail", "-c"] -COPY ../stuff/start.sh start +COPY ./stuff/start.sh start RUN curl -L -o /chainquery.zip $(curl -s https://api.github.com/repos/lbryio/chainquery/releases | grep -F 'Linux_x86_64.zip' | grep download | head -n 1 | cut -d'"' -f4) && \ unzip /chainquery.zip && \ rm /chainquery.zip diff --git a/chainquery/README.md b/chainquery/README.md index f9fce36..1068f14 100644 --- a/chainquery/README.md +++ b/chainquery/README.md @@ -77,6 +77,8 @@ Now that you're done syncing your own copy of the lbry blockchain into the `lbry `cd ../chainquery/` +`cat ./compose/docker-compose.yml-prod-example > docker-compose.yml` + `./quick-bootstrap.sh getdata` `./quick-bootstrap.sh extract` diff --git a/chainquery/compose/docker-compose.yml-prod-example b/chainquery/compose/docker-compose.yml-prod-example index 5c885a8..ff36354 100644 --- a/chainquery/compose/docker-compose.yml-prod-example +++ b/chainquery/compose/docker-compose.yml-prod-example @@ -19,7 +19,7 @@ services: aliases: - mysql env_file: - - env + - ../environment/chainquery.env expose: - 3306 ## TODO: I want to find a way that is acceptable to everyone to lock this up @@ -42,7 +42,7 @@ services: ipv4_address: 10.6.1.3 env_file: - env - - ../lbrycrd/env + - ../environment/lbrycrd.env labels: - "traefik.expose=false" expose: @@ -52,5 +52,5 @@ services: depends_on: - mysql ## TODO: Uncomment this in a docker-compose.override.yml to allow for external configurations. - # volumes: - # - ../persist/chainquery/config/chainqueryconfig.toml:/etc/chainquery/chainqueryconfig.toml + volumes: + - ../persist/chainquery/config/chainqueryconfig.toml:/etc/chainquery/chainqueryconfig.toml diff --git a/contrib/systemd-cloud-init.md b/contrib/systemd-cloud-init.md index a6f967b..5bb7763 100644 --- a/contrib/systemd-cloud-init.md +++ b/contrib/systemd-cloud-init.md @@ -26,7 +26,7 @@ for that. This tutorial will use cloud-init and systemd to control docker. * Select a Standard droplet with 8GB of memory ($40 per month in 2019.) * Select whatever datacenter you want. * Mark the checkbox called `User data`, and paste the following into the box: - + ``` #cloud-config @@ -57,10 +57,10 @@ write_files: Description=lbrycrd docker container After=snap.docker.dockerd.service Requires=snap.docker.dockerd.service - + [Service] Environment=SERVICE=lbrycrd - Environment=IMAGE=lbry/lbry-docker:lbrycrd-production + Environment=IMAGE=lbry/lbrycrd:linux-x86-64-production TimeoutStartSec=0 ExecStartPre=-/snap/bin/docker stop $SERVICE ExecStartPre=-/snap/bin/docker rm -f $SERVICE @@ -77,14 +77,14 @@ write_files: ExecStop=/snap/bin/docker stop $SERVICE Restart=always RestartSec=60 - + [Install] WantedBy=multi-user.target - path: "/etc/mysql/conf.d/chainquery.cnf" content: | # Put mysql optimizations specific to chainquery here - + - path: "/etc/systemd/system/mysql.service" content: | [Unit] @@ -112,7 +112,7 @@ write_files: ExecStop=/snap/bin/docker stop $SERVICE Restart=always RestartSec=60 - + [Install] WantedBy=multi-user.target @@ -122,7 +122,7 @@ write_files: lbrycrdurl="rpc://lbry:lbry@lbrycrd:9245" mysqldsn="chainquery:chainquery@tcp(mysql:3306)/chainquery" apimysqldsn="chainquery:chainquery@tcp(mysql:3306)/chainquery" - + - path: "/etc/systemd/system/chainquery.service" content: | [Unit] @@ -133,7 +133,7 @@ write_files: [Service] Environment=SERVICE=chainquery - Environment=IMAGE=lbry/lbry-docker:chainquery-production + Environment=IMAGE=lbry/chainquery:linux-x86-64-production TimeoutStartSec=0 ExecStartPre=-/snap/bin/docker stop $SERVICE ExecStartPre=-/snap/bin/docker rm -f $SERVICE @@ -149,7 +149,7 @@ write_files: ExecStop=/snap/bin/docker stop $SERVICE Restart=always RestartSec=60 - + [Install] WantedBy=multi-user.target @@ -157,7 +157,7 @@ write_files: - path: "/root/.bash_aliases" content: | alias lbrycrd-cli="docker run --rm -it --link lbrycrd:lbrycrd --mount type=bind,source=/etc/lbry/lbrycrd.conf,target=/etc/lbry/lbrycrd.conf \ - lbry/lbry-docker:lbrycrd-production lbrycrd-cli -conf=/etc/lbry/lbrycrd.conf -rpcconnect=lbrycrd" + lbry/lbrycrd:linux-x86-64-production lbrycrd-cli -conf=/etc/lbry/lbrycrd.conf -rpcconnect=lbrycrd" alias mysql="docker run --rm -it --link mysql:mysql mysql:5 mysql -hmysql -u chainquery --password=chainquery" runcmd: @@ -169,7 +169,7 @@ runcmd: - /snap/bin/docker volume create mysql-data - systemctl enable --now lbrycrd - echo "Good to go." -``` +``` * You can leave everything above as it is, to use the default configuration, OR you may edit the config in the box to your own liking. * For instance, if you wanted to run in [regtest @@ -199,7 +199,7 @@ startup. You can tail the log to monitor the install progress: ``` -tail -f /var/log/cloud-init-output.log +tail -f /var/log/cloud-init-output.log ``` Wait for the final `Good to go` message to know that the installer has finished. @@ -304,7 +304,7 @@ journalctl --unit chainquery (optionally use `-f` if you want to tail/follow the logs) -##### Disabling chainquery service +##### Disabling chainquery service ``` systemctl disable --now chainquery @@ -319,4 +319,3 @@ In the future, this tutorial may replace the snap version of docker with the regular PPA version of docker-ce, which has a more predictable update strategy (apt-get) rather than auto-updates. More long term testing is needed to know which way is better. - diff --git a/chainquery/env b/environment/chainquery.env similarity index 83% rename from chainquery/env rename to environment/chainquery.env index ca64fcf..7d338d2 100644 --- a/chainquery/env +++ b/environment/chainquery.env @@ -1,4 +1,4 @@ -COMPOSE_PROJECT_NAME=chainquery +## Contains environment variables for chainquery containers ######################### ## Chainquery Settings ## diff --git a/lbrycrd/env b/environment/lbrycrd.env similarity index 86% rename from lbrycrd/env rename to environment/lbrycrd.env index 57647ae..0f9551f 100644 --- a/lbrycrd/env +++ b/environment/lbrycrd.env @@ -1,5 +1,4 @@ -COMPOSE_PROJECT_NAME=lbrycrd - +## Contains environment variables for lbrycrd containers ############# ## Lbrycrd ## ############# diff --git a/environment/lbrynet.env b/environment/lbrynet.env new file mode 100644 index 0000000..3e7b5f9 --- /dev/null +++ b/environment/lbrynet.env @@ -0,0 +1,2 @@ +## Contains environment variables for lbrynet containers +RUN_MODE=default diff --git a/lbrycrd/linux-x86_64-production/Dockerfile b/lbrycrd/Dockerfile-linux-x86_64-production similarity index 87% rename from lbrycrd/linux-x86_64-production/Dockerfile rename to lbrycrd/Dockerfile-linux-x86_64-production index 636c52e..2a745a1 100644 --- a/lbrycrd/linux-x86_64-production/Dockerfile +++ b/lbrycrd/Dockerfile-linux-x86_64-production @@ -7,10 +7,10 @@ RUN apt-get update && \ rm -rf /var/lib/apt/lists/* WORKDIR / SHELL ["/bin/bash", "-o", "pipefail", "-c"] -COPY ../stuff/start.sh start -COPY ../stuff/healthcheck.sh healthcheck -COPY ../stuff/advance_blocks.sh advance -COPY ../stuff/fix-permissions.c fix-permissions.c +COPY ./stuff/start.sh start +COPY ./stuff/healthcheck.sh healthcheck +COPY ./stuff/advance_blocks.sh advance +COPY ./stuff/fix-permissions.c fix-permissions.c RUN curl -L -o ./lbrycrd-linux.zip $(curl -s https://api.github.com/repos/lbryio/lbrycrd/releases | grep -F 'lbrycrd-linux.zip' | grep download | head -n 1 | cut -d'"' -f4) && \ unzip ./lbrycrd-linux.zip && \ gcc fix-permissions.c -o fix-permissions && \ diff --git a/lbrycrd/compose/docker-compose.yml-regtest b/lbrycrd/compose/docker-compose.yml-regtest index 424b6a2..8b3fe14 100644 --- a/lbrycrd/compose/docker-compose.yml-regtest +++ b/lbrycrd/compose/docker-compose.yml-regtest @@ -5,7 +5,7 @@ services: ## Lbrycrd ## ############# lbrycrd: - image: lbry/lbry-docker:lbrycrd-linux-x86_64-production + image: lbry/lbrycrd:linux-x86_64-production restart: always ports: - "11336:9246" diff --git a/lbrycrd/compose/docker-compose.yml-testnet b/lbrycrd/compose/docker-compose.yml-testnet index 6ccbba8..9bf4bdc 100644 --- a/lbrycrd/compose/docker-compose.yml-testnet +++ b/lbrycrd/compose/docker-compose.yml-testnet @@ -5,7 +5,7 @@ services: ## Lbrycrd ## ############# lbrycrd: - image: lbry/lbry-docker:lbrycrd-linux-x86_64-production + image: lbry/lbrycrd:linux-x86_64-production restart: always ports: - "11336:9246" diff --git a/lbrynet/Dockerfile-compiler-linux b/lbrynet/Dockerfile-linux-multiarch-compiler similarity index 100% rename from lbrynet/Dockerfile-compiler-linux rename to lbrynet/Dockerfile-linux-multiarch-compiler diff --git a/lbrynet/Dockerfile-x86_64-production b/lbrynet/Dockerfile-linux-x86_64-production similarity index 100% rename from lbrynet/Dockerfile-x86_64-production rename to lbrynet/Dockerfile-linux-x86_64-production diff --git a/lbrynet/README.md b/lbrynet/README.md index b9ad0d8..3e4d48a 100644 --- a/lbrynet/README.md +++ b/lbrynet/README.md @@ -2,7 +2,7 @@ ## Compiler container -The [Dockerfile-compiler-linux](Dockerfile-compiler-linux) is for building lbrynet for any architecture supported +The [Dockerfile-linux-multiarch-compiler](Dockerfile-linux-multiarch-compiler) is for building lbrynet for any architecture supported by an Ubuntu 18.04 base image. ### Register qemu to run docker images built for platforms other than your host @@ -14,17 +14,17 @@ docker run --rm --privileged multiarch/qemu-user-static:register ### Build for the default x86_64 platform: ``` -docker build -t lbrynet -f Dockerfile-compiler-linux . +docker build -t lbrynet -f Dockerfile-linux-multiarch-compiler . ``` ### Build for an ARM 32-bit platform: ``` -docker build -t lbrynet-armhf -f Dockerfile-compiler-linux --build-arg BASE_IMAGE=multiarch/ubuntu-core:armhf-bionic . +docker build -t lbrynet-armhf -f Dockerfile-linux-multiarch-compiler --build-arg BASE_IMAGE=multiarch/ubuntu-core:armhf-bionic . ``` ### Build for an ARM 64-bit platform: ``` -docker build -t lbrynet-arm64 -f Dockerfile-compiler-linux --build-arg BASE_IMAGE=multiarch/ubuntu-core:arm64-bionic . +docker build -t lbrynet-arm64 -f Dockerfile-linux-multiarch-compiler --build-arg BASE_IMAGE=multiarch/ubuntu-core:arm64-bionic . ``` diff --git a/lbrynet/docker-compose.yml b/lbrynet/compose/docker-compose.yml-prod-example similarity index 58% rename from lbrynet/docker-compose.yml rename to lbrynet/compose/docker-compose.yml-prod-example index 9d65a9b..fb54840 100644 --- a/lbrynet/docker-compose.yml +++ b/lbrynet/compose/docker-compose.yml-prod-example @@ -9,19 +9,18 @@ services: ## Lbrynet ## ############# lbrynet: - build: . + image: lbry/lbrynet:Linux-x86_64-production restart: always - ## TODO: Does this container ever need a static network address or is it better to be dynamic? networks: # lbrynet: # ipv4_address: 10.6.1.4 - lbry-network - labels: - - "traefik.expose=false" - ## TODO: Verify the need for the following port binds on host networking. ports: - 4444:4444 - 50001:50001 + env_file: + - ../environment/lbrynet.env ## host volumes for persistent data such as wallet private keys. + # This container will not run without a persistent wallet as it may contain a LBC balance. volumes: - - ./data:/lbrynet + - ../persist/lbrynet/data:/lbrynet diff --git a/lbrynet/linux-armhf-production/.gitkeep b/lbrynet/linux-armhf-production/.gitkeep deleted file mode 100644 index a4b008f..0000000 --- a/lbrynet/linux-armhf-production/.gitkeep +++ /dev/null @@ -1 +0,0 @@ -# TODO: Prepare a container for deployment on armhf architecture. diff --git a/lbrynet/stuff/checkmount.sh b/lbrynet/stuff/checkmount.sh index d47d8d3..637e037 100644 --- a/lbrynet/stuff/checkmount.sh +++ b/lbrynet/stuff/checkmount.sh @@ -1,11 +1,12 @@ #!/bin/bash +## TODO: Make a bit more aware of the run mode of this appliance in case there is ever a test mode enabled in the start.sh mountpoint=/home/lbrynet if ! grep -qs ".* $mountpoint " /proc/mounts; then echo "$mountpoint not mounted, refusing to run." + ## TODO: We should have documentation that this error references directly with a URL as to why it won't run without a volume. exit 1 else `$@` fi - diff --git a/persist/.gitkeep b/persist/.gitkeep new file mode 100644 index 0000000..e69de29