37 Setting up a Hub

Jack Robison edited this page 2023-03-03 13:16:13 -05:00

Table of Contents

• Server Setup
• create a VPS
• create lbry user
• logout and log in as lbry user
• install docker
• create lbcd and rocksdb volumes
• LBCD
• download lbcd snapshot
• download docker-compose.yml
• start lbcd first and let it catch up
• Scribe
• download scribe snapshot
• start scribe and let it sync
• Elastic and Elastic Sync
• download snapshot
• start elastic first and wait for it to load the snapshot
• once the count stops going up, start elastic sync
• Herald
• Firewall
• Connect to Your Hub
• Monitoring
• Federation

Server Setup

create a VPS

We recommend a server running Ubuntu 20.04 LTS with 16 vCPU, 32G RAM, and a 500GB SSD drive. As a benchmark, on Jan 2023 this setup costs about $100 a month and uses about 400GB of disk space.

For the setup below we used a c5.4xlarge AWS instance, but we've also used this same setup on OVH servers. Most providers should work.

create lbry user

sudo adduser lbry --disabled-password --gecos ""
sudo -H -u lbry bash -c 'mkdir -p /home/lbry/.ssh && echo "YOUR-SSH-KEY" >> /home/lbry/.ssh/authorized_keys'
sudo -H -u root bash -c 'echo "lbry ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers'

logout and log in as lbry user

install docker

https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository

sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg lsb-release
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo service docker start
sudo usermod -aG docker $USER
newgrp docker
sudo systemctl enable docker.service
sudo systemctl enable containerd.service

create lbcd and rocksdb volumes

mkdir -p /home/lbry/docker-volumes/lbcd /home/lbry/docker-volumes/rocksdb /home/lbry/docker-volumes/es01
# sudo chown -R 999:999 /home/lbry/docker-volumes  ## not needed unless you encounter problems 
docker volume create --driver local --opt type=none --opt device=/home/lbry/docker-volumes/lbcd --opt o=bind lbry_lbcd
docker volume create --driver local --opt type=none --opt device=/home/lbry/docker-volumes/rocksdb --opt o=bind lbry_rocksdb
docker volume create --driver local --opt type=none --opt device=/home/lbry/docker-volumes/es01 --opt o=bind es01

LBCD

download lbcd snapshot

sudo apt install -y zstd
cd /home/lbry/docker-volumes/lbcd
wget -c https://snapshots.lbry.com/blockchain/lbcd_snapshot_1269305_v0.22.119_2022-12-03.tar.zst -O - | tar --zstd -x
sudo chown -R 999:999 .
cd ~

download docker-compose.yml

https://github.com/lbryio/hub/wiki/docker-compose.yml

remember to remove markdown if downloading directly from that url

start lbcd first and let it catch up

docker compose up -d lbcd

wait for it to sync fully (how will they know?)

Scribe

download scribe snapshot

cd /home/lbry/docker-volumes/rocksdb
wget -c https://snapshots.lbry.com/hub/block_1312050/lbry-rocksdb.tar -O - | tar -x
sudo chown -R 999:999 .
cd ~

start scribe and let it sync

scribe sync takes about a week if you sync from scratch, or an hour or two if you use a recent snapshot

docker compose up -d scribe

wait for it to sync

Elastic and Elastic Sync

download snapshot

mkdir /home/lbry/elastic-snapshot
cd /home/lbry/elastic-snapshot
wget -c https://snapshots.lbry.com/hub/es_snap.tar -O - | tar x
sudo mv es_snap/lbry_es01/_data/nodes /home/lbry/docker-volumes/es01/
sudo chown -R lbry:root /home/lbry/docker-volumes/es01
sudo chmod -R 775 /home/lbry/docker-volumes/es01
sudo cp es_snap/es_info /home/lbry/docker-volumes/rocksdb/
sudo chown -R 999:999 /home/lbry/docker-volumes/rocksdb/es_info

start elastic first and wait for it to load the snapshot

docker compose up -d es01

you can check the loading progress by running curl localhost:9200/claims/_count

once the count stops going up, start elastic sync

docker compose up -d scribe_elastic_sync

it should take a few hours to sync. you'll know its done when the log messages (docker compose logs) say they're advancing to the latest block. here's an example:

scribe_elastic_sync_1  | 2023-01-12 15:02:50,801 INFO hub.service.ElasticSyncService:236: advancing to 1292266
scribe_elastic_sync_1  | 2023-01-12 15:02:59,100 INFO hub.service.ElasticSyncService:330: Indexing block 1292266 done. 240/240 successful

Herald

docker compose up -d herald

Firewall

open the following ports in your firewall

• 22 for ssh
• 9246 for lbcd
• 50001 for herald
• 2112 and 2113 for prometheus (if you want external monitoring)

Connect to Your Hub

Congratulations! You have a full hub up and running. Now you can add your hub's IP or hostname to the lbryum_servers config in the LBRY Desktop app, and connect to it.

To check how many clients are connected to your hub, use:

# count the number of connected clients, grouped by client version
curl -s localhost:2112/metrics | grep session_count
# count the total number of clients across all versions
curl -s localhost:2112/metrics | grep session_count | grep -v '#' | cut -d' ' -f2- | paste -sd+ | bc -l

Monitoring

start prometheus and grafana

do this to check that everything is working

Federation