LBRYCommunity/ytsync
1
0
Fork 0
Code Issues 39 Pull requests 8 Projects Releases 197 Packages Wiki Activity
An overly complex tool to mirror youtube content to LBRY
494 commits 17 branches 205 tags 26 MiB
Go 92.4%
Shell 5.7%
Dockerfile 1.4%
Makefile 0.4%
C 0.1%
Find a file
Niko Storni 28791f317b update gitignore 
improve error logging to slack
fix regression in dev builds
2021-11-24 18:37:04 +01:00
blobs_reflector upgrade reflector to switch to wasabi 2021-04-12 23:31:37 +02:00
configs switch from env vars to a config file 2021-11-24 05:54:08 +01:00
downloader switch from env vars to a config file 2021-11-24 05:54:08 +01:00
e2e switch from env vars to a config file 2021-11-24 05:54:08 +01:00
ip_manager revert aria2c usage 2021-06-25 19:09:00 +02:00
manager update gitignore 2021-11-24 18:37:04 +01:00
metrics fix prom allowed chars 2020-05-19 23:24:16 +02:00
namer fix issues with unicode claim names 2021-03-23 01:18:26 +01:00
scripts fix modules pollution 2019-02-27 20:19:29 +01:00
sdk switch from env vars to a config file 2021-11-24 05:54:08 +01:00
shared switch from env vars to a config file 2021-11-24 05:54:08 +01:00
sources switch from env vars to a config file 2021-11-24 05:54:08 +01:00
tags_manager refactor tags_mapping name 2019-10-21 15:44:24 +02:00
thumbs switch from env vars to a config file 2021-11-24 05:54:08 +01:00
timing fix semver 2020-06-11 18:45:56 +02:00
util update gitignore 2021-11-24 18:37:04 +01:00
ytapi switch from env vars to a config file 2021-11-24 05:54:08 +01:00
.gitignore update gitignore 2021-11-24 18:37:04 +01:00
.goreleaser.yml update goreleaser 2020-01-11 22:11:44 -05:00
.travis.yml switch to yt-dlp 2021-06-17 17:51:21 +02:00
0001-lbry-patch.patch add new throttling detection 2019-12-16 00:29:56 +01:00
go.mod switch from env vars to a config file 2021-11-24 05:54:08 +01:00
go.sum switch from env vars to a config file 2021-11-24 05:54:08 +01:00
LICENSE Update LICENSE 2020-02-03 04:08:28 +07:00
main.go switch from env vars to a config file 2021-11-24 05:54:08 +01:00
Makefile Add retry on IP failures 2020-08-05 15:47:53 -04:00
README.md Update README.md 2021-09-28 10:20:17 -04:00

README.md

YTSync Tool

Build Status

This tool serves lbry to parse youtube channels that want their content mirrored on LBRY.

The tool downloads the entire set of public videos from a given channel, publishes them to LBRY and populates our private database in order to keep track of what's publishes. With the support of said database, the tool is also able to keep all the channels updated.

Requirements

  • lbrynet SDK https://github.com/lbryio/lbry/releases (We strive to keep the latest release of ytsync compatible with the latest major release of the SDK)
  • a lbrycrd node running (localhost or on a remote machine) with credits in it

Setup

  • make sure daemon is stopped and can be controlled through systemctl (find example below)
  • extract the ytsync binary anywhere
  • add the environment variables necessary to the tool
    • export SLACK_TOKEN="a-token-to-spam-your-slack"
    • export SLACK_CHANNEL="youtube-status"
    • export YOUTUBE_API_KEY="youtube-api-key"
    • export LBRY_WEB_API="https://lbry-api-url-here"
    • export LBRY_API_TOKEN="internal-apis-token-for-ytsync-user"
    • export LBRYCRD_STRING="tcp://user:password@host:5429"
    • export AWS_S3_ID="THE-ID-LIES-HERE"
    • export AWS_S3_SECRET="THE-SECRET-LIES-HERE"
    • export AWS_S3_REGION="us-east-1"
    • export AWS_S3_BUCKET="ytsync-wallets"

systemd script example

/etc/systemd/system/lbrynet.service

[Unit]
Description="LBRYnet daemon"
After=network.target

[Service]
Environment="HOME=/home/lbry"
ExecStart=/opt/lbry/lbrynet start
User=lbry
Group=lbry
Restart=on-failure
KillMode=process

[Install]
WantedBy=multi-user.target

Instructions

Publish youtube channels into LBRY network automatically.

Usage:
  ytsync [flags]

Flags:
      --after int                   Specify from when to pull jobs [Unix time](Default: 0)
      --before int                  Specify until when to pull jobs [Unix time](Default: current Unix time) (default current timestamp)
      --channelID string            If specified, only this channel will be synced.
      --concurrent-jobs int         how many jobs to process concurrently (default 1)
  -h, --help                        help for ytsync
      --limit int                   limit the amount of channels to sync
      --max-length float            Maximum video length to process (in hours) (default 2)
      --max-size int                Maximum video size to process (in MB) (default 2048)
      --max-tries int               Number of times to try a publish that fails (default 3)
      --remove-db-unpublished       Remove videos from the database that are marked as published but aren't really published
      --run-once                    Whether the process should be stopped after one cycle or not
      --skip-space-check            Do not perform free space check on startup
      --status string               Specify which queue to pull from. Overrides --update
      --stop-on-error               If a publish fails, stop all publishing and exit
      --takeover-existing-channel   If channel exists and we don't own it, take over the channel
      --update                      Update previously synced channels instead of syncing new ones
      --upgrade-metadata            Upgrade videos if they're on the old metadata version
      --videos-limit int            how many videos to process per channel (default 1000)

Running from Source

Clone the repository and run make

License

This project is MIT licensed. For the full license, see LICENSE.

Contributing

Contributions to this project are welcome, encouraged, and compensated. For more details, see CONTRIBUTING.

Security

We take security seriously. Please contact security@lbry.io regarding any security issues. Our PGP key is here if you need it.

Contact

The primary contact for this project is Niko Storni (niko@lbry.io).

Additional Info and Links