handle throttling automatically #98

Open
opened 2021-02-10 22:35:14 +01:00 by tzarebczan · 1 comment
tzarebczan commented 2021-02-10 22:35:14 +01:00 (Migrated from github.com)

Today we need to shutdown servers, wait, fix yt sync records so they are picked up by other channels, etc.

Today we need to shutdown servers, wait, fix yt sync records so they are picked up by other channels, etc.
tzarebczan commented 2021-04-21 21:29:16 +02:00 (Migrated from github.com)

23 recently got stuck at eleting original file /tmp/ytsync318973938/QTWjcTZI5ko/jennifer-love-hewitt-stories.f140 (pass -k to keep) ERRO[694297] 2604:2dc0:101:200::e9f set to throttled DEBU[694297] slack: ytsync-status: :sos: 2604:2dc0:101:200::e9f set to throttled INFO[694297] Command finished with error: WARNING: Unable to download webpage: HTTP Error 429: Too Many Requests INFO[694297] /tmp/ytsync318973938/QTWjcTZI5ko DEBU[694297] QTWjcTZI5ko deleted from disk for 'due to error' (/tmp/ytsync318973938/QTWjcTZI5ko/jennifer-love-hewitt-stories.mp4) INFO[694312] Running command youtube-dl --no-progress -o/tmp/ytsync318973938/NE6xHQGFJSc/jennifer-love-hewitt-stories --merge-output-format mp4 --rm-cache-dir --postprocessor-args -movflags faststart --abort-on-unavailable-fragment --fragment-retries 1 --cookies cookies.txt --max-filesize 2048M --match-filter duration <= 7200 --source-address 2604:2dc0:101:200::e9f https://www.youtube.com/watch?v=NE6xHQGFJSc -fbestvideo[ext=mp4][height<=1080]+bestaudio[ext!=webm] --user-agent Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36 DEBU[694315] Removing cache dir /home/lbry/.cache/youtube-dl .. [youtube] NE6xHQGFJSc: Downloading webpage [youtube] NE6xHQGFJSc: Downloading API JSON [youtube] NE6xHQGFJSc: Downloading MPD manifest [youtube] NE6xHQGFJSc: Downloading API JSON [download] Destination: /tmp/ytsync318973938/NE6xHQGFJSc/jennifer-love-hewitt-stories.f135 [download] Download completed [download] Destination: /tmp/ytsync318973938/NE6xHQGFJSc/jennifer-love-hewitt-stories.f140 [download] Download completed [ffmpeg] Merging formats into "/tmp/ytsync318973938/NE6xHQGFJSc/jennifer-love-hewitt-stories.mp4" Deleting original file /tmp/ytsync318973938/NE6xHQGFJSc/jennifer-love-hewitt-stories.f135 (pass -k to keep) Deleting original file /tmp/ytsync318973938/NE6xHQGFJSc/jennifer-love-hewitt-stories.f140 (pass -k to keep) INFO[694315] Command finished with error: WARNING: Unable to download webpage: HTTP Error 429: Too Many Requests INFO[694315] /tmp/ytsync318973938/NE6xHQGFJSc DEBU[694315] NE6xHQGFJSc deleted from disk for 'due to error' (/tmp/ytsync318973938/NE6xHQGFJSc/jennifer-love-hewitt-stories.mp4)

it detects it - it should stop syncing so that another server can pick it up and then wait a day.

23 recently got stuck at ```eleting original file /tmp/ytsync318973938/QTWjcTZI5ko/jennifer-love-hewitt-stories.f140 (pass -k to keep) ERRO[694297] 2604:2dc0:101:200::e9f set to throttled DEBU[694297] slack: ytsync-status: :sos: 2604:2dc0:101:200::e9f set to throttled INFO[694297] Command finished with error: WARNING: Unable to download webpage: HTTP Error 429: Too Many Requests INFO[694297] /tmp/ytsync318973938/QTWjcTZI5ko DEBU[694297] QTWjcTZI5ko deleted from disk for 'due to error' (/tmp/ytsync318973938/QTWjcTZI5ko/jennifer-love-hewitt-stories.mp4) INFO[694312] Running command youtube-dl --no-progress -o/tmp/ytsync318973938/NE6xHQGFJSc/jennifer-love-hewitt-stories --merge-output-format mp4 --rm-cache-dir --postprocessor-args -movflags faststart --abort-on-unavailable-fragment --fragment-retries 1 --cookies cookies.txt --max-filesize 2048M --match-filter duration <= 7200 --source-address 2604:2dc0:101:200::e9f https://www.youtube.com/watch?v=NE6xHQGFJSc -fbestvideo[ext=mp4][height<=1080]+bestaudio[ext!=webm] --user-agent Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36 DEBU[694315] Removing cache dir /home/lbry/.cache/youtube-dl .. [youtube] NE6xHQGFJSc: Downloading webpage [youtube] NE6xHQGFJSc: Downloading API JSON [youtube] NE6xHQGFJSc: Downloading MPD manifest [youtube] NE6xHQGFJSc: Downloading API JSON [download] Destination: /tmp/ytsync318973938/NE6xHQGFJSc/jennifer-love-hewitt-stories.f135 [download] Download completed [download] Destination: /tmp/ytsync318973938/NE6xHQGFJSc/jennifer-love-hewitt-stories.f140 [download] Download completed [ffmpeg] Merging formats into "/tmp/ytsync318973938/NE6xHQGFJSc/jennifer-love-hewitt-stories.mp4" Deleting original file /tmp/ytsync318973938/NE6xHQGFJSc/jennifer-love-hewitt-stories.f135 (pass -k to keep) Deleting original file /tmp/ytsync318973938/NE6xHQGFJSc/jennifer-love-hewitt-stories.f140 (pass -k to keep) INFO[694315] Command finished with error: WARNING: Unable to download webpage: HTTP Error 429: Too Many Requests INFO[694315] /tmp/ytsync318973938/NE6xHQGFJSc DEBU[694315] NE6xHQGFJSc deleted from disk for 'due to error' (/tmp/ytsync318973938/NE6xHQGFJSc/jennifer-love-hewitt-stories.mp4)``` it detects it - it should stop syncing so that another server can pick it up and then wait a day.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: LBRYCommunity/ytsync#98
No description provided.