LBRYCommunity/ytsync
1
0
Fork 0
Code Issues 39 Pull requests 8 Projects Releases 197 Packages Wiki Activity
511 commits 17 branches 205 tags 26 MiB
Commit graph

143 commits

Author SHA1 Message Date
Niko Storni
62dfdc1adb fix possible resource leak 2019-12-14 05:49:25 +01:00
Niko Storni
36f539ef5d change ip throttling management 2019-12-10 23:02:56 +01:00
Niko Storni
7bdc117ba0
upgrade ytsync to latest SDK 
upgrade e2e tests
 2019-12-08 11:49:27 -05:00
Niko Storni
77d0efcf3c refactor tags_mapping name 
remove failure state for transfers
 2019-10-21 15:44:24 +02:00
Niko Storni
30af4a0136 update lbry.go to v2 library 
update modules
add slightly more LBCs for the channel claim to ensure fees are covered
 2019-10-10 16:51:40 +02:00
Niko Storni
7d38aa7b29 add republishing on non-available videos 
enhanced logging
 2019-08-30 19:35:04 +02:00
Mark Beamer Jr
e477759395 change to master tag of internal-apis docker image 2019-08-14 04:22:59 +02:00
Mark Beamer Jr
73937a541b Add var/tmp volume for ytsync tmp files communicated between lbrynet and ytsync 
Add reference to wallet dir env var for when the wallet file is uploaded so it is cleared properly when testing locally
 2019-08-14 04:22:59 +02:00
Niko Storni
03161a5a02 prefer 1080p videos 2019-08-05 20:08:46 +02:00
Niko Storni
6dc2c39ee6 limit description length 
rather than limiting lines count
 2019-08-02 10:01:33 -04:00
Niko Storni
3ba0c97634 fix flags 2019-07-25 18:33:05 -04:00
Niko Storni
9d65fce6bc do not apply limits if disabled 2019-07-25 18:22:56 -04:00
Niko Storni
d5ed4ce753 facilitate streaming by moving the moov atom 2019-07-25 18:15:14 -04:00
Niko Storni
0c47510785 revert quoting paths 2019-07-22 02:51:13 +02:00
Niko Storni
8f556a86c8 fix missing fragments for good 
never retry them again
 2019-07-22 02:45:38 +02:00
Niko Storni
64ab2490fe fix bugs 
improve debugging
 2019-07-22 02:27:14 +02:00
Niko Storni
1760935edf don't retry missing fragments 
lower syncing quality for now
 2019-07-22 02:09:18 +02:00
Niko Storni
df08d42d9b improve ip re-selection 2019-07-13 12:15:19 +02:00
Niko Storni
1b55033bb8 don't retry big videos 2019-07-13 11:40:50 +02:00
Niko Storni
1331e5769f actually use the functions I create... 2019-07-13 11:33:40 +02:00
Niko Storni
8c2a8262e6 fix bugs 2019-07-12 23:20:01 +02:00
Niko Storni
6f486717da fix stop pattern 
fix noretry string
fix youtube-dl param
 2019-07-12 22:58:34 +02:00
Niko Storni
a3fcd67611 better handle interruptions by user 
refactor IP throttling in its own package
 2019-07-12 21:32:49 +02:00
Niko Storni
ae1ffb60c5 add ip throttling and better pooling 
fix nil pointer dereference
 2019-07-12 20:42:44 +02:00
Niko Storni
fa24d83ae9 add max length filter 2019-07-12 02:39:24 +02:00
Niko Storni
bab01a6354 prefer IPv4 over IPv6 
youtube sucks and they ban /64 subnet blocks on IPv6
 2019-07-12 01:54:34 +02:00
Niko Storni
89fe1e99d6 improve ipv6 validation 
fix audio merge issue
 2019-07-12 01:28:00 +02:00
Niko Storni
ec81af0857 fix download filters 2019-07-12 01:04:45 +02:00
Niko Storni
51cb7f5e34 improve youtube antithrottle 
fix logging
 2019-07-11 19:14:15 +02:00
Niko Storni
8ad2acc936 remove original downloader 
add locking to IP selection to avoid concurrent access
 2019-07-11 17:32:46 +02:00
Niko Storni
b8caf3f82b add IPv6 source pooling to bypass youtube throttling 2019-07-11 16:22:58 +02:00
Niko Storni
b5c9806dd0 improve space management 2019-06-24 20:43:50 -04:00
Niko Storni
6668840b11 fix description not containing youtube link 
add workaround to SDK bug not spending the right UTXOs
fix bug in assignments
 2019-06-13 19:33:58 +02:00
Niko Storni
dce71d2451 add adjustments to video tracking 
reduce minimum refill amount (min account balance supersedes)
revert account_fund change as --everything isn't supported with outputs
fix bugs
 2019-06-12 04:44:56 +02:00
Niko Storni
f4e75cf221 rewrite account funding algorithm 
lock publishing when UTXOs management is in progress
spend everything when funding UTXOs
update lbry.go
 2019-06-12 03:17:59 +02:00
Niko Storni
f7f0231c80 add VideoUpgradeFailed staus 2019-06-10 21:59:42 +02:00
Niko Storni
10ad46f464 various bug fixes 2019-06-06 23:25:31 +02:00
Niko Storni
76e653fb9b add Fee support 
lowercase all tags
refactor code
implement and test regexps for tags
 2019-06-06 02:16:07 +02:00
Niko Storni
cfe8ff0879
add flag to remove unpublished videos 
refactor code
add workaround for SDK bug
add code to remove unpublished videos
fix alt downloader to only produce mp4
add missing height and width
add tags sorting
add a few channel wide tags
 2019-06-04 22:21:44 +02:00
Niko Storni
3389a8be93 fix many bugs 
calculate size for claims missing in the database
fix reprocessing for videos no longer on youtube
pass metadata instead of hardcoding it
add video height when updating streams
add tags to channels
add test for tags
 2019-06-01 01:46:16 +02:00
Niko Storni
913c31a9f7 add channelID to video property 
work around missing data for mocked videos
normalize tags
sort tags
 2019-05-31 23:15:22 +02:00
Niko Storni
b184f033ab merge conflicts 2019-05-31 18:02:55 +02:00
Niko Storni
6767fdcc40 reprocess videos that are no longer on youtube (part 1) 2019-05-31 10:38:31 -04:00
Niko Storni
7df4137445 fix issues with metadata 2019-05-29 23:07:26 +02:00
Niko Storni
74d5e2606d fix cover URL 
fix fallback download
fix thumbnail permissions
 2019-05-28 22:40:31 +02:00
Niko Storni
b8d2b9335b implement fallback downloader 
fix bugs
update lbry.go
add tags manager/mapper
 2019-05-28 19:28:35 +02:00
Niko Storni
aecfc67f84 bug fixes 
first working version of re-processing
add metadata version to api updates
 2019-05-08 23:12:13 +02:00
Niko Storni
1f7017f3d0 videos never published should not be processed instead published 2019-05-07 22:07:44 +02:00
Niko Storni
75364805a5 more refactoring 2019-05-07 21:15:43 +02:00
Niko Storni
07f66a089f update dependency and fix bugs 2019-05-07 18:28:25 +02:00
Niko Storni
36abd1e471 remove lambda dependency 
fix error checking
 2019-05-07 10:01:11 -04:00
Niko Storni
502bfa924b change minimum refill amount 
remove more berkeley code
add address reuse code
remove strPtr function in favor of utils
refactor code
 2019-05-06 21:56:56 +02:00
Niko Storni
23b8fb4e54 add support for new SDK 
add support for direct thumbnails upload
remove UCB code
add support for republishing
refactoring
 2019-05-03 05:11:52 +02:00
Niko Storni
01590306a1 Add support for SDK 0.35 
Implement thumbnails uploader
Fetch more data from youtube for additional metadata
 2019-04-18 21:22:51 -04:00
Niko Storni
91fe8ba459 use go mod 
update lbry.go deps
rename LBRY_API env var as the SDK is now using it too
fix startup routine
replace ytdl library
add hardcoded support for khan academy
 2019-02-22 19:33:00 +01:00
Niko Storni
1a3a587514 hotfixes for imported channels 
convert to go mod
bug fixes
 2019-02-15 14:11:38 +01:00
Niko Storni
5e125fa641 upgrade ytsync to latest SDK 2019-01-30 13:42:23 +01:00
Mark Beamer Jr
4789f894b8 upgraded to latest lbry.go reorg 2019-01-16 12:51:15 +01:00
Niko Storni
9e102a3298 patch youtube API lies 2019-01-11 20:15:17 +01:00
Niko Storni
94c9e91bc7 add configurable size and length limits 
swap safety logic
fix bugs
add hardcoded exceptions
remove unused code
 2019-01-03 19:55:27 +01:00
Niko Storni
a7b2bc816d
fix update not picking up fresh accounts 
fix huge videos clogging up the queue
 2018-10-11 17:21:11 -04:00
Niko Storni
ea06ed54a6
fix nil pointer dereference 2018-10-09 15:57:10 -04:00
Alex Grintsvayg
bdcc1c62d3 missing files and fixes after splitting from lbry.go 2018-10-08 16:20:10 -04:00
Niko Storni
5a7d59a499 remove unused deps 
remove unused code
handle edge case causing a publish error
address a few previous review notes
 2018-10-02 20:53:54 -04:00
Niko Storni
58fef45a5f pair programming follow up 2018-10-02 20:53:54 -04:00
Alex Grintsvayg
5ec5412191 grin/niko coworking changes 2018-10-02 20:53:53 -04:00
Niko Storni
10f9cc1ba0 completely remove redis db dependency 
improve download process
 2018-09-26 16:27:32 -04:00
Niko Storni
3493a42216 update youtube download lib 
drop sync server dependency
 2018-09-26 16:27:25 -04:00
Niko Storni
a486ab6080 address memory leak 2018-09-05 13:54:36 -04:00
Niko Storni
2588e6dc6b switch to MP4 over webm 2018-09-05 13:44:25 -04:00
Niko Storni
06e9dd9a98 detect name collisions based on api data 
improve mutex usage
remove unnecessary line breaks
 2018-09-05 13:44:24 -04:00
Niko Storni
83b0e40ec5 user pointers! 
bug fixes
now it works...
 2018-09-05 13:44:24 -04:00
Niko Storni
6b4a1b13b4 add size support 2018-09-05 13:44:24 -04:00
Niko Storni
3e32372dbe fix nastiest bug ever 2018-08-10 08:41:26 -04:00
Niko Storni
d946d6ee76 add function to wait for a new block 
bad bug fixes
remove half broken code used to wait for UTXOs
use channel IDs rather than names (skip lbryum)
port changes to UCB uploader
 2018-08-09 20:13:53 -04:00
Niko Storni
e5017892d8 add more startup parameters 2018-08-01 08:56:07 -04:00
Niko Storni
302e080b95 address review comments 2018-07-23 20:01:38 -04:00
Niko Storni
dfe3e8a078 rename selfsync to ytsync 
add video marking support
 2018-07-20 19:56:38 -04:00
Niko Storni
e0ec334af9 remove invalid tests 
move functions out of utils
address review comments
revert lockfile changes
 2018-07-18 08:47:12 -04:00
Niko Storni
ad8a9c3003 remove launch parameter 
improve failure handling
adjust slack logging
increase file size in string
increase publish timeout to 40 minutes
 2018-07-12 08:26:46 -04:00
Niko Storni
920ee956ac fix slack logging 
add failure cases
change interrupt behavior for fatal errors
bump up timeout to 20 minutes
change max file size to 2GB
remove unnecessary utxo wait
 2018-07-12 08:26:46 -04:00
Niko Storni
9acd4b08c2 supply hostname for job listing 
fix download race condition
adjusted error handling
 2018-07-12 08:26:45 -04:00
Niko Storni
decb57bbed add concurrency support 
fix error handling (FINALLY)
fix slack message formatting
add locking for wallet setup to support concurrency
remove UTXO bug workaround (lbryumx should fix it)
add limits to filesize
 2018-07-12 08:26:45 -04:00
Niko Storni
ba5c1c6519 fix selfsync issues with the queue 
remove margin for fees
handle non latin names
adjusted error handling
 2018-07-12 08:26:45 -04:00
Niko Storni
3bbf46e139 move params to env vars 
consider fees of 0.1 while calculating setup funds
add change address to speed up publishing
rebase
 2018-07-12 08:26:45 -04:00
Niko Storni
8959393def improve continuos self sync 
fix bug in error handling while deleting videos
refactoring of code
 2018-07-12 08:24:43 -04:00
Niko Storni
611dcbc3bc add slack logging 
reduce claim name length (fees are too high)
delete videos in case of failure too
reduce synced videos to 1000 most recent
 2018-07-12 08:21:15 -04:00
Niko Storni
72039623e0 human readable video names 2018-04-25 15:43:20 -04:00
Niko Storni
1c6428f1ab add playlist position 
add delete function
 2018-04-25 14:56:26 -04:00
Niko Storni
0725700708 update youtube download library 2018-03-13 17:48:13 -04:00
Alex Grintsvayg
daffded4f0 recover ytsync panics when processing videos, switch to our errors everywhere 2018-03-09 11:47:38 -05:00
Alex Grintsvayg
2cd8552702 ucb 2018-03-01 14:23:25 -05:00
Alex Grintsvayg
039ed1a7eb big refactor, abort on ctrl-c, proper wallet init and backup 2017-12-28 12:18:01 -05:00