Concurrency while publishing files with the same name #38

New issue

Closed

opened 2018-08-22 20:44:17 +02:00 by nikooo777 · 2 comments

nikooo777 commented

2018-08-22 20:44:17 +02:00

(Migrated from github.com)

consider the following videos:

My awesome dog and its friend speedy (part 1)
My awesome dog and its friend speedy (part 2)
My awesome dog and its friend speedy (part 3)
My awesome dog and its friend speedy (part 4)
My awesome dog and its friend speedy (part 5)

the claim name is based on the title, a limit is enforced thus trimming out a part of the name.

claim name for the 5 videos as computed by the algorithm:

my-awesome-dog-and-its-friend-sp
my-awesome-dog-and-its-friend-sp
my-awesome-dog-and-its-friend-sp
my-awesome-dog-and-its-friend-sp
my-awesome-dog-and-its-friend-sp

lbry.go attempts to publish all videos concurrently. The end result is ambiguous. If they are of similar size they will end up being published in the same block. some will be rejected, some will not. If a video goes through on block 100, the other 4 risk either failing and attempting the very subsequent my-awesome-dog-and-its-friend-sp-1... or updating the previous claim. If they go through in the same block, separate claims with the same claim_name are sent through (I believe).

All in all it's a chaos.
A solution should ensure that no 2 videos can attempt publishing using the same name ever.

My idea: the video tracking table contains the claim names used, said claim names should be returned (and probably are already returned) together with the channel status response payload.
lbry.go needs to check against that list before attempting a publication, if a collision is detected, another name should be attempted.

consider the following videos: 1. My awesome dog and its friend speedy (part 1) 2. My awesome dog and its friend speedy (part 2) 3. My awesome dog and its friend speedy (part 3) 4. My awesome dog and its friend speedy (part 4) 5. My awesome dog and its friend speedy (part 5) the claim name is based on the title, a limit is enforced thus trimming out a part of the name. claim name for the 5 videos as computed by the algorithm: 1. my-awesome-dog-and-its-friend-sp 2. my-awesome-dog-and-its-friend-sp 3. my-awesome-dog-and-its-friend-sp 4. my-awesome-dog-and-its-friend-sp 5. my-awesome-dog-and-its-friend-sp lbry.go attempts to publish all videos concurrently. The end result is ambiguous. If they are of similar size they will end up being published in the same block. some will be rejected, some will not. If a video goes through on block 100, the other 4 risk either failing and attempting the very subsequent `my-awesome-dog-and-its-friend-sp-1...` or updating the previous claim. If they go through in the same block, separate claims with the same claim_name are sent through (I believe). All in all it's a chaos. A solution should ensure that no 2 videos can attempt publishing using the same name ever. My idea: the video tracking table contains the claim names used, said claim names should be returned (and probably are already returned) together with the channel status response payload. lbry.go needs to check against that list before attempting a publication, if a collision is detected, another name should be attempted.

kauffj commented

2018-08-22 20:57:38 +02:00

(Migrated from github.com)

If you search related to generating a unique slug with maximum length, you can find code samples with good patterns to solve this.

👍 1

nikooo777 commented

2018-08-23 00:49:43 +02:00

(Migrated from github.com)

solved with 9424dce8ed

i didn't touch the name generation pattern as it was working already. However now I'm supplying the function with complete data about the previous claims, it was using a local version that would reset at every startup.

solved with https://github.com/lbryio/lbry.go/commit/9424dce8ed63c914a033725400e0e731448e5749 i didn't touch the name generation pattern as it was working already. However now I'm supplying the function with complete data about the previous claims, it was using a local version that would reset at every startup.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: LBRYCommunity/lbry.go#38

No description provided.