Source file error when editing existing claim

tzarebczan commented

2017-08-11 00:09:18 +02:00

(Migrated from github.com)

The Issue

When copying data from an existing claim, you should not need to select the source file again since the original blobs should be used in the updated claim. Thanks @reillysmith for finding this one.

Steps to reproduce

Type existing claim name in Publish screen
Use "Use data from my existing claim" option to copy claim data
Publish and get error message about source

Expected behaviour

Edit claim without adding source

Actual behaviour

Error about source

System Configuration

LBRY Daemon version: 0.14.2
LBRY App version: 0.14.3
LBRY Installation ID:
Operating system: Windows

Anything Else

Screenshots

## The Issue When copying data from an existing claim, you should not need to select the source file again since the original blobs should be used in the updated claim. Thanks @reillysmith for finding this one. ### Steps to reproduce 1. Type existing claim name in Publish screen 2. Use "Use data from my existing claim" option to copy claim data 3. Publish and get error message about source ![copy claim](https://user-images.githubusercontent.com/8120721/29194321-00cd3c24-7df7-11e7-96a9-c56a1a811bef.JPG) ### Expected behaviour Edit claim without adding source ### Actual behaviour Error about source ## System Configuration   - LBRY Daemon version: 0.14.2 - LBRY App version: 0.14.3 - LBRY Installation ID: - Operating system: Windows ## Anything Else  ## Screenshots

kauffj commented

2017-08-11 04:43:52 +02:00

(Migrated from github.com)

I didn't test, but I doubt #464 fixes this. I actually think that error message is coming from the daemon, though I agree it's likely an app issue.

👍 1

tzarebczan commented

2017-08-21 16:30:23 +02:00

(Migrated from github.com)

Discussion with Kay regarding changing of SD-hash. Want to test this scenario out.

tomzarebczan
When we allow for claim updates, is it normal that you can either use the same file (same sd-hash) or pick a new file (new sd-hash?) What would happen if someone downloaded the file and then it's updated by the owner - it will have the same claim id but different contents...does that screw things up somehow?

kaykurokawa 
Yes, its normal. I think we solved this issue, but I would not be too suprised if there are still lingering problems.

tomzarebczan 
I haven't actually tried it, was just thinking out loud. I could see how it would be problematic. I'll try it in practice and see what happens sometime. I just feel like 1 claimid can point to 2 sets of blobs...so unless we are tracking that carefully, could lead to issues.

Discussion with Kay regarding changing of SD-hash. Want to test this scenario out. ``` tomzarebczan When we allow for claim updates, is it normal that you can either use the same file (same sd-hash) or pick a new file (new sd-hash?) What would happen if someone downloaded the file and then it's updated by the owner - it will have the same claim id but different contents...does that screw things up somehow? kaykurokawa Yes, its normal. I think we solved this issue, but I would not be too suprised if there are still lingering problems. tomzarebczan I haven't actually tried it, was just thinking out loud. I could see how it would be problematic. I'll try it in practice and see what happens sometime. I just feel like 1 claimid can point to 2 sets of blobs...so unless we are tracking that carefully, could lead to issues. ```

tzarebczan commented

2017-08-28 14:47:34 +02:00

(Migrated from github.com)

@btzr-io - any update on this one? I know you made some progress, but did you get stuck?

tzarebczan commented

2017-08-28 18:06:38 +02:00

(Migrated from github.com)

Discussion with @btzr-io:

btzr [28 minutes ago
I know  is the app forces you to add a path every time otherwise you get an error:
`error occurred when attempting to publish your file: source`

jiggytom [27 minutes ago] 
yep, and that's exactly what we are trying to solve. So it cold be a backend issue, or maybe we aren't passing something correctly when a path is not selected.
btzr [19 minutes ago] 
that's where I'm stuck, how / why does it stores the path if isn't need it?
jiggytom 
[13 minutes ago] 
Yea I'm not sure, sorry. I think it still needs the path, but if the path is the same, don't re-upload the file. What happens now is that the content gets re-uploaded as new data even though nothing changed with the file itself. Should be able to update claim and leave file intact. Want me to  comment on the issue
btzr [5 minutes ago] 
`https://lbryio.github.io/lbry/#publish`

btzr [4 minutes ago] 
it says  -> `'file_path'(optional)`
btzr [3 minutes ago] 
>  If no path is given but a metadata dict is provided, the source from the given metadata will be used.
(edited)
jiggytom 
[3 minutes ago] 
yea, so it is an app issue then
jiggytom 
[3 minutes ago] 
or should be...just need to check if its an update and if the file is not selected (optional), don't pass/check for the file_path?
btzr [2 minutes ago] 
https://github.com/lbryio/lbry-app/blob/master/ui/js/component/publishForm/view.jsx#L118
btzr [< 1 minute ago] 
I guess this is what don't get passed -> `metadata dict`, not sure what it's that yet ?

Discussion with @btzr-io: ``` btzr [28 minutes ago I know is the app forces you to add a path every time otherwise you get an error: `error occurred when attempting to publish your file: source` jiggytom [27 minutes ago] yep, and that's exactly what we are trying to solve. So it cold be a backend issue, or maybe we aren't passing something correctly when a path is not selected. btzr [19 minutes ago] that's where I'm stuck, how / why does it stores the path if isn't need it? jiggytom [13 minutes ago] Yea I'm not sure, sorry. I think it still needs the path, but if the path is the same, don't re-upload the file. What happens now is that the content gets re-uploaded as new data even though nothing changed with the file itself. Should be able to update claim and leave file intact. Want me to comment on the issue btzr [5 minutes ago] `https://lbryio.github.io/lbry/#publish` btzr [4 minutes ago] it says -> `'file_path'(optional)` btzr [3 minutes ago] > If no path is given but a metadata dict is provided, the source from the given metadata will be used. (edited) jiggytom [3 minutes ago] yea, so it is an app issue then jiggytom [3 minutes ago] or should be...just need to check if its an update and if the file is not selected (optional), don't pass/check for the file_path? btzr [2 minutes ago] https://github.com/lbryio/lbry-app/blob/master/ui/js/component/publishForm/view.jsx#L118 btzr [< 1 minute ago] I guess this is what don't get passed -> `metadata dict`, not sure what it's that yet ? ```

kauffj commented

2017-08-28 23:01:54 +02:00

(Migrated from github.com)

@btzr-io @tzarebczan to debug further:

Open console.
Perform broken publish.
Observe request. Is it making it to daemon properly? Is it returning errors?
If daemon is erroring, what is being passed? Is it passing a signature that the API docs report as valid?
If yes, it's a bug in the daemon and has to be fixed there. But could just as easily be app-side.

@btzr-io @tzarebczan to debug further: 1. Open console. 1. Perform broken publish. 1. Observe request. Is it making it to daemon properly? Is it returning errors? 1. If daemon is erroring, what is being passed? Is it passing a signature that the API docs report as valid? 1. If yes, it's a bug in the daemon and has to be fixed there. But could just as easily be app-side.

btzr-io commented

2017-08-29 03:29:12 +02:00

(Migrated from github.com)

KeyError: 'source'

2017-08-28 19:28:08,168 INFO     lbrynet.daemon.Daemon:1851: Publish: {'claim_address': None, 'name': u'fullprice', 'bid': 1, 'claim_dict': {'version': '_0_0_1', 'claimType': 'streamType', 'stream': {'version': '_0_0_1', 'metadata': {u'description': u'Lbry dev - fixing full price format', u'license': u'Public Domain', u'title': u'Lbry dev - fix price ', u'language': u'en', 'version': '_0_1_0', u'nsfw': False, u'licenseUrl': None}}}, 'file_path': None, 'change_address': None}
2017-08-28 19:28:08,187 ERROR    lbrynet.daemon.auth.server:353: Failed to process publish: 'source'
Traceback (most recent call last):
  File "site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
  File "site-packages/lbrynet/core/Wallet.py", line 934, in claim_name
  File "site-packages/lbryschema/claim.py", line 139, in load_dict
  File "site-packages/lbryschema/encoding.py", line 36, in decode_fields
KeyError: 'source'

Python raises a KeyError whenever a dict() object is requested (using the format a = adict[key]) and the key is not in the dictionary.

https://wiki.python.org/moin/KeyError

### KeyError: 'source' ``` 2017-08-28 19:28:08,168 INFO lbrynet.daemon.Daemon:1851: Publish: {'claim_address': None, 'name': u'fullprice', 'bid': 1, 'claim_dict': {'version': '_0_0_1', 'claimType': 'streamType', 'stream': {'version': '_0_0_1', 'metadata': {u'description': u'Lbry dev - fixing full price format', u'license': u'Public Domain', u'title': u'Lbry dev - fix price ', u'language': u'en', 'version': '_0_1_0', u'nsfw': False, u'licenseUrl': None}}}, 'file_path': None, 'change_address': None} 2017-08-28 19:28:08,187 ERROR lbrynet.daemon.auth.server:353: Failed to process publish: 'source' Traceback (most recent call last): File "site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks File "site-packages/lbrynet/core/Wallet.py", line 934, in claim_name File "site-packages/lbryschema/claim.py", line 139, in load_dict File "site-packages/lbryschema/encoding.py", line 36, in decode_fields KeyError: 'source' ``` > Python raises a KeyError whenever a dict() object is requested (using the format a = adict[key]) and the key is not in the dictionary. https://wiki.python.org/moin/KeyError

btzr-io commented

2017-08-29 03:33:42 +02:00

(Migrated from github.com)

@kauffj As far as I know when there is no path provided it should use source but it never get passed ^^

@kauffj As far as I know when there is no path provided it should use `source` but it never get passed ^^

btzr-io commented

2017-08-29 03:54:52 +02:00

(Migrated from github.com)

https://github.com/lbryio/lbry/blob/master/lbrynet/daemon/Publisher.py#L41

btzr-io commented

2017-08-29 18:46:13 +02:00

(Migrated from github.com)

Also the API docs aren't really clear about this:
https://lbryio.github.io/lbry/#publish

If provided, a lbry stream of this file will be used in 'sources'.
If no path is given but a metadata dict is provided, the source
from the given metadata will be used.

Also the API docs aren't really clear about this: https://lbryio.github.io/lbry/#publish > If provided, a lbry stream of this file will be used in 'sources'. If no path is given but a metadata dict is provided, the source from the given metadata will be used.

kauffj commented

2017-08-30 00:46:44 +02:00

(Migrated from github.com)

@tzarebczan can you confirm this exists purely daemon side by determining a precise API call that fails when it shouldn't? Then we can open this on the daemon.

tzarebczan commented

2017-08-30 01:33:42 +02:00

(Migrated from github.com)

@kauffj sure, will take a look tomorrow. I thought it was fixed already, had a comment on slack.

btzr-io commented

2017-08-30 16:58:40 +02:00

(Migrated from github.com)

@kauffj Already mention this but here is it again,
daemon expects a fallback source when no file_path is passed,
the app only pass file_path, check this possible fix: 7a853f1

@kauffj Already mention this but here is it again, daemon expects a fallback `source` when no `file_path` is passed, the app only pass `file_path`, check this possible fix: 7a853f1

kauffj commented

2017-08-30 17:06:41 +02:00

(Migrated from github.com)

But if the daemon says a source isn't required, a source shouldn't be required. I'm not inclined to add a fix for this app side (which also later ought to be removed), when we can just fix it in the right place.

😕 1

tzarebczan commented

2017-08-30 17:25:15 +02:00

(Migrated from github.com)

Can confirm I am seeing the same as @btzr-io

Original publish:
{"jsonrpc":"2.0","method":"publish","params":{"name":"lbryio","bid":0.01,"metadata":{"title":"test pub 1","language":"en","license":"","licenseUrl":null,"nsfw":false},"file_path":"C:\\Users\\Tom-LaptopVM\\Downloads\\lbryio.jpg"},"id":126}

daemon log:
2017-08-30 08:12:44,973 INFO lbrynet.daemon.Daemon:1838: Publish: {'claim_address': None, 'name': u'lbryio', 'bid': 0.01, 'claim_dict': {'version': '_0_0_1', 'claimType': 'streamType', 'stream': {'version': '_0_0_1', 'metadata': {u'language': u'en', u'license': u'', u'title': u'test pub 1', 'version': '_0_1_0', u'nsfw': False, u'licenseUrl': None}}}, 'file_path': u'C:\\Users\\Tom-LaptopVM\\Downloads\\lbryio.jpg', 'change_address': None}

Update publish:
{"jsonrpc":"2.0","method":"publish","params":{"name":"lbryio","bid":0.02,"metadata":{"title":"test pub 2","description":"update to publish","language":"en","license":"","licenseUrl":null,"nsfw":false}},"id":211}

daemon log:

2017-08-30 08:18:48,490 INFO     lbrynet.daemon.Daemon:1838: Publish: {'claim_address': None, 'name': u'lbryio', 'bid': 0.02, 'claim_dict': {'version': '_0_0_1', 'claimType': 'streamType', 'stream': {'version': '_0_0_1', 'metadata': {u'description': u'update to publish', u'license': u'', u'title': u'test pub 2', u'language': u'en', 'version': '_0_1_0', u'nsfw': False, u'licenseUrl': None}}}, 'file_path': None, 'change_address': None}
2017-08-30 08:18:48,605 ERROR    lbrynet.daemon.auth.server:353: Failed to process publish: 'source'
Traceback (most recent call last):
  File "site-packages\twisted\internet\defer.py", line 1299, in _inlineCallbacks
  File "site-packages\lbrynet\core\Wallet.py", line 933, in claim_name
  File "site-packages\lbryschema\claim.py", line 139, in load_dict
  File "site-packages\lbryschema\encoding.py", line 36, in decode_fields
KeyError: 'source'

@kauffj I think the daemon is saying that it does require the source to be specified (from the original claim) if a new file isn't selected. Are you saying that the daemon should just figure that part out on its own as opposed to filling it from the app?

Can confirm I am seeing the same as @btzr-io Original publish: `{"jsonrpc":"2.0","method":"publish","params":{"name":"lbryio","bid":0.01,"metadata":{"title":"test pub 1","language":"en","license":"","licenseUrl":null,"nsfw":false},"file_path":"C:\\Users\\Tom-LaptopVM\\Downloads\\lbryio.jpg"},"id":126}` daemon log: `2017-08-30 08:12:44,973 INFO lbrynet.daemon.Daemon:1838: Publish: {'claim_address': None, 'name': u'lbryio', 'bid': 0.01, 'claim_dict': {'version': '_0_0_1', 'claimType': 'streamType', 'stream': {'version': '_0_0_1', 'metadata': {u'language': u'en', u'license': u'', u'title': u'test pub 1', 'version': '_0_1_0', u'nsfw': False, u'licenseUrl': None}}}, 'file_path': u'C:\\Users\\Tom-LaptopVM\\Downloads\\lbryio.jpg', 'change_address': None} ` Update publish: `{"jsonrpc":"2.0","method":"publish","params":{"name":"lbryio","bid":0.02,"metadata":{"title":"test pub 2","description":"update to publish","language":"en","license":"","licenseUrl":null,"nsfw":false}},"id":211}` daemon log: ``` 2017-08-30 08:18:48,490 INFO lbrynet.daemon.Daemon:1838: Publish: {'claim_address': None, 'name': u'lbryio', 'bid': 0.02, 'claim_dict': {'version': '_0_0_1', 'claimType': 'streamType', 'stream': {'version': '_0_0_1', 'metadata': {u'description': u'update to publish', u'license': u'', u'title': u'test pub 2', u'language': u'en', 'version': '_0_1_0', u'nsfw': False, u'licenseUrl': None}}}, 'file_path': None, 'change_address': None} 2017-08-30 08:18:48,605 ERROR lbrynet.daemon.auth.server:353: Failed to process publish: 'source' Traceback (most recent call last): File "site-packages\twisted\internet\defer.py", line 1299, in _inlineCallbacks File "site-packages\lbrynet\core\Wallet.py", line 933, in claim_name File "site-packages\lbryschema\claim.py", line 139, in load_dict File "site-packages\lbryschema\encoding.py", line 36, in decode_fields KeyError: 'source' ``` @kauffj I think the daemon is saying that it does require the source to be specified (from the original claim) if a new file isn't selected. Are you saying that the daemon should just figure that part out on its own as opposed to filling it from the app?

👍 1

btzr-io commented

2017-08-30 17:41:19 +02:00

(Migrated from github.com)

@kauffj where does it says the source isn't required?

If provided, a lbry stream of this file will be used in 'sources'.
If no path is given but a metadata dict is provided, the source
from the given metadata will be used.

Publisher.py#L41

       if 'source' not in claim_dict['stream']:
            claim_dict['stream']['source'] = {}

so what happens when both are missing? 😕

@kauffj where does it says the source isn't required? > If provided, a lbry stream of this file will be used in 'sources'. If no path is given but a metadata dict is provided, the source from the given metadata will be used. ### [Publisher.py#L41](https://github.com/lbryio/lbry/blob/master/lbrynet/daemon/Publisher.py#L41) ``` if 'source' not in claim_dict['stream']: claim_dict['stream']['source'] = {} ``` so what happens when both are missing? :confused:

kauffj commented

2017-08-30 23:04:05 +02:00

(Migrated from github.com)

@btzr-io you're right, my mistake.

Source file error when editing existing claim #467

The Issue

Steps to reproduce

Expected behaviour

Actual behaviour

System Configuration

Anything Else

Screenshots

KeyError: 'source'

Publisher.py#L41