Decode protobuf-encoded claim values and return them in JSONRPC responses

lyoshenka commented

2018-09-26 18:37:53 +02:00

(Migrated from github.com)

RPC commands such as getclaimbyid return the value of a claim. For most claims, this value is a protobuf-encoded bytestring that was created using the types in https://github.com/lbryio/types. Such values should be decoded and returned as JSON in a new value_decoded field.

The protobuf definitions for c++ are here: https://github.com/lbryio/types/tree/master/cpp

Acceptance Criteria

Definition of Done

Tested against acceptance criteria
Tested against the assumptions of the user story
The project builds without errors
Unit tests are written and passing
Tests on devices/browsers listed in the issue have passed
QA performed & issues resolved
Refactoring completed
Any configuration or build changes documented
Documentation updated
Peer Code Review performed

RPC commands such as `getclaimbyid` return the value of a claim. For most claims, this value is a protobuf-encoded bytestring that was created using the types in https://github.com/lbryio/types. Such values should be decoded and returned as JSON in a new `value_decoded` field. The protobuf definitions for c++ are here: https://github.com/lbryio/types/tree/master/cpp ### Acceptance Criteria 1. 2. 3. ### Definition of Done - [ ] Tested against acceptance criteria - [ ] Tested against the assumptions of the user story - [ ] The project builds without errors - [ ] Unit tests are written and passing - [ ] Tests on devices/browsers listed in the issue have passed - [ ] QA performed & issues resolved - [ ] Refactoring completed - [ ] Any configuration or build changes documented - [ ] Documentation updated - [ ] Peer Code Review performed

👍 2

ConnorBach commented

2018-10-05 04:09:43 +02:00

(Migrated from github.com)

Hi! I'd like to tackle this issue. Should the new value_decoded field be returned in the UniValue claim object?

🎉 1

BrannonKing commented

2018-10-05 16:50:16 +02:00

(Migrated from github.com)

Yes, the new values should be returned in the existing UniValue result. You can safely add fields; try not to modify the other fields already coming out.

LucianaMarques commented

2018-10-17 16:52:01 +02:00

(Migrated from github.com)

Hello there,

Has any pull request been made on this issue? I'd like to give it a try!

Hello there, Has any pull request been made on this issue? I'd like to give it a try!

BrannonKing commented

2018-10-17 16:57:52 +02:00

(Migrated from github.com)

I have seen no PRs for this yet.

LucianaMarques commented

2018-10-17 19:48:15 +02:00

(Migrated from github.com)

Ok, I think I may have a solution in mind.

Just a question: whenever commands like getclaimid are called, there is a string that represents a value of the concerned claim. Isn't what I need to do with this number basically convert the represented number in an integer?

Ok, I think I may have a solution in mind. Just a question: whenever commands like getclaimid are called, there is a string that represents a value of the concerned claim. Isn't what I need to do with this number basically convert the represented number in an integer?

BrannonKing commented

2018-10-18 17:54:54 +02:00

(Migrated from github.com)

This issue is somewhat more complicated than numerical conversion. The claim values are passed in as binary data (hex-encoded). The binary data is output from a protobuf serializer. You won't be able to deserialize that data without the necessary protobuf definitions. The protobuf definitions are defined in an alternate project (mentioned in the first post here). Lbrycrd does have an existing reference to protobuf, so you should be able to use that directly once the types are also referenced. At present lbrycrd has no other lbryio dependencies.

And in thinking about it, I actually think this story would be better implemented as a python script. The script would run the current export command (like the import scripts), process that data some more using the already-existing python metadata defs, and output the results. The script could even pull the metadata definition in from the web when it runs; it wouldn't need a downloaded dependency.

This issue is somewhat more complicated than numerical conversion. The claim values are passed in as binary data (hex-encoded). The binary data is output from a protobuf serializer. You won't be able to deserialize that data without the necessary protobuf definitions. The protobuf definitions are defined in an alternate project (mentioned in the first post here). Lbrycrd does have an existing reference to protobuf, so you should be able to use that directly once the types are also referenced. At present lbrycrd has no other lbryio dependencies. And in thinking about it, I actually think this story would be better implemented as a python script. The script would run the current export command (like the import scripts), process that data some more using the already-existing python metadata defs, and output the results. The script could even pull the metadata definition in from the web when it runs; it wouldn't need a downloaded dependency.

LucianaMarques commented

2018-10-21 19:34:58 +02:00

(Migrated from github.com)

Hi, just to inform that I'm working on this. Still trying to understand some things.

lyoshenka commented

2018-10-22 16:28:32 +02:00

(Migrated from github.com)

@BrannonKing can you explain how this would work as a python script? when i call getclaimbyid, would lbrycrd-cli call the python script somehow?

@BrannonKing can you explain how this would work as a python script? when i call `getclaimbyid`, would `lbrycrd-cli` call the python script somehow?

BrannonKing commented

2018-10-22 18:38:27 +02:00

(Migrated from github.com)

I was picturing a python script that would call lbrycrd-cli (like this script: https://github.com/lbryio/lbrycrd/blob/add_claim_import_scripts/contrib/devtools/import_claims_from_claimsintrie_output.py). It would load that data (with json.loads, etc.). It would then pull down the current type definitions from lbryio/types. For each item in the data it would run the metadata through the protobuf parser, specifying the right protobuf def as it went. It would modify the JSON and reoutput it.

I was picturing a python script that would call lbrycrd-cli (like this script: https://github.com/lbryio/lbrycrd/blob/add_claim_import_scripts/contrib/devtools/import_claims_from_claimsintrie_output.py). It would load that data (with `json.loads`, etc.). It would then pull down the current type definitions from lbryio/types. For each item in the data it would run the metadata through the protobuf parser, specifying the right protobuf def as it went. It would modify the JSON and reoutput it.

lyoshenka commented

2018-10-22 21:13:33 +02:00

(Migrated from github.com)

why is this better than doing the decoding in lbrycrd? it seems like i'd basically always want the json output instead of the protobuf-serialized bytes.

BrannonKing commented

2018-10-22 21:25:32 +02:00

(Migrated from github.com)

why is this better than doing the decoding in lbrycrd?

Please see the non-cpp changes here for an example of what was involved in bringing in an additional dependency: https://github.com/lbryio/lbrycrd/pull/159
lbrycrdd is compiled, and that not very often. Pulling the protobuf spec in at runtime (easy in Python but hard in C++) will ensure that the applied spec is always up to date, that there is no dependency that can drive a new release of lbrycrd.
It's not metadata if we know what it means. Invariably someone will be tempted to use that data definition inside lbrycrd -- something that it appears we have tried hard to avoid.

> why is this better than doing the decoding in lbrycrd? 1. Please see the non-cpp changes here for an example of what was involved in bringing in an additional dependency: https://github.com/lbryio/lbrycrd/pull/159 2. lbrycrdd is compiled, and that not very often. Pulling the protobuf spec in at runtime (easy in Python but hard in C++) will ensure that the applied spec is always up to date, that there is no dependency that can drive a new release of lbrycrd. 3. It's not metadata if we know what it means. Invariably someone will be tempted to use that data definition inside lbrycrd -- something that it appears we have tried hard to avoid.

kauffj commented

2018-10-22 21:35:36 +02:00

(Migrated from github.com)

@BrannonKing agreed on historical aspect of (3) but not convinced this is right long-term.

ghost commented

2018-10-28 04:57:57 +01:00

(Migrated from github.com)

@BrannonKing Can you tell me where to get/how to produce claims for testing. I'm now in phase of testing my python script. I'm using -regtest to produce block. However, the blocks are currently empty.

BrannonKing commented

2019-03-12 19:06:33 +01:00

(Migrated from github.com)

@vancam-pham , I apologize for never replying to your query last October. That was an oversight.

BrannonKing commented

2019-05-08 17:09:37 +02:00

(Migrated from github.com)

I'm closing this as rejected. It's out of scope for the foreseeable future. It is easily done in Python3 utilizing the SDK:

from lbrynet.wallet.transaction import Transaction
tx = Transaction(unhexlify(raw_tx['hex']))
if tx.outputs[0].is_claim and tx.outputs[0].claim.is_stream:
	metadata = tx.outputs[0].claim.stream

I'm closing this as rejected. It's out of scope for the foreseeable future. It is easily done in Python3 utilizing the SDK: ```python from lbrynet.wallet.transaction import Transaction tx = Transaction(unhexlify(raw_tx['hex'])) if tx.outputs[0].is_claim and tx.outputs[0].claim.is_stream: metadata = tx.outputs[0].claim.stream ```

Decode protobuf-encoded claim values and return them in JSONRPC responses #206

Acceptance Criteria

Definition of Done