proposal: metadata extensions #32

New issue

Closed

opened 2022-03-22 16:52:58 +01:00 by lyoshenka · 1 comment

lyoshenka commented

2022-03-22 16:52:58 +01:00

(Migrated from github.com)

LBRY has a base schema for claim metadata. This schema is very general. Different file types and communities will want to include metadata that is specific to them. It would be good to have a mechanism to suppor this.

One idea for this is to allow publishing of metadata schemas to LBRY itself. This would be a new claim type (Schema). The schema would be a protobuf file (or similar, if protobufs cannot be compiled live), plus other information that hubs need (e.g. which fields to index). Then a claim can reference a published schema to tell hubs how to decode extra metadata contained in the claim.

Schema claim type

schema structure
how to specify indexes
schema versioning?
can schemas be changed? that complicates things
limit who can create schemas at first, for testing?

Stream and Channel claim changes

add a way to add arbitrary data
add a way to specify the schema for the arbitrary data

SDK search changes

The SDK would switch from using flags for search to using a simple DSL. This way the SDK does not need to know all possible metadata fields in advance.

what DSL to use for search?
how to reference fields in search?

Concerns

this is an avenue for protocol users to inject code into hubs and/or control their behavior. It's possible to DDoS hubs or impose unsafe or expensive computations on them (eg indexing fields for search), esp in combination with user-created claims.
i doubt that protobufs can be compiled on the fly (at least in Go). will we need another schema description language?

LBRY has a base [schema](https://github.com/lbryio/types) for claim metadata. This schema is very general. Different file types and communities will want to include metadata that is specific to them. It would be good to have a mechanism to suppor this. One idea for this is to allow publishing of metadata schemas to LBRY itself. This would be a new claim type (`Schema`). The schema would be a protobuf file (or similar, if protobufs cannot be compiled live), plus other information that hubs need (e.g. which fields to index). Then a claim can reference a published schema to tell hubs how to decode extra metadata contained in the claim. ## Schema claim type - schema structure - how to specify indexes - schema versioning? - can schemas be changed? that complicates things - limit who can create schemas at first, for testing? ## Stream and Channel claim changes - add a way to add arbitrary data - add a way to specify the schema for the arbitrary data ## SDK search changes The SDK would switch from using flags for search to using a simple DSL. This way the SDK does not need to know all possible metadata fields in advance. - what DSL to use for search? - how to reference fields in search? ## Concerns - this is an avenue for protocol users to inject code into hubs and/or control their behavior. It's possible to DDoS hubs or impose unsafe or expensive computations on them (eg indexing fields for search), esp in combination with user-created claims. - i doubt that protobufs can be compiled on the fly (at least in Go). will we need another schema description language?

lyoshenka commented

2022-11-29 15:09:49 +01:00

(Migrated from github.com)

we're doing a limited version of this here https://github.com/lbryio/lbry-sdk/pull/3706

will revisit after thats been out in the wild

we're doing a limited version of this here https://github.com/lbryio/lbry-sdk/pull/3706 will revisit after thats been out in the wild

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: LBRYCommunity/spec#32

No description provided.