LBRYCommunity/spec
1
0
Fork 0
Code Issues 5 Pull requests Projects Releases Packages Wiki Activity

Compare commits

base: LBRYCommunity:master
Branches Tags
LBRYCommunity:master
LBRYCommunity:signing
LBRYCommunity:kauffj-patch-1
LBRYCommunity:justification
LBRYCommunity:v0.9
..
compare: LBRYCommunity:kauffj-patch-1
Branches Tags
LBRYCommunity:master
LBRYCommunity:signing
LBRYCommunity:kauffj-patch-1
LBRYCommunity:justification
LBRYCommunity:v0.9

1 commit
master ... kauffj-pat

Author SHA1 Message Date
Jeremy Kauffman
3eee8fbafa
additional resources section 
the people want links!
 2019-02-14 12:56:31 -05:00
8 changed files with 90 additions and 102 deletions
Show stats Expand all files Collapse all files

2
CNAME
View file

@ -1 +1 @@
spec.lbry.com spec.lbry.io

2
LICENSE
View file

@ -1,6 +1,6 @@
The MIT License (MIT) The MIT License (MIT)
Copyright (c) 2015-2020 LBRY Inc Copyright (c) 2015-2018 LBRY Inc
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish,

2
_config.yml
View file

@ -1,5 +1,5 @@
title: LBRY Protocol Specification title: LBRY Protocol Specification
email: grin@lbry.com email: grin@lbry.io
baseurl: "" # the subpath of your site, e.g. /blog baseurl: "" # the subpath of your site, e.g. /blog
url: "" # the base hostname & protocol for your site, e.g. http://example.com url: "" # the base hostname & protocol for your site, e.g. http://example.com

2
_layouts/spec.html
View file

@ -15,7 +15,7 @@
<main> <main>
<h1>LBRY: A Decentralized Digital Content Marketplace</h1> <h1>LBRY: A Decentralized Digital Content Marketplace</h1>
<div class="byline"> <div class="byline">
Alex Grintsvayg (<a href="mailto:grin@lbry.com">grin@lbry.com</a>), Jeremy Kauffman (<a href="mailto:jeremy@lbry.com">jeremy@lbry.com</a>) Alex Grintsvayg (<a href="mailto:grin@lbry.io">grin@lbry.io</a>), Jeremy Kauffman (<a href="mailto:jeremy@lbry.io">jeremy@lbry.io</a>)
</div> </div>
<a href="/lbry-spec.pdf" class="pdf-hide" target="_blank" rel="noopener">PDF Version</a> <a href="/lbry-spec.pdf" class="pdf-hide" target="_blank" rel="noopener">PDF Version</a>
<div class="toc-menu pdf-hide">Menu</div> <div class="toc-menu pdf-hide">Menu</div>

17
assets/style.css
View file

@ -106,18 +106,16 @@ h1 {
} }
h2 { h2 {
font-size: 3rem; font-size: 3rem;
border-bottom: solid 1px black;
padding-bottom: 5px;
} }
h3 { h3 {
font-size: 2.5rem; font-size: 2.25rem;
} }
h4 { h4 {
font-size: 2rem; font-size: 2rem;
margin-bottom: 1.5rem; margin-bottom: 1.5rem;
} }
h5 { h5 {
font-size: 1.5rem; font-size: 1.75rem;
margin-bottom: 1rem; margin-bottom: 1rem;
} }
@ -126,7 +124,7 @@ a {
color: #0074D9; color: #0074D9;
} }
a.external-link::after { a.external-link::after {
content: url('data:image/svg+xml;utf8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 576 512" color="rgb(0,116,217)" width="13px" height="13px" class="ext-link-icon"><path fill="currentColor" d="M576 24v127.984c0 21.461-25.96 31.98-40.971 16.971l-35.707-35.709-243.523 243.523c-9.373 9.373-24.568 9.373-33.941 0l-22.627-22.627c-9.373-9.373-9.373-24.569 0-33.941L442.756 76.676l-35.703-35.705C391.982 25.9 402.656 0 424.024 0H552c13.255 0 24 10.745 24 24zM407.029 270.794l-16 16A23.999 23.999 0 0 0 384 303.765V448H64V128h264a24.003 24.003 0 0 0 16.97-7.029l16-16C376.089 89.851 365.381 64 344 64H48C21.49 64 0 85.49 0 112v352c0 26.51 21.49 48 48 48h352c26.51 0 48-21.49 48-48V287.764c0-21.382-25.852-32.09-40.971-16.97z" class=""></path></svg>'); content: url('data:image/svg+xml;utf8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 576 512" color="#0074D9" width="13px" height="13px" class="ext-link-icon"><path fill="currentColor" d="M576 24v127.984c0 21.461-25.96 31.98-40.971 16.971l-35.707-35.709-243.523 243.523c-9.373 9.373-24.568 9.373-33.941 0l-22.627-22.627c-9.373-9.373-9.373-24.569 0-33.941L442.756 76.676l-35.703-35.705C391.982 25.9 402.656 0 424.024 0H552c13.255 0 24 10.745 24 24zM407.029 270.794l-16 16A23.999 23.999 0 0 0 384 303.765V448H64V128h264a24.003 24.003 0 0 0 16.97-7.029l16-16C376.089 89.851 365.381 64 344 64H48C21.49 64 0 85.49 0 112v352c0 26.51 21.49 48 48 48h352c26.51 0 48-21.49 48-48V287.764c0-21.382-25.852-32.09-40.971-16.97z" class=""></path></svg>');
padding-left: 3px; padding-left: 3px;
} }
@ -185,18 +183,9 @@ table {
table tr { table tr {
border-bottom: 1px solid #dee2e6; border-bottom: 1px solid #dee2e6;
} }
td {
border-left: solid 1px #dee2e6;
border-right: solid 1px #dee2e6;
}
th, td { th, td {
padding-left: 5px; padding-left: 5px;
} }
th {
font-size: 1.5rem;
font-weight: normal;
text-transform: uppercase;
}
table thead tr { table thead tr {
border-bottom: 2px solid #dee2e6; border-bottom: 2px solid #dee2e6;
} }

137
index.md
View file

@ -8,23 +8,19 @@ LBRY is a protocol for accessing and publishing digital content in a global, dec
Clients can use LBRY to publish, host, find, download, and pay for content — books, movies, music, or anything else that can be represented as a stream of bits. The protocol is permissionless and censorship-resistant, which means that participation is open to everyone and no one can unilaterally block or remove content. Clients can use LBRY to publish, host, find, download, and pay for content — books, movies, music, or anything else that can be represented as a stream of bits. The protocol is permissionless and censorship-resistant, which means that participation is open to everyone and no one can unilaterally block or remove content.
Before LBRY, publishers had to choose between a centralized host such as Amazon or Youtube, or a protocol like Bittorrent. Centralized platforms suffer from several problems because their incentives are not aligned with the incentives of their users. Hosts engage in rent-seeking behavior, often extracting 30-55% of creator profits. They enforce opaque and arbitrary rules on creators, and change those rules without warning or community input. They choose to censor content at the behest of repressive regimes around the world, in exchange for access to more users and higher profits for themselves. LBRY is a step forward from previous generations of decentralized networks, which provide no discovery or payment mechanisms. For creators, LBRY is unparalleled in trust and earning potential. For consumers, LBRY is the first system that provides end-to-end digital content consumption that does not require trusting a third-party. For the world, LBRY is designed to engender the most complete catalog of information to ever exist, and to be controlled by the only party that could be trusted with such monumental responsibility: no one.
Bittorrent does not have these faults, but it has problems of its own. It is only useful if one already knows the infohash of the content they seek, and there is no way to discover these hashes within the protocol. Even using an external search engine does not provide a comprehensive list of what is available on the network. There are no incentives for users to seed content, and Bittorrent largely works because users earn status through private communities, are nice, or simply fail to understand what their client is doing. Finally, a lot of content on BitTorrent infringes on copyright, which taints the protocol's public perception and overshadows the many positives it has. ### Assumptions
LBRY offers a significant improvement over both options. It uses a blockchain to provide the good parts of a centralized host (a single place to store data, find interesting content, build a brand, and get rewarded for contributing), while removing the downsides (opaque and arbitrary rules, rent extraction, censorship). It is public and no one can be censored or blocked from using it. Its rules are clearly defined and cannot be changed without community consensus. The blockchain records everything that is published to LBRY, so interesting content is easy to find and infringing content is difficult to hide. Accessing the blockchain data is free, the costs for downloading content are transparent, and publishers earn 100% of the price they set. This document assumes that the reader is familiar with distributed hash tables (DHTs), the BitTorrent protocol, Bitcoin, and blockchain technology in general. It does not attempt to document these technologies or explain how they work. The [Bitcoin developer reference](https://bitcoin.org/en/developer-reference) and [BitTorrent protocol specification](http://www.bittorrent.org/beps/bep_0003.html) are recommended for anyone wishing to understand the technical details.
### Status
LBRY has been in public use since June 2016. As of May 2020, over 3.3 million pieces of digital content have been published via the protocol. Millions of users access this content each month, downloading and uploading terabytes of data. Graphical browsers and wallets are available for all major operating systems and can be downloaded at [lbry.com/get](https://lbry.com/get).
### Overview ### Overview
This document defines the LBRY protocol, its components, and how they fit together. LBRY consists of several discrete components that are used together in order to provide the end-to-end capabilities of the protocol. There are two distributed data stores (blockchain and DHT), a peer-to-peer protocol for exchanging data, and specifications for data structure, encoding, and retrieval. This document defines the LBRY protocol, its components, and how they fit together. LBRY consists of several discrete components that are used together in order to provide the end-to-end capabilities of the protocol. There are two distributed data stores (blockchain and DHT), a peer-to-peer protocol for exchanging data, and specifications for data structure, encoding, and retrieval.
### Assumptions ### Status
This document assumes that the reader is familiar with distributed hash tables (DHTs), the BitTorrent protocol, Bitcoin, and blockchain technology in general. It does not attempt to document these technologies or explain how they work. The [Bitcoin developer reference](https://bitcoin.org/en/developer-reference) and [BitTorrent protocol specification](http://www.bittorrent.org/beps/bep_0003.html) are recommended for anyone wishing to understand the technical details. LBRY has been in public use since June 2016. As of February 2019, approximately 750,000 pieces of digital content have been published via the protocol. Tens of thousands of users access hundreds of thousands of pieces of content each month, downloading and uploading terrabytes of data. Graphical browsers and wallets are available for all major operating systems and can be downloaded on LBRY's user-facing portal at [lbry.io/get](https://lbry.io/get).
### Conventions and Terminology ### Conventions and Terminology
@ -64,7 +60,7 @@ This document assumes that the reader is familiar with distributed hash tables (
## Blockchain ## Blockchain
The LBRY blockchain is a public proof-of-work blockchain. The design is based on [Bitcoin](https://bitcoin.org/bitcoin.pdf), with substantial modifications. This document does not cover or specify any aspects of LBRY that are identical to Bitcoin and instead focuses on the differences, primarily the claim operations and claimtrie. The LBRY blockchain is a public, proof-of-work blockchain. The design is based on the [Bitcoin](https://bitcoin.org/bitcoin.pdf) blockchain, with substantial modifications. This document does not cover or specify any aspects of LBRY that are identical to Bitcoin and instead focuses on the differences, primarily the claim operations and claimtrie.
Our blockchain serves three key purposes: Our blockchain serves three key purposes:
@ -75,7 +71,7 @@ Our blockchain serves three key purposes:
### Stakes ### Stakes
A _stake_ is a single entry in the blockchain that commits credits toward a name. The two types of stakes are [_claims_](#claims) and [_supports_](#supports). A _stake_ is a a single entry in the blockchain that commits credits toward a name. The two types of stakes are [_claims_](#claims) and [_supports_](#supports).
All stakes have these properties: All stakes have these properties:
@ -89,7 +85,14 @@ All stakes have these properties:
#### Claims #### Claims
A _claim_ is a stake that stores metadata. There are two types of claims. _Stream claims_ declare the availability, access method, and publisher of a stream. _Channel claims_ create a pseudonym that can be used as the publisher of stream claims. A _claim_ is a stake that stores metadata. There are two types of claims:
<dl>
<dt>stream claim</dt>
<dd>Declares the availability, access method, and publisher of a stream.</dd>
<dt>channel claim</dt>
<dd>Creates a pseudonym that can be declared as the publisher of stream claims.</dd>
</dl>
##### Claim Properties ##### Claim Properties
@ -179,7 +182,7 @@ The _root hash_ is the hash of the root node. It is stored in the header of each
Multiple claims can exist for the same name. They are all stored in the leaf node for that name. See [Claim Ordering](#claim-ordering) Multiple claims can exist for the same name. They are all stored in the leaf node for that name. See [Claim Ordering](#claim-ordering)
For more details on the specific claimtrie implementation, see [the source code](https://github.com/lbryio/lbrycrd/blob/v0.17.3.2/src/claimtrie.cpp). For more details on the specific claimtrie implementation, see [the source code](https://github.com/lbryio/lbrycrd/blob/master/src/claimtrie.cpp).
#### Statuses {#stake-statuses} #### Statuses {#stake-statuses}
@ -256,10 +259,6 @@ See the [example](#claim-activation-example) in the appendix for more informatio
Names in the claimtrie are normalized when performing any comparisons. This is necessary to avoid confusion due to Unicode equivalence or casing. When names are being compared, they are first converted using [Unicode Normalization Form D](http://unicode.org/reports/tr15/#Norm_Forms) (NFD), then lowercased using the en_US locale. This means names are effectively case-insensitive. Since claims competing for the same name are stored in the same node in the claimtrie, names are also normalized to determine the claimtrie path to the node. Names in the claimtrie are normalized when performing any comparisons. This is necessary to avoid confusion due to Unicode equivalence or casing. When names are being compared, they are first converted using [Unicode Normalization Form D](http://unicode.org/reports/tr15/#Norm_Forms) (NFD), then lowercased using the en_US locale. This means names are effectively case-insensitive. Since claims competing for the same name are stored in the same node in the claimtrie, names are also normalized to determine the claimtrie path to the node.
#### Expiration
In an earlier version of the protocol, stakes would expire (i.e. automatically become abandoned) 262974 blocks after they were accepted. A hard fork was deployed that effectively disables expiration. Any stakes that expired before the fork took effect are treated as if they were abandoned. For details see [this pull request](https://github.com/lbryio/lbrycrd/pull/137).
### URLs ### URLs
URLs are memorable references to claims. All URLs: URLs are memorable references to claims. All URLs:
@ -303,25 +302,23 @@ lbry://@lbry/meet-lbry
A claim for this name with this claim ID. Partial prefix matches are allowed (see [URL Resolution](#url-resolution)). A claim for this name with this claim ID. Partial prefix matches are allowed (see [URL Resolution](#url-resolution)).
``` ```
lbry://meet-lbry:7a0aa95c5023c21c098 lbry://meet-lbry#7a0aa95c5023c21c098
lbry://meet-lbry:7a lbry://meet-lbry#7a
lbry://@lbry:3f/meet-lbry lbry://@lbry#3f/meet-lbry
``` ```
*Note: in a previous version of this spec, the `#` character was used to signify the claim ID portion of the url. This character is now deprecated and will stop being supported in the future.* ##### Claim Sequence
##### Sequence The _n_th accepted claim for this name. _n_ must be a positive number. This can be used to resolve claims in the order in which they were made, rather than by the amount of credits backing a claim.
The _n_th accepted claim for this name. _n_ must be a positive number. This can be used to reference claims in the order in which they were made, rather than by the amount of credits backing a claim.
``` ```
lbry://meet-lbry*1 lbry://meet-lbry:1
lbry://@lbry*1/meet-lbry lbry://@lbry:1/meet-lbry
``` ```
##### Amount Order ##### Bid Position
The _n_th claim for this name, ordered by total amount (highest first). _n_ must be a positive number. This is useful for resolving non-controlling claims that may become controlling. The _n_th claim for this name, ordered by total amount (highest first). _n_ must be a positive number. This is useful for resolving non-winning bids in bid order.
``` ```
lbry://meet-lbry$2 lbry://meet-lbry$2
@ -356,10 +353,10 @@ ChannelClaimNameAndModifier ::= ChannelClaimName Modifier?
StreamClaimName ::= NameChar+ StreamClaimName ::= NameChar+
ChannelClaimName ::= '@' NameChar+ ChannelClaimName ::= '@' NameChar+
Modifier ::= ClaimID | Sequence | AmountOrder Modifier ::= ClaimID | ClaimSequence | BidPosition
ClaimID ::= ':' Hex+ ClaimID ::= '#' Hex+
Sequence ::= '*' PositiveNumber ClaimSequence ::= ':' PositiveNumber
AmountOrder ::= '$' PositiveNumber BidPosition ::= '$' PositiveNumber
Query ::= '?' QueryParameterList Query ::= '?' QueryParameterList
QueryParameterList ::= QueryParameter ( '&' QueryParameterList )* QueryParameterList ::= QueryParameter ( '&' QueryParameterList )*
@ -374,7 +371,7 @@ PositiveNumber ::= PositiveDigit Digit*
HexAlpha ::= [abcdef] HexAlpha ::= [abcdef]
Hex ::= (Digit | HexAlpha)+ Hex ::= (Digit | HexAlpha)+
NameChar ::= Char - [=&#:*$@%?/] /* any character that is not reserved */ NameChar ::= Char - [=&#:$@%?/] /* any character that is not reserved */
Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */ Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */
``` ```
@ -390,13 +387,13 @@ Return the controlling claim for the name. Stream claims and channel claims are
Get all claims for the claim name whose IDs start with the given `ClaimID`. Sort the claims in ascending order by block height and position within the block. Return the first claim. Get all claims for the claim name whose IDs start with the given `ClaimID`. Sort the claims in ascending order by block height and position within the block. Return the first claim.
##### Sequence ##### ClaimSequence
Get all claims for the claim name. Sort the claims in ascending order by block height and position within the block. Return the _n_th claim, where _n_ is the given `Sequence` value. Get all claims for the claim name. Sort the claims in ascending order by block height and position within the block. Return the _n_th claim, where _n_ is the given `ClaimSequence` value.
##### AmountOrder ##### BidPosition
Get all claims for the claim name. Sort the claims in descending order by total effective amount. Return the _n_th claim, where _n_ is the given `AmountOrder` value. Get all claims for the claim name. Sort the claims in descending order by total effective amount. Return the _n_th claim, where _n_ is the given `BidSequence` value.
##### ChannelClaimName and StreamClaimName ##### ChannelClaimName and StreamClaimName
@ -456,7 +453,7 @@ OP_CLAIM_NAME Fruit Apple OP_2DROP OP_DROP OP_DUP OP_HASH160 <address> OP_EQUALV
##### OP\_UPDATE\_CLAIM ##### OP\_UPDATE\_CLAIM
`OP_UPDATE_CLAIM` updates a claim by replacing its metadata. An update transaction has an added requirement that it must spend the output for the existing claim that it wishes to update. Otherwise, it is considered invalid and will not make it into the claimtrie. Thus, it must have the following redeem script: `OP_UPDATE_CLAIM` updates a claim by replacing its metadata. An update transaction has an added requirement that it must spend the output for the existing claim that it wishes to update. Otherwise, it is considered invalid and will not make it into the claimtrie. Thus it must have the following redeem script:
``` ```
<signature> <pubKeyForPreviousAddress> <signature> <pubKeyForPreviousAddress>
@ -512,21 +509,21 @@ The target block time was lowered from 10 minutes to 2.5 minutes to facilitate f
#### Difficulty Adjustment #### Difficulty Adjustment
The proof-of-work target is adjusted every block to better adapt to sudden changes in hash rate. The exact adjustment algorithm can be seen [here](https://github.com/lbryio/lbrycrd/blob/v0.17.3.2/src/lbry.cpp). The proof-of-work target is adjusted every block to better adapt to sudden changes in hash rate. The exact adjustment algorithm can be seen [here](https://github.com/lbryio/lbrycrd/blob/master/src/lbry.cpp).
#### Block Hash Algorithm #### Block Hash Algorithm
LBRY uses a combination of SHA-256, SHA-512, and RIPEMD-160. The exact hashing algorithm can be seen [here](https://github.com/lbryio/lbrycrd/blob/v0.17.3.2/src/hash.cpp#L15). LBRY uses a combination of SHA-256, SHA-512, and RIPEMD-160. The exact hashing algorithm can be seen [here](https://github.com/lbryio/lbrycrd/blob/master/src/hash.cpp#L18).
#### Block Rewards #### Block Rewards
The block reward schedule was adjusted to provide an initial testing period, a quick ramp-up to max block rewards, then a logarithmic decay to 0. The source for the algorithm is [here](https://github.com/lbryio/lbrycrd/blob/v0.17.3.2/src/validation.cpp#L1025). The block reward schedule was adjusted to provide an initial testing period, a quick ramp-up to max block rewards, then a logarithmic decay to 0. The source for the algorithm is [here](https://github.com/lbryio/lbrycrd/blob/master/src/main.cpp#L1594).
#### Addresses #### Addresses
The address version byte is set to `0x55` for standard (pay-to-public-key-hash) addresses and `0x7a` for multisig (pay-to-script-hash) addresses. P2PKH addresses start with the letter `b`, and P2SH addresses start with `r`. The address version byte is set to `0x55` for standard (pay-to-public-key-hash) addresses and `0x7a` for multisig (pay-to-script-hash) addresses. P2PKH addresses start with the letter `b`, and P2SH addresses start with `r`.
All the chain parameters are defined [here](https://github.com/lbryio/lbrycrd/blob/v0.17.3.2/src/chainparams.cpp). All the chain parameters are defined [here](https://github.com/lbryio/lbrycrd/blob/master/src/chainparams.cpp).
## Metadata ## Metadata
@ -535,9 +532,9 @@ Metadata is structured information about a stream or channel separate from the c
Metadata is stored in a serialized binary format using [Protocol Buffers](https://developers.google.com/protocol-buffers/). This allows for metadata to be: Metadata is stored in a serialized binary format using [Protocol Buffers](https://developers.google.com/protocol-buffers/). This allows for metadata to be:
- **Extensible**. Metadata can encompass thousands of fields for dozens of types of content. It must be efficient to both modify the structure and maintain backward compatibility. - **Extensibile**. Metadata can encompass thousands of fields for dozens of types of content. It must be efficient to both modify the structure and maintain backward compatibility.
- **Compact**. Blockchain space is expensive. Data must be stored as compactly as possible. - **Compact**. Blockchain space is expensive. Data must be stored as compactly as possible.
- **Interoperable**. Metadata will be used by many projects written in different languages. - **Interoperabile**. Metadata will be used by many projects written in different languages.
The serialized metadata may be cryptographically signed to indicate membership in a channel. See [Channels](#channels) for more info. The serialized metadata may be cryptographically signed to indicate membership in a channel. See [Channels](#channels) for more info.
@ -630,15 +627,15 @@ The purpose of channels is to allow content to be clustered under a single pseud
#### Signing #### Signing
A claim is considered part of a channel when the metadata in its value is signed by the channel's private key. Here's the structure of a signed value: A claim is considered part of a channel when its metadata is signed by the channel's private key. Here's the structure of a signed metadata value:
field | size | description field | size | description
:--- | :--- | :--- :--- | :--- | :---
Version | 1 byte | Format version. See [Format Versions](#format-versions). Version | 1 byte | Format version. See [Format Versions](#format-versions).
Channel Claim ID | 20 bytes | Claim ID of the channel claim that contains the matching public key. _Skip this field if there is no signature._ Channel ID | 20 bytes | Claim ID of the channel claim that contains the matching public key. _Skip this field if there is no signature._
Signature | 64 bytes | The signature. _Skip this field if there is no signature._ Signature | 64 bytes | The signature. _Skip this field if there is no signature._
Metadata | variable | The protobuf-encoded metadata. Payload | variable | The protobuf-encoded metadata.
##### Format Versions ##### Format Versions
@ -653,13 +650,9 @@ format | description
##### Signing Process ##### Signing Process
1. Encode the metadata using protobuf. 1. Encode the metadata using protobuf.
1. Create the payload to be signed by concatenating the following: 2. Hash the encoded claim using SHA-256.
- The outpoint hash the first transaction input of this claim's transaction. This is in the payload to prevent replay attacks. 3. Sign the hash using the private key associated with the channel.
- The claim ID of the channel. See <a href="#stake-identifier-generation">this</a> for more on outpoints and claim IDs. 4. Append all the values (the version, the claim ID of the corresponding channel claim, the signature, and the protobuf-encoded metadata).
- The encoded metadata
1. Hash the payload using SHA-256.
1. Sign the payload hash using the private key associated with the channel.
1. Concatenate the version, the channel claim ID, the payload signature, and the protobuf-encoded metadata. This is the claim value.
##### Signature Validation ##### Signature Validation
@ -667,8 +660,7 @@ format | description
1. Check the version field. If it indicates that there is no signature, then no validation is necessary. 1. Check the version field. If it indicates that there is no signature, then no validation is necessary.
1. Split out the channel ID and signature from the rest of the data. 1. Split out the channel ID and signature from the rest of the data.
1. Look up the channel claim to ensure it exists and contains a public key. 1. Look up the channel claim to ensure it exists and contains a public key.
1. Create the payload hash as described <a href="#signing-process">above</a>. 1. Use the public key to verify the signature.
1. Use the public key to verify that the payload hash signature is valid.
### Validation {#metadata-validation} ### Validation {#metadata-validation}
@ -676,6 +668,7 @@ format | description
The blockchain treats metadata as an opaque series of bytes. Clients should not trust the metadata they read from the blockchain. Each client is responsible for correctly encoding and decoding the metadata, and for validating its structure and signatures. This allows evolution of the metadata definition without changes to blockchain consensus rules. The blockchain treats metadata as an opaque series of bytes. Clients should not trust the metadata they read from the blockchain. Each client is responsible for correctly encoding and decoding the metadata, and for validating its structure and signatures. This allows evolution of the metadata definition without changes to blockchain consensus rules.
## Data ## Data
Files published using LBRY are stored in a distributed fashion by the clients participating in the network. Each file is split into many small pieces. Each piece is encrypted and [announced](#announce) to the network. The pieces may also be uploaded to other hosts on the network that specialize in rehosting content. Files published using LBRY are stored in a distributed fashion by the clients participating in the network. Each file is split into many small pieces. Each piece is encrypted and [announced](#announce) to the network. The pieces may also be uploaded to other hosts on the network that specialize in rehosting content.
@ -749,6 +742,7 @@ The `version` field is always 1. It is intended to signal structure changes in f
Every stream must have at least two blobs - the manifest blob and a content blob. Consequently, zero-length streams are not allowed. Every stream must have at least two blobs - the manifest blob and a content blob. Consequently, zero-length streams are not allowed.
#### Stream Encoding #### Stream Encoding
A file must be encoded into a stream before it can be published. Encoding involves breaking the file into chunks, encrypting the chunks into content blobs, and creating the manifest blob. Here are the steps: A file must be encoded into a stream before it can be published. Encoding involves breaking the file into chunks, encrypting the chunks into content blobs, and creating the manifest blob. Here are the steps:
@ -760,7 +754,7 @@ A file must be encoded into a stream before it can be published. Encoding involv
##### Content Blobs ##### Content Blobs
1. Break the file into chunks of at most 2097151 bytes. 1. Break the file into chunks of at most 2097151 bytes.
1. Generate a random 32-byte initialization vector (IV) for each chunk. 1. Generate a random 32-byte initialization vector (IV) for each chuck.
1. Pad each chunk using PKCS7 padding. 1. Pad each chunk using PKCS7 padding.
1. Encrypt each chunk with AES-CBC using the stream key and the IV for that chunk. 1. Encrypt each chunk with AES-CBC using the stream key and the IV for that chunk.
1. An encrypted chunk is a blob. 1. An encrypted chunk is a blob.
@ -780,13 +774,14 @@ An implementation of this process is available [here](https://github.com/lbryio/
Decoding a stream is like encoding in reverse, and with the added step of verifying that the expected blob hashes match the actual data. Decoding a stream is like encoding in reverse, and with the added step of verifying that the expected blob hashes match the actual data.
1. Verify that the hash of the manifest blob matches the stream hash. 1. Verify that the hash of the manifest blob and matches the stream hash.
2. Parse the JSON in the manifest blob. 2. Parse the JSON in manifest blob.
3. Verify the hashes of the content blobs. 3. Verify the hashes of the content blobs.
4. Decrypt and remove the padding from each content blob using the stream key and IVs in the manifest. 4. Decrypt and remove the padding from each content blob using the stream key and IVs in the manifest.
5. Concatenate the decrypted chunks in order. 5. Concatenate the decrypted chunks in order.
### Announce ### Announce
After a stream is encoded, it must be _announced_ to the network. Announcing is the process of letting other nodes on the network know that a client has content available for download. LBRY tracks announced content using a distributed hash table. After a stream is encoded, it must be _announced_ to the network. Announcing is the process of letting other nodes on the network know that a client has content available for download. LBRY tracks announced content using a distributed hash table.
@ -846,6 +841,8 @@ Upload sends a blob to the server. If uploading many blobs, the client should us
The protocol methods and message types are defined in detail [here](https://github.com/lbryio/lbry.go/blob/master/blobex/blobex.proto). The protocol methods and message types are defined in detail [here](https://github.com/lbryio/lbry.go/blob/master/blobex/blobex.proto).
### Reflectors and Data Markets ### Reflectors and Data Markets
In order for a client to download content, there must be hosts online that have the content the client wants, when the client wants it. To incentivize the continued hosting of data, the blob exchange protocol supports data upload and payment for data. _Reflectors_ are hosts that accept data uploads. They rehost (reflect) the uploaded data and charge for downloads. In order for a client to download content, there must be hosts online that have the content the client wants, when the client wants it. To incentivize the continued hosting of data, the blob exchange protocol supports data upload and payment for data. _Reflectors_ are hosts that accept data uploads. They rehost (reflect) the uploaded data and charge for downloads.
@ -896,7 +893,7 @@ _--_ | apple | 690eea | 10
@Bryan | _--_ | 0da517 | 1 @Bryan | _--_ | 0da517 | 1
@Chris | _--_ | b3f7b1 | 1 @Chris | _--_ | b3f7b1 | 1
@Chris | banana | fc861c | 1 @Chris | banana | fc861c | 1
@Arthur | apple | a37ee1 | 20 @Arthur | apple | 37ee1 | 20
@Bryan | cherry | a18bca | 10 @Bryan | cherry | a18bca | 10
@Chris | _--_ | 005a7d | 100 @Chris | _--_ | 005a7d | 100
@Arthur | cherry | d39aa0 | 20 @Arthur | cherry | d39aa0 | 20
@ -909,23 +906,25 @@ URL | Claim ID
`lbry://banana` | 714a3f `lbry://banana` | 714a3f
`lbry://@Chris` | 005a7d `lbry://@Chris` | 005a7d
`lbry://@Chris/banana` | _not found_ (the controlling `@Chris` does not have a `banana`) `lbry://@Chris/banana` | _not found_ (the controlling `@Chris` does not have a `banana`)
`lbry://@Chris*1/banana` | fc861c `lbry://@Chris:1/banana` | fc861c
`lbry://@Chris:fc8/banana` | fc861c `lbry://@Chris:#fc8/banana` | fc861c
`lbry://cherry` | bfaabb `lbry://cherry` | bfaabb
`lbry://@Arthur/cherry` | d39aa0 `lbry://@Arthur/cherry` | d39aa0
`lbry://@Bryan` | 0da517 `lbry://@Bryan` | 0da517
`lbry://banana$1` | 714a3f `lbry://banana$1` | 714a3f
`lbry://banana$2` | fc861c `lbry://banana$2` | fc861c
`lbry://banana$3` | _not found_ `lbry://banana$3` | _not found_
`lbry://@Arthur*1` | b7bab5 `lbry://@Arthur:1` | b7bab5
### Additional Resources ### Additional Resources
- [lbry.tech](https://lbry.tech) is designed specifically for a technical audience. There you will find detailed explanations and examples, ways to interact with the LBRY community, and guidance for anyone who wants to get more involved. - [lbry.tech](https://lbry.tech) is LBRY's website designed specifically for anyone capable of understanding this document.
- [github.com/lbryio](https://github.com/lbryio) contains MIT-licensed source code and implementation details for the protocol and related software. - [github.com/lbryio](https://github.com/lbryio) contains MIT-licensed source code and implementation details for everything explained by this document.
- [lbry.com](https://lbry.com) is an application built on top of LBRY. It's one example of the type of end-to-end experience that is possible to create using this technology. - [lbry.io](https://lbry.io) is LBRY's consumer, creator, and end-user website.
- [lbry.io/get](https://lbry.io/get) provides downloads for LBRY's end-user applications. They're significantly less stuffy than this spec.
- [lbry.tech/resources](https://lbry.tech/resources) provides API specifications, additional resources, and other items of technical interest.
- [lbry.tech/contribute](https://lbry.tech/contribute) explains how you can get involved. If you're still reading at this point then you're probably interested, but if you're on the fence then [please watch this video](http://spee.ch/doit.mp4).
- [lbry.tech/community](https://lbry.tech/community) lists how you can interact with us, other users, and keep up-to-date.
<pre class="pdf-hide" style="font: 10px/5px monospace;overflow:hidden;text-align: center;margin: 10rem 0"> <pre class="pdf-hide" style="font: 10px/5px monospace;overflow:hidden;text-align: center;margin: 10rem 0">
@ -997,4 +996,4 @@ URL | Claim ID
--- ---
_[Improve this page on Github](https://github.com/lbryio/spec/blob/master/index.md)_ _[Edit this on Github](https://github.com/lbryio/spec/blob/master/index.md)_

BIN
lbry-spec.pdf
View file

Binary file not shown.

8
readme.md
View file

@ -7,7 +7,7 @@ This repo contains the official protocol specification for LBRY. It is built wit
## Usage ## Usage
The spec is available at https://spec.lbry.com. The spec is available at https://spec.lbry.io.
To view the content locally, you must have [Ruby](https://www.ruby-lang.org/en/documentation/installation/) and [bundler](https://bundler.io#getting-started) installed. Then run `bundle install` and `bundle exec jekyll serve`, and open `localhost:4000` in your browser. To view the content locally, you must have [Ruby](https://www.ruby-lang.org/en/documentation/installation/) and [bundler](https://bundler.io#getting-started) installed. Then run `bundle install` and `bundle exec jekyll serve`, and open `localhost:4000` in your browser.
@ -20,7 +20,7 @@ To edit the spec:
- make changes to the markdown - make changes to the markdown
- refresh `localhost:4000` in your browser to see the effect of your changes. - refresh `localhost:4000` in your browser to see the effect of your changes.
Contributions to this project are welcome, encouraged, and compensated. For more details, [click here](https://lbry.com/faq/contributing). Contributions to this project are welcome, encouraged, and compensated. For more details, [click here](https://lbry.io/faq/contributing).
## License ## License
@ -30,9 +30,9 @@ This project is MIT licensed. For the full license, see [LICENSE](LICENSE).
## Security ## Security
We take security seriously. Please contact security@lbry.com regarding any security issues. [Our PGP key is here](https://lbry.com/faq/pgp-key) if you need it. We take security seriously. Please contact security@lbry.io regarding any security issues. [Our GPG key is here](https://lbry.io/faq/gpg-key) if you need it.
## Contact ## Contact
The primary contact for this project is [@lyoshenka](https://github.com/lyoshenka) (grin@lbry.com). The primary contact for this project is [@lyoshenka](https://github.com/lyoshenka) (grin@lbry.io).