LBRYCommunity/spec
1
0
Fork 0
Code Issues 5 Pull requests Projects Releases Packages Wiki Activity

cleaned up some fixmes

Browse source
This commit is contained in:
Alex Grintsvayg 2018-11-27 13:48:31 -05:00
parent 4187928e6a
commit 71e67e5e99
2 changed files with 189 additions and 205 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

284
index.html
View file

@ -85,99 +85,97 @@
<!-- this TOC is autogenerated for github preview or js-challenged browsers -->
<!--ts-->
- [LBRY: A Decentralized Digital Content Marketplace](#lbry-a-decentralized-digital-content-marketplace)
- [Table of Contents](#table-of-contents)
- [Introduction](#introduction)
- [Overview](#overview)
- [Assumptions](#assumptions)
- [Conventions and Terminology](#conventions-and-terminology)
- [Blockchain](#blockchain)
- [Stakes](#stakes)
- [Claims](#claims)
- [Claim Properties](#claim-properties)
- [Example Claim](#example-claim)
- [Claim Operations](#claim-operations)
- [Supports](#supports)
- [Claimtrie](#claimtrie)
- [Statuses {#stake-statuses}](#statuses-stake-statuses)
- [Accepted](#accepted)
- [Abandoned](#abandoned)
- [Active](#active)
- [Controlling (claims only) {#controlling}](#controlling-claims-only-controlling)
- [Claimtrie Transitions](#claimtrie-transitions)
- [Stake Activation](#stake-activation)
- [Example {#transition-example}](#example-transition-example)
- [Normalization](#normalization)
- [URLs](#urls)
- [Components](#components)
- [Stream Claim Name](#stream-claim-name)
- [Channel Claim Name](#channel-claim-name)
- [Channel Claim Name and Stream Claim Name](#channel-claim-name-and-stream-claim-name)
- [Claim ID](#claim-id)
- [Claim Sequence](#claim-sequence)
- [Bid Position](#bid-position)
- [Query Params](#query-params)
- [Grammar](#grammar)
- [Resolution](#resolution)
- [No Modifier](#no-modifier)
- [Claim ID](#claim-id-1)
- [Claim Sequence](#claim-sequence-1)
- [Bid Position](#bid-position-1)
- [ChannelName and ClaimName](#channelname-and-claimname)
- [Examples {#url-resolution-examples}](#examples-url-resolution-examples)
- [Design Notes](#design-notes)
- [Transactions](#transactions)
- [Operations and Opcodes](#operations-and-opcodes)
- [Claim Identifier Generation](#claim-identifier-generation)
- [OP\_CLAIM\_NAME](#opclaimname)
- [OP\_UPDATE\_CLAIM](#opupdateclaim)
- [OP\_SUPPORT\_CLAIM](#opsupportclaim)
- [Tips](#tips)
- [Addresses](#addresses)
- [Proof of Payment](#proof-of-payment)
- [Consensus](#consensus)
- [Block Timing](#block-timing)
- [Difficulty Adjustment](#difficulty-adjustment)
- [Block Hash Algorithm](#block-hash-algorithm)
- [Block Rewards](#block-rewards)
- [Metadata](#metadata)
- [Specification](#specification)
- [Example {#metadata-example}](#example-metadata-example)
- [Key Fields](#key-fields)
- [Source and Stream Hashes](#source-and-stream-hashes)
- [Fees and Fee Structure](#fees-and-fee-structure)
- [Title, Author, Description](#title-author-description)
- [Language](#language)
- [Thumbnail](#thumbnail)
- [Media Type](#media-type)
- [Channels (Identities) {#channels}](#channels-identities-channels)
- [Signing](#signing)
- [Format Versions](#format-versions)
- [Signing Process](#signing-process)
- [Signature Validation](#signature-validation)
- [Validation {#metadata-validation}](#validation-metadata-validation)
- [Data](#data)
- [Encoding](#encoding)
- [Blobs](#blobs)
- [Streams](#streams)
- [Manifest Contents](#manifest-contents)
- [Stream Encoding](#stream-encoding)
- [Setup](#setup)
- [Content Blobs](#content-blobs)
- [Manifest Blob](#manifest-blob)
- [Stream Decoding](#stream-decoding)
- [Announce](#announce)
- [Distributed Hash Table](#distributed-hash-table)
- [Announcing to the DHT](#announcing-to-the-dht)
- [Download](#download)
- [Querying the DHT](#querying-the-dht)
- [Blob Exchange Protocol](#blob-exchange-protocol)
- [PriceCheck](#pricecheck)
- [DownloadCheck](#downloadcheck)
- [Download](#download-1)
- [UploadCheck](#uploadcheck)
- [Upload](#upload)
- [Reflectors and Data Markets](#reflectors-and-data-markets)
* [Introduction](#introduction)
* [Overview](#overview)
* [Assumptions](#assumptions)
* [Conventions and Terminology](#conventions-and-terminology)
* [Blockchain](#blockchain)
* [Stakes](#stakes)
* [Claims](#claims)
* [Claim Properties](#claim-properties)
* [Example Claim](#example-claim)
* [Claim Operations](#claim-operations)
* [Supports](#supports)
* [Claimtrie](#claimtrie)
* [Statuses](#stake-statuses)
* [Accepted](#accepted)
* [Abandoned](#abandoned)
* [Active](#active)
* [Controlling (claims only)](#controlling)
* [Claimtrie Transitions](#claimtrie-transitions)
* [Stake Activation](#stake-activation)
* [Example](#transition-example)
* [Normalization](#normalization)
* [URLs](#urls)
* [Components](#components)
* [Stream Claim Name](#stream-claim-name)
* [Channel Claim Name](#channel-claim-name)
* [Channel Claim Name and Stream Claim Name](#channel-claim-name-and-stream-claim-name)
* [Claim ID](#claim-id)
* [Claim Sequence](#claim-sequence)
* [Bid Position](#bid-position)
* [Query Params](#query-params)
* [Grammar](#grammar)
* [Resolution](#resolution)
* [No Modifier](#no-modifier)
* [Claim ID](#claim-id-1)
* [Claim Sequence](#claim-sequence-1)
* [Bid Position](#bid-position-1)
* [ChannelName and ClaimName](#channelname-and-claimname)
* [Examples](#url-resolution-examples)
* [Design Notes](#design-notes)
* [Transactions](#transactions)
* [Operations and Opcodes](#operations-and-opcodes)
* [Claim Identifier Generation](#claim-identifier-generation)
* [OP_CLAIM_NAME](#op-claim-name)
* [OP_UPDATE_CLAIM](#op-update-claim)
* [OP_SUPPORT_CLAIM](#op-support-claim)
* [Tips](#tips)
* [Proof of Payment](#proof-of-payment)
* [Consensus](#consensus)
* [Block Timing](#block-timing)
* [Difficulty Adjustment](#difficulty-adjustment)
* [Block Hash Algorithm](#block-hash-algorithm)
* [Block Rewards](#block-rewards)
* [Addresses](#addresses)
* [Metadata](#metadata)
* [Specification](#specification)
* [Example](#metadata-example)
* [Key Fields](#key-fields)
* [Stream Hash](#stream-hash)
* [Fee](#fee)
* [Title, Author, Description](#title-author-description)
* [Language](#language)
* [Thumbnail](#thumbnail)
* [Media Type](#media-type)
* [Channels (Identities)](#channels)
* [Signing](#signing)
* [Format Versions](#format-versions)
* [Signing Process](#signing-process)
* [Signature Validation](#signature-validation)
* [Validation](#metadata-validation)
* [Data](#data)
* [Encoding](#encoding)
* [Blobs](#blobs)
* [Streams](#streams)
* [Manifest Contents](#manifest-contents)
* [Stream Encoding](#stream-encoding)
* [Setup](#setup)
* [Content Blobs](#content-blobs)
* [Manifest Blob](#manifest-blob)
* [Stream Decoding](#stream-decoding)
* [Announce](#announce)
* [Distributed Hash Table](#distributed-hash-table)
* [Announcing to the DHT](#announcing-to-the-dht)
* [Download](#download)
* [Querying the DHT](#querying-the-dht)
* [Blob Exchange Protocol](#blob-exchange-protocol)
* [PriceCheck](#pricecheck)
* [DownloadCheck](#downloadcheck)
* [Download](#download-1)
* [UploadCheck](#uploadcheck)
* [Upload](#upload)
* [Reflectors and Data Markets](#reflectors-and-data-markets)
<!--te-->
</noscript>
@ -189,7 +187,9 @@ fixme final polish checklist:
- standardize when we say "we do X" vs "LBRY does X"
- check that all anchors work
- check css across browsers/mobile
-
- create links for [[terms]]
- ensure that all italicized terms are defined before they are used, or if that doesn't work, that they are linked
- don't say "the LBRY network". instead say "LBRY" or say nothing.
-->
@ -209,19 +209,15 @@ fixme final polish checklist:
<h3 id="overview">Overview</h3>
<!-- fix me -->
<p>This document defines the LBRY protocol, its components, and how they fit together. LBRY consists of several discrete components that are used together in order to provide the end-to-end capabilities of the protocol. There are two distributed data stores (blockchain and DHT), a peer-to-peer protocol for exchanging data, and several specifications for data structure, transformation, and retrieval.</p>
<p>This document defines the LBRY protocol, its components, and how they fit together. LBRY consists of several discrete components that are used together in order to provide the end-to-end capabilities of the protocol. There are two distributed data stores (blockchain and DHT), a peer-to-peer protocol for exchanging data, and several specifications for data structure, encoding, and retrieval.</p>
<h3 id="assumptions">Assumptions</h3>
<!-- fix me -->
<p>This document assumes that the reader is familiar with Bitcoin and blockchain technology. It does not attempt to document the Bitcoin protocol or explain how it works. The <a href="https://bitcoin.org/en/developer-reference">Bitcoin developer reference</a> is recommended for anyone wishing to understand the technical details.</p>
<h3 id="conventions-and-terminology">Conventions and Terminology</h3>
<!-- fix me - rather than inline this here, I think we should use lbry.tech glossary definitions and the [[keyword]] syntax -->
<!-- fixme - rather than inline this here, I think we should use lbry.tech glossary definitions and the [[keyword]] syntax -->
<dl>
<dt>file</dt>
@ -268,7 +264,7 @@ fixme final polish checklist:
<li>Trustful publisher identities</li>
</ol>
<p>The LBRY blockchain is a fork of the <a href="https://bitcoin.org/bitcoin.pdf">Bitcoin</a> blockchain, with substantial modifications. This document will not cover or specify any aspects of LBRY that are identical to Bitcoin, and will instead focus on the differences.</p>
<p>The blockchain is a fork of the <a href="https://bitcoin.org/bitcoin.pdf">Bitcoin</a> blockchain, with substantial modifications. This document will not cover or specify any aspects of LBRY that are identical to Bitcoin, and will instead focus on the differences.</p>
<h3 id="stakes">Stakes</h3>
@ -347,7 +343,7 @@ fixme final polish checklist:
<h4 id="supports">Supports</h4>
<p>A <em>support</em> is a stake that lends its <em>amount</em> to an existing claim.</p>
<p>A <em>support</em> is a stake that lends its amount to an existing claim.</p>
<p>Supports have one extra property on top of the basic stake properties: a <code>claim_id</code>. This is the ID of the claim that the support is bolstering.</p>
@ -359,11 +355,9 @@ fixme final polish checklist:
<p>The claimtrie is implemented as a <a href="https://en.wikipedia.org/wiki/Merkle_tree">Merkle tree</a> that maps names to claims. Claims are stored as leaf nodes in the tree. Names are stored as the path from the root node to the leaf node.</p>
<p>The <em>root hash</em> is the hash of the root node. It is stored in the header of each block in the blockchain. Nodes in the LBRY network use the root hash to efficiently and securely validate the state of the claimtrie.</p>
<p>The <em>root hash</em> is the hash of the root node. It is stored in the header of each block in the blockchain. Nodes use the root hash to efficiently and securely validate the state of the claimtrie.</p>
<p>Multiple claims can exist for the same name. They are all stored in the leaf node for that name, sorted by the amount of credits backing the claim (descending), then by block height (ascending), then by transaction order in the block (ascending).</p>
<!-- fix me above? "amount of credits backing each claim" is the effective amount, but that's not defined till later -->
<p>Multiple claims can exist for the same name. They are all stored in the leaf node for that name, sorted by <a href="descending">[effective amount]</a>, then by block height (ascending), then by transaction order in the block (ascending).</p>
<p>For more details on the specific claimtrie implementation, see <a href="https://github.com/lbryio/lbrycrd/blob/master/src/claimtrie.cpp">the source code</a>.</p>
@ -387,15 +381,13 @@ fixme final polish checklist:
<h5 id="active">Active</h5>
<!-- fix me a lot -->
<p>An <em>active</em> stake is an accepted and non-abandoned stake that has been in the blockchain for an algorithmically determined number of blocks. This length of time required is called the <em>activation delay</em>.</p>
<p>If the stake is an update to an already active claim, is the first claim for a name, or does not cause a change in which claim is controlling the name, the activation delay is 0 (i.e. the stake becomes active in the same block it is accepted).</p>
<p>If the stake is an update to an active claim, is the first claim for a name, or does not cause a change in which claim is controlling the name, the activation delay is 0 (i.e. the stake becomes active in the same block it is accepted).</p>
<p>Otherwise, the activation delay is determined by a formula covered in <a href="#claimtrie-transitions">Claimtrie Transitions</a>. The formula&rsquo;s inputs are the height of the current block, the height at which the stake was accepted, and the height at which the controlling claim for that name last changed.</p>
<p>The sum of the amount of an active claim and all active supports is called it&rsquo;s <em>effective amount</em>. Only the effective amount affects the sort order of claims in a leaf node. Claims that are not active have an effective amount of 0.</p>
<p>The sum of the amount of an active claim and all active supports is called it&rsquo;s <em>effective amount</em>. The effective amount affects the sort order of claims in a leaf node, and which claim is controlling for that name. Claims that are not active have an effective amount of 0.</p>
<h5 id="controlling">Controlling (claims only)</h5>
@ -471,7 +463,7 @@ fixme final polish checklist:
<h3 id="urls">URLs</h3>
<!-- fix me:
<!-- fixme:
jeremy: @grin does SPV need a mention inside of the document?
grin: no, but we should probably include an example for how to do the validation using the root hash. its not strictly necessary because its similar to how bitcoin does it. so maybe link to https://lbry.tech/resources/claimtrie (which needs an update) and add a validation example there?
-->
@ -607,9 +599,7 @@ Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]
<h5 id="channelname-and-claimname">ChannelName and ClaimName</h5>
<!-- fix me: explain how claim signing works, and what it means to be **in** a channel -->
<p>If both a channel name and a claim name is present, resolution happens in two steps. First, remove the <code>/</code> and <code>StreamClaimNameAndModifier</code> from the path, and resolve the URL as if it only had a <code>ChannelClaimNameAndModifier</code>. Then get the list of all claims in that channel. Finally, resolve the <code>StreamClaimNameAndModifier</code> as if it was its own URL, but instead of considering all claims, only consider the set of claims in the channel.</p>
<p>If both a channel name and a claim name are present, resolution happens in two steps. First, remove the <code>/</code> and <code>StreamClaimNameAndModifier</code> from the path, and resolve the URL as if it only had a <code>ChannelClaimNameAndModifier</code>. Then get the list of all claims in that channel. Finally, resolve the <code>StreamClaimNameAndModifier</code> as if it was its own URL, but instead of considering all claims, only consider the set of claims in the channel.</p>
<p>If multiple claims for the same name exist inside the same channel, they are resolved via the same resolution rules applied entirely within the sub-scope of the channel.</p>
@ -867,13 +857,7 @@ OP_SUPPORT_CLAIM &lt;name&gt; &lt;claimId&gt; OP_2DROP OP_DROP &lt;outputScript&
<h4 id="tips">Tips</h4>
<!-- fixme: describe how tips are different from supports -->
<h4 id="addresses">Addresses</h4>
<p>The address version byte is set to <code>0x55</code> for standard (pay-to-public-key-hash) addresses and <code>0x7a</code> for multisig (pay-to-script-hash) addresses. P2PKH addresses start with the letter <code>b</code>, and P2SH addresses start with <code>r</code>.</p>
<p>All the chain parameters are defined <a href="https://github.com/lbryio/lbrycrd/blob/master/src/chainparams.cpp">here</a>.</p>
<p>Since a claim is updated or abandoned by spending the claim&rsquo;s transaction, most claimtrie transactions spend to an address controlled by the transaction&rsquo;s creator. However, a claimtrie transaction can spend to any address. This can be used to create a <em>tip</em>. A tip is a support that also sends credits to the claim&rsquo;s creator. Tips in LBRY are just like tips in real life. They can be used to express gratitude or make an optional payment to a content creator.</p>
<h4 id="proof-of-payment">Proof of Payment</h4>
@ -901,11 +885,17 @@ OP_SUPPORT_CLAIM &lt;name&gt; &lt;claimId&gt; OP_2DROP OP_DROP &lt;outputScript&
<p>The block reward schedule was adjusted to provide an initial testing period, a quick ramp-up to max block rewards, then a logarithmic decay to 0. The source for the algorithm is <a href="https://github.com/lbryio/lbrycrd/blob/master/src/main.cpp#L1594">here</a>.</p>
<h4 id="addresses">Addresses</h4>
<p>The address version byte is set to <code>0x55</code> for standard (pay-to-public-key-hash) addresses and <code>0x7a</code> for multisig (pay-to-script-hash) addresses. P2PKH addresses start with the letter <code>b</code>, and P2SH addresses start with <code>r</code>.</p>
<p>All the chain parameters are defined <a href="https://github.com/lbryio/lbrycrd/blob/master/src/chainparams.cpp">here</a>.</p>
<h2 id="metadata">Metadata</h2>
<p>Metadata is structured information about the stream or channel separate from the content itself (e.g. the title, language, media type, etc.). It is stored in the <a href="#claim-properties">value property</a> of a claim.</p>
<p>Metadata is structured information about a stream or channel separate from the content itself (e.g. the title, language, media type, etc.). It is stored in the blockchain as the <a href="#claim-properties">value property</a> of a claim.</p>
<p>Metadata is stored in a serialized binary format via <a href="https://developers.google.com/protocol-buffers/">Protocol Buffers</a>. This allows for metadata to be:</p>
<p>Metadata is stored in a serialized binary format using <a href="https://developers.google.com/protocol-buffers/">Protocol Buffers</a>. This allows for metadata to be:</p>
<ul>
<li><strong>Extensibile</strong>. Metadata can encompass thousands of fields for dozens of types of content. It must be efficient to both modify the structure and maintain backward compatibility.</li>
@ -913,11 +903,11 @@ OP_SUPPORT_CLAIM &lt;name&gt; &lt;claimId&gt; OP_2DROP OP_DROP &lt;outputScript&
<li><strong>Interoperabile</strong>. Metadata will be used by many projects written in different languages.</li>
</ul>
<p>The serialized metadata may be signed to indicate membership in a channel. See <a href="#channels">Channels</a> for more info.</p>
<p>The serialized metadata may be cryptographically signed to indicate membership in a channel. See <a href="#channels">Channels</a> for more info.</p>
<h3 id="specification">Specification</h3>
<p>As the metadata specification is designed to grow and change frequently, the full specification will not be examined here. The <a href="https://github.com/lbryio/types">types</a> repository is considered the precise specification.</p>
<p>The metadata specification is designed to grow and change frequently. The full specification will not be detailed here. The <a href="https://github.com/lbryio/types">types</a> repository is considered the precise specification.</p>
<p>Instead, let&rsquo;s look at an example and some key fields.</p>
@ -943,19 +933,22 @@ OP_SUPPORT_CLAIM &lt;name&gt; &lt;claimId&gt; OP_2DROP OP_DROP &lt;outputScript&
<p>Some important metadata fields are highlighted below.</p>
<h4 id="source-and-stream-hashes">Source and Stream Hashes</h4>
<h4 id="stream-hash">Stream Hash</h4>
<p>The <code>source</code> property contains information about how to fetch the data from the network. Within the <code>source</code> is a unique identifier to locate and find the content in the data network. More in [[Data]].</p>
<p>A unique identifier that is used to locate and fetch the content from the data network. More in <a href="#data">Data</a>.</p>
<h4 id="fees-and-fee-structure">Fees and Fee Structure</h4>
<h4 id="fee">Fee</h4>
<!-- fix me extensively -->
<p>Information on how to pay for the content. It includes the address that will receive the payment (the <em>fee address</em>), the the amount to be paid, and the currency. Only LBC and USD currencies are supported, though others may be added in the future.</p>
<ul>
<li>LBC</li>
<li>Currencies?</li>
<li>channel signatures and private keys</li>
</ul>
<p>Example fee:</p>
<pre><code>&quot;fee&quot;: {
&quot;address&quot;:&quot;bNz8Va7xMyK9eHA5APzLph6cCTjBtGgmDN&quot;,
&quot;amount&quot;:&quot;99.95&quot;,
&quot;currency&quot;:&quot;LBC&quot;
}
</code></pre>
<h4 id="title-author-description">Title, Author, Description</h4>
@ -975,7 +968,7 @@ OP_SUPPORT_CLAIM &lt;name&gt; &lt;claimId&gt; OP_2DROP OP_DROP &lt;outputScript&
<h3 id="channels">Channels (Identities)</h3>
<p>Channels are the unit of identity in the LBRY system. A channel is a claim for a name beginning with <code>@</code> that contains a metadata structure for identity rather than content. Included in the metadata is the channel&rsquo;s public key. Here&rsquo;s an example:</p>
<p>Channels are the unit of identity. A channel is a claim for a name beginning with <code>@</code> that contains a metadata structure for identity rather than content. Included in the metadata is the channel&rsquo;s public key. Here&rsquo;s an example:</p>
<pre><code>&quot;claim_id&quot;: &quot;6e56325c5351ceda2dd0795a30e864492910ccbf&quot;,
&quot;name&quot;: &quot;@lbry&quot;,
@ -1084,19 +1077,17 @@ OP_SUPPORT_CLAIM &lt;name&gt; &lt;claimId&gt; OP_2DROP OP_DROP &lt;outputScript&
<h2 id="data">Data</h2>
<!-- fixme this section -->
<p>Files published using LBRY are stored in a distributed fashion by the clients participating in the network. Each file split into multiple small pieces, encrypted, and announced to the network. It may also be uploaded to other hosts on the network that specialize in rehosting content.</p>
<p>Data refers to the full binary data which is ultimate distributed by blah blah blah.</p>
<p>The purpose of blah blah blah is to blah blah.</p>
<p>The purpose of this process is to enable file storage without relying on centralized infrastructure, and to create a marketplace for data that allows hosts to be paid for their services. The design is strongly influenced by the Bittorrent protocol and network.</p>
<h3 id="encoding">Encoding</h3>
<p>Content on the LBRY network is encoded to facilitate distribution.</p>
<p>Content on LBRY is encoded to facilitate distribution.</p>
<h4 id="blobs">Blobs</h4>
<p>The unit of data in the LBRY network is called a <em>blob</em>. A blob is an encrypted chunk of data up to 2MiB in size. Each blob is indexed by its <em>blob hash</em>, which is a SHA384 hash of the blob contents. Addressing blobs by their hash protects against naming collisions and ensures that the content you get is what you expect.</p>
<p>The smallest unit of data is called a <em>blob</em>. A blob is an encrypted chunk of data up to 2MiB in size. Each blob is indexed by its <em>blob hash</em>, which is a SHA384 hash of the blob contents. Addressing blobs by their hash protects against naming collisions and ensures that the content you get is what you expect.</p>
<p>Blobs are encrypted using AES-256 in CBC mode and PKCS7 padding. In order to keep each encrypted blob at 2MiB max, a blob can hold at most 2097151 bytes (2MiB minus 1 byte) of plaintext data. The source code for the exact algorithm is available <a href="https://github.com/lbryio/lbry.go/blob/master/stream/blob.go">here</a>. The encryption key and IV for each blob is stored as described below.</p>
@ -1184,8 +1175,9 @@ OP_SUPPORT_CLAIM &lt;name&gt; &lt;claimId&gt; OP_2DROP OP_DROP &lt;outputScript&
<li>Compute the stream hash.</li>
</ol>
<p>An implementation of this process is available <a href="https://github.com/lbryio/lbry.go/tree/master/stream">here</a>.
fixme: this link is for v0, not v1. need to implement v1 or drop the link.</p>
<p>An implementation of this process is available <a href="https://github.com/lbryio/lbry.go/tree/master/stream">here</a>.</p>
<!-- fixme: the above link is for v0, not v1. need to implement v1 or drop the link. -->
<h4 id="stream-decoding">Stream Decoding</h4>
@ -1201,7 +1193,7 @@ fixme: this link is for v0, not v1. need to implement v1 or drop the link.</p>
<h3 id="announce">Announce</h3>
<p>After a [[stream]] is encoded, it must be <em>announced</em> to the network. Announcing is the process of letting other nodes on the network know that you have content available for download. The LBRY network tracks announced content using a distributed hash table.</p>
<p>After a [[stream]] is encoded, it must be <em>announced</em> to the network. Announcing is the process of letting other nodes on the network know that you have content available for download. LBRY tracks announced content using a distributed hash table.</p>
<h4 id="distributed-hash-table">Distributed Hash Table</h4>
@ -1230,9 +1222,7 @@ specification fairly closely, with some modifications.</p>
<h4 id="blob-exchange-protocol">Blob Exchange Protocol</h4>
<p>Downloading a blob from a peer is governed by the <em>Blob Exchange Protocol</em>. It is used by hosts and clients to check blob availability, exchange blobs, and negotiate data prices. The protocol is an RPC protocol using Protocol Buffers and the gRPC framework. It has five types of requests.</p>
<p>fixme: protocol does not <strong>negotiate</strong> anything right now. It just checks the price. Should we include negotiation in v1?</p>
<p>Downloading a blob from a peer is governed by the <em>Blob Exchange Protocol</em>. It is used by hosts and clients to exchange blobs and check data pricing and blob availability. The protocol is an RPC protocol using Protocol Buffers and the gRPC framework. It has five types of requests.</p>
<h5 id="pricecheck">PriceCheck</h5>

110
index.md
View file

@ -66,19 +66,19 @@
* [OP_UPDATE_CLAIM](#op-update-claim)
* [OP_SUPPORT_CLAIM](#op-support-claim)
* [Tips](#tips)
* [Addresses](#addresses)
* [Proof of Payment](#proof-of-payment)
* [Consensus](#consensus)
* [Block Timing](#block-timing)
* [Difficulty Adjustment](#difficulty-adjustment)
* [Block Hash Algorithm](#block-hash-algorithm)
* [Block Rewards](#block-rewards)
* [Addresses](#addresses)
* [Metadata](#metadata)
* [Specification](#specification)
* [Example](#metadata-example)
* [Key Fields](#key-fields)
* [Source and Stream Hashes](#source-and-stream-hashes)
* [Fees and Fee Structure](#fees-and-fee-structure)
* [Stream Hash](#stream-hash)
* [Fee](#fee)
* [Title, Author, Description](#title-author-description)
* [Language](#language)
* [Thumbnail](#thumbnail)
@ -122,7 +122,9 @@ fixme final polish checklist:
- standardize when we say "we do X" vs "LBRY does X"
- check that all anchors work
- check css across browsers/mobile
-
- create links for [[terms]]
- ensure that all italicized terms are defined before they are used, or if that doesn't work, that they are linked
- don't say "the LBRY network". instead say "LBRY" or say nothing.
-->
@ -141,19 +143,15 @@ TODO:
### Overview
<!-- fix me -->
This document defines the LBRY protocol, its components, and how they fit together. LBRY consists of several discrete components that are used together in order to provide the end-to-end capabilities of the protocol. There are two distributed data stores (blockchain and DHT), a peer-to-peer protocol for exchanging data, and several specifications for data structure, transformation, and retrieval.
This document defines the LBRY protocol, its components, and how they fit together. LBRY consists of several discrete components that are used together in order to provide the end-to-end capabilities of the protocol. There are two distributed data stores (blockchain and DHT), a peer-to-peer protocol for exchanging data, and several specifications for data structure, encoding, and retrieval.
### Assumptions
<!-- fix me -->
This document assumes that the reader is familiar with Bitcoin and blockchain technology. It does not attempt to document the Bitcoin protocol or explain how it works. The [Bitcoin developer reference](https://bitcoin.org/en/developer-reference) is recommended for anyone wishing to understand the technical details.
### Conventions and Terminology
<!-- fix me - rather than inline this here, I think we should use lbry.tech glossary definitions and the [[keyword]] syntax -->
<!-- fixme - rather than inline this here, I think we should use lbry.tech glossary definitions and the [[keyword]] syntax -->
<dl>
<dt>file</dt>
@ -200,7 +198,7 @@ The LBRY blockchain is a public, proof-of-work blockchain. It serves three key p
2. A payment system and record of purchases for priced content
3. Trustful publisher identities
The LBRY blockchain is a fork of the [Bitcoin](https://bitcoin.org/bitcoin.pdf) blockchain, with substantial modifications. This document will not cover or specify any aspects of LBRY that are identical to Bitcoin, and will instead focus on the differences.
The blockchain is a fork of the [Bitcoin](https://bitcoin.org/bitcoin.pdf) blockchain, with substantial modifications. This document will not cover or specify any aspects of LBRY that are identical to Bitcoin, and will instead focus on the differences.
### Stakes
@ -278,7 +276,7 @@ There are three claim operations: _create_, _update_, and _abandon_.
#### Supports
A _support_ is a stake that lends its _amount_ to an existing claim.
A _support_ is a stake that lends its amount to an existing claim.
Supports have one extra property on top of the basic stake properties: a `claim_id`. This is the ID of the claim that the support is bolstering.
@ -290,11 +288,9 @@ The _claimtrie_ is the data structure used to store the set of all claims and pr
The claimtrie is implemented as a [Merkle tree](https://en.wikipedia.org/wiki/Merkle_tree) that maps names to claims. Claims are stored as leaf nodes in the tree. Names are stored as the path from the root node to the leaf node.
The _root hash_ is the hash of the root node. It is stored in the header of each block in the blockchain. Nodes in the LBRY network use the root hash to efficiently and securely validate the state of the claimtrie.
The _root hash_ is the hash of the root node. It is stored in the header of each block in the blockchain. Nodes use the root hash to efficiently and securely validate the state of the claimtrie.
Multiple claims can exist for the same name. They are all stored in the leaf node for that name, sorted by the amount of credits backing the claim (descending), then by block height (ascending), then by transaction order in the block (ascending).
<!-- fix me above? "amount of credits backing each claim" is the effective amount, but that's not defined till later -->
Multiple claims can exist for the same name. They are all stored in the leaf node for that name, sorted by [[effective amount]] (descending), then by block height (ascending), then by transaction order in the block (ascending).
For more details on the specific claimtrie implementation, see [the source code](https://github.com/lbryio/lbrycrd/blob/master/src/claimtrie.cpp).
@ -318,15 +314,13 @@ While data related to abandoned stakes still resides in the blockchain, it is im
##### Active
<!-- fix me a lot -->
An _active_ stake is an accepted and non-abandoned stake that has been in the blockchain for an algorithmically determined number of blocks. This length of time required is called the _activation delay_.
If the stake is an update to an already active claim, is the first claim for a name, or does not cause a change in which claim is controlling the name, the activation delay is 0 (i.e. the stake becomes active in the same block it is accepted).
If the stake is an update to an active claim, is the first claim for a name, or does not cause a change in which claim is controlling the name, the activation delay is 0 (i.e. the stake becomes active in the same block it is accepted).
Otherwise, the activation delay is determined by a formula covered in [Claimtrie Transitions](#claimtrie-transitions). The formula's inputs are the height of the current block, the height at which the stake was accepted, and the height at which the controlling claim for that name last changed.
The sum of the amount of an active claim and all active supports is called it's _effective amount_. Only the effective amount affects the sort order of claims in a leaf node. Claims that are not active have an effective amount of 0.
The sum of the amount of an active claim and all active supports is called it's _effective amount_. The effective amount affects the sort order of claims in a leaf node, and which claim is controlling for that name. Claims that are not active have an effective amount of 0.
##### Controlling (claims only) {#controlling}
@ -400,7 +394,7 @@ Names in the claimtrie are normalized prior to comparison to avoid confusion due
### URLs
<!-- fix me:
<!-- fixme:
jeremy: @grin does SPV need a mention inside of the document?
grin: no, but we should probably include an example for how to do the validation using the root hash. its not strictly necessary because its similar to how bitcoin does it. so maybe link to https://lbry.tech/resources/claimtrie (which needs an update) and add a validation example there?
-->
@ -543,9 +537,7 @@ Get all claims for the claim name. Sort the claims in descending order by total
##### ChannelName and ClaimName
<!-- fix me: explain how claim signing works, and what it means to be **in** a channel -->
If both a channel name and a claim name is present, resolution happens in two steps. First, remove the `/` and `StreamClaimNameAndModifier` from the path, and resolve the URL as if it only had a `ChannelClaimNameAndModifier`. Then get the list of all claims in that channel. Finally, resolve the `StreamClaimNameAndModifier` as if it was its own URL, but instead of considering all claims, only consider the set of claims in the channel.
If both a channel name and a claim name are present, resolution happens in two steps. First, remove the `/` and `StreamClaimNameAndModifier` from the path, and resolve the URL as if it only had a `ChannelClaimNameAndModifier`. Then get the list of all claims in that channel. Finally, resolve the `StreamClaimNameAndModifier` as if it was its own URL, but instead of considering all claims, only consider the set of claims in the channel.
If multiple claims for the same name exist inside the same channel, they are resolved via the same resolution rules applied entirely within the sub-scope of the channel.
@ -669,14 +661,7 @@ The `<address>` in this script may be the same as the address in the original tr
#### Tips
<!-- fixme: describe how tips are different from supports -->
#### Addresses
The address version byte is set to `0x55` for standard (pay-to-public-key-hash) addresses and `0x7a` for multisig (pay-to-script-hash) addresses. P2PKH addresses start with the letter `b`, and P2SH addresses start with `r`.
All the chain parameters are defined [here](https://github.com/lbryio/lbrycrd/blob/master/src/chainparams.cpp).
Since a claim is updated or abandoned by spending the claim's transaction, most claimtrie transactions spend to an address controlled by the transaction's creator. However, a claimtrie transaction can spend to any address. This can be used to create a _tip_. A tip is a support that also sends credits to the claim's creator. Tips in LBRY are just like tips in real life. They can be used to express gratitude or make an optional payment to a content creator.
#### Proof of Payment
@ -705,22 +690,28 @@ LBRY uses a combination of SHA-256, SHA-512, and RIPEMD-160. The exact hashing a
The block reward schedule was adjusted to provide an initial testing period, a quick ramp-up to max block rewards, then a logarithmic decay to 0. The source for the algorithm is [here](https://github.com/lbryio/lbrycrd/blob/master/src/main.cpp#L1594).
#### Addresses
The address version byte is set to `0x55` for standard (pay-to-public-key-hash) addresses and `0x7a` for multisig (pay-to-script-hash) addresses. P2PKH addresses start with the letter `b`, and P2SH addresses start with `r`.
All the chain parameters are defined [here](https://github.com/lbryio/lbrycrd/blob/master/src/chainparams.cpp).
## Metadata
Metadata is structured information about the stream or channel separate from the content itself (e.g. the title, language, media type, etc.). It is stored in the [value property](#claim-properties) of a claim.
Metadata is structured information about a stream or channel separate from the content itself (e.g. the title, language, media type, etc.). It is stored in the blockchain as the [value property](#claim-properties) of a claim.
Metadata is stored in a serialized binary format via [Protocol Buffers](https://developers.google.com/protocol-buffers/). This allows for metadata to be:
Metadata is stored in a serialized binary format using [Protocol Buffers](https://developers.google.com/protocol-buffers/). This allows for metadata to be:
- **Extensibile**. Metadata can encompass thousands of fields for dozens of types of content. It must be efficient to both modify the structure and maintain backward compatibility.
- **Compact**. Blockchain space is expensive. Data must be stored as compactly as possible.
- **Interoperabile**. Metadata will be used by many projects written in different languages.
The serialized metadata may be signed to indicate membership in a channel. See [Channels](#channels) for more info.
The serialized metadata may be cryptographically signed to indicate membership in a channel. See [Channels](#channels) for more info.
### Specification
As the metadata specification is designed to grow and change frequently, the full specification will not be examined here. The [types](https://github.com/lbryio/types) repository is considered the precise specification.
The metadata specification is designed to grow and change frequently. The full specification will not be detailed here. The [types](https://github.com/lbryio/types) repository is considered the precise specification.
Instead, let's look at an example and some key fields.
@ -747,17 +738,23 @@ Here’s some example metadata:
Some important metadata fields are highlighted below.
#### Source and Stream Hashes
#### Stream Hash
The `source` property contains information about how to fetch the data from the network. Within the `source` is a unique identifier to locate and find the content in the data network. More in [[Data]].
A unique identifier that is used to locate and fetch the content from the data network. More in [Data](#data).
#### Fees and Fee Structure
#### Fee
<!-- fix me extensively -->
Information on how to pay for the content. It includes the address that will receive the payment (the _fee address_), the the amount to be paid, and the currency. Only LBC and USD currencies are supported, though others may be added in the future.
- LBC
- Currencies?
- channel signatures and private keys
Example fee:
```
"fee": {
"address":"bNz8Va7xMyK9eHA5APzLph6cCTjBtGgmDN",
"amount":"99.95",
"currency":"LBC"
}
```
#### Title, Author, Description
@ -778,7 +775,7 @@ The media type of the item as [defined](https://www.iana.org/assignments/media-t
### Channels (Identities) {#channels}
Channels are the unit of identity in the LBRY system. A channel is a claim for a name beginning with `@` that contains a metadata structure for identity rather than content. Included in the metadata is the channel's public key. Here's an example:
Channels are the unit of identity. A channel is a claim for a name beginning with `@` that contains a metadata structure for identity rather than content. Included in the metadata is the channel's public key. Here's an example:
```
"claim_id": "6e56325c5351ceda2dd0795a30e864492910ccbf",
@ -822,9 +819,9 @@ format | description
##### Signing Process
1. Encode the metadata using protobuf.
1. Hash the encoded claim using SHA-256.
1. Sign the hash using the private key associated with the channel.
1. Append all the values (the version, the claim ID of the corresponding channel claim, the signature, and the protobuf-encoded metadata).
2. Hash the encoded claim using SHA-256.
3. Sign the hash using the private key associated with the channel.
4. Append all the values (the version, the claim ID of the corresponding channel claim, the signature, and the protobuf-encoded metadata).
##### Signature Validation
@ -842,20 +839,18 @@ Clients are responsible for validating metadata, including data structure and si
## Data
<!-- fixme this section -->
Files published using LBRY are stored in a distributed fashion by the clients participating in the network. Each file split into multiple small pieces, encrypted, and announced to the network. It may also be uploaded to other hosts on the network that specialize in rehosting content.
Data refers to the full binary data which is ultimate distributed by blah blah blah.
The purpose of blah blah blah is to blah blah.
The purpose of this process is to enable file storage without relying on centralized infrastructure, and to create a marketplace for data that allows hosts to be paid for their services. The design is strongly influenced by the Bittorrent protocol and network.
### Encoding
Content on the LBRY network is encoded to facilitate distribution.
Content on LBRY is encoded to facilitate distribution.
#### Blobs
The unit of data in the LBRY network is called a _blob_. A blob is an encrypted chunk of data up to 2MiB in size. Each blob is indexed by its _blob hash_, which is a SHA384 hash of the blob contents. Addressing blobs by their hash protects against naming collisions and ensures that the content you get is what you expect.
The smallest unit of data is called a _blob_. A blob is an encrypted chunk of data up to 2MiB in size. Each blob is indexed by its _blob hash_, which is a SHA384 hash of the blob contents. Addressing blobs by their hash protects against naming collisions and ensures that the content you get is what you expect.
Blobs are encrypted using AES-256 in CBC mode and PKCS7 padding. In order to keep each encrypted blob at 2MiB max, a blob can hold at most 2097151 bytes (2MiB minus 1 byte) of plaintext data. The source code for the exact algorithm is available [here](https://github.com/lbryio/lbry.go/blob/master/stream/blob.go). The encryption key and IV for each blob is stored as described below.
@ -940,7 +935,8 @@ A file must be encoded into a stream before it can be published. Encoding involv
1. Compute the stream hash.
An implementation of this process is available [here](https://github.com/lbryio/lbry.go/tree/master/stream).
fixme: this link is for v0, not v1. need to implement v1 or drop the link.
<!-- fixme: the above link is for v0, not v1. need to implement v1 or drop the link. -->
#### Stream Decoding
@ -957,7 +953,7 @@ Decoding a stream is like encoding in reverse, and with the added step of verify
### Announce
After a [[stream]] is encoded, it must be _announced_ to the network. Announcing is the process of letting other nodes on the network know that you have content available for download. The LBRY network tracks announced content using a distributed hash table.
After a [[stream]] is encoded, it must be _announced_ to the network. Announcing is the process of letting other nodes on the network know that you have content available for download. LBRY tracks announced content using a distributed hash table.
#### Distributed Hash Table
@ -988,9 +984,7 @@ Querying works almost the same way as [[announcing]]. A client looking for a tar
#### Blob Exchange Protocol
Downloading a blob from a peer is governed by the _Blob Exchange Protocol_. It is used by hosts and clients to check blob availability, exchange blobs, and negotiate data prices. The protocol is an RPC protocol using Protocol Buffers and the gRPC framework. It has five types of requests.
fixme: protocol does not **negotiate** anything right now. It just checks the price. Should we include negotiation in v1?
Downloading a blob from a peer is governed by the _Blob Exchange Protocol_. It is used by hosts and clients to exchange blobs and check data pricing and blob availability. The protocol is an RPC protocol using Protocol Buffers and the gRPC framework. It has five types of requests.
##### PriceCheck