LBRYCommunity/spec
1
0
Fork 0
Code Issues 5 Pull requests Projects Releases Packages Wiki Activity

work

Browse source
This commit is contained in:
Alex Grintsvayg 2018-10-24 16:41:52 -04:00
parent ad5d9bb22e
commit 4e317556ea
4 changed files with 260 additions and 79 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

BIN
bin/reflex Executable file
View file

Binary file not shown.

197
index.html
View file

@ -44,9 +44,11 @@
<li><a href="#claims">Claims</a> <li><a href="#claims">Claims</a>
<ul> <ul>
<li><a href="#claim-properties">Claim Properties</a></li>
<li><a href="#claim-example">Claim Example</a></li>
<li><a href="#claim-operations">Claim Operations</a></li> <li><a href="#claim-operations">Claim Operations</a></li>
<li><a href="#claimtrie">Claimtrie</a></li> <li><a href="#claimtrie">Claimtrie</a></li>
<li><a href="#claim-properties">Claim Properties</a> <li><a href="#claim-statuses">Claim Statuses</a>
<ul> <ul>
<li><a href="#accepted">Accepted</a></li> <li><a href="#accepted">Accepted</a></li>
@ -60,6 +62,7 @@
<ul> <ul>
<li><a href="#components">Components</a></li> <li><a href="#components">Components</a></li>
<li><a href="#grammar">Grammar</a></li>
<li><a href="#design-notes">Design Notes</a></li> <li><a href="#design-notes">Design Notes</a></li>
</ul></li> </ul></li>
<li><a href="#transactions">Transactions</a> <li><a href="#transactions">Transactions</a>
@ -87,7 +90,6 @@
<ul> <ul>
<li><a href="#streams-and-stream-hashes">Streams and Stream Hashes</a></li> <li><a href="#streams-and-stream-hashes">Streams and Stream Hashes</a></li>
<li><a href="#fees-and-fee-structure">Fees and Fee Structure</a></li> <li><a href="#fees-and-fee-structure">Fees and Fee Structure</a></li>
<li><a href="#more">More?</a></li>
</ul></li> </ul></li>
<li><a href="#identities">Identities</a></li> <li><a href="#identities">Identities</a></li>
<li><a href="#metadata-validation">Metadata Validation</a></li> <li><a href="#metadata-validation">Metadata Validation</a></li>
@ -99,8 +101,18 @@
<ul> <ul>
<li><a href="#blobs">Blobs</a></li> <li><a href="#blobs">Blobs</a></li>
<li><a href="#streams">Streams</a></li> <li><a href="#streams">Streams</a>
<li><a href="#how-to-turn-files-into-streams-and-vice-versa">How to Turn Files into Streams, and Vice Versa</a></li>
<ul>
<li><a href="#manifest-encoding">Manifest Encoding</a></li>
</ul></li>
<li><a href="#stream-creation">Stream Creation</a>
<ul>
<li><a href="#setup">Setup</a></li>
<li><a href="#content-blobs">Content Blobs</a></li>
<li><a href="#manifest-blob">Manifest Blob</a></li>
</ul></li>
</ul></li> </ul></li>
<li><a href="#download">Download</a> <li><a href="#download">Download</a>
@ -164,23 +176,45 @@
<h2 id="blockchain">Blockchain</h2> <h2 id="blockchain">Blockchain</h2>
<!-- done -->
<p>The LBRY blockchain is a public, proof-of-work blockchain. It serves three key purposes:</p> <p>The LBRY blockchain is a public, proof-of-work blockchain. It serves three key purposes:</p>
<ol> <ol>
<li>An index of the content available on the network</li> <li>An index of the content available on the network</li>
<li>A payment system and record of purchases for priced content</li> <li>A payment system and record of purchases for priced content</li>
<li>Trustful publisher identities (fixme: should this even be listed here?)</li> <li>Trustful publisher identities</li>
</ol> </ol>
<p>The LBRY blockchain is a fork of the <a href="https://bitcoin.org/bitcoin.pdf">Bitcoin</a> blockchain, with substantial modifications. This document will not cover or specify any aspects of LBRY that are identical to Bitcoin, and will instead focus on the differences.</p> <p>The LBRY blockchain is a fork of the <a href="https://bitcoin.org/bitcoin.pdf">Bitcoin</a> blockchain, with substantial modifications. This document will not cover or specify any aspects of LBRY that are identical to Bitcoin, and will instead focus on the differences.</p>
<h3 id="claims">Claims</h3> <h3 id="claims">Claims</h3>
<p>A single metadata entry in the blockchain is called a <code>claim</code>. It records an item that was published to the network or a publisher&rsquo;s identity.</p> <!-- done -->
<p>Every claim has a globally-unique <code>claimID</code>, an <code>amount</code> (how many credits were set aside to back the claim), and a <code>value</code>. The value may contain metadata about a piece of content, a publisher&rsquo;s public key, or other information. See the <a href="#metadata">Metadata</a> section for more information about what may be stored in the value.</p> <p>A single metadata entry in the blockchain is called a <em>claim</em>. It records a file that was published to the network or a publisher&rsquo;s identity.</p>
<p>Every claim is associated with a <code>name</code>, which is a bytestring of 0-255 bytes. Every name must be a valid UTF8 string.</p> <h4 id="claim-properties">Claim Properties</h4>
<!-- done -->
<p>Every claim contains 4 properties:</p>
<dl>
<dt>claimId</dt>
<dd>A 20-byte hash unique among all claims. See [Claim Identifier Generation](#claim-identifier-generation).</dd>
<dt>name</dt>
<dd>A normalized UTF-8 string of up to 255 bytes used to address the claim. See [URLs](#urls) and [Normalization](#normalization).</dd>
<dt>amount</dt>
<dd>A quantity of tokens used to stake the claim. See [Controlling](#controlling).</dd>
<dt>value</dt>
<dd>Metadata about a piece of content, a publisher's public key, or other information. See [Metadata](#metadata).</dd>
</dl>
<h4 id="claim-example">Claim Example</h4>
<!-- done -->
<p>Here is an example claim:</p> <p>Here is an example claim:</p>
@ -202,19 +236,28 @@
<h4 id="claim-operations">Claim Operations</h4> <h4 id="claim-operations">Claim Operations</h4>
<p>There are four claim operations: <code>create</code>, <code>support</code>, <code>update</code>, and <code>abandon</code>.</p> <!-- done -->
<p>A <code>create</code> operation makes a new claim for a name, or submits a competing claim on an existing name.</p> <p>There are four claim operations: <em>create</em>, <em>support</em>, <em>update</em>, and <em>abandon</em>.</p>
<p>A <code>support</code> is a claim that adds to the credit total of an existing claim. A support does not have it’s own claim ID or data. Instead, it has the claim ID of the claim to which its amount will be added.</p> <dl>
<dt>create</dt>
<p>An <code>update</code> changes the data or the amount stored in an existing claim or support. Updates do not change the claim ID, so an updated claim retains any supports attached to it.</p> <dd>Makes a new claim.</dd>
<dt>support</dt>
<p>An <code>abandon</code> withdraws a claim or support, freeing the associated credits to be used for other purposes.</p> <dd>Adds its [[amount]] to the stake of an already existing claim. It contains no metadata.</dd>
<dt>update</dt>
<dd>Changes the data or the amount stored in an existing claim or support. Updates do not change the claim ID, so an updated claim retains any supports attached to it. </dd>
<dt>abandon</dt>
<dd>Withdraws a claim or support, freeing the associated credits to be used for other purposes.</dd>
</dl>
<h4 id="claimtrie">Claimtrie</h4> <h4 id="claimtrie">Claimtrie</h4>
<p>The <code>claimtrie</code> is the data structure that LBRY uses to store claims and prove the correctness of name resolution. It is a <a href="https://en.wikipedia.org/wiki/Merkle_tree">Merkle tree</a> that maps names to claims. Claims are stored as leaf nodes in the tree. Names are stored as the path from the root node to the leaf node.</p> <!-- done -->
<p>The <em>claimtrie</em> is the data structure used to store the set of all claims and prove the correctness of claim resolution.</p>
<p>The claimtrie is implemented as a <a href="https://en.wikipedia.org/wiki/Merkle_tree">Merkle tree</a> that maps names to claims. Claims are stored as leaf nodes in the tree. Names are stored as the path from the root node to the leaf node.</p>
<p>The hash of the root node (the <code>root hash</code>) is stored in the header of each block in the blockchain. Nodes in the LBRY network use the root hash to efficiently and securely validate the state of the claimtrie.</p> <p>The hash of the root node (the <code>root hash</code>) is stored in the header of each block in the blockchain. Nodes in the LBRY network use the root hash to efficiently and securely validate the state of the claimtrie.</p>
@ -222,17 +265,23 @@
<p>For more details on the specific claimtrie implementation, see <a href="https://github.com/lbryio/lbrycrd/blob/master/src/claimtrie.cpp">the source code</a>.</p> <p>For more details on the specific claimtrie implementation, see <a href="https://github.com/lbryio/lbrycrd/blob/master/src/claimtrie.cpp">the source code</a>.</p>
<h4 id="claim-properties">Claim Properties</h4> <h4 id="claim-statuses">Claim Statuses</h4>
<p>A claim can have one or more the following properties at a given block:</p> <!-- done -->
<p>A claim can have one or more the following properties at a given block.</p>
<h5 id="accepted">Accepted</h5> <h5 id="accepted">Accepted</h5>
<p>An accepted claim or support is simply one that has been entered into the blockchain. This happens when the transaction containing the claim is included in a block.</p> <!-- done -->
<p>An accepted claim or support is one that has been entered into the blockchain. This happens when the transaction containing the claim is included in a block.</p>
<h5 id="abandoned">Abandoned</h5> <h5 id="abandoned">Abandoned</h5>
<p>An abandoned claim or support is one that was withdrawn by its creator. It is no longer in contention to control a name. Spending the transaction that contains the claim will also cause the claim to become abandoned.</p> <!-- done -->
<p>An abandoned claim or support is one that was withdrawn by its creator. It is no longer in contention to control a name. Spending a transaction that contains a claim will cause that claim to become abandoned.</p>
<p>While data related to abandoned claims technically still resides in the blockchain, it is considered inappropriate to use this data to fetch the associated content.</p> <p>While data related to abandoned claims technically still resides in the blockchain, it is considered inappropriate to use this data to fetch the associated content.</p>
@ -299,9 +348,7 @@
<h4 id="normalization">Normalization</h4> <h4 id="normalization">Normalization</h4>
<p>TODO: Talk about how claim names are normalized.</p> <p>Names in the claimtrie are normalized to avoid confusion due to Unicode equivalence or casing. All names are normalized using the NFD normalization form, then lowercased using the en_US locale.</p>
<p><a href="https://github.com/lbryio/lbrycrd/issues/208">https://github.com/lbryio/lbrycrd/issues/208</a></p>
<h3 id="urls">URLs</h3> <h3 id="urls">URLs</h3>
@ -348,20 +395,22 @@ lbry:meet-LBRY$3
<pre><code>lbry:@lbry/meet-LBRY <pre><code>lbry:@lbry/meet-LBRY
</code></pre> </code></pre>
<p>The full URL grammar is defined below using <a href="https://www.w3.org/TR/2017/REC-xquery-31-20170321/#EBNFNotation">Xquery EBNF notation</a>:</p> <h4 id="grammar">Grammar</h4>
<p>The full URL grammar is defined using <a href="https://www.w3.org/TR/2017/REC-xquery-31-20170321/#EBNFNotation">Xquery EBNF notation</a>:</p>
<!-- see http://bottlecaps.de/rr/ui for visuals--> <!-- see http://bottlecaps.de/rr/ui for visuals-->
<pre><code>URL ::= Scheme Path Query? <pre><code>URL ::= Scheme Path Query?
Scheme ::= 'lbry:' Scheme ::= 'lbry://'
Path ::= ClaimNameAndModifier | ChannelAndModifier ( '/' ClaimNameAndModifier )? Path ::= ClaimNameAndModifier | ChannelAndModifier ( '/' ClaimNameAndModifier )?
ClaimNameAndModifier ::= ClaimName Modifier? ClaimNameAndModifier ::= ClaimName Modifier?
ChannelAndModifier ::= Channel Modifier? ChannelAndModifier ::= Channel Modifier?
ClaimName ::= AllowedChar+ ClaimName ::= NameChar+
Channel ::= '@' ClaimName Channel ::= '@' ClaimName
Modifier ::= ClaimID | ClaimSequence | BidPosition Modifier ::= ClaimID | ClaimSequence | BidPosition
@ -372,8 +421,8 @@ BidPosition ::= '$' PositiveNumber
Query ::= '?' QueryParameterList Query ::= '?' QueryParameterList
QueryParameterList ::= QueryParameter ( '&amp;' QueryParameterList )* QueryParameterList ::= QueryParameter ( '&amp;' QueryParameterList )*
QueryParameter ::= QueryParameterName ( '=' QueryParameterValue )? QueryParameter ::= QueryParameterName ( '=' QueryParameterValue )?
QueryParameterName ::= AllowedChar+ QueryParameterName ::= NameChar+
QueryParameterValue ::= AllowedChar+ QueryParameterValue ::= NameChar+
PositiveDigit ::= [123456789] PositiveDigit ::= [123456789]
Digit ::= '0' | PositiveDigit Digit ::= '0' | PositiveDigit
@ -382,7 +431,8 @@ PositiveNumber ::= PositiveDigit Digit*
HexAlpha ::= [abcdef] HexAlpha ::= [abcdef]
Hex ::= (Digit | HexAlpha)+ Hex ::= (Digit | HexAlpha)+
AllowedChar ::= [^=&amp;#:$@?/] /* any UTF8 character that is not reserved */ NameChar ::= Char - [=&amp;#:$@?/] /* any character that is not reserved */
Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */
</code></pre> </code></pre>
<h4 id="design-notes">Design Notes</h4> <h4 id="design-notes">Design Notes</h4>
@ -512,17 +562,16 @@ OP_SUPPORT_CLAIM &lt;name&gt; &lt;claimId&gt; OP_2DROP OP_DROP &lt;pubKey&gt;
<h4 id="streams-and-stream-hashes">Streams and Stream Hashes</h4> <h4 id="streams-and-stream-hashes">Streams and Stream Hashes</h4>
<p>(The metadata property <code>sd_hash</code> contains a unique identifier to locate and find the content in the data network. Reference [[Data]].)</p> <p>(The metadata property <code>lbry_sd_hash</code> contains a unique identifier to locate and find the content in the data network. Reference [[Data]].)</p>
<h4 id="fees-and-fee-structure">Fees and Fee Structure</h4> <h4 id="fees-and-fee-structure">Fees and Fee Structure</h4>
<ul> <ul>
<li>LBC</li> <li>LBC</li>
<li>Currencies?</li> <li>Currencies?</li>
<li>channel signatures and private keys</li>
</ul> </ul>
<h4 id="more">More?</h4>
<h3 id="identities">Identities</h3> <h3 id="identities">Identities</h3>
<p>Channels are the unit of identity in the LBRY system. A channel is a claim that start with <code>@</code> and contains a metadata structure for identities rather than content. The most important part of channel&rsquo;s metadata is the public key. Claims belonging to a channel are signed with the corresponding private key. A valid signature proves channel membership.</p> <p>Channels are the unit of identity in the LBRY system. A channel is a claim that start with <code>@</code> and contains a metadata structure for identities rather than content. The most important part of channel&rsquo;s metadata is the public key. Claims belonging to a channel are signed with the corresponding private key. A valid signature proves channel membership.</p>
@ -555,6 +604,8 @@ OP_SUPPORT_CLAIM &lt;name&gt; &lt;claimId&gt; OP_2DROP OP_DROP &lt;pubKey&gt;
<h3 id="metadata-validation">Metadata Validation</h3> <h3 id="metadata-validation">Metadata Validation</h3>
<p>Clients are responsible for validating metadata, including data structure and signatures.</p>
<p>(expand)</p> <p>(expand)</p>
<ul> <ul>
@ -570,13 +621,89 @@ OP_SUPPORT_CLAIM &lt;name&gt; &lt;claimId&gt; OP_2DROP OP_DROP &lt;pubKey&gt;
<h4 id="blobs">Blobs</h4> <h4 id="blobs">Blobs</h4>
<p>The unit of content in our network is called a <code>blob</code>. A blob is an encrypted chunk of data up to 2MB in size. Each blob is indexed by its <code>blob hash</code>, which is a SHA384 hash of the blob contents. Addressing blobs by their hashes simultaneously protects against naming collisions and ensures that the content you get is what you expect.</p> <p>The unit of data in our network is called a <em>blob</em>. A blob is an encrypted chunk of data up to 2MB in size. Each blob is indexed by its <em>blob hash</em>, which is a SHA384 hash of the blob contents. Addressing blobs by their hashes simultaneously protects against naming collisions and ensures that the content you get is what you expect.</p>
<p>Blobs are encrypted using AES-256 in CBC mode and PKCS7 padding. In order to keep each encrypted blob at 2MB max, a blob can hold at most 2097151 bytes (2MB minus 1 byte) of plaintext data. The source code for exact algorithm is available <a href="https://github.com/lbryio/lbry.go/blob/master/stream/blob.go">here</a>. The encryption key and IV for each blob is stored as described below.</p>
<h4 id="streams">Streams</h4> <h4 id="streams">Streams</h4>
<p>Multiple blobs may be combined into a <code>stream</code>. A stream may be a book, a movie, a CAD file, etc. All content on the network is shared as streams. Every stream begins with the <code>stream descriptor</code> blob, which contains a JSON list of the hashes and keys of the <code>content blobs</code>. The content blobs hold the actual content of the stream. Every stream ends with an empty content blob, to signify that the stream has finished (this is similar to a null-terminated string, and is necessary to support streaming content).</p> <p>Multiple blobs are combined into a <em>stream</em>. A stream may be a book, a movie, a CAD file, etc. All content on the network is shared as streams. Every stream begins with the <em>manifest blob</em>, followed by one or more <em>content blobs</em>. The content blobs hold the actual content of the stream. The manifest blob contains information necessary to find the content blobs and convert them into a file. This includes the hashes of the content blobs, their order in the stream, and cryptographic material for decrypting them.</p>
<h4 id="how-to-turn-files-into-streams-and-vice-versa">How to Turn Files into Streams, and Vice Versa</h4> <p>The blob hash of the manifest blob is called the <em>stream hash</em>. It uniquely identifies each stream.</p>
<h5 id="manifest-encoding">Manifest Encoding</h5>
<p>A manifest blob&rsquo;s contents are encoded using a canonical JSON encoding. The JSON encoding must be canonical to support consistent hashing and validation. The encoding is the same as standard JSON, but adds the following rules:</p>
<ul>
<li>Object keys must be quoted and lexicographically sorted.</li>
<li>All strings are hex-encoded. Hex letters must be lowercase.</li>
<li>Whitespace before, after, or between tokens is not permitted.</li>
<li>Floating point numbers, leading zeros, and &ldquo;minus 0&rdquo; for integers are not permitted.</li>
<li>Trailing commas after the last item in an array or object are not permitted.</li>
</ul>
<p>Here&rsquo;s an example manifest, with whitespace added for readability:</p>
<!-- originally from 053b2f0f0e82e7f022837382733d5f5817dcd67027103fe43f00fa7a6f9fa8742c1022a851616c1ac15d1c60e89db3f4 -->
<pre><code>{
&quot;blobs&quot;:[
{
&quot;blob_hash&quot;:&quot;a6daea71be2bb89fab29a2a10face08143411a5245edcaa5efff48c2e459e7ec01ad20edfde6da43a932aca45b2cec61&quot;,
&quot;iv&quot;:&quot;ef6caef207a207ca5b14c0282d25ce21&quot;,
&quot;length&quot;:2097152
},
{
&quot;blob_hash&quot;:&quot;bf2717e2c445052366d35bcd58edb108cbe947af122d8f76b4856db577aeeaa2def5b57dbb80f7b1531296bd3e0256fc&quot;,
&quot;iv&quot;:&quot;a37b291a37337fc1ff90ae655c244c1d&quot;,
&quot;length&quot;:2097152
},
...,
{
&quot;blob_hash&quot;:&quot;322973617221ddfec6e53bff4b74b9c21c968cd32ba5a5094d84210e660c4b2ed0882b114a2392a08b06183f19330aaf&quot;,
&quot;iv&quot;: &quot;a00f5f458695bdc9d50d3dbbc7905abc&quot;,
&quot;length&quot;: 600160
}
],
&quot;filename&quot;:&quot;6b706a7977755477704d632e6d7034&quot;,
&quot;key&quot;:&quot;94d89c0493c576057ac5f32eb0871180&quot;
}
</code></pre>
<p>The <code>key</code> field contains the key to decrypt the stream, and is optional. The key may be stored by a third party and made available to a client when presented with proof that the content was purchased. The <code>length</code> field for each blob is the length of the encrypted blob, not the original file chunk.</p>
<p>Every stream must have at least two blobs - the manifest blob and a content blob. Consequently, zero-length streams are not allowed.</p>
<h4 id="stream-creation">Stream Creation</h4>
<p>A file must be converted into a stream before it can be published. Conversion involves breaking the file into chunks, encrypting the chunks into content blobs, and creating the manifest blob. Here are the steps:</p>
<h5 id="setup">Setup</h5>
<ol>
<li>Generate a random 32-byte key for the stream.</li>
</ol>
<h5 id="content-blobs">Content Blobs</h5>
<ol>
<li>Break the file into chunks of at most 2097151 bytes.</li>
<li>Generate a random IV for each chuck.</li>
<li>Pad each chunk using PKCS7 padding</li>
<li>Encrypt each chunk with AES-CBC using the stream key and the IV for that chunk.</li>
<li>An encrypted chunk is a blob.</li>
</ol>
<h5 id="manifest-blob">Manifest Blob</h5>
<ol>
<li>Fill in the manifest data.</li>
<li>Encode the data using the canonical JSON encoding described above.</li>
<li>Compute the stream hash</li>
</ol>
<p>An implementation of this process is available <a href="https://github.com/lbryio/lbry.go/tree/master/stream">here</a>.</p>
<h3 id="download">Download</h3> <h3 id="download">Download</h3>

139
index.md
View file

@ -47,9 +47,11 @@ TODO:
* [Conventions and Terminology](#conventions-and-terminology) * [Conventions and Terminology](#conventions-and-terminology)
* [Blockchain](#blockchain) * [Blockchain](#blockchain)
* [Claims](#claims) * [Claims](#claims)
* [Claim Properties](#claim-properties)
* [Claim Example](#claim-example)
* [Claim Operations](#claim-operations) * [Claim Operations](#claim-operations)
* [Claimtrie](#claimtrie) * [Claimtrie](#claimtrie)
* [Claim Properties](#claim-properties) * [Claim Statuses](#claim-statuses)
* [Accepted](#accepted) * [Accepted](#accepted)
* [Abandoned](#abandoned) * [Abandoned](#abandoned)
* [Active](#active) * [Active](#active)
@ -57,6 +59,7 @@ TODO:
* [Normalization](#normalization) * [Normalization](#normalization)
* [URLs](#urls) * [URLs](#urls)
* [Components](#components) * [Components](#components)
* [Grammar](#grammar)
* [Design Notes](#design-notes) * [Design Notes](#design-notes)
* [Transactions](#transactions) * [Transactions](#transactions)
* [Operations and Opcodes](#operations-and-opcodes) * [Operations and Opcodes](#operations-and-opcodes)
@ -72,14 +75,17 @@ TODO:
* [Key Metadata Fields](#key-metadata-fields) * [Key Metadata Fields](#key-metadata-fields)
* [Streams and Stream Hashes](#streams-and-stream-hashes) * [Streams and Stream Hashes](#streams-and-stream-hashes)
* [Fees and Fee Structure](#fees-and-fee-structure) * [Fees and Fee Structure](#fees-and-fee-structure)
* [More?](#more)
* [Identities](#identities) * [Identities](#identities)
* [Metadata Validation](#metadata-validation) * [Metadata Validation](#metadata-validation)
* [Data](#data) * [Data](#data)
* [Encoding and Decoding](#encoding-and-decoding) * [Encoding and Decoding](#encoding-and-decoding)
* [Blobs](#blobs) * [Blobs](#blobs)
* [Streams](#streams) * [Streams](#streams)
* [How to Turn Files into Streams, and Vice Versa](#how-to-turn-files-into-streams-and-vice-versa) * [Manifest Encoding](#manifest-encoding)
* [Stream Creation](#stream-creation)
* [Setup](#setup)
* [Content Blobs](#content-blobs)
* [Manifest Blob](#manifest-blob)
* [Download](#download) * [Download](#download)
* [Distributed Hash Table](#distributed-hash-table) * [Distributed Hash Table](#distributed-hash-table)
* [Blob Exchange Protocol](#blob-exchange-protocol) * [Blob Exchange Protocol](#blob-exchange-protocol)
@ -149,11 +155,15 @@ The LBRY blockchain is a public, proof-of-work blockchain. It serves three key p
The LBRY blockchain is a fork of the [Bitcoin](https://bitcoin.org/bitcoin.pdf) blockchain, with substantial modifications. This document will not cover or specify any aspects of LBRY that are identical to Bitcoin, and will instead focus on the differences. The LBRY blockchain is a fork of the [Bitcoin](https://bitcoin.org/bitcoin.pdf) blockchain, with substantial modifications. This document will not cover or specify any aspects of LBRY that are identical to Bitcoin, and will instead focus on the differences.
### Claims <!-- done --> ### Claims
<!-- done -->
A single metadata entry in the blockchain is called a _claim_. It records a file that was published to the network or a publisher's identity. A single metadata entry in the blockchain is called a _claim_. It records a file that was published to the network or a publisher's identity.
#### Claim Properties <!-- done --> #### Claim Properties
<!-- done -->
Every claim contains 4 properties: Every claim contains 4 properties:
@ -168,7 +178,9 @@ Every claim contains 4 properties:
<dd>Metadata about a piece of content, a publisher's public key, or other information. See [Metadata](#metadata).</dd> <dd>Metadata about a piece of content, a publisher's public key, or other information. See [Metadata](#metadata).</dd>
</dl> </dl>
#### Claim Example <!-- done --> #### Claim Example
<!-- done -->
Here is an example claim: Here is an example claim:
@ -189,7 +201,9 @@ Here is an example claim:
} }
``` ```
#### Claim Operations <!-- done --> #### Claim Operations
<!-- done -->
There are four claim operations: _create_, _support_, _update_, and _abandon_. There are four claim operations: _create_, _support_, _update_, and _abandon_.
@ -204,7 +218,9 @@ There are four claim operations: _create_, _support_, _update_, and _abandon_.
<dd>Withdraws a claim or support, freeing the associated credits to be used for other purposes.</dd> <dd>Withdraws a claim or support, freeing the associated credits to be used for other purposes.</dd>
</dl> </dl>
#### Claimtrie <!-- done --> #### Claimtrie
<!-- done -->
The _claimtrie_ is the data structure used to store the set of all claims and prove the correctness of claim resolution. The _claimtrie_ is the data structure used to store the set of all claims and prove the correctness of claim resolution.
@ -216,15 +232,21 @@ Multiple claims can exist for the same name. They are all stored in the leaf nod
For more details on the specific claimtrie implementation, see [the source code](https://github.com/lbryio/lbrycrd/blob/master/src/claimtrie.cpp). For more details on the specific claimtrie implementation, see [the source code](https://github.com/lbryio/lbrycrd/blob/master/src/claimtrie.cpp).
#### Claim Statuses <!-- done --> #### Claim Statuses
<!-- done -->
A claim can have one or more the following properties at a given block. A claim can have one or more the following properties at a given block.
##### Accepted <!-- done --> ##### Accepted
<!-- done -->
An accepted claim or support is one that has been entered into the blockchain. This happens when the transaction containing the claim is included in a block. An accepted claim or support is one that has been entered into the blockchain. This happens when the transaction containing the claim is included in a block.
##### Abandoned <!-- done --> ##### Abandoned
<!-- done -->
An abandoned claim or support is one that was withdrawn by its creator. It is no longer in contention to control a name. Spending a transaction that contains a claim will cause that claim to become abandoned. An abandoned claim or support is one that was withdrawn by its creator. It is no longer in contention to control a name. Spending a transaction that contains a claim will cause that claim to become abandoned.
@ -592,53 +614,82 @@ Clients are responsible for validating metadata, including data structure and si
#### Blobs #### Blobs
The unit of content in our network is called a `blob`. A blob is an encrypted chunk of data up to 2MB in size. Each blob is indexed by its `blob hash`, which is a SHA384 hash of the blob contents. Addressing blobs by their hashes simultaneously protects against naming collisions and ensures that the content you get is what you expect. The unit of data in our network is called a _blob_. A blob is an encrypted chunk of data up to 2MB in size. Each blob is indexed by its _blob hash_, which is a SHA384 hash of the blob contents. Addressing blobs by their hashes simultaneously protects against naming collisions and ensures that the content you get is what you expect.
Blobs are encrypted using AES-256 in CBC mode and PKCS7 padding. In order to keep each encrypted blob at 2MB max, a blob can hold at most 2097151 bytes (2MB minus 1 byte) of plaintext data. The source code for exact algorithm is available [here](https://github.com/lbryio/lbry.go/blob/master/stream/blob.go). The encryption key and IV for each blob is stored as described below.
#### Streams #### Streams
Multiple blobs are combined into a `stream`. A stream may be a book, a movie, a CAD file, etc. All content on the network is shared as streams. Every stream begins with the `stream descriptor blob` (or SD blob), followed by one or more `content blobs`. The content blobs hold the actual content of the stream. The SD blob contains information necessary to find the content blobs and assemble them into a file. This includes the hashes of the content blobs, their order in the stream, and cryptographic material for decrypting them. Multiple blobs are combined into a _stream_. A stream may be a book, a movie, a CAD file, etc. All content on the network is shared as streams. Every stream begins with the _manifest blob_, followed by one or more _content blobs_. The content blobs hold the actual content of the stream. The manifest blob contains information necessary to find the content blobs and convert them into a file. This includes the hashes of the content blobs, their order in the stream, and cryptographic material for decrypting them.
Here's an example SD blob. It's hash is `053b2f0f0e82e7f022837382733d5f5817dcd67027103fe43f00fa7a6f9fa8742c1022a851616c1ac15d1c60e89db3f4`. The blob hash of the manifest blob is called the _stream hash_. It uniquely identifies each stream.
##### Manifest Encoding
A manifest blob's contents are encoded using a canonical JSON encoding. The JSON encoding must be canonical to support consistent hashing and validation. The encoding is the same as standard JSON, but adds the following rules:
- Object keys must be quoted and lexicographically sorted.
- All strings are hex-encoded. Hex letters must be lowercase.
- Whitespace before, after, or between tokens is not permitted.
- Floating point numbers, leading zeros, and "minus 0" for integers are not permitted.
- Trailing commas after the last item in an array or object are not permitted.
Here's an example manifest, with whitespace added for readability:
<!-- originally from 053b2f0f0e82e7f022837382733d5f5817dcd67027103fe43f00fa7a6f9fa8742c1022a851616c1ac15d1c60e89db3f4 -->
``` ```
{ {
"stream_type":"lbryfile", "blobs":[
"key":"94d89c0493c576057ac5f32eb0871180", {
"suggested_file_name":"6b706a7977755477704d632e6d7034", "blob_hash":"a6daea71be2bb89fab29a2a10face08143411a5245edcaa5efff48c2e459e7ec01ad20edfde6da43a932aca45b2cec61",
"stream_hash":"8cef6280f36f7e6590a6218da6b2eb8184ab1435c3f8d77f008088f5d2bc6bd2252a2beb9cfa3d9d40b9ce36d2d7b2ce" "iv":"ef6caef207a207ca5b14c0282d25ce21",
"stream_name":"6b706a7977755477704d632e6d7034", "length":2097152
"blobs":[ },
{ {
"length":2097152, "blob_hash":"bf2717e2c445052366d35bcd58edb108cbe947af122d8f76b4856db577aeeaa2def5b57dbb80f7b1531296bd3e0256fc",
"blob_num":0, "iv":"a37b291a37337fc1ff90ae655c244c1d",
"blob_hash":"a6daea71be2bb89fab29a2a10face08143411a5245edcaa5efff48c2e459e7ec01ad20edfde6da43a932aca45b2cec61", "length":2097152
"iv":"ef6caef207a207ca5b14c0282d25ce21" },
}, ...,
{ {
"length":2097152, "blob_hash":"322973617221ddfec6e53bff4b74b9c21c968cd32ba5a5094d84210e660c4b2ed0882b114a2392a08b06183f19330aaf",
"blob_num":1, "iv": "a00f5f458695bdc9d50d3dbbc7905abc",
"blob_hash":"bf2717e2c445052366d35bcd58edb108cbe947af122d8f76b4856db577aeeaa2def5b57dbb80f7b1531296bd3e0256fc", "length": 600160
"iv":"a37b291a37337fc1ff90ae655c244c1d" }
}, ],
..., "filename":"6b706a7977755477704d632e6d7034",
{ "key":"94d89c0493c576057ac5f32eb0871180"
"length":0,
"blob_num":45,
"iv":"53677e463ddb3bf060a40b99f8236432"
}
]
} }
``` ```
Every field except 'stream_type' is either an integer or a hex-encoded string. The `key` field contains the key to decrypt the stream, and is optional. The key may be stored externally on a keyserver. The keyserver would make the key available to a client when presented with proof that the content was purchased. The `key` field contains the key to decrypt the stream, and is optional. The key may be stored by a third party and made available to a client when presented with proof that the content was purchased. The `length` field for each blob is the length of the encrypted blob, not the original file chunk.
The last blob in the `blobs` list of the SD hash is always an empty blob with no hash. This signifies the end of the stream. This is similar to a null-terminated string, and is necessary to support content where the length is not known in advance (e.g. live video). Every stream must have at least two blobs - the manifest blob and a content blob. Consequently, zero-length streams are not allowed.
Every stream must have at least two blobs - an SD blob and a content blob. Zero-length streams are not allowed. #### Stream Creation
#### How to Turn Files into Streams, and Vice Versa A file must be converted into a stream before it can be published. Conversion involves breaking the file into chunks, encrypting the chunks into content blobs, and creating the manifest blob. Here are the steps:
https://github.com/lbryio/lbry.go/tree/master/stream ##### Setup
1. Generate a random 32-byte key for the stream.
##### Content Blobs
1. Break the file into chunks of at most 2097151 bytes.
1. Generate a random IV for each chuck.
1. Pad each chunk using PKCS7 padding
1. Encrypt each chunk with AES-CBC using the stream key and the IV for that chunk.
1. An encrypted chunk is a blob.
##### Manifest Blob
1. Fill in the manifest data.
1. Encode the data using the canonical JSON encoding described above.
1. Compute the stream hash
An implementation of this process is available [here](https://github.com/lbryio/lbry.go/tree/master/stream).
### Download ### Download

3
watch.sh Normal file
View file

@ -0,0 +1,3 @@
#!/bin/bash
./bin/reflex --decoration=none --start-service=true --inverse-regex='index\.html' -- sh -c "make"