work
This commit is contained in:
Alex Grintsvayg
parent
ad5d9bb22e
commit
4e317556ea
4 changed files with 260 additions and 79 deletions
BIN
bin/reflex
Executable file
BIN
bin/reflex
Executable file
Binary file not shown.
197
index.html
197
index.html
|
|
@ -44,9 +44,11 @@
|
|||
<li><a href="#claims">Claims</a>
|
||||
|
||||
<ul>
|
||||
<li><a href="#claim-properties">Claim Properties</a></li>
|
||||
<li><a href="#claim-example">Claim Example</a></li>
|
||||
<li><a href="#claim-operations">Claim Operations</a></li>
|
||||
<li><a href="#claimtrie">Claimtrie</a></li>
|
||||
<li><a href="#claim-properties">Claim Properties</a>
|
||||
<li><a href="#claim-statuses">Claim Statuses</a>
|
||||
|
||||
<ul>
|
||||
<li><a href="#accepted">Accepted</a></li>
|
||||
|
|
@ -60,6 +62,7 @@
|
|||
|
||||
<ul>
|
||||
<li><a href="#components">Components</a></li>
|
||||
<li><a href="#grammar">Grammar</a></li>
|
||||
<li><a href="#design-notes">Design Notes</a></li>
|
||||
</ul></li>
|
||||
<li><a href="#transactions">Transactions</a>
|
||||
|
|
@ -87,7 +90,6 @@
|
|||
<ul>
|
||||
<li><a href="#streams-and-stream-hashes">Streams and Stream Hashes</a></li>
|
||||
<li><a href="#fees-and-fee-structure">Fees and Fee Structure</a></li>
|
||||
<li><a href="#more">More?</a></li>
|
||||
</ul></li>
|
||||
<li><a href="#identities">Identities</a></li>
|
||||
<li><a href="#metadata-validation">Metadata Validation</a></li>
|
||||
|
|
@ -99,8 +101,18 @@
|
|||
|
||||
<ul>
|
||||
<li><a href="#blobs">Blobs</a></li>
|
||||
<li><a href="#streams">Streams</a></li>
|
||||
<li><a href="#how-to-turn-files-into-streams-and-vice-versa">How to Turn Files into Streams, and Vice Versa</a></li>
|
||||
<li><a href="#streams">Streams</a>
|
||||
|
||||
<ul>
|
||||
<li><a href="#manifest-encoding">Manifest Encoding</a></li>
|
||||
</ul></li>
|
||||
<li><a href="#stream-creation">Stream Creation</a>
|
||||
|
||||
<ul>
|
||||
<li><a href="#setup">Setup</a></li>
|
||||
<li><a href="#content-blobs">Content Blobs</a></li>
|
||||
<li><a href="#manifest-blob">Manifest Blob</a></li>
|
||||
</ul></li>
|
||||
</ul></li>
|
||||
<li><a href="#download">Download</a>
|
||||
|
||||
|
|
@ -164,23 +176,45 @@
|
|||
|
||||
<h2 id="blockchain">Blockchain</h2>
|
||||
|
||||
<!-- done -->
|
||||
|
||||
<p>The LBRY blockchain is a public, proof-of-work blockchain. It serves three key purposes:</p>
|
||||
|
||||
<ol>
|
||||
<li>An index of the content available on the network</li>
|
||||
<li>A payment system and record of purchases for priced content</li>
|
||||
<li>Trustful publisher identities (fixme: should this even be listed here?)</li>
|
||||
<li>Trustful publisher identities</li>
|
||||
</ol>
|
||||
|
||||
<p>The LBRY blockchain is a fork of the <a href="https://bitcoin.org/bitcoin.pdf">Bitcoin</a> blockchain, with substantial modifications. This document will not cover or specify any aspects of LBRY that are identical to Bitcoin, and will instead focus on the differences.</p>
|
||||
|
||||
<h3 id="claims">Claims</h3>
|
||||
|
||||
<p>A single metadata entry in the blockchain is called a <code>claim</code>. It records an item that was published to the network or a publisher’s identity.</p>
|
||||
<!-- done -->
|
||||
|
||||
<p>Every claim has a globally-unique <code>claimID</code>, an <code>amount</code> (how many credits were set aside to back the claim), and a <code>value</code>. The value may contain metadata about a piece of content, a publisher’s public key, or other information. See the <a href="#metadata">Metadata</a> section for more information about what may be stored in the value.</p>
|
||||
<p>A single metadata entry in the blockchain is called a <em>claim</em>. It records a file that was published to the network or a publisher’s identity.</p>
|
||||
|
||||
<p>Every claim is associated with a <code>name</code>, which is a bytestring of 0-255 bytes. Every name must be a valid UTF8 string.</p>
|
||||
<h4 id="claim-properties">Claim Properties</h4>
|
||||
|
||||
<!-- done -->
|
||||
|
||||
<p>Every claim contains 4 properties:</p>
|
||||
|
||||
<dl>
|
||||
<dt>claimId</dt>
|
||||
<dd>A 20-byte hash unique among all claims. See [Claim Identifier Generation](#claim-identifier-generation).</dd>
|
||||
<dt>name</dt>
|
||||
<dd>A normalized UTF-8 string of up to 255 bytes used to address the claim. See [URLs](#urls) and [Normalization](#normalization).</dd>
|
||||
<dt>amount</dt>
|
||||
<dd>A quantity of tokens used to stake the claim. See [Controlling](#controlling).</dd>
|
||||
<dt>value</dt>
|
||||
<dd>Metadata about a piece of content, a publisher's public key, or other information. See [Metadata](#metadata).</dd>
|
||||
</dl>
|
||||
|
||||
|
||||
<h4 id="claim-example">Claim Example</h4>
|
||||
|
||||
<!-- done -->
|
||||
|
||||
<p>Here is an example claim:</p>
|
||||
|
||||
|
|
@ -202,19 +236,28 @@
|
|||
|
||||
<h4 id="claim-operations">Claim Operations</h4>
|
||||
|
||||
<p>There are four claim operations: <code>create</code>, <code>support</code>, <code>update</code>, and <code>abandon</code>.</p>
|
||||
<!-- done -->
|
||||
|
||||
<p>A <code>create</code> operation makes a new claim for a name, or submits a competing claim on an existing name.</p>
|
||||
<p>There are four claim operations: <em>create</em>, <em>support</em>, <em>update</em>, and <em>abandon</em>.</p>
|
||||
|
||||
<p>A <code>support</code> is a claim that adds to the credit total of an existing claim. A support does not have it’s own claim ID or data. Instead, it has the claim ID of the claim to which its amount will be added.</p>
|
||||
|
||||
<p>An <code>update</code> changes the data or the amount stored in an existing claim or support. Updates do not change the claim ID, so an updated claim retains any supports attached to it.</p>
|
||||
|
||||
<p>An <code>abandon</code> withdraws a claim or support, freeing the associated credits to be used for other purposes.</p>
|
||||
<dl>
|
||||
<dt>create</dt>
|
||||
<dd>Makes a new claim.</dd>
|
||||
<dt>support</dt>
|
||||
<dd>Adds its [[amount]] to the stake of an already existing claim. It contains no metadata.</dd>
|
||||
<dt>update</dt>
|
||||
<dd>Changes the data or the amount stored in an existing claim or support. Updates do not change the claim ID, so an updated claim retains any supports attached to it. </dd>
|
||||
<dt>abandon</dt>
|
||||
<dd>Withdraws a claim or support, freeing the associated credits to be used for other purposes.</dd>
|
||||
</dl>
|
||||
|
||||
<h4 id="claimtrie">Claimtrie</h4>
|
||||
|
||||
<p>The <code>claimtrie</code> is the data structure that LBRY uses to store claims and prove the correctness of name resolution. It is a <a href="https://en.wikipedia.org/wiki/Merkle_tree">Merkle tree</a> that maps names to claims. Claims are stored as leaf nodes in the tree. Names are stored as the path from the root node to the leaf node.</p>
|
||||
<!-- done -->
|
||||
|
||||
<p>The <em>claimtrie</em> is the data structure used to store the set of all claims and prove the correctness of claim resolution.</p>
|
||||
|
||||
<p>The claimtrie is implemented as a <a href="https://en.wikipedia.org/wiki/Merkle_tree">Merkle tree</a> that maps names to claims. Claims are stored as leaf nodes in the tree. Names are stored as the path from the root node to the leaf node.</p>
|
||||
|
||||
<p>The hash of the root node (the <code>root hash</code>) is stored in the header of each block in the blockchain. Nodes in the LBRY network use the root hash to efficiently and securely validate the state of the claimtrie.</p>
|
||||
|
||||
|
|
@ -222,17 +265,23 @@
|
|||
|
||||
<p>For more details on the specific claimtrie implementation, see <a href="https://github.com/lbryio/lbrycrd/blob/master/src/claimtrie.cpp">the source code</a>.</p>
|
||||
|
||||
<h4 id="claim-properties">Claim Properties</h4>
|
||||
<h4 id="claim-statuses">Claim Statuses</h4>
|
||||
|
||||
<p>A claim can have one or more the following properties at a given block:</p>
|
||||
<!-- done -->
|
||||
|
||||
<p>A claim can have one or more the following properties at a given block.</p>
|
||||
|
||||
<h5 id="accepted">Accepted</h5>
|
||||
|
||||
<p>An accepted claim or support is simply one that has been entered into the blockchain. This happens when the transaction containing the claim is included in a block.</p>
|
||||
<!-- done -->
|
||||
|
||||
<p>An accepted claim or support is one that has been entered into the blockchain. This happens when the transaction containing the claim is included in a block.</p>
|
||||
|
||||
<h5 id="abandoned">Abandoned</h5>
|
||||
|
||||
<p>An abandoned claim or support is one that was withdrawn by its creator. It is no longer in contention to control a name. Spending the transaction that contains the claim will also cause the claim to become abandoned.</p>
|
||||
<!-- done -->
|
||||
|
||||
<p>An abandoned claim or support is one that was withdrawn by its creator. It is no longer in contention to control a name. Spending a transaction that contains a claim will cause that claim to become abandoned.</p>
|
||||
|
||||
<p>While data related to abandoned claims technically still resides in the blockchain, it is considered inappropriate to use this data to fetch the associated content.</p>
|
||||
|
||||
|
|
@ -299,9 +348,7 @@
|
|||
|
||||
<h4 id="normalization">Normalization</h4>
|
||||
|
||||
<p>TODO: Talk about how claim names are normalized.</p>
|
||||
|
||||
<p><a href="https://github.com/lbryio/lbrycrd/issues/208">https://github.com/lbryio/lbrycrd/issues/208</a></p>
|
||||
<p>Names in the claimtrie are normalized to avoid confusion due to Unicode equivalence or casing. All names are normalized using the NFD normalization form, then lowercased using the en_US locale.</p>
|
||||
|
||||
<h3 id="urls">URLs</h3>
|
||||
|
||||
|
|
@ -348,20 +395,22 @@ lbry:meet-LBRY$3
|
|||
<pre><code>lbry:@lbry/meet-LBRY
|
||||
</code></pre>
|
||||
|
||||
<p>The full URL grammar is defined below using <a href="https://www.w3.org/TR/2017/REC-xquery-31-20170321/#EBNFNotation">Xquery EBNF notation</a>:</p>
|
||||
<h4 id="grammar">Grammar</h4>
|
||||
|
||||
<p>The full URL grammar is defined using <a href="https://www.w3.org/TR/2017/REC-xquery-31-20170321/#EBNFNotation">Xquery EBNF notation</a>:</p>
|
||||
|
||||
<!-- see http://bottlecaps.de/rr/ui for visuals-->
|
||||
|
||||
<pre><code>URL ::= Scheme Path Query?
|
||||
|
||||
Scheme ::= 'lbry:'
|
||||
Scheme ::= 'lbry://'
|
||||
|
||||
Path ::= ClaimNameAndModifier | ChannelAndModifier ( '/' ClaimNameAndModifier )?
|
||||
|
||||
ClaimNameAndModifier ::= ClaimName Modifier?
|
||||
ChannelAndModifier ::= Channel Modifier?
|
||||
|
||||
ClaimName ::= AllowedChar+
|
||||
ClaimName ::= NameChar+
|
||||
Channel ::= '@' ClaimName
|
||||
|
||||
Modifier ::= ClaimID | ClaimSequence | BidPosition
|
||||
|
|
@ -372,8 +421,8 @@ BidPosition ::= '$' PositiveNumber
|
|||
Query ::= '?' QueryParameterList
|
||||
QueryParameterList ::= QueryParameter ( '&' QueryParameterList )*
|
||||
QueryParameter ::= QueryParameterName ( '=' QueryParameterValue )?
|
||||
QueryParameterName ::= AllowedChar+
|
||||
QueryParameterValue ::= AllowedChar+
|
||||
QueryParameterName ::= NameChar+
|
||||
QueryParameterValue ::= NameChar+
|
||||
|
||||
PositiveDigit ::= [123456789]
|
||||
Digit ::= '0' | PositiveDigit
|
||||
|
|
@ -382,7 +431,8 @@ PositiveNumber ::= PositiveDigit Digit*
|
|||
HexAlpha ::= [abcdef]
|
||||
Hex ::= (Digit | HexAlpha)+
|
||||
|
||||
AllowedChar ::= [^=&#:$@?/] /* any UTF8 character that is not reserved */
|
||||
NameChar ::= Char - [=&#:$@?/] /* any character that is not reserved */
|
||||
Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */
|
||||
</code></pre>
|
||||
|
||||
<h4 id="design-notes">Design Notes</h4>
|
||||
|
|
@ -512,17 +562,16 @@ OP_SUPPORT_CLAIM <name> <claimId> OP_2DROP OP_DROP <pubKey>
|
|||
|
||||
<h4 id="streams-and-stream-hashes">Streams and Stream Hashes</h4>
|
||||
|
||||
<p>(The metadata property <code>sd_hash</code> contains a unique identifier to locate and find the content in the data network. Reference [[Data]].)</p>
|
||||
<p>(The metadata property <code>lbry_sd_hash</code> contains a unique identifier to locate and find the content in the data network. Reference [[Data]].)</p>
|
||||
|
||||
<h4 id="fees-and-fee-structure">Fees and Fee Structure</h4>
|
||||
|
||||
<ul>
|
||||
<li>LBC</li>
|
||||
<li>Currencies?</li>
|
||||
<li>channel signatures and private keys</li>
|
||||
</ul>
|
||||
|
||||
<h4 id="more">More?</h4>
|
||||
|
||||
<h3 id="identities">Identities</h3>
|
||||
|
||||
<p>Channels are the unit of identity in the LBRY system. A channel is a claim that start with <code>@</code> and contains a metadata structure for identities rather than content. The most important part of channel’s metadata is the public key. Claims belonging to a channel are signed with the corresponding private key. A valid signature proves channel membership.</p>
|
||||
|
|
@ -555,6 +604,8 @@ OP_SUPPORT_CLAIM <name> <claimId> OP_2DROP OP_DROP <pubKey>
|
|||
|
||||
<h3 id="metadata-validation">Metadata Validation</h3>
|
||||
|
||||
<p>Clients are responsible for validating metadata, including data structure and signatures.</p>
|
||||
|
||||
<p>(expand)</p>
|
||||
|
||||
<ul>
|
||||
|
|
@ -570,13 +621,89 @@ OP_SUPPORT_CLAIM <name> <claimId> OP_2DROP OP_DROP <pubKey>
|
|||
|
||||
<h4 id="blobs">Blobs</h4>
|
||||
|
||||
<p>The unit of content in our network is called a <code>blob</code>. A blob is an encrypted chunk of data up to 2MB in size. Each blob is indexed by its <code>blob hash</code>, which is a SHA384 hash of the blob contents. Addressing blobs by their hashes simultaneously protects against naming collisions and ensures that the content you get is what you expect.</p>
|
||||
<p>The unit of data in our network is called a <em>blob</em>. A blob is an encrypted chunk of data up to 2MB in size. Each blob is indexed by its <em>blob hash</em>, which is a SHA384 hash of the blob contents. Addressing blobs by their hashes simultaneously protects against naming collisions and ensures that the content you get is what you expect.</p>
|
||||
|
||||
<p>Blobs are encrypted using AES-256 in CBC mode and PKCS7 padding. In order to keep each encrypted blob at 2MB max, a blob can hold at most 2097151 bytes (2MB minus 1 byte) of plaintext data. The source code for exact algorithm is available <a href="https://github.com/lbryio/lbry.go/blob/master/stream/blob.go">here</a>. The encryption key and IV for each blob is stored as described below.</p>
|
||||
|
||||
<h4 id="streams">Streams</h4>
|
||||
|
||||
<p>Multiple blobs may be combined into a <code>stream</code>. A stream may be a book, a movie, a CAD file, etc. All content on the network is shared as streams. Every stream begins with the <code>stream descriptor</code> blob, which contains a JSON list of the hashes and keys of the <code>content blobs</code>. The content blobs hold the actual content of the stream. Every stream ends with an empty content blob, to signify that the stream has finished (this is similar to a null-terminated string, and is necessary to support streaming content).</p>
|
||||
<p>Multiple blobs are combined into a <em>stream</em>. A stream may be a book, a movie, a CAD file, etc. All content on the network is shared as streams. Every stream begins with the <em>manifest blob</em>, followed by one or more <em>content blobs</em>. The content blobs hold the actual content of the stream. The manifest blob contains information necessary to find the content blobs and convert them into a file. This includes the hashes of the content blobs, their order in the stream, and cryptographic material for decrypting them.</p>
|
||||
|
||||
<h4 id="how-to-turn-files-into-streams-and-vice-versa">How to Turn Files into Streams, and Vice Versa</h4>
|
||||
<p>The blob hash of the manifest blob is called the <em>stream hash</em>. It uniquely identifies each stream.</p>
|
||||
|
||||
<h5 id="manifest-encoding">Manifest Encoding</h5>
|
||||
|
||||
<p>A manifest blob’s contents are encoded using a canonical JSON encoding. The JSON encoding must be canonical to support consistent hashing and validation. The encoding is the same as standard JSON, but adds the following rules:</p>
|
||||
|
||||
<ul>
|
||||
<li>Object keys must be quoted and lexicographically sorted.</li>
|
||||
<li>All strings are hex-encoded. Hex letters must be lowercase.</li>
|
||||
<li>Whitespace before, after, or between tokens is not permitted.</li>
|
||||
<li>Floating point numbers, leading zeros, and “minus 0” for integers are not permitted.</li>
|
||||
<li>Trailing commas after the last item in an array or object are not permitted.</li>
|
||||
</ul>
|
||||
|
||||
<p>Here’s an example manifest, with whitespace added for readability:</p>
|
||||
|
||||
<!-- originally from 053b2f0f0e82e7f022837382733d5f5817dcd67027103fe43f00fa7a6f9fa8742c1022a851616c1ac15d1c60e89db3f4 -->
|
||||
|
||||
<pre><code>{
|
||||
"blobs":[
|
||||
{
|
||||
"blob_hash":"a6daea71be2bb89fab29a2a10face08143411a5245edcaa5efff48c2e459e7ec01ad20edfde6da43a932aca45b2cec61",
|
||||
"iv":"ef6caef207a207ca5b14c0282d25ce21",
|
||||
"length":2097152
|
||||
},
|
||||
{
|
||||
"blob_hash":"bf2717e2c445052366d35bcd58edb108cbe947af122d8f76b4856db577aeeaa2def5b57dbb80f7b1531296bd3e0256fc",
|
||||
"iv":"a37b291a37337fc1ff90ae655c244c1d",
|
||||
"length":2097152
|
||||
},
|
||||
...,
|
||||
{
|
||||
"blob_hash":"322973617221ddfec6e53bff4b74b9c21c968cd32ba5a5094d84210e660c4b2ed0882b114a2392a08b06183f19330aaf",
|
||||
"iv": "a00f5f458695bdc9d50d3dbbc7905abc",
|
||||
"length": 600160
|
||||
}
|
||||
],
|
||||
"filename":"6b706a7977755477704d632e6d7034",
|
||||
"key":"94d89c0493c576057ac5f32eb0871180"
|
||||
}
|
||||
</code></pre>
|
||||
|
||||
<p>The <code>key</code> field contains the key to decrypt the stream, and is optional. The key may be stored by a third party and made available to a client when presented with proof that the content was purchased. The <code>length</code> field for each blob is the length of the encrypted blob, not the original file chunk.</p>
|
||||
|
||||
<p>Every stream must have at least two blobs - the manifest blob and a content blob. Consequently, zero-length streams are not allowed.</p>
|
||||
|
||||
<h4 id="stream-creation">Stream Creation</h4>
|
||||
|
||||
<p>A file must be converted into a stream before it can be published. Conversion involves breaking the file into chunks, encrypting the chunks into content blobs, and creating the manifest blob. Here are the steps:</p>
|
||||
|
||||
<h5 id="setup">Setup</h5>
|
||||
|
||||
<ol>
|
||||
<li>Generate a random 32-byte key for the stream.</li>
|
||||
</ol>
|
||||
|
||||
<h5 id="content-blobs">Content Blobs</h5>
|
||||
|
||||
<ol>
|
||||
<li>Break the file into chunks of at most 2097151 bytes.</li>
|
||||
<li>Generate a random IV for each chuck.</li>
|
||||
<li>Pad each chunk using PKCS7 padding</li>
|
||||
<li>Encrypt each chunk with AES-CBC using the stream key and the IV for that chunk.</li>
|
||||
<li>An encrypted chunk is a blob.</li>
|
||||
</ol>
|
||||
|
||||
<h5 id="manifest-blob">Manifest Blob</h5>
|
||||
|
||||
<ol>
|
||||
<li>Fill in the manifest data.</li>
|
||||
<li>Encode the data using the canonical JSON encoding described above.</li>
|
||||
<li>Compute the stream hash</li>
|
||||
</ol>
|
||||
|
||||
<p>An implementation of this process is available <a href="https://github.com/lbryio/lbry.go/tree/master/stream">here</a>.</p>
|
||||
|
||||
<h3 id="download">Download</h3>
|
||||
|
||||
|
|
|
|||
119
index.md
119
index.md
|
|
@ -47,9 +47,11 @@ TODO:
|
|||
* [Conventions and Terminology](#conventions-and-terminology)
|
||||
* [Blockchain](#blockchain)
|
||||
* [Claims](#claims)
|
||||
* [Claim Properties](#claim-properties)
|
||||
* [Claim Example](#claim-example)
|
||||
* [Claim Operations](#claim-operations)
|
||||
* [Claimtrie](#claimtrie)
|
||||
* [Claim Properties](#claim-properties)
|
||||
* [Claim Statuses](#claim-statuses)
|
||||
* [Accepted](#accepted)
|
||||
* [Abandoned](#abandoned)
|
||||
* [Active](#active)
|
||||
|
|
@ -57,6 +59,7 @@ TODO:
|
|||
* [Normalization](#normalization)
|
||||
* [URLs](#urls)
|
||||
* [Components](#components)
|
||||
* [Grammar](#grammar)
|
||||
* [Design Notes](#design-notes)
|
||||
* [Transactions](#transactions)
|
||||
* [Operations and Opcodes](#operations-and-opcodes)
|
||||
|
|
@ -72,14 +75,17 @@ TODO:
|
|||
* [Key Metadata Fields](#key-metadata-fields)
|
||||
* [Streams and Stream Hashes](#streams-and-stream-hashes)
|
||||
* [Fees and Fee Structure](#fees-and-fee-structure)
|
||||
* [More?](#more)
|
||||
* [Identities](#identities)
|
||||
* [Metadata Validation](#metadata-validation)
|
||||
* [Data](#data)
|
||||
* [Encoding and Decoding](#encoding-and-decoding)
|
||||
* [Blobs](#blobs)
|
||||
* [Streams](#streams)
|
||||
* [How to Turn Files into Streams, and Vice Versa](#how-to-turn-files-into-streams-and-vice-versa)
|
||||
* [Manifest Encoding](#manifest-encoding)
|
||||
* [Stream Creation](#stream-creation)
|
||||
* [Setup](#setup)
|
||||
* [Content Blobs](#content-blobs)
|
||||
* [Manifest Blob](#manifest-blob)
|
||||
* [Download](#download)
|
||||
* [Distributed Hash Table](#distributed-hash-table)
|
||||
* [Blob Exchange Protocol](#blob-exchange-protocol)
|
||||
|
|
@ -149,11 +155,15 @@ The LBRY blockchain is a public, proof-of-work blockchain. It serves three key p
|
|||
|
||||
The LBRY blockchain is a fork of the [Bitcoin](https://bitcoin.org/bitcoin.pdf) blockchain, with substantial modifications. This document will not cover or specify any aspects of LBRY that are identical to Bitcoin, and will instead focus on the differences.
|
||||
|
||||
### Claims <!-- done -->
|
||||
### Claims
|
||||
|
||||
<!-- done -->
|
||||
|
||||
A single metadata entry in the blockchain is called a _claim_. It records a file that was published to the network or a publisher's identity.
|
||||
|
||||
#### Claim Properties <!-- done -->
|
||||
#### Claim Properties
|
||||
|
||||
<!-- done -->
|
||||
|
||||
Every claim contains 4 properties:
|
||||
|
||||
|
|
@ -168,7 +178,9 @@ Every claim contains 4 properties:
|
|||
<dd>Metadata about a piece of content, a publisher's public key, or other information. See [Metadata](#metadata).</dd>
|
||||
</dl>
|
||||
|
||||
#### Claim Example <!-- done -->
|
||||
#### Claim Example
|
||||
|
||||
<!-- done -->
|
||||
|
||||
Here is an example claim:
|
||||
|
||||
|
|
@ -189,7 +201,9 @@ Here is an example claim:
|
|||
}
|
||||
```
|
||||
|
||||
#### Claim Operations <!-- done -->
|
||||
#### Claim Operations
|
||||
|
||||
<!-- done -->
|
||||
|
||||
There are four claim operations: _create_, _support_, _update_, and _abandon_.
|
||||
|
||||
|
|
@ -204,7 +218,9 @@ There are four claim operations: _create_, _support_, _update_, and _abandon_.
|
|||
<dd>Withdraws a claim or support, freeing the associated credits to be used for other purposes.</dd>
|
||||
</dl>
|
||||
|
||||
#### Claimtrie <!-- done -->
|
||||
#### Claimtrie
|
||||
|
||||
<!-- done -->
|
||||
|
||||
The _claimtrie_ is the data structure used to store the set of all claims and prove the correctness of claim resolution.
|
||||
|
||||
|
|
@ -216,15 +232,21 @@ Multiple claims can exist for the same name. They are all stored in the leaf nod
|
|||
|
||||
For more details on the specific claimtrie implementation, see [the source code](https://github.com/lbryio/lbrycrd/blob/master/src/claimtrie.cpp).
|
||||
|
||||
#### Claim Statuses <!-- done -->
|
||||
#### Claim Statuses
|
||||
|
||||
<!-- done -->
|
||||
|
||||
A claim can have one or more the following properties at a given block.
|
||||
|
||||
##### Accepted <!-- done -->
|
||||
##### Accepted
|
||||
|
||||
<!-- done -->
|
||||
|
||||
An accepted claim or support is one that has been entered into the blockchain. This happens when the transaction containing the claim is included in a block.
|
||||
|
||||
##### Abandoned <!-- done -->
|
||||
##### Abandoned
|
||||
|
||||
<!-- done -->
|
||||
|
||||
An abandoned claim or support is one that was withdrawn by its creator. It is no longer in contention to control a name. Spending a transaction that contains a claim will cause that claim to become abandoned.
|
||||
|
||||
|
|
@ -592,53 +614,82 @@ Clients are responsible for validating metadata, including data structure and si
|
|||
|
||||
#### Blobs
|
||||
|
||||
The unit of content in our network is called a `blob`. A blob is an encrypted chunk of data up to 2MB in size. Each blob is indexed by its `blob hash`, which is a SHA384 hash of the blob contents. Addressing blobs by their hashes simultaneously protects against naming collisions and ensures that the content you get is what you expect.
|
||||
The unit of data in our network is called a _blob_. A blob is an encrypted chunk of data up to 2MB in size. Each blob is indexed by its _blob hash_, which is a SHA384 hash of the blob contents. Addressing blobs by their hashes simultaneously protects against naming collisions and ensures that the content you get is what you expect.
|
||||
|
||||
Blobs are encrypted using AES-256 in CBC mode and PKCS7 padding. In order to keep each encrypted blob at 2MB max, a blob can hold at most 2097151 bytes (2MB minus 1 byte) of plaintext data. The source code for exact algorithm is available [here](https://github.com/lbryio/lbry.go/blob/master/stream/blob.go). The encryption key and IV for each blob is stored as described below.
|
||||
|
||||
#### Streams
|
||||
|
||||
Multiple blobs are combined into a `stream`. A stream may be a book, a movie, a CAD file, etc. All content on the network is shared as streams. Every stream begins with the `stream descriptor blob` (or SD blob), followed by one or more `content blobs`. The content blobs hold the actual content of the stream. The SD blob contains information necessary to find the content blobs and assemble them into a file. This includes the hashes of the content blobs, their order in the stream, and cryptographic material for decrypting them.
|
||||
Multiple blobs are combined into a _stream_. A stream may be a book, a movie, a CAD file, etc. All content on the network is shared as streams. Every stream begins with the _manifest blob_, followed by one or more _content blobs_. The content blobs hold the actual content of the stream. The manifest blob contains information necessary to find the content blobs and convert them into a file. This includes the hashes of the content blobs, their order in the stream, and cryptographic material for decrypting them.
|
||||
|
||||
Here's an example SD blob. It's hash is `053b2f0f0e82e7f022837382733d5f5817dcd67027103fe43f00fa7a6f9fa8742c1022a851616c1ac15d1c60e89db3f4`.
|
||||
The blob hash of the manifest blob is called the _stream hash_. It uniquely identifies each stream.
|
||||
|
||||
##### Manifest Encoding
|
||||
|
||||
A manifest blob's contents are encoded using a canonical JSON encoding. The JSON encoding must be canonical to support consistent hashing and validation. The encoding is the same as standard JSON, but adds the following rules:
|
||||
|
||||
- Object keys must be quoted and lexicographically sorted.
|
||||
- All strings are hex-encoded. Hex letters must be lowercase.
|
||||
- Whitespace before, after, or between tokens is not permitted.
|
||||
- Floating point numbers, leading zeros, and "minus 0" for integers are not permitted.
|
||||
- Trailing commas after the last item in an array or object are not permitted.
|
||||
|
||||
Here's an example manifest, with whitespace added for readability:
|
||||
|
||||
<!-- originally from 053b2f0f0e82e7f022837382733d5f5817dcd67027103fe43f00fa7a6f9fa8742c1022a851616c1ac15d1c60e89db3f4 -->
|
||||
|
||||
```
|
||||
{
|
||||
"stream_type":"lbryfile",
|
||||
"key":"94d89c0493c576057ac5f32eb0871180",
|
||||
"suggested_file_name":"6b706a7977755477704d632e6d7034",
|
||||
"stream_hash":"8cef6280f36f7e6590a6218da6b2eb8184ab1435c3f8d77f008088f5d2bc6bd2252a2beb9cfa3d9d40b9ce36d2d7b2ce"
|
||||
"stream_name":"6b706a7977755477704d632e6d7034",
|
||||
"blobs":[
|
||||
{
|
||||
"length":2097152,
|
||||
"blob_num":0,
|
||||
"blob_hash":"a6daea71be2bb89fab29a2a10face08143411a5245edcaa5efff48c2e459e7ec01ad20edfde6da43a932aca45b2cec61",
|
||||
"iv":"ef6caef207a207ca5b14c0282d25ce21"
|
||||
"iv":"ef6caef207a207ca5b14c0282d25ce21",
|
||||
"length":2097152
|
||||
},
|
||||
{
|
||||
"length":2097152,
|
||||
"blob_num":1,
|
||||
"blob_hash":"bf2717e2c445052366d35bcd58edb108cbe947af122d8f76b4856db577aeeaa2def5b57dbb80f7b1531296bd3e0256fc",
|
||||
"iv":"a37b291a37337fc1ff90ae655c244c1d"
|
||||
"iv":"a37b291a37337fc1ff90ae655c244c1d",
|
||||
"length":2097152
|
||||
},
|
||||
...,
|
||||
{
|
||||
"length":0,
|
||||
"blob_num":45,
|
||||
"iv":"53677e463ddb3bf060a40b99f8236432"
|
||||
"blob_hash":"322973617221ddfec6e53bff4b74b9c21c968cd32ba5a5094d84210e660c4b2ed0882b114a2392a08b06183f19330aaf",
|
||||
"iv": "a00f5f458695bdc9d50d3dbbc7905abc",
|
||||
"length": 600160
|
||||
}
|
||||
]
|
||||
],
|
||||
"filename":"6b706a7977755477704d632e6d7034",
|
||||
"key":"94d89c0493c576057ac5f32eb0871180"
|
||||
}
|
||||
```
|
||||
|
||||
Every field except 'stream_type' is either an integer or a hex-encoded string. The `key` field contains the key to decrypt the stream, and is optional. The key may be stored externally on a keyserver. The keyserver would make the key available to a client when presented with proof that the content was purchased.
|
||||
The `key` field contains the key to decrypt the stream, and is optional. The key may be stored by a third party and made available to a client when presented with proof that the content was purchased. The `length` field for each blob is the length of the encrypted blob, not the original file chunk.
|
||||
|
||||
The last blob in the `blobs` list of the SD hash is always an empty blob with no hash. This signifies the end of the stream. This is similar to a null-terminated string, and is necessary to support content where the length is not known in advance (e.g. live video).
|
||||
Every stream must have at least two blobs - the manifest blob and a content blob. Consequently, zero-length streams are not allowed.
|
||||
|
||||
Every stream must have at least two blobs - an SD blob and a content blob. Zero-length streams are not allowed.
|
||||
#### Stream Creation
|
||||
|
||||
#### How to Turn Files into Streams, and Vice Versa
|
||||
A file must be converted into a stream before it can be published. Conversion involves breaking the file into chunks, encrypting the chunks into content blobs, and creating the manifest blob. Here are the steps:
|
||||
|
||||
https://github.com/lbryio/lbry.go/tree/master/stream
|
||||
##### Setup
|
||||
|
||||
1. Generate a random 32-byte key for the stream.
|
||||
|
||||
##### Content Blobs
|
||||
|
||||
1. Break the file into chunks of at most 2097151 bytes.
|
||||
1. Generate a random IV for each chuck.
|
||||
1. Pad each chunk using PKCS7 padding
|
||||
1. Encrypt each chunk with AES-CBC using the stream key and the IV for that chunk.
|
||||
1. An encrypted chunk is a blob.
|
||||
|
||||
##### Manifest Blob
|
||||
|
||||
1. Fill in the manifest data.
|
||||
1. Encode the data using the canonical JSON encoding described above.
|
||||
1. Compute the stream hash
|
||||
|
||||
An implementation of this process is available [here](https://github.com/lbryio/lbry.go/tree/master/stream).
|
||||
|
||||
### Download
|
||||
|
||||
|
|
|
|||
3
watch.sh
Normal file
3
watch.sh
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
#!/bin/bash
|
||||
|
||||
./bin/reflex --decoration=none --start-service=true --inverse-regex='index\.html' -- sh -c "make"
|
||||
Loading…
Reference in a new issue