LBRYCommunity/spec
1
0
Fork 0
Code Issues 5 Pull requests Projects Releases Packages Wiki Activity

a bunch of small corrections

Browse source
This commit is contained in:
Alex Grintsvayg 2018-11-28 14:44:19 -05:00
parent b0f7b00a18
commit 4beac146fb
2 changed files with 117 additions and 130 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

119
index.html
View file

@ -218,8 +218,6 @@ fixme final polish checklist:
<h3 id="conventions-and-terminology">Conventions and Terminology</h3> <h3 id="conventions-and-terminology">Conventions and Terminology</h3>
<!-- fixme - rather than inline this here, I think we should use lbry.tech glossary definitions and the [[keyword]] syntax -->
<dl> <dl>
<dt>file</dt> <dt>file</dt>
<dd>A single piece of content published using LBRY.</dd> <dd>A single piece of content published using LBRY.</dd>
@ -228,16 +226,16 @@ fixme final polish checklist:
<dd>The unit of data transmission on the data network. A published file is split into many blobs.</dd> <dd>The unit of data transmission on the data network. A published file is split into many blobs.</dd>
<dt>stream</dt> <dt>stream</dt>
<dd>A set of blobs that can be reassembled into a file. Every stream has a descriptor blob and one or more content blobs.</dd> <dd>A set of blobs that can be reassembled into a file. Every stream has a manifest blob and one or more content blobs.</dd>
<dt>blob hash</dt> <dt>blob hash</dt>
<dd>The output of a cryptographic hash function is applied to a blob. Hashes are used to uniquely identify blobs and to verify that the contents of the blob are correct. Unless otherwise specified, LBRY uses SHA384 as the hash function.</dd> <dd>The output of a cryptographic hash function is applied to a blob. Hashes are used to uniquely identify blobs and to verify that the contents of the blob are correct. Unless otherwise specified, LBRY uses SHA384 as the hash function.</dd>
<dt>metadata</dt> <dt>metadata</dt>
<dd>Information about the contents of a stream (e.g. creator, description, stream descriptor hash, etc). Metadata is stored in the blockchain.</dd> <dd>Information about the contents of a stream (e.g. creator, description, stream hash, etc). Metadata is stored in the blockchain.</dd>
<dt>name</dt> <dt>name</dt>
<dd>A human-readable UTF8 string that is associated with a stream.</dd> <dd>A human-readable UTF8 string that is associated with a claim.</dd>
<dt>stake</dt> <dt>stake</dt>
<dd>An entry in the blockchain that commits credits toward a name.</dd> <dd>An entry in the blockchain that commits credits toward a name.</dd>
@ -262,10 +260,10 @@ fixme final polish checklist:
<ol> <ol>
<li>An index of the content available on the network</li> <li>An index of the content available on the network</li>
<li>A payment system and record of purchases for priced content</li> <li>A payment system and record of purchases for priced content</li>
<li>Trustful publisher identities</li> <li>Cryptographic publisher identities</li>
</ol> </ol>
<p>The blockchain is a fork of the <a href="https://bitcoin.org/bitcoin.pdf">Bitcoin</a> blockchain, with substantial modifications. This document will not cover or specify any aspects of LBRY that are identical to Bitcoin, and will instead focus on the differences.</p> <p>The LBRY blockchain is a fork of the <a href="https://bitcoin.org/bitcoin.pdf">Bitcoin</a> blockchain, with substantial modifications. This document does not cover or specify any aspects of LBRY that are identical to Bitcoin, and instead focuses on the differences.</p>
<h3 id="stakes">Stakes</h3> <h3 id="stakes">Stakes</h3>
@ -297,7 +295,7 @@ fixme final polish checklist:
<dl> <dl>
<dt>name</dt> <dt>name</dt>
<dd>A normalized UTF-8 string of up to 255 bytes used to address the claim. See <a href="#urls">URLs</a> and <a href="#normalization">Normalization</a>.</dd> <dd>A normalized UTF-8 string of up to 255 bytes used to address the claim. See <a href="#urls">URLs</a>.</dd>
<dt>value</dt> <dt>value</dt>
<dd>Metadata about a stream or a channel. See <a href="#metadata">Metadata</a>.</dd> <dd>Metadata about a stream or a channel. See <a href="#metadata">Metadata</a>.</dd>
</dl> </dl>
@ -308,7 +306,7 @@ fixme final polish checklist:
<p>Here is an example stream claim:</p> <p>Here is an example stream claim:</p>
<figure> <figure>
<pre><code>{ <pre><code>{
&quot;claim_id&quot;: &quot;6e56325c5351ceda2dd0795a30e864492910ccbf&quot;, &quot;claimID&quot;: &quot;6e56325c5351ceda2dd0795a30e864492910ccbf&quot;,
&quot;name&quot;: &quot;lbry&quot;, &quot;name&quot;: &quot;lbry&quot;,
&quot;amount&quot;: 1.0, &quot;amount&quot;: 1.0,
&quot;value&quot;: { &quot;value&quot;: {
@ -320,12 +318,12 @@ fixme final polish checklist:
&quot;license&quot;: &quot;LBRY inc&quot;, &quot;license&quot;: &quot;LBRY inc&quot;,
&quot;thumbnail&quot;: &quot;https://s3.amazonaws.com/files.lbry.io/logo.png&quot;, &quot;thumbnail&quot;: &quot;https://s3.amazonaws.com/files.lbry.io/logo.png&quot;,
&quot;mediaType&quot;: &quot;video/mp4&quot;, &quot;mediaType&quot;: &quot;video/mp4&quot;,
&quot;stream_hash&quot;: &quot;232068af6d51325c4821ac897d13d7837265812164021ec832cb7f18b9caf6c77c23016b31bac9747e7d5d9be7f4b752&quot;, &quot;streamHash&quot;: &quot;232068af6d51325c4821ac897d13d7837265812164021ec832cb7f18b9caf6c77c23016b31bac9747e7d5d9be7f4b752&quot;,
}, },
} }
} }
</code></pre> </code></pre>
<figcaption>Note: the blockchain treats the <code>value</code> as an opaque byte string and does not impose any structure on it. Structure is applied and validated <a href="#metadata-validation">higher in the stack</a>. In this example, the value is shown for demonstration purposes only. <figcaption>Note: the blockchain treats the <code>value</code> as an opaque byte string and does not impose any structure on it. Structure is applied and validated <a href="#metadata-validation">higher in the stack</a>. The value is shown here for demonstration purposes only.
</figcaption> </figcaption>
</figure> </figure>
@ -346,19 +344,19 @@ fixme final polish checklist:
<p>A <em>support</em> is a stake that lends its amount to an existing claim.</p> <p>A <em>support</em> is a stake that lends its amount to an existing claim.</p>
<p>Supports have one extra property on top of the basic stake properties: a <code>claim_id</code>. This is the ID of the claim that the support is bolstering.</p> <p>Supports have one extra property on top of the basic stake properties: a claim ID. This is the ID of the claim that the support is bolstering.</p>
<p>Supports are created and abandoned just like claims (see <a href="#claim-operations">Claim Operations</a>). They cannot be updated or themselves supported.</p> <p>Supports are created and abandoned just like claims (see <a href="#claim-operations">Claim Operations</a>). They cannot be updated or themselves supported.</p>
<h4 id="claimtrie">Claimtrie</h4> <h4 id="claimtrie">Claimtrie</h4>
<p>The <em>claimtrie</em> is the data structure used to store the set of all claims and prove the correctness of claim resolution.</p> <p>A <em>claimtrie</em> is a data structure used to store the set of all claims and prove the correctness of claim resolution.</p>
<p>The claimtrie is implemented as a <a href="https://en.wikipedia.org/wiki/Merkle_tree">Merkle tree</a> that maps names to claims. Claims are stored as leaf nodes in the tree. Names are stored as the path from the root node to the leaf node.</p> <p>The claimtrie is implemented as a <a href="https://en.wikipedia.org/wiki/Merkle_tree">Merkle tree</a> that maps names to claims. Claims are stored as leaf nodes in the tree. Names are stored as the <a href="#normalization">normalized</a> path from the root node to the leaf node.</p>
<p>The <em>root hash</em> is the hash of the root node. It is stored in the header of each block in the blockchain. Nodes use the root hash to efficiently and securely validate the state of the claimtrie.</p> <p>The <em>root hash</em> is the hash of the root node. It is stored in the header of each block in the blockchain. Nodes use the root hash to efficiently and securely validate the state of the claimtrie.</p>
<p>Multiple claims can exist for the same name. They are all stored in the leaf node for that name, sorted by <a href="descending">[effective amount]</a>, then by block height (ascending), then by transaction order in the block (ascending).</p> <p>Multiple claims can exist for the same name. They are all stored in the leaf node for that name, sorted by <a href="#active">effective amount</a> (descending), then by block height (ascending), then by transaction order in the block (ascending).</p>
<p>For more details on the specific claimtrie implementation, see <a href="https://github.com/lbryio/lbrycrd/blob/master/src/claimtrie.cpp">the source code</a>.</p> <p>For more details on the specific claimtrie implementation, see <a href="https://github.com/lbryio/lbrycrd/blob/master/src/claimtrie.cpp">the source code</a>.</p>
@ -372,13 +370,13 @@ fixme final polish checklist:
<p>Accepted stakes do not affect the intra-leaf claim order until they are <a href="#active">active</a>.</p> <p>Accepted stakes do not affect the intra-leaf claim order until they are <a href="#active">active</a>.</p>
<p>The sum of the amount of a claim stake and all accepted supports is called the <em>total amount</em>.</p> <p>The sum of the amount of a claim stake and all of its accepted supports is called its <em>total amount</em>.</p>
<h5 id="abandoned">Abandoned</h5> <h5 id="abandoned">Abandoned</h5>
<p>An <em>abandoned</em> stake is one that was withdrawn by its creator or current owner. Spending a transaction that contains a stake will cause that claim to become abandoned. Abandoned claim stakes are removed from the claimtrie.</p> <p>An <em>abandoned</em> stake is one that was withdrawn by its creator or current owner. Spending a transaction that contains a stake will cause that claim to become abandoned. Abandoned claim stakes are removed from the claimtrie.</p>
<p>While data related to abandoned stakes still resides in the blockchain, it is improper to use this data to fetch the associated content. Active claim stakes signed by abandoned identities will no longer be reported as valid.</p> <p>While data related to abandoned stakes still resides in the blockchain, it should be considered invalid and should not be used to resolve URLs or fetch the associated content. Active claim stakes signed by abandoned identities are also considered invalid.</p>
<h5 id="active">Active</h5> <h5 id="active">Active</h5>
@ -388,7 +386,7 @@ fixme final polish checklist:
<p>Otherwise, the activation delay is determined by a formula covered in <a href="#claimtrie-transitions">Claimtrie Transitions</a>. The formula&rsquo;s inputs are the height of the current block, the height at which the stake was accepted, and the height at which the controlling claim for that name last changed.</p> <p>Otherwise, the activation delay is determined by a formula covered in <a href="#claimtrie-transitions">Claimtrie Transitions</a>. The formula&rsquo;s inputs are the height of the current block, the height at which the stake was accepted, and the height at which the controlling claim for that name last changed.</p>
<p>The sum of the amount of an active claim and all active supports is called it&rsquo;s <em>effective amount</em>. The effective amount affects the sort order of claims in a leaf node, and which claim is controlling for that name. Claims that are not active have an effective amount of 0.</p> <p>The sum of the amount of an active claim and all of its active supports is called its <em>effective amount</em>. The effective amount affects the sort order of claims in a leaf node, and which claim is controlling for that name. Claims that are not active have an effective amount of 0.</p>
<h5 id="controlling">Controlling (claims only)</h5> <h5 id="controlling">Controlling (claims only)</h5>
@ -403,11 +401,11 @@ fixme final polish checklist:
<ol> <ol>
<li><p>For each claim, recalculate the effective amount.</p></li> <li><p>For each claim, recalculate the effective amount.</p></li>
<li><p>Sort the claims by effective amount in descending order. Claims tied for the same amount are ordered by block height (ascending), then by transaction order within the block (ascending).</p></li> <li><p>Sort the claims by effective amount in descending order. Claims tied for the same amount are ordered by block height (lowest first), then by transaction order within the block.</p></li>
<li><p>If the controlling claim from the previous block is still first in the order, then the sort is finished.</p></li> <li><p>If the controlling claim from the previous block is still first in the order, then the sort is finished.</p></li>
<li><p>Otherwise, a takeover is occurring. Set the takeover height for this name to the current height, recalculate which stakes are now active, and return to step 1.</p></li> <li><p>Otherwise, a takeover is occurring. Set the takeover height for this name to the current height, recalculate which stakes are now active, and redo steps 1 and 2.</p></li>
<li><p>At this point, the claim with the greatest effective amount is the controlling claim at this block.</p></li> <li><p>At this point, the claim with the greatest effective amount is the controlling claim at this block.</p></li>
</ol> </ol>
@ -416,22 +414,21 @@ fixme final polish checklist:
<h5 id="stake-activation">Stake Activation</h5> <h5 id="stake-activation">Stake Activation</h5>
<p>If a stake does not become active immediately, it becomes active at the block heigh determined by the following formula:</p> <p>If a stake does not become active immediately, it becomes active at the block height determined by the following formula:</p>
<pre><code>C + min(4032, floor((H-T) / 32)) <pre><code>ActivationHeight = AcceptedHeight + min(4032, floor( (AcceptedHeight-TakeoverHeight)/32 ))
</code></pre> </code></pre>
<p>Where:</p> <p>Where:</p>
<ul> <ul>
<li>C = stake height (height when the stake was accepted)</li> <li><code>AcceptedHeight</code> is the height when the stake was accepted</li>
<li>H = current height</li> <li><code>TakeoverHeight</code> is the most recent height at which the controlling claim for the name changed</li>
<li>T = takeover height (the most recent height at which the controlling claim for the name changed)</li>
</ul> </ul>
<p>In written form, the delay before a stake becomes active is equal to the stake&rsquo;s height minus height of the last takeover, divided by 32. The delay is capped at 4032 blocks, which is 7 days of blocks at 2.5 minutes per block (our target block time). The max delay is reached 224 (7x32) days after the last takeover.</p> <p>In written form, the delay before a stake becomes active is equal to the height at which the stake was accepted minus height of the last takeover, divided by 32. This delay is capped at a maximum of 4032 blocks, which is 7 days of blocks at 2.5 minutes per block (our target block time). It takes approximately 224 days without a takeover to reach the max delay.</p>
<p>The purpose of this delay function is to give long-standing claimants time to respond to changes, while still keeping takeover times reasonable and allowing recent or contentious claims to change state quickly.</p> <p>The purpose of this delay is to give long-standing claimants time to respond to changes, while still keeping takeover times reasonable and allowing recent or contentious claims to change state quickly.</p>
<h5 id="transition-example">Example</h5> <h5 id="transition-example">Example</h5>
@ -460,29 +457,27 @@ fixme final polish checklist:
<h4 id="normalization">Normalization</h4> <h4 id="normalization">Normalization</h4>
<p>Names in the claimtrie are normalized prior to comparison to avoid confusion due to Unicode equivalence or casing. When names are being compared, they are first converted using <a href="http://unicode.org/reports/tr15/#Norm_Forms">Unicode Normalization Form D</a> (NFD), then lowercased using the en_US locale when possible. Since claims competing for the same name are stored in the same node in the claimtrie, names are also normalized to determine the claimtrie path to the node.</p> <p>Names in the claimtrie are normalized when performing any comparisons. This is necessary to avoid confusion due to Unicode equivalence or casing. When names are being compared, they are first converted using <a href="http://unicode.org/reports/tr15/#Norm_Forms">Unicode Normalization Form D</a> (NFD), then lowercased using the en_US locale. This means names are effectively case-insensitive. Since claims competing for the same name are stored in the same node in the claimtrie, names are also normalized to determine the claimtrie path to the node.</p>
<h3 id="urls">URLs</h3> <h3 id="urls">URLs</h3>
<!-- fixme: <!-- fixme:
jeremy: @grin does SPV need a mention inside of the document? jeremy: @grin does SPV need a mention inside of the document?
grin: no, but we should probably include an example for how to do the validation using the root hash. its not strictly necessary because its similar to how bitcoin does it. so maybe link to https://lbry.tech/resources/claimtrie (which needs an update) and add a validation example there? grin: no, but we should probably include an example for how to do the validation using the root hash. its not strictly necessary because its similar to how bitcoin does it. so maybe link to https://lbry.tech/resources/claimtrie (which needs an update) and add a validation example there?
--> -->
<p>URLs are human-readable references to claims. All URLs:</p> <p>URLs are human-readable references to claims. All URLs:</p>
<ol> <ol>
<li>must contain a name (see <a href="#claim-properties">Claim Properties</a>), and</li> <li>contain a name (see <a href="#claim-properties">Claim Properties</a>), and</li>
<li>resolve to a single, specific claim for that name</li> <li>resolve to a single, specific claim for that name</li>
</ol> </ol>
<p>The ultimate purpose of much of the claim and blockchain design is to provide human-readable URLs that can be provably resolved by clients without a full copy of the blockchain (i.e. <a href="https://lbry.tech/glossary#spv">Simplified Payment Verification</a> wallets).</p> <p>The ultimate purpose of much of the claim and blockchain design is to provide human-readable URLs that can be provably resolved by clients without a full copy of the blockchain (i.e. <a href="https://lbry.tech/glossary#spv">Simplified Payment Verification</a> wallets).</p>
<p>It is possible to write short, human-readable, and memorable URLs.</p>
<h4 id="components">Components</h4> <h4 id="components">Components</h4>
<p>A URL is a name with one or more modifiers. A bare name on its own will resolve to the <a href="#controlling">controlling claim</a> at the latest block height. Common URL structures are:</p> <p>A URL is a name with one or more modifiers. A bare name on its own resolves to the <a href="#controlling">controlling claim</a> at the latest block height. Here are some common URL structures.</p>
<h5 id="stream-claim-name">Stream Claim Name</h5> <h5 id="stream-claim-name">Stream Claim Name</h5>
@ -516,7 +511,7 @@ lbry://@lbry#3f/meet-lbry
<h5 id="claim-sequence">Claim Sequence</h5> <h5 id="claim-sequence">Claim Sequence</h5>
<p>The <em>n</em>th claim for this name, in the order the claims entered the blockchain. <em>n</em> must be a positive number. This can be used to resolve claims in the order in which they were accepted, rather than by which claim has the most credits backing it.</p> <p>The <em>n</em>th accepted claim for this name. <em>n</em> must be a positive number. This can be used to resolve claims in the order in which they were made, rather than by the amount of credits backing a claim.</p>
<pre><code>lbry://meet-lbry:1 <pre><code>lbry://meet-lbry:1
lbry://@lbry:1/meet-lbry lbry://@lbry:1/meet-lbry
@ -533,7 +528,7 @@ lbry://@lbry$2/meet-lbry
<h5 id="query-params">Query Params</h5> <h5 id="query-params">Query Params</h5>
<p>These parameters are reserved for future use.</p> <p>These parameters have no meaning within the LBRY protocol. They are for use by upstream applications.</p>
<pre><code>lbry://meet-lbry?arg=value+arg2=value2 <pre><code>lbry://meet-lbry?arg=value+arg2=value2
</code></pre> </code></pre>
@ -574,7 +569,7 @@ PositiveNumber ::= PositiveDigit Digit*
HexAlpha ::= [abcdef] HexAlpha ::= [abcdef]
Hex ::= (Digit | HexAlpha)+ Hex ::= (Digit | HexAlpha)+
NameChar ::= Char - [=&amp;#:$@?/] /* any character that is not reserved */ NameChar ::= Char - [=&amp;#:$@%?/] /* any character that is not reserved */
Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */ Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */
</code></pre> </code></pre>
@ -785,21 +780,21 @@ Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]
<h4 id="design-notes">Design Notes</h4> <h4 id="design-notes">Design Notes</h4>
<p>The most contentious aspect of this design has been the choice to resolve naked names (sometimes called <em>vanity names</em>) to the claim with the highest effective amount.</p> <p>The most contentious aspect of this design is the choice to resolve naked names (sometimes called <em>vanity names</em>) to the claim with the highest effective amount.</p>
<p>First, it is important to note the problems in existing domain allocation design. Most existing public name schemes are first-come, first-serve with a fixed price. This leads to several bad outcomes:</p> <p>First, it is important to note the problems in existing name allocation designs. Most existing public name schemes are first-come, first-serve with a fixed price. This leads to several bad outcomes:</p>
<ol> <ol>
<li><p>Speculation and extortion. Entrepreneurs are incentivized to register common names even if they don&rsquo;t intend to use them, in hopes of selling them to the proper owner in the future for an exorbitant price. While speculation in general can have positive externalities (stable prices and price signals), in this case it is pure value extraction. Speculation also harms the user experience, who will see the vast majority of URLs sitting unused (c.f. Namecoin).</p></li> <li><p>Speculation and extortion. Entrepreneurs are incentivized to register common names even if they don&rsquo;t intend to use them, in hopes of selling them to the proper owner in the future for an exorbitant price. While speculation in general can have positive externalities (stable prices and price signals), in this case it is pure value extraction. Speculation also harms the user experience, who will see the vast majority of URLs sitting unused (c.f. Namecoin).</p></li>
<li><p>Bureaucracy and transaction costs. While a centralized system can allow for an authority to use a process to reassign names based on trademark or other common use reasons, this system is also imperfect. Most importantly, it is a censorship point and an avenue for complete exclusion. Additionally, such processes are often arbitrary, change over time, involve significant transaction costs, and <em>still</em> lead to names being used in ways that are contrary to user expectation (e.g. <a href="http://nissan.com">nissan.com</a>).</p></li> <li><p>Bureaucracy and transaction costs. While a centralized system can allow for an authority to use a process to reassign names based on trademark or other common use reasons, this system is also imperfect. Most importantly, it is a censorship point and an avenue for complete exclusion. Additionally, such processes are often arbitrary, change over time, involve significant transaction costs, and <em>still</em> lead to names being used in ways that are contrary to user expectation (e.g. <a href="http://nissan.com">nissan.com</a>).</p></li>
<li><p>Inefficiencies from price controls. Any system that does not allow a price to float freely creates inefficiencies. If the set price is too low, we facilitate speculation and rent-seeking. If the price is too high, we see people excluded from a good that it would otherwise be beneficial for them to purchase.</p></li> <li><p>Inefficiencies from price controls. Any system that does not allow a price to float freely creates inefficiencies. If the set price is too low, there is speculation and rent-seeking. If the price is too high, people are excluded from a good that it would otherwise be beneficial for them to purchase.</p></li>
</ol> </ol>
<p>We sought an algorithmic design built into consensus that would allow URLs to flow to their highest valued use. Following <a href="https://en.wikipedia.org/wiki/Coase_theorem">Coase</a>, this staking design allows for clearly defined rules, low transaction costs, and no information asymmetry, minimizing inefficiency in URL allocation.</p> <p>Instead, LBRY has an algorithmic design built into consensus that encourage URLs to flow to their highest valued use. Following <a href="https://en.wikipedia.org/wiki/Coase_theorem">Coase</a>, this staking design allows for clearly defined rules, low transaction costs, and no information asymmetry, minimizing inefficiency in URL allocation.</p>
<p>Finally, it&rsquo;s important to note that <em>only</em> vanity URLs have this property. Extremely short, memorable URLs like <code>lbry://myclaimname#a</code> exist and are available for the minimal cost of issuing a transaction.</p> <p>Finally, it&rsquo;s important to note that <em>only</em> vanity URLs have this property. Short, memorable URLs like <code>lbry://myclaimname#a</code> exist and are available for the minimal cost of issuing a transaction.</p>
<h3 id="transactions">Transactions</h3> <h3 id="transactions">Transactions</h3>
@ -807,38 +802,38 @@ Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]
<h4 id="operations-and-opcodes">Operations and Opcodes</h4> <h4 id="operations-and-opcodes">Operations and Opcodes</h4>
<p>To enable <a href="#claim-operations">claim operations</a>, we added three new opcodes to the scripting language: <code>OP_CLAIM_NAME</code>, <code>OP_UPDATE_CLAIM</code>, and <code>OP_SUPPORT_CLAIM</code>. In Bitcoin they are respectively <code>OP_NOP6</code>, <code>OP_NOP7</code>, and <code>OP_NOP8</code>. The opcodes are used in output scripts to interact with the claimtrie. Each opcode is followed by one or more parameters. Here&rsquo;s how these opcodes are used:</p> <p>To enable <a href="#claim-operations">claim operations</a>, three new opcodes were added to the scripting language: <code>OP_CLAIM_NAME</code>, <code>OP_UPDATE_CLAIM</code>, and <code>OP_SUPPORT_CLAIM</code>. In Bitcoin they are respectively <code>OP_NOP6</code>, <code>OP_NOP7</code>, and <code>OP_NOP8</code>. The opcodes are used in output scripts to interact with the claimtrie. Each opcode is followed by one or more parameters. Here&rsquo;s how these opcodes are used:</p>
<pre><code>OP_CLAIM_NAME &lt;name&gt; &lt;value&gt; OP_2DROP OP_DROP &lt;outputScript&gt; <pre><code>OP_CLAIM_NAME &lt;name&gt; &lt;value&gt; OP_2DROP OP_DROP &lt;outputScript&gt;
OP_UPDATE_CLAIM &lt;name&gt; &lt;claimId&gt; &lt;value&gt; OP_2DROP OP_2DROP &lt;outputScript&gt; OP_UPDATE_CLAIM &lt;name&gt; &lt;claimID&gt; &lt;value&gt; OP_2DROP OP_2DROP &lt;outputScript&gt;
OP_SUPPORT_CLAIM &lt;name&gt; &lt;claimId&gt; OP_2DROP OP_DROP &lt;outputScript&gt; OP_SUPPORT_CLAIM &lt;name&gt; &lt;claimID&gt; OP_2DROP OP_DROP &lt;outputScript&gt;
</code></pre> </code></pre>
<p>The <code>&lt;name&gt;</code> parameter is the [[name]] that the claim is associated with. <code>&lt;value&gt;</code> is the protobuf-encoded claim metadata and optional channel signature (see <a href="#metadata">Metadata</a> for more about this value). The <code>&lt;claimId&gt;</code> is the claim ID of a previous claim that is being updated or supported.</p> <p>The <code>&lt;name&gt;</code> parameter is the [[name]] that the claim is associated with. <code>&lt;value&gt;</code> is the protobuf-encoded claim metadata and optional channel signature (see <a href="#metadata">Metadata</a> for more about this value). The <code>&lt;claimID&gt;</code> is the claim ID of a previous claim that is being updated or supported.</p>
<p>Each opcode will push a zero on to the execution stack. Those zeros, as well as any additional parameters after the opcodes, are all dropped by <code>OP_2DROP</code> and <code>OP_DROP</code>. <code>&lt;outputScript&gt;</code> can be any valid script, so a script using these opcodes is also a pay-to-pubkey script. This means that claim scripts can be spent just like regular Bitcoin output scripts.</p> <p>Each opcode will push a zero on to the execution stack. Those zeros, as well as any additional parameters after the opcodes, are all dropped by <code>OP_2DROP</code> and <code>OP_DROP</code>. <code>&lt;outputScript&gt;</code> can be any valid script, so a script using these opcodes is also a pay-to-pubkey script. This means that claim scripts can be spent just like regular Bitcoin output scripts.</p>
<h5 id="claim-identifier-generation">Claim Identifier Generation</h5> <h5 id="claim-identifier-generation">Claim Identifier Generation</h5>
<p>Like any standard Bitcoin output script, a claim script will be associated with a transaction hash and output index. This combination of transaction hash and index is called an <em>outpoint</em>. Each claim script has a unique outpoint. The outpoint is hashed using SHA-256 and RIPEMD-160 to generate the claim ID for a claim. For the example above, let&rsquo;s say claim script is included in transaction <code>7560111513bea7ec38e2ce58a58c1880726b1515497515fd3f470d827669ed43</code> at the output index <code>1</code>. Then the claim ID would be <code>529357c3422c6046d3fec76be2358004ba22e323</code>. An implementation of this is available <a href="https://github.com/lbryio/lbry.go/blob/master/lbrycrd/blockchain.go">here</a>.</p> <p>Like any standard Bitcoin output script, a claim script is associated with a transaction hash and output index. This combination of transaction hash and index is called an <em>outpoint</em>. Each claim script has a unique outpoint. The outpoint is hashed using SHA-256 and RIPEMD-160 to generate the claim ID for a claim. For the example above, let&rsquo;s say claim script is included in transaction <code>7560111513bea7ec38e2ce58a58c1880726b1515497515fd3f470d827669ed43</code> at the output index <code>1</code>. Then the claim ID would be <code>529357c3422c6046d3fec76be2358004ba22e323</code>. An implementation of this is available <a href="https://github.com/lbryio/lbry.go/blob/master/lbrycrd/blockchain.go">here</a>.</p>
<h5 id="op-claim-name">OP_CLAIM_NAME</h5> <h5 id="op-claim-name">OP_CLAIM_NAME</h5>
<p>New claims are created using <code>OP_CLAIM_NAME</code>. For example, a claim transaction setting the name <code>Fruit</code> to the value <code>Apple</code> will look like this:</p> <p>New claims are created using <code>OP_CLAIM_NAME</code>. For example, a claim transaction setting the name <code>Fruit</code> to the value <code>Apple</code> looks like this:</p>
<pre><code>OP_CLAIM_NAME Fruit Apple OP_2DROP OP_DROP OP_DUP OP_HASH160 &lt;address&gt; OP_EQUALVERIFY OP_CHECKSIG <pre><code>OP_CLAIM_NAME Fruit Apple OP_2DROP OP_DROP OP_DUP OP_HASH160 &lt;address&gt; OP_EQUALVERIFY OP_CHECKSIG
</code></pre> </code></pre>
<h5 id="op-update-claim">OP_UPDATE_CLAIM</h5> <h5 id="op-update-claim">OP_UPDATE_CLAIM</h5>
<p><code>OP_UPDATE_CLAIM</code> updates a claim by replacing its metadata. An update transaction has an added requirement that it must spend the output for the existing claim that it wishes to update. Otherwise, it will be considered invalid and will not make it into the claimtrie. Thus it must have the following redeem script:</p> <p><code>OP_UPDATE_CLAIM</code> updates a claim by replacing its metadata. An update transaction has an added requirement that it must spend the output for the existing claim that it wishes to update. Otherwise, it is considered invalid and will not make it into the claimtrie. Thus it must have the following redeem script:</p>
<pre><code>&lt;signature&gt; &lt;pubKeyForPreviousAddress&gt; <pre><code>&lt;signature&gt; &lt;pubKeyForPreviousAddress&gt;
</code></pre> </code></pre>
<p>The syntax is identical to the standard way of redeeming a pay-to-pubkey script in Bitcoin, with the caveat that <code>&lt;pubKeyForPreviousAddress&gt;</code> must be the public key for the address of the output that contains the claim that will be updated.</p> <p>The syntax is identical to the standard way of redeeming a pay-to-pubkey script in Bitcoin, with the caveat that <code>&lt;pubKeyForPreviousAddress&gt;</code> must be the public key for the address of the output that contains the claim that is being updated.</p>
<p>To change the value of the previous example claim to “Banana”, the payout script would be</p> <p>To change the value of the previous example claim to “Banana”, the payout script would be</p>
@ -849,7 +844,7 @@ OP_SUPPORT_CLAIM &lt;name&gt; &lt;claimId&gt; OP_2DROP OP_DROP &lt;outputScript&
<h5 id="op-support-claim">OP_SUPPORT_CLAIM</h5> <h5 id="op-support-claim">OP_SUPPORT_CLAIM</h5>
<p>A support for the original example claim will have the following payout script:</p> <p>A support for the original example claim has the following payout script:</p>
<pre><code>OP_SUPPORT_CLAIM Fruit 529357c3422c6046d3fec76be2358004ba22e323 OP_2DROP OP_DROP OP_DUP OP_HASH160 &lt;address&gt; OP_EQUALVERIFY OP_CHECKSIG <pre><code>OP_SUPPORT_CLAIM Fruit 529357c3422c6046d3fec76be2358004ba22e323 OP_2DROP OP_DROP OP_DUP OP_HASH160 &lt;address&gt; OP_EQUALVERIFY OP_CHECKSIG
</code></pre> </code></pre>
@ -908,7 +903,7 @@ OP_SUPPORT_CLAIM &lt;name&gt; &lt;claimId&gt; OP_2DROP OP_DROP &lt;outputScript&
<h3 id="specification">Specification</h3> <h3 id="specification">Specification</h3>
<p>The metadata specification is designed to grow and change frequently. The full specification will not be detailed here. The <a href="https://github.com/lbryio/types">types</a> repository is considered the precise specification.</p> <p>The metadata specification is designed to grow and change frequently. The full specification is not detailed here. The <a href="https://github.com/lbryio/types">types</a> repository is considered the precise specification.</p>
<p>Instead, let&rsquo;s look at an example and some key fields.</p> <p>Instead, let&rsquo;s look at an example and some key fields.</p>
@ -971,7 +966,7 @@ OP_SUPPORT_CLAIM &lt;name&gt; &lt;claimId&gt; OP_2DROP OP_DROP &lt;outputScript&
<p>Channels are the unit of identity. A channel is a claim for a name beginning with <code>@</code> that contains a metadata structure for identity rather than content. Included in the metadata is the channel&rsquo;s public key. Here&rsquo;s an example:</p> <p>Channels are the unit of identity. A channel is a claim for a name beginning with <code>@</code> that contains a metadata structure for identity rather than content. Included in the metadata is the channel&rsquo;s public key. Here&rsquo;s an example:</p>
<pre><code>&quot;claim_id&quot;: &quot;6e56325c5351ceda2dd0795a30e864492910ccbf&quot;, <pre><code>&quot;claimID&quot;: &quot;6e56325c5351ceda2dd0795a30e864492910ccbf&quot;,
&quot;name&quot;: &quot;@lbry&quot;, &quot;name&quot;: &quot;@lbry&quot;,
&quot;amount&quot;: 6.26, &quot;amount&quot;: 6.26,
&quot;value&quot;: { &quot;value&quot;: {
@ -1072,15 +1067,13 @@ OP_SUPPORT_CLAIM &lt;name&gt; &lt;claimId&gt; OP_2DROP OP_DROP &lt;outputScript&
<h3 id="metadata-validation">Validation</h3> <h3 id="metadata-validation">Validation</h3>
<p>No enforcement or validation on metadata happens at the blockchain level. Instead, metadata encoding, decoding, and validation is done by clients. This allows evolution of the metadata without changes to consensus rules.</p> <p>No enforcement or validation on metadata happens at the blockchain level. Instead, metadata encoding, decoding, and validation is done by clients. This allows evolution of the metadata without changes to consensus rules. Clients are responsible for validating metadata, including data structure and signatures.</p>
<p>Clients are responsible for validating metadata, including data structure and signatures. This typically happens when the raw binary data stored in the blockchain is decoded client side via Protocol Buffers.</p>
<h2 id="data">Data</h2> <h2 id="data">Data</h2>
<p>Files published using LBRY are stored in a distributed fashion by the clients participating in the network. Each file split into multiple small pieces, encrypted, and announced to the network. It may also be uploaded to other hosts on the network that specialize in rehosting content.</p> <p>Files published using LBRY are stored in a distributed fashion by the clients participating in the network. Each file is split into multiple small pieces. Each piece is encrypted and announced to the network. The pieces may also be uploaded to other hosts on the network that specialize in rehosting content.</p>
<p>The purpose of this process is to enable file storage without relying on centralized infrastructure, and to create a marketplace for data that allows hosts to be paid for their services. The design is strongly influenced by the Bittorrent protocol and network.</p> <p>The purpose of this process is to enable file storage and access without relying on centralized infrastructure, and to create a marketplace for data that allows hosts to be paid for their services. The design is strongly influenced by the Bittorrent protocol.</p>
<h3 id="encoding">Encoding</h3> <h3 id="encoding">Encoding</h3>
@ -1088,7 +1081,7 @@ OP_SUPPORT_CLAIM &lt;name&gt; &lt;claimId&gt; OP_2DROP OP_DROP &lt;outputScript&
<h4 id="blobs">Blobs</h4> <h4 id="blobs">Blobs</h4>
<p>The smallest unit of data is called a <em>blob</em>. A blob is an encrypted chunk of data up to 2MiB in size. Each blob is indexed by its <em>blob hash</em>, which is a SHA384 hash of the blob contents. Addressing blobs by their hash protects against naming collisions and ensures that the content you get is what you expect.</p> <p>The smallest unit of data is called a <em>blob</em>. A blob is an encrypted chunk of data up to 2MiB in size. Each blob is indexed by its <em>blob hash</em>, which is a SHA384 hash of the blob. Addressing blobs by their hash protects against naming collisions and ensures that data cannot be accidentally or maliciously modified.</p>
<p>Blobs are encrypted using AES-256 in CBC mode and PKCS7 padding. In order to keep each encrypted blob at 2MiB max, a blob can hold at most 2097151 bytes (2MiB minus 1 byte) of plaintext data. The source code for the exact algorithm is available <a href="https://github.com/lbryio/lbry.go/blob/master/stream/blob.go">here</a>. The encryption key and IV for each blob is stored as described below.</p> <p>Blobs are encrypted using AES-256 in CBC mode and PKCS7 padding. In order to keep each encrypted blob at 2MiB max, a blob can hold at most 2097151 bytes (2MiB minus 1 byte) of plaintext data. The source code for the exact algorithm is available <a href="https://github.com/lbryio/lbry.go/blob/master/stream/blob.go">here</a>. The encryption key and IV for each blob is stored as described below.</p>
@ -1185,7 +1178,7 @@ OP_SUPPORT_CLAIM &lt;name&gt; &lt;claimId&gt; OP_2DROP OP_DROP &lt;outputScript&
<p>Decoding a stream is like encoding in reverse, and with the added step of verifying that the expected blob hashes match the actual data.</p> <p>Decoding a stream is like encoding in reverse, and with the added step of verifying that the expected blob hashes match the actual data.</p>
<ol> <ol>
<li>Verify that the manifest blob hash matches the stream hash you expect.</li> <li>Compute a SHA384 has of the manifest blob and verify that it matches the stream hash.</li>
<li>Parse the manifest blob contents.</li> <li>Parse the manifest blob contents.</li>
<li>Verify the hashes of the content blobs.</li> <li>Verify the hashes of the content blobs.</li>
<li>Decrypt and remove the padding from each content blob using the key and IVs in the manifest.</li> <li>Decrypt and remove the padding from each content blob using the key and IVs in the manifest.</li>
@ -1194,7 +1187,7 @@ OP_SUPPORT_CLAIM &lt;name&gt; &lt;claimId&gt; OP_2DROP OP_DROP &lt;outputScript&
<h3 id="announce">Announce</h3> <h3 id="announce">Announce</h3>
<p>After a [[stream]] is encoded, it must be <em>announced</em> to the network. Announcing is the process of letting other nodes on the network know that you have content available for download. LBRY tracks announced content using a distributed hash table.</p> <p>After a [[stream]] is encoded, it must be <em>announced</em> to the network. Announcing is the process of letting other nodes on the network know that a client has content available for download. LBRY tracks announced content using a distributed hash table.</p>
<h4 id="distributed-hash-table">Distributed Hash Table</h4> <h4 id="distributed-hash-table">Distributed Hash Table</h4>
@ -1219,7 +1212,7 @@ specification fairly closely, with some modifications.</p>
<h4 id="querying-the-dht">Querying the DHT</h4> <h4 id="querying-the-dht">Querying the DHT</h4>
<p>Querying works almost the same way as [[announcing]]. A client looking for a target hash will start by sending iterative <code>FindValue(target_hash)</code> requests to the nodes it knows that are closest to the target hash. If a node receives a <code>FindValue</code> request and knows of any peers for the target hash, it will respond with a list of those peers. Otherwise, it will respond with the closest nodes to the target hash that it knows about. The client then queries those closer nodes using the same <code>FindValue</code> call. This way, each call either finds the client some peers, or brings it closer to finding those peers. If no peers are found and no closer nodes are being returned, the client will determine that the target hash is not available and will give up.</p> <p>Querying works almost the same way as [[announcing]]. A client looking for a target hash starts by sending iterative <code>FindValue(target_hash)</code> requests to the nodes it knows that are closest to the target hash. If a node receives a <code>FindValue</code> request and knows of any peers for the target hash, it responds with a list of those peers. Otherwise, it responds with the closest nodes to the target hash that it knows about. The client then queries those closer nodes using the same <code>FindValue</code> call. This way, each call either finds the client some peers, or brings it closer to finding those peers. If no peers are found and no closer nodes are being returned, the client determines that the target hash is not available and gives up.</p>
<h4 id="blob-exchange-protocol">Blob Exchange Protocol</h4> <h4 id="blob-exchange-protocol">Blob Exchange Protocol</h4>
@ -1235,7 +1228,7 @@ specification fairly closely, with some modifications.</p>
<h5 id="download-1">Download</h5> <h5 id="download-1">Download</h5>
<p>Download requests the blob for a given hash. The response contains the blob, its hash, and the address where to send payment for the data transfer. If the blob is not available on the server, the response will instead contain an error.</p> <p>Download requests the blob for a given hash. The response contains the blob, its hash, and the address where to send payment for the data transfer. If the blob is not available on the server, the response instead contains an error.</p>
<h5 id="uploadcheck">UploadCheck</h5> <h5 id="uploadcheck">UploadCheck</h5>
@ -1251,7 +1244,7 @@ specification fairly closely, with some modifications.</p>
<p>In order for a client to download content, there must be hosts online that have the content the client wants, when the client wants it. To incentivize the continued hosting of data, the blob exchange protocol supports data upload and payment for data. <em>Reflectors</em> are hosts that accept data uploads. They rehost (reflect) the uploaded data and charge for downloads. Using a reflector is optional, but most publishers will probably choose to use them. Doing so obviates the need for the publisher&rsquo;s server to be online and connectable, which can be especially useful for mobile clients or those behind a firewall.</p> <p>In order for a client to download content, there must be hosts online that have the content the client wants, when the client wants it. To incentivize the continued hosting of data, the blob exchange protocol supports data upload and payment for data. <em>Reflectors</em> are hosts that accept data uploads. They rehost (reflect) the uploaded data and charge for downloads. Using a reflector is optional, but most publishers will probably choose to use them. Doing so obviates the need for the publisher&rsquo;s server to be online and connectable, which can be especially useful for mobile clients or those behind a firewall.</p>
<p>The current version of the protocol does not support sophisticated price negotiation between clients and hosts. The host simply chooses the price it will charge. Clients check this price before downloading, and pay the price after the download is complete. Future protocol versions will include more options for price negotiation, as well as stronger proofs of payment.</p> <p>The current version of the protocol does not support sophisticated price negotiation between clients and hosts. The host simply chooses the price it wants to charge. Clients check this price before downloading, and pay the price after the download is complete. Future protocol versions will include more options for price negotiation, as well as stronger proofs of payment.</p>
<pre style="font: 10px/5px monospace;overflow:hidden;text-align: center;margin: 10rem 0"> <pre style="font: 10px/5px monospace;overflow:hidden;text-align: center;margin: 10rem 0">

128
index.md
View file

@ -151,8 +151,6 @@ This document assumes that the reader is familiar with Bitcoin and blockchain te
### Conventions and Terminology ### Conventions and Terminology
<!-- fixme - rather than inline this here, I think we should use lbry.tech glossary definitions and the [[keyword]] syntax -->
<dl> <dl>
<dt>file</dt> <dt>file</dt>
<dd>A single piece of content published using LBRY.</dd> <dd>A single piece of content published using LBRY.</dd>
@ -161,16 +159,16 @@ This document assumes that the reader is familiar with Bitcoin and blockchain te
<dd>The unit of data transmission on the data network. A published file is split into many blobs.</dd> <dd>The unit of data transmission on the data network. A published file is split into many blobs.</dd>
<dt>stream</dt> <dt>stream</dt>
<dd>A set of blobs that can be reassembled into a file. Every stream has a descriptor blob and one or more content blobs.</dd> <dd>A set of blobs that can be reassembled into a file. Every stream has a manifest blob and one or more content blobs.</dd>
<dt>blob hash</dt> <dt>blob hash</dt>
<dd>The output of a cryptographic hash function is applied to a blob. Hashes are used to uniquely identify blobs and to verify that the contents of the blob are correct. Unless otherwise specified, LBRY uses SHA384 as the hash function.</dd> <dd>The output of a cryptographic hash function is applied to a blob. Hashes are used to uniquely identify blobs and to verify that the contents of the blob are correct. Unless otherwise specified, LBRY uses SHA384 as the hash function.</dd>
<dt>metadata</dt> <dt>metadata</dt>
<dd>Information about the contents of a stream (e.g. creator, description, stream descriptor hash, etc). Metadata is stored in the blockchain.</dd> <dd>Information about the contents of a stream (e.g. creator, description, stream hash, etc). Metadata is stored in the blockchain.</dd>
<dt>name</dt> <dt>name</dt>
<dd>A human-readable UTF8 string that is associated with a stream.</dd> <dd>A human-readable UTF8 string that is associated with a claim.</dd>
<dt>stake</dt> <dt>stake</dt>
<dd>An entry in the blockchain that commits credits toward a name.</dd> <dd>An entry in the blockchain that commits credits toward a name.</dd>
@ -196,9 +194,9 @@ The LBRY blockchain is a public, proof-of-work blockchain. It serves three key p
1. An index of the content available on the network 1. An index of the content available on the network
2. A payment system and record of purchases for priced content 2. A payment system and record of purchases for priced content
3. Trustful publisher identities 3. Cryptographic publisher identities
The blockchain is a fork of the [Bitcoin](https://bitcoin.org/bitcoin.pdf) blockchain, with substantial modifications. This document will not cover or specify any aspects of LBRY that are identical to Bitcoin, and will instead focus on the differences. The LBRY blockchain is a fork of the [Bitcoin](https://bitcoin.org/bitcoin.pdf) blockchain, with substantial modifications. This document does not cover or specify any aspects of LBRY that are identical to Bitcoin, and instead focuses on the differences.
### Stakes ### Stakes
@ -231,7 +229,7 @@ In addition to the properties that all stakes have, claims have two more propert
<dl> <dl>
<dt>name</dt> <dt>name</dt>
<dd>A normalized UTF-8 string of up to 255 bytes used to address the claim. See <a href="#urls">URLs</a> and <a href="#normalization">Normalization</a>.</dd> <dd>A normalized UTF-8 string of up to 255 bytes used to address the claim. See <a href="#urls">URLs</a>.</dd>
<dt>value</dt> <dt>value</dt>
<dd>Metadata about a stream or a channel. See <a href="#metadata">Metadata</a>.</dd> <dd>Metadata about a stream or a channel. See <a href="#metadata">Metadata</a>.</dd>
</dl> </dl>
@ -242,7 +240,7 @@ Here is an example stream claim:
``` ```
{ {
"claim_id": "6e56325c5351ceda2dd0795a30e864492910ccbf", "claimID": "6e56325c5351ceda2dd0795a30e864492910ccbf",
"name": "lbry", "name": "lbry",
"amount": 1.0, "amount": 1.0,
"value": { "value": {
@ -254,12 +252,12 @@ Here is an example stream claim:
"license": "LBRY inc", "license": "LBRY inc",
"thumbnail": "https://s3.amazonaws.com/files.lbry.io/logo.png", "thumbnail": "https://s3.amazonaws.com/files.lbry.io/logo.png",
"mediaType": "video/mp4", "mediaType": "video/mp4",
"stream_hash": "232068af6d51325c4821ac897d13d7837265812164021ec832cb7f18b9caf6c77c23016b31bac9747e7d5d9be7f4b752", "streamHash": "232068af6d51325c4821ac897d13d7837265812164021ec832cb7f18b9caf6c77c23016b31bac9747e7d5d9be7f4b752",
}, },
} }
} }
``` ```
Figure: Note: the blockchain treats the `value` as an opaque byte string and does not impose any structure on it. Structure is applied and validated [higher in the stack](#metadata-validation). In this example, the value is shown for demonstration purposes only. Figure: Note: the blockchain treats the `value` as an opaque byte string and does not impose any structure on it. Structure is applied and validated [higher in the stack](#metadata-validation). The value is shown here for demonstration purposes only.
#### Claim Operations #### Claim Operations
@ -278,19 +276,19 @@ There are three claim operations: _create_, _update_, and _abandon_.
A _support_ is a stake that lends its amount to an existing claim. A _support_ is a stake that lends its amount to an existing claim.
Supports have one extra property on top of the basic stake properties: a `claim_id`. This is the ID of the claim that the support is bolstering. Supports have one extra property on top of the basic stake properties: a claim ID. This is the ID of the claim that the support is bolstering.
Supports are created and abandoned just like claims (see [Claim Operations](#claim-operations)). They cannot be updated or themselves supported. Supports are created and abandoned just like claims (see [Claim Operations](#claim-operations)). They cannot be updated or themselves supported.
#### Claimtrie #### Claimtrie
The _claimtrie_ is the data structure used to store the set of all claims and prove the correctness of claim resolution. A _claimtrie_ is a data structure used to store the set of all claims and prove the correctness of claim resolution.
The claimtrie is implemented as a [Merkle tree](https://en.wikipedia.org/wiki/Merkle_tree) that maps names to claims. Claims are stored as leaf nodes in the tree. Names are stored as the path from the root node to the leaf node. The claimtrie is implemented as a [Merkle tree](https://en.wikipedia.org/wiki/Merkle_tree) that maps names to claims. Claims are stored as leaf nodes in the tree. Names are stored as the [normalized](#normalization) path from the root node to the leaf node.
The _root hash_ is the hash of the root node. It is stored in the header of each block in the blockchain. Nodes use the root hash to efficiently and securely validate the state of the claimtrie. The _root hash_ is the hash of the root node. It is stored in the header of each block in the blockchain. Nodes use the root hash to efficiently and securely validate the state of the claimtrie.
Multiple claims can exist for the same name. They are all stored in the leaf node for that name, sorted by [[effective amount]] (descending), then by block height (ascending), then by transaction order in the block (ascending). Multiple claims can exist for the same name. They are all stored in the leaf node for that name, sorted by [effective amount](#active) (descending), then by block height (ascending), then by transaction order in the block (ascending).
For more details on the specific claimtrie implementation, see [the source code](https://github.com/lbryio/lbrycrd/blob/master/src/claimtrie.cpp). For more details on the specific claimtrie implementation, see [the source code](https://github.com/lbryio/lbrycrd/blob/master/src/claimtrie.cpp).
@ -304,13 +302,13 @@ An _accepted_ stake is one that has been been entered into the blockchain. This
Accepted stakes do not affect the intra-leaf claim order until they are [active](#active). Accepted stakes do not affect the intra-leaf claim order until they are [active](#active).
The sum of the amount of a claim stake and all accepted supports is called the _total amount_. The sum of the amount of a claim stake and all of its accepted supports is called its _total amount_.
##### Abandoned ##### Abandoned
An _abandoned_ stake is one that was withdrawn by its creator or current owner. Spending a transaction that contains a stake will cause that claim to become abandoned. Abandoned claim stakes are removed from the claimtrie. An _abandoned_ stake is one that was withdrawn by its creator or current owner. Spending a transaction that contains a stake will cause that claim to become abandoned. Abandoned claim stakes are removed from the claimtrie.
While data related to abandoned stakes still resides in the blockchain, it is improper to use this data to fetch the associated content. Active claim stakes signed by abandoned identities will no longer be reported as valid. While data related to abandoned stakes still resides in the blockchain, it should be considered invalid and should not be used to resolve URLs or fetch the associated content. Active claim stakes signed by abandoned identities are also considered invalid.
##### Active ##### Active
@ -320,7 +318,7 @@ If the stake is an update to an active claim, is the first claim for a name, or
Otherwise, the activation delay is determined by a formula covered in [Claimtrie Transitions](#claimtrie-transitions). The formula's inputs are the height of the current block, the height at which the stake was accepted, and the height at which the controlling claim for that name last changed. Otherwise, the activation delay is determined by a formula covered in [Claimtrie Transitions](#claimtrie-transitions). The formula's inputs are the height of the current block, the height at which the stake was accepted, and the height at which the controlling claim for that name last changed.
The sum of the amount of an active claim and all active supports is called it's _effective amount_. The effective amount affects the sort order of claims in a leaf node, and which claim is controlling for that name. Claims that are not active have an effective amount of 0. The sum of the amount of an active claim and all of its active supports is called its _effective amount_. The effective amount affects the sort order of claims in a leaf node, and which claim is controlling for that name. Claims that are not active have an effective amount of 0.
##### Controlling (claims only) {#controlling} ##### Controlling (claims only) {#controlling}
@ -334,11 +332,11 @@ To determine the sort order of claims in a leaf node, the following algorithm is
1. For each claim, recalculate the effective amount. 1. For each claim, recalculate the effective amount.
2. Sort the claims by effective amount in descending order. Claims tied for the same amount are ordered by block height (ascending), then by transaction order within the block (ascending). 2. Sort the claims by effective amount in descending order. Claims tied for the same amount are ordered by block height (lowest first), then by transaction order within the block.
3. If the controlling claim from the previous block is still first in the order, then the sort is finished. 3. If the controlling claim from the previous block is still first in the order, then the sort is finished.
4. Otherwise, a takeover is occurring. Set the takeover height for this name to the current height, recalculate which stakes are now active, and return to step 1. 4. Otherwise, a takeover is occurring. Set the takeover height for this name to the current height, recalculate which stakes are now active, and redo steps 1 and 2.
5. At this point, the claim with the greatest effective amount is the controlling claim at this block. 5. At this point, the claim with the greatest effective amount is the controlling claim at this block.
@ -346,21 +344,20 @@ The purpose of 4 is to handle the case when multiple competing claims are made o
##### Stake Activation ##### Stake Activation
If a stake does not become active immediately, it becomes active at the block heigh determined by the following formula: If a stake does not become active immediately, it becomes active at the block height determined by the following formula:
``` ```
C + min(4032, floor((H-T) / 32)) ActivationHeight = AcceptedHeight + min(4032, floor( (AcceptedHeight-TakeoverHeight)/32 ))
``` ```
Where: Where:
- C = stake height (height when the stake was accepted) - `AcceptedHeight` is the height when the stake was accepted
- H = current height - `TakeoverHeight` is the most recent height at which the controlling claim for the name changed
- T = takeover height (the most recent height at which the controlling claim for the name changed)
In written form, the delay before a stake becomes active is equal to the stake's height minus height of the last takeover, divided by 32. The delay is capped at 4032 blocks, which is 7 days of blocks at 2.5 minutes per block (our target block time). The max delay is reached 224 (7x32) days after the last takeover. In written form, the delay before a stake becomes active is equal to the height at which the stake was accepted minus height of the last takeover, divided by 32. This delay is capped at a maximum of 4032 blocks, which is 7 days of blocks at 2.5 minutes per block (our target block time). It takes approximately 224 days without a takeover to reach the max delay.
The purpose of this delay function is to give long-standing claimants time to respond to changes, while still keeping takeover times reasonable and allowing recent or contentious claims to change state quickly. The purpose of this delay is to give long-standing claimants time to respond to changes, while still keeping takeover times reasonable and allowing recent or contentious claims to change state quickly.
##### Example {#transition-example} ##### Example {#transition-example}
@ -390,28 +387,26 @@ Here is a step-by-step example to illustrate the different scenarios. All stakes
#### Normalization #### Normalization
Names in the claimtrie are normalized prior to comparison to avoid confusion due to Unicode equivalence or casing. When names are being compared, they are first converted using [Unicode Normalization Form D](http://unicode.org/reports/tr15/#Norm_Forms) (NFD), then lowercased using the en_US locale when possible. Since claims competing for the same name are stored in the same node in the claimtrie, names are also normalized to determine the claimtrie path to the node. Names in the claimtrie are normalized when performing any comparisons. This is necessary to avoid confusion due to Unicode equivalence or casing. When names are being compared, they are first converted using [Unicode Normalization Form D](http://unicode.org/reports/tr15/#Norm_Forms) (NFD), then lowercased using the en_US locale. This means names are effectively case-insensitive. Since claims competing for the same name are stored in the same node in the claimtrie, names are also normalized to determine the claimtrie path to the node.
### URLs ### URLs
<!-- fixme: <!-- fixme:
jeremy: @grin does SPV need a mention inside of the document? jeremy: @grin does SPV need a mention inside of the document?
grin: no, but we should probably include an example for how to do the validation using the root hash. its not strictly necessary because its similar to how bitcoin does it. so maybe link to https://lbry.tech/resources/claimtrie (which needs an update) and add a validation example there? grin: no, but we should probably include an example for how to do the validation using the root hash. its not strictly necessary because its similar to how bitcoin does it. so maybe link to https://lbry.tech/resources/claimtrie (which needs an update) and add a validation example there?
--> -->
URLs are human-readable references to claims. All URLs: URLs are human-readable references to claims. All URLs:
1. must contain a name (see [Claim Properties](#claim-properties)), and 1. contain a name (see [Claim Properties](#claim-properties)), and
2. resolve to a single, specific claim for that name 2. resolve to a single, specific claim for that name
The ultimate purpose of much of the claim and blockchain design is to provide human-readable URLs that can be provably resolved by clients without a full copy of the blockchain (i.e. [Simplified Payment Verification](https://lbry.tech/glossary#spv) wallets). The ultimate purpose of much of the claim and blockchain design is to provide human-readable URLs that can be provably resolved by clients without a full copy of the blockchain (i.e. [Simplified Payment Verification](https://lbry.tech/glossary#spv) wallets).
It is possible to write short, human-readable, and memorable URLs.
#### Components #### Components
A URL is a name with one or more modifiers. A bare name on its own will resolve to the [controlling claim](#controlling) at the latest block height. Common URL structures are: A URL is a name with one or more modifiers. A bare name on its own resolves to the [controlling claim](#controlling) at the latest block height. Here are some common URL structures.
##### Stream Claim Name ##### Stream Claim Name
@ -449,7 +444,7 @@ lbry://@lbry#3f/meet-lbry
##### Claim Sequence ##### Claim Sequence
The _n_th claim for this name, in the order the claims entered the blockchain. _n_ must be a positive number. This can be used to resolve claims in the order in which they were accepted, rather than by which claim has the most credits backing it. The _n_th accepted claim for this name. _n_ must be a positive number. This can be used to resolve claims in the order in which they were made, rather than by the amount of credits backing a claim.
``` ```
lbry://meet-lbry:1 lbry://meet-lbry:1
@ -468,7 +463,7 @@ lbry://@lbry$2/meet-lbry
##### Query Params ##### Query Params
These parameters are reserved for future use. These parameters have no meaning within the LBRY protocol. They are for use by upstream applications.
``` ```
lbry://meet-lbry?arg=value+arg2=value2 lbry://meet-lbry?arg=value+arg2=value2
@ -511,7 +506,7 @@ PositiveNumber ::= PositiveDigit Digit*
HexAlpha ::= [abcdef] HexAlpha ::= [abcdef]
Hex ::= (Digit | HexAlpha)+ Hex ::= (Digit | HexAlpha)+
NameChar ::= Char - [=&#:$@?/] /* any character that is not reserved */ NameChar ::= Char - [=&#:$@%?/] /* any character that is not reserved */
Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */ Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */
``` ```
@ -581,19 +576,19 @@ URL | Claim ID | Note
#### Design Notes #### Design Notes
The most contentious aspect of this design has been the choice to resolve naked names (sometimes called _vanity names_) to the claim with the highest effective amount. The most contentious aspect of this design is the choice to resolve naked names (sometimes called _vanity names_) to the claim with the highest effective amount.
First, it is important to note the problems in existing domain allocation design. Most existing public name schemes are first-come, first-serve with a fixed price. This leads to several bad outcomes: First, it is important to note the problems in existing name allocation designs. Most existing public name schemes are first-come, first-serve with a fixed price. This leads to several bad outcomes:
1. Speculation and extortion. Entrepreneurs are incentivized to register common names even if they don't intend to use them, in hopes of selling them to the proper owner in the future for an exorbitant price. While speculation in general can have positive externalities (stable prices and price signals), in this case it is pure value extraction. Speculation also harms the user experience, who will see the vast majority of URLs sitting unused (c.f. Namecoin). 1. Speculation and extortion. Entrepreneurs are incentivized to register common names even if they don't intend to use them, in hopes of selling them to the proper owner in the future for an exorbitant price. While speculation in general can have positive externalities (stable prices and price signals), in this case it is pure value extraction. Speculation also harms the user experience, who will see the vast majority of URLs sitting unused (c.f. Namecoin).
2. Bureaucracy and transaction costs. While a centralized system can allow for an authority to use a process to reassign names based on trademark or other common use reasons, this system is also imperfect. Most importantly, it is a censorship point and an avenue for complete exclusion. Additionally, such processes are often arbitrary, change over time, involve significant transaction costs, and _still_ lead to names being used in ways that are contrary to user expectation (e.g. [nissan.com](http://nissan.com)). 2. Bureaucracy and transaction costs. While a centralized system can allow for an authority to use a process to reassign names based on trademark or other common use reasons, this system is also imperfect. Most importantly, it is a censorship point and an avenue for complete exclusion. Additionally, such processes are often arbitrary, change over time, involve significant transaction costs, and _still_ lead to names being used in ways that are contrary to user expectation (e.g. [nissan.com](http://nissan.com)).
3. Inefficiencies from price controls. Any system that does not allow a price to float freely creates inefficiencies. If the set price is too low, we facilitate speculation and rent-seeking. If the price is too high, we see people excluded from a good that it would otherwise be beneficial for them to purchase. 3. Inefficiencies from price controls. Any system that does not allow a price to float freely creates inefficiencies. If the set price is too low, there is speculation and rent-seeking. If the price is too high, people are excluded from a good that it would otherwise be beneficial for them to purchase.
We sought an algorithmic design built into consensus that would allow URLs to flow to their highest valued use. Following [Coase](https://en.wikipedia.org/wiki/Coase_theorem), this staking design allows for clearly defined rules, low transaction costs, and no information asymmetry, minimizing inefficiency in URL allocation. Instead, LBRY has an algorithmic design built into consensus that encourage URLs to flow to their highest valued use. Following [Coase](https://en.wikipedia.org/wiki/Coase_theorem), this staking design allows for clearly defined rules, low transaction costs, and no information asymmetry, minimizing inefficiency in URL allocation.
Finally, it's important to note that _only_ vanity URLs have this property. Extremely short, memorable URLs like `lbry://myclaimname#a` exist and are available for the minimal cost of issuing a transaction. Finally, it's important to note that _only_ vanity URLs have this property. Short, memorable URLs like `lbry://myclaimname#a` exist and are available for the minimal cost of issuing a transaction.
### Transactions ### Transactions
@ -602,28 +597,28 @@ The LBRY blockchain includes the following changes to Bitcoin's transaction scri
#### Operations and Opcodes #### Operations and Opcodes
To enable [claim operations](#claim-operations), we added three new opcodes to the scripting language: `OP_CLAIM_NAME`, `OP_UPDATE_CLAIM`, and `OP_SUPPORT_CLAIM`. In Bitcoin they are respectively `OP_NOP6`, `OP_NOP7`, and `OP_NOP8`. The opcodes are used in output scripts to interact with the claimtrie. Each opcode is followed by one or more parameters. Here's how these opcodes are used: To enable [claim operations](#claim-operations), three new opcodes were added to the scripting language: `OP_CLAIM_NAME`, `OP_UPDATE_CLAIM`, and `OP_SUPPORT_CLAIM`. In Bitcoin they are respectively `OP_NOP6`, `OP_NOP7`, and `OP_NOP8`. The opcodes are used in output scripts to interact with the claimtrie. Each opcode is followed by one or more parameters. Here's how these opcodes are used:
``` ```
OP_CLAIM_NAME <name> <value> OP_2DROP OP_DROP <outputScript> OP_CLAIM_NAME <name> <value> OP_2DROP OP_DROP <outputScript>
OP_UPDATE_CLAIM <name> <claimId> <value> OP_2DROP OP_2DROP <outputScript> OP_UPDATE_CLAIM <name> <claimID> <value> OP_2DROP OP_2DROP <outputScript>
OP_SUPPORT_CLAIM <name> <claimId> OP_2DROP OP_DROP <outputScript> OP_SUPPORT_CLAIM <name> <claimID> OP_2DROP OP_DROP <outputScript>
``` ```
The `<name>` parameter is the [[name]] that the claim is associated with. `<value>` is the protobuf-encoded claim metadata and optional channel signature (see [Metadata](#metadata) for more about this value). The `<claimId>` is the claim ID of a previous claim that is being updated or supported. The `<name>` parameter is the [[name]] that the claim is associated with. `<value>` is the protobuf-encoded claim metadata and optional channel signature (see [Metadata](#metadata) for more about this value). The `<claimID>` is the claim ID of a previous claim that is being updated or supported.
Each opcode will push a zero on to the execution stack. Those zeros, as well as any additional parameters after the opcodes, are all dropped by `OP_2DROP` and `OP_DROP`. `<outputScript>` can be any valid script, so a script using these opcodes is also a pay-to-pubkey script. This means that claim scripts can be spent just like regular Bitcoin output scripts. Each opcode will push a zero on to the execution stack. Those zeros, as well as any additional parameters after the opcodes, are all dropped by `OP_2DROP` and `OP_DROP`. `<outputScript>` can be any valid script, so a script using these opcodes is also a pay-to-pubkey script. This means that claim scripts can be spent just like regular Bitcoin output scripts.
##### Claim Identifier Generation ##### Claim Identifier Generation
Like any standard Bitcoin output script, a claim script will be associated with a transaction hash and output index. This combination of transaction hash and index is called an _outpoint_. Each claim script has a unique outpoint. The outpoint is hashed using SHA-256 and RIPEMD-160 to generate the claim ID for a claim. For the example above, let's say claim script is included in transaction `7560111513bea7ec38e2ce58a58c1880726b1515497515fd3f470d827669ed43` at the output index `1`. Then the claim ID would be `529357c3422c6046d3fec76be2358004ba22e323`. An implementation of this is available [here](https://github.com/lbryio/lbry.go/blob/master/lbrycrd/blockchain.go). Like any standard Bitcoin output script, a claim script is associated with a transaction hash and output index. This combination of transaction hash and index is called an _outpoint_. Each claim script has a unique outpoint. The outpoint is hashed using SHA-256 and RIPEMD-160 to generate the claim ID for a claim. For the example above, let's say claim script is included in transaction `7560111513bea7ec38e2ce58a58c1880726b1515497515fd3f470d827669ed43` at the output index `1`. Then the claim ID would be `529357c3422c6046d3fec76be2358004ba22e323`. An implementation of this is available [here](https://github.com/lbryio/lbry.go/blob/master/lbrycrd/blockchain.go).
##### OP\_CLAIM\_NAME ##### OP\_CLAIM\_NAME
New claims are created using `OP_CLAIM_NAME`. For example, a claim transaction setting the name `Fruit` to the value `Apple` will look like this: New claims are created using `OP_CLAIM_NAME`. For example, a claim transaction setting the name `Fruit` to the value `Apple` looks like this:
``` ```
OP_CLAIM_NAME Fruit Apple OP_2DROP OP_DROP OP_DUP OP_HASH160 <address> OP_EQUALVERIFY OP_CHECKSIG OP_CLAIM_NAME Fruit Apple OP_2DROP OP_DROP OP_DUP OP_HASH160 <address> OP_EQUALVERIFY OP_CHECKSIG
@ -632,13 +627,13 @@ OP_CLAIM_NAME Fruit Apple OP_2DROP OP_DROP OP_DUP OP_HASH160 <address> OP_EQUALV
##### OP\_UPDATE\_CLAIM ##### OP\_UPDATE\_CLAIM
`OP_UPDATE_CLAIM` updates a claim by replacing its metadata. An update transaction has an added requirement that it must spend the output for the existing claim that it wishes to update. Otherwise, it will be considered invalid and will not make it into the claimtrie. Thus it must have the following redeem script: `OP_UPDATE_CLAIM` updates a claim by replacing its metadata. An update transaction has an added requirement that it must spend the output for the existing claim that it wishes to update. Otherwise, it is considered invalid and will not make it into the claimtrie. Thus it must have the following redeem script:
``` ```
<signature> <pubKeyForPreviousAddress> <signature> <pubKeyForPreviousAddress>
``` ```
The syntax is identical to the standard way of redeeming a pay-to-pubkey script in Bitcoin, with the caveat that `<pubKeyForPreviousAddress>` must be the public key for the address of the output that contains the claim that will be updated. The syntax is identical to the standard way of redeeming a pay-to-pubkey script in Bitcoin, with the caveat that `<pubKeyForPreviousAddress>` must be the public key for the address of the output that contains the claim that is being updated.
To change the value of the previous example claim to “Banana”, the payout script would be To change the value of the previous example claim to “Banana”, the payout script would be
@ -650,7 +645,7 @@ The `<address>` in this script may be the same as the address in the original tr
##### OP\_SUPPORT\_CLAIM ##### OP\_SUPPORT\_CLAIM
A support for the original example claim will have the following payout script: A support for the original example claim has the following payout script:
``` ```
OP_SUPPORT_CLAIM Fruit 529357c3422c6046d3fec76be2358004ba22e323 OP_2DROP OP_DROP OP_DUP OP_HASH160 <address> OP_EQUALVERIFY OP_CHECKSIG OP_SUPPORT_CLAIM Fruit 529357c3422c6046d3fec76be2358004ba22e323 OP_2DROP OP_DROP OP_DUP OP_HASH160 <address> OP_EQUALVERIFY OP_CHECKSIG
@ -711,7 +706,7 @@ The serialized metadata may be cryptographically signed to indicate membership i
### Specification ### Specification
The metadata specification is designed to grow and change frequently. The full specification will not be detailed here. The [types](https://github.com/lbryio/types) repository is considered the precise specification. The metadata specification is designed to grow and change frequently. The full specification is not detailed here. The [types](https://github.com/lbryio/types) repository is considered the precise specification.
Instead, let's look at an example and some key fields. Instead, let's look at an example and some key fields.
@ -778,7 +773,7 @@ The media type of the item as [defined](https://www.iana.org/assignments/media-t
Channels are the unit of identity. A channel is a claim for a name beginning with `@` that contains a metadata structure for identity rather than content. Included in the metadata is the channel's public key. Here's an example: Channels are the unit of identity. A channel is a claim for a name beginning with `@` that contains a metadata structure for identity rather than content. Included in the metadata is the channel's public key. Here's an example:
``` ```
"claim_id": "6e56325c5351ceda2dd0795a30e864492910ccbf", "claimID": "6e56325c5351ceda2dd0795a30e864492910ccbf",
"name": "@lbry", "name": "@lbry",
"amount": 6.26, "amount": 6.26,
"value": { "value": {
@ -833,24 +828,23 @@ format | description
### Validation {#metadata-validation} ### Validation {#metadata-validation}
No enforcement or validation on metadata happens at the blockchain level. Instead, metadata encoding, decoding, and validation is done by clients. This allows evolution of the metadata without changes to consensus rules. No enforcement or validation on metadata happens at the blockchain level. Instead, metadata encoding, decoding, and validation is done by clients. This allows evolution of the metadata without changes to consensus rules. Clients are responsible for validating metadata, including data structure and signatures.
Clients are responsible for validating metadata, including data structure and signatures. This typically happens when the raw binary data stored in the blockchain is decoded client side via Protocol Buffers.
## Data ## Data
Files published using LBRY are stored in a distributed fashion by the clients participating in the network. Each file split into multiple small pieces, encrypted, and announced to the network. It may also be uploaded to other hosts on the network that specialize in rehosting content. Files published using LBRY are stored in a distributed fashion by the clients participating in the network. Each file is split into multiple small pieces. Each piece is encrypted and announced to the network. The pieces may also be uploaded to other hosts on the network that specialize in rehosting content.
The purpose of this process is to enable file storage without relying on centralized infrastructure, and to create a marketplace for data that allows hosts to be paid for their services. The design is strongly influenced by the Bittorrent protocol and network. The purpose of this process is to enable file storage and access without relying on centralized infrastructure, and to create a marketplace for data that allows hosts to be paid for their services. The design is strongly influenced by the Bittorrent protocol.
### Encoding ### Encoding
Content on LBRY is encoded to facilitate distribution. Content on LBRY is encoded to facilitate distribution.
#### Blobs #### Blobs
The smallest unit of data is called a _blob_. A blob is an encrypted chunk of data up to 2MiB in size. Each blob is indexed by its _blob hash_, which is a SHA384 hash of the blob contents. Addressing blobs by their hash protects against naming collisions and ensures that the content you get is what you expect. The smallest unit of data is called a _blob_. A blob is an encrypted chunk of data up to 2MiB in size. Each blob is indexed by its _blob hash_, which is a SHA384 hash of the blob. Addressing blobs by their hash protects against naming collisions and ensures that data cannot be accidentally or maliciously modified.
Blobs are encrypted using AES-256 in CBC mode and PKCS7 padding. In order to keep each encrypted blob at 2MiB max, a blob can hold at most 2097151 bytes (2MiB minus 1 byte) of plaintext data. The source code for the exact algorithm is available [here](https://github.com/lbryio/lbry.go/blob/master/stream/blob.go). The encryption key and IV for each blob is stored as described below. Blobs are encrypted using AES-256 in CBC mode and PKCS7 padding. In order to keep each encrypted blob at 2MiB max, a blob can hold at most 2097151 bytes (2MiB minus 1 byte) of plaintext data. The source code for the exact algorithm is available [here](https://github.com/lbryio/lbry.go/blob/master/stream/blob.go). The encryption key and IV for each blob is stored as described below.
@ -943,17 +937,17 @@ An implementation of this process is available [here](https://github.com/lbryio/
Decoding a stream is like encoding in reverse, and with the added step of verifying that the expected blob hashes match the actual data. Decoding a stream is like encoding in reverse, and with the added step of verifying that the expected blob hashes match the actual data.
1. Verify that the manifest blob hash matches the stream hash you expect. 1. Compute a SHA384 has of the manifest blob and verify that it matches the stream hash.
1. Parse the manifest blob contents. 2. Parse the manifest blob contents.
1. Verify the hashes of the content blobs. 3. Verify the hashes of the content blobs.
1. Decrypt and remove the padding from each content blob using the key and IVs in the manifest. 4. Decrypt and remove the padding from each content blob using the key and IVs in the manifest.
1. Concatenate the decrypted chunks in order. 5. Concatenate the decrypted chunks in order.
### Announce ### Announce
After a [[stream]] is encoded, it must be _announced_ to the network. Announcing is the process of letting other nodes on the network know that you have content available for download. LBRY tracks announced content using a distributed hash table. After a [[stream]] is encoded, it must be _announced_ to the network. Announcing is the process of letting other nodes on the network know that a client has content available for download. LBRY tracks announced content using a distributed hash table.
#### Distributed Hash Table #### Distributed Hash Table
@ -979,7 +973,7 @@ A client wishing to download a [[stream]] must first query the [[DHT]] to find [
#### Querying the DHT #### Querying the DHT
Querying works almost the same way as [[announcing]]. A client looking for a target hash will start by sending iterative `FindValue(target_hash)` requests to the nodes it knows that are closest to the target hash. If a node receives a `FindValue` request and knows of any peers for the target hash, it will respond with a list of those peers. Otherwise, it will respond with the closest nodes to the target hash that it knows about. The client then queries those closer nodes using the same `FindValue` call. This way, each call either finds the client some peers, or brings it closer to finding those peers. If no peers are found and no closer nodes are being returned, the client will determine that the target hash is not available and will give up. Querying works almost the same way as [[announcing]]. A client looking for a target hash starts by sending iterative `FindValue(target_hash)` requests to the nodes it knows that are closest to the target hash. If a node receives a `FindValue` request and knows of any peers for the target hash, it responds with a list of those peers. Otherwise, it responds with the closest nodes to the target hash that it knows about. The client then queries those closer nodes using the same `FindValue` call. This way, each call either finds the client some peers, or brings it closer to finding those peers. If no peers are found and no closer nodes are being returned, the client determines that the target hash is not available and gives up.
#### Blob Exchange Protocol #### Blob Exchange Protocol
@ -996,7 +990,7 @@ DownloadCheck checks whether the server has certain blobs available for download
##### Download ##### Download
Download requests the blob for a given hash. The response contains the blob, its hash, and the address where to send payment for the data transfer. If the blob is not available on the server, the response will instead contain an error. Download requests the blob for a given hash. The response contains the blob, its hash, and the address where to send payment for the data transfer. If the blob is not available on the server, the response instead contains an error.
##### UploadCheck ##### UploadCheck
@ -1016,7 +1010,7 @@ The protocol calls and message types are defined in detail [here](https://github
In order for a client to download content, there must be hosts online that have the content the client wants, when the client wants it. To incentivize the continued hosting of data, the blob exchange protocol supports data upload and payment for data. _Reflectors_ are hosts that accept data uploads. They rehost (reflect) the uploaded data and charge for downloads. Using a reflector is optional, but most publishers will probably choose to use them. Doing so obviates the need for the publisher's server to be online and connectable, which can be especially useful for mobile clients or those behind a firewall. In order for a client to download content, there must be hosts online that have the content the client wants, when the client wants it. To incentivize the continued hosting of data, the blob exchange protocol supports data upload and payment for data. _Reflectors_ are hosts that accept data uploads. They rehost (reflect) the uploaded data and charge for downloads. Using a reflector is optional, but most publishers will probably choose to use them. Doing so obviates the need for the publisher's server to be online and connectable, which can be especially useful for mobile clients or those behind a firewall.
The current version of the protocol does not support sophisticated price negotiation between clients and hosts. The host simply chooses the price it will charge. Clients check this price before downloading, and pay the price after the download is complete. Future protocol versions will include more options for price negotiation, as well as stronger proofs of payment. The current version of the protocol does not support sophisticated price negotiation between clients and hosts. The host simply chooses the price it wants to charge. Clients check this price before downloading, and pay the price after the download is complete. Future protocol versions will include more options for price negotiation, as well as stronger proofs of payment.
<pre style="font: 10px/5px monospace;overflow:hidden;text-align: center;margin: 10rem 0"> <pre style="font: 10px/5px monospace;overflow:hidden;text-align: center;margin: 10rem 0">