1229 lines
57 KiB
HTML
1229 lines
57 KiB
HTML
<!DOCTYPE html>
|
||
<html>
|
||
<head>
|
||
<title>LBRY: A Decentralized Digital Content Marketplace</title>
|
||
<meta name="GENERATOR" content="github.com/mmarkdown/mmark Mmark Markdown Processor - mmark.nl">
|
||
<meta charset="utf-8">
|
||
<link rel="stylesheet" type="text/css" href="normalize.css">
|
||
<link rel="stylesheet" type="text/css" href="tocbot.css">
|
||
<link rel="stylesheet" type="text/css" href="style.css">
|
||
<script src="tocbot.min.js"></script>
|
||
<script>
|
||
function ready(fn) {
|
||
if (document.attachEvent ? document.readyState === "complete" : document.readyState !== "loading"){
|
||
fn();
|
||
} else {
|
||
document.addEventListener('DOMContentLoaded', fn);
|
||
}
|
||
}
|
||
|
||
function toggleClass(el, className) {
|
||
if (el.classList) {
|
||
el.classList.toggle(className);
|
||
} else {
|
||
var classes = el.className.split(' ');
|
||
var existingIndex = classes.indexOf(className);
|
||
|
||
if (existingIndex >= 0)
|
||
classes.splice(existingIndex, 1);
|
||
else
|
||
classes.push(className);
|
||
|
||
el.className = classes.join(' ');
|
||
}
|
||
}
|
||
|
||
ready(function() {
|
||
var options = {
|
||
tocSelector: '.toc',
|
||
contentSelector: '#content',
|
||
headingSelector: 'h2, h3, h4, h5, h6',
|
||
collapseDepth: 3,
|
||
positionFixedSelector: ".toc",
|
||
};
|
||
|
||
tocbot.init(options);
|
||
|
||
document.querySelector('.toc-menu').addEventListener("click", function() {
|
||
toggleClass(document.querySelector('.toc'), "open");
|
||
});
|
||
|
||
|
||
// need this part to fix toc offset height if content above toc changes height
|
||
var resizeTimer;
|
||
window.addEventListener('resize', function(e) {
|
||
clearTimeout(resizeTimer);
|
||
resizeTimer = setTimeout(function() {
|
||
o = options;
|
||
o.fixedSidebarOffset = "auto";
|
||
tocbot.refresh(o);
|
||
}, 250);
|
||
});
|
||
|
||
|
||
})
|
||
</script></head>
|
||
<body>
|
||
|
||
<p><main></p>
|
||
|
||
<h1 id="lbry-a-decentralized-digital-content-marketplace">LBRY: A Decentralized Digital Content Marketplace</h1>
|
||
|
||
<aside>
|
||
<p>Please excuse the unfinished state of this paper. It is being actively worked on. The content here is made available early because it contains useful information for developers.</p>
|
||
|
||
<p>For more technical information about LBRY, visit <a href="https://lbry.tech">lbry.tech</a>.</p>
|
||
</aside>
|
||
|
||
<p><div class="toc-menu">Menu</div>
|
||
<nav class="toc"></nav>
|
||
<div id="content"></p>
|
||
|
||
<noscript>
|
||
|
||
## Table of Contents
|
||
|
||
<!-- this TOC is only here for github or js-challenged browsers -->
|
||
|
||
<!--ts-->
|
||
* [Introduction](#introduction)
|
||
* [Overview](#overview)
|
||
* [Conventions and Terminology](#conventions-and-terminology)
|
||
* [Blockchain](#blockchain)
|
||
* [Claims](#claims)
|
||
* [Properties {#claim-properties}](#properties-claim-properties)
|
||
* [Example Claim](#example-claim)
|
||
* [Operations {#claim-operations}](#operations-claim-operations)
|
||
* [Supports](#supports)
|
||
* [Claimtrie](#claimtrie)
|
||
* [Statuses {#claim-statuses}](#statuses-claim-statuses)
|
||
* [Accepted](#accepted)
|
||
* [Abandoned](#abandoned)
|
||
* [Active](#active)
|
||
* [Controlling](#controlling)
|
||
* [Claimtrie Transitions](#claimtrie-transitions)
|
||
* [Determining Active Claims](#determining-active-claims)
|
||
* [Claim Transition Example](#claim-transition-example)
|
||
* [Normalization](#normalization)
|
||
* [URLs](#urls)
|
||
* [Components](#components)
|
||
* [Stream Claim Name](#stream-claim-name)
|
||
* [Channel Claim Name](#channel-claim-name)
|
||
* [Channel Claim Name and Stream Claim Name](#channel-claim-name-and-stream-claim-name)
|
||
* [Claim ID](#claim-id)
|
||
* [Claim Sequence](#claim-sequence)
|
||
* [Bid Position](#bid-position)
|
||
* [Query Params](#query-params)
|
||
* [Grammar](#grammar)
|
||
* [Resolution](#resolution)
|
||
* [No Modifier](#no-modifier)
|
||
* [Claim ID](#claim-id-1)
|
||
* [Claim Sequence](#claim-sequence-1)
|
||
* [Bid Position](#bid-position-1)
|
||
* [ChannelName and ClaimName](#channelname-and-claimname)
|
||
* [Examples {#url-resolution-examples}](#examples-url-resolution-examples)
|
||
* [Design Notes](#design-notes)
|
||
* [Transactions](#transactions)
|
||
* [Operations and Opcodes](#operations-and-opcodes)
|
||
* [Addresses](#addresses)
|
||
* [Proof of Payment](#proof-of-payment)
|
||
* [Consensus](#consensus)
|
||
* [Block Timing](#block-timing)
|
||
* [Difficulty Adjustment](#difficulty-adjustment)
|
||
* [Block Hash Algorithm](#block-hash-algorithm)
|
||
* [Block Rewards](#block-rewards)
|
||
* [Metadata](#metadata)
|
||
* [Specification](#specification)
|
||
* [Example {#metadata-example}](#example-metadata-example)
|
||
* [Key Fields](#key-fields)
|
||
* [Source and Stream Hashes](#source-and-stream-hashes)
|
||
* [Fees and Fee Structure](#fees-and-fee-structure)
|
||
* [Title](#title)
|
||
* [Thumbnail](#thumbnail)
|
||
* [Content Type](#content-type)
|
||
* [Certificate](#certificate)
|
||
* [Channels (Identities) {#channels}](#channels-identities-channels)
|
||
* [Example Channel Metadata](#example-channel-metadata)
|
||
* [Validation {#metadata-validation}](#validation-metadata-validation)
|
||
* [Data](#data)
|
||
* [Encoding](#encoding)
|
||
* [Blobs](#blobs)
|
||
* [Streams](#streams)
|
||
* [Manifest Contents](#manifest-contents)
|
||
* [Stream Encoding](#stream-encoding)
|
||
* [Setup](#setup)
|
||
* [Content Blobs](#content-blobs)
|
||
* [Manifest Blob](#manifest-blob)
|
||
* [Stream Decoding](#stream-decoding)
|
||
* [Announce](#announce)
|
||
* [Distributed Hash Table](#distributed-hash-table)
|
||
* [Announcing to the DHT](#announcing-to-the-dht)
|
||
* [Download](#download)
|
||
* [Querying the DHT](#querying-the-dht)
|
||
* [Blob Exchange Protocol](#blob-exchange-protocol)
|
||
* [PriceCheck](#pricecheck)
|
||
* [DownloadCheck](#downloadcheck)
|
||
* [Download](#download-1)
|
||
* [UploadCheck](#uploadcheck)
|
||
* [Upload](#upload)
|
||
* [Reflectors and Data Markets](#reflectors-and-data-markets)
|
||
<!--te-->
|
||
|
||
</noscript>
|
||
|
||
<h2 id="introduction">Introduction</h2>
|
||
|
||
<p>LBRY is a protocol for accessing and publishing digital content in a global, decentralized marketplace. Clients can use LBRY to publish, host, find, download, and pay for content — books, movies, music, or anything else. Anyone can participate and no permission is required, nor can anyone be blocked from participating. The system is distributed, so no single entity has unilateral control, nor will the removal of any single entity prevent the system from functioning.</p>
|
||
|
||
<p>TODO:</p>
|
||
|
||
<ul>
|
||
<li>why is it significant</li>
|
||
<li>whom does it help</li>
|
||
<li>why is it different/better than what existed before</li>
|
||
</ul>
|
||
|
||
<h2 id="overview">Overview</h2>
|
||
|
||
<p>This document defines the LBRY protocol, its components, and how they fit together. At its core, LBRY consists of several discrete components that are used together in order to provide the end-to-end capabilities of the protocol. There are two distributed data stores (blockchain and DHT), a peer-to-peer protocol for exchanging data, and several specifications for data structure, transformation, and retrieval.</p>
|
||
|
||
<p>This document assumes that the reader is familiar with Bitcoin and blockchain technology. It does not attempt to document the Bitcoin protocol or explain how it works. The <a href="https://bitcoin.org/en/developer-reference">Bitcoin developer reference</a> is recommended for anyone wishing to understand the technical details.</p>
|
||
|
||
<h2 id="conventions-and-terminology">Conventions and Terminology</h2>
|
||
|
||
<p>(Rather than this section, maybe we can use a syntax like brackets around keywords to inline key definitions?)</p>
|
||
|
||
<dl>
|
||
<dt>file</dt>
|
||
<dd>A single piece of content published using LBRY.</dd>
|
||
|
||
<dt>blob</dt>
|
||
<dd>The unit of data transmission on the data network. A published file is split into many blobs.</dd>
|
||
|
||
<dt>stream</dt>
|
||
<dd>A set of blobs that can be reassembled into a file. Every stream has a descriptor blob and one or more content blobs.</dd>
|
||
|
||
<dt>blob hash</dt>
|
||
<dd>The output of a cryptographic hash function is applied to a blob. Hashes are used to uniquely identify blobs and to verify that the contents of the blob are correct. Unless otherwise specified, LBRY uses SHA384 as the hash function.</dd>
|
||
|
||
<dt>metadata</dt>
|
||
<dd>Information about the contents of a stream (e.g. creator, description, stream descriptor hash, etc). Metadata is stored in the blockchain.</dd>
|
||
|
||
<dt>claim</dt>
|
||
<dd>A single metadata entry in the blockchain.</dd>
|
||
|
||
<dt>name</dt>
|
||
<dd>A human-readable UTF8 string that is associated with a published claim.</dd>
|
||
|
||
<dt>channel</dt>
|
||
<dd>The unit of pseudonymous publisher identity. Claims may be part of a channel.</dd>
|
||
|
||
<dt>URL</dt>
|
||
<dd>A reference to a claim that specifies how to retrieve it.</dd>
|
||
</dl>
|
||
|
||
<h2 id="blockchain">Blockchain</h2>
|
||
|
||
<!-- done -->
|
||
|
||
<p>The LBRY blockchain is a public, proof-of-work blockchain. It serves three key purposes:</p>
|
||
|
||
<ol>
|
||
<li>An index of the content available on the network</li>
|
||
<li>A payment system and record of purchases for priced content</li>
|
||
<li>Trustful publisher identities</li>
|
||
</ol>
|
||
|
||
<p>The LBRY blockchain is a fork of the <a href="https://bitcoin.org/bitcoin.pdf">Bitcoin</a> blockchain, with substantial modifications. This document will not cover or specify any aspects of LBRY that are identical to Bitcoin, and will instead focus on the differences.</p>
|
||
|
||
<h3 id="claims">Claims</h3>
|
||
|
||
<!-- done -->
|
||
|
||
<p>A <em>claim</em> is a single entry in the blockchain that stores metadata. There are two types of claims:</p>
|
||
|
||
<dl>
|
||
<dt>stream</dt>
|
||
<dd>Declares the availability, access method, and publisher of a stream of bytes (typically a file).</dd>
|
||
<dt>channel</dt>
|
||
<dd>Creates a pseudonym that can be declared as the publisher of a set of stream claims.</dd>
|
||
</dl>
|
||
|
||
<h4 id="claim-properties">Properties</h4>
|
||
|
||
<p>Claims have four properties:</p>
|
||
|
||
<dl>
|
||
<dt>claimId</dt>
|
||
<dd>A 20-byte hash unique among all claims. See [Claim Identifier Generation](#claim-identifier-generation).</dd>
|
||
<dt>name</dt>
|
||
<dd>A normalized UTF-8 string of up to 255 bytes used to address the claim. See [URLs](#urls) and [Normalization](#normalization).</dd>
|
||
<dt>amount</dt>
|
||
<dd>A quantity of tokens used to stake the claim. See [Controlling](#controlling).</dd>
|
||
<dt>value</dt>
|
||
<dd>Metadata about a stream or a channel. See [Metadata](#metadata).</dd>
|
||
</dl>
|
||
|
||
|
||
<h4 id="example-claim">Example Claim</h4>
|
||
|
||
<!-- done -->
|
||
|
||
<p>Here is an example stream claim:</p>
|
||
|
||
<pre><code>{
|
||
"claimId": "fa3d002b67c4ff439463fcc0d4c80758e38a0aed",
|
||
"name": "lbry",
|
||
"amount": 100000000,
|
||
"value": {
|
||
"title": "What is LBRY?",
|
||
"description": "What is LBRY? An introduction with Alex Tabarrok",
|
||
"license": "LBRY inc",
|
||
"author": "Samuel Bryan",
|
||
"language": "en",
|
||
"content_type": "video/mp4",
|
||
"thumbnail": "https://s3.amazonaws.com/files.lbry.io/logo.png",
|
||
"sources": {
|
||
"lbry_sd_hash": "e1e324bce7437540fac6707fa142cca44d76fc4e8e65060139a88ff7cdb218b4540cb9cff8bb3d5e06157ae6b08e5cb5"
|
||
}
|
||
},
|
||
"txid": "53ed05d9dfd728a94bedf952d67783bbe9da5d2ab436a84338bb53f0b85301b5",
|
||
"n": 0,
|
||
"height": 146117
|
||
}
|
||
</code></pre>
|
||
|
||
<h4 id="claim-operations">Operations</h4>
|
||
|
||
<!-- done -->
|
||
|
||
<p>There are three claim operations: <em>create</em>, <em>update</em>, and <em>abandon</em>.</p>
|
||
|
||
<dl>
|
||
<dt>create</dt>
|
||
<dd>Makes a new claim.</dd>
|
||
<dt>update</dt>
|
||
<dd>Changes the value or amount of an existing claim, without changing the claim ID.</dd>
|
||
<dt>abandon</dt>
|
||
<dd>Withdraws a claim, freeing the associated credits to be used for other purposes.</dd>
|
||
</dl>
|
||
|
||
<h4 id="supports">Supports</h4>
|
||
|
||
<p>A <em>support</em> is an additional transaction type that lends its <em>amount</em> to an existing claim.</p>
|
||
|
||
<p>A support contains only a <code>claimID</code> and an <code>amount</code>, no other properties. Supports function analogously to claims in terms of <a href="#claim-operations">Claim Operations</a> and <a href="#claim-statuses">Claim Statuses</a>, with the exception that they cannot be updated.</p>
|
||
|
||
<h4 id="claimtrie">Claimtrie</h4>
|
||
|
||
<!-- done -->
|
||
|
||
<p>The <em>claimtrie</em> is the data structure used to store the set of all claims and prove the correctness of claim resolution.</p>
|
||
|
||
<p>The claimtrie is implemented as a <a href="https://en.wikipedia.org/wiki/Merkle_tree">Merkle tree</a> that maps names to claims. Claims are stored as leaf nodes in the tree. Names are stored as the path from the root node to the leaf node.</p>
|
||
|
||
<p>The <em>root hash</em> is the hash of the root node. It is stored in the header of each block in the blockchain. Nodes in the LBRY network use the root hash to efficiently and securely validate the state of the claimtrie.</p>
|
||
|
||
<p>Multiple claims can exist for the same name. They are all stored in the leaf node for that name, sorted in decreasing order by the amount of credits backing each claim.</p>
|
||
|
||
<!-- fix me above? "amount of credits backing each claim" is the effective amount, but that's not defined till later -->
|
||
|
||
<p>For more details on the specific claimtrie implementation, see <a href="https://github.com/lbryio/lbrycrd/blob/master/src/claimtrie.cpp">the source code</a>.</p>
|
||
|
||
<h4 id="claim-statuses">Statuses</h4>
|
||
|
||
<!-- fix me? is using claims to mean claims and supports okay? -->
|
||
|
||
<p>Throughout this section we use the word “claim” to refer to both claims and supports, unless otherwise specified.</p>
|
||
|
||
<p>Claims and supports can have one or more of the following statuses at a given block.</p>
|
||
|
||
<h5 id="accepted">Accepted</h5>
|
||
|
||
<!-- done -->
|
||
|
||
<p>An <em>accepted</em> claim is one that has been been entered into the blockchain. This happens when the transaction containing it is included in a block.</p>
|
||
|
||
<p>Accepted claims do not affect the claimtrie leaf order until they are <a href="#active">active</a>.</p>
|
||
|
||
<p>The sum of the amount of a claim and all accepted supports is called the <em>total amount</em>.</p>
|
||
|
||
<h5 id="abandoned">Abandoned</h5>
|
||
|
||
<!-- done -->
|
||
|
||
<p>An <em>abandoned</em> claim is one that was withdrawn by its creator or current owner. Spending a transaction that contains a claim will cause that claim to become abandoned. Abandoned claims are removed from the claimtrie.</p>
|
||
|
||
<p>While data related to abandoned claims technically still resides in the blockchain, it is improper to use this data to fetch the associated content, and active claims signed by abandoned identities will no longer be reported as valid.</p>
|
||
|
||
<h5 id="active">Active</h5>
|
||
|
||
<!-- fix me a lot -->
|
||
|
||
<p>An <em>active</em> claim is an accepted and non-abandoned claim that has been in the blockchain for an algorithmically determined number of blocks. This length of time required is called the <em>activation delay</em>.</p>
|
||
|
||
<p>If the claim is an update to an already active claim, is the first claim for a name, or does not affect the sort order at the leaf for a name, the activation delay is 0 (i.e. the claim becomes active in the same block it is accepted).</p>
|
||
|
||
<p>Otherwise, the activation delay is determined by a formula covered in <a href="#claimtrie-transitions">Claimtrie Transitions</a>. The formula’s inputs are the height of the current block, the height at which the claim was accepted, and the height at which the relevant claimtrie state for the name being considered last changed.</p>
|
||
|
||
<p>The sum of the amount of an active claim and all active supports is called it’s <em>effective amount</em>. Only the effective amount affects the sort order of claims in a claimtrie leaf.</p>
|
||
|
||
<h5 id="controlling">Controlling</h5>
|
||
|
||
<!-- done -->
|
||
|
||
<p>A <em>controlling</em> claim is the active claim that is first in the sort order at a leaf. That is, it has the highest effective amount of all claims with the same name.</p>
|
||
|
||
<p>Only one claim can be controlling for a given name at a given block.</p>
|
||
|
||
<h4 id="claimtrie-transitions">Claimtrie Transitions</h4>
|
||
|
||
<!-- fix me. it doesnt make sense for non-first claim takeovers to depend on top claim. also be clear that amount is effective amount -->
|
||
|
||
<p>To determine the sort order of claims in a claimtrie leaf, the following algorithm is used:</p>
|
||
|
||
<ol>
|
||
<li><p>For each active claim, add up the amount of the claim and the amount of all the active supports for that claim.</p></li>
|
||
|
||
<li><p>If all of the claims are in the same order (appending new claims allowed), then nothing is changing.</p></li>
|
||
|
||
<li><p>Otherwise, a takeover is occurring. Set the takeover height for this name to the current height, recalculate which claims and supports are now active, and return to step 1.</p></li>
|
||
|
||
<li><p>At this point, the claim with the greatest effective amount is the controlling claim at this block.</p></li>
|
||
</ol>
|
||
|
||
<p>The purpose of 3 is to handle the case when multiple competing claims are made on the same name in different blocks, and one of those claims becomes active but another still-inactive claim has the greatest amount. Step 3 will cause the greater claim to also activate and become the controlling claim.</p>
|
||
|
||
<h5 id="determining-active-claims">Determining Active Claims</h5>
|
||
|
||
<p>If a claim does not become active immediately, it becomes active at the block heigh determined by the following formula:</p>
|
||
|
||
<pre><code>C + min(4032, floor((H-T) / 32))
|
||
</code></pre>
|
||
|
||
<p>Where:</p>
|
||
|
||
<ul>
|
||
<li>C = claim height (height when the claim was accepted)</li>
|
||
<li>H = current height</li>
|
||
<li>T = takeover height (the most recent height at which the relevant claimtrie state for the name changed)</li>
|
||
</ul>
|
||
|
||
<!-- fixme: relevant claimtrie state??? -->
|
||
|
||
<p>In written form, the delay before a claim becomes active is equal to the claim’s height minus height of the last takeover, divided by 32. The delay is capped at 4032 blocks, which is 7 days of blocks at 2.5 minutes per block (our target block time). The max delay is reached 224 (7x32) days after the last takeover.</p>
|
||
|
||
<p>The purpose of this delay function is to give long-standing claimants time to respond to changes, while still keeping takeover times reasonable and allowing recent or contentious claims to change state quickly.</p>
|
||
|
||
<h6 id="claim-transition-example">Claim Transition Example</h6>
|
||
|
||
<!-- done -->
|
||
|
||
<p>Here is a step-by-step example to illustrate the different scenarios. All claims are for the same name.</p>
|
||
|
||
<p><strong>Block 13:</strong> Claim A for 10LBC is accepted. It is the first claim, so it immediately becomes active and controlling.
|
||
<br>State: A(10) is controlling</p>
|
||
|
||
<p><strong>Block 1001:</strong> Claim B for 20LBC is accepted. It’s activation height is <code>1001 + min(4032, floor((1001-13) / 32)) = 1001 + 30 = 1031</code>.
|
||
<br>State: A(10) is controlling, B(20) is accepted.</p>
|
||
|
||
<p><strong>Block 1010:</strong> Support X for 14LBC for claim A is accepted. Since it is a support for the controlling claim, it activates immediately.
|
||
<br>State: A(10+14) is controlling, B(20) is accepted.</p>
|
||
|
||
<p><strong>Block 1020:</strong> Claim C for 50LBC is accepted. The activation height is <code>1020 + min(4032, floor((1020-13) / 32)) = 1020 + 31 = 1051</code>.
|
||
<br>State: A(10+14) is controlling, B(20) is accepted, C(50) is accepted.</p>
|
||
|
||
<p><strong>Block 1031:</strong> Claim B activates. It has 20LBC, while claim A has 24LBC (10 original + 14 from support X). There is no takeover, and claim A remains controlling.
|
||
<br>State: A(10+14) is controlling, B(20) is active, C(50) is accepted.</p>
|
||
|
||
<p><strong>Block 1040:</strong> Claim D for 300LBC is accepted. The activation height is <code>1040 + min(4032, floor((1040-13) / 32)) = 1040 + 32 = 1072</code>.
|
||
<br>State: A(10+14) is controlling, B(20) is active, C(50) is accepted, D(300) is accepted.</p>
|
||
|
||
<p><strong>Block 1051:</strong> Claim C activates. It has 50LBC, while claim A has 24LBC, so a takeover is initiated. The takeover height for this name is set to 1051, and therefore the activation delay for all the claims becomes <code>min(4032, floor((1051-1051) / 32)) = 0</code>. All the claims become active. The totals for each claim are recalculated, and claim D becomes controlling because it has the highest total.
|
||
<br>State: A(10+14) is active, B(20) is active, C(50) is active, D(300) is controlling.</p>
|
||
|
||
<h4 id="normalization">Normalization</h4>
|
||
|
||
<!-- done -->
|
||
|
||
<p>Names in the claimtrie are normalized to avoid confusion due to Unicode equivalence or casing. All names are converted using <a href="http://unicode.org/reports/tr15/#Norm_Forms">Unicode Normalization Form D</a> (NFD), then lowercased using the en_US locale when possible.</p>
|
||
|
||
<h3 id="urls">URLs</h3>
|
||
|
||
<!-- fix me:
|
||
jeremy: @grin does SPV need a mention inside of the document?
|
||
grin: no, but we should probably include an example for how to do the validation using the root hash. its not strictly necessary because its similar to how bitcoin does it. so maybe link to https://lbry.tech/resources/claimtrie (which needs an update) and add a validation example there?
|
||
-->
|
||
|
||
<p>URLs are human-readable references to claims. All URLs:</p>
|
||
|
||
<ol>
|
||
<li>must contain a name (see <a href="#claim-properties">Claim Properties</a>)</li>
|
||
<li>and resolve to a single, specific claim for that name</li>
|
||
</ol>
|
||
|
||
<p>The ultimate purpose of much of the claim and blockchain design is to provide human-readable URLs that can be provably resolved by clients without a full copy of the blockchain (i.e. <a href="https://lbry.tech/glossary#spv">Simplified Payment Verification</a> wallets).</p>
|
||
|
||
<p>It is possible to write extremely short, human-readable and memorabl</p>
|
||
|
||
<h4 id="components">Components</h4>
|
||
|
||
<!-- done -->
|
||
|
||
<p>A URL is a name with one or more modifiers. A bare name on its own will resolve to the <a href="#controlling">controlling claim</a> at the latest block height. Common URL structures are:</p>
|
||
|
||
<h5 id="stream-claim-name">Stream Claim Name</h5>
|
||
|
||
<p>A basic claim for a name.</p>
|
||
|
||
<pre><code>lbry://meet-lbry
|
||
</code></pre>
|
||
|
||
<h5 id="channel-claim-name">Channel Claim Name</h5>
|
||
|
||
<p>A basic claim for a channel.</p>
|
||
|
||
<pre><code>lbry://@lbry
|
||
</code></pre>
|
||
|
||
<h5 id="channel-claim-name-and-stream-claim-name">Channel Claim Name and Stream Claim Name</h5>
|
||
|
||
<p>A URL containing both a channel and a stream claim name. URLs containing both are resolved in two steps. First, the channel is resolved to it’s associated claim. Then the stream claim name is resolved to get the appropriate claim from among the claims in the channel.</p>
|
||
|
||
<pre><code>lbry://@lbry/meet-lbry
|
||
</code></pre>
|
||
|
||
<h5 id="claim-id">Claim ID</h5>
|
||
|
||
<p>A claim for this name with this claim ID. Partial prefix matches are allowed (see <a href="#resolution">Resolution</a>).</p>
|
||
|
||
<pre><code>lbry://meet-lbry#7a0aa95c5023c21c098
|
||
lbry://meet-lbry#7a
|
||
lbry://@lbry#3f/meet-lbry
|
||
</code></pre>
|
||
|
||
<h5 id="claim-sequence">Claim Sequence</h5>
|
||
|
||
<p>The <em>n</em>th claim for this name, in the order the claims entered the blockchain. <em>n</em> must be a positive number. This can be used to resolve claims in the order in which they were recorded, rather than by which claim has the most support.</p>
|
||
|
||
<pre><code>lbry://meet-lbry:1
|
||
lbry://@lbry:1/meet-lbry
|
||
</code></pre>
|
||
|
||
<h5 id="bid-position">Bid Position</h5>
|
||
|
||
<p>The <em>n</em>th claim for this name, in order of most support to least total support. <em>n</em> must be a positive number. This is useful for resolving non-winning bids in bid order.</p>
|
||
|
||
<pre><code>lbry://meet-lbry$2
|
||
lbry://meet-lbry$3
|
||
lbry://@lbry$2/meet-lbry
|
||
</code></pre>
|
||
|
||
<h5 id="query-params">Query Params</h5>
|
||
|
||
<p>These parameters are reserved for future use.</p>
|
||
|
||
<pre><code>lbry://meet-lbry?arg=value+arg2=value2
|
||
</code></pre>
|
||
|
||
<h4 id="grammar">Grammar</h4>
|
||
|
||
<p>The full URL grammar is defined using <a href="https://www.w3.org/TR/2017/REC-xquery-31-20170321/#EBNFNotation">Xquery EBNF notation</a>:</p>
|
||
|
||
<!-- use http://bottlecaps.de/rr/ui for visuals -->
|
||
|
||
<pre><code>URL ::= Scheme Path Query?
|
||
|
||
Scheme ::= 'lbry://'
|
||
|
||
Path ::= StreamClaimNameAndModifier | ChannelClaimNameAndModifier ( '/' StreamClaimNameAndModifier )?
|
||
|
||
StreamClaimNameAndModifier ::= StreamClaimName Modifier?
|
||
ChannelClaimNameAndModifier ::= ChannelClaimName Modifier?
|
||
|
||
StreamClaimName ::= NameChar+
|
||
ChannelClaimName ::= '@' NameChar+
|
||
|
||
Modifier ::= ClaimID | ClaimSequence | BidPosition
|
||
ClaimID ::= '#' Hex+
|
||
ClaimSequence ::= ':' PositiveNumber
|
||
BidPosition ::= '$' PositiveNumber
|
||
|
||
Query ::= '?' QueryParameterList
|
||
QueryParameterList ::= QueryParameter ( '&' QueryParameterList )*
|
||
QueryParameter ::= QueryParameterName ( '=' QueryParameterValue )?
|
||
QueryParameterName ::= NameChar+
|
||
QueryParameterValue ::= NameChar+
|
||
|
||
PositiveDigit ::= [123456789]
|
||
Digit ::= '0' | PositiveDigit
|
||
PositiveNumber ::= PositiveDigit Digit*
|
||
|
||
HexAlpha ::= [abcdef]
|
||
Hex ::= (Digit | HexAlpha)+
|
||
|
||
NameChar ::= Char - [=&#:$@?/] /* any character that is not reserved */
|
||
Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */
|
||
</code></pre>
|
||
|
||
<h4 id="resolution">Resolution</h4>
|
||
|
||
<p>URL <em>resolution</em> is the process of translating a URL into it’s associated claim id and metadata.</p>
|
||
|
||
<h5 id="no-modifier">No Modifier</h5>
|
||
|
||
<p>Return the controlling claim for the name. Stream claims and channel claims are resolved the same way.</p>
|
||
|
||
<h5 id="claim-id-1">Claim ID</h5>
|
||
|
||
<p>Get all claims for the claim name whose IDs start with the given <code>ClaimID</code>. Sort the claims in ascending order by block height and position within the block. Return the first claim.</p>
|
||
|
||
<h5 id="claim-sequence-1">Claim Sequence</h5>
|
||
|
||
<p>Get all claims for the claim name. Sort the claims in ascending order by block height and position within the block. Return the Nth claim, where N is the given <code>ClaimSequence</code> value.</p>
|
||
|
||
<h5 id="bid-position-1">Bid Position</h5>
|
||
|
||
<p>Get all claims for the claim name. Sort the claims in descending order by total effective amount. Return the Nth claim, where N is the given <code>BidSequence</code> value.</p>
|
||
|
||
<h5 id="channelname-and-claimname">ChannelName and ClaimName</h5>
|
||
|
||
<!-- fix me: explain how claim signing works, and what it means to be **in** a channel -->
|
||
|
||
<p>If both a channel name and a claim name is present, resolution happens in two steps. First, remove the <code>/</code> and <code>StreamClaimNameAndModifier</code> from the path, and resolve the URL as if it only had a <code>ChannelClaimNameAndModifier</code>. Then get the list of all claims in that channel. Finally, resolve the <code>StreamClaimNameAndModifier</code> as if it was its own URL, but instead of considering all claims, only consider the set of claims in the channel.</p>
|
||
|
||
<p>Technically, if multiple claims for the same name exist inside the same channel, they are resolved via the same resolution rules applied entirely within the sub-scope of the channel. Pragmatically, it rarely makes sense for channels to use the same name twice and support for this functionality may be unreliable in current tooling.</p>
|
||
|
||
<h5 id="url-resolution-examples">Examples</h5>
|
||
|
||
<p>Suppose the following names were claimed in the following order:</p>
|
||
|
||
<table>
|
||
<thead>
|
||
<tr>
|
||
<th align="left">Name</th>
|
||
<th align="left">Claim ID</th>
|
||
<th align="left">Amount</th>
|
||
</tr>
|
||
</thead>
|
||
|
||
<tbody>
|
||
<tr>
|
||
<td align="left">apple</td>
|
||
<td align="left">690eea</td>
|
||
<td align="left">1</td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td align="left">banana</td>
|
||
<td align="left">714a3f</td>
|
||
<td align="left">2</td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td align="left">cherry</td>
|
||
<td align="left">bfaabb</td>
|
||
<td align="left">100</td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td align="left">apple</td>
|
||
<td align="left">690eea</td>
|
||
<td align="left">10</td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td align="left">@Arthur</td>
|
||
<td align="left">b7bab5</td>
|
||
<td align="left">1</td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td align="left">@Bryan</td>
|
||
<td align="left">0da517</td>
|
||
<td align="left">1</td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td align="left">@Chris</td>
|
||
<td align="left">b3f7b1</td>
|
||
<td align="left">1</td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td align="left">@Chris/banana</td>
|
||
<td align="left">fc861c</td>
|
||
<td align="left">1</td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td align="left">@Arthur/apple</td>
|
||
<td align="left">37ee1</td>
|
||
<td align="left">20</td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td align="left">@Bryan/cherry</td>
|
||
<td align="left">a18bca</td>
|
||
<td align="left">10</td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td align="left">@Chris</td>
|
||
<td align="left">005a7d</td>
|
||
<td align="left">100</td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td align="left">@Arthur/cherry</td>
|
||
<td align="left">d39aa0</td>
|
||
<td align="left">20</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p>Here is how the following URLs should resolve:</p>
|
||
|
||
<table>
|
||
<thead>
|
||
<tr>
|
||
<th align="left">URL</th>
|
||
<th align="left">Claim ID</th>
|
||
<th align="left">Note</th>
|
||
</tr>
|
||
</thead>
|
||
|
||
<tbody>
|
||
<tr>
|
||
<td align="left"><code>lbry://apple</code></td>
|
||
<td align="left">a37ee1</td>
|
||
<td align="left"></td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td align="left"><code>lbry://banana</code></td>
|
||
<td align="left">714a3f</td>
|
||
<td align="left"></td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td align="left"><code>lbry://@Chris</code></td>
|
||
<td align="left">005a7d</td>
|
||
<td align="left"></td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td align="left"><code>lbry://@Chris/banana</code></td>
|
||
<td align="left"><em>not found</em></td>
|
||
<td align="left">the controlling <code>@Chris</code> does not have a <code>banana</code></td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td align="left"><code>lbry://@Chris:1/banana</code></td>
|
||
<td align="left">fc861c</td>
|
||
<td align="left"></td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td align="left"><code>lbry://@Chris:#fc8/banana</code></td>
|
||
<td align="left">fc861c</td>
|
||
<td align="left"></td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td align="left"><code>lbry://cherry</code></td>
|
||
<td align="left">bfaabb</td>
|
||
<td align="left"></td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td align="left"><code>lbry://@Arthur/cherry</code></td>
|
||
<td align="left">d39aa0</td>
|
||
<td align="left"></td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td align="left"><code>lbry://@Bryan</code></td>
|
||
<td align="left">0da517</td>
|
||
<td align="left"></td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td align="left"><code>lbry://banana$1</code></td>
|
||
<td align="left">714a3f</td>
|
||
<td align="left"></td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td align="left"><code>lbry://banana$2</code></td>
|
||
<td align="left">fc861c</td>
|
||
<td align="left"></td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td align="left"><code>lbry://banana$3</code></td>
|
||
<td align="left"><em>not found</em></td>
|
||
<td align="left"></td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td align="left"><code>lbry://@Arthur:1</code></td>
|
||
<td align="left">b7bab5</td>
|
||
<td align="left"></td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
|
||
<h4 id="design-notes">Design Notes</h4>
|
||
|
||
<p>The most contentious aspect of this design has been the choice to resolve naked names (sometimes called <em>vanity names</em>) to the claim with the largest number of staked credits.</p>
|
||
|
||
<p>First, it is important to note the problems in existing domain allocation design. Most existing public name schemes are first-come, first-serve with a fixed price. This leads to several bad outcomes:</p>
|
||
|
||
<ol>
|
||
<li><p>Speculation and extortion. Entrepreneurs are incentivized to register common names even if they don’t intend to use them, in hopes of selling them to the proper owner in the future for an exorbitant price. While speculation in general can have positive externalities (stable prices and price signals), in this case it is pure value extraction. Speculation also harms the user experience, who will see the vast majority of URLs sitting unused (c.f. Namecoin).</p></li>
|
||
|
||
<li><p>Bureaucracy and transaction costs. While a centralized system can allow for an authority to use a process to reassign names based on trademark or other common use reasons, this system is also imperfect. Most importantly, it is a censorship point and an avenue for complete exclusion. Additionally, such processes are often arbitrary, change over time, involve significant transaction costs, and <em>still</em> lead to names being used in ways that are contrary to user expectation (e.g. <a href="http://nissan.com">nissan.com</a>).</p></li>
|
||
|
||
<li><p>Inefficiencies from price controls. Any system that does not allow a price to float freely creates inefficiencies. If the set price is too low, we facilitate speculation and rent-seeking. If the price is too high, we see people excluded from a good that it would otherwise be beneficial for them to purchase.</p></li>
|
||
</ol>
|
||
|
||
<p>We sought an algorithmic design built into consensus that would allow URLs to flow to their highest valued use. Following <a href="https://en.wikipedia.org/wiki/Coase_theorem">Coase</a>, this staking design allows for clearly defined rules, low transaction costs, and no information asymmetry, minimizing inefficiency in URL allocation.</p>
|
||
|
||
<p>Finally, it’s important to note that <em>only</em> vanity URLs have this property. Extremely short, memorable URLs like <code>lbry://myclaimname#a</code> exist and are available for the minimal cost of issuing a transaction.</p>
|
||
|
||
<h3 id="transactions">Transactions</h3>
|
||
|
||
<p>To support claims, the LBRY blockchain makes the following changes on top of Bitcoin.</p>
|
||
|
||
<h4 id="operations-and-opcodes">Operations and Opcodes</h4>
|
||
|
||
<!-- fixme: grin: i need to go over this section -->
|
||
|
||
<p>To enable <a href="#claim-operations">claim operations</a>, three new opcodes were added to the blockchain scripting language: <code>OP_CLAIM_NAME</code>, <code>OP_SUPPORT_CLAIM</code>, and <code>OP_UPDATE_CLAIM</code> (in Bitcoin they are respectively <code>OP_NOP6</code>, <code>OP_NOP7</code>, and <code>OP_NOP8</code>). Each op code will push a zero on to the execution stack, and will trigger the claimtrie to perform calculations necessary for each operation. Below are the three supported transactions scripts using these opcodes.</p>
|
||
|
||
<pre><code>OP_CLAIM_NAME <name> <value> OP_2DROP OP_DROP <pubKey>
|
||
|
||
OP_UPDATE_CLAIM <name> <claimId> <value> OP_2DROP OP_2DROP <pubKey>
|
||
|
||
OP_SUPPORT_CLAIM <name> <claimId> OP_2DROP OP_DROP <pubKey>
|
||
</code></pre>
|
||
|
||
<p><code><pubKey></code> can be any valid Bitcoin payout script, so a claimtrie script is also a pay-to-pubkey script to a user-controlled address. Note that the zeros pushed onto the stack by the claimtrie opcodes and vectors are all dropped by <code>OP_2DROP</code> and <code>OP_DROP</code>. This means that claimtrie transactions exist as prefixes to Bitcoin payout scripts and can be spent just like standard transactions.</p>
|
||
|
||
<p>For example, a claim transaction setting the name “Fruit” to “Apple” and using a pay-to-pubkey script will have the following payout script:</p>
|
||
|
||
<pre><code>OP_CLAIM_NAME Fruit Apple OP_2DROP OP_DROP OP_DUP OP_HASH160 <addressOne> OP_EQUALVERIFY OP_CHECKSIG
|
||
</code></pre>
|
||
|
||
<p>Like any standard Bitcoin transaction output script, it will be associated with a transaction hash and output index. The transaction hash and output index are concatenated and hashed to create the claimID for this claim. For the example above, let’s say the above transaction hash is <code>7560111513bea7ec38e2ce58a58c1880726b1515497515fd3f470d827669ed43</code> and the output index is <code>1</code>. Then the claimID would be <code>529357c3422c6046d3fec76be2358004ba22e323</code>.</p>
|
||
|
||
<p>A support for this bid will have the following payout script:</p>
|
||
|
||
<pre><code>OP_SUPPORT_CLAIM Fruit 529357c3422c6046d3fec76be2358004ba22e323 OP_2DROP OP_DROP OP_DUP OP_HASH160 <addressTwo> OP_EQUALVERIFY OP_CHECKSIG
|
||
</code></pre>
|
||
|
||
<p>And now let’s say we want to update the original claim to change the value to “Banana”. An update transaction has a special requirement that it must spend the existing claim that it wishes to update in its redeem script. Otherwise, it will be considered invalid and will not make it into the claimtrie. Thus it will have the following redeem script:</p>
|
||
|
||
<pre><code><signature> <pubKeyForAddressOne>
|
||
</code></pre>
|
||
|
||
<p>This is identical to the standard way of redeeming a pay-to-pubkey script in Bitcoin.</p>
|
||
|
||
<p>The payout script for the update transaction is:</p>
|
||
|
||
<pre><code>OP_UPDATE_CLAIM Fruit 529357c3422c6046d3fec76be2358004ba22e323 Banana OP_2DROP OP_2DROP OP_DUP OP_HASH160 <addressThree> OP_EQUALVERIFY OP_CHECKSIG
|
||
</code></pre>
|
||
|
||
<p>An update transaction must spend the [[outpoint]] of the original claim that it is updating.</p>
|
||
|
||
<h4 id="addresses">Addresses</h4>
|
||
|
||
<p>The address version byte is set to <code>0x55</code> for standard (pay-to-public-key-hash) addresses and <code>0x7a</code> for multisig (pay-to-script-hash) addresses. P2PKH addresses start with the letter <code>b</code>, and P2SH addresses start with <code>r</code>.</p>
|
||
|
||
<p>All the chain parameters are defined <a href="https://github.com/lbryio/lbrycrd/blob/master/src/chainparams.cpp">here</a>.</p>
|
||
|
||
<h4 id="proof-of-payment">Proof of Payment</h4>
|
||
|
||
<p>TODO: Explain how transactions serve as proof that a client has made a valid payment for a piece of content.</p>
|
||
|
||
<h3 id="consensus">Consensus</h3>
|
||
|
||
<p>LBRY makes a few small changes to consensus rules.</p>
|
||
|
||
<h4 id="block-timing">Block Timing</h4>
|
||
|
||
<p>The target block time was lowered from 10 minutes to 2.5 minutes to facilitate faster transaction confirmation.</p>
|
||
|
||
<h4 id="difficulty-adjustment">Difficulty Adjustment</h4>
|
||
|
||
<p>The proof-of-work target is adjusted every block to better adapt to sudden changes in hash rate. The exact adjustment algorithm can be seen <a href="https://github.com/lbryio/lbrycrd/blob/master/src/lbry.cpp">here</a>.</p>
|
||
|
||
<h4 id="block-hash-algorithm">Block Hash Algorithm</h4>
|
||
|
||
<p>LBRY uses a combination of SHA256, SHA512, and RIPEMD160. The exact hashing algorithm can be seen <a href="https://github.com/lbryio/lbrycrd/blob/master/src/hash.cpp#L18">here</a>.</p>
|
||
|
||
<h4 id="block-rewards">Block Rewards</h4>
|
||
|
||
<p>The block reward schedule was adjusted to provide an initial testing period, a quick ramp-up to max block rewards, then a logarithmic decay to 0. The source for the algorithm is <a href="https://github.com/lbryio/lbrycrd/blob/master/src/main.cpp#L1594">here</a>.</p>
|
||
|
||
<h2 id="metadata">Metadata</h2>
|
||
|
||
<!-- done -->
|
||
|
||
<p>Metadata is structured information about the stream or channel separate from the content itself (e.g. the title, language, media type, etc.). It is stored in the <a href="#claim-properties">value property</a> of a claim.</p>
|
||
|
||
<p>Metadata is stored in a serialized format via <a href="https://developers.google.com/protocol-buffers/">Protocol Buffers</a>. This allows for metadata to be:</p>
|
||
|
||
<ul>
|
||
<li><strong>Extensibile</strong>. Metadata can encompass thousands of fields for dozens of types of content. It must be efficient to both modify the structure and maintain backward compatibility.</li>
|
||
<li><strong>Compact</strong>. Blockchain space is expensive. Data must be stored as compactly as possible.</li>
|
||
<li><strong>Interoperabile</strong>. Metadata will be used by many projects written in different languages.</li>
|
||
</ul>
|
||
|
||
<h3 id="specification">Specification</h3>
|
||
|
||
<p>As the metadata specification is designed to grow and change frequently, the full specification will not be examined here. The <a href="https://github.com/lbryio/types/issues">types</a> repository is considered the precise specification.</p>
|
||
|
||
<p>Instead, let’s look at an example and some key fields.</p>
|
||
|
||
<h4 id="metadata-example">Example</h4>
|
||
|
||
<p>Here’s some example metadata:</p>
|
||
|
||
<!-- fix me better example data -->
|
||
|
||
<pre><code>"metadata": {
|
||
"description": "All proceeds go to Holly for buying toys.",
|
||
"language": "en",
|
||
"nsfw": false,
|
||
"thumbnail": "http://www.thetoydiscounter.com/happy.jpg",
|
||
"title": "Holly singing The Happy Working Song",
|
||
"source": {
|
||
"contentType": "video/mp4",
|
||
"source": "92b8aae7a901c56901fd5602c1f1acc0e63fb5492ef2a3cd5b9c631d92cab2e060e2a908baa922c24dee6c5229a98136",
|
||
"sourceType": "lbry_sd_hash",
|
||
}
|
||
}
|
||
</code></pre>
|
||
|
||
<h3 id="key-fields">Key Fields</h3>
|
||
|
||
<p>Some important metadata fields are highlighted below.</p>
|
||
|
||
<h4 id="source-and-stream-hashes">Source and Stream Hashes</h4>
|
||
|
||
<p>The <code>source</code> property contains information about how to fetch the data from the network. Within the <code>source</code> is a unique identifier to locate and find the content in the data network. More in [[Data]].</p>
|
||
|
||
<h4 id="fees-and-fee-structure">Fees and Fee Structure</h4>
|
||
|
||
<!-- fix me extensively -->
|
||
|
||
<ul>
|
||
<li>LBC</li>
|
||
<li>Currencies?</li>
|
||
<li>channel signatures and private keys</li>
|
||
</ul>
|
||
|
||
<h4 id="title">Title</h4>
|
||
|
||
<p>Used to label the content.</p>
|
||
|
||
<h4 id="thumbnail">Thumbnail</h4>
|
||
|
||
<p>An http or lbry URL to be used to display an image associated with the content.</p>
|
||
|
||
<h4 id="content-type">Content Type</h4>
|
||
|
||
<p>The media type of the item as <a href="https://www.iana.org/assignments/media-types/media-types.xhtml">defined</a> by the IANA.</p>
|
||
|
||
<h4 id="certificate">Certificate</h4>
|
||
|
||
<p>Information related to signing the claim as belonging to a specific channel. Covered more in <a href="#channels">Channels</a>.</p>
|
||
|
||
<h3 id="channels">Channels (Identities)</h3>
|
||
|
||
<p>Channels are the unit of identity in the LBRY system. A channel is a claim that:</p>
|
||
|
||
<ul>
|
||
<li>Has name beginning with <code>@</code></li>
|
||
<li>Contains a metadata structure for identity rather than content</li>
|
||
</ul>
|
||
|
||
<p>Included in the metadata is the channel’s public key. Claims belonging to a channel are signed with the corresponding private key. A valid signature proves channel origin and ownership.</p>
|
||
|
||
<p>The purpose of channels is to allow content to be clustered under a single pseudonym or identity. This allows publishers to easily list all their content, maintain attribution, and build their brand.</p>
|
||
|
||
<h4 id="example-channel-metadata">Example Channel Metadata</h4>
|
||
|
||
<p>Included in a channel’s metadata is the following structure:</p>
|
||
|
||
<pre><code>"certificate": {
|
||
"keyType": "SECP256k1",
|
||
"publicKey": "3056301006072a8648ce3d020106052b8104000a0342
|
||
0004180488ffcb3d1825af538b0b952f0eba6933faa6
|
||
d8229609ac0aeadfdbcf49C59363aa5d77ff2b7ff06c
|
||
ddc07116b335a4a0849b1b524a4a69d908d69f1bcebb",
|
||
}
|
||
</code></pre>
|
||
|
||
<p>When a claim published into a channel, the claim data is signed and the following is added to the claim metadata:</p>
|
||
|
||
<pre><code>"publisherSignature": {
|
||
"channelClaimID": "2996b9a087c18456402b57cba6085b2a8fcc136d",
|
||
"signature": "bf82d53143155bb0cac1fd3d917c000322244b5aD17
|
||
e7865124db2ed33812ea66c9b0c3f390a65a9E2d452
|
||
e315e91ae695642847d88e90348ef3c1fa283a36a8",
|
||
"signatureType": "SECP256k1",
|
||
}
|
||
</code></pre>
|
||
|
||
<h3 id="metadata-validation">Validation</h3>
|
||
|
||
<p>No enforcement or validation on metadata happens at the blockchain level. Instead, metadata encoding, decoding, and validation is done by clients. This allows evolution of the metadata without changes to consensus rules.</p>
|
||
|
||
<p>Clients are responsible for validating metadata, including data structure and signatures. This typically happens when the raw binary data stored in the blockchain is decoded client side via Protocol Buffers.</p>
|
||
|
||
<h2 id="data">Data</h2>
|
||
|
||
<p>(This portion covers how content is actually encoded and decoded, fetched, and announced. Expand/fix.)</p>
|
||
|
||
<h3 id="encoding">Encoding</h3>
|
||
|
||
<!-- done -->
|
||
|
||
<p>Content on the LBRY network is encoded to facilitate distribution.</p>
|
||
|
||
<h4 id="blobs">Blobs</h4>
|
||
|
||
<!-- done -->
|
||
|
||
<p>The unit of data in the LBRY network is called a <em>blob</em>. A blob is an encrypted chunk of data up to 2MiB in size. Each blob is indexed by its <em>blob hash</em>, which is a SHA384 hash of the blob contents. Addressing blobs by their hash protects against naming collisions and ensures that the content you get is what you expect.</p>
|
||
|
||
<p>Blobs are encrypted using AES-256 in CBC mode and PKCS7 padding. In order to keep each encrypted blob at 2MiB max, a blob can hold at most 2097151 bytes (2MiB minus 1 byte) of plaintext data. The source code for the exact algorithm is available <a href="https://github.com/lbryio/lbry.go/blob/master/stream/blob.go">here</a>. The encryption key and IV for each blob is stored as described below.</p>
|
||
|
||
<h4 id="streams">Streams</h4>
|
||
|
||
<!-- done -->
|
||
|
||
<p>Multiple blobs are combined into a <em>stream</em>. A stream may be a book, a movie, a CAD file, etc. All content on the network is shared as streams. Every stream begins with the <em>manifest blob</em>, followed by one or more <em>content blobs</em>. The content blobs hold the actual content of the stream. The manifest blob contains information necessary to find the content blobs and decode them into a file. This includes the hashes of the content blobs, their order in the stream, and cryptographic material for decrypting them.</p>
|
||
|
||
<p>The blob hash of the manifest blob is called the <em>stream hash</em>. It uniquely identifies each stream.</p>
|
||
|
||
<h4 id="manifest-contents">Manifest Contents</h4>
|
||
|
||
<!-- done -->
|
||
|
||
<p>A manifest blob’s contents are encoded using a canonical JSON encoding. The JSON encoding must be canonical to support consistent hashing and validation. The encoding is the same as standard JSON, but adds the following rules:</p>
|
||
|
||
<ul>
|
||
<li>Object keys must be quoted and lexicographically sorted.</li>
|
||
<li>All strings are hex-encoded. Hex letters must be lowercase.</li>
|
||
<li>Whitespace before, after, or between tokens is not permitted.</li>
|
||
<li>Floating point numbers, leading zeros, and “minus 0” for integers are not permitted.</li>
|
||
<li>Trailing commas after the last item in an array or object are not permitted.</li>
|
||
</ul>
|
||
|
||
<p>Here’s an example manifest:</p>
|
||
|
||
<!-- originally from 053b2f0f0e82e7f022837382733d5f5817dcd67027103fe43f00fa7a6f9fa8742c1022a851616c1ac15d1c60e89db3f4 -->
|
||
|
||
<pre><code>{"blobs":[{"blob_hash":"a6daea71be2bb89fab29a2a10face08143411a5245edcaa5efff48c2e459e7ec01ad20edfde6da43a932aca45b2cec61","iv":"ef6caef207a207ca5b14c0282d25ce21","length":2097152},{"blob_hash":"bf2717e2c445052366d35bcd58edb108cbe947af122d8f76b4856db577aeeaa2def5b57dbb80f7b1531296bd3e0256fc","iv":"a37b291a37337fc1ff90ae655c244c1d","length":2097152},...,{"blob_hash":"322973617221ddfec6e53bff4b74b9c21c968cd32ba5a5094d84210e660c4b2ed0882b114a2392a08b06183f19330aaf","iv": "a00f5f458695bdc9d50d3dbbc7905abc","length":600160}],"filename":"6b706a7977755477704d632e6d7034","key":"94d89c0493c576057ac5f32eb0871180","version":1}
|
||
</code></pre>
|
||
|
||
<p>Here’s the same manifest, with whitespace added for readability:</p>
|
||
|
||
<pre><code>{
|
||
"blobs":[
|
||
{
|
||
"blob_hash":"a6daea71be2bb89fab29a2a10face08143411a5245edcaa5efff48c2e459e7ec01ad20edfde6da43a932aca45b2cec61",
|
||
"iv":"ef6caef207a207ca5b14c0282d25ce21",
|
||
"length":2097152
|
||
},
|
||
{
|
||
"blob_hash":"bf2717e2c445052366d35bcd58edb108cbe947af122d8f76b4856db577aeeaa2def5b57dbb80f7b1531296bd3e0256fc",
|
||
"iv":"a37b291a37337fc1ff90ae655c244c1d",
|
||
"length":2097152
|
||
},
|
||
...,
|
||
{
|
||
"blob_hash":"322973617221ddfec6e53bff4b74b9c21c968cd32ba5a5094d84210e660c4b2ed0882b114a2392a08b06183f19330aaf",
|
||
"iv": "a00f5f458695bdc9d50d3dbbc7905abc",
|
||
"length": 600160
|
||
}
|
||
],
|
||
"filename":"6b706a7977755477704d632e6d7034",
|
||
"key":"94d89c0493c576057ac5f32eb0871180",
|
||
"version":1
|
||
}
|
||
</code></pre>
|
||
|
||
<p>The <code>key</code> field contains the key to decrypt the stream, and is optional. The key may be stored by a third party and made available to a client when presented with proof that the content was purchased. The <code>version</code> field is always 1. It is intended to signal structure changes in future versions of this protocol. The <code>length</code> field for each blob is the length of the encrypted blob, not the original file chunk.</p>
|
||
|
||
<p>Every stream must have at least two blobs - the manifest blob and a content blob. Consequently, zero-length streams are not allowed.</p>
|
||
|
||
<h4 id="stream-encoding">Stream Encoding</h4>
|
||
|
||
<!-- done -->
|
||
|
||
<p>A file must be encoded into a stream before it can be published. Encoding involves breaking the file into chunks, encrypting the chunks into content blobs, and creating the manifest blob. Here are the steps:</p>
|
||
|
||
<h5 id="setup">Setup</h5>
|
||
|
||
<!-- done -->
|
||
|
||
<ol>
|
||
<li>Generate a random 32-byte key for the stream.</li>
|
||
</ol>
|
||
|
||
<h5 id="content-blobs">Content Blobs</h5>
|
||
|
||
<!-- done -->
|
||
|
||
<ol>
|
||
<li>Break the file into chunks of at most 2097151 bytes.</li>
|
||
<li>Generate a random IV for each chuck.</li>
|
||
<li>Pad each chunk using PKCS7 padding</li>
|
||
<li>Encrypt each chunk with AES-CBC using the stream key and the IV for that chunk.</li>
|
||
<li>An encrypted chunk is a blob.</li>
|
||
</ol>
|
||
|
||
<h5 id="manifest-blob">Manifest Blob</h5>
|
||
|
||
<!-- done -->
|
||
|
||
<ol>
|
||
<li>Fill in the manifest data.</li>
|
||
<li>Encode the data using the canonical JSON encoding described above.</li>
|
||
<li>Compute the stream hash.</li>
|
||
</ol>
|
||
|
||
<p>An implementation of this process is available <a href="https://github.com/lbryio/lbry.go/tree/master/stream">here</a>.
|
||
fixme: this link is for v0, not v1. need to implement v1 or drop the link.</p>
|
||
|
||
<h4 id="stream-decoding">Stream Decoding</h4>
|
||
|
||
<!-- done -->
|
||
|
||
<p>Decoding a stream is like encoding in reverse, and with the added step of verifying that the expected blob hashes match the actual data.</p>
|
||
|
||
<ol>
|
||
<li>Verify that the manifest blob hash matches the stream hash you expect.</li>
|
||
<li>Parse the manifest blob contents.</li>
|
||
<li>Verify the hashes of the content blobs.</li>
|
||
<li>Decrypt and remove the padding from each content blob using the key and IVs in the manifest.</li>
|
||
<li>Concatenate the decrypted chunks in order.</li>
|
||
</ol>
|
||
|
||
<h3 id="announce">Announce</h3>
|
||
|
||
<p>After a [[stream]] is encoded, it must be <em>announced</em> to the network. Announcing is the process of letting other nodes on the network know that you have content available for download. The LBRY network tracks announced content using a distributed hash table.</p>
|
||
|
||
<h4 id="distributed-hash-table">Distributed Hash Table</h4>
|
||
|
||
<p><em>Distributed hash tables</em> (or DHTs) have proven to be an effective way to build a decentralized content network. Our DHT implementation follows the <a href="https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf">Kademlia</a>
|
||
specification fairly closely, with some modifications.</p>
|
||
|
||
<p>A distributed hash table is a key-value store that is spread over multiple nodes in a network. Nodes may join or leave the network anytime, with no central coordination necessary. Nodes communicate with each other using a peer-to-peer protocol to advertise what data they have and what they are best positioned to store.</p>
|
||
|
||
<p>When a host connects to the DHT, it announces the hash for every [[blob]] it wishes to share. Downloading a blob from the network requires querying the DHT for a list of hosts that announced that blob’s hash (called <em>peers</em>), then requesting the blob from the peers directly.</p>
|
||
|
||
<h4 id="announcing-to-the-dht">Announcing to the DHT</h4>
|
||
|
||
<p>A host announces a hash to the DHT in two steps. First, the host looks for nodes that are closest to the target hash. Then the host asks those nodes to store the fact that the host has the target hash available for download.</p>
|
||
|
||
<p>Finding the closest nodes is done via iterative <code>FindNode</code> DHT requests. The host starts with the closest nodes it knows about and sends a <code>FindNode(target_hash)</code> request to each of them. If any of the requests return nodes that are closer to the target hash, the host sends <code>FindNode</code> requests to those nodes to try to get even closer. When the <code>FindNode</code> requests no longer return nodes that are closer, the search ends.</p>
|
||
|
||
<p>Once the search is over, the host takes the 8 closest nodes it found and sends a <code>Store(target_hash)</code> request to them. The nodes receiving this request store the fact that the host is a peer for the target hash.</p>
|
||
|
||
<h3 id="download">Download</h3>
|
||
|
||
<p>A client wishing to download a [[stream]] must first query the [[DHT]] to find [[peers]] hosting the [[blobs]] in that stream, then contact those peers to download the blobs directly.</p>
|
||
|
||
<h4 id="querying-the-dht">Querying the DHT</h4>
|
||
|
||
<p>Querying works almost the same way as [[announcing]]. A client looking for a target hash will start by sending iterative <code>FindValue(target_hash)</code> requests to the nodes it knows that are closest to the target hash. If a node receives a <code>FindValue</code> request and knows of any peers for the target hash, it will respond with a list of those peers. Otherwise, it will respond with the closest nodes to the target hash that it knows about. The client then queries those closer nodes using the same <code>FindValue</code> call. This way, each call either finds the client some peers, or brings it closer to finding those peers. If no peers are found and no closer nodes are being returned, the client will determine that the target hash is not available and will give up.</p>
|
||
|
||
<h4 id="blob-exchange-protocol">Blob Exchange Protocol</h4>
|
||
|
||
<p>Downloading a blob from a peer is governed by the <em>Blob Exchange Protocol</em>. It is used by hosts and clients to check blob availability, exchange blobs, and negotiate data prices. The protocol is an RPC protocol using Protocol Buffers and the gRPC framework. It has five types of requests.</p>
|
||
|
||
<p>fixme: protocol does not <strong>negotiate</strong> anything right now. It just checks the price. Should we include negotiation in v1?</p>
|
||
|
||
<h5 id="pricecheck">PriceCheck</h5>
|
||
|
||
<p>PriceCheck gets the price that the server is charging for data transfer. It returns the prices in [[deweys]] per KB.</p>
|
||
|
||
<h5 id="downloadcheck">DownloadCheck</h5>
|
||
|
||
<p>DownloadCheck checks whether the server has certain blobs available for download. For each hash in the request, the server returns a true or false to indicate whether the blob is available.</p>
|
||
|
||
<h5 id="download-1">Download</h5>
|
||
|
||
<p>Download requests the blob for a given hash. The response contains the blob, its hash, and the address where to send payment for the data transfer. If the blob is not available on the server, the response will instead contain an error.</p>
|
||
|
||
<h5 id="uploadcheck">UploadCheck</h5>
|
||
|
||
<p>UploadCheck asks the server whether blobs can be uploaded to it. For each hash in the request, the server returns a true or false to indicate whether it would accept a given blob for upload. In addition, if any of the hashes in the request is a stream hash and the server has the manifest blob for that stream but is missing some content blobs, it may include the hashes of those content blobs in the response.</p>
|
||
|
||
<h5 id="upload">Upload</h5>
|
||
|
||
<p>Upload sends a blob to the server. If uploading many blobs, the client should use the UploadCheck request to check which blobs the server actually needs. This avoids needlessly uploading blobs that the server already has. If a client tries to upload too many blobs that the server does not want, this may be consider a denial of service attack.</p>
|
||
|
||
<p>The protocol calls and message types are defined in detail <a href="https://github.com/lbryio/lbry.go/blob/master/blobex/blobex.proto">here</a>.</p>
|
||
|
||
<h3 id="reflectors-and-data-markets">Reflectors and Data Markets</h3>
|
||
|
||
<p>In order for a client to download content, there must be hosts online that have the content the client wants, when the client wants it. To incentivize the continued hosting of data, the blob exchange protocol supports data upload and payment for data. <em>Reflectors</em> are hosts that accept data uploads. They rehost (reflect) the uploaded data and charge for downloads. Using a reflector is optional, but most publishers will probably choose to use them. Doing so obviates the need for the publisher’s server to be online and connectable, which can be especially useful for mobile clients or those behind a firewall.</p>
|
||
|
||
<p>The current version of the protocol does not support sophisticated price negotiation between clients and hosts. The host simply chooses the price it will charge. Clients check this price before downloading, and pay the price after the download is complete. Future protocol versions will include more options for price negotiation, as well as stronger proofs of payment.</p>
|
||
|
||
<hr>
|
||
|
||
<p><em>Edit this on Github: <a href="https://github.com/lbryio/spec">https://github.com/lbryio/spec</a></em></p>
|
||
|
||
<p></div></main> <!-- DONT DELETE THIS, its for the TOC --></p>
|
||
|
||
<!---
|
||
|
||
### Supports, Tips
|
||
|
||
Supports add weight to name claims. They are kind of like voting. You retain control of the credits.
|
||
Tips are supports where person you tip gains control of the credits.
|
||
|
||
### Discovery
|
||
|
||
### Search
|
||
|
||
Search will be handled primarily by external indexing services. There are many existing search solutions
|
||
that would ingest the blockchain data and build an index of the content. The novel aspect of our system is
|
||
that the credits committed to a claim are a strong signal of relevance.
|
||
|
||
### Tagging
|
||
|
||
Tags provide extra information for content discovery. A tag has a claim ID and a name. Tags can be created,
|
||
supported, updated, and abandoned, just like claims. One key difference is that tag supports may be
|
||
labeled “negative” supports. A negative support reduces the effective amount of credits attached to a
|
||
tag. This is a signal that the tag is not a good fit for the content of the claim.
|
||
|
||
## Trust and Security
|
||
|
||
We believe that the
|
||
|
||
## Combatting the Ugly
|
||
|
||
Use this section to rebut some of the most common concerns regarding the nature of LBRY.
|
||
|
||
One of our core beliefs is that people want to pay the legitimate content owners and creators, as
|
||
long as the content reasonably-priced and the payment process is convenient.
|
||
|
||
|
||
Conclusion
|
||
|
||
Summary
|
||
|
||
|
||
-->
|
||
|
||
</body>
|
||
</html>
|
||
|