Download
@@ -164,23 +176,45 @@
Blockchain
+
+
The LBRY blockchain is a public, proof-of-work blockchain. It serves three key purposes:
- An index of the content available on the network
- A payment system and record of purchases for priced content
-- Trustful publisher identities (fixme: should this even be listed here?)
+- Trustful publisher identities
The LBRY blockchain is a fork of the Bitcoin blockchain, with substantial modifications. This document will not cover or specify any aspects of LBRY that are identical to Bitcoin, and will instead focus on the differences.
Claims
-A single metadata entry in the blockchain is called a claim. It records an item that was published to the network or a publisher’s identity.
+
-Every claim has a globally-unique claimID, an amount (how many credits were set aside to back the claim), and a value. The value may contain metadata about a piece of content, a publisher’s public key, or other information. See the Metadata section for more information about what may be stored in the value.
+A single metadata entry in the blockchain is called a claim. It records a file that was published to the network or a publisher’s identity.
-Every claim is associated with a name, which is a bytestring of 0-255 bytes. Every name must be a valid UTF8 string.
+Claim Properties
+
+
+
+Every claim contains 4 properties:
+
+
+ - claimId
+ - A 20-byte hash unique among all claims. See [Claim Identifier Generation](#claim-identifier-generation).
+ - name
+ - A normalized UTF-8 string of up to 255 bytes used to address the claim. See [URLs](#urls) and [Normalization](#normalization).
+ - amount
+ - A quantity of tokens used to stake the claim. See [Controlling](#controlling).
+ - value
+ - Metadata about a piece of content, a publisher's public key, or other information. See [Metadata](#metadata).
+
+
+
+Claim Example
+
+
Here is an example claim:
@@ -202,19 +236,28 @@
Claim Operations
-There are four claim operations: create, support, update, and abandon.
+
-A create operation makes a new claim for a name, or submits a competing claim on an existing name.
+There are four claim operations: create, support, update, and abandon.
-A support is a claim that adds to the credit total of an existing claim. A support does not have it’s own claim ID or data. Instead, it has the claim ID of the claim to which its amount will be added.
-
-An update changes the data or the amount stored in an existing claim or support. Updates do not change the claim ID, so an updated claim retains any supports attached to it.
-
-An abandon withdraws a claim or support, freeing the associated credits to be used for other purposes.
+
+ - create
+ - Makes a new claim.
+ - support
+ - Adds its [[amount]] to the stake of an already existing claim. It contains no metadata.
+ - update
+ - Changes the data or the amount stored in an existing claim or support. Updates do not change the claim ID, so an updated claim retains any supports attached to it.
+ - abandon
+ - Withdraws a claim or support, freeing the associated credits to be used for other purposes.
+
Claimtrie
-The claimtrie is the data structure that LBRY uses to store claims and prove the correctness of name resolution. It is a Merkle tree that maps names to claims. Claims are stored as leaf nodes in the tree. Names are stored as the path from the root node to the leaf node.
+
+
+The claimtrie is the data structure used to store the set of all claims and prove the correctness of claim resolution.
+
+The claimtrie is implemented as a Merkle tree that maps names to claims. Claims are stored as leaf nodes in the tree. Names are stored as the path from the root node to the leaf node.
The hash of the root node (the root hash) is stored in the header of each block in the blockchain. Nodes in the LBRY network use the root hash to efficiently and securely validate the state of the claimtrie.
@@ -222,17 +265,23 @@
For more details on the specific claimtrie implementation, see the source code.
-Claim Properties
+Claim Statuses
-A claim can have one or more the following properties at a given block:
+
+
+A claim can have one or more the following properties at a given block.
Accepted
-An accepted claim or support is simply one that has been entered into the blockchain. This happens when the transaction containing the claim is included in a block.
+
+
+An accepted claim or support is one that has been entered into the blockchain. This happens when the transaction containing the claim is included in a block.
Abandoned
-An abandoned claim or support is one that was withdrawn by its creator. It is no longer in contention to control a name. Spending the transaction that contains the claim will also cause the claim to become abandoned.
+
+
+An abandoned claim or support is one that was withdrawn by its creator. It is no longer in contention to control a name. Spending a transaction that contains a claim will cause that claim to become abandoned.
While data related to abandoned claims technically still resides in the blockchain, it is considered inappropriate to use this data to fetch the associated content.
@@ -299,9 +348,7 @@
Normalization
-TODO: Talk about how claim names are normalized.
-
-https://github.com/lbryio/lbrycrd/issues/208
+Names in the claimtrie are normalized to avoid confusion due to Unicode equivalence or casing. All names are normalized using the NFD normalization form, then lowercased using the en_US locale.
URLs
@@ -348,20 +395,22 @@ lbry:meet-LBRY$3
lbry:@lbry/meet-LBRY
-The full URL grammar is defined below using Xquery EBNF notation:
+Grammar
+
+The full URL grammar is defined using Xquery EBNF notation:
URL ::= Scheme Path Query?
-Scheme ::= 'lbry:'
+Scheme ::= 'lbry://'
Path ::= ClaimNameAndModifier | ChannelAndModifier ( '/' ClaimNameAndModifier )?
ClaimNameAndModifier ::= ClaimName Modifier?
ChannelAndModifier ::= Channel Modifier?
-ClaimName ::= AllowedChar+
+ClaimName ::= NameChar+
Channel ::= '@' ClaimName
Modifier ::= ClaimID | ClaimSequence | BidPosition
@@ -372,8 +421,8 @@ BidPosition ::= '$' PositiveNumber
Query ::= '?' QueryParameterList
QueryParameterList ::= QueryParameter ( '&' QueryParameterList )*
QueryParameter ::= QueryParameterName ( '=' QueryParameterValue )?
-QueryParameterName ::= AllowedChar+
-QueryParameterValue ::= AllowedChar+
+QueryParameterName ::= NameChar+
+QueryParameterValue ::= NameChar+
PositiveDigit ::= [123456789]
Digit ::= '0' | PositiveDigit
@@ -382,7 +431,8 @@ PositiveNumber ::= PositiveDigit Digit*
HexAlpha ::= [abcdef]
Hex ::= (Digit | HexAlpha)+
-AllowedChar ::= [^=&#:$@?/] /* any UTF8 character that is not reserved */
+NameChar ::= Char - [=&#:$@?/] /* any character that is not reserved */
+Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */
Design Notes
@@ -512,17 +562,16 @@ OP_SUPPORT_CLAIM <name> <claimId> OP_2DROP OP_DROP <pubKey>
Streams and Stream Hashes
-(The metadata property sd_hash contains a unique identifier to locate and find the content in the data network. Reference [[Data]].)
+(The metadata property lbry_sd_hash contains a unique identifier to locate and find the content in the data network. Reference [[Data]].)
Fees and Fee Structure
- LBC
- Currencies?
+- channel signatures and private keys
-More?
-
Identities
Channels are the unit of identity in the LBRY system. A channel is a claim that start with @ and contains a metadata structure for identities rather than content. The most important part of channel’s metadata is the public key. Claims belonging to a channel are signed with the corresponding private key. A valid signature proves channel membership.
@@ -555,6 +604,8 @@ OP_SUPPORT_CLAIM <name> <claimId> OP_2DROP OP_DROP <pubKey>
+Clients are responsible for validating metadata, including data structure and signatures.
+
(expand)
@@ -570,13 +621,89 @@ OP_SUPPORT_CLAIM <name> <claimId> OP_2DROP OP_DROP <pubKey>
Blobs
-The unit of content in our network is called a blob. A blob is an encrypted chunk of data up to 2MB in size. Each blob is indexed by its blob hash, which is a SHA384 hash of the blob contents. Addressing blobs by their hashes simultaneously protects against naming collisions and ensures that the content you get is what you expect.
+The unit of data in our network is called a blob. A blob is an encrypted chunk of data up to 2MB in size. Each blob is indexed by its blob hash, which is a SHA384 hash of the blob contents. Addressing blobs by their hashes simultaneously protects against naming collisions and ensures that the content you get is what you expect.
+
+Blobs are encrypted using AES-256 in CBC mode and PKCS7 padding. In order to keep each encrypted blob at 2MB max, a blob can hold at most 2097151 bytes (2MB minus 1 byte) of plaintext data. The source code for exact algorithm is available here. The encryption key and IV for each blob is stored as described below.
Streams
-Multiple blobs may be combined into a stream. A stream may be a book, a movie, a CAD file, etc. All content on the network is shared as streams. Every stream begins with the stream descriptor blob, which contains a JSON list of the hashes and keys of the content blobs. The content blobs hold the actual content of the stream. Every stream ends with an empty content blob, to signify that the stream has finished (this is similar to a null-terminated string, and is necessary to support streaming content).
+Multiple blobs are combined into a stream. A stream may be a book, a movie, a CAD file, etc. All content on the network is shared as streams. Every stream begins with the manifest blob, followed by one or more content blobs. The content blobs hold the actual content of the stream. The manifest blob contains information necessary to find the content blobs and convert them into a file. This includes the hashes of the content blobs, their order in the stream, and cryptographic material for decrypting them.
-How to Turn Files into Streams, and Vice Versa
+The blob hash of the manifest blob is called the stream hash. It uniquely identifies each stream.
+
+Manifest Encoding
+
+A manifest blob’s contents are encoded using a canonical JSON encoding. The JSON encoding must be canonical to support consistent hashing and validation. The encoding is the same as standard JSON, but adds the following rules:
+
+
+- Object keys must be quoted and lexicographically sorted.
+- All strings are hex-encoded. Hex letters must be lowercase.
+- Whitespace before, after, or between tokens is not permitted.
+- Floating point numbers, leading zeros, and “minus 0” for integers are not permitted.
+- Trailing commas after the last item in an array or object are not permitted.
+
+
+Here’s an example manifest, with whitespace added for readability:
+
+
+
+
{
+ "blobs":[
+ {
+ "blob_hash":"a6daea71be2bb89fab29a2a10face08143411a5245edcaa5efff48c2e459e7ec01ad20edfde6da43a932aca45b2cec61",
+ "iv":"ef6caef207a207ca5b14c0282d25ce21",
+ "length":2097152
+ },
+ {
+ "blob_hash":"bf2717e2c445052366d35bcd58edb108cbe947af122d8f76b4856db577aeeaa2def5b57dbb80f7b1531296bd3e0256fc",
+ "iv":"a37b291a37337fc1ff90ae655c244c1d",
+ "length":2097152
+ },
+ ...,
+ {
+ "blob_hash":"322973617221ddfec6e53bff4b74b9c21c968cd32ba5a5094d84210e660c4b2ed0882b114a2392a08b06183f19330aaf",
+ "iv": "a00f5f458695bdc9d50d3dbbc7905abc",
+ "length": 600160
+ }
+ ],
+ "filename":"6b706a7977755477704d632e6d7034",
+ "key":"94d89c0493c576057ac5f32eb0871180"
+}
+
+
+The key field contains the key to decrypt the stream, and is optional. The key may be stored by a third party and made available to a client when presented with proof that the content was purchased. The length field for each blob is the length of the encrypted blob, not the original file chunk.
+
+Every stream must have at least two blobs - the manifest blob and a content blob. Consequently, zero-length streams are not allowed.
+
+Stream Creation
+
+A file must be converted into a stream before it can be published. Conversion involves breaking the file into chunks, encrypting the chunks into content blobs, and creating the manifest blob. Here are the steps:
+
+Setup
+
+
+- Generate a random 32-byte key for the stream.
+
+
+Content Blobs
+
+
+- Break the file into chunks of at most 2097151 bytes.
+- Generate a random IV for each chuck.
+- Pad each chunk using PKCS7 padding
+- Encrypt each chunk with AES-CBC using the stream key and the IV for that chunk.
+- An encrypted chunk is a blob.
+
+
+Manifest Blob
+
+
+- Fill in the manifest data.
+- Encode the data using the canonical JSON encoding described above.
+- Compute the stream hash
+
+
+An implementation of this process is available here.
Download
diff --git a/index.md b/index.md
index afd983c..44b55b6 100644
--- a/index.md
+++ b/index.md
@@ -47,9 +47,11 @@ TODO:
* [Conventions and Terminology](#conventions-and-terminology)
* [Blockchain](#blockchain)
* [Claims](#claims)
+ * [Claim Properties](#claim-properties)
+ * [Claim Example](#claim-example)
* [Claim Operations](#claim-operations)
* [Claimtrie](#claimtrie)
- * [Claim Properties](#claim-properties)
+ * [Claim Statuses](#claim-statuses)
* [Accepted](#accepted)
* [Abandoned](#abandoned)
* [Active](#active)
@@ -57,6 +59,7 @@ TODO:
* [Normalization](#normalization)
* [URLs](#urls)
* [Components](#components)
+ * [Grammar](#grammar)
* [Design Notes](#design-notes)
* [Transactions](#transactions)
* [Operations and Opcodes](#operations-and-opcodes)
@@ -72,14 +75,17 @@ TODO:
* [Key Metadata Fields](#key-metadata-fields)
* [Streams and Stream Hashes](#streams-and-stream-hashes)
* [Fees and Fee Structure](#fees-and-fee-structure)
- * [More?](#more)
* [Identities](#identities)
* [Metadata Validation](#metadata-validation)
* [Data](#data)
* [Encoding and Decoding](#encoding-and-decoding)
* [Blobs](#blobs)
* [Streams](#streams)
- * [How to Turn Files into Streams, and Vice Versa](#how-to-turn-files-into-streams-and-vice-versa)
+ * [Manifest Encoding](#manifest-encoding)
+ * [Stream Creation](#stream-creation)
+ * [Setup](#setup)
+ * [Content Blobs](#content-blobs)
+ * [Manifest Blob](#manifest-blob)
* [Download](#download)
* [Distributed Hash Table](#distributed-hash-table)
* [Blob Exchange Protocol](#blob-exchange-protocol)
@@ -149,11 +155,15 @@ The LBRY blockchain is a public, proof-of-work blockchain. It serves three key p
The LBRY blockchain is a fork of the [Bitcoin](https://bitcoin.org/bitcoin.pdf) blockchain, with substantial modifications. This document will not cover or specify any aspects of LBRY that are identical to Bitcoin, and will instead focus on the differences.
-### Claims
+### Claims
+
+
A single metadata entry in the blockchain is called a _claim_. It records a file that was published to the network or a publisher's identity.
-#### Claim Properties
+#### Claim Properties
+
+
Every claim contains 4 properties:
@@ -168,7 +178,9 @@ Every claim contains 4 properties:
Metadata about a piece of content, a publisher's public key, or other information. See [Metadata](#metadata).
-#### Claim Example
+#### Claim Example
+
+
Here is an example claim:
@@ -189,7 +201,9 @@ Here is an example claim:
}
```
-#### Claim Operations
+#### Claim Operations
+
+
There are four claim operations: _create_, _support_, _update_, and _abandon_.
@@ -204,7 +218,9 @@ There are four claim operations: _create_, _support_, _update_, and _abandon_.
Withdraws a claim or support, freeing the associated credits to be used for other purposes.
-#### Claimtrie
+#### Claimtrie
+
+
The _claimtrie_ is the data structure used to store the set of all claims and prove the correctness of claim resolution.
@@ -216,15 +232,21 @@ Multiple claims can exist for the same name. They are all stored in the leaf nod
For more details on the specific claimtrie implementation, see [the source code](https://github.com/lbryio/lbrycrd/blob/master/src/claimtrie.cpp).
-#### Claim Statuses
+#### Claim Statuses
+
+
A claim can have one or more the following properties at a given block.
-##### Accepted
+##### Accepted
+
+
An accepted claim or support is one that has been entered into the blockchain. This happens when the transaction containing the claim is included in a block.
-##### Abandoned
+##### Abandoned
+
+
An abandoned claim or support is one that was withdrawn by its creator. It is no longer in contention to control a name. Spending a transaction that contains a claim will cause that claim to become abandoned.
@@ -592,53 +614,82 @@ Clients are responsible for validating metadata, including data structure and si
#### Blobs
-The unit of content in our network is called a `blob`. A blob is an encrypted chunk of data up to 2MB in size. Each blob is indexed by its `blob hash`, which is a SHA384 hash of the blob contents. Addressing blobs by their hashes simultaneously protects against naming collisions and ensures that the content you get is what you expect.
+The unit of data in our network is called a _blob_. A blob is an encrypted chunk of data up to 2MB in size. Each blob is indexed by its _blob hash_, which is a SHA384 hash of the blob contents. Addressing blobs by their hashes simultaneously protects against naming collisions and ensures that the content you get is what you expect.
+
+Blobs are encrypted using AES-256 in CBC mode and PKCS7 padding. In order to keep each encrypted blob at 2MB max, a blob can hold at most 2097151 bytes (2MB minus 1 byte) of plaintext data. The source code for exact algorithm is available [here](https://github.com/lbryio/lbry.go/blob/master/stream/blob.go). The encryption key and IV for each blob is stored as described below.
#### Streams
-Multiple blobs are combined into a `stream`. A stream may be a book, a movie, a CAD file, etc. All content on the network is shared as streams. Every stream begins with the `stream descriptor blob` (or SD blob), followed by one or more `content blobs`. The content blobs hold the actual content of the stream. The SD blob contains information necessary to find the content blobs and assemble them into a file. This includes the hashes of the content blobs, their order in the stream, and cryptographic material for decrypting them.
+Multiple blobs are combined into a _stream_. A stream may be a book, a movie, a CAD file, etc. All content on the network is shared as streams. Every stream begins with the _manifest blob_, followed by one or more _content blobs_. The content blobs hold the actual content of the stream. The manifest blob contains information necessary to find the content blobs and convert them into a file. This includes the hashes of the content blobs, their order in the stream, and cryptographic material for decrypting them.
-Here's an example SD blob. It's hash is `053b2f0f0e82e7f022837382733d5f5817dcd67027103fe43f00fa7a6f9fa8742c1022a851616c1ac15d1c60e89db3f4`.
+The blob hash of the manifest blob is called the _stream hash_. It uniquely identifies each stream.
+
+##### Manifest Encoding
+
+A manifest blob's contents are encoded using a canonical JSON encoding. The JSON encoding must be canonical to support consistent hashing and validation. The encoding is the same as standard JSON, but adds the following rules:
+
+- Object keys must be quoted and lexicographically sorted.
+- All strings are hex-encoded. Hex letters must be lowercase.
+- Whitespace before, after, or between tokens is not permitted.
+- Floating point numbers, leading zeros, and "minus 0" for integers are not permitted.
+- Trailing commas after the last item in an array or object are not permitted.
+
+Here's an example manifest, with whitespace added for readability:
+
+
```
{
- "stream_type":"lbryfile",
- "key":"94d89c0493c576057ac5f32eb0871180",
- "suggested_file_name":"6b706a7977755477704d632e6d7034",
- "stream_hash":"8cef6280f36f7e6590a6218da6b2eb8184ab1435c3f8d77f008088f5d2bc6bd2252a2beb9cfa3d9d40b9ce36d2d7b2ce"
- "stream_name":"6b706a7977755477704d632e6d7034",
- "blobs":[
- {
- "length":2097152,
- "blob_num":0,
- "blob_hash":"a6daea71be2bb89fab29a2a10face08143411a5245edcaa5efff48c2e459e7ec01ad20edfde6da43a932aca45b2cec61",
- "iv":"ef6caef207a207ca5b14c0282d25ce21"
- },
- {
- "length":2097152,
- "blob_num":1,
- "blob_hash":"bf2717e2c445052366d35bcd58edb108cbe947af122d8f76b4856db577aeeaa2def5b57dbb80f7b1531296bd3e0256fc",
- "iv":"a37b291a37337fc1ff90ae655c244c1d"
- },
- ...,
- {
- "length":0,
- "blob_num":45,
- "iv":"53677e463ddb3bf060a40b99f8236432"
- }
- ]
+ "blobs":[
+ {
+ "blob_hash":"a6daea71be2bb89fab29a2a10face08143411a5245edcaa5efff48c2e459e7ec01ad20edfde6da43a932aca45b2cec61",
+ "iv":"ef6caef207a207ca5b14c0282d25ce21",
+ "length":2097152
+ },
+ {
+ "blob_hash":"bf2717e2c445052366d35bcd58edb108cbe947af122d8f76b4856db577aeeaa2def5b57dbb80f7b1531296bd3e0256fc",
+ "iv":"a37b291a37337fc1ff90ae655c244c1d",
+ "length":2097152
+ },
+ ...,
+ {
+ "blob_hash":"322973617221ddfec6e53bff4b74b9c21c968cd32ba5a5094d84210e660c4b2ed0882b114a2392a08b06183f19330aaf",
+ "iv": "a00f5f458695bdc9d50d3dbbc7905abc",
+ "length": 600160
+ }
+ ],
+ "filename":"6b706a7977755477704d632e6d7034",
+ "key":"94d89c0493c576057ac5f32eb0871180"
}
```
-Every field except 'stream_type' is either an integer or a hex-encoded string. The `key` field contains the key to decrypt the stream, and is optional. The key may be stored externally on a keyserver. The keyserver would make the key available to a client when presented with proof that the content was purchased.
+The `key` field contains the key to decrypt the stream, and is optional. The key may be stored by a third party and made available to a client when presented with proof that the content was purchased. The `length` field for each blob is the length of the encrypted blob, not the original file chunk.
-The last blob in the `blobs` list of the SD hash is always an empty blob with no hash. This signifies the end of the stream. This is similar to a null-terminated string, and is necessary to support content where the length is not known in advance (e.g. live video).
+Every stream must have at least two blobs - the manifest blob and a content blob. Consequently, zero-length streams are not allowed.
-Every stream must have at least two blobs - an SD blob and a content blob. Zero-length streams are not allowed.
+#### Stream Creation
-#### How to Turn Files into Streams, and Vice Versa
+A file must be converted into a stream before it can be published. Conversion involves breaking the file into chunks, encrypting the chunks into content blobs, and creating the manifest blob. Here are the steps:
-https://github.com/lbryio/lbry.go/tree/master/stream
+##### Setup
+
+1. Generate a random 32-byte key for the stream.
+
+##### Content Blobs
+
+1. Break the file into chunks of at most 2097151 bytes.
+1. Generate a random IV for each chuck.
+1. Pad each chunk using PKCS7 padding
+1. Encrypt each chunk with AES-CBC using the stream key and the IV for that chunk.
+1. An encrypted chunk is a blob.
+
+##### Manifest Blob
+
+1. Fill in the manifest data.
+1. Encode the data using the canonical JSON encoding described above.
+1. Compute the stream hash
+
+An implementation of this process is available [here](https://github.com/lbryio/lbry.go/tree/master/stream).
### Download
diff --git a/watch.sh b/watch.sh
new file mode 100644
index 0000000..16bc5e1
--- /dev/null
+++ b/watch.sh
@@ -0,0 +1,3 @@
+#!/bin/bash
+
+./bin/reflex --decoration=none --start-service=true --inverse-regex='index\.html' -- sh -c "make"