Remove claim expiration

kaykurokawa commented

2016-07-18 23:45:33 +02:00

(Migrated from github.com)

As discovered by @jimmykiselak

Expired claims are not kept track of by claimtrie.cpp , thus they are not updateable using OP_UPDATE_CLAIM. Name owners can only reclaim the expired claim through OP_CLAIM_NAME , which means that all supports must be updated as well.

Expired claims should be kept track of, so they can be updated.

This is a hard fork and will require reindexing. We have roughly till the end of August to fix this issue in a simple way (see related issue https://github.com/lbryio/lbrycrd/issues/20 ) . Otherwise there needs to be exceptions made for early blocks.

As discovered by @jimmykiselak Expired claims are not kept track of by claimtrie.cpp , thus they are not updateable using OP_UPDATE_CLAIM. Name owners can only reclaim the expired claim through OP_CLAIM_NAME , which means that all supports must be updated as well. Expired claims should be kept track of, so they can be updated. This is a hard fork and will require reindexing. We have roughly till the end of August to fix this issue in a simple way (see related issue https://github.com/lbryio/lbrycrd/issues/20 ) . Otherwise there needs to be exceptions made for early blocks.

kauffj commented

2017-08-14 18:07:47 +02:00

(Migrated from github.com)

@kaykurokawa looks like we'd need to do something this month if we're going to address this.

What would it look like to just have claims never expire? It seems like this may be more desirable. Doing some quick math, 1e9 pieces (1 billion) would require 5000 updates per block just to keep content published.

@kaykurokawa looks like we'd need to do something this month if we're going to address this. What would it look like to just have claims never expire? It seems like this may be more desirable. Doing some quick math, 1e9 pieces (1 billion) would require 5000 updates per block just to keep content published.

tzarebczan commented

2017-09-13 16:50:17 +02:00

(Migrated from github.com)

@kauffj / @kaykurokawa / @jackrobison I remember discussing this on one of the calls, not sure if we had to take any action.

lyoshenka commented

2017-09-19 17:59:57 +02:00

(Migrated from github.com)

Our oldest claim was in block 2806, so it expires at 265780, which will be roughly November 1st.

lyoshenka commented

2017-10-31 13:45:21 +01:00

(Migrated from github.com)

@kaykurokawa @jackrobison what are the downsides to having claims never expire? The only one I can think of is that its nice to be able to prune expired claims from the cliamtrie. Anything else?

kaykurokawa commented

2017-10-31 20:23:24 +01:00

(Migrated from github.com)

a) Already stated but worth stating again: Old names that people claimed with a lot of credits and lost the key for can be reclaimed

b) There are several ways in which the value of a claim is no longer of any use, and is effectively "expired". For example : claims can become effectively "expired" by us updating the lbry schema in a non backwards compatible manner. Claims can also become effectively "expired" if they no longer exist in the DHT. We don't need to take any action when these things happen, because the claims will just expire on its own.

c) storing things in the claimtrie is expensive because every full node has to bring the entire claimtrie into memory for every block in order to calculate the claimtrie hash. (this is more expensive than storing an unspent from a transaction because only the unspents that are used in the block need to be brought into memory ). If we just let it grow for all the lifespan of lbrycrd and nothing expired, eventually it will reach a point where it will take too long to calculate the claim trie hash ( Note that we have to store 32 bytes per claim). Miners can more effectively control the growth of the claim trie by imposing transaction fees IF the claims have a set expiration date. If claims never expire, you are essentially asking the miner "Can you store this thing in your memory during the entirety of lbrycrd existence", which miners will have a hard time pricing.

In conclusion, claims should definitely expire.

a) Already stated but worth stating again: Old names that people claimed with a lot of credits and lost the key for can be reclaimed b) There are several ways in which the value of a claim is no longer of any use, and is effectively "expired". For example : claims can become effectively "expired" by us updating the lbry schema in a non backwards compatible manner. Claims can also become effectively "expired" if they no longer exist in the DHT. We don't need to take any action when these things happen, because the claims will just expire on its own. c) storing things in the claimtrie is expensive because every full node has to bring the entire claimtrie into memory for every block in order to calculate the claimtrie hash. (this is more expensive than storing an unspent from a transaction because only the unspents that are used in the block need to be brought into memory ). If we just let it grow for all the lifespan of lbrycrd and nothing expired, eventually it will reach a point where it will take too long to calculate the claim trie hash ( Note that we have to store 32 bytes per claim). Miners can more effectively control the growth of the claim trie by imposing transaction fees IF the claims have a set expiration date. If claims never expire, you are essentially asking the miner "Can you store this thing in your memory during the entirety of lbrycrd existence", which miners will have a hard time pricing. In conclusion, claims should definitely expire.

kaykurokawa commented

2017-10-31 21:20:48 +01:00

(Migrated from github.com)

Also should note that the original question on this issue was:
Should expired claims be update able , as opposed to should claims expire.

Making expired claims update able is not that costly since expired claims do not enter the claim trie, they just need to be kept track of the same way an unspent transaction is.

Also should note that the original question on this issue was: Should expired claims be update able , as opposed to should claims expire. Making expired claims update able is not that costly since expired claims do not enter the claim trie, they just need to be kept track of the same way an unspent transaction is.

kauffj commented

2017-10-31 22:13:34 +01:00

(Migrated from github.com)

@kaykurokawa thank you for writing up these comments, useful and informative.

lyoshenka commented

2017-11-01 18:59:15 +01:00

(Migrated from github.com)

It seems like there's a tradeoff here between having a large claimtrie, and having to issue lots of updates to keep names from expiring. Maybe we should figure out what our claimtrie "capacity" is? e.g. how many claims can we have before we exceed some limit (either miner RAM or txns/second). Then we'll know which bottleneck we're likely to hit first.

kauffj commented

2017-11-02 22:20:06 +01:00

(Migrated from github.com)

@kaykurokawa do you also see there as being scaling issues around requiring regular claim updates? If LBRY is supporting 1e9+ pieces of content and they need to be updated every X seconds, what values make sense so that the system works from a total and transactional capacity perspective?

(At 32 bytes per claim, 1e9 pieces of content is only 32GB, which doesn't seem crazy to require of miners at that scale.)

@kaykurokawa do you also see there as being scaling issues around _requiring_ regular claim updates? If LBRY is supporting 1e9+ pieces of content and they need to be updated every X seconds, what values make sense so that the system works from a total and transactional capacity perspective? (At 32 bytes per claim, 1e9 pieces of content is only 32GB, which doesn't seem crazy to require of miners at that scale.)

kaykurokawa commented

2018-01-08 05:31:45 +01:00

(Migrated from github.com)

I think @lyoshenka is right, lets start by verifying/measuring how well the lbrycrd code responds to claimtrie size increases (memory usage, claimtrie update speed, discover any possible bugs that can occur from being stress tested, there should probably be an RPC command to return the amount of memory claimtrie is using) There has really been no testing done at all in this area so I think its necessary. This is about 40 hours of work.

The code changes to remove expiration and thorough testing will take about 80 to 120 hours worth of work

I think @lyoshenka is right, lets start by verifying/measuring how well the lbrycrd code responds to claimtrie size increases (memory usage, claimtrie update speed, discover any possible bugs that can occur from being stress tested, there should probably be an RPC command to return the amount of memory claimtrie is using) There has really been no testing done at all in this area so I think its necessary. This is about 40 hours of work. The code changes to remove expiration and thorough testing will take about 80 to 120 hours worth of work

lyoshenka commented

2018-01-16 15:44:46 +01:00

(Migrated from github.com)

@kaykurokawa could you break down the 80-120 hours into smaller items with time estimates on those items. aim for items that are a 5 at most, and break up anything bigger than that into subitems. make a separate issue for each of those subitems and add them to this epic. lemme know if you want help with this

lyoshenka commented

2018-04-25 19:59:59 +02:00

(Migrated from github.com)

Claim expiration will be removed by https://github.com/lbryio/lbrycrd/pull/115

Rows
Columns

Remove claim expiration #21