Chainquery integration

marcdeb1 commented

2018-12-21 17:47:25 +01:00

(Migrated from github.com)

In this version, the block explorer is directly connected to Chainquery to retrieve all data.
What is changed here is mainly the SQL queries because of the different structures between Chainquery data and the block explorer database. Now the block sync process becomes useless, but the local database is still used to keep Price History.
There may still be some bugs, please share them here so that I can fix them.

Some data is inaccurate, such as the block nonces for first blocks, but it's reflecting Chainquery's data, which is now the reference.
Issues due to data inaccuracy such as #3, #45, #8, #31 are now solved.

Also, it seems that the circulating supply is closer to 200M and not 212M like shown in the previous version. This is because of a repeated address in the old calculation (3eab838967/src/Controller/MainController.php (L905)) and wrong balances for addresses.

In this version, the block explorer is directly connected to Chainquery to retrieve all data. What is changed here is mainly the SQL queries because of the different structures between Chainquery data and the block explorer database. Now the block sync process becomes useless, but the local database is still used to keep Price History. There may still be some bugs, please share them here so that I can fix them. Some data is inaccurate, such as the block nonces for first blocks, but it's reflecting Chainquery's data, which is now the reference. Issues due to data inaccuracy such as #3, #45, #8, #31 are now solved. Also, it seems that the circulating supply is closer to 200M and not 212M like shown in the previous version. This is because of a repeated address in the old calculation (https://github.com/lbryio/block-explorer/blob/3eab8389671328113badb0e315fb2bfa37fb7cea/src/Controller/MainController.php#L905) and wrong balances for addresses.

👍 1

tzarebczan commented

2019-01-03 02:43:26 +01:00

(Migrated from github.com)

The repeated address would have made the circulating supply smaller, not larger. Also, balance issues only on those addresses would have affected this.

What's the total supply look like in comparison to 584,781,965 (current). 200M does sound closer if subtracting our current reserved balance from the 584M figure, given that we have a few million LBC on hot wallets.

@tiger5226 do you think this is ready to go as is?

Thanks @marcdeb1 for taking this on, can't wait to see what other cool things we can do with chainquery here!

The repeated address would have made the circulating supply smaller, not larger. Also, balance issues only on those addresses would have affected this. What's the total supply look like in comparison to `584,781,965` (current). 200M does sound closer if subtracting our current reserved balance from the 584M figure, given that we have a few million LBC on hot wallets. @tiger5226 do you think this is ready to go as is? Thanks @marcdeb1 for taking this on, can't wait to see what other cool things we can do with chainquery here!

tiger5226 commented

2019-01-03 04:23:00 +01:00

(Migrated from github.com)

@tzarebczan Yes, it is ready to put in a dev environment to play with for a week or two. There are a few known issues with the Chainquery data that the full sync @nikooo777 will be doing with his machine will solve and a few I need to solve. Once that is done, moving it to production should be fine. The price history needs to be moved to internal-apis. There is an issue for that already, I will move it up in priority so it does not block this.

The next step here is for @nikooo777 to provide a machine to run this on, exposed so we can play with it for a bit to find any issues with the data. Most of this data has been in Chainquery for some time now, however, visualizing the data like this is new. So if there are any bugs in this area we will notice them right away.

@tzarebczan Yes, it is ready to put in a dev environment to play with for a week or two. There are a few known issues with the Chainquery data that the full sync @nikooo777 will be doing with his machine will solve and a few I need to solve. Once that is done, moving it to production should be fine. The price history needs to be moved to internal-apis. There is an issue for that already, I will move it up in priority so it does not block this. The next step here is for @nikooo777 to provide a machine to run this on, exposed so we can play with it for a bit to find any issues with the data. Most of this data has been in Chainquery for some time now, however, visualizing the data like this is new. So if there are any bugs in this area we will notice them right away.

tiger5226 commented

2019-01-03 04:24:36 +01:00

(Migrated from github.com)

@marcdeb1 Looks like travis doesn't like this. Can you make sure this still works with travis with your changes? Feel free to adjust the travis.yml so it considers this new integration.

nikooo777 commented

2019-01-04 14:00:52 +01:00

(Migrated from github.com)

Could you please rebase onto master to get the latest changes in? I'm deploying this on a test server

nikooo777 commented

2019-01-04 15:04:39 +01:00

(Migrated from github.com)

you can see a running version here: https://beta.explorer.lbry.io/

👍 1

tzarebczan commented

2019-01-04 15:25:57 +01:00

(Migrated from github.com)

Sweet! Some things I noticed:

first seen is incorrect on some addresses (@tiger5226 we noticed this before and fixed one case I think).

4 | bZSJ8LYMEvxtkBNMkSGmFr9tWCgsyzb8Yv | 49,186,488.62235134 | $983,729.77 | 01 Jan 1970 00:00:00 UTC | 9.06959

blocksize chart is not working
Suppy is incorrect: https://beta.explorer.lbry.io/api/v1/supply - {"success":true,"utxosupply":{"total":null,"circulating":-381116086.0784012}}

Sweet! Some things I noticed: 1) first seen is incorrect on some addresses (@tiger5226 we noticed this before and fixed one case I think). ``` 4 | bZSJ8LYMEvxtkBNMkSGmFr9tWCgsyzb8Yv | 49,186,488.62235134 | $983,729.77 | 01 Jan 1970 00:00:00 UTC | 9.06959 ``` 2) blocksize chart is not working 3) Suppy is incorrect: https://beta.explorer.lbry.io/api/v1/supply - {"success":true,"utxosupply":{"total":null,"circulating":-381116086.0784012}}

marcdeb1 commented

2019-01-04 15:49:32 +01:00

(Migrated from github.com)

Thanks for the beta link, looks good !
@tzarebczan for the circulating supply it is close to 200M (584 - 381).
It is negative because total amount is taken from redis that might not be setup on the server. 381M is the total balance in reserved addresses.

I will check what's wrong with the blocksize chart.

Thanks for the beta link, looks good ! @tzarebczan for the circulating supply it is close to 200M (584 - 381). It is negative because total amount is taken from redis that might not be setup on the server. 381M is the total balance in reserved addresses. I will check what's wrong with the blocksize chart.

👍 1

nikooo777 commented

2019-01-04 16:06:46 +01:00

(Migrated from github.com)

Yes redis isn't configured on the server for this beta. I left the config default to use "file" rather than redis. shouldn't that work too?

tzarebczan commented

2019-01-04 18:06:35 +01:00

(Migrated from github.com)

Network on mainpage shows NA, should show GH/s.

tzarebczan commented

2019-01-04 18:53:57 +01:00

(Migrated from github.com)

When searching for "one" , the controlling claim (https://beta.explorer.lbry.io/claims/008401a48cecef2a13cf5376b696169bb36b18c6) doesn't show. We should show controlling first, mark as such, and then allow sorting of rest:

marcdeb1 commented

2019-01-05 23:06:45 +01:00

(Migrated from github.com)

Added Controlling claim tag and sorting by Controlling claim in search page.
There are several data sources in the explorer :
- Chainquery (new)
- API via JSON RPC (url in config/lbry.php) : used to get network hashrate and total LBC in circulation (nothing changed)
- Redis to cache LBC price (nothing changed)
- Local DB to store price history, only used for blocksize chart. I removed price line in the chart for the moment. To set local db, add details in config/app.php, where 'default' db is chainquery, and 'localdb' is the old one.

- Added Controlling claim tag and sorting by Controlling claim in search page. - There are several data sources in the explorer : - Chainquery (new) - API via JSON RPC (url in config/lbry.php) : used to get network hashrate and total LBC in circulation (nothing changed) - Redis to cache LBC price (nothing changed) - Local DB to store price history, only used for blocksize chart. I removed price line in the chart for the moment. To set local db, add details in config/app.php, where 'default' db is chainquery, and 'localdb' is the old one.

nikooo777 commented

2019-01-09 05:03:45 +01:00

(Migrated from github.com)

I'm having issues deploying this last version. is it working?

marcdeb1 commented

2019-01-10 11:05:22 +01:00

(Migrated from github.com)

Hey Niko, what are the issues ?

tiger5226 commented

2019-01-11 01:22:46 +01:00

(Migrated from github.com)

@marcdeb1 I just pulled the latest changes in and I am getting the following. Hope this helps:

@marcdeb1 I just pulled the latest changes in and I am getting the following. Hope this helps: <img width="1436" alt="screen shot 2019-01-10 at 7 22 07 pm" src="https://user-images.githubusercontent.com/3402064/51005503-186c0000-150d-11e9-970b-8785dced17ea.png">

nikooo777 commented

2019-01-11 17:24:31 +01:00

(Migrated from github.com)

Yep that's the issue I was getting

tzarebczan commented

2019-01-14 22:16:40 +01:00

(Migrated from github.com)

@shyba noticed that going to https://beta.explorer.lbry.io/claims?page=6360 (last page), takes a really long time to load. Not sure why that would be - maybe pagination not being called correctly somewhere?

marcdeb1 commented

2019-01-26 20:23:18 +01:00

(Migrated from github.com)

@shyba noticed that going to https://beta.explorer.lbry.io/claims?page=6360 (last page), takes a really long time to load. Not sure why that would be - maybe pagination not being called correctly somewhere?

Query takes a long time on Chainquery, try : https://chainquery.lbry.io/api/sql?query=SELECT%20*%20FROM%20claim%20LIMIT%20100%20OFFSET%20300000

> > > @shyba noticed that going to https://beta.explorer.lbry.io/claims?page=6360 (last page), takes a really long time to load. Not sure why that would be - maybe pagination not being called correctly somewhere? Query takes a long time on Chainquery, try : https://chainquery.lbry.io/api/sql?query=SELECT%20*%20FROM%20claim%20LIMIT%20100%20OFFSET%20300000

marcdeb1 commented

2019-01-26 23:08:07 +01:00

(Migrated from github.com)

Please check if there are any issues.
I cannot get Travis to work (For PHP 7.0, check is failing because some packages require 7.1, for 7.1 it's because of the database name chainquery.lbry.io that cannot be parsed).

Please check if there are any issues. I cannot get Travis to work (For PHP 7.0, check is failing because some packages require 7.1, for 7.1 it's because of the database name chainquery.lbry.io that cannot be parsed).

nikooo777 commented

2019-01-28 19:30:14 +01:00

(Migrated from github.com)

Updated! good job!
API works too: https://beta.explorer.lbry.io/api/v1/supply?/api/v1/supply

Updated! good job! API works too: https://beta.explorer.lbry.io/api/v1/supply?/api/v1/supply

marcdeb1 commented

2019-01-28 20:14:01 +01:00

(Migrated from github.com)

Awesome ! And it's really fast too

tiger5226 commented

2019-01-29 02:34:01 +01:00

(Migrated from github.com)

Very Fast!!! Very impressive! This is really exciting! I think this was the last block explorer item blocking this(besides CI). The rest are Chainquery items.

For the CI failures, I think we just need to run the composer updates for the ./composer.json. Then it should be able to find the combination of packages. It sounds like it hit a deadlock on package dependencies. @akinwale Can you take a look at the CI logs? We should make sure this is fixed before we merge.

Very Fast!!! Very impressive! This is really exciting! I think this was the last block explorer item blocking this(besides CI). The rest are Chainquery items. For the CI failures, I think we just need to run the composer updates for the `./composer.json`. Then it should be able to find the combination of packages. It sounds like it hit a deadlock on package dependencies. @akinwale Can you take a look at the CI logs? We should make sure this is fixed before we merge.

👍 1

nikooo777 commented

2019-01-30 22:01:31 +01:00

(Migrated from github.com)

Please let me know when this is ready to be merged/deployed so I can coordinate.

akinwale commented

2019-02-06 12:37:20 +01:00

(Migrated from github.com)

@nikooo777 This can be merged if there are no other changes required. The Travis build isn't a blocker.
cc @tiger5226 @tzarebczan

@nikooo777 This can be merged if there are no other changes required. The Travis build isn't a blocker. cc @tiger5226 @tzarebczan

tiger5226 commented

2019-02-06 17:54:07 +01:00

(Migrated from github.com)

@marcdeb1 For the slow query it is because offset is being used. see here https://www.eversql.com/faster-pagination-in-mysql-why-order-by-with-limit-and-offset-is-slow/

We could adjust it to use a seek strategy, but I dont think this is a blocker as the original takes 3 times as long.

@marcdeb1 For the slow query it is because offset is being used. see here https://www.eversql.com/faster-pagination-in-mysql-why-order-by-with-limit-and-offset-is-slow/ We could adjust it to use a seek strategy, but I dont think this is a blocker as the original takes 3 times as long.

nikooo777 commented

2019-02-13 19:01:40 +01:00

(Migrated from github.com)

alright I merged it, i'm deploying too. Please continue with the non blocking issues if any

🎉 1 🚀 1

Chainquery integration #49