Add chainquery dependencies for Spee.ch, does not include migrations #593

Merged
skhameneh merged 15 commits from add-chainquery into master 2018-10-10 02:12:18 +02:00
skhameneh commented 2018-09-05 00:41:56 +02:00 (Migrated from github.com)

Ready for review, uses chainquery first then falls back to local; this is needed for my next PRs for trending content and metrics.

Ready for review, uses chainquery first then falls back to local; this is needed for my next PRs for trending content and metrics.
tiger5226 (Migrated from github.com) reviewed 2018-09-05 01:22:03 +02:00
tiger5226 (Migrated from github.com) commented 2018-09-05 01:22:02 +02:00

this is only used for database migrations. Not of use to speech. I would ignore this table.

this is only used for database migrations. Not of use to speech. I would ignore this table.
tiger5226 (Migrated from github.com) reviewed 2018-09-05 01:38:51 +02:00
tiger5226 (Migrated from github.com) commented 2018-09-05 01:22:43 +02:00

This table is used for tracking background job statuses internal to chainquery. Most likely not useful for speech.

This table is used for tracking background job statuses internal to chainquery. Most likely not useful for speech.
tiger5226 (Migrated from github.com) commented 2018-09-05 01:23:52 +02:00

same thing. this is an internal table of no importance to speech.

same thing. this is an internal table of no importance to speech.
@ -0,0 +1,123 @@
const Sequelize = require('sequelize');
const logger = require('winston');
import abnormalClaimTable from './tables/abnormalClaimTable';
tiger5226 (Migrated from github.com) commented 2018-09-05 01:23:23 +02:00

Is this useful to you? This is for claims that were not made through the app. Claims that are not structured according to the lbry protocol.

Is this useful to you? This is for claims that were not made through the app. Claims that are not structured according to the lbry protocol.
tiger5226 (Migrated from github.com) commented 2018-09-05 01:25:52 +02:00

These are timestamps, is it ok to create it as an integer?

These are timestamps, is it ok to create it as an integer?
tiger5226 (Migrated from github.com) commented 2018-09-05 01:28:59 +02:00

This is a boolean that is a TINYINT(1) in the database.

This is a boolean that is a `TINYINT(1)` in the database.
tiger5226 (Migrated from github.com) commented 2018-09-05 01:29:58 +02:00

This is also an enumeration. I see that you set an enumeration above. Should this be listed as well? Active,Expired,Controlling,Spent,Accepted are the enumerations.

This is also an enumeration. I see that you set an enumeration above. Should this be listed as well? `Active,Expired,Controlling,Spent,Accepted` are the enumerations.
@ -0,0 +18,4 @@
case 'video/mp4':
return 'mp4';
default:
logger.debug('setting unknown file type as file extension jpg');
tiger5226 (Migrated from github.com) commented 2018-09-05 01:27:16 +02:00

Is jpg really a good default? Why not blank or unknown?

Is jpg really a good default? Why not blank or unknown?
tiger5226 (Migrated from github.com) commented 2018-09-05 01:28:09 +02:00

Also how do you handle nulls? A channel claim is not going to have any of the meta data fields. They will be null in the DB.

Also how do you handle nulls? A channel claim is not going to have any of the meta data fields. They will be null in the DB.
tiger5226 (Migrated from github.com) commented 2018-09-05 01:30:22 +02:00

boolean

boolean
tiger5226 (Migrated from github.com) commented 2018-09-05 01:30:46 +02:00

this is unsigned. Does it matter that you treat it as just an integer?

this is unsigned. Does it matter that you treat it as just an integer?
tiger5226 (Migrated from github.com) commented 2018-09-05 01:33:23 +02:00

boolean

boolean
@ -0,0 +49,4 @@
type: INTEGER,
set() { },
},
address_list: {
tiger5226 (Migrated from github.com) commented 2018-09-05 01:33:13 +02:00

This is a JSON array. Not sure if you can specify that here or not.

This is a JSON array. Not sure if you can specify that here or not.
tiger5226 commented 2018-09-05 01:41:30 +02:00 (Migrated from github.com)

Generally, I think there are some tables here that are just not needed for speech and are internal tables for Chainquery. They also don't fall under the major version promise of Chainquery. Those are subject to change in the future. Also there are fields that appear to not have mapped over properly. It may be intentional but I flagged them just in case.

Generally, I think there are some tables here that are just not needed for speech and are internal tables for Chainquery. They also don't fall under the major version promise of Chainquery. Those are subject to change in the future. Also there are fields that appear to not have mapped over properly. It may be intentional but I flagged them just in case.
skhameneh (Migrated from github.com) reviewed 2018-09-05 02:33:43 +02:00
@ -0,0 +18,4 @@
case 'video/mp4':
return 'mp4';
default:
logger.debug('setting unknown file type as file extension jpg');
skhameneh (Migrated from github.com) commented 2018-09-05 02:33:43 +02:00

This is the same implementation as existing code in Spee.ch; it's implemented as a generated field.
I may need to revisit this for reusability.

This is the same implementation as existing code in Spee.ch; it's implemented as a generated field. I may need to revisit this for reusability.
skhameneh (Migrated from github.com) reviewed 2018-09-05 02:36:40 +02:00
skhameneh (Migrated from github.com) commented 2018-09-05 02:36:40 +02:00

This will work for now, I need to verify that Sequelize.DATE(6) is fully compatible first.
There's a few things like INT(1) that I'm using INTEGER for as well that could probably be switched to Sequelize.BOOLEAN. I know there's a number of BIGINT as well that I need to adjust.

This will work for now, I need to verify that `Sequelize.DATE(6)` is fully compatible first. There's a few things like `INT(1)` that I'm using `INTEGER` for as well that could probably be switched to `Sequelize.BOOLEAN`. I know there's a number of `BIGINT` as well that I need to adjust.
skhameneh commented 2018-09-05 02:38:27 +02:00 (Migrated from github.com)

@tiger5226 which fall under major version promise? I'm aiming to keep this independent of the implementations on Spee.ch

I may need to rethink some of generated fields.

@tiger5226 which fall under `major version promise`? I'm aiming to keep this independent of the implementations on Spee.ch I may need to rethink some of generated fields.
skhameneh (Migrated from github.com) reviewed 2018-09-05 02:42:19 +02:00
skhameneh (Migrated from github.com) commented 2018-09-05 02:42:19 +02:00

Ha, alright! I didn't realize TINYINT(1) is exactly the same as BOOL
https://dev.mysql.com/doc/refman/5.7/en/numeric-type-overview.html

BOOL, BOOLEAN
These types are synonyms for TINYINT(1). A value of zero is considered false. Nonzero values are considered true

Ha, alright! I didn't realize `TINYINT(1)` is exactly the same as `BOOL` https://dev.mysql.com/doc/refman/5.7/en/numeric-type-overview.html > **BOOL, BOOLEAN** > These types are synonyms for TINYINT(1). A value of zero is considered false. Nonzero values are considered true
tiger5226 commented 2018-09-05 02:57:37 +02:00 (Migrated from github.com)

It is more about which ones don't. The below tables are used internally by chainquery:
gorpMigrationsTable
applicationStatusTable
jobStatusTable

It is more about which ones don't. The below tables are used internally by chainquery: gorpMigrationsTable applicationStatusTable jobStatusTable
tiger5226 (Migrated from github.com) reviewed 2018-09-26 03:56:29 +02:00
tiger5226 (Migrated from github.com) left a comment

A lot of what I commented on was probably taken from what was here already. However, I think it was still good to review those things.

A lot of what I commented on was probably taken from what was here already. However, I think it was still good to review those things.
@ -0,0 +1,198 @@
const logger = require('winston');
const returnShortId = (claimsArray, longId) => {
tiger5226 (Migrated from github.com) commented 2018-09-26 03:39:53 +02:00

Is this the canonical url?

Is this the canonical url?
tiger5226 (Migrated from github.com) commented 2018-09-26 03:42:43 +02:00

If so this doesn't seem right. You want the claim_name included here. Also the claim_id is a hash not based on the height. Ignore if this is not the case. @kauffj has a preference for how we do these shortened URLs.

If so this doesn't seem right. You want the claim_name included here. Also the claim_id is a hash not based on the height. Ignore if this is not the case. @kauffj has a preference for how we do these shortened URLs.
@ -0,0 +65,4 @@
getAllChannelClaims: async (channelClaimId) => {
logger.debug(`claim.getAllChannelClaims for ${channelClaimId}`);
return await table.findAll({
tiger5226 (Migrated from github.com) commented 2018-09-26 03:45:08 +02:00

you will want to select specific pieces of data. Otherwise you will get all of it no? The metadata is also in this table as columns, json, and as hex. You probably only want 1 of these versions for your query.

you will want to select specific pieces of data. Otherwise you will get all of it no? The metadata is also in this table as columns, json, and as hex. You probably only want 1 of these versions for your query.
@ -0,0 +82,4 @@
logger.debug(`finding claim id for claim ${claimName} from channel ${channelClaimId}`);
return await table.findAll({
where: { name: claimName, publisher_id: channelClaimId },
order: [['id', 'ASC']],
tiger5226 (Migrated from github.com) commented 2018-09-26 03:48:29 +02:00

any reason for ordering? id doesn't mean much so it may be wasteful to order the results, then again I think this is the default ordering for mysql anyway.

any reason for ordering? `id` doesn't mean much so it may be wasteful to order the results, then again I think this is the default ordering for mysql anyway.
@ -0,0 +91,4 @@
case 1:
return result[0].claim_id;
default:
// Does this actually happen??? (from converted code)
tiger5226 (Migrated from github.com) commented 2018-09-26 03:50:37 +02:00

This is a good question actually. Technically, I believe this can happen in the blockchain. In the end every claim has a bid state. So you can always have a claim for a name and you can have 100 of them. Only one can be in the Controlling state though.

This is a good question actually. Technically, I believe this can happen in the blockchain. In the end every claim has a bid state. So you can always have a claim for a name and you can have 100 of them. Only one can be in the `Controlling` state though.
@ -0,0 +101,4 @@
validateLongClaimId: async (name, claimId) => {
return await table.findOne({
where: {
name,
tiger5226 (Migrated from github.com) commented 2018-09-26 03:45:40 +02:00

why is name here?

why is name here?
@ -0,0 +117,4 @@
where: {
name,
claim_id: {
[sequelize.Op.like]: `${shortId}%`,
tiger5226 (Migrated from github.com) commented 2018-09-26 03:46:53 +02:00

This is a very expensive call. Maybe if you really need to use this, I add this feature to Chainquery so it can simply be matched against.

This is a very expensive call. Maybe if you really need to use this, I add this feature to Chainquery so it can simply be matched against.
@ -0,0 +134,4 @@
return await table.findAll({
// TODO: Limit 1
where: { name },
order: [['effective_amount', 'DESC'], ['height', 'ASC']],
tiger5226 (Migrated from github.com) commented 2018-09-26 03:52:01 +02:00

So not sure we should use this. The bid_state = Controlling is better. This information comes directly from the blockchain.

So not sure we should use this. The `bid_state` = `Controlling` is better. This information comes directly from the blockchain.
@ -0,0 +158,4 @@
resolveClaim: async (name, claimId) => {
logger.debug(`Claim.resolveClaim: ${name} ${claimId}`);
return table.findAll({
where: { name, claim_id: claimId },
tiger5226 (Migrated from github.com) commented 2018-09-26 03:52:48 +02:00

not sure why name is here like this.

not sure why name is here like this.
@ -0,0 +174,4 @@
logger.debug(`finding outpoint for ${name}#${claimId}`);
return await table.findAll({
where : { name, claim_id: claimId },
tiger5226 (Migrated from github.com) commented 2018-09-26 03:54:11 +02:00

I think I get it. You are not specifying name because they are identical. However, the claim table is unique on claim_id. So you do not need to ever you both. There is no benefit that I can think of. If there is let me know.

I think I get it. You are not specifying name because they are identical. However, the claim table is unique on `claim_id`. So you do not need to ever you both. There is no benefit that I can think of. If there is let me know.
@ -8,3 +10,3 @@
*/
const claimData = ({ ip, originalUrl, body, params }, res) => {
const claimData = async ({ ip, originalUrl, body, params }, res) => {
tiger5226 (Migrated from github.com) commented 2018-09-26 03:55:34 +02:00

I like the backup for now. Good for resilience for intial cutover.

I like the backup for now. Good for resilience for intial cutover.
skhameneh (Migrated from github.com) reviewed 2018-09-26 04:46:57 +02:00
@ -0,0 +101,4 @@
validateLongClaimId: async (name, claimId) => {
return await table.findOne({
where: {
name,
skhameneh (Migrated from github.com) commented 2018-09-26 04:46:57 +02:00

The old code validates against the name as well as the claim ID.
I kept the existing functionality - I might remove this.

The old code validates against the name as well as the claim ID. I kept the existing functionality - I might remove this.
skhameneh (Migrated from github.com) reviewed 2018-09-26 04:47:30 +02:00
@ -0,0 +65,4 @@
getAllChannelClaims: async (channelClaimId) => {
logger.debug(`claim.getAllChannelClaims for ${channelClaimId}`);
return await table.findAll({
skhameneh (Migrated from github.com) commented 2018-09-26 04:47:30 +02:00

Good call

Good call
skhameneh (Migrated from github.com) reviewed 2018-09-26 04:48:37 +02:00
@ -0,0 +82,4 @@
logger.debug(`finding claim id for claim ${claimName} from channel ${channelClaimId}`);
return await table.findAll({
where: { name: claimName, publisher_id: channelClaimId },
order: [['id', 'ASC']],
skhameneh (Migrated from github.com) commented 2018-09-26 04:48:37 +02:00

Retained from the old query, I'll remove.

Retained from the old query, I'll remove.
skhameneh (Migrated from github.com) reviewed 2018-09-26 04:50:28 +02:00
@ -8,3 +10,3 @@
*/
const claimData = ({ ip, originalUrl, body, params }, res) => {
const claimData = async ({ ip, originalUrl, body, params }, res) => {
skhameneh (Migrated from github.com) commented 2018-09-26 04:50:28 +02:00

It's also, unfortunately, required since we don't wait for chainquery to have the claim before rendering the content page. We have a number of nuances that require the fallback and prevent us from fully dropping tables.

It's also, unfortunately, required since we don't wait for `chainquery` to have the claim before rendering the content page. We have a number of nuances that require the fallback and prevent us from fully dropping tables.
skhameneh (Migrated from github.com) reviewed 2018-09-28 06:05:12 +02:00
@ -7,1 +8,4 @@
module.exports = {
'/:identifier/:claim': { controller: serveByIdentifierAndClaim, action: Actions.onHandleShowPageUri, saga: Sagas.handleShowPageUri },
'/:claim': { controller: serveByClaim, action: Actions.onHandleShowPageUri, saga: Sagas.handleShowPageUri },
};
skhameneh (Migrated from github.com) commented 2018-09-28 06:04:59 +02:00

@daovist Take a look at the action/saga binding I've added.
You can also use httpContext.set('key', value) as well as httpContext.get('key') in controllers. I'm using this to communicate route information through the stack programmatically.

@daovist Take a look at the action/saga binding I've added. You can also use `httpContext.set('key', value)` as well as `httpContext.get('key')` in controllers. I'm using this to communicate route information through the stack programmatically.
daovist (Migrated from github.com) reviewed 2018-09-28 14:57:16 +02:00
@ -7,1 +8,4 @@
module.exports = {
'/:identifier/:claim': { controller: serveByIdentifierAndClaim, action: Actions.onHandleShowPageUri, saga: Sagas.handleShowPageUri },
'/:claim': { controller: serveByClaim, action: Actions.onHandleShowPageUri, saga: Sagas.handleShowPageUri },
};
daovist (Migrated from github.com) commented 2018-09-28 14:57:16 +02:00

Is this for SSR?

Is this for SSR?
skhameneh (Migrated from github.com) reviewed 2018-10-01 17:01:39 +02:00
@ -7,1 +8,4 @@
module.exports = {
'/:identifier/:claim': { controller: serveByIdentifierAndClaim, action: Actions.onHandleShowPageUri, saga: Sagas.handleShowPageUri },
'/:claim': { controller: serveByClaim, action: Actions.onHandleShowPageUri, saga: Sagas.handleShowPageUri },
};
skhameneh (Migrated from github.com) commented 2018-10-01 17:01:39 +02:00

Some of it is

Some of it is
tiger5226 (Migrated from github.com) reviewed 2018-10-09 02:09:08 +02:00
tiger5226 (Migrated from github.com) commented 2018-10-09 02:09:08 +02:00

Not sure what Date(6) is but the db stores a DATETIME

Not sure what `Date(6)` is but the db stores a `DATETIME`
skhameneh (Migrated from github.com) reviewed 2018-10-09 02:10:35 +02:00
skhameneh (Migrated from github.com) commented 2018-10-09 02:10:35 +02:00

DATE(6) is the equivalent

`DATE(6)` is the equivalent
tiger5226 (Migrated from github.com) reviewed 2018-10-09 02:11:05 +02:00
@ -0,0 +146,4 @@
type: ENUM('Active', 'Expired', 'Controlling', 'Spent', 'Accepted'),
set() { },
},
created_at: {
tiger5226 (Migrated from github.com) commented 2018-10-09 02:11:05 +02:00

This is an old column. I need to delete this. Not sure why it is in the schema. I will create an issue for it.

This is an old column. I need to delete this. Not sure why it is in the schema. I will create an issue for it.
tiger5226 (Migrated from github.com) reviewed 2018-10-09 02:11:32 +02:00
tiger5226 (Migrated from github.com) left a comment

just a few comments

just a few comments
skhameneh (Migrated from github.com) reviewed 2018-10-09 16:58:43 +02:00
@ -0,0 +117,4 @@
where: {
name,
claim_id: {
[sequelize.Op.like]: `${shortId}%`,
skhameneh (Migrated from github.com) commented 2018-10-09 16:58:43 +02:00

It's an old query, these can be adjusted over time

It's an old query, these can be adjusted over time
skhameneh (Migrated from github.com) reviewed 2018-10-09 20:11:58 +02:00
@ -0,0 +1,123 @@
const Sequelize = require('sequelize');
const logger = require('winston');
import abnormalClaimTable from './tables/abnormalClaimTable';
skhameneh (Migrated from github.com) commented 2018-10-09 20:11:58 +02:00

Not at the moment, retained for full functionality if this is to be split out into a separate lib

Not at the moment, retained for full functionality if this is to be split out into a separate lib
skhameneh (Migrated from github.com) reviewed 2018-10-09 20:13:32 +02:00
@ -0,0 +49,4 @@
type: INTEGER,
set() { },
},
address_list: {
skhameneh (Migrated from github.com) commented 2018-10-09 20:13:32 +02:00

Can't, not supported unless using a virtual field. I might add a custom getter method.

Can't, not supported unless using a virtual field. I might add a custom getter method.
kauffj (Migrated from github.com) approved these changes 2018-10-09 20:48:58 +02:00
kauffj (Migrated from github.com) left a comment
  1. I didn't test or run this directly.
  2. I'm not familiar enough with sequalize to have strong opinions or know best practices.
  3. It does seem we shouldn't need to hand create sequalize model files since the SQL is defined by chainquery. It's worth looking at sequalize-auto or another method that doesn't require hand modifying or syncing these.
1) I didn't test or run this directly. 2) I'm not familiar enough with sequalize to have strong opinions or know best practices. 3) It does seem we shouldn't need to hand create sequalize model files since the SQL is defined by chainquery. It's worth looking at sequalize-auto or another method that doesn't require hand modifying or syncing these.
@ -0,0 +1,198 @@
const logger = require('winston');
const returnShortId = (claimsArray, longId) => {
kauffj (Migrated from github.com) commented 2018-10-09 20:45:08 +02:00

This is the existing logic, I believe. I think there are some issues with it, but we're going to handle it through https://github.com/lbryio/spee.ch/issues/184

This is the existing logic, I believe. I think there are some issues with it, but we're going to handle it through https://github.com/lbryio/spee.ch/issues/184
daovist (Migrated from github.com) approved these changes 2018-10-09 21:26:53 +02:00
daovist (Migrated from github.com) left a comment

Looks good! Some great changes in here.

(I didn't follow everything going on with sagas but I can see how to use them in routes)

Looks good! Some great changes in here. (I didn't follow everything going on with sagas but I can see how to use them in routes)
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: LBRYCommunity/spee.ch#593
No description provided.