Add support for filtering by file type #111

Closed
opened 2018-08-30 16:37:58 +02:00 by kauffj · 7 comments
kauffj commented 2018-08-30 16:37:58 +02:00 (Migrated from github.com)

I believe this involves the following:

  1. Reading the file's content type into the elastic search document. We currently store the file's media type (mime type) in the content_type column on a claim.

  2. Creating a mapping between mime/media types and more usable file types. While it's fine/good to support direct querying by media type, it is more useful to search by a generic type. So "video" might correspond to video/mp4, video/x-msvideo, etc.

Some suggested first types are: video, audio, written, CAD, archive

Presumably, these human readable types should not be written to the document, instead the API can map/unmap these more usable terms in business logic.

  1. Adding API support to perform queries by both the direct media type and the more generic types described in 2.

Acceptance Criteria

Definition of Done

  • Tested against acceptance criteria
  • Tested against the assumptions of the user story
  • The project builds without errors
  • Unit tests are written and passing
  • Tests on devices/browsers listed in the issue have passed
  • QA performed & issues resolved
  • Refactoring completed
  • Any configuration or build changes documented
  • Documentation updated
  • Peer Code Review performed
I believe this involves the following: 1) Reading the file's content type into the elastic search document. We currently store the file's media type (mime type) in the [content_type](https://github.com/lbryio/chainquery/blob/master/db/chainquery_schema.sql#L154) column on a claim. 2) Creating a mapping between mime/media types and more usable file types. While it's fine/good to support direct querying by media type, it is more useful to search by a generic type. So "video" might correspond to `video/mp4`, `video/x-msvideo`, etc. Some suggested first types are: video, audio, written, CAD, archive Presumably, these human readable types should _not_ be written to the document, instead the API can map/unmap these more usable terms in business logic. 3) Adding API support to perform queries by both the direct media type and the more generic types described in 2. ### Acceptance Criteria 1. 2. 3. ### Definition of Done - [ ] Tested against acceptance criteria - [ ] Tested against the assumptions of the user story - [ ] The project builds without errors - [ ] Unit tests are written and passing - [ ] Tests on devices/browsers listed in the issue have passed - [ ] QA performed & issues resolved - [ ] Refactoring completed - [ ] Any configuration or build changes documented - [ ] Documentation updated - [ ] Peer Code Review performed
alyssaoc commented 2018-08-30 21:30:41 +02:00 (Migrated from github.com)

@tiger5226 and @filipnyquist first we need a basic triage, is this issue workable/doable? What else do we need to know? Do you(either one or both) want to do it? There is a community member who may be interested, is this issue suitable for that situation?
We need to either:

  1. complete 111
  2. Agree that 111 is workable and that lighthouse is in reasonable shape for a new person to attempt tackling this

Thoughts?

@tiger5226 and @filipnyquist first we need a basic triage, is this issue workable/doable? What else do we need to know? Do you(either one or both) want to do it? There is a community member who may be interested, is this issue suitable for that situation? We need to either: 1) complete 111 2) Agree that 111 is workable and that lighthouse is in reasonable shape for a new person to attempt tackling this Thoughts?
tiger5226 commented 2018-08-30 21:33:08 +02:00 (Migrated from github.com)

@alyssaoc We need to to know the precise mappings referenced by @kauffj. As long as how we do this is up to do us, this should be enough information to complete the issue.

As for a new person tackling this, is this what we want? Why not have @filipnyquist or I work on it?

@alyssaoc We need to to know the precise mappings referenced by @kauffj. As long as how we do this is up to do us, this should be enough information to complete the issue. As for a new person tackling this, is this what we want? Why not have @filipnyquist or I work on it?
alyssaoc commented 2018-08-30 21:38:32 +02:00 (Migrated from github.com)

@tiger5226 That is choice one, either one of you works on it. If you want to pass, you can, just confirm that it is workable. It opens up a lot of other development opportunities and is important but not urgent.

@tiger5226 That is choice one, either one of you works on it. If you want to pass, you can, just confirm that it is workable. It opens up a lot of other development opportunities and is important but not urgent.
kauffj commented 2018-08-30 21:45:49 +02:00 (Migrated from github.com)

I'm fine with anyone doing it. I believe @filipnyquist is expected to be unavailable for a few weeks as he is beginning university.

The background here is that I encouraged @jamesbiller to seek out a technical contributor who was familiar with maker (3d printing and more) use-cases. After some discussion, we realized that a lot of desired directions are blocked by this issue, like the ability to construct views that only show CAD content. Additionally, since I consider pursuing this domain to be in the experimental area, I'm hesitant to dedicate significant amounts of core contributor time to maker-specific issues. However, this feature is clearly of broad and general benefit, so I don't think that applies here.

So basically the options are:

  1. Ensure this is well-spec'ed and use as an evaluation task
  2. Beamer tackles this now (I'm fine with this if your slate isn't too full, I think several API features are already pending work at other levels)
  3. Filip tackles once he is back to a more regular cadence
I'm fine with anyone doing it. I believe @filipnyquist is expected to be unavailable for a few weeks as he is beginning university. The background here is that I encouraged @jamesbiller to seek out a technical contributor who was familiar with maker (3d printing and more) use-cases. After some discussion, we realized that a lot of desired directions are blocked by this issue, like the ability to construct views that only show CAD content. Additionally, since I consider pursuing this domain to be in the experimental area, I'm hesitant to dedicate significant amounts of core contributor time to maker-specific issues. However, this feature is clearly of broad and general benefit, so I don't think that applies here. So basically the options are: 1) Ensure this is well-spec'ed and use as an evaluation task 2) Beamer tackles this now (I'm fine with this if your slate isn't too full, I think several API features are already pending work at other levels) 3) Filip tackles once he is back to a more regular cadence
kauffj commented 2018-08-30 21:46:31 +02:00 (Migrated from github.com)

Regarding creating the file type mappings, I think my initial list is a fine start. The engineer working on this should construct the specific mime type list.

Regarding creating the file type mappings, I think my initial list is a fine start. The engineer working on this should construct the specific mime type list.
tiger5226 commented 2018-08-31 00:40:02 +02:00 (Migrated from github.com)

@kauffj All clear! I will put this on my list. I run into situations sometimes where I am temporarily out of work for milestones and I pick from the queue. I like to have a secondary queue to fallback on. Right now that is api-forms ( did some more work there last weekend), chainquery, and lighthouse. So this is actually perfect for that fall back queue when I need to wait for responses or new issues.

@kauffj All clear! I will put this on my list. I run into situations sometimes where I am temporarily out of work for milestones and I pick from the queue. I like to have a secondary queue to fallback on. Right now that is api-forms ( did some more work there last weekend), chainquery, and lighthouse. So this is actually perfect for that fall back queue when I need to wait for responses or new issues.
tiger5226 commented 2019-02-09 23:33:37 +01:00 (Migrated from github.com)

merged and deployed with #144

merged and deployed with #144
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: LBRYCommunity/lighthouse.js#111
No description provided.