With the throughput tweaks at the backend, it seems like the number of "file is locked" errors have reduced.
The next thing to try is to reduce the chunk size, hoping that file writes would be faster, reducing the lock duration from causing a timeout.
Completely remove any assumptions of multi-tab uploading from server status (should have done it previously, but wanted to be conservative). This should make it less confusing to the user.
The real issue still remains -- the upload is somehow locked at the backend.
Also, when we override the error to present a user-friendly message to the user, pass the original error to the log (just in case it gives extra info).
From the logs, it seems like the second retry (5s) fixes the "normal" cases, so just remove the first retry (0s).
Also from the logs, if a retry doesn't work by the third attempt (10s), it's most likely the "locked" case and retrying further doesn't help. So, reduce one more useless retry attemp.
Removed logging since we've gathered enough data, and that this hook is expected to be hit a lot, so we don't want to clog the logs.
## Background
Per developer of `tus-js-client`, it is normal to occasionally encounter upload errors. The auto-retry mechanism is meant to address this.
While implementing tab-lock to prevent multiple uploads of the same file, 423_locked was used to detect this scenario. But 423_locked could also mean "the server is busy writing the chunk" (per discussion with Randy), so we kind of disabled the auto-retry mechanism accidentally.
Meanwhile, from a prior discussion with Randy, one of the chunk-writing duration took 3 minutes. Our current maximum of "retry after 15s" wouldn't help.
## Change
1. Given that tab-locking was improved recently and no longer reliant on the server error messages (we use secure storage to mark a file as locked), reverted the change to "skip retry on 409/423". This is now back to normal recommended behavior.
2. `tus-js-client` currently does not support variable retry delay, otherwise we could prolong the delay if the error was 423. Since we know it could take up to 3 minutes, and that we don't know if it's file-size dependant, just add another 30s retry and put a friendlier message asking the user to retry themselves after waiting a bit.
Our current chunk size is 25,000,000.
Google and S3 documentation suggests the chunk size to be multiples of 256KiB. MongoDB too. We aren't using any of those, but I guess no harm doing the same. From the logs, the values "25,000,000" and "50,000,000" seems to be common.
f0cd1592 did an additional call instead of replacement.
Aside: the 1 hour value only has effect in dev instances. For prod, CloudFlare seems to override that to 4 hours.
## Issue
In the original Desktop code, new strings encountered during runtime will be automatically added to the local `app-strings.json` file. The feature is unavailable in Web because writing to file would require explicit permissions.
## Change
Partially restore the functionality by saving the strings to memory and retrieving it from the console via `copy(window.new_strings)`. It's a little bit of manual work, but I think it is good as it forces a sanity check before committing (previously, experimental/developmental strings are committed and being translated in Transifex).
## Ticket
418 TUS: skip fingerprint storage
- Fingerprints for canceled uploads are not being cleared by tus-js-client. It's in localStorage, and there is limit for that.
- We are storing the confirmed fingerprint (from the backend) in redux anyway, so we don't need that functionality.
* Upload: fix redux key clash
## Issue
`params` is the "final" value that will be passed to the SDK and `channel` is not a valid argument (it should be `channel_name`). Also, it seems like we only pass the channel ID now and skip the channel name entirely.
For the anonymous case, a clash will still happen when since the channel part is hardcoded to `anonymous`.
## Approach
Generate a guid in `params` and use that as the key to handle all the cases above. We couldn't use the `uploadUrl` because v1 doesn't have it.
The old formula is retained to allow users to retry or cancel their existing uploads one last time (otherwise it will persist forever). The next upload will be using the new key.
* Upload: add tab-locking
## Issue
- The previous code does detect uploads from multiple tabs, but it was done by handling the CONFLICT error message from the backend. At certain corner-cases, this does not work well. A better way is to not allow resumption while the same file is being uploading from another tab.
- When an upload from 1 tab finishes, the GUI on the other tab does not remove the completed item. User either have to refresh or click Cancel. Clicking Cancel results in the 404 backend error. This should be avoided.
## Approach
- Added tab synchronization and locking by passing the "locked" and "removed" information through `localStorage`.
## Other considered approaches
- Wallet sync -- but decided not to pollute the wallet.
- 3rd-party redux tab syncing -- but decided it's not worth adding another module for 1 usage.
* Upload: check if locked before confirming delete
## Reproduce
Have 2 tabs + paused upload
Open "cancel" dialog in one of the tabs.
Continue upload in other tab
Confirm cancellation in first tab
Upload disappears from both tabs, but based on network traffic the upload keeps happening.
(If upload finishes the claim seems to get created)
* Fix query selection
* Fix xml format
* Fix link url and author_url
* Refactor repeated components
* Refactor repeated embed iframe string
* Add support for passing referrer queries to src
* Change iframe id from lbry to odysee
* Improve replace logic understanding
* Fix URL
Co-authored-by: Thomas Zarebczan <tzarebczan@users.noreply.github.com>
## Issue
Tom seeing crashes on the line that was trying to remove the script, saying it's not a child of that node.
## Changes
- I'm guessing the found `fjs` sometimes is not in `head`, but we always remove from `head` during cleanup. Just append to the bottom of head, and remove from head. I think script order doesn't matter if we are injecting at runtime?
- Fixed effect dependency while at it (the latest PR removed the need to check for `type`).
* coming along well
* coming along well
* adding custom react element
* coming along well
* coming along well
* coming along well
* working pretty well
* almost done
* essentially working just could use a couple touchups
* cleanup and lint errors
* fix lint errors
* fix flow errors
* possible bugfix
* dynamically set width and height
* only run when rowdata is populated
* trying using ref
* better way to check for card population
* working implementation
* working implementation
* clean up flow and clean up script
* fix typo in comment and logs
* Route recommendation search to recsys 5% of the time + add `user_id`
## Ticket
334 send some recommended requests to recsys
## Approach
`doSearch`:
- If the search options include `related_to`, route that to the new `searchRecommendations` which performs the 5% check + appends `user_id` at the end. This way, we don't need to alter the function signature of `doSearch`.
- Else, run proceed as normal.
* Always go to alt provider
f
Co-authored-by: Thomas Zarebczan <thomas.zarebczan@gmail.com>
* add gdpr support
* only run on production
* testing implementation
* just needs last touches then ready
* ready for merge
* add cookies to sidebar
* hide button when secureprivacy not available
* switch over to loading script as a react hook
* conditionally add secureprivacy script
* save gdpr status on session
* better design
* SyncFatalError: show nag instead of hard-crashing.
## Issue
When sync fails, we crash the app.
## Ticket
Maybe closes 39 "Better handle both internal and web backend interruptions / downtime"
## Approach
I'm tackling this from the standpoint that (1) sync errors are not that fatal -- we'll just lost a few recent changes (2) network disconnection is the common cause.
## Changes
- If we are offline:
- Inform user through a nag. All other status is meaningless if we are offline.
- If we are online:
- If api is STATUS_DOWN, show the existing crash page.
- If there is a sync error, show a nag saying settings are now potentially unsynchronized, and add a button to retry sync.
- If there is a chunk error, nag to reload.
* Attempt to detect `status=DOWN`
Previous code resolves the status to either "ok" or "not", which makes the app unable to differentiate between the "degraded" (nag) and "down" (crash) states.
## Issue
The TUS client automatically removes the upload fingerprint whenever there is a 4xx error. When we try to resume later, we couldn't find the the fingerprint and ended up creating a new upload ID.
## Changes
Since we are also storing the uploadUrl ourselves, provided that to override the tus client's default behavior of restarting a new session on 4xx errors.
The stalling behavior has changed a bit, probably with the removal of CF.
The stall difference between 10MB and 50MB is not too noticable, so picking 25MB as a start.
## Issue
The status = 0 is due to unresponsive backend right after the tus-upload. No root-cause found yet.
## Change
It may or may not help, but adding a delay to account for the unresponsive stage for now.
## Issue/Steps
From Randy:
- started the upload then open a new tab of the same page
- one of the tab finished the upload and successfully published the file, and the other tab received 404 error on patch and head request, because the file is already removed on the server
## Changes
Use the default onRetry code that ignores all 4xx, except for LOCKED and CONFLICT. Had to duplicate some code from tus because I still need to inject the 'retry' progress for the GUI to update the string.
## Issue
If you make 2 claims from the same source file, the second upload thinks it's trying to resume from the first one. They should be unique uploads.
## Approach
Stash the upload url for comparison when looking up existing uploads to resume.
Stash that in `params` to minimize code changes. We'll just need to ensure it is cleared before we generate the SDK payload.
* Publish button: use spinner instead of "Publishing..."
Looks better, plus the preview could take a while sometimes.
* Refactor `doPublish`. No functional change
This is to allow `doPublish` to accept a custom payload as an input (for resuming uploads), instead of always resolving it from the redux data.
* Add doPublishResume
* Support resume-able upload via tus
## Issue
38 Handle resumable file upload
## Notes
Since we can't serialize a File object, we'll need to the user to re-select the file to resume.
* Exclude "modified date" for Firefox/Android
## Issue
It appears that the modification date of the Android file changes when selected, so that file was deemed "different" when trying to resume upload.
## Change
Exclude modification date for now. Let's assume a smart user.
* Move 'currentUploads' to 'publish' reducer
`publish` is currently rehydrated, so we can ride on that and don't need to store the `currentUploads` in `localStorage` for persistence. This would allow us to store Markdown Post data too, as `localStorage` has a 5MB limit per app.
We could have also made `webReducer` rehydrate, but in this repo, there is no need to split it to another reducer. It also makes more sense to be part of publish anyway (at least to me).
This change is mostly moving items between files, with the exception of
1. An additional REHYDRATE in the publish reducer to clean up the tusUploader.
2. Not clearing `currentUploads` in CLEAR_PUBLISH.
* Restore v1 code for livestream replay, etc.
v2 (tus) does not handle `remote_url`, so the app still needs v1 for that. Since we'll still have v1 code, use v1 for previews as well.
## Issue
211 - CSS load-order problem
## Notes
It is unlikely that we'll need to support different brands in the future, so simplifying the code and number of files so that we don't have to handle the various import paths. Will probably make things easier for the css-splitting work too.
* fix type error
fix is subscribed check
- Persist subscription data locally
- add / remove subscription during log in / out
- Use store directly in hook
Add toast error if subscription fails
Revert removal of v2
hotfix linting issue
Add custom notification handler
- fix isSupported flag
- make icon color compatible with light/dark theme
- fix icon on notifications blocked banner
wip: add push notification banner to notifications page.
- ignore failed deletions via internal API
- add ua parsing package
- add more robust meta data to token save
refactor naming + add push toggle to notification button
shift some code around
update css naming o proper BEM notation
update notifications UI
remove now unneeded util function
Update push notification system to sue firebase sdk
separate service worker webpack bundling
update service worker to use firebase sdk
Add firebase config
Add firebase and remove filemanager
Stub out the basics for browser push notifications.
* fix safari
* try smaller image for badge
* add token validation with server, refactor code
* remove param
* add special icon for web notification badge
* add translations
* add missing trans for toast error
* add pushRequest method that will not prompt users who have subscribed but since disabled notifications in the settings.
* Fix CSS for live chat embeds
* Fix Markdown Lists in Comments
* Disable copy link menu option on livestream comments
* Fix nested indents in Live Chat
* Fix mentions and timestamps not parsed in bullet lists
* Highlight livestream comment and menu button on hover
* Fix mention parsing
* adding functionality to detect user download speed
* calculating bandwidth speed more intelligently
* saving download speed and updating it every 30s
* all the functionality should be done needs testing
* fix linting
* use a 1mb file for calculating bandwidth
* add optional chaining plugin to babel and get bitrate from texttrack
* allow optional chaining for flow
* ignore flow error
* disable bandwidth checking functionality
* fix flow error
## Issue
Part of `7166 improve search metadata`
## Notes
This is an experiment to influence the Sitelinks in our search results. Our current sitemap only consists of claims, so claims appear in Sitelinks more often. We (Julian) want categories to have higher priority, if possible.
For now, the sitemap will be defined in Google Console instead of robots.txt.
If it works, the file should be uploaded to sitemap.odysee.com, alongside the claim list sitemap.
## Issue
Category cards are showing up as "odysee.com" cards in Facebook.
## Change
- `og:url` is supposed to be the canonical URL. It was hardcoded to "odysee.com", so every category was being redirected when the card is being generated.
- Removed `twitter:url`. The documentation says it will fall back to `og:url`, so there is not need to define both if it's the same.
- A side-quest from "7166 improve search metadata".
- The favicon must be from the same domain as the homepage, so the CDN URL couldn't be used, hence the additional upload.
- The favicon also needs to be multiples of 48x48 and above.
- Wanted to use SVG for the smallest size possible, but seems like Safari does not fully support it. Got Dejan to give me a reasonably-sized PNG.
## Reference
https://developers.google.com/search/docs/advanced/appearance/favicon-in-search#guidelines
## Issue
7166 improve search metadata
Depending on the search term and timing, Google extracts data from the sidebar or page content to use as the search-result description.
## Change
Defined `description` (on top of the existing `og:description` and `twitter:description`.
While I couldn't find a definitive doc saying that this is the solution, this is present in all other sites (and matches their description in a Google Search results).
* Add Channel Mention selection ability
* Fix mentioned user name being smaller than other text
* Improve logic for locating a mention
* Fix mentioning with enter on livestream
* Fix breaking for invalid URI query
* Handle punctuation after mention
* Fix name display and appeareance
* Use canonical url
* Fix missing search
* Embed: add replay button
Also, changed "Rewatch or Discuss" to "Discuss + external arrow" since there is a dedicated re-watch button.
* Embed: resize "ended message" based on container height
## Issue
5848 Remove scroll-bar in update prompt
## Changes
- Make scrollbars appear only when necessary.
- Style the scrollbar. Similar to other areas, we skip this for Mac (can't recall why we need to skip).
## Issue
`https://github.com/lbryio/lbry-desktop/issues/6369#issuecomment-882081892`
## Changes
- Replace 'feed' with 'node-rss' for itunes spec support.
- Replace content type from 'rss+xml' to 'xml' so that the browser will display it nicely using the document tree without us having to re-format it ('node-rss' does not). Seems like all feeds that I found does it this way.
- There is no need to escape characters now that 'node-rss' does it. Nice.
## Issue
5951: OG messed up if `$"` is in title
The replacement string would contain `...$"...`. The character `$&` has special meaning in `replace`, so the output was messed up.
## Approach
Use a direct slice approach. A bit less elegant.
- Moved the thumbnail from 'enclosure' to 'description' by appending the thumbnail in front.
- Generate enclosure for media. It's up to readers on how they will handle it. Some will display the inline video player, some will just provide a download link.
## Changes
- Change to canonical_url (with ':').
- The old format of 'claimName/claimId' needs to be supported still, since we shipped with it.
## Notes
- It would be nice to use regex instead of 2 separate paths, but I couldn't figure out how to make the koa variables work.
## Issue
3779 RSS feed for channels
## Initial implementation details
- RSS only (not atom)
- Grabs latest 10 entries (Beamer have concerns)
## Credit
Referenced the community version mentioned in 3779
Even with the caching changes, a 150kB thumbnail still takes 1-2s to fetch. This impacts the score.
## Change
(1) Start with a large-enough placeholder image (has to be larger than the final image -- inflating doesn't count), then delay just enough for scoring, then switch to the real thumbnail.
(2) Since we are now doing post-mount stuff, we have the exact dimensions to optimize the claim thumbnail. This reduces the typically-several-MBs thumbnail to kBs.
wip
wip - everything but publish, autoplay, and styling
collection publishing
add channel to collection publish
cleanup
wip
bump
clear mass add after success
move collection item management controls
redirect replace to published collection id
bump
playlist selector on create
bump
use new collection add ui element
bump
wip
gitignore
add content json
wip
bump
context add to playlist
basic collections page style pass wip
wip: edits, buttons, styles...
change fileAuthor to claimAuthor
update, pending bugfixes, delete modal progress, collection header, other bugfixes
bump
cleaning
show page bugfix
builtin collection headers
no playlists, no grid title
wip
style tweaks
use normal looking claim previews for collection tiles
add collection changes
style library previews
collection menulist for delete/view on library
delete modal works for unpublished
rearrange collection publish tabs
clean up collection publishing and items
show on odysee
begin collectoin edit header and css renaming
better thumbnails
bump
fix collection publish redirect
view collection in menu does something
copy and thumbs
list previews, pending, context menus, list page
enter to add collection, lists page empty state
playable lists only, delete feature, bump
put fileListDownloaded back
better collection titles
improve collection claim details
fix horiz more icon
fix up channel page
style, copy, bump
refactor preview overlay properties,
fix reposts showing as floppydisk
add watch later toast,
small overlay properties on wunderbar results,
fix collection actions buttons
bump
cleanup
cleaning, refactoring
bump
preview thumb styling, cleanup
support discover page lists search
sync, bump
bump, fix sync more
enforce builtin order for now
new lists page empty state
try to indicate unpublished edits in lists
bump
fix autoplay and linting
consts, fix autoplay
bugs
fixes
cleanup
fix, bump
lists experimental ui, fixes
refactor listIndex out
hack in collection fallback thumb
bump
Since lbry.tv is going away, I tried not to touch the css file much so that the merge into `odysee` won't affect it. The side effect is that the links are now white instead of the regular link color.
## Issue
Closes 5923 video metadata issues
## Notes
For thumbnails, future claims should have correct thumbnails after `5925 Fix thumbnail-checking and block 'data:image'`. For existing claims, we'll just skip the metadata to avoid the crawler error. Users can easily update their thumbnails.
For the `OlivOliv` thumbnail url case, I wasn't sure whether to go all out to fetch and verify each thumbnail. I ended up just checking whether it starts with `http(s)`
