Victor Shyba
|
fdb0e22656
|
cache search_ahead
|
2021-04-27 22:57:04 -03:00 |
|
Victor Shyba
|
cc2852cd48
|
new implementation for limit_claims_per_channel
|
2021-04-14 18:32:16 -04:00 |
|
shubhendra
|
601f99ac16
|
Remove unnecessary generator
Signed-off-by: shubhendra <withshubh@gmail.com>
|
2021-04-06 21:22:27 -04:00 |
|
shubhendra
|
68399ca31c
|
Iterate dictionary directly
Signed-off-by: shubhendra <withshubh@gmail.com>
|
2021-04-06 21:22:27 -04:00 |
|
Victor Shyba
|
46dd389d0d
|
add elasticsearch service to gitlab
|
2021-03-30 17:00:15 -04:00 |
|
Jack Robison
|
7832c62c5d
|
add ELASTIC_HOST and ELASTIC_PORT settings to hub
|
2021-03-30 12:48:13 -04:00 |
|
Victor Shyba
|
aa727cb9b1
|
show channels regardless of no_source
|
2021-03-30 09:47:08 -04:00 |
|
Victor Shyba
|
b8c9a99f20
|
fix no_source for reposts
|
2021-03-30 09:47:08 -04:00 |
|
Victor Shyba
|
5235a150b1
|
add prog name to sync arg parser
|
2021-03-24 17:07:17 -03:00 |
|
Victor Shyba
|
7df4cc44c4
|
fixes from review
|
2021-03-24 16:30:33 -03:00 |
|
Victor Shyba
|
d09663c066
|
remove flush call
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
d855e6c8b1
|
move elasticsearch things into its own module
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
cd66f7eb43
|
if not no_totals, use default page size
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
a3e146dc68
|
sort on index time
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
b81305a4a9
|
index and allow has_source
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
73884b34bc
|
apply no_totals
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
6166a34db2
|
check cache item before locking
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
6fa7da4b1c
|
less slices
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
c3e426c491
|
fix search by channel for invalid channel
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
21e023f0db
|
fix search by channel
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
063be001b3
|
cache inner parsing
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
5dff02e8bc
|
on resolve, get all claims at once
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
60a59407d8
|
cache the encoded output instead
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
20a5aecfca
|
fix lib exception to asyncio TimeoutError
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
8f32303d07
|
apply search timeout
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
891b1e7782
|
track results up to 200
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
f26394fd3b
|
report deletions on docs that doesnt exist, but dont raise
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
4d83d42b4c
|
fix equality instead of mod
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
57f1108df2
|
fix query being json serializable
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
2641a9abe5
|
make better resolve cache
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
6b193ab350
|
make indexing cooperative
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
b1bb37511c
|
use right key on cache
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
02eb789f84
|
caching for resolve
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
eb6924277f
|
round time to 10 minutes and fetch referenced by id
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
bd8f371fd5
|
bump referenced rows query limit up
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
1783ff2845
|
dont delete claims on reorg
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
d388527ffa
|
log indexing errors
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
19494088bd
|
generate from queue
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
920dad524a
|
simplify sync and use asyncio Queue instead
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
ec89bcac8e
|
improve sync script for no-downtime maintenance
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
a916c1f4ad
|
check if db file exists before sync
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
a9a0ac92d7
|
ignore unset flag
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
da8a8bd1ef
|
filter+fts and tests for edge cases
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
67817005b5
|
check ES synced without a process and wait for ES
|
2021-03-19 19:58:13 -03:00 |
|
Jack Robison
|
24d11de5a7
|
torba-elastic-sync
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
e12fab90d1
|
docker compose update
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
0a194b5b01
|
claim_ids query
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
8d028adc53
|
be a writer by default
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
dfca15395e
|
claim id is also a keyword
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
e21f2362fe
|
apply reorg deletion as well
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
5d3704c7ea
|
reader mode
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
87037c06c9
|
remove reader code
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
dd412c0f50
|
delete sqlite fts
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
bf44befff6
|
backport fixes from server
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
e61874bb6f
|
only repeat search if it has blocked items
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
ec9a3a4f7c
|
do not page filtered
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
e439a3a8dc
|
advanced resolve
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
19f70d7a11
|
create changelog trigger
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
d4bf004d74
|
use a thread pool to sync changes
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
e4d06a088b
|
include the channel being filtered/blocked
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
0929088b12
|
missing refresh step
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
7b4838fc9b
|
dont update more than 400 items a time
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
0cf9533248
|
narrow update by query
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
84ff0b8a9f
|
general timeout
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
d467dcfeaf
|
increase sync queue
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
8e68ba4751
|
fix join, refresh before update
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
0f2a85ba9f
|
simplify sync
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
7674a0a91e
|
backport fixes from testing server
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
5bc1a66572
|
32 slices and add censor type to fields
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
9b56067213
|
raise request timeout for content filtering
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
9a9df2fc3c
|
apply filtering only to whats unfiltered
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
9989d8d1d4
|
refresh after delete
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
f9471f297e
|
apply filter and block from ES script lang
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
146b693e4a
|
exclude title and description
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
e2441ea3e7
|
use prefix from ES docs
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
119e51912e
|
fix partial id
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
dd950f5b0d
|
tag can have empty space
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
0c6eaf5484
|
fix resolve partial id
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
1010068ddb
|
disable refresh interval. start with 3 shards
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
ee7b37d3f3
|
also normalize the name supplied by user
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
143d82d242
|
normalized, not normalized_name
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
8b91b38855
|
update winners in one go
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
1098f0d2a3
|
use normalized name instead
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
ab53cec022
|
fix is_controlling sync
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
90106f5f08
|
all test_claim_commands tests green
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
9924b7b438
|
reposts and tag inheritance
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
aa37faab0a
|
use porter analyzer with weights on full text search
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
dc10f8ce72
|
ignore errors when deleting
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
996686c1da
|
claim search and resolve translated to ES queries
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
488785d013
|
add indexer task
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
3abdc01230
|
index ES during sync
|
2021-03-19 19:58:13 -03:00 |
|
Victor Shyba
|
74df4fab83
|
change column to has_source and document both flags
|
2021-03-14 10:11:42 -04:00 |
|
Victor Shyba
|
1e5cd3d7a1
|
typo, fix tests
|
2021-03-14 10:11:42 -04:00 |
|
Victor Shyba
|
a54e9b64aa
|
add no_source claim_search filter
|
2021-03-14 10:11:42 -04:00 |
|
Lex Berezhny
|
b153e4bb9f
|
added support to claim_search for filtering collections via --claim_type
|
2021-02-16 11:08:54 -05:00 |
|
Jack Robison
|
0758b85179
|
skip null args given to channel_ids in claim search
|
2020-12-29 21:41:07 -05:00 |
|
Jack Robison
|
bbed9b94c1
|
null language filtering for claim search
|
2020-11-05 21:23:49 -05:00 |
|
Brendon J. Brewer
|
1cdff47477
|
Rewrite of variable_decay.py for speed improvements
|
2020-11-05 21:15:55 -05:00 |
|
Jack Robison
|
925a458abe
|
tags
|
2020-11-03 10:39:08 -05:00 |
|
Jack Robison
|
46dc15dd29
|
use language indexes
|
2020-09-28 14:45:51 -04:00 |
|
Jack Robison
|
8dc654b513
|
add language table and indexes
|
2020-09-28 14:45:51 -04:00 |
|
Jack Robison
|
7000ac3f3f
|
update indexed tags
|
2020-09-28 14:45:51 -04:00 |
|
Lex Berezhny
|
ec24ebf2cf
|
added --limit_claims_per_channel argument to claim_search to only return up to the specified number of claims per channel
|
2020-08-19 10:51:31 -04:00 |
|
Tom
|
49458d1085
|
fix: reposts being returned for single tags
|
2020-04-22 23:16:30 -04:00 |
|
Jack Robison
|
7ffdfd12f8
|
faster not tags
|
2020-04-17 10:47:01 -04:00 |
|
Jack Robison
|
cb9a30f285
|
faster query
|
2020-04-17 10:47:01 -04:00 |
|
Jack Robison
|
87cdf1e3a0
|
improve test_reorg_change_claim_height
|
2020-04-05 16:58:36 -04:00 |
|
Jack Robison
|
64f7f837e7
|
delete claims above reorg height from the database
|
2020-04-03 13:39:44 -04:00 |
|
Brendon J. Brewer
|
5f043b9a78
|
variable decay
|
2020-03-12 10:33:15 +13:00 |
|
Lex Berezhny
|
3cbeadfbc3
|
lbrynet status command returns features available of currenty connected wallet server, including thetrending algorithm used
|
2020-03-03 20:04:57 -05:00 |
|
Lex Berezhny
|
33679b56fd
|
update claim_type_* indexes to be more peformant
|
2020-02-27 12:07:38 -05:00 |
|
Lex Berezhny
|
8c7908b200
|
reposts inherit the tags of the original claim
|
2020-02-22 12:23:11 -05:00 |
|
Lex Berezhny
|
d57900a069
|
claim_search --not_channel_ids excludes the channel itself and not just claims in the channel
|
2020-02-21 21:59:46 -05:00 |
|
Lex Berezhny
|
0b2d9e15b4
|
added order bys for fee_amount
|
2020-02-17 17:24:48 -05:00 |
|
Lex Berezhny
|
d14e5e75e8
|
all three order bys for duration filter
|
2020-02-17 17:24:48 -05:00 |
|
Lex Berezhny
|
df11ef34d2
|
index cleanup
|
2020-02-17 17:24:48 -05:00 |
|
Lex Berezhny
|
4fc619262c
|
replacing indexes on trending_global, trending_mixed to be trending_group, trending_mixed
|
2020-02-17 17:24:48 -05:00 |
|
Lex Berezhny
|
71cb03345b
|
removed default order_by for claim_search
|
2020-02-17 16:26:29 -05:00 |
|
Jack Robison
|
78eef25f1a
|
add reposted_claim_hash index
fixes https://github.com/lbryio/lbry-sdk/issues/2782
|
2020-02-12 17:39:57 -05:00 |
|
Lex Berezhny
|
605fc8ecd8
|
claim_search by --claim_type=repost support
|
2020-02-12 11:51:35 -05:00 |
|
Lex Berezhny
|
be4515ec9a
|
use sets for __in constraint values
|
2020-02-12 10:39:25 -05:00 |
|
Brendon J. Brewer
|
3697d9e1f0
|
Tweaked some parameters, back to 134
|
2020-02-09 17:38:34 -05:00 |
|
Brendon J. Brewer
|
94524f7330
|
lint
|
2020-02-09 17:38:34 -05:00 |
|
Brendon J. Brewer
|
17d8a3e5d6
|
Use claim_hash not claim_id
|
2020-02-09 17:38:24 -05:00 |
|
Brendon J. Brewer
|
39996d7612
|
Only create file during install if logging is turned on
|
2020-02-09 17:37:55 -05:00 |
|
Lex Berezhny
|
9607d21828
|
censored searches/resolves include metadata of channel which did the censoring
|
2020-02-07 18:50:29 -05:00 |
|
Lex Berezhny
|
1e81d6f848
|
added database index for various channel_hash based queries
|
2020-02-03 20:28:39 -05:00 |
|
Lex Berezhny
|
bfe711bd42
|
fixing tests
|
2020-02-01 13:29:55 -05:00 |
|
Lex Berezhny
|
15abf49211
|
censored resolve responses return appropriate error
|
2020-02-01 12:53:39 -05:00 |
|
Brendon J. Brewer
|
22bca7a16e
|
Length zero could also just mean there aren't any claims
|
2020-01-31 07:34:43 +13:00 |
|
jessop
|
73c958222b
|
add duration to claim search params
|
2020-01-29 15:46:47 -05:00 |
|
Lex Berezhny
|
790bbfcc99
|
integrate ar trending algorithm into configurable trending system
|
2020-01-28 18:11:32 -05:00 |
|
Brendon J. Brewer
|
994e70d43a
|
ar trending algorithm
|
2020-01-28 18:11:32 -05:00 |
|
Jack Robison
|
3b2d635390
|
Revert "Revert "configurable trending algorithms""
This reverts commit 55e08f96a6.
|
2020-01-28 15:53:57 -05:00 |
|
Jack Robison
|
55e08f96a6
|
Revert "configurable trending algorithms"
|
2020-01-28 15:03:09 -05:00 |
|
Lex Berezhny
|
01ed71b243
|
configurable trending
|
2020-01-28 11:58:27 -05:00 |
|
Lex Berezhny
|
595c058517
|
moved trending.py to trending/zscore.py
|
2020-01-28 11:58:27 -05:00 |
|
Lex Berezhny
|
8d8a5b36b6
|
Merge pull request #2737 from lbryio/fix_repost_resolve_bug
fix bug where repost claims were not being fully resolved
|
2020-01-22 23:32:55 -05:00 |
|
Lex Berezhny
|
a70cc7beb1
|
fix bug where repost claims were not being fully resolved
|
2020-01-22 22:59:05 -05:00 |
|
Lex Berezhny
|
bb23f509d7
|
rename BLOCKING_CHANNELS_IDS/FILTERING_CHANNELS_IDS to remove double S
|
2020-01-22 11:02:01 -05:00 |
|
Lex Berezhny
|
2cd7ea257c
|
Added support to differentiate between filtering and blocking for content censoring
|
2020-01-22 08:39:14 -05:00 |
|
Lex Berezhny
|
9d79c52d20
|
add index for claim_type and release_time
|
2020-01-20 12:50:20 -05:00 |
|
Lex Berezhny
|
801f05f45e
|
fix json api generator
|
2020-01-20 12:50:16 -05:00 |
|
Lex Berezhny
|
345196aa3b
|
add pagination for claim_search
|
2020-01-20 12:49:42 -05:00 |
|
Lex Berezhny
|
86cedfe8b2
|
using multiprocessing.Manager to keep blocked content synced between readers
|
2020-01-20 12:49:41 -05:00 |
|
Lex Berezhny
|
fb1af9e3d2
|
update imports and more merging
|
2020-01-03 03:08:15 -05:00 |
|
Lex Berezhny
|
2968f74c6c
|
moved lbry up one level
|
2020-01-01 15:57:56 -05:00 |
|