Jack Robison
cacbe30871
rebase
2021-10-05 16:44:49 -04:00
Jack Robison
04bb7b4919
add wrapper for getnamesintrie
...
-used for verifying db state against lbrycrd
2021-10-05 16:44:49 -04:00
Jack Robison
b7df277a5c
db state struct
...
-remove dead code
2021-10-05 16:44:49 -04:00
Jack Robison
c681041b48
claim expiration
2021-10-05 16:44:49 -04:00
Jack Robison
923834c784
get_claim_by_claim_id
2021-10-05 16:44:49 -04:00
Jack Robison
588edf98be
claims db
...
-move all leveldb prefixes to DB_PREFIXES enum
-add serializable RevertableOp interface for key/value puts and deletes
-resolve urls from leveldb
2021-10-05 16:44:49 -04:00
Jack Robison
28c603ad5f
transaction_num_mapping
2021-10-05 16:44:49 -04:00
Jack Robison
6988a47e02
disable sqlite in block processor
2021-10-05 16:44:49 -04:00
Jack Robison
2c8ceb1217
named tuples
2021-10-05 16:44:49 -04:00
Jack Robison
ccac4ffa24
consolidate flush_backup
2021-10-05 16:44:49 -04:00
Jack Robison
4258cef9bd
remove lbry.wallet.server.history
2021-10-05 16:44:49 -04:00
Jack Robison
62cc6dfe76
consolidate leveldb block advance/reorg
...
-move methods from History to LevelDB
2021-10-05 16:44:49 -04:00
Jack Robison
9f224a971b
atomic flush_dbs
2021-10-05 16:44:49 -04:00
Jack Robison
cf5dba9157
combine leveldb databases
2021-10-05 16:44:49 -04:00
Jack Robison
23035b9aa0
Merkle staticmethods
2021-10-05 16:44:49 -04:00
Victor Shyba
c2dcc4c898
avoid [''] on peers list
2021-09-13 15:57:21 -03:00
Victor Shyba
5061a35e66
remove ignored output from hub node
2021-09-10 18:57:20 -03:00
Victor Shyba
cd9a1e8c9e
default to legacy search for this release
2021-09-10 18:57:20 -03:00
Victor Shyba
b64aa51c0c
fix stream_types being an integer
2021-09-10 18:57:20 -03:00
Victor Shyba
8206441834
run CI for old and new setups
2021-09-10 18:57:20 -03:00
Victor Shyba
57dffaa2ce
update hub to beta release
2021-09-10 18:57:20 -03:00
Victor Shyba
9e81dd2360
refactor arguments fixup
2021-09-10 18:57:20 -03:00
Victor Shyba
1c31ec66f2
simplify operator handling
2021-09-10 18:57:20 -03:00
Victor Shyba
270192486a
translate grpc errors to RPCError
2021-09-10 18:57:20 -03:00
Victor Shyba
a799503c97
update fields from hub
2021-09-10 18:57:20 -03:00
Victor Shyba
9685928087
there is no first_search
2021-09-10 18:57:20 -03:00
Victor Shyba
0e4b2fad99
specify index name
2021-09-10 18:57:20 -03:00
Jeffrey Picard
046147eb1d
updates for fields
2021-09-10 18:57:20 -03:00
Jeffrey Picard
4544a074d9
Move the go hub settings from network to ledger config and hook reset
...
correctly.
2021-09-10 18:57:20 -03:00
Jeffrey Picard
9b78501392
Set default server to the networks default and use go hub by default
2021-09-10 18:57:20 -03:00
Jeffrey Picard
a4955a2b79
remove uneeded prints
2021-09-10 18:57:20 -03:00
Jeffrey Picard
15a56ca25e
tons of small changes squashed together
2021-09-10 18:57:20 -03:00
Jeffrey Picard
9f65799a3d
uncomment tests, add remove_duplicates param
...
Cleanup prints and commented out code
remove print
don't do list claims
cleanup
2021-09-10 18:57:20 -03:00
Jeffrey Picard
a97fc6dba8
cleanup and reorgnazing some stuff
...
Fixing tests
relabel failing tests properly
run all the tests for the hub
cleanup HubNode
2021-09-10 18:57:20 -03:00
Jeffrey Picard
c124e88d12
grpc client for python
2021-09-10 18:57:20 -03:00
Jeffrey Picard
17f3870296
Add tests for hub
...
Have the basic starting /stopping / querying. Still don't have the hub
jsonrpc stuff working right and from the looks of it I need to clearify
some of the logic in the claim search function itself because it's not
returning the correct number of claims anyways.
get the integration working with grpcurl
Got tests working, still need to port the rest of them
ported all of the claim search tests
still a few failing due to not having inflation working, and there's something weird
with limit_claims_per_channel that needs to be fixed.
2021-09-10 18:57:20 -03:00
Cristian Vicas
45bf6c3bf3
Drop comment_* apis.
...
Refactored dangling functions.
Added unit test.
2021-09-02 11:38:29 +03:00
Lex Berezhny
dc1d9e1c84
upgrade pylint and fix lint errors
2021-08-20 22:36:35 -04:00
Jack Robison
2445c00c7e
raise WalletNotLoadedError in get_wallet_or_error instead of ValueError
2021-08-17 11:30:58 -04:00
Victor Shyba
21c71bfac1
update sync utility
2021-08-09 18:33:47 -03:00
Victor Shyba
6baaed3581
refactor query with new fields
2021-08-09 18:33:47 -03:00
Victor Shyba
152dbfd5d1
reflect fee_currency, fee_amount and duration on repost searches
2021-08-09 18:33:47 -03:00
Victor Shyba
a56d14086b
reflect media_type on repost searches
2021-08-09 18:33:47 -03:00
Victor Shyba
aee87693f8
reflect stream_type on repost searches
2021-08-09 18:33:47 -03:00
Victor Shyba
e222b6ad9c
log that a invalid query happened
2021-08-09 15:07:44 -03:00
Victor Shyba
19b17374e8
throttle instead of disconnecting
2021-08-09 15:07:44 -03:00
Victor Shyba
43989122bb
add error type and message to error readme and update code
2021-08-09 15:07:44 -03:00
Victor Shyba
72712d6047
raise and disconnect if too many parameters are used on search
2021-08-09 15:07:44 -03:00
Victor Shyba
c591792de9
has_source is a special case
2021-07-22 16:25:55 -03:00
Victor Shyba
3108543ae5
3 missing fields
2021-07-22 16:25:55 -03:00
Victor Shyba
1eb221c743
translate reposted, signature_valid and normalized
2021-07-22 16:25:55 -03:00
Alex Grin
bebf6bc2e7
Update constants.py
2021-07-22 16:25:55 -03:00
Alex Grin
9e91cc2138
Update constants.py
2021-07-22 16:25:55 -03:00
Victor Shyba
5bd411ca27
filtering hash->id
2021-07-22 16:25:55 -03:00
Victor Shyba
a533cda6f0
ES: all _hash to _id
2021-07-22 16:25:55 -03:00
Jack Robison
9a8f9f0a94
fix stuck notification due to mempool/notification race
2021-07-20 15:14:10 -04:00
Victor Shyba
0ccafd5b53
make get_or_create_usable_address respect the generator lock
2021-07-19 14:09:52 -04:00
Victor Shyba
3de0982a4a
limit request error logging to 16k
2021-07-07 18:39:38 -03:00
Victor Shyba
c2184fb3bf
run migration on history db open
2021-07-07 18:39:38 -03:00
Victor Shyba
919c09fcb0
add migration
2021-07-07 18:39:38 -03:00
Victor Shyba
1d9dbd40ec
increase flush counter to 32 bits
2021-07-07 18:39:38 -03:00
Victor Shyba
309e957a85
add concurrent_hub_requests conf
2021-06-24 21:21:19 -03:00
Victor Shyba
d7007e402e
move request semaphore to session and apply to all requests
2021-06-24 21:02:41 -03:00
Victor Shyba
91323a21cf
add hub_timeout and propagate it to network code
2021-06-24 21:02:41 -03:00
Lex Berezhny
485b958599
revert release
2021-06-22 12:50:11 -04:00
Lex Berezhny
c39195488a
bug fix
2021-06-22 11:07:58 -04:00
Lex Berezhny
0946c72b88
lint
2021-06-22 11:07:58 -04:00
Lex Berezhny
7d49b046d4
added support to config for determining if value is set and implemented hub selection logic
2021-06-22 11:07:58 -04:00
Lex Berezhny
5f0426c840
country jurisdiction added to hub UDP protocol
2021-06-22 11:07:58 -04:00
Lex Berezhny
73e239cc5f
client side hub discovery pub/sub and hub metadata stored, removed old peers implementation
2021-06-22 11:07:58 -04:00
Lex Berezhny
ad670f721a
working client peer hub
2021-06-22 11:07:58 -04:00
Lex Berezhny
028a4a70cf
wallet server federation, client portion
2021-06-22 11:07:58 -04:00
Lex Berezhny
e457b2f0d6
fix trending to use built-in sqlite instead of apsw
2021-06-15 18:22:42 -04:00
Lex Berezhny
c9cf7fd4d4
drop apsw in wallet.server.db.elasticsearch.sync
2021-06-15 18:22:42 -04:00
Lex Berezhny
25e16c3565
dropping apsw
2021-06-15 18:22:42 -04:00
Victor Shyba
a638aa9d53
add and test support for support_create anonymous --comment
2021-06-02 12:05:36 -04:00
Victor Shyba
67cce0ef7e
test+implement --comment for support_create
2021-06-02 12:05:36 -04:00
Lex Berezhny
464cfd475e
properly format scripthash address on output
2021-06-02 11:34:21 -04:00
Lex Berezhny
fe469ae57f
create appropriate script for scripthash address
2021-06-02 11:34:21 -04:00
Lex Berezhny
550ef9a1c4
allows script addresses (beginning with r) to be used
2021-06-02 11:34:21 -04:00
Victor Shyba
3974df4a62
fix interaction between two modes
2021-05-27 20:14:12 -03:00
Victor Shyba
ca28de02d8
test and implementation for remove_duplicates on post-search filtering
2021-05-27 20:14:12 -03:00
Victor Shyba
bfc15ea029
handle limit being 0 and skip reordering if 0/none
2021-05-27 20:14:12 -03:00
Victor Shyba
6e8b8a5920
always call search_ahead
2021-05-27 20:14:12 -03:00
Victor Shyba
09f6637fe0
remove unused multiprocessin.Manager
2021-05-17 15:07:32 -03:00
Victor Shyba
ccadd88af5
fix cache call
2021-05-13 22:40:21 -03:00
Victor Shyba
cc02a0efc2
fix es migration bug, expand test case
2021-05-13 19:00:53 -03:00
Victor Shyba
43a1385b79
test sync helper
2021-05-13 19:00:53 -03:00
Jack Robison
eba0c9be34
fix typo
2021-05-07 16:51:19 -04:00
Jack Robison
81ebde88db
resync ES search index on version bumps
...
-bump ES search index to version 1
2021-05-07 14:36:53 -04:00
Victor Shyba
a4058b84ce
clean out unused sharding
2021-05-07 15:03:37 -03:00
Victor Shyba
7bf211a52b
apply reposted_claim_type on es sync
2021-05-07 15:03:37 -03:00
Victor Shyba
d5f722792f
fix and test has_source for channel reposts
2021-05-07 15:03:37 -03:00
Victor Shyba
0f02906c9b
fix has_source for reposted channels
2021-05-07 15:03:37 -03:00
Jack Robison
b5dfce7861
Revert "finished switch from using hash # in URLs to colon :"
...
This reverts commit 888aa558
2021-05-07 11:31:28 -04:00
Jack Robison
2ca5a65544
Revert "FindShortestID updated"
...
This reverts commit 8f04a50c
2021-05-07 11:30:31 -04:00
Jack Robison
32f7ecb261
fix invalid claim fees breaking the block processor
2021-05-06 11:18:58 -04:00
Jack Robison
21d0038ff2
add timestamps to hub log
2021-04-28 16:47:00 -04:00
Jack Robison
c094d8f2e8
add ALLOW_LAN_UDP hub setting
2021-04-28 16:47:00 -04:00
Jack Robison
c465d6a6c2
ignore udp packets with low source ports
2021-04-28 16:47:00 -04:00
Lex Berezhny
8f04a50ce1
FindShortestID updated
2021-04-28 16:21:01 -04:00
Lex Berezhny
888aa5586b
finished switch from using hash # in URLs to colon :
2021-04-28 16:21:01 -04:00
Victor Shyba
f0d334d3e2
refactor from review
2021-04-28 13:28:38 -03:00
Victor Shyba
5f829b048f
use separator to avoid cache key conflicts
2021-04-27 22:57:04 -03:00
Victor Shyba
1a961e66ff
invalidate short_id cache on new block
2021-04-27 22:57:04 -03:00
Victor Shyba
fdb0e22656
cache search_ahead
2021-04-27 22:57:04 -03:00
Victor Shyba
44bf4f3c8f
fix if statement from always evaluating a string
2021-04-27 17:10:04 -03:00
Lex Berezhny
2280fe8e8e
default has_source to 1
2021-04-23 14:54:51 -04:00
Lex Berezhny
575d6dcd2d
migration specifically for upgrading from client db v1.5 to v1.6
2021-04-23 14:54:51 -04:00
Lex Berezhny
f729490c6b
pending claims ordered towards top in claim_list
2021-04-23 11:00:58 -04:00
Lex Berezhny
3d4321ee38
added --has_source/--has_no_source filters to claim_list
2021-04-23 10:24:48 -04:00
Lex Berezhny
e54c31d2d5
fix bug in how reserved balance is calculated
2021-04-16 11:17:51 -04:00
Victor Shyba
cc2852cd48
new implementation for limit_claims_per_channel
2021-04-14 18:32:16 -04:00
Lex Berezhny
3cfc292d84
lint
2021-04-14 11:24:58 -04:00
Lex Berezhny
6acf94a810
moved balance calculation to SQL
2021-04-14 11:24:58 -04:00
Jack Robison
31367fb4c4
show hostnames of spvs
2021-04-13 11:51:27 -04:00
Jack Robison
12d6074e3b
fix typing
2021-04-13 11:51:27 -04:00
shubhendra
601f99ac16
Remove unnecessary generator
...
Signed-off-by: shubhendra <withshubh@gmail.com>
2021-04-06 21:22:27 -04:00
shubhendra
68399ca31c
Iterate dictionary directly
...
Signed-off-by: shubhendra <withshubh@gmail.com>
2021-04-06 21:22:27 -04:00
shubhendra
2a6d7fd80f
Remove methods with unnecessary super delegation.
...
Signed-off-by: shubhendra <withshubh@gmail.com>
2021-04-06 21:22:27 -04:00
shubhendra
4725f510d8
Remove unnecessary use of comprehension
...
Signed-off-by: shubhendra <withshubh@gmail.com>
2021-04-06 21:22:27 -04:00
shubhendra
be0ba22222
Remove unnecessary comprehension
...
Signed-off-by: shubhendra <withshubh@gmail.com>
2021-04-06 21:22:27 -04:00
Victor Shyba
46dd389d0d
add elasticsearch service to gitlab
2021-03-30 17:00:15 -04:00
Jack Robison
7832c62c5d
add ELASTIC_HOST and ELASTIC_PORT settings to hub
2021-03-30 12:48:13 -04:00
Victor Shyba
aa727cb9b1
show channels regardless of no_source
2021-03-30 09:47:08 -04:00
Victor Shyba
b8c9a99f20
fix no_source for reposts
2021-03-30 09:47:08 -04:00
Lex Berezhny
aff995b0d0
temporary fix for mempool sync failing during reorg
2021-03-29 16:11:03 -04:00
Victor Shyba
5235a150b1
add prog name to sync arg parser
2021-03-24 17:07:17 -03:00
Victor Shyba
7df4cc44c4
fixes from review
2021-03-24 16:30:33 -03:00
Victor Shyba
7f5d88e95c
remove dead/broken/unused API
2021-03-19 19:58:13 -03:00
Victor Shyba
d09663c066
remove flush call
2021-03-19 19:58:13 -03:00
Victor Shyba
ef97c9b69f
torba-server -> hub
2021-03-19 19:58:13 -03:00
Victor Shyba
d855e6c8b1
move elasticsearch things into its own module
2021-03-19 19:58:13 -03:00
Victor Shyba
cd66f7eb43
if not no_totals, use default page size
2021-03-19 19:58:13 -03:00
Victor Shyba
a3e146dc68
sort on index time
2021-03-19 19:58:13 -03:00
Victor Shyba
b81305a4a9
index and allow has_source
2021-03-19 19:58:13 -03:00
Victor Shyba
73884b34bc
apply no_totals
2021-03-19 19:58:13 -03:00
Victor Shyba
6166a34db2
check cache item before locking
2021-03-19 19:58:13 -03:00
Victor Shyba
6fa7da4b1c
less slices
2021-03-19 19:58:13 -03:00
Victor Shyba
c3e426c491
fix search by channel for invalid channel
2021-03-19 19:58:13 -03:00
Victor Shyba
21e023f0db
fix search by channel
2021-03-19 19:58:13 -03:00
Victor Shyba
063be001b3
cache inner parsing
2021-03-19 19:58:13 -03:00
Victor Shyba
5dff02e8bc
on resolve, get all claims at once
2021-03-19 19:58:13 -03:00
Victor Shyba
60a59407d8
cache the encoded output instead
2021-03-19 19:58:13 -03:00
Victor Shyba
20a5aecfca
fix lib exception to asyncio TimeoutError
2021-03-19 19:58:13 -03:00
Victor Shyba
c2e7b5a67d
restore some of the interrupt metrics
2021-03-19 19:58:13 -03:00
Victor Shyba
8f32303d07
apply search timeout
2021-03-19 19:58:13 -03:00
Victor Shyba
891b1e7782
track results up to 200
2021-03-19 19:58:13 -03:00
Victor Shyba
f26394fd3b
report deletions on docs that doesnt exist, but dont raise
2021-03-19 19:58:13 -03:00
Victor Shyba
4d83d42b4c
fix equality instead of mod
2021-03-19 19:58:13 -03:00
Victor Shyba
57f1108df2
fix query being json serializable
2021-03-19 19:58:13 -03:00
Victor Shyba
2641a9abe5
make better resolve cache
2021-03-19 19:58:13 -03:00
Victor Shyba
6b193ab350
make indexing cooperative
2021-03-19 19:58:13 -03:00
Victor Shyba
b1bb37511c
use right key on cache
2021-03-19 19:58:13 -03:00
Victor Shyba
319187d6d6
log mempool task exceptions
2021-03-19 19:58:13 -03:00
Victor Shyba
02eb789f84
caching for resolve
2021-03-19 19:58:13 -03:00
Victor Shyba
eb6924277f
round time to 10 minutes and fetch referenced by id
2021-03-19 19:58:13 -03:00
Victor Shyba
bd8f371fd5
bump referenced rows query limit up
2021-03-19 19:58:13 -03:00
Victor Shyba
1783ff2845
dont delete claims on reorg
2021-03-19 19:58:13 -03:00
Victor Shyba
d388527ffa
log indexing errors
2021-03-19 19:58:13 -03:00
Victor Shyba
19494088bd
generate from queue
2021-03-19 19:58:13 -03:00
Victor Shyba
920dad524a
simplify sync and use asyncio Queue instead
2021-03-19 19:58:13 -03:00
Victor Shyba
ec89bcac8e
improve sync script for no-downtime maintenance
2021-03-19 19:58:13 -03:00
Victor Shyba
a916c1f4ad
check if db file exists before sync
2021-03-19 19:58:13 -03:00
Victor Shyba
a9a0ac92d7
ignore unset flag
2021-03-19 19:58:13 -03:00
Victor Shyba
da8a8bd1ef
filter+fts and tests for edge cases
2021-03-19 19:58:13 -03:00
Victor Shyba
67817005b5
check ES synced without a process and wait for ES
2021-03-19 19:58:13 -03:00
Jack Robison
24d11de5a7
torba-elastic-sync
2021-03-19 19:58:13 -03:00
Victor Shyba
e12fab90d1
docker compose update
2021-03-19 19:58:13 -03:00
Victor Shyba
0a194b5b01
claim_ids query
2021-03-19 19:58:13 -03:00
Victor Shyba
8d028adc53
be a writer by default
2021-03-19 19:58:13 -03:00
Victor Shyba
dfca15395e
claim id is also a keyword
2021-03-19 19:58:13 -03:00
Victor Shyba
e21f2362fe
apply reorg deletion as well
2021-03-19 19:58:13 -03:00
Victor Shyba
1ce328e8a9
cache signature inspection
2021-03-19 19:58:13 -03:00
Victor Shyba
038a5f999f
cache encoded headers
2021-03-19 19:58:13 -03:00
Victor Shyba
5d3704c7ea
reader mode
2021-03-19 19:58:13 -03:00
Victor Shyba
87037c06c9
remove reader code
2021-03-19 19:58:13 -03:00
Victor Shyba
dd412c0f50
delete sqlite fts
2021-03-19 19:58:13 -03:00
Victor Shyba
bf44befff6
backport fixes from server
2021-03-19 19:58:13 -03:00
Victor Shyba
e61874bb6f
only repeat search if it has blocked items
2021-03-19 19:58:13 -03:00
Victor Shyba
ec9a3a4f7c
do not page filtered
2021-03-19 19:58:13 -03:00
Victor Shyba
e439a3a8dc
advanced resolve
2021-03-19 19:58:13 -03:00
Victor Shyba
19f70d7a11
create changelog trigger
2021-03-19 19:58:13 -03:00
Victor Shyba
afe7ed5b05
adjust size
2021-03-19 19:58:13 -03:00
Victor Shyba
d4bf004d74
use a thread pool to sync changes
2021-03-19 19:58:13 -03:00
Victor Shyba
e4d06a088b
include the channel being filtered/blocked
2021-03-19 19:58:13 -03:00
Victor Shyba
0929088b12
missing refresh step
2021-03-19 19:58:13 -03:00
Victor Shyba
7b4838fc9b
dont update more than 400 items a time
2021-03-19 19:58:13 -03:00
Victor Shyba
0cf9533248
narrow update by query
2021-03-19 19:58:13 -03:00
Victor Shyba
84ff0b8a9f
general timeout
2021-03-19 19:58:13 -03:00
Victor Shyba
d467dcfeaf
increase sync queue
2021-03-19 19:58:13 -03:00
Victor Shyba
8e68ba4751
fix join, refresh before update
2021-03-19 19:58:13 -03:00
Victor Shyba
0f2a85ba9f
simplify sync
2021-03-19 19:58:13 -03:00
Victor Shyba
7674a0a91e
backport fixes from testing server
2021-03-19 19:58:13 -03:00
Victor Shyba
5bc1a66572
32 slices and add censor type to fields
2021-03-19 19:58:13 -03:00
Victor Shyba
9b56067213
raise request timeout for content filtering
2021-03-19 19:58:13 -03:00
Victor Shyba
9a9df2fc3c
apply filtering only to whats unfiltered
2021-03-19 19:58:13 -03:00
Victor Shyba
9989d8d1d4
refresh after delete
2021-03-19 19:58:13 -03:00
Victor Shyba
f9471f297e
apply filter and block from ES script lang
2021-03-19 19:58:13 -03:00
Victor Shyba
146b693e4a
exclude title and description
2021-03-19 19:58:13 -03:00
