integration testing scripts #64
No reviewers
Labels
No labels
consider soon
documentation
good first issue
hacktoberfest
help wanted
priority: blocker
priority: high
priority: low
priority: medium
type: bug
type: discussion
type: improvement
type: new feature
type: refactor
type: task
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: LBRYCommunity/herald.go#64
Loading…
Reference in a new issue
No description provided.
Delete branch "integration_testing"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
some scripts for integration testing and a docker file for an action. Still need to figure out how to properly run a more realistic version in ci.
Also, this appears to fall back on
/mnt/d/data/snapshot_1072108/lbry-rocksdb/which I still don't know how to obtain, and I don't know how the GitHub runner gets a copy. I can live without it for now, but it would be great to be able to run locally.This is missing herald launch, and calling integration_tests.sh.
Suggest substituting the value here, and eliminating
CHUNK_TEST_RESas it's not reused by other tests.Greater than or equal to (
-ge). At least for the one test using this, you wouldn't want GOT=0 to pass.@ -0,0 +164,4 @@--data '{"id": 1, "method": "blockchain.scripthash.get_mempool", "params":[{"scripthash": "bGqWuXRVm5bBqLvLPEQQpvsNxJ5ubc6bwN"}]}'| jq .error | sed 's/"//g'EOMWANT="encoding/hex: invalid byte: U+0047 'G'"Yeah.... the scripthash ones are tricky. I don't have an example of constructing one. Lbry-sdk does not send them.
I haven't figured it out yet, but see:
https://github.com/btcsuite/btcd/search?p=1&q=AddressScriptHash&type=Code
@ -0,0 +164,4 @@--data '{"id": 1, "method": "blockchain.scripthash.get_mempool", "params":[{"scripthash": "bGqWuXRVm5bBqLvLPEQQpvsNxJ5ubc6bwN"}]}'| jq .error | sed 's/"//g'EOMWANT="encoding/hex: invalid byte: U+0047 'G'"Just found this. Purports to convert addr to scripthash:
https://github.com/checksum0/go-electrum/blob/b862ac442cf9/electrum/address.go#L14
This definitely requires a local and complete snapshot of the current db at this point. Which requires running your own hub to have in general. Another big piece of what else I'm working on, but not something that in general will be documented here. Getting this running as part of a ci runner we do need to figure out, there's a dependency on hundreds of gigabytes of data to do a "real" smoke test on, so it's not clear the best way to do this.
I think I've just transposed the WANT and GOT, but the
-gtis correct. I want a number greater than 0.Actually, nvm, you are right.
This should be before
db.NewIteratorCF()and possibly beforegrocksdb.NewDefaultReadOptions().This whole block (
if opts.DB != nil {...) should be afterit.Close()and possibly afterro.Destroy().If a thread is waiting on
ShutdownChanindb.Shutdown()and holding the lock, this line (364) can't be reached. Send the done message just before locking, sodb.Shutdown()can proceed.Position
[0]isDoneChan, and position[1]isShutdownChan. I think you mean to send toShutdownChan, then receive fromDoneChan.I think this scheme could be simplified by having a single bidirectional chan. Send the shutdown message, then receive the done message on the same chan.
My first thought: Yes of course. But...
...on investigating how this works,
ShutdownChanworks to stopRunDetectChanges(). Since that's a source of activity that may spawn new iterators, it's a good idea to stop that first so no new iterators are created. Also, consider waiting to receive the done message (<- db.DoneChan) before waiting on the iterators to be done.The real grocksdb shutdown stuff that would break running iterators probably lies in
db.Cleanup().II agree i used the wrong channels. I don't think you can really do the bi-directional thing though. It would create a new race conditional where I would send the notification then immediately read it off before the thread it was intended for did.
I agree.
sure
good call
OK
The
opts.DoneChan <- struct{}{}is still inside the critical section. It should be beforeLock().Edit: Oops, I didn't notice. It is outside the critical section. But it needs to be before.
Edit Edit: It moved from after the critical section to inside in the last change I think. The code displayed above is the original form where the done message is sent after the critical section.
I don't see anything using
WithDB, so the new logic never becomes active. Perhaps makeNewIterateOptions()takedbas an argument?Good call. I made the first change, and did changed the existing NewIterateOptions to use WithDB
One more tiny thing... At this point, it might be that db.Shutdown() has already been called. Because we are not yet in the map, the interruptRequested() func can't tell us anything. There should be a bool in DB that tells us Shutdown() has been called and no more registrations are being accepted. Set it in Shutdown() and check it here, exiting early if needed.