Repository navigation
Conversation
Yeah, it's a junky boyyy... A lot of the added lines are test vectors. But lots of Go code as well, I agree. |
351d6df to
3f5e7b6
Compare
|
I ran a full sweep with Kimi K3 that found 19 issues (1 critical, 6 high, 5 medium and 7 low) which are now fixed. The two READMEs now point out exactly what the supported features and known divergences are against Core and/or |
|
It’s a nice feature! Do you think it would be feasible to split this PR into a few smaller ones without losing too much context? That way we could review it in individual chunks. The new GitHub stacked PRs feature also seems like a good fit for this kind of change. |
Yes, you're right, splitting up the PR does make sense. I'll give it a try. |
|
I noticed |
Okay, PRs are now split. Only two parts, but way smaller commits each. I wanted to avoid needing to close this one, so I didn't end up using the stacked PR feature directly. But then I made a mistake which required me to close part 1 and re-open again anyway, so it wouldn't have mattered in the first place 😅 |
Yes, the feature parity with |
f4cc1de to
59961ba
Compare
59961ba to
e4a68fc
Compare
e4a68fc to
bd248cd
Compare
|
I ran a full security and correctness review with GPT 5.6 Sol and fixed a couple of issues (here and in the miniscript PR). @rogeriob2br the latest version also fully supports hash preimages in the |
8eef6ef to
c4161c4
Compare
|
@Lrifton92 thank you very much for your review and for reporting the problem. I also did a full differential diff of all test vectors against the latest |
c4161c4 to
45bfbdf
Compare
Coverage Report for CI Build 37457660869Coverage increased (+1.0%) to 56.721%Details
Uncovered Changes
Coverage Regressions19 previously-covered lines in 3 files lost coverage.
Coverage Stats
💛 - Coveralls |
fametrano
left a comment
There was a problem hiding this comment.
I reviewed 94b7925, on top of #2592 at 45bfbdf, against Bitcoin Core (master 4bacf21a13, and the 31.1 release), the BIP380-389 test vectors and the Python descriptors in btclib-wallet (efa2d7d). Apart from what the README lists and what I raised on #2592, they agree, with three exceptions:
- A key repeated under another spelling passes
NewDescriptor, and then no address derives. - Hybrid public keys (prefix 06 or 07) are accepted.
- A repeated key is rejected in a
multi_aleaf but accepted inmulti().
I ran the BIP vectors, Core's descriptor_tests.cpp cases, and 20,000 seeded random descriptors compared with btclib-wallet on the scriptPubKeys at indexes 0-2 of every path. I also passed the descriptors below to getdescriptorinfo on a Core 31.1 regtest node, with the WIF and xpub re-encoded for testnet.
1. A key repeated under another spelling passes NewDescriptor, and then no address derives
miniscript.go:607 compares keys as written. Derived keys are compared only in ApplyVars, at derivation, and checkMsSanity adds nothing. So NewDescriptor accepts each of these, AddressAt fails with "duplicate key found at pk_k", and MaxWeightToSatisfy still returns 217. Here K is 03a34b99f22c790c4e36b2b3c2c35a36db06226e41c692fc82b8b56ac1c540c5bd and X is BIP387's xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL:
wsh(and_v(v:pk(L4rK1yDtCWekvXuE6oXD9jCYfFNV2cWRpVuPLBcCU2z8TrisoyY1),pk(K))), the WIF and the hex of one key;wsh(and_v(v:pk([deadbeef/1]K),pk(K)));wsh(and_v(v:pk(X/0/*),pk([01020304/5]X/0/*)));wsh(and_v(v:pk(X/<0;1>/*),pk(X/0/*))), where path 0 fails and path 1 derives.
Core 31.1 rejects all four at parse with "contains duplicate public keys", because its KeyCompare (descriptor.cpp:2304) compares derived keys. btclib-wallet rejects them too. This contradicts the README's "A descriptor that parses can be paid to". checkMsSanity could compare the derived keys of each path.
2. Hybrid public keys are accepted
pk(07a34b99…a235) and pkh(07a34b99…a235) parse, and the script holds the 04 encoding of the key instead (key.go:588): pk(07…) gives 4104a34b…ac. With the wrong parity, pk(06a34b99…a235) parses and then derivation fails. BIP380 allows only 02, 03 and 04. Core rejects these ("Hybrid public keys are not allowed", also among its descriptor_tests.cpp unparsable cases), and so does btclib-wallet. key.go:172-173 checks only the length; checkRawKey could reject any other prefix.
3. A repeated key is rejected in a multi_a leaf but accepted in multi()
tr(K,multi_a(1,A,A)) and tr(K,sortedmulti_a(1,A,A)) fail with "duplicate key". The leaf is parsed as miniscript (descriptor.go:816), and the sanity check refuses it. wsh(multi(1,A,A)), sh(multi(1,A,A)) and bare multi(1,A,A) are accepted. Core 31.1 accepts all five, and so does btclib-wallet. Either accept the leaf, or add it to the divergence table.
|
All three points from my review are fixed at 5bc4530. I re-ran the review's descriptors: the four repeated-key cases and the hybrid keys are now rejected by |
45bfbdf to
9c290de
Compare
|
@fametrano thank you very much for the reviews! The original version oriented itself toward |
A descriptor names its keys through key expressions: a raw public key, a WIF private key, or an extended key with an optional origin and a derivation path that may end in a wildcard, and that may hold a multipath element. This parses them and derives the concrete key of a path, which is the part of a descriptor that turns one string into the many scripts of a wallet.
Output script descriptors carry an optional checksum which catches the transcription errors a raw descriptor string is prone to: any single character substitution, and a swap of characters that are not too far apart. BIP380 specifies it. This is the first piece of the descriptors package, and the only one that needs nothing else.
The descriptor grammar of BIP380 and its script expressions: the key based ones (pk, pkh, wpkh), the wrappers (sh, wsh), the multisigs (multi, sortedmulti) and a miniscript expression as the script of a wsh. Which expression may appear where is part of the grammar, so the parser tracks the position it is in and rejects, say, a wsh inside a wsh. The result is a tree of nodes with the keys of the descriptor collected in the order they appear. Nothing is derived from it yet.
A tr() descriptor has an internal key and an optional script tree, written as a nested pair of branches with a script expression at every leaf. BIP386 defines it. Parsing the tree is what the rest of this commit is about, including the two limits that keep a pathological descriptor from exhausting the process: the depth of the tree, which BIP341 caps at 128 anyway, and the depth of the nesting of the descriptor itself.
With the tree parsed and the keys derivable, a descriptor can produce what a wallet needs from it: the address of the output at a derivation index, and the script code that a signature over that input commits to. Both are per multipath and derivation index, and both compile a miniscript expression through the package of the previous commits where the descriptor holds one.
What a spend of a descriptor's output costs, in weight units: the witness or scriptSig that satisfies it, sized without the signatures being available. The estimate is the one a fee calculation needs, and for a taproot descriptor it is the cheapest of the key path and the leaves of the script tree, since that is the path a spender would take.
The redeem script of a P2SH output is pushed as a single element of the spending scriptSig, so the 520-byte limit on a script element applies to it: an output whose redeem script is larger can never be spent, and a descriptor that describes one is a mistake worth rejecting at parse time. The same holds for the number of keys of a bare multisig, which the standardness rules cap at three.
The normative vectors of BIP380 to BIP386: the descriptors those documents list as valid or invalid, with the scripts and addresses they must produce. The features this package does not support are classified explicitly rather than skipped silently, so that the gaps are visible in the test output instead of hiding in a skip.
The semantic policy of a whole descriptor rather than of a single miniscript: a taproot descriptor lifts to the choice between its key path and its leaves, a multisig to a threshold, and a wsh to the policy of its script. It is what a wallet shows a user as the spending conditions of an address.
A plan is the spending path a descriptor takes given what a signer holds: which signatures, preimages and time locks are available decides which branch of the descriptor can be satisfied, and of the branches that can be, the cheapest one is chosen. The plan carries the weight of that satisfaction and knows how to complete it once the real signatures arrive, which is the two-phase flow a wallet needs to build a transaction, size its fee and only then sign it.
For a taproot descriptor the plan chooses between the key path and every leaf of the script tree, which requires the control block of a leaf and therefore its merkle proof. The proofs are collected in one walk over the tree, and the cheapest of the candidates wins, as a spender would have it.
The differential vectors come from rust-miniscript and descriptors-go: for about 500 descriptors they compare the canonical string, the type, the keys, the addresses of a grid of derivation indices, the script code, the satisfaction weight, the lifted policy and the plan. Everything this package computes about a descriptor is in there, which is what makes the comparison worth having.
The descriptor parser takes strings from the outside as well, so it is fuzzed like the miniscript one, with the invariant that everything a descriptor derives stays consistent with the string it round trips to. The benchmarks cover parsing, address derivation and planning, and the recursion tests pin that a deeply nested descriptor is rejected rather than crashing the process.
A reader judging this package needs to know what it does, what it does not, and where it deliberately differs from Bitcoin Core and rust-miniscript. That is what these two documents are: the supported expressions, the unsupported ones, the divergences with the commit of each implementation they were compared against, and the behaviours that are worth knowing before using the API.
Add implementation-independent JSON fixtures and a public-API Go harness for planning and completing descriptor spends. Cover exact scripts and stacks, size estimates, derivation, negative completions, fixed-path selection and serialization boundaries, including independently signed transaction verification cases. Document the vector format and retain its standalone generator.
9c290de to
6eafd22
Compare

I wasn't able to reopen the old PR, so I'm resurrecting it here.
After using https://github.com/benma/descriptors-go in a mobile project, the limitations of using a WASM interpreter showed to be too restricting. So I'm giving it another try, this time with quite a bit of agentic help (I gave up on #1987 because I'm not smart enough for compiler-level stuff, but apparently Claude is quite capable).
Description
Depends on #2592.
Part 1 added a
miniscriptlibrary (originally authored by @benma in https://github.com/benma/miniscript-go) and this second part now adds full Output Descriptor support on top.The code is modeled after the
rust-miniscriptimplementation and a lot of test vectors for diferential unit tests were extracted from that implementation.The
descriptorsfeatures are currently limited to feature parity with https://github.com/benma/descriptors-go.Implementation disclaimer
The code in this PR is more or less a supervised re-implementation of
rust-miniscript. I spent a lot of tokens on differential code review and bug hunting, including large amounts of test vectors extracted from the original Rust code.There are also seeded fuzz tests that have been running for several hours on a 12-core machine.
All commits are hand-crafted and conform to the repository's code style. But apart from that, the code is not mine (I'd guess about 60% original miniscript code by @benma and the rest added by Claude Opus 4.8).