Skip to content

[2/2] Add Output Descriptor library - #2568

Open
guggero wants to merge 15 commits into
btcsuite:miniscript-part-1from
guggero:miniscript
Open

guggero wants to merge 15 commits into
btcsuite:miniscript-part-1from
guggero:miniscript

Conversation

@guggero

@guggero guggero commented Jul 6, 2026 •

Copy link
Copy Markdown
Collaborator

I wasn't able to reopen the old PR, so I'm resurrecting it here.
After using https://github.com/benma/descriptors-go in a mobile project, the limitations of using a WASM interpreter showed to be too restricting. So I'm giving it another try, this time with quite a bit of agentic help (I gave up on #1987 because I'm not smart enough for compiler-level stuff, but apparently Claude is quite capable).

Description

Depends on #2592.

Part 1 added a miniscript library (originally authored by @benma in https://github.com/benma/miniscript-go) and this second part now adds full Output Descriptor support on top.
The code is modeled after the rust-miniscript implementation and a lot of test vectors for diferential unit tests were extracted from that implementation.

The descriptors features are currently limited to feature parity with https://github.com/benma/descriptors-go.

Implementation disclaimer

The code in this PR is more or less a supervised re-implementation of rust-miniscript. I spent a lot of tokens on differential code review and bug hunting, including large amounts of test vectors extracted from the original Rust code.
There are also seeded fuzz tests that have been running for several hours on a 12-core machine.
All commits are hand-crafted and conform to the repository's code style. But apart from that, the code is not mine (I'd guess about 60% original miniscript code by @benma and the rest added by Claude Opus 4.8).

@kcalvinalvin

kcalvinalvin commented Jul 7, 2026 •

Copy link
Copy Markdown
Collaborator

Me looking at the number of lines changed

IMG_0802

.
.
.
.
.
.
.
.
.

Would be awesome to have though! I'll start slowly going through...

@guggero

guggero commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator Author

Me looking at the number of lines changed

Yeah, it's a junky boyyy... A lot of the added lines are test vectors. But lots of Go code as well, I agree.
Would be nice if you could take a look, thanks!

@guggero

guggero commented Aug 6, 2026 •

Copy link
Copy Markdown
Collaborator Author

I ran a full sweep with Kimi K3 that found 19 issues (1 critical, 6 high, 5 medium and 7 low) which are now fixed.
Also did a full validation against all BIPs which found a couple of small bugs and divergences in the BIPs themselves (bitcoin/bips#2239).
The full differential testing also found a small inaccuracy in the rust-miniscript library, which I'm going to create a PR for separately (rust-bitcoin/rust-miniscript#1015).

The two READMEs now point out exactly what the supported features and known divergences are against Core and/or rust-miniscript.

@GustavoStingelin

Copy link
Copy Markdown

It’s a nice feature! Do you think it would be feasible to split this PR into a few smaller ones without losing too much context? That way we could review it in individual chunks. The new GitHub stacked PRs feature also seems like a good fit for this kind of change.

@guggero

guggero commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator Author

It’s a nice feature! Do you think it would be feasible to split this PR into a few smaller ones without losing too much context? That way we could review it in individual chunks. The new GitHub stacked PRs feature also seems like a good fit for this kind of change.

Yes, you're right, splitting up the PR does make sense. I'll give it a try.

@rogeriob2br

Copy link
Copy Markdown

I noticed descriptors.Assets doesn't include a lookup for hash preimages, making PlanAt hardcode them as unavailable. Was this kept intentionally for parity with benma/descriptors-go?

@guggero guggero changed the title Add Output Descriptor library [2/2Add Output Descriptor library Aug 12, 2026
@guggero
guggero changed the base branch from master to miniscript-part-1 August 12, 2026 13:51
@guggero guggero changed the title [2/2Add Output Descriptor library [2/2] Add Output Descriptor library Aug 12, 2026
@guggero

guggero commented Aug 12, 2026 •

Copy link
Copy Markdown
Collaborator Author

Do you think it would be feasible to split this PR into a few smaller ones without losing too much context? That way we could review it in individual chunks. The new GitHub stacked PRs feature also seems like a good fit for this kind of change.

Okay, PRs are now split. Only two parts, but way smaller commits each. I wanted to avoid needing to close this one, so I didn't end up using the stacked PR feature directly. But then I made a mistake which required me to close part 1 and re-open again anyway, so it wouldn't have mattered in the first place 😅

@guggero

guggero commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator Author

I noticed descriptors.Assets doesn't include a lookup for hash preimages, making PlanAt hardcode them as unavailable. Was this kept intentionally for parity with benma/descriptors-go?

Yes, the feature parity with benma/descriptors-go is the reason that's missing. I think we can try to get to feature parity with rust-miniscript in a later PR, as these two are already super large.
Or do you have a use case that depends on preimages and would block you from using the code as is?

@guggero

guggero commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator Author

I ran a full security and correctness review with GPT 5.6 Sol and fixed a couple of issues (here and in the miniscript PR).

@rogeriob2br the latest version also fully supports hash preimages in the Assets.

@guggero

guggero commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator Author

@Lrifton92 thank you very much for your review and for reporting the problem.
My latest push now addresses the issue: NewDescriptor() now rejects expressions that have no satisfaction.
A new NewDescriptorInsane() allows you to still parse such expressions if you need to analyze them for some reason. I decided to add that since this PR intends to add a generic and re-usable library (and not a narrow wallet-specific implementation such as Core has it).

I also did a full differential diff of all test vectors against the latest rust-miniscript master branch.
This resulted in an update to some test vectors, mainly because rust-bitcoin/rust-miniscript#1040 and rust-bitcoin/rust-miniscript#1037 were merged recently.

@coveralls

coveralls commented Oct 6, 2026 •

Copy link
Copy Markdown

Coverage Report for CI Build 37457660869

Coverage increased (+1.0%) to 56.721%

Details

  • Coverage increased (+1.0%) from the base build.
  • Patch coverage: 170 uncovered changes across 6 files (1830 of 2000 lines covered, 91.5%).
  • 19 coverage regressions across 3 files.

Uncovered Changes

File Changed Covered %
descriptors/plan.go 596 535 89.77%
descriptors/address.go 203 168 82.76%
descriptors/descriptor.go 417 393 94.24%
descriptors/lift.go 182 162 89.01%
descriptors/weight.go 185 169 91.35%
descriptors/key.go 309 295 95.47%
Total (8 files) 2000 1830 91.5%

Coverage Regressions

19 previously-covered lines in 3 files lost coverage.

File Lines Losing Coverage Coverage
btcec/v2/ellswift/ellswift.go 16 78.06%
rpcclient/infrastructure.go 2 62.33%
btcutil/gcs/gcs.go 1 80.95%

Coverage Stats

Coverage Status
Relevant Lines: 75802
Covered Lines: 42996
Line Coverage: 56.72%
Coverage Strength: 316273.83 hits per line

💛 - Coveralls

@fametrano fametrano left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed 94b7925, on top of #2592 at 45bfbdf, against Bitcoin Core (master 4bacf21a13, and the 31.1 release), the BIP380-389 test vectors and the Python descriptors in btclib-wallet (efa2d7d). Apart from what the README lists and what I raised on #2592, they agree, with three exceptions:

  1. A key repeated under another spelling passes NewDescriptor, and then no address derives.
  2. Hybrid public keys (prefix 06 or 07) are accepted.
  3. A repeated key is rejected in a multi_a leaf but accepted in multi().

I ran the BIP vectors, Core's descriptor_tests.cpp cases, and 20,000 seeded random descriptors compared with btclib-wallet on the scriptPubKeys at indexes 0-2 of every path. I also passed the descriptors below to getdescriptorinfo on a Core 31.1 regtest node, with the WIF and xpub re-encoded for testnet.

1. A key repeated under another spelling passes NewDescriptor, and then no address derives

miniscript.go:607 compares keys as written. Derived keys are compared only in ApplyVars, at derivation, and checkMsSanity adds nothing. So NewDescriptor accepts each of these, AddressAt fails with "duplicate key found at pk_k", and MaxWeightToSatisfy still returns 217. Here K is 03a34b99f22c790c4e36b2b3c2c35a36db06226e41c692fc82b8b56ac1c540c5bd and X is BIP387's xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL:

  • wsh(and_v(v:pk(L4rK1yDtCWekvXuE6oXD9jCYfFNV2cWRpVuPLBcCU2z8TrisoyY1),pk(K))), the WIF and the hex of one key;
  • wsh(and_v(v:pk([deadbeef/1]K),pk(K)));
  • wsh(and_v(v:pk(X/0/*),pk([01020304/5]X/0/*)));
  • wsh(and_v(v:pk(X/<0;1>/*),pk(X/0/*))), where path 0 fails and path 1 derives.

Core 31.1 rejects all four at parse with "contains duplicate public keys", because its KeyCompare (descriptor.cpp:2304) compares derived keys. btclib-wallet rejects them too. This contradicts the README's "A descriptor that parses can be paid to". checkMsSanity could compare the derived keys of each path.

2. Hybrid public keys are accepted

pk(07a34b99…a235) and pkh(07a34b99…a235) parse, and the script holds the 04 encoding of the key instead (key.go:588): pk(07…) gives 4104a34b…ac. With the wrong parity, pk(06a34b99…a235) parses and then derivation fails. BIP380 allows only 02, 03 and 04. Core rejects these ("Hybrid public keys are not allowed", also among its descriptor_tests.cpp unparsable cases), and so does btclib-wallet. key.go:172-173 checks only the length; checkRawKey could reject any other prefix.

3. A repeated key is rejected in a multi_a leaf but accepted in multi()

tr(K,multi_a(1,A,A)) and tr(K,sortedmulti_a(1,A,A)) fail with "duplicate key". The leaf is parsed as miniscript (descriptor.go:816), and the sanity check refuses it. wsh(multi(1,A,A)), sh(multi(1,A,A)) and bare multi(1,A,A) are accepted. Core 31.1 accepts all five, and so does btclib-wallet. Either accept the leaf, or add it to the divergence table.

@fametrano

Copy link
Copy Markdown
Contributor

All three points from my review are fixed at 5bc4530. I re-ran the review's descriptors: the four repeated-key cases and the hybrid keys are now rejected by NewDescriptor, and multi_a/sortedmulti_a with a repeated key are accepted like multi(), which is what Core 31.1 does with each of them. The same test shows the old behaviour at 94b7925, and go test and go vet pass in descriptors.

@guggero

guggero commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator Author

@fametrano thank you very much for the reviews! The original version oriented itself toward rust-miniscript's behavior. But IMO it makes sense to adopt Core's behavior where the two disagree.

A descriptor names its keys through key expressions: a raw public key,
a WIF private key, or an extended key with an optional origin and a
derivation path that may end in a wildcard, and that may hold a multipath
element. This parses them and derives the concrete key of a path, which
is the part of a descriptor that turns one string into the many scripts
of a wallet.
Output script descriptors carry an optional checksum which catches the
transcription errors a raw descriptor string is prone to: any single
character substitution, and a swap of characters that are not too far
apart. BIP380 specifies it. This is the first piece of the descriptors
package, and the only one that needs nothing else.
The descriptor grammar of BIP380 and its script expressions: the key
based ones (pk, pkh, wpkh), the wrappers (sh, wsh), the multisigs (multi,
sortedmulti) and a miniscript expression as the script of a wsh. Which
expression may appear where is part of the grammar, so the parser tracks
the position it is in and rejects, say, a wsh inside a wsh.

The result is a tree of nodes with the keys of the descriptor collected
in the order they appear. Nothing is derived from it yet.
A tr() descriptor has an internal key and an optional script tree,
written as a nested pair of branches with a script expression at every
leaf. BIP386 defines it. Parsing the tree is what the rest of this commit
is about, including the two limits that keep a pathological descriptor
from exhausting the process: the depth of the tree, which BIP341 caps at
128 anyway, and the depth of the nesting of the descriptor itself.
With the tree parsed and the keys derivable, a descriptor can produce
what a wallet needs from it: the address of the output at a derivation
index, and the script code that a signature over that input commits to.
Both are per multipath and derivation index, and both compile a
miniscript expression through the package of the previous commits where
the descriptor holds one.
What a spend of a descriptor's output costs, in weight units: the
witness or scriptSig that satisfies it, sized without the signatures
being available. The estimate is the one a fee calculation needs, and for
a taproot descriptor it is the cheapest of the key path and the leaves of
the script tree, since that is the path a spender would take.
The redeem script of a P2SH output is pushed as a single element of the
spending scriptSig, so the 520-byte limit on a script element applies to
it: an output whose redeem script is larger can never be spent, and a
descriptor that describes one is a mistake worth rejecting at parse time.
The same holds for the number of keys of a bare multisig, which the
standardness rules cap at three.
The normative vectors of BIP380 to BIP386: the descriptors those
documents list as valid or invalid, with the scripts and addresses they
must produce. The features this package does not support are classified
explicitly rather than skipped silently, so that the gaps are visible in
the test output instead of hiding in a skip.
The semantic policy of a whole descriptor rather than of a single
miniscript: a taproot descriptor lifts to the choice between its key path
and its leaves, a multisig to a threshold, and a wsh to the policy of its
script. It is what a wallet shows a user as the spending conditions of an
address.
A plan is the spending path a descriptor takes given what a signer
holds: which signatures, preimages and time locks are available decides
which branch of the descriptor can be satisfied, and of the branches that
can be, the cheapest one is chosen. The plan carries the weight of that
satisfaction and knows how to complete it once the real signatures
arrive, which is the two-phase flow a wallet needs to build a
transaction, size its fee and only then sign it.
For a taproot descriptor the plan chooses between the key path and
every leaf of the script tree, which requires the control block of a leaf
and therefore its merkle proof. The proofs are collected in one walk over
the tree, and the cheapest of the candidates wins, as a spender would
have it.
The differential vectors come from rust-miniscript and
descriptors-go: for about 500 descriptors they compare the canonical
string, the type, the keys, the addresses of a grid of derivation
indices, the script code, the satisfaction weight, the lifted policy and
the plan. Everything this package computes about a descriptor is in
there, which is what makes the comparison worth having.
The descriptor parser takes strings from the outside as well, so it is
fuzzed like the miniscript one, with the invariant that everything a
descriptor derives stays consistent with the string it round trips to.
The benchmarks cover parsing, address derivation and planning, and the
recursion tests pin that a deeply nested descriptor is rejected rather
than crashing the process.
A reader judging this package needs to know what it does, what it does
not, and where it deliberately differs from Bitcoin Core and
rust-miniscript. That is what these two documents are: the supported
expressions, the unsupported ones, the divergences with the commit of
each implementation they were compared against, and the behaviours that
are worth knowing before using the API.
Add implementation-independent JSON fixtures and a public-API Go harness for planning and completing descriptor spends. Cover exact scripts and stacks, size estimates, derivation, negative completions, fixed-path selection and serialization boundaries, including independently signed transaction verification cases. Document the vector format and retain its standalone generator.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants