Nostr notes by matejcik [ARCHIVE]

📅 Original date posted:2018-06-27 📝 Original message:hello, ...

2023-06-07T18:13:13Z

In reply to nevent1q…hx60
_________________________

📅 Original date posted:2018-06-27
📝 Original message:hello,

On 26.6.2018 18:58, William Casarin wrote:
> as a data point, I was able to build a simple serializer[1] in about an
> afternoon. I would much prefer to use this lib in say, clightning (my
> original goal), without having to have the larger protobuf dependency.

To drive my point home, here's a PR converting the `writer` part of your
code to a protobuf-compatible version. It took me less than an hour to
write, the bigger part of which was spent orienting myself in unfamiliar
code. I assume I could do `reader` in less than that, if your
deserialization code was complete.

https://github.com/jb55/libpsbt/pull/3/files

This code produces PSBTs that my "bip174 playground" can correctly parse.

regards
m.

>
> Cheers,
>
> [1] https://github.com/jb55/libpsbt
>
>
> --
> https://jb55.com
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180627/4d38374e/attachment.sig>;

📅 Original date posted:2018-06-27 📝 Original message:hello, ...

2023-06-07T18:13:13Z

In reply to nevent1q…aqd5
_________________________

📅 Original date posted:2018-06-27
📝 Original message:hello,

On 26.6.2018 22:30, Pieter Wuille wrote:
>> (Moreover, as I wrote previously, the Combiner seems like a weirdly
>> placed role. I still don't see its significance and why is it important
>> to correctly combine PSBTs by agents that don't understand them. If you
>> have a usecase in mind, please explain.
>
> Forward compatibility with new script types. A transaction may spend
> inputs from different outputs, with different script types. Perhaps
> some of these are highly specialized things only implemented by some
> software (say HTLCs of a particular structure), in non-overlapping
> ways where no piece of software can handle all scripts involved in a
> single transaction. If Combiners cannot deal with unknown fields, they
> won't be able to deal with unknown scripts.

Record-based Combiners *can* deal with unknown fields. Either by
including both versions, or by including one selected at random. This is
the same in k-v model.

> combining must be done independently by Combiner implementations for
> each script type involved. As this is easily avoided by adding a
> slight bit of structure (parts of the fields that need to be unique -
> "keys"), this seems the preferable option.

IIUC, you're proposing a "semi-smart Combiner" that understands and
processes some fields but not others? That doesn't seem to change
things. Either the "dumb" combiner throws data away before the "smart"
one sees it, or it needs to include all of it anyway.

> No, a Combiner can pick any of the values in case different PSBTs have
> different values for the same key. That's the point: by having a
> key-value structure the choice of fields can be made such that
> Combiners don't need to care about the contents. Finalizers do need to
> understand the contents, but they only operate once at the end.
> Combiners may be involved in any PSBT passing from one entity to
> another.

Yes. Combiners don't need to care about the contents.
So why is it important that a Combiner properly de-duplicates the case
where keys are the same but values are different? This is a job that,
AFAICT so far, can be safely left to someone along the chain who
understands that particular record.

Say we have field F(key,value), and several Signers produce F(1,1),
F(1,2), F(1,3).

A key-based Combiner will pick exactly one to pass along. A record-based
Combiner will pass all three.

It seems that you consider the latter PSBT "invalid". But it is well
formed and doesn't contain duplicate records. A Finalizer, or a
different Combiner that understands field F, can as well have the rule
"throw away all but one" for this case.

To repeat and restate my central question:
Why is it important, that an agent which doesn't understand a particular
field structure, can nevertheless make decisions about its inclusion or
omission from the result (based on a repeated prefix)?

Actually, I can imagine the opposite: having fields with same "key"
(identifying data), and wanting to combine their "values" intelligently
without losing any of the data. Say, two Signers producing separate
parts of a combined-signature under the same common public key?

> In case of BIP32 derivation, computing the pubkeys is possibly
> expensive. A simple signer can choose to just sign with whatever keys
> are present, but they're not the only way to implement a signer, and
> even less the only software interacting with this format. Others may
> want to use a matching approach to find keys that are relevant;
> without pubkeys in the format, they're forced to perform derivations
> for all keys present.

I'm going to search for relevant keys by comparing master fingerprint; I
would expect HWWs generally don't have index based on leaf pubkeys.
OTOH, Signers with lots of keys probably aren't resource-constrained and
can do the derivations in case of collisions.

Also, you need to do the derivation and checking anyway, because what if
there is a mismatch between the key and the value?

I liked @achow101's idea about supporting non-derived keys, but I
assumed that you would match them based on the master fingerprint too?

I wouldn't be against including the full master public key (probably
without chaincode) instead of the fingerprint, as you proposed earlier.
But including both the leaf pubkey and the fingerprint seems weird.

> If you take the records model, and then additionally drop the
> whole-record uniqueness constraint, yes, though that seems pushing it
> a bit by moving even more guarantees from the file format to
> application level code.

The "file format" makes no guarantees, because the parsing code and
application code is the same anyway. You could say I'm proposing to
separate these concerns ;)

regards
m.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180627/386870f6/attachment.sig>;

📅 Original date posted:2018-06-26 📝 Original message:hello, ...

2023-06-07T18:13:12Z

In reply to nevent1q…eklz
_________________________

📅 Original date posted:2018-06-26
📝 Original message:hello,

in general, I agree with my colleague Tomas, the proposed changes are
good and achieve the most important things that we wanted. We'll review
the proposal in more detail later.

For now a couple minor things I have run into:

- valid test vector 2 ("one P2PKH input and one P2SH-P2WPKH input")
seems broken, at least its hex version; a delimiter seems to be missing,
misplaced or corrupted
- at least the first signing vector is not updated, but you probably
know that
- BIP32 derivation fields don't specify size of the "master public key",
which would make it un-parsable :)
- "Transaction format is specified as follows" and its table need to be
updated.

I'm still going to argue against the key-value model though.

It's true that this is not significant in terms of space. But I'm more
concerned about human readability, i.e., confusing future implementers.
At this point, the key-value model is there "for historical reasons",
except these aren't valid even before finalizing the format. The
original rationale for using key-values seems to be gone (no key-based
lookups are necessary). As for combining and deduplication, whether key
data is present or not is now purely a stand-in for a "repeatable" flag.
We could just as easily say, e.g., that the high bit of "type" specifies
whether this record can be repeated.

(Moreover, as I wrote previously, the Combiner seems like a weirdly
placed role. I still don't see its significance and why is it important
to correctly combine PSBTs by agents that don't understand them. If you
have a usecase in mind, please explain.
ISTM a Combiner could just as well combine based on whole-record
uniqueness, and leave the duplicate detection to the Finalizer. In case
the incoming PSBTs have incompatible unique fields, the Combiner would
have to fail anyway, so the Finalizer might as well do it. Perhaps it
would be good to leave out the Combiner role entirely?)

There's two remaining types where key data is used: BIP32 derivations
and partial signatures. In case of BIP32 derivation, the key data is
redundant ( pubkey = derive(value) ), so I'd argue we should leave that
out and save space. In case of partial signatures, it's simple enough to
make the pubkey part of the value.

Re breaking change, we are proposing breaking changes anyway, existing
code *will* need to be touched, and given that this is a hand-parsed
format, changing `parse_keyvalue` to `parse_record` seems like a small
matter?

---

At this point I'm obliged to again argue for using protobuf.

Thing is: BIP174 *is basically protobuf* (v2) as it stands. If I'm
succesful in convincing you to switch to a record set model, it's going
to be "protobuf with different varint".

I mean this very seriously: (the relevant subset of) protobuf is a set
of records in the following format:
<record type><varint field length><field data>
Record types can repeat, the schema specifies whether a field is
repeatable or not - if it's not, the last parsed value is used.

BIP174 is a ad-hoc format, simple to parse by hand; but that results in
_having to_ parse it by hand. In contrast, protobuf has a huge
collection of implementations that will do the job of sorting record
types into relevant struct fields, proper delimiting of records, etc.
...while at the same time, implementing "protobuf-based-BIP174" by hand
is roughly equally difficult as implementing the current BIP174.

N.B., it's possible to write a parser for protobuf-BIP174 without
needing a general protobuf library. Protobuf formats are designed with
forwards- and backwards- compatibility in mind, so having a hand-written
implementation should not lead to incompatibilities.

I did an experiment with this and other variants of the BIP174 format.
You can see them here:
[1] https://github.com/matejcik/bip174-playground
see in particular:
[2] https://github.com/matejcik/bip174-playground/blob/master/bip174.proto

The tool at [1] does size comparisons. On the test vectors, protobuf is
slightly smaller than key-value, and roughly equal to record-set, losing
out a little when BIP32 fields are used.
(I'm also leaving out key-data for BIP32 fields.)

There's some technical points to consider about protobuf, too:

- I decided to structure the message as a single "PSBT" type, where
"InputType" and "OutputType" are repeatable embedded fields. This seemed
better in terms of extensibility - we could add more sections in the
future. But the drawback is that you need to know the size of
Input/OutputType record in advance.
The other option is sending the messages separated by NUL bytes, same as
now, in which case you don't need to specify size.

- in InputType, i'm using "uint32" for sighash. This type is not
length-delimited, so non-protobuf consumers would have to understand it
specially. We could also declare that all fields must be
length-delimited[1] and implement sighash as a separate message type
with one field.

- non-protobuf consumers will also need to understand both protobuf
varint and bitcoin compact uint, which is a little ugly

best regards
matejcik

[1] https://developers.google.com/protocol-buffers/docs/encoding#structure

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180626/a1c4e6e5/attachment.sig>;

📅 Original date posted:2018-06-19 📝 Original message:hello, ...

2023-06-07T18:13:07Z

In reply to nevent1q…7rch
_________________________

📅 Original date posted:2018-06-19
📝 Original message:hello,
this is our second e-mail with replies to Pieter's suggestions.

On 16.6.2018 01:34, pieter.wuille at gmail.com (Pieter Wuille) wrote:
> * Key-value map model or set model.
>
> This was suggested in this thread:
> https://twitter.com/matejcik/status/1002618633472892929
>
> The motivation behind using a key-value model rather than a simple
> list of records was that PSBTs can be duplicated (given to multiple
> people for signing, for example), and merged back together after
> signing. With a generic key-value model, any implementation can remove
> the duplication even if they don't understand fields that may have
> been added in future extensions.
>
> However, almost the same can be accomplished by using the simpler set
> model (the file consists of a set of records, with no duplication
> allowed). This would mean that it would technically be legal to have
> two partial signatures with the same key for the same input, if a
> non-deterministic signer is used.

Strongly agree with this.

Just to note, we should probably use varint for the <type> field - this
allows us, e.g., to create “namespaces” for future extensions by using
one byte as namespace identifier and one as field identifier.

>
> On the other hand, this means that certain data currently encoded
> inside keys can be dropped, reducing the PSBT size. This is
> particularly true for redeemscripts and witnessscripts, as they can
> just be computed by the client when deserializing. The two types could
> even be merged into just "scripts" records - as they don't need to be
> separated based on the way they're looked up (Hash160 for P2SH, SHA256
> for P2WSH). The same could be done for the BIP32 derivation paths,
> though this may be expensive, as the client would need to derive all
> keys before being able to figure out which one(s) it needs.

It could be nice if the output scripts records would be ordered the same
as their corresponding outputs. But what if the Creator doesn’t want to
include a script for an output? Perhaps the Script record should have a
<vout> field to match it to the appropriate output.

As for input scripts, we suggest that they are per-input and not
included in the global record, see the other thread.

>
> One exception is the "transaction" record, which needs to be unique.
> That can either be done by adding an exception ("there can only be one
> transaction record"), or by encoding it separately outside the normal
> records (that may also be useful to make it clear that it is always
> required).

This seems to be the case for some fields already - i.e., an input field
must have exactly one of Non-witness UTXO or Witness Output. So “adding
an exception” is probably just a matter of language?

We’d also like to note that the “number of inputs” field should be
mandatory - and as such, possibly also a candidate for outside-record field.

>
> * Ability for Combiners to verify two PSBT are for the same transaction
>
> Clearly two PSBTs for incompatible transactions cannot be combined,
> and this should not be allowed.
>
> It may be easier to enforce this if the "transaction" record inside a
> PSBT was required to be in a canonical form, meaning with empty
> scriptSigs and witnesses. In order to do so, there could be per-input
> records for "finalized scriptSig" and "finalized witness". Actually
> placing those inside the transaction itself would only be allowed when
> all inputs are finalized.

Agreed! Also increases clarity, which is desired.

> * Derivation from xpub or fingerprint
>
> For BIP32 derivation paths, the spec currently only encodes the 32-bit
> fingerprint of the parent or master xpub. When the Signer only has a
> single xprv from which everything is derived, this is obviously
> sufficient. When there are many xprv, or when they're not available
> indexed by fingerprint, this may be less convenient for the signer.
> Furthermore, it violates the "PSBT contains all information necessary
> for signing, excluding private keys" idea - at least if we don't treat
> the chaincode as part of the private key.
>
> For that reason I would suggest that the derivation paths include the
> full public key and chaincode of the parent or master things are
> derived from. This does mean that the Creator needs to know the full
> xpub which things are derived from, rather than just its fingerprint.

We don’t understand the rationale for this idea. Do you see a scenario
where an index on master fingerprint is not available but index by xpubs
is? In our envisioned use cases at least, indexing private keys by xpubs
(as opposed to deriving from a BIP32 path) makes no sense.

Maybe this folds into the proposal for generic derivation below, or
something like implementation-specific derivation methods?

best regards

Jan Matejek
Tomas Susanka

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180619/70447694/attachment-0001.sig>;

📅 Original date posted:2018-06-21 📝 Original message:On ...

2023-06-07T18:13:05Z

In reply to nevent1q…qgm9
_________________________

📅 Original date posted:2018-06-21
📝 Original message:On 19.6.2018 19:16, Pieter Wuille wrote:
>> 1) Why isn't the global type 0x03 (BIP-32 path) per-input? How do we
>> know, which BIP-32 path goes to which input? The only idea that comes to
>> my mind is that we should match the input's scriptPubKey's pubkey to
>> this 0x03's key (the public key).
>
>> If our understanding is correct, the BIP-32 path is global to save space
>> in case two inputs share the same BIP-32 path? How often does that
>> happen? And in case it does, doesn't it mean an address reuse which is
>> discouraged?
>
> Yes, the reason is address reuse. It may be discouraged, but it still
> happens in practice (and unfortunately it's very hard to prevent
> people from sending to the same address twice).
>
> It's certainly possible to make them per-input (and even per-output as
> suggested below), but I don't think it gains you much. At least when a
> signer supports any kind of multisig, it needs to match up public keys
> with derivation paths. If several can be provided, looking them up
> from a global table or a per-input table shouldn't fundamentally
> change anything.

So here’s a thing I’m still confused about.

Imagine two cases, for a naive Signer:
- either all data is global
- or most data is per input.

Now, the general signing flow is this:
1. Pre-serialize the transaction
2. Prepare the current input - fill out scriptPubKey (or equivalent for
segwit)
3. find a secret key
4. output public key + signature

Step (3) is the main issue here.

In the case of everything per-input, the naive Signer can do this:
1. (in the global section) pre-serialize the transaction
2. (in each input) find and fill out scriptPubKey from the provided UTXO
3. (for a given BIP32 path) check if the master fingerprint matches
mine, if yes, derive secret key, output pubkey, signature
4. goto 3 (more keys per input), goto 2 (next input)

Note that this flow works perfectly for multisig; it’s going to be the
job of a Finalizer to build the final scriptSig, but each input can have
multiple partial signatures -- and, interestingly, the naive Signer
doesn’t even need to know about multisig.

A less naive Signer will want to check things, maybe derive a scriptSig
itself and check if it matches the given hash, etc., but it can do this
all in place. You go linearly through the signing flow and place a
couple strategic assertions along the way.

However, if the data is global, as is now, it gets more complicated:
1. (in the global section) pre-serialize the transaction, prefill lookup
tables
2. (for a given BIP32 path) check if mine, then derive public key and
store in a dictionary
3. (for each input) find _and parse_ scriptPubKey, extract (PK or)
script hash
4. lookup redeem script based on script-hash; if not found, goto 2; if
found, parse out public key
5. lookup public key in the BIP32 dictionary; if not found, goto 2
6. output pubkey, signature

In addition to being more steps and lookups, it requires the Signer to
understand the redeem script. A strict Signer will want that anyway, but
in the first case, the Signer can regenerate the scripts and compare
specificaly the ones it's working with; here, you need to parse them
even before you know what you're comparing to.

Is there something I’m missing? Because as I see it, there is literally
no advantage to the more complicated flow; that’s why we assumed that
the format is space-saving, because saving space was the only reason we
could imagine.

> If we go down this route, if a field is marked as mandatory, can you
> still act as a combiner for it? Future extensions should always
> maintain the invariant that a simple combiner which just merges all
> the fields and deduplicates should always be correct, I think. So such
> a mandatory field should only apply to signers?

(...)

> However, perhaps we do want to enforce at-most one UTXO per input. If
> there are more potential extensions like this, perhaps a key-value
> model is better, as it's much easier to enforce no duplicate keys than
> it is to add field-specific logic to combiners (especially for
> extensions they don't know about yet).

In general, you seem to focus a lot on the role of Combiners, esp.
simple Combiners. To me, that doesn’t look like a significant role. As I
envision it, a Combiner really doesn’t need to do anything more
complicated than merge and deduplicate records, simply based on the
uniqueness of the whole record.
It’s the Finalizer’s job to reconstruct and validate the result. Also
ISTM if something messes up the PSBT (such as including multiple
conflicting fields anywhere), it’s OK to leave it to Finalizer to fail.

Are the Combiners supposed to be separate from Finalizers? (Is there a
risk of a Combiner passing along a bad PSBT, Finalizer rejecting it, and
the other parties not finding out?)

> If we go with the "not put signatures/witnesses inside the transaction
> until all of them are finalized" suggestion, perhaps the number of
> inputs field can be dropped. There would be always one exactly for
> each input (but some may have the "final script/witness" field and
> others won't).

Strongly agree with this. A guarantee that number of inputs in the
transaction corresponds to number of input fields for PBST looks cleaner
than specifying it separately. This way we can also drop the "input index".

> Right now, the BIP32 fields are of the form <master
> fingerprint><childidx><childidx><childidx>...
>
> Instead, I suggest fields of the form <master pubkey><master
> chaincode><childidx><childidx><childidx>...
>
> The fingerprint in this case is identical to the first 32 bit of the
> Hash160 of <master pubkey>, so certainly no information is lost by
> making this change.
>
> This may be advantageous for three reasons:
> * It permits signers to have ~thousands of master keys (at which point
> 32-bit fingerprints would start having reasonable chance for
> collisions, meaning multiple derivation attempts would be needed to
> figure out which one to use).
> * It permits signers to index their master keys by whatever they like
> (for example, SHA256 rather than Hash160 or prefix thereof)> * It permits signers who don't store a chaincode at all, and just
> protect a single private key.

I like this last usecase a lot, but perhaps that's a role for a
"sub-Creator"? see below.

Also, is there a reason to publish the chain code, wouldn't just the
public key be sufficient to accomplish all three usecases you list?
I sort of dislike the notion that you need to give all this information
to a possibly untrusted Creator.

An aside to this in particular, I’ve been thinking about the requirement
to share derivation paths and public keys with the Creator. The spec
assumes that this will happen; you’re talking about providing full
xpub+chaincode too. At least, the Creator must prefill BIP32 paths and
master key fingerprints. Possibly also prefill public keys in the redeem
scripts?

This might not be an improvement proposal, but a point worth being
raised and maybe explained in the spec. Perhaps the original Creator
doesn’t have access to this data, and delegates this to some
“sub-Creators” - I imagine a coordinator sending a PSBT to signing
parties, each of which acts as a sub-Creator (fills out derivation paths
and public keys) and a Signer (forwarding to a HWW). Some of the
discussion even suggests some sort of generic “key derivation field”
with arbitrary contents - fingerprint + bip32 path? xpub + chain code?
derivation points? encrypted xprv?

thank you for your comments

regards
m.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180621/92c941f5/attachment-0001.sig>;

📅 Original date posted:2018-06-19 📝 Original message:Hello, ...

2023-06-07T18:13:03Z

In reply to nevent1q…gqts
_________________________

📅 Original date posted:2018-06-19
📝 Original message:Hello,

First of all, we'd like to apologize for such a late feedback, since
there is a PR for this already. We've come up with a few more notes on
this, so we are introducing those in this message and replying on
Pieter's points in another one.

1) Why isn't the global type 0x03 (BIP-32 path) per-input? How do we
know, which BIP-32 path goes to which input? The only idea that comes to
my mind is that we should match the input's scriptPubKey's pubkey to
this 0x03's key (the public key).

If our understanding is correct, the BIP-32 path is global to save space
in case two inputs share the same BIP-32 path? How often does that
happen? And in case it does, doesn't it mean an address reuse which is
discouraged?

Also, we believe that if the public key is to be used as "spent to by an
output" it should be in an output section. If the public key is to be
used to sign an input, it should be in the input section. Again, how
often are those the same? We understand creating another section might
be cumbersome, but it'd significantly increase clarity to have global,
input and output section.

Alternately, we could keep “spend to” public keys in the global section,
and put the input public keys to the per-input sections. This is less
clear, but doesn’t introduce another section. A question to consider is,
will there be more per-output data? If yes, it might make sense to have
an output section.

2) The global items 0x01 (redeem script) and 0x02 (witness script) are
somewhat confusing. Let's consider only the redeem script (0x01) to make
it simple. The value description says: "A redeem script that will be
needed to sign a Pay-To-Script-Hash input or is spent to by an output.".
Does this mean that the record includes both input's redeem script
(because we need to sign it), but also a redeem script for the output
(to verify we are sending to a correct P2SH)? To mix those two seems
really confusing.

Yet again, adding a new output section would make this more readable. We
would include the input’s redeem script in the input section and the
output’s redeem script again in the output section, because they’ll most
likely differ anyway.

The rationale says that the scripts are global to avoid duplication.
However, how often is this the case? The scripts include a hash of some
OP codes and the recipient's public key for example. So a) how often are
two scripts equal to justify this? b) if they're the same, doesn't it
yet again signalize address reuse?

3) The sighash type 0x03 says the sighash is only a recommendation. That
seems rather ambiguous. If the field is specified shouldn't it be binding?

4) Is it a good idea to skip records which types we are unaware of? We
can't come up with a reasonable example, but intuitively this seems as a
potential security issue. We think we should consider introducing a
flag, which would define if the record is "optional". In case the signer
encounters a record it doesn't recognize and such flag is not set, it
aborts the procedure. If we assume the set model we could change the
structure to <type><optional flag><length>{data}. We are not keen on
this, but we wanted to include this idea to see what you think.

-----------

In general, the standard is trying to be very space-conservative,
however is that really necessary? We would argue for clarity and ease of
use over space constraints. We think more straightforward approach is
desired, although more space demanding. What are the arguments to make
this as small as possible? If we understand correctly, this format is
not intended for blockchain nor for persistent storage, so size doesn’t
matter nearly as much.

Thank you,

Tomas Susanka
Jan Matejek

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20180619/57fc30e2/attachment.sig>;