profile
viewpoint
George Tankersley gtank https://twitter.com/gtank__ You can't hide secrets from the future with math, but I try.

gtank/cryptopasta 1495

copy & paste-friendly golang crypto

FiloSottile/hstools 66

Library and tools to interact with and analyze Tor HSDirs.

gtank/captcha-draft 25

proposal for blinded-token captchas

alecmuffett/videonion 19

video onion hackery (osx scripts)

gtank/blake2s 12

Optimized Go implementation of blake2s with salting and personalization support

gtank/bloomfilter 12

Bloom filter in Golang with hashing efficiency trick

gtank/bytecrypto 8

little-endian cryptobyte

gtank/csrctl 8

debugging tool for working with the k8s certificates api

gtank/awsid 4

tool to verify EC2 identity metadata

gtank/corepki 1

support scripts for TLS work

issue commentZcashFoundation/zebra

Implement Sapling Note Commitment Function

Related: https://github.com/ZcashFoundation/zebra/issues/279

hdevalence

comment created time in 20 hours

issue openedZcashFoundation/zebra

Implement Sapling note hashing functions

The Sapling note commitment tree depends on

  • [ ] 5.4.1.8 Mixing Pedersen Hash Function

and Sapling note commitments, themselves, which need

  • [ ] 5.4.1.7 Pedersen Hash Function
  • [ ] 5.4.7.2 Windowed Pedersen commitments

These are unrelated to the homomorphic Pedersen hashes used for value commitments, which are described in section 5.4.7.3.

Rust implementations of these primitives should already exist somewhere inside librustzcash.

created time in 20 hours

issue commentZcashFoundation/zebra

Sapling keys

A reference implementation of these can be found in https://github.com/zcash-hackworks/zcash-test-vectors

dconnolly

comment created time in 21 hours

startedNikVolf/zcash-mmr

started time in 6 days

pull request commentzcash/zips

Mutually agreed modifications to zip-1014

Note: ZIP editors' review is not yet complete. Will remove this comment later.

acityinohio

comment created time in 8 days

pull request commentzcash/zips

Mutually agreed modifications to zip-1014

  1. The new text omits the explicit responsibility to maintain and report on reserves that was added to @tromer's version in 9336a78 . Is this intentional? (That wording was suggested by me in https://forum.zcashcommunity.com/t/community-sentiment-polling-results-nu4-and-draft-zip-1014/35560/498 and received support in the forum thread.)

In the final negotiations between ECC and ZF over the application of the trademark to ZIP 1014, we decided not to make this change because we did not feel it necessary, and only necessary changes should be made to the ZIP.

This leaves us with an unspecified requirement in that section of the document. It says:

In case the value of ZEC jumps, the Dev Fund recipients should not wastefully use excessive amounts of funds. Conversely, given market volatility and eventual halvings, it is desirable to create rainy-day reserves.

If we don't at least add @tromer's paragraph from https://github.com/zcash/zips/commit/9336a78e3504d0dd1d6b49562adfec9caa7c7deb, then the rest of the document says nothing about this supposed "requirement". As ZIP Editor, I think it IS necessary to add Eran's paragraph to at least minimally specify what this requirement means.

acityinohio

comment created time in 8 days

pull request commentZcashFoundation/zips

Editorial changes to Dev Fund ZIP

We should change the title to disambiguate from ZIP-1012, which is called the same.

daira

comment created time in 8 days

create barnchgtank/zebra

branch : version_handshake

created branch time in 14 days

fork gtank/zebra

An ongoing Rust implementation of a Zcash node. 🦓

fork in 14 days

issue commentzcash/lightwalletd

Improve logging

It would be nice to enable selective logging by category, rather than just level (an integer). This is how zcashd logging is done, see the --debug command-line argument.

It may also be nice to allow logging to be enabled and disabled at runtime (using a gRPC), rather than only when the process starts.

In Zebra we're using the https://github.com/tokio-rs/tracing library, which has those capabilities and is indeed quite nice. Unfortunately I don't know of an equivalent for Go.

gtank

comment created time in 22 days

startedinitc3/babySNARK

started time in a month

Pull request review commentzcash/zips

Add ZIP 1014, based on ZIP 1012 with modifications from the Zcash Foundation board

+::++  ZIP: 1014+  Title: Dev Fund to ECC + ZF + Major Grants+  Owner: Josh Cincinnati <josh@zfnd.org>+  Original-Author: Eran Tromer+  Credits: Matt Luongo+           Howard Loo+           @aristarchus+           @dontbeevil+  Status: Draft+  Category: Consensus / Process+  Created: 2019-11-10+  License: MIT+  Discussions-To: <https://forum.zcashcommunity.com/t/dev-fund-proposal-dev-fund-to-ecc-zfnd-major-grants/35364>+++Terminology+===========++The key words "MUST", "MUST NOT", "SHALL", "SHALL NOT", "SHOULD", and "MAY"+in this document are to be interpreted as described in RFC 2119. [#RFC2119]_++Abstract+========++This proposal describes a structure for the Zcash Development Fund, to be+enacted in Network Upgrade 4 and last for 4 years. This Dev Fund would consist+of 20% of the block rewards, split into 3 slices:++* 35% for the Electric Coin Company;+* 25% for Zcash Foundation (for internal work and grants);+* 40% for additional "Major Grants" for large-scale long-term projects (decided+  by the Zcash Foundation, with extra community input and scrutiny).++Funding is capped at $700k/month per slice. Governance and accountability are+based on existing entities and legal mechanisms, and increasingly decentralized+governance is encouraged.+++Motivation+==========++Starting at Zcash's first halving in October 2020, by default 100% of the block+rewards will be allocated to miners, and no further funds will be automatically+allocated to research, development, and outreach. Consequently, no substantial+new funding may be available to existing teams dedicated to Zcash: the Electric+Coin Company (ECC), the Zcash Foundation (ZF), and the many entities funded by+the ZF grant program.++There is a need to strike a balance between incentivizing the security of the+consensus protocol (i.e., mining) versus other crucial aspects of the Zcash+security and functionality, such as development and outreach.++Furthermore, there is a need to balance the sustenance of ongoing work by the+current teams dedicated to Zcash, with encouraging decentralization and growth+of independent development teams.++Difference from Matt Luongo's proposal+--------------------------------------++This proposal is based on Matt Luongo's `Decentralizing the Dev Fee`_ proposal,+which has similar motivations. The major changes are as follows:++* The Dev Fund slice intended for external recipients (beyond ECC, ZF and+  existing ZF grants) may be used to fund ECC if no competitive alternatives+  present themselves, to mitigate unwarranted loss of existing capabilities.+* For simplicity, the above slice is combined with the Foundation's existing+  grant system; but is accompanied by explicit requirements to achieve its+  goals, an independent body to disburse funds, and a Restricted Funds+  mechanism to enforce these requirements.+* The "easing function" coin value cap is removed, in favor of capping each+  slice at $700k/month funding target. Any excess is kept in a reserve by each+  organization, from which it can be withdrawn only to maintain the funding+  target in the future.+* Strengthened the transparency and accountability requirements, and+  harmonized them across ECC, ZF, and major grantees.+* Removed ZF's supervisory role in determining the "principal developer",+  fixing it to be ECC (changing this would be sufficiently dramatic to merit a+  fork).+* Calls for the development of decentralized voting and governance.+* Clarity and brevity.++.. _Decentralizing the Dev Fee: https://forum.zcashcommunity.com/t/decentralizing-the-dev-fee/35252+++Requirements+============++The Dev Fund should encourage decentralization of the work and funding, by+supporting new teams dedicated to Zcash.++The Dev Fund should maintain the existing teams and capabilities in the Zcash+ecosystem, unless and until concrete opportunities arise to create even greater+value for the Zcash ecosystem.++There should not be any single entity which is a single point of failure, i.e.,+whose capture or failure will effectively prevent effective use of the funds.++Major funding decisions should be based, to the extent feasible, on inputs from+domain experts and pertinent stakeholders.++The Dev Fund mechanism should not modify the monetary emission curve (and in+particular, should not irrevocably burn coins).++In case the value of ZEC jumps, the Dev Fund recipients should not be allowed+to wastefully use excessive amounts of funds. Conversely, given market volatility+and eventual halvings, it is desirable to create rainy-day reserves.++The Dev Fund mechanism should not reduce users' financial privacy or security.+In particular, it should not cause them to expose their coin holdings, or to+maintain access to secret keys for much longer than they would otherwise. (This+rules out some forms of voting, and of disbursing coins to past/future miners).++The new Dev Fund system should be simple to understand and realistic to+implement. In particular, it should not assume the creation of new mechanisms+(e.g., election systems) or entities (for governance or development) for its+execution; but it should strive to support and use these once they are built.++Comply with legal, regulatory, and taxation constraints in pertinent+jurisdictions.+++Non-requirements+================++General on-chain governance is outside the scope of this proposal.++Rigorous voting mechanisms (whether coin-weighted, holding-time-weighted or+one-person-one-vote) are outside the scope of this proposal, though there is+prescribed room for integrating them once available.+++Specification+=============++Dev Fund allocation+-------------------++Starting at the first Zcash halving in 2020, until the second halving in 2024,+20% of the block rewards SHALL be allocated to a "Dev Fund" that consists of+the following three slices:++* 35% for the Electric Coin Company (denoted **ECC slice**);+* 25% for the Zcash Foundation, for general use (denoted **ZF-GU slice**);+* 40% for additional "Major Grants" for large-scale long-term projects+  (denoted **ZF-MG slice**).++Details below. The fund flow will be implemented at the consensus-rule layer,+by sending the corresponding ZEC to the designated address in each block. This+Dev Fund will end at the second halving (unless extended/modified by a future+ZIP).+++ECC slice (Electric Coin Company)+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~++This slice of the Dev Fund will flow to ECC.++ECC MUST undertake a firm obligation to use the Dev Fund only in support of the+Zcash cryptocurrency and its community.++In particular, ECC MUST commit to not distribute the Dev Fund proceeds to its+partners ("shareholders"), other than:++1. In fair-market-value compensation for specific new work (e.g., to employees+   and contractors).+2. For covering pass-through tax obligations to partners caused by ECC's receipt+   of the Dev Fund.++(ECC is encouraged to transition to a corporate structure that would avoid the+latter taxes.)++This obligation MUST be made irrevocable, e.g., within ECC's corporate+governance structure (i.e., its Operating Agreement) or contractual obligations.+++ZF-GU slice (Zcash Foundation, for general use)+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~++This slice of the Dev Fund will flow to ZF, to be used at its discretion for+any purpose within its mandate to support Zcash and financial privacy,+including: development, education, support community communication online+and via events, gathering community sentiment, and external awarding grants+for all of the above.+++ZF-MG slice (Zcash Foundation, for major grants)+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~++This slice of the Dev Fund is intended to fund independent teams entering the+Zcash ecosystem, to perform major ongoing development (or other work) for the+public good of Zcash ecosystem, to the extent that such teams are available+and effective.++The funds SHALL be received and administered by ZF. ZF MUST disburse them as+"Major Grants", within the framework of ZF's grant program but subject to the+following additional constraints:++1. These funds MUST only be used to issue Major Grants to external parties+   that are independent of ZF. They MUST NOT be used by ZF for its internal+   operations and direct expenses.++2. Major Grants SHOULD support well-specified work proposed by the grantee,+   at reasonable market-rate costs. They can be of any duration, or ongoing+   without a duration limit, but have semiannual review points for+   continuation of funding.++3. Major Grants may be issued to ECC only if there are no other proposals+   to perform the specified work with similar capabilities, effectiveness and+   cost. (The intent is that eventually ECC will not receive Major Grants.)++4. Priority SHOULD be given to Major Grants that bolster teams with+   substantial (current or prospective) continual existence, and set them up+   for long-term success, subject to the usual grant award considerations+   (impact, ability, risks, team, cost-effectiveness, etc.). Priority should be+   given to Major Grants that support ecosystem growth by mentorship, coaching,+   technical resources, creating entrepreneurial opportunities, etc. If one+   proposal substantially duplicates anothers' plans, priority should be+   given to the originator of the plans.++5. Major Grants SHOULD be awarded based on ZF's mission_ and values_, restricted+   to furthering of the Zcash cryptocurrency and its ecosystem (which is more+   specific than furthering financial privacy in general).++6. Major Grants awarding is subject to approval by a five-seat Major Grant+   Review Committee. The Major Grant Review Committee SHALL be selected by the+   ZF's Community Panel. The Major Grant Review Committee's funding+   decisions will be final, requiring no approval from the ZF Board, but are+   subject to veto if the Foundation judges them to violate the ZF's operating+   documents or U.S. law.++7. Major Grant Review Committee members SHALL have a one-year term and MAY sit+   for reelection. The Major Grant Review Committee is subject to the same+   conflict of interest policy that governs the ZF board of directors+   (i.e. they MUST recuse themselves when voting on proposals where they have+   a financial interest). Additionally, no one with interest in or association+   with the ECC may sit on the Major Grant Review Committee --- since the ECC+   can be a beneficiary, this avoids those potential conflicts altogether.+   The ZF SHALL continue to operate the Community Panel and SHOULD work+   toward making it more representative and independent (more on that below).++ZF SHALL recognize the ZF-MG slice of the Dev Fund as a Restricted Fund+donation under the above constraints (suitably formalized), and keep separate+accounting of its balance and usage under its Transparency and Accountability+obligations defined below.++From grant proposers' side, proposals for such grants SHALL be submitted+through ZF's usual grant process, allowing for public discussion and public+funding. It is intended that small one-time grants will be funded by drawing+on the ZF-GU slice (where they also compete with other ZF activities), whereas+large long-duration will be funded from the dedicated ZF-MG slice; though+this is at ZF's discretion (e.g. if there are no Major Grant applications the+ZF may opt to direct the ZF-MG to smaller grants).++ZF SHALL strive to define target metrics and key performance indicators, and+the Major Grant Review Committee SHOULD utilize these in its funding+decisions.++.. _mission: https://www.zfnd.org/about/#mission+.. _values: https://www.zfnd.org/about/#values++Direct-grant option+'''''''''''''''''''++It may be deemed better, operationally or legally, if the Major Grant funds+are not accepted and disbursed by ZF, but rather directly assigned to the+grantees. Thus, the following mechanism MAY be used in perpetuity, if agreed+upon by both ECC and ZF before NU4 activation:++Prior to each Network Upgrade, the Foundation SHALL publish a list of+grantees' addresses and the total number of Dev Fund ZEC per block they+should receive. ECC and ZF SHALL implement this list in any implementations+of the Zcash consensus rules they maintain. This decision will then be,+effectively, ratified by the miners as the network upgrade activates.+++Funding Target and Volatility Reserve+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~++Each Dev Fund slice has a Funding Target, initially US $700,000 for each+slice. At the end of each calendar month, the fair market value of the Dev

How will fair market value be calculated? The ZIP editors propose that each grant slice recipient should describe their methodology in the relevant transparency report.

acityinohio

comment created time in 2 months

PR opened daira/zips

zip-1014: clarify review of indefinite duration grants

The previous wording was ambiguous as to whether both types of grants or only the ongoing ones should have review points.

+3 -3

0 comment

1 changed file

pr created time in 2 months

push eventgtank/zips

George Tankersley

commit sha 08e8b0f273aac404ea6e426c3c40096250415804

zip-1014: clarify review of indefinite duration grants

view details

push time in 2 months

push eventgtank/coredns-zcash

George Tankersley

commit sha 54a67e5e7206160e7324688b16d261327244ff2b

zcash: working but unpleasant refactoring of address handling

view details

push time in 2 months

starteddenismerigoux/rust-secret-integers

started time in 2 months

startedreHackable/scripts

started time in 2 months

issue commentgtank/merlin

Transcript RNG

Hey, I'm clearly not going to get around to this as quickly as I'd hoped. I didn't mean to lick the cookie! If you would like to work on it, please go ahead. I'll be happy to look at an API or review a PR @noot

noot

comment created time in 2 months

startedsignalapp/SecureValueRecovery

started time in 2 months

push eventgtank/papers

George Tankersley

commit sha 74e85f464b58ed4967df9ab16484f1b566f4aebf

partial snarktember, threshold signatures, private metrics, and other things

view details

push time in 2 months

startedmmou/threshold-OPAQUE

started time in 3 months

issue openedgtank/ristretto255

Write comparative tests for target-specific field arithmetic

Speaking of https://golang.org/wiki/TargetSpecific (see #30), I don't think we have tests that explicitly compare our math routines.

created time in 3 months

push eventgtank/ristretto255

Filippo Valsorda

commit sha 6db84dcfdcc0cd1b8556f145c39d230bf2a5ffef

internal/radix51: restructure according to golang.org/wiki/TargetSpecific

view details

Filippo Valsorda

commit sha a679c261e4df42455c830458dee32512a3e08211

internal/radix51: implement (*FieldElement).Mul32 This pure Go implementation of Mul32 is more than twice as fast as the assembly Mul implementation, and four times faster than the pure Go Mul. Mul32 7.91ns ± 1% Mul 18.6ns ± 1% Mul [purego] 33.4ns ± 0% Before Go 1.13, where we can't use math/bits because the fallbacks might not be constant time, Mul32 is a little slower, but not nearly as much as the pure Go Mul. Mul32 9.74ns ± 0% Mul [purego] 75.4ns ± 1%

view details

Filippo Valsorda

commit sha 502122d1258906c8f71e68ffb3a914362cc67743

all: ensure compatibility with older Go versions

view details

push time in 3 months

PR merged gtank/ristretto255

internal/radix51: implement (*FieldElement).Mul32

This pure Go implementation of Mul32 is more than twice as fast as the assembly Mul implementation, and four times faster than the pure Go Mul.

Mul32          7.91ns ± 1%
Mul            18.6ns ± 1%
Mul [purego]   33.4ns ± 0%

Before Go 1.13, where we can't use math/bits because the fallbacks might not be constant time, Mul32 is a little slower, but not nearly as much as the pure Go Mul.

Mul32          9.74ns ± 0%
Mul [purego]   75.4ns ± 1%

/cc @hdevalence

+348 -290

1 comment

14 changed files

FiloSottile

pr closed time in 3 months

Pull request review commentgtank/ristretto255

internal/radix51: implement (*FieldElement).Mul32

 // Copyright (c) 2019 George Tankersley. All rights reserved.+// Copyright (c) 2019 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. -// +build go1.12+// +build go1.13

Ah, I see. bits was vartime on platforms without the intrinsics?

FiloSottile

comment created time in 3 months

Pull request review commentgtank/ristretto255

internal/radix51: implement (*FieldElement).Mul32

 func BenchmarkMul(b *testing.B) { 		x.Mul(&x, &y) 	} }++func BenchmarkMul32(b *testing.B) {+	var x radix51.FieldElement+	x.One()+	b.ResetTimer()+	for i := 0; i < b.N; i++ {+		x.Mul32(&x, 0b10101010_10101010_10101010_10101010)

This is invalid syntax prior to go1.13 but the module supports anything from go1.12. Integer and a comment?

Aside: we actually support older versions than that via the _compat implementations. Should we change the module back to at least 1.11?

FiloSottile

comment created time in 3 months

Pull request review commentgtank/ristretto255

internal/radix51: implement (*FieldElement).Mul32

 // Copyright (c) 2017 George Tankersley. All rights reserved.+// Copyright (c) 2019 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. -// +build !go1.12+// +build !go1.13

Same as above

FiloSottile

comment created time in 3 months

Pull request review commentgtank/ristretto255

internal/radix51: implement (*FieldElement).Mul32

 // Copyright (c) 2019 George Tankersley. All rights reserved.+// Copyright (c) 2019 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. -// +build go1.12+// +build go1.13

This kicks go1.12 over to _compat even though it has the bits package.

FiloSottile

comment created time in 3 months

pull request commentzcash/zips

ZIP 1003: 20% Split Evenly Between the ECC and the Zcash Foundation, and a Voting System Mandate

I have reviewed this as well. As a ZIP editor, looks good!

daira

comment created time in 3 months

startedchiphuyen/machine-learning-systems-design

started time in 3 months

issue closedgtank/merlin

Cut new release for v0.1.1

Since the v0.1.0 has the bad module descriptor.

closed time in 3 months

s-rah

issue commentgtank/merlin

Cut new release for v0.1.1

Done!

s-rah

comment created time in 3 months

release gtank/merlin

v0.1.1

released time in 3 months

startedw3f/schnorrkel

started time in 3 months

startedmit-dci/utreexo

started time in 3 months

created taggtank/merlin

tagv0.1.1

dalek-compatible implementation of the merlin transcript protocol

created time in 3 months

startedstr4d/rage

started time in 3 months

startedbogatyy/grin-linkability

started time in 3 months

created taggtank/ristretto255

tagv0.1.1

Implements ristretto255, a fast prime-order group.

created time in 3 months

Pull request review commentzcash/zips

Merging a number of ZIPS

+::++  ZIP: unassigned.+  Title: Enforce DevFund Commitments with a Legal Charter+  Owner: unassigned+  Author: @lex-node (zcash forums)+  Advocates: @lex-node (zcash forums) /  @mistfpga  (zcash forums) <steve@mistfpga.net>+  Category: ?+  Created: 2019-08-24+  License: CC BY-SA 4.0 (Creative Commons Attribution-ShareAlike 4.0) [1]+  ++Originally posted and discussed `on the Zcash Community Forum <https://forum.zcashcommunity.com/t/dev-fund-supplemental-proposal-enforce-devfund-commitments-with-legal-charter/34709>`__.+++Terminology+===========++`RFC2119 <https://tools.ietf.org/html/rfc2119>` refrences will be in CAPS.++**The key words include "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and "OPTIONAL"**++For clarity in this ZIP defines these terms:++-  Covenant is defined as a legally binding agreement upon which a specific aspect of development of the zcash protocol and/or adoption is scheduled and agreed upon.++Abstract+========++Supplemental proposal to ensure feature selection and work is community driven.++Hopefully it will be compatible with a number of other zips and can be worked into them.++Out of Scope for this proposal+==============================++-  This proposal does not address the merits, motivations or terms of any particular development funding proposal.+-  Requirements & Implementation+-  The current trademark and sign off issues that just popped up.+++Motivation+==========++This proposal is supplemental to any Development Fund Proposal which places or purports to place conditions on how the Electric Coin Company (ECC) and the Zcash Foundation use development funds or take other related off-chain actions (such requirements, Covenants).++For example, the proposal 20% to a 2-of-3 multisig with community-involved governance provides that “[f]unds accruing to the Zcash Development Fund MUST be used only for … technical work directly connected to the various software implementations of the Zcash protocol.” However, once development funding is approved and implemented via a hardfork, there will be no enforcement mechanism to ensure that the ZF and ECC abide by this requirement.++This proposal aims to provide such an enforcement mechanism. If this proposal is adopted, then the ECC and/or ZF, as applicable MUST enter into a legal agreement which would entitle ZCash (ZEC) holders to enforce ECC’s/ZF’s performance of any Covenants. For purposes of this proposal we will refer to the legal agreement as the “DevFund Charter” or “Charter” for short, but it MAY also be styled in other ways–e.g,. as a Constitution, Bylaws, Fund Governance Agreement, etc.++The DevFund Charter should be used to the benefit of all ZEC users, but the DevFund Charter MAY provide that an enforcement action requires the support of the holders of a plurality, majority or supermajority of ZEC. ZEC held by the ZF, ECC and their officers, directors, employees and/or affiliates SHOULD be excluded from the denominator in calculating the requisite plurality, majority or supermajority of ZEC.++A quorum of yet to be decided number of representatives from a number specified groups from which the DevFund Charter MAY provide that an enforcement action requires the support of the assigned representatives of each.  One such mechanism SHOULD be ZEC balance, however this would require a 66% majority or a 85% super majority. (These numbers are not binding and are up for discussion)++It is assumed that the ECC, Foundation or party with a direct conflict of interest WILL identify their vote/signal - which MAY be rejected from the vote.++Legal enforcement MAY occur in a court of law, non-binding mediation or binding arbitration. The DevFund Charter MAY also provide rights to other ZCash community constituencies, such as specified miners or the “Third Entity” contemplated by the 20% to a 2-of-3 multisig with community-involved governance proposal referenced above.++Rationale+=========++Because ZEC holders do not have specific legal rights against the ECC or ZF, but MAY wish to condition renewed on-chain development funding on the ECC’s or ZF’s agreement to use the development funds in certain ways, ZEC holders SHOULD have the legal right to enforce ECC’s/ZF’s compliance with those agreements.

This touches on a generally larger problem of how exactly to structure this legal arrangement. Given how difficult the trademark assignment was, the ZIP editors are doubtful that this is possible. To merge this proposal, we'd need to see some proposed specific legal mechanism in one of the relevant jurisdictions.

This objection is partially discussed below, but it suggests crowdsourcing and that isn't really an answer.

mistfpga

comment created time in 3 months

Pull request review commentzcash/zips

Merging a number of ZIPS

+::++  ZIP: unassigned.+  Title: Enforce DevFund Commitments with a Legal Charter+  Owner: unassigned+  Author: @lex-node (zcash forums)+  Advocates: @lex-node (zcash forums) /  @mistfpga  (zcash forums) <steve@mistfpga.net>+  Category: ?+  Created: 2019-08-24+  License: CC BY-SA 4.0 (Creative Commons Attribution-ShareAlike 4.0) [1]+  ++Originally posted and discussed `on the Zcash Community Forum <https://forum.zcashcommunity.com/t/dev-fund-supplemental-proposal-enforce-devfund-commitments-with-legal-charter/34709>`__.+++Terminology+===========++`RFC2119 <https://tools.ietf.org/html/rfc2119>` refrences will be in CAPS.++**The key words include "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and "OPTIONAL"**++For clarity in this ZIP defines these terms:++-  Covenant is defined as a legally binding agreement upon which a specific aspect of development of the zcash protocol and/or adoption is scheduled and agreed upon.++Abstract+========++Supplemental proposal to ensure feature selection and work is community driven.++Hopefully it will be compatible with a number of other zips and can be worked into them.++Out of Scope for this proposal+==============================++-  This proposal does not address the merits, motivations or terms of any particular development funding proposal.+-  Requirements & Implementation+-  The current trademark and sign off issues that just popped up.+++Motivation+==========++This proposal is supplemental to any Development Fund Proposal which places or purports to place conditions on how the Electric Coin Company (ECC) and the Zcash Foundation use development funds or take other related off-chain actions (such requirements, Covenants).++For example, the proposal 20% to a 2-of-3 multisig with community-involved governance provides that “[f]unds accruing to the Zcash Development Fund MUST be used only for … technical work directly connected to the various software implementations of the Zcash protocol.” However, once development funding is approved and implemented via a hardfork, there will be no enforcement mechanism to ensure that the ZF and ECC abide by this requirement.++This proposal aims to provide such an enforcement mechanism. If this proposal is adopted, then the ECC and/or ZF, as applicable MUST enter into a legal agreement which would entitle ZCash (ZEC) holders to enforce ECC’s/ZF’s performance of any Covenants. For purposes of this proposal we will refer to the legal agreement as the “DevFund Charter” or “Charter” for short, but it MAY also be styled in other ways–e.g,. as a Constitution, Bylaws, Fund Governance Agreement, etc.++The DevFund Charter should be used to the benefit of all ZEC users, but the DevFund Charter MAY provide that an enforcement action requires the support of the holders of a plurality, majority or supermajority of ZEC. ZEC held by the ZF, ECC and their officers, directors, employees and/or affiliates SHOULD be excluded from the denominator in calculating the requisite plurality, majority or supermajority of ZEC.++A quorum of yet to be decided number of representatives from a number specified groups from which the DevFund Charter MAY provide that an enforcement action requires the support of the assigned representatives of each.  One such mechanism SHOULD be ZEC balance, however this would require a 66% majority or a 85% super majority. (These numbers are not binding and are up for discussion)++It is assumed that the ECC, Foundation or party with a direct conflict of interest WILL identify their vote/signal - which MAY be rejected from the vote.

WILL is not one of the defined key words. SHOULD?

mistfpga

comment created time in 3 months

Pull request review commentzcash/zips

Merging a number of ZIPS

+::++  ZIP: unassigned+  Title: Genuine Protocol opt-in/out donation feature+  Owner: unassigned+  Advocate: mistfpga (zcash forums) <steve@mistfpga.net>+  Category: Protocol+  Created: 2019-07-17+  License: CC BY-SA 4.0 (Creative Commons Attribution-ShareAlike 4.0) [1]++Originally posted and discussed `on the Zcash Community Forum <https://forum.zcashcommunity.com/t/zip-proposal-a-genuine-opt-in-protocol-level-development-donation-option/33846>`__.

This version appears to differ significantly from the one on the forum. It's unclear which is meant to be canonical, and the ZIP editors believe either would need additionaly specificity in terms of motivation and proposed protocol changes, if any. If no consensus protocol changes are required, as seems possible from our reading, then this should also be clearly stated.

mistfpga

comment created time in 3 months

issue commentgtank/merlin

Transcript RNG

Yes, definitely! This is a side project for me so it's slow going, but I've been trying to think of an idiomatic Go way to expose that. At the moment I'm favoring a builder type that will downscope itself to io.Reader when you're ready to start reading from it.

Did you have a favored approach?

noot

comment created time in 4 months

pull request commentgtank/merlin

Update go.mod to have correct module name

Whoops, I guess we started the module before we decided on the repo location? Thanks!

s-rah

comment created time in 4 months

push eventgtank/merlin

Sarah Jamie Lewis

commit sha 8318aed1a79f2b4c16cf2d7c2f09d991ec0731d1

Update go.mod to have correct module name

view details

push time in 4 months

PR merged gtank/merlin

Update go.mod to have correct module name

Currently go mod errors:

go: github.com/gtank/merlin@v0.1.0: parsing go.mod: unexpected module path "merlin"
go: error loading module requirements

+1 -1

1 comment

1 changed file

s-rah

pr closed time in 4 months

startedZcashFoundation/zebra

started time in 4 months

issue commentzcash-hackworks/lightwalletd

Accept runtime arguments from environment variables and config files

We do already use go-ini for parsing Zcash config files over in the RPC client, so that's not out of our way.

This was more referring to allowing cluster orchestrators to pass port assignments and such through environment variables or mounting a config, whatever is idiomatic. Someone's going to need it someday, but not until they start deploying lots of lightwalletd instances on k8s or marathon or something.

gtank

comment created time in 4 months

issue commentzcash/librustzcash

Data Access API

Pinging @dconnolly and @hdevalence, who may be interested in this question for Zebra.

gmale

comment created time in 4 months

startedstarkware-industries/stark101

started time in 4 months

startedscipr-lab/marlin

started time in 4 months

startedadityapk00/zecwallet-light-cli

started time in 4 months

startedFiloSottile/age

started time in 4 months

create barnchgtank/coredns-zcash

branch : address_book

created branch time in 4 months

push eventgtank/coredns-zcash

George Tankersley

commit sha f6ce02e5b860b211ceee4cf23883ce3a1c004a98

zcash: try worker pool instead of goroutine-per-addr

view details

push time in 4 months

create barnchgtank/coredns-zcash

branch : worker_queue

created branch time in 4 months

push eventgtank/coredns-zcash

George Tankersley

commit sha 6b0ca0e25a72e56186ce41c7f628bf29c56619b4

zcash: time out underlying TCP connection

view details

push time in 4 months

push eventgtank/coredns-zcash

George Tankersley

commit sha fdb8953ac20f96d06c2e9c8e6b4b63844af91b18

zcash: make better use of address metadata

view details

push time in 4 months

push eventgtank/coredns-zcash

George Tankersley

commit sha 21b4c750a3fc48913cd0a3b9e9ef44c2f4ae1112

zcash: add address request handling

view details

push time in 4 months

push eventgtank/coredns-zcash

George Tankersley

commit sha a8d0179cfba9c9dda196c404160a9a1d59006115

zcash: add address request handling

view details

push time in 4 months

push eventgtank/coredns-zcash

George Tankersley

commit sha a232eb38e808f090c02ab85c424f9abe4be13cab

zcash: add address request handling

view details

push time in 4 months

push eventgtank/coredns-zcash

George Tankersley

commit sha 82c3ee8aeedfb4f661b7387ece9209e872746bc6

zcash: add address request handling

view details

push time in 4 months

push eventgtank/coredns-zcash

George Tankersley

commit sha 7d12f5e57dc8106ace566d33d6ba7b1a66c53cea

zcash: support alternative ports and clarify APIs

view details

George Tankersley

commit sha d8de354d87711f1c825207341c1e522e4729e207

zcash: add preliminary address crawling

view details

push time in 4 months

fork gtank/bitcoin

Bitcoin Core integration/staging tree

https://bitcoincore.org/en/download

fork in 4 months

push eventgtank/coredns-zcash

George Tankersley

commit sha 7716305c89599df58163d206a85e67e7789190c5

zcash: implement address requests and improve test reliability

view details

push time in 4 months

push eventgtank/coredns-zcash

George Tankersley

commit sha f38c87dec46cfebc48593ed1fcb297ebffdc95d2

zcash: implement address requests and improve test reliability

view details

push time in 4 months

push eventgtank/coredns-zcash

George Tankersley

commit sha e5267c800bb92ef9919604350a49e107d1c2bdac

zcash: implement address requests and improve test reliability

view details

push time in 4 months

push eventgtank/coredns-zcash

George Tankersley

commit sha c6c3f2ca537bd12920b8e0559f5e75389c042cff

zcash: replace mutex tangle with too many sync.Maps

view details

push time in 4 months

push eventgtank/coredns-zcash

George Tankersley

commit sha 56570ba6babcdd5b483c674dbaee2edb8127950c

zcash: replace mutex tangle with too many sync.Maps

view details

push time in 4 months

push eventgtank/coredns-zcash

George Tankersley

commit sha 0f1773c0d1290ecfa8682fca4309a5a8e18c3958

zcash: implement peer handshakes and state tracking

view details

George Tankersley

commit sha 352b865775d742a87d7d4b2f9598fbda793c9544

module: use fork of btcd with mockable peers

view details

push time in 4 months

issue commentbtcsuite/btcd

Example of NewOutboundPeer doesn't work

I've also run into this: I'm trying to use just the peer library (but not the entire node) and would like to be able to generate mock peers. I opened #1481 with a fix.

arberiii

comment created time in 4 months

push eventgtank/btcd

George Tankersley

commit sha b43c61a68604271eba6eb384cdf2c7aaa91e7ec8

peer: add AllowSelfConns to peer config This allows external users of the peer library to mock peers for tests.

view details

push time in 4 months

pull request commentbtcsuite/btcd

peer: Move AllowSelfConns into peer config

Another approach to this would be to move sentNonces out of the global namespace. I'm not sure what approach the maintainers prefer, so I started with the one that seemed lowest impact.

gtank

comment created time in 4 months

PR opened btcsuite/btcd

peer: Move AllowSelfConns into peer config

Currently, the NewOutboundPeer example in the docs can't actually work outside of the peer_test package. This change allows external users of the peer library to designate peer configs as explicitly for testing that will allow self-connections.

Closes https://github.com/btcsuite/btcd/issues/1464

+15 -30

0 comment

4 changed files

pr created time in 4 months

create barnchgtank/btcd

branch : allow_local_tests

created branch time in 4 months

push eventgtank/ristretto255

Sarah Jamie Lewis

commit sha af147e8e15b6e65066ea600a5225ec9e9769305e

Add Stringer & TextMarshaler interface

view details

push time in 4 months

PR merged gtank/ristretto255

Add Stringer & TextMarshaler interface

For more human-friendly logging/debugging & integrating with go's json serialization.

+79 -0

1 comment

3 changed files

s-rah

pr closed time in 4 months

startedstellar/slingshot

started time in 5 months

PublicEvent

Pull request review commentgtank/ristretto255

Add Stringer & TextMarshaler interface

 func TestRistrettoFromUniformBytesTestVectors(t *testing.T) { 		} 	} }++func TestMarshalScalar(t *testing.T) {+	x := new(Scalar)+	// generate an arbitrary scalar+	xbytes := sha512.Sum512([]byte("Hello World"))+	x.FromUniformBytes(xbytes[:])+	text,err := json.Marshal(x)+	if err !=nil {+		t.Fatalf("Could not marshal json: %v", err)+	}+	t.Logf("json: %s", text)+	y := new(Scalar)+	json.Unmarshal(text, y)+	if err !=nil || y.Equal(x) == 0 {

Should this be the err from Unmarshal, or is it meant to be the err from Marshal above?

s-rah

comment created time in 5 months

Pull request review commentgtank/ristretto255

Add Stringer & TextMarshaler interface

 func TestRistrettoFromUniformBytesTestVectors(t *testing.T) { 		} 	} }++func TestMarshalScalar(t *testing.T) {+	x := new(Scalar)+	// generate an arbitrary scalar+	xbytes := sha512.Sum512([]byte("Hello World"))+	x.FromUniformBytes(xbytes[:])+	text,err := json.Marshal(x)+	if err !=nil {+		t.Fatalf("Could not marshal json: %v", err)+	}+	t.Logf("json: %s", text)+	y := new(Scalar)+	json.Unmarshal(text, y)+	if err !=nil || y.Equal(x) == 0 {+		t.Fatalf("Error unmarshaling scalar from json: %s %v", text, err)+	}+}++func TestMarshalElement(t *testing.T) {+	x := new(Element)+	// generate an arbitrary element+	xbytes := sha512.Sum512([]byte("Hello World"))+	x.FromUniformBytes(xbytes[:])+	text,err := json.Marshal(x)+	if err !=nil {+		t.Fatalf("Could not marshal json: %v", err)+	}+	t.Logf("json: %s", text)+	y := new(Element)+	json.Unmarshal(text, y)+	if err !=nil || y.Equal(x) == 0 {

Same question as the other test?

s-rah

comment created time in 5 months

PR opened btcsuite/btcd

peer: fix small typo
+1 -1

0 comment

1 changed file

pr created time in 5 months

push eventgtank/btcd

George Tankersley

commit sha 80d47d4e3246f1a1cb97e1c9e6f9c5cdace6bfed

peer: fix small typo

view details

push time in 5 months

create barnchgtank/btcd

branch : fix_typo

created branch time in 5 months

fork gtank/btcd

An alternative full node bitcoin implementation written in Go (golang)

https://github.com/btcsuite/btcd/blob/master/docs/README.md

fork in 5 months

PublicEvent

startedadityapk00/lightwalletclient

started time in 5 months

startedgoogle/go-attestation

started time in 5 months

Pull request review commentzcash/zips

Add ZIP Draft: Addressing mempool denial-of-service

+::++  ZIP: Unassigned+  Title: Addressing mempool denial-of-service+  Owners: Daira Hopwood <daira@electriccoin.co>+  Status: Draft+  Category: Network+  Created: 2019-09-09+  License: MIT+++Terminology+===========++The key words "MUST", "SHOULD", and "MAY" in this document are to be interpreted+as described in RFC 2119. [#RFC2119]_+++Abstract+========++This proposal specifies a change to the behaviour of zcashd nodes intended to+mitigate denial-of-service from transaction flooding.+++Motivation+==========++Adoption of this proposal would increase robustness of Zcash nodes against+denial-of-service attack, in particular attacks that attempt to exhaust node+memory.++Bitcoin Core added size limitation for the mempool in version 0.12+[#BitcoinCore-PR6722]_, defaulting to 300 MB. This was after Zcash forked from+Bitcoin Core.+++Requirements+============++The memory usage of a node’s mempool should be bounded.++The eviction policy should as far as possible not be “gameable” by an adversary,+i.e. an adversary should not be able to cause legitimate transactions (that do not+themselves present any denial-of-service problem) to be preferentially evicted+relative to its own transactions.++Any configuration options should have reasonable defaults, i.e. without changing+relevant configuration, a node should be adequately protected from denial-of-service+via mempool memory exhaustion.+++Non-requirements+================++The current architecture of Zcash imposes fundamental limits on scaling of+transaction throughput. This proposal does not increase the aggregate transaction+capacity of the network. (The Blossom upgrade does increase transaction capacity,+by a factor of two [#zip-0208]_.)++Denial-of-service issues in the messaging layer of the peer-to-peer protocol are+out of scope for this proposal.++This proposal is focused primarily on memory exhaustion attacks. It does not+attempt to use fees to make denial-of-service economically prohibitive, since that+is unlikely to be feasible while maintaining low fees for legitimate users. It+does not preclude changes in fee policy.+++Specification+=============++This specification describes the intended behaviour of zcashd nodes. Other node+implementations MAY implement the same or similar behaviour, but this is not a+requirement of the network protocol. Thus, RFC 2119 conformance keywords below are+to be interpreted only as placing requirements on the zcashd implementation (and
to be interpreted only as placing requirements on the ``zcashd`` implementation (and
daira

comment created time in 5 months

Pull request review commentzcash/zips

Add ZIP Draft: Addressing mempool denial-of-service

+::++  ZIP: Unassigned+  Title: Addressing mempool denial-of-service+  Owners: Daira Hopwood <daira@electriccoin.co>+  Status: Draft+  Category: Network+  Created: 2019-09-09+  License: MIT+++Terminology+===========++The key words "MUST", "SHOULD", and "MAY" in this document are to be interpreted+as described in RFC 2119. [#RFC2119]_+++Abstract+========++This proposal specifies a change to the behaviour of zcashd nodes intended to+mitigate denial-of-service from transaction flooding.+++Motivation+==========++Adoption of this proposal would increase robustness of Zcash nodes against+denial-of-service attack, in particular attacks that attempt to exhaust node+memory.++Bitcoin Core added size limitation for the mempool in version 0.12+[#BitcoinCore-PR6722]_, defaulting to 300 MB. This was after Zcash forked from+Bitcoin Core.+++Requirements+============++The memory usage of a node’s mempool should be bounded.++The eviction policy should as far as possible not be “gameable” by an adversary,+i.e. an adversary should not be able to cause legitimate transactions (that do not+themselves present any denial-of-service problem) to be preferentially evicted+relative to its own transactions.++Any configuration options should have reasonable defaults, i.e. without changing+relevant configuration, a node should be adequately protected from denial-of-service+via mempool memory exhaustion.+++Non-requirements+================++The current architecture of Zcash imposes fundamental limits on scaling of+transaction throughput. This proposal does not increase the aggregate transaction+capacity of the network. (The Blossom upgrade does increase transaction capacity,+by a factor of two [#zip-0208]_.)++Denial-of-service issues in the messaging layer of the peer-to-peer protocol are+out of scope for this proposal.++This proposal is focused primarily on memory exhaustion attacks. It does not+attempt to use fees to make denial-of-service economically prohibitive, since that+is unlikely to be feasible while maintaining low fees for legitimate users. It+does not preclude changes in fee policy.+++Specification+=============++This specification describes the intended behaviour of zcashd nodes. Other node+implementations MAY implement the same or similar behaviour, but this is not a+requirement of the network protocol. Thus, RFC 2119 conformance keywords below are+to be interpreted only as placing requirements on the zcashd implementation (and+potentially other implementations that have adopted this specification in full).++The mempool of a node holds a set of transactions. Each transaction has a cost,+which is an integer defined as:++  max(serialized transaction size in bytes, 4000) + fee_penalty++where ``fee_penalty`` is 16000 if the transaction pays a fee less than+10000 zatoshi, otherwise 0.++Each node also MUST hold a FIFO queue RecentlyEvicted of pairs (txid, time), where+the time indicates when the given txid was evicted. This SHOULD be empty on node+startup. The size of RecentlyEvicted MUST never exceed 10000 entries.++There MUST be a configuration option ``mempool.tx_cost_limit``, which SHOULD default+to 80000000.++There MUST be a configuration option ``mempool.eviction_memory_minutes``, which+SHOULD default to 60.++On receiving a transaction:++* If it is in RecentlyEvicted, the transaction MUST be dropped.+* Calculate its cost. If the total cost of transactions in the mempool including+  this one would exceed ``mempool.tx_cost_limit``, then the node MUST repeatedly+  call EvictTransaction (with the new transaction included as a candidate to evict)+  until the total cost does not exceed ``mempool.tx_cost_limit``.++EvictTransaction MUST do the following:++* Select a random transaction to evict, weighted by cost.+* Add the txid and the current time to RecentlyEvicted, dropping the oldest entry+  in RecentlyEvicted if necessary to keep it to at most 10000 entries.+* Remove it from the mempool.++Nodes SHOULD remove transactions from RecentlyEvicted that were evicted more than+``mempool.eviction_memory_minutes`` minutes ago. This MAY be done periodically,+and/or just before RecentlyEvicted is accessed when receiving a transaction.+++Rationale+=========++The accounting for transaction size should include some overhead per transaction,+to reflect the cost to the network of processing them (proof and signature+verification; networking overheads; size of in-memory data structures). The+implication of not including overhead is that a denial-of-service attacker would+be likely to use minimum-size transactions so that more of them would fit in a+block, increasing the unaccounted-for overhead. A possible counterargument would+be that the complexity of accounting for this overhead is unwarranted given that+the format of a transaction already imposes a minimum size. However, the proposed+cost function is almost as simple as using transaction size directly.++The threshold 4000 for the cost function is chosen so that the size in bytes of a+typical fully shielded Sapling transaction (with, say, 2 shielded outputs and up+to 5 shielded inputs) will fall below the threshold. This has the effect of+ensuring that such transactions are not evicted preferentially to typical+transparent transactions because of their size.++The proposed eviction policy differs significantly from that of Bitcoin Core+[#BitcoinCore-PR6722]_, which is primarily fee-based. This reflects differing+philosophies about the motivation for fees and the level of fee that legitimate+users can reasonably be expected to pay. The proposed cost function does involve+a penalty for transactions with a fee lower than the standard (0.0001 ZEC) value,+but since there is no further benefit to increasing the fee above the standard+value, it creates no pressure toward escalating fees. For transactions up to+4000 bytes, this penalty makes a transaction that pays less than the standard fee+value five times as likely to be chosen for eviction.++The default value of 80000000 for ``mempool.tx_cost_limit`` represents no more+than 40 blocks’ worth of transactions in the worst case, which is the default

Consider expanding this explanation.

daira

comment created time in 5 months

Pull request review commentzcash/zips

Add ZIP Draft: Addressing mempool denial-of-service

+::++  ZIP: Unassigned+  Title: Addressing mempool denial-of-service+  Owners: Daira Hopwood <daira@electriccoin.co>+  Status: Draft+  Category: Network+  Created: 2019-09-09+  License: MIT+++Terminology+===========++The key words "MUST", "SHOULD", and "MAY" in this document are to be interpreted+as described in RFC 2119. [#RFC2119]_+++Abstract+========++This proposal specifies a change to the behaviour of zcashd nodes intended to+mitigate denial-of-service from transaction flooding.+++Motivation+==========++Adoption of this proposal would increase robustness of Zcash nodes against+denial-of-service attack, in particular attacks that attempt to exhaust node+memory.++Bitcoin Core added size limitation for the mempool in version 0.12+[#BitcoinCore-PR6722]_, defaulting to 300 MB. This was after Zcash forked from+Bitcoin Core.+++Requirements+============++The memory usage of a node’s mempool should be bounded.++The eviction policy should as far as possible not be “gameable” by an adversary,+i.e. an adversary should not be able to cause legitimate transactions (that do not+themselves present any denial-of-service problem) to be preferentially evicted+relative to its own transactions.++Any configuration options should have reasonable defaults, i.e. without changing+relevant configuration, a node should be adequately protected from denial-of-service+via mempool memory exhaustion.+++Non-requirements+================++The current architecture of Zcash imposes fundamental limits on scaling of+transaction throughput. This proposal does not increase the aggregate transaction+capacity of the network. (The Blossom upgrade does increase transaction capacity,+by a factor of two [#zip-0208]_.)++Denial-of-service issues in the messaging layer of the peer-to-peer protocol are+out of scope for this proposal.++This proposal is focused primarily on memory exhaustion attacks. It does not+attempt to use fees to make denial-of-service economically prohibitive, since that+is unlikely to be feasible while maintaining low fees for legitimate users. It+does not preclude changes in fee policy.+++Specification+=============++This specification describes the intended behaviour of zcashd nodes. Other node+implementations MAY implement the same or similar behaviour, but this is not a+requirement of the network protocol. Thus, RFC 2119 conformance keywords below are+to be interpreted only as placing requirements on the zcashd implementation (and+potentially other implementations that have adopted this specification in full).++The mempool of a node holds a set of transactions. Each transaction has a cost,+which is an integer defined as:++  max(serialized transaction size in bytes, 4000) + fee_penalty++where ``fee_penalty`` is 16000 if the transaction pays a fee less than+10000 zatoshi, otherwise 0.++Each node also MUST hold a FIFO queue RecentlyEvicted of pairs (txid, time), where+the time indicates when the given txid was evicted. This SHOULD be empty on node+startup. The size of RecentlyEvicted MUST never exceed 10000 entries.++There MUST be a configuration option ``mempool.tx_cost_limit``, which SHOULD default+to 80000000.++There MUST be a configuration option ``mempool.eviction_memory_minutes``, which+SHOULD default to 60.++On receiving a transaction:++* If it is in RecentlyEvicted, the transaction MUST be dropped.+* Calculate its cost. If the total cost of transactions in the mempool including+  this one would exceed ``mempool.tx_cost_limit``, then the node MUST repeatedly+  call EvictTransaction (with the new transaction included as a candidate to evict)+  until the total cost does not exceed ``mempool.tx_cost_limit``.++EvictTransaction MUST do the following:++* Select a random transaction to evict, weighted by cost.+* Add the txid and the current time to RecentlyEvicted, dropping the oldest entry+  in RecentlyEvicted if necessary to keep it to at most 10000 entries.
  in RecentlyEvicted if necessary to keep it to at most 100000 entries.
daira

comment created time in 5 months

Pull request review commentzcash/zips

Add ZIP Draft: Addressing mempool denial-of-service

+::++  ZIP: Unassigned

400 series

daira

comment created time in 5 months

Pull request review commentzcash/zips

Add ZIP Draft: Addressing mempool denial-of-service

+::++  ZIP: Unassigned+  Title: Addressing mempool denial-of-service+  Owners: Daira Hopwood <daira@electriccoin.co>+  Status: Draft+  Category: Network+  Created: 2019-09-09+  License: MIT+++Terminology+===========++The key words "MUST", "SHOULD", and "MAY" in this document are to be interpreted+as described in RFC 2119. [#RFC2119]_+++Abstract+========++This proposal specifies a change to the behaviour of zcashd nodes intended to+mitigate denial-of-service from transaction flooding.+++Motivation+==========++Adoption of this proposal would increase robustness of Zcash nodes against+denial-of-service attack, in particular attacks that attempt to exhaust node+memory.++Bitcoin Core added size limitation for the mempool in version 0.12+[#BitcoinCore-PR6722]_, defaulting to 300 MB. This was after Zcash forked from+Bitcoin Core.+++Requirements+============++The memory usage of a node’s mempool should be bounded.++The eviction policy should as far as possible not be “gameable” by an adversary,+i.e. an adversary should not be able to cause legitimate transactions (that do not+themselves present any denial-of-service problem) to be preferentially evicted+relative to its own transactions.++Any configuration options should have reasonable defaults, i.e. without changing+relevant configuration, a node should be adequately protected from denial-of-service+via mempool memory exhaustion.+++Non-requirements+================++The current architecture of Zcash imposes fundamental limits on scaling of+transaction throughput. This proposal does not increase the aggregate transaction+capacity of the network. (The Blossom upgrade does increase transaction capacity,+by a factor of two [#zip-0208]_.)++Denial-of-service issues in the messaging layer of the peer-to-peer protocol are+out of scope for this proposal.++This proposal is focused primarily on memory exhaustion attacks. It does not+attempt to use fees to make denial-of-service economically prohibitive, since that+is unlikely to be feasible while maintaining low fees for legitimate users. It+does not preclude changes in fee policy.+++Specification+=============++This specification describes the intended behaviour of zcashd nodes. Other node+implementations MAY implement the same or similar behaviour, but this is not a+requirement of the network protocol. Thus, RFC 2119 conformance keywords below are+to be interpreted only as placing requirements on the zcashd implementation (and+potentially other implementations that have adopted this specification in full).++The mempool of a node holds a set of transactions. Each transaction has a cost,+which is an integer defined as:++  max(serialized transaction size in bytes, 4000) + fee_penalty++where ``fee_penalty`` is 16000 if the transaction pays a fee less than+10000 zatoshi, otherwise 0.++Each node also MUST hold a FIFO queue RecentlyEvicted of pairs (txid, time), where+the time indicates when the given txid was evicted. This SHOULD be empty on node+startup. The size of RecentlyEvicted MUST never exceed 10000 entries.++There MUST be a configuration option ``mempool.tx_cost_limit``, which SHOULD default+to 80000000.++There MUST be a configuration option ``mempool.eviction_memory_minutes``, which+SHOULD default to 60.++On receiving a transaction:++* If it is in RecentlyEvicted, the transaction MUST be dropped.+* Calculate its cost. If the total cost of transactions in the mempool including+  this one would exceed ``mempool.tx_cost_limit``, then the node MUST repeatedly+  call EvictTransaction (with the new transaction included as a candidate to evict)+  until the total cost does not exceed ``mempool.tx_cost_limit``.++EvictTransaction MUST do the following:++* Select a random transaction to evict, weighted by cost.+* Add the txid and the current time to RecentlyEvicted, dropping the oldest entry+  in RecentlyEvicted if necessary to keep it to at most 10000 entries.+* Remove it from the mempool.++Nodes SHOULD remove transactions from RecentlyEvicted that were evicted more than+``mempool.eviction_memory_minutes`` minutes ago. This MAY be done periodically,+and/or just before RecentlyEvicted is accessed when receiving a transaction.+++Rationale+=========++The accounting for transaction size should include some overhead per transaction,+to reflect the cost to the network of processing them (proof and signature+verification; networking overheads; size of in-memory data structures). The+implication of not including overhead is that a denial-of-service attacker would+be likely to use minimum-size transactions so that more of them would fit in a+block, increasing the unaccounted-for overhead. A possible counterargument would+be that the complexity of accounting for this overhead is unwarranted given that+the format of a transaction already imposes a minimum size. However, the proposed+cost function is almost as simple as using transaction size directly.++The threshold 4000 for the cost function is chosen so that the size in bytes of a+typical fully shielded Sapling transaction (with, say, 2 shielded outputs and up+to 5 shielded inputs) will fall below the threshold. This has the effect of+ensuring that such transactions are not evicted preferentially to typical+transparent transactions because of their size.++The proposed eviction policy differs significantly from that of Bitcoin Core+[#BitcoinCore-PR6722]_, which is primarily fee-based. This reflects differing+philosophies about the motivation for fees and the level of fee that legitimate+users can reasonably be expected to pay. The proposed cost function does involve+a penalty for transactions with a fee lower than the standard (0.0001 ZEC) value,+but since there is no further benefit to increasing the fee above the standard+value, it creates no pressure toward escalating fees. For transactions up to+4000 bytes, this penalty makes a transaction that pays less than the standard fee+value five times as likely to be chosen for eviction.++The default value of 80000000 for ``mempool.tx_cost_limit`` represents no more+than 40 blocks’ worth of transactions in the worst case, which is the default+expiration height after Blossom network upgrade [#zip-0208]_. It would serve no+purpose to make it larger.++The ``mempool.tx_cost_limit`` is a per-node configurable parameter in order to+provide flexibility for node operators to change it either in response to+attempted denial-of-service attacks, or if needed to handle spikes in transaction+demand. It may also be useful for nodes running in memory-constrained environments+to reduce this parameter.++The limit of 100000 entries in RecentlyEvicted bounds the memory needed for this+data structure. Since a txid is 32 bytes and a timestamp 8 bytes, 100000 entries+can be stored in ~4 MB, which is small compared to other node memory usage (in+particular, small compared to the maximum memory usage of the mempool itself under+the default ``mempool.tx_cost_limit``). 100000 entries should be sufficient to+mitigate any performance loss caused by re-accepting transactions that were +previously evicted. In particular, since a transaction has a minimum cost of 1000,+and the default ``mempool.tx_cost_limit`` is 80000000, at most 80000 transactions

80M / 4000 = 20000

daira

comment created time in 5 months

Pull request review commentzcash/zips

Add ZIP Draft: Addressing mempool denial-of-service

+::++  ZIP: Unassigned+  Title: Addressing mempool denial-of-service+  Owners: Daira Hopwood <daira@electriccoin.co>+  Status: Draft+  Category: Network+  Created: 2019-09-09+  License: MIT+++Terminology+===========++The key words "MUST", "SHOULD", and "MAY" in this document are to be interpreted+as described in RFC 2119. [#RFC2119]_+++Abstract+========++This proposal specifies a change to the behaviour of zcashd nodes intended to+mitigate denial-of-service from transaction flooding.+++Motivation+==========++Adoption of this proposal would increase robustness of Zcash nodes against+denial-of-service attack, in particular attacks that attempt to exhaust node+memory.++Bitcoin Core added size limitation for the mempool in version 0.12+[#BitcoinCore-PR6722]_, defaulting to 300 MB. This was after Zcash forked from+Bitcoin Core.+++Requirements+============++The memory usage of a node’s mempool should be bounded.++The eviction policy should as far as possible not be “gameable” by an adversary,+i.e. an adversary should not be able to cause legitimate transactions (that do not+themselves present any denial-of-service problem) to be preferentially evicted+relative to its own transactions.++Any configuration options should have reasonable defaults, i.e. without changing+relevant configuration, a node should be adequately protected from denial-of-service+via mempool memory exhaustion.+++Non-requirements+================++The current architecture of Zcash imposes fundamental limits on scaling of+transaction throughput. This proposal does not increase the aggregate transaction+capacity of the network. (The Blossom upgrade does increase transaction capacity,+by a factor of two [#zip-0208]_.)++Denial-of-service issues in the messaging layer of the peer-to-peer protocol are+out of scope for this proposal.++This proposal is focused primarily on memory exhaustion attacks. It does not+attempt to use fees to make denial-of-service economically prohibitive, since that+is unlikely to be feasible while maintaining low fees for legitimate users. It+does not preclude changes in fee policy.+++Specification+=============++This specification describes the intended behaviour of zcashd nodes. Other node+implementations MAY implement the same or similar behaviour, but this is not a+requirement of the network protocol. Thus, RFC 2119 conformance keywords below are+to be interpreted only as placing requirements on the zcashd implementation (and+potentially other implementations that have adopted this specification in full).++The mempool of a node holds a set of transactions. Each transaction has a cost,+which is an integer defined as:++  max(serialized transaction size in bytes, 4000) + fee_penalty++where ``fee_penalty`` is 16000 if the transaction pays a fee less than+10000 zatoshi, otherwise 0.++Each node also MUST hold a FIFO queue RecentlyEvicted of pairs (txid, time), where+the time indicates when the given txid was evicted. This SHOULD be empty on node+startup. The size of RecentlyEvicted MUST never exceed 10000 entries.++There MUST be a configuration option ``mempool.tx_cost_limit``, which SHOULD default+to 80000000.++There MUST be a configuration option ``mempool.eviction_memory_minutes``, which+SHOULD default to 60.++On receiving a transaction:++* If it is in RecentlyEvicted, the transaction MUST be dropped.+* Calculate its cost. If the total cost of transactions in the mempool including+  this one would exceed ``mempool.tx_cost_limit``, then the node MUST repeatedly+  call EvictTransaction (with the new transaction included as a candidate to evict)+  until the total cost does not exceed ``mempool.tx_cost_limit``.++EvictTransaction MUST do the following:++* Select a random transaction to evict, weighted by cost.+* Add the txid and the current time to RecentlyEvicted, dropping the oldest entry+  in RecentlyEvicted if necessary to keep it to at most 10000 entries.+* Remove it from the mempool.++Nodes SHOULD remove transactions from RecentlyEvicted that were evicted more than+``mempool.eviction_memory_minutes`` minutes ago. This MAY be done periodically,+and/or just before RecentlyEvicted is accessed when receiving a transaction.+++Rationale+=========++The accounting for transaction size should include some overhead per transaction,+to reflect the cost to the network of processing them (proof and signature+verification; networking overheads; size of in-memory data structures). The+implication of not including overhead is that a denial-of-service attacker would+be likely to use minimum-size transactions so that more of them would fit in a+block, increasing the unaccounted-for overhead. A possible counterargument would+be that the complexity of accounting for this overhead is unwarranted given that+the format of a transaction already imposes a minimum size. However, the proposed+cost function is almost as simple as using transaction size directly.++The threshold 4000 for the cost function is chosen so that the size in bytes of a+typical fully shielded Sapling transaction (with, say, 2 shielded outputs and up+to 5 shielded inputs) will fall below the threshold. This has the effect of+ensuring that such transactions are not evicted preferentially to typical+transparent transactions because of their size.++The proposed eviction policy differs significantly from that of Bitcoin Core+[#BitcoinCore-PR6722]_, which is primarily fee-based. This reflects differing+philosophies about the motivation for fees and the level of fee that legitimate+users can reasonably be expected to pay. The proposed cost function does involve+a penalty for transactions with a fee lower than the standard (0.0001 ZEC) value,+but since there is no further benefit to increasing the fee above the standard+value, it creates no pressure toward escalating fees. For transactions up to+4000 bytes, this penalty makes a transaction that pays less than the standard fee+value five times as likely to be chosen for eviction.++The default value of 80000000 for ``mempool.tx_cost_limit`` represents no more+than 40 blocks’ worth of transactions in the worst case, which is the default+expiration height after Blossom network upgrade [#zip-0208]_. It would serve no+purpose to make it larger.++The ``mempool.tx_cost_limit`` is a per-node configurable parameter in order to+provide flexibility for node operators to change it either in response to+attempted denial-of-service attacks, or if needed to handle spikes in transaction+demand. It may also be useful for nodes running in memory-constrained environments+to reduce this parameter.++The limit of 100000 entries in RecentlyEvicted bounds the memory needed for this+data structure. Since a txid is 32 bytes and a timestamp 8 bytes, 100000 entries+can be stored in ~4 MB, which is small compared to other node memory usage (in+particular, small compared to the maximum memory usage of the mempool itself under+the default ``mempool.tx_cost_limit``). 100000 entries should be sufficient to+mitigate any performance loss caused by re-accepting transactions that were +previously evicted. In particular, since a transaction has a minimum cost of 1000,
previously evicted. In particular, since a transaction has a minimum cost of 4000,
daira

comment created time in 5 months

Pull request review commentzcash/zips

Add ZIP Draft: Addressing mempool denial-of-service

+::++  ZIP: Unassigned+  Title: Addressing mempool denial-of-service+  Owners: Daira Hopwood <daira@electriccoin.co>+  Status: Draft+  Category: Network+  Created: 2019-09-09+  License: MIT+++Terminology+===========++The key words "MUST", "SHOULD", and "MAY" in this document are to be interpreted+as described in RFC 2119. [#RFC2119]_+++Abstract+========++This proposal specifies a change to the behaviour of zcashd nodes intended to+mitigate denial-of-service from transaction flooding.+++Motivation+==========++Adoption of this proposal would increase robustness of Zcash nodes against+denial-of-service attack, in particular attacks that attempt to exhaust node+memory.++Bitcoin Core added size limitation for the mempool in version 0.12+[#BitcoinCore-PR6722]_, defaulting to 300 MB. This was after Zcash forked from+Bitcoin Core.+++Requirements+============++The memory usage of a node’s mempool should be bounded.++The eviction policy should as far as possible not be “gameable” by an adversary,+i.e. an adversary should not be able to cause legitimate transactions (that do not+themselves present any denial-of-service problem) to be preferentially evicted+relative to its own transactions.++Any configuration options should have reasonable defaults, i.e. without changing+relevant configuration, a node should be adequately protected from denial-of-service+via mempool memory exhaustion.+++Non-requirements+================++The current architecture of Zcash imposes fundamental limits on scaling of+transaction throughput. This proposal does not increase the aggregate transaction+capacity of the network. (The Blossom upgrade does increase transaction capacity,+by a factor of two [#zip-0208]_.)++Denial-of-service issues in the messaging layer of the peer-to-peer protocol are+out of scope for this proposal.++This proposal is focused primarily on memory exhaustion attacks. It does not+attempt to use fees to make denial-of-service economically prohibitive, since that+is unlikely to be feasible while maintaining low fees for legitimate users. It+does not preclude changes in fee policy.+++Specification+=============++This specification describes the intended behaviour of zcashd nodes. Other node+implementations MAY implement the same or similar behaviour, but this is not a+requirement of the network protocol. Thus, RFC 2119 conformance keywords below are+to be interpreted only as placing requirements on the zcashd implementation (and+potentially other implementations that have adopted this specification in full).++The mempool of a node holds a set of transactions. Each transaction has a cost,+which is an integer defined as:++  max(serialized transaction size in bytes, 4000) + fee_penalty++where ``fee_penalty`` is 16000 if the transaction pays a fee less than+10000 zatoshi, otherwise 0.++Each node also MUST hold a FIFO queue RecentlyEvicted of pairs (txid, time), where+the time indicates when the given txid was evicted. This SHOULD be empty on node+startup. The size of RecentlyEvicted MUST never exceed 10000 entries.++There MUST be a configuration option ``mempool.tx_cost_limit``, which SHOULD default+to 80000000.++There MUST be a configuration option ``mempool.eviction_memory_minutes``, which+SHOULD default to 60.++On receiving a transaction:++* If it is in RecentlyEvicted, the transaction MUST be dropped.+* Calculate its cost. If the total cost of transactions in the mempool including+  this one would exceed ``mempool.tx_cost_limit``, then the node MUST repeatedly+  call EvictTransaction (with the new transaction included as a candidate to evict)+  until the total cost does not exceed ``mempool.tx_cost_limit``.++EvictTransaction MUST do the following:++* Select a random transaction to evict, weighted by cost.+* Add the txid and the current time to RecentlyEvicted, dropping the oldest entry+  in RecentlyEvicted if necessary to keep it to at most 10000 entries.+* Remove it from the mempool.++Nodes SHOULD remove transactions from RecentlyEvicted that were evicted more than+``mempool.eviction_memory_minutes`` minutes ago. This MAY be done periodically,+and/or just before RecentlyEvicted is accessed when receiving a transaction.+++Rationale+=========++The accounting for transaction size should include some overhead per transaction,+to reflect the cost to the network of processing them (proof and signature+verification; networking overheads; size of in-memory data structures). The+implication of not including overhead is that a denial-of-service attacker would+be likely to use minimum-size transactions so that more of them would fit in a+block, increasing the unaccounted-for overhead. A possible counterargument would+be that the complexity of accounting for this overhead is unwarranted given that+the format of a transaction already imposes a minimum size. However, the proposed+cost function is almost as simple as using transaction size directly.++The threshold 4000 for the cost function is chosen so that the size in bytes of a+typical fully shielded Sapling transaction (with, say, 2 shielded outputs and up+to 5 shielded inputs) will fall below the threshold. This has the effect of+ensuring that such transactions are not evicted preferentially to typical+transparent transactions because of their size.++The proposed eviction policy differs significantly from that of Bitcoin Core+[#BitcoinCore-PR6722]_, which is primarily fee-based. This reflects differing+philosophies about the motivation for fees and the level of fee that legitimate+users can reasonably be expected to pay. The proposed cost function does involve+a penalty for transactions with a fee lower than the standard (0.0001 ZEC) value,+but since there is no further benefit to increasing the fee above the standard+value, it creates no pressure toward escalating fees. For transactions up to+4000 bytes, this penalty makes a transaction that pays less than the standard fee+value five times as likely to be chosen for eviction.
value five times as likely to be chosen for eviction (because 4000 + 16000 = 20000 = 4000 \* 5).
daira

comment created time in 5 months

Pull request review commentzcash/zips

Add ZIP Draft: Addressing mempool denial-of-service

+::++  ZIP: Unassigned+  Title: Addressing mempool denial-of-service+  Owners: Daira Hopwood <daira@electriccoin.co>+  Status: Draft+  Category: Network+  Created: 2019-09-09+  License: MIT+++Terminology+===========++The key words "MUST", "SHOULD", and "MAY" in this document are to be interpreted+as described in RFC 2119. [#RFC2119]_+++Abstract+========++This proposal specifies a change to the behaviour of zcashd nodes intended to+mitigate denial-of-service from transaction flooding.+++Motivation+==========++Adoption of this proposal would increase robustness of Zcash nodes against+denial-of-service attack, in particular attacks that attempt to exhaust node+memory.++Bitcoin Core added size limitation for the mempool in version 0.12+[#BitcoinCore-PR6722]_, defaulting to 300 MB. This was after Zcash forked from+Bitcoin Core.+++Requirements+============++The memory usage of a node’s mempool should be bounded.++The eviction policy should as far as possible not be “gameable” by an adversary,+i.e. an adversary should not be able to cause legitimate transactions (that do not+themselves present any denial-of-service problem) to be preferentially evicted+relative to its own transactions.++Any configuration options should have reasonable defaults, i.e. without changing+relevant configuration, a node should be adequately protected from denial-of-service+via mempool memory exhaustion.+++Non-requirements+================++The current architecture of Zcash imposes fundamental limits on scaling of+transaction throughput. This proposal does not increase the aggregate transaction+capacity of the network. (The Blossom upgrade does increase transaction capacity,+by a factor of two [#zip-0208]_.)++Denial-of-service issues in the messaging layer of the peer-to-peer protocol are+out of scope for this proposal.++This proposal is focused primarily on memory exhaustion attacks. It does not+attempt to use fees to make denial-of-service economically prohibitive, since that+is unlikely to be feasible while maintaining low fees for legitimate users. It+does not preclude changes in fee policy.+++Specification+=============++This specification describes the intended behaviour of zcashd nodes. Other node+implementations MAY implement the same or similar behaviour, but this is not a+requirement of the network protocol. Thus, RFC 2119 conformance keywords below are+to be interpreted only as placing requirements on the zcashd implementation (and+potentially other implementations that have adopted this specification in full).++The mempool of a node holds a set of transactions. Each transaction has a cost,+which is an integer defined as:++  max(serialized transaction size in bytes, 4000) + fee_penalty++where ``fee_penalty`` is 16000 if the transaction pays a fee less than+10000 zatoshi, otherwise 0.++Each node also MUST hold a FIFO queue RecentlyEvicted of pairs (txid, time), where+the time indicates when the given txid was evicted. This SHOULD be empty on node+startup. The size of RecentlyEvicted MUST never exceed 10000 entries.++There MUST be a configuration option ``mempool.tx_cost_limit``, which SHOULD default+to 80000000.++There MUST be a configuration option ``mempool.eviction_memory_minutes``, which+SHOULD default to 60.++On receiving a transaction:++* If it is in RecentlyEvicted, the transaction MUST be dropped.+* Calculate its cost. If the total cost of transactions in the mempool including+  this one would exceed ``mempool.tx_cost_limit``, then the node MUST repeatedly+  call EvictTransaction (with the new transaction included as a candidate to evict)+  until the total cost does not exceed ``mempool.tx_cost_limit``.++EvictTransaction MUST do the following:++* Select a random transaction to evict, weighted by cost.+* Add the txid and the current time to RecentlyEvicted, dropping the oldest entry+  in RecentlyEvicted if necessary to keep it to at most 10000 entries.+* Remove it from the mempool.++Nodes SHOULD remove transactions from RecentlyEvicted that were evicted more than+``mempool.eviction_memory_minutes`` minutes ago. This MAY be done periodically,+and/or just before RecentlyEvicted is accessed when receiving a transaction.+++Rationale+=========++The accounting for transaction size should include some overhead per transaction,+to reflect the cost to the network of processing them (proof and signature+verification; networking overheads; size of in-memory data structures). The+implication of not including overhead is that a denial-of-service attacker would+be likely to use minimum-size transactions so that more of them would fit in a+block, increasing the unaccounted-for overhead. A possible counterargument would+be that the complexity of accounting for this overhead is unwarranted given that+the format of a transaction already imposes a minimum size. However, the proposed+cost function is almost as simple as using transaction size directly.++The threshold 4000 for the cost function is chosen so that the size in bytes of a+typical fully shielded Sapling transaction (with, say, 2 shielded outputs and up+to 5 shielded inputs) will fall below the threshold. This has the effect of+ensuring that such transactions are not evicted preferentially to typical+transparent transactions because of their size.++The proposed eviction policy differs significantly from that of Bitcoin Core+[#BitcoinCore-PR6722]_, which is primarily fee-based. This reflects differing+philosophies about the motivation for fees and the level of fee that legitimate+users can reasonably be expected to pay. The proposed cost function does involve+a penalty for transactions with a fee lower than the standard (0.0001 ZEC) value,+but since there is no further benefit to increasing the fee above the standard+value, it creates no pressure toward escalating fees. For transactions up to+4000 bytes, this penalty makes a transaction that pays less than the standard fee+value five times as likely to be chosen for eviction.++The default value of 80000000 for ``mempool.tx_cost_limit`` represents no more+than 40 blocks’ worth of transactions in the worst case, which is the default+expiration height after Blossom network upgrade [#zip-0208]_. It would serve no+purpose to make it larger.++The ``mempool.tx_cost_limit`` is a per-node configurable parameter in order to+provide flexibility for node operators to change it either in response to+attempted denial-of-service attacks, or if needed to handle spikes in transaction+demand. It may also be useful for nodes running in memory-constrained environments+to reduce this parameter.++The limit of 100000 entries in RecentlyEvicted bounds the memory needed for this+data structure. Since a txid is 32 bytes and a timestamp 8 bytes, 100000 entries+can be stored in ~4 MB, which is small compared to other node memory usage (in+particular, small compared to the maximum memory usage of the mempool itself under+the default ``mempool.tx_cost_limit``). 100000 entries should be sufficient to
the default ``mempool.total_cost_limit``). 100000 entries should be sufficient to
daira

comment created time in 5 months

Pull request review commentzcash/zips

Add ZIP Draft: Addressing mempool denial-of-service

+::++  ZIP: Unassigned+  Title: Addressing mempool denial-of-service+  Owners: Daira Hopwood <daira@electriccoin.co>+  Status: Draft+  Category: Network+  Created: 2019-09-09+  License: MIT+++Terminology+===========++The key words "MUST", "SHOULD", and "MAY" in this document are to be interpreted+as described in RFC 2119. [#RFC2119]_+++Abstract+========++This proposal specifies a change to the behaviour of zcashd nodes intended to+mitigate denial-of-service from transaction flooding.+++Motivation+==========++Adoption of this proposal would increase robustness of Zcash nodes against+denial-of-service attack, in particular attacks that attempt to exhaust node+memory.++Bitcoin Core added size limitation for the mempool in version 0.12+[#BitcoinCore-PR6722]_, defaulting to 300 MB. This was after Zcash forked from+Bitcoin Core.+++Requirements+============++The memory usage of a node’s mempool should be bounded.++The eviction policy should as far as possible not be “gameable” by an adversary,+i.e. an adversary should not be able to cause legitimate transactions (that do not+themselves present any denial-of-service problem) to be preferentially evicted+relative to its own transactions.++Any configuration options should have reasonable defaults, i.e. without changing+relevant configuration, a node should be adequately protected from denial-of-service+via mempool memory exhaustion.+++Non-requirements+================++The current architecture of Zcash imposes fundamental limits on scaling of+transaction throughput. This proposal does not increase the aggregate transaction+capacity of the network. (The Blossom upgrade does increase transaction capacity,+by a factor of two [#zip-0208]_.)++Denial-of-service issues in the messaging layer of the peer-to-peer protocol are+out of scope for this proposal.++This proposal is focused primarily on memory exhaustion attacks. It does not+attempt to use fees to make denial-of-service economically prohibitive, since that+is unlikely to be feasible while maintaining low fees for legitimate users. It+does not preclude changes in fee policy.+++Specification+=============++This specification describes the intended behaviour of zcashd nodes. Other node+implementations MAY implement the same or similar behaviour, but this is not a+requirement of the network protocol. Thus, RFC 2119 conformance keywords below are+to be interpreted only as placing requirements on the zcashd implementation (and+potentially other implementations that have adopted this specification in full).++The mempool of a node holds a set of transactions. Each transaction has a cost,+which is an integer defined as:++  max(serialized transaction size in bytes, 4000) + fee_penalty++where ``fee_penalty`` is 16000 if the transaction pays a fee less than+10000 zatoshi, otherwise 0.++Each node also MUST hold a FIFO queue RecentlyEvicted of pairs (txid, time), where+the time indicates when the given txid was evicted. This SHOULD be empty on node+startup. The size of RecentlyEvicted MUST never exceed 10000 entries.++There MUST be a configuration option ``mempool.tx_cost_limit``, which SHOULD default+to 80000000.++There MUST be a configuration option ``mempool.eviction_memory_minutes``, which+SHOULD default to 60.++On receiving a transaction:++* If it is in RecentlyEvicted, the transaction MUST be dropped.+* Calculate its cost. If the total cost of transactions in the mempool including+  this one would exceed ``mempool.tx_cost_limit``, then the node MUST repeatedly+  call EvictTransaction (with the new transaction included as a candidate to evict)+  until the total cost does not exceed ``mempool.tx_cost_limit``.++EvictTransaction MUST do the following:++* Select a random transaction to evict, weighted by cost.+* Add the txid and the current time to RecentlyEvicted, dropping the oldest entry+  in RecentlyEvicted if necessary to keep it to at most 10000 entries.+* Remove it from the mempool.++Nodes SHOULD remove transactions from RecentlyEvicted that were evicted more than+``mempool.eviction_memory_minutes`` minutes ago. This MAY be done periodically,+and/or just before RecentlyEvicted is accessed when receiving a transaction.+++Rationale+=========++The accounting for transaction size should include some overhead per transaction,+to reflect the cost to the network of processing them (proof and signature+verification; networking overheads; size of in-memory data structures). The+implication of not including overhead is that a denial-of-service attacker would+be likely to use minimum-size transactions so that more of them would fit in a+block, increasing the unaccounted-for overhead. A possible counterargument would+be that the complexity of accounting for this overhead is unwarranted given that+the format of a transaction already imposes a minimum size. However, the proposed+cost function is almost as simple as using transaction size directly.++The threshold 4000 for the cost function is chosen so that the size in bytes of a+typical fully shielded Sapling transaction (with, say, 2 shielded outputs and up+to 5 shielded inputs) will fall below the threshold. This has the effect of+ensuring that such transactions are not evicted preferentially to typical+transparent transactions because of their size.

We know of at least two legitimate use patterns that substantially diverge from this:

  1. Exchanges sweeping from user-specific ingress addresses to other parts of their fund management infrastructure will likely have a lot of inputs from consolidating small notes

  2. Miner payouts will have lots of outputs

However, it seems reasonable to address the problem we have now based on the behavior we see now. Since this is not a consensus-critical ZIP it will be simple to address future problems that may arise.

daira

comment created time in 5 months

Pull request review commentzcash/zips

Add ZIP Draft: Addressing mempool denial-of-service

+::++  ZIP: Unassigned+  Title: Addressing mempool denial-of-service+  Owners: Daira Hopwood <daira@electriccoin.co>+  Status: Draft+  Category: Network+  Created: 2019-09-09+  License: MIT+++Terminology+===========++The key words "MUST", "SHOULD", and "MAY" in this document are to be interpreted+as described in RFC 2119. [#RFC2119]_+++Abstract+========++This proposal specifies a change to the behaviour of zcashd nodes intended to+mitigate denial-of-service from transaction flooding.+++Motivation+==========++Adoption of this proposal would increase robustness of Zcash nodes against+denial-of-service attack, in particular attacks that attempt to exhaust node+memory.++Bitcoin Core added size limitation for the mempool in version 0.12+[#BitcoinCore-PR6722]_, defaulting to 300 MB. This was after Zcash forked from+Bitcoin Core.+++Requirements+============++The memory usage of a node’s mempool should be bounded.++The eviction policy should as far as possible not be “gameable” by an adversary,+i.e. an adversary should not be able to cause legitimate transactions (that do not+themselves present any denial-of-service problem) to be preferentially evicted+relative to its own transactions.++Any configuration options should have reasonable defaults, i.e. without changing+relevant configuration, a node should be adequately protected from denial-of-service+via mempool memory exhaustion.+++Non-requirements+================++The current architecture of Zcash imposes fundamental limits on scaling of+transaction throughput. This proposal does not increase the aggregate transaction+capacity of the network. (The Blossom upgrade does increase transaction capacity,+by a factor of two [#zip-0208]_.)++Denial-of-service issues in the messaging layer of the peer-to-peer protocol are+out of scope for this proposal.++This proposal is focused primarily on memory exhaustion attacks. It does not+attempt to use fees to make denial-of-service economically prohibitive, since that+is unlikely to be feasible while maintaining low fees for legitimate users. It+does not preclude changes in fee policy.+++Specification+=============++This specification describes the intended behaviour of zcashd nodes. Other node+implementations MAY implement the same or similar behaviour, but this is not a+requirement of the network protocol. Thus, RFC 2119 conformance keywords below are+to be interpreted only as placing requirements on the zcashd implementation (and+potentially other implementations that have adopted this specification in full).++The mempool of a node holds a set of transactions. Each transaction has a cost,+which is an integer defined as:++  max(serialized transaction size in bytes, 4000) + fee_penalty++where ``fee_penalty`` is 16000 if the transaction pays a fee less than+10000 zatoshi, otherwise 0.++Each node also MUST hold a FIFO queue RecentlyEvicted of pairs (txid, time), where+the time indicates when the given txid was evicted. This SHOULD be empty on node+startup. The size of RecentlyEvicted MUST never exceed 10000 entries.++There MUST be a configuration option ``mempool.tx_cost_limit``, which SHOULD default+to 80000000.++There MUST be a configuration option ``mempool.eviction_memory_minutes``, which+SHOULD default to 60.++On receiving a transaction:++* If it is in RecentlyEvicted, the transaction MUST be dropped.+* Calculate its cost. If the total cost of transactions in the mempool including+  this one would exceed ``mempool.tx_cost_limit``, then the node MUST repeatedly+  call EvictTransaction (with the new transaction included as a candidate to evict)+  until the total cost does not exceed ``mempool.tx_cost_limit``.++EvictTransaction MUST do the following:++* Select a random transaction to evict, weighted by cost.+* Add the txid and the current time to RecentlyEvicted, dropping the oldest entry+  in RecentlyEvicted if necessary to keep it to at most 10000 entries.+* Remove it from the mempool.++Nodes SHOULD remove transactions from RecentlyEvicted that were evicted more than+``mempool.eviction_memory_minutes`` minutes ago. This MAY be done periodically,+and/or just before RecentlyEvicted is accessed when receiving a transaction.+++Rationale+=========++The accounting for transaction size should include some overhead per transaction,+to reflect the cost to the network of processing them (proof and signature+verification; networking overheads; size of in-memory data structures). The+implication of not including overhead is that a denial-of-service attacker would+be likely to use minimum-size transactions so that more of them would fit in a+block, increasing the unaccounted-for overhead. A possible counterargument would+be that the complexity of accounting for this overhead is unwarranted given that+the format of a transaction already imposes a minimum size. However, the proposed+cost function is almost as simple as using transaction size directly.++The threshold 4000 for the cost function is chosen so that the size in bytes of a+typical fully shielded Sapling transaction (with, say, 2 shielded outputs and up+to 5 shielded inputs) will fall below the threshold. This has the effect of+ensuring that such transactions are not evicted preferentially to typical+transparent transactions because of their size.++The proposed eviction policy differs significantly from that of Bitcoin Core+[#BitcoinCore-PR6722]_, which is primarily fee-based. This reflects differing+philosophies about the motivation for fees and the level of fee that legitimate+users can reasonably be expected to pay. The proposed cost function does involve+a penalty for transactions with a fee lower than the standard (0.0001 ZEC) value,+but since there is no further benefit to increasing the fee above the standard+value, it creates no pressure toward escalating fees. For transactions up to+4000 bytes, this penalty makes a transaction that pays less than the standard fee+value five times as likely to be chosen for eviction.++The default value of 80000000 for ``mempool.tx_cost_limit`` represents no more+than 40 blocks’ worth of transactions in the worst case, which is the default+expiration height after Blossom network upgrade [#zip-0208]_. It would serve no+purpose to make it larger.++The ``mempool.tx_cost_limit`` is a per-node configurable parameter in order to+provide flexibility for node operators to change it either in response to+attempted denial-of-service attacks, or if needed to handle spikes in transaction+demand. It may also be useful for nodes running in memory-constrained environments+to reduce this parameter.++The limit of 100000 entries in RecentlyEvicted bounds the memory needed for this+data structure. Since a txid is 32 bytes and a timestamp 8 bytes, 100000 entries+can be stored in ~4 MB, which is small compared to other node memory usage (in+particular, small compared to the maximum memory usage of the mempool itself under+the default ``mempool.tx_cost_limit``). 100000 entries should be sufficient to+mitigate any performance loss caused by re-accepting transactions that were +previously evicted. In particular, since a transaction has a minimum cost of 1000,+and the default ``mempool.tx_cost_limit`` is 80000000, at most 80000 transactions
and the default ``mempool.total_cost_limit`` is 80000000, at most 80000 transactions
daira

comment created time in 5 months

Pull request review commentzcash/zips

Add ZIP Draft: Addressing mempool denial-of-service

+::++  ZIP: Unassigned+  Title: Addressing mempool denial-of-service+  Owners: Daira Hopwood <daira@electriccoin.co>+  Status: Draft+  Category: Network+  Created: 2019-09-09+  License: MIT+++Terminology+===========++The key words "MUST", "SHOULD", and "MAY" in this document are to be interpreted+as described in RFC 2119. [#RFC2119]_+++Abstract+========++This proposal specifies a change to the behaviour of zcashd nodes intended to+mitigate denial-of-service from transaction flooding.+++Motivation+==========++Adoption of this proposal would increase robustness of Zcash nodes against+denial-of-service attack, in particular attacks that attempt to exhaust node+memory.++Bitcoin Core added size limitation for the mempool in version 0.12+[#BitcoinCore-PR6722]_, defaulting to 300 MB. This was after Zcash forked from+Bitcoin Core.+++Requirements+============++The memory usage of a node’s mempool should be bounded.++The eviction policy should as far as possible not be “gameable” by an adversary,+i.e. an adversary should not be able to cause legitimate transactions (that do not+themselves present any denial-of-service problem) to be preferentially evicted+relative to its own transactions.++Any configuration options should have reasonable defaults, i.e. without changing+relevant configuration, a node should be adequately protected from denial-of-service+via mempool memory exhaustion.+++Non-requirements+================++The current architecture of Zcash imposes fundamental limits on scaling of+transaction throughput. This proposal does not increase the aggregate transaction+capacity of the network. (The Blossom upgrade does increase transaction capacity,+by a factor of two [#zip-0208]_.)++Denial-of-service issues in the messaging layer of the peer-to-peer protocol are+out of scope for this proposal.++This proposal is focused primarily on memory exhaustion attacks. It does not+attempt to use fees to make denial-of-service economically prohibitive, since that+is unlikely to be feasible while maintaining low fees for legitimate users. It+does not preclude changes in fee policy.+++Specification+=============++This specification describes the intended behaviour of zcashd nodes. Other node+implementations MAY implement the same or similar behaviour, but this is not a+requirement of the network protocol. Thus, RFC 2119 conformance keywords below are+to be interpreted only as placing requirements on the zcashd implementation (and+potentially other implementations that have adopted this specification in full).++The mempool of a node holds a set of transactions. Each transaction has a cost,+which is an integer defined as:++  max(serialized transaction size in bytes, 4000) + fee_penalty++where ``fee_penalty`` is 16000 if the transaction pays a fee less than+10000 zatoshi, otherwise 0.++Each node also MUST hold a FIFO queue RecentlyEvicted of pairs (txid, time), where+the time indicates when the given txid was evicted. This SHOULD be empty on node+startup. The size of RecentlyEvicted MUST never exceed 10000 entries.++There MUST be a configuration option ``mempool.tx_cost_limit``, which SHOULD default+to 80000000.++There MUST be a configuration option ``mempool.eviction_memory_minutes``, which+SHOULD default to 60.++On receiving a transaction:++* If it is in RecentlyEvicted, the transaction MUST be dropped.+* Calculate its cost. If the total cost of transactions in the mempool including+  this one would exceed ``mempool.tx_cost_limit``, then the node MUST repeatedly+  call EvictTransaction (with the new transaction included as a candidate to evict)+  until the total cost does not exceed ``mempool.tx_cost_limit``.
  until the total cost does not exceed ``mempool.total_cost_limit``.
daira

comment created time in 5 months

Pull request review commentzcash/zips

Add ZIP Draft: Addressing mempool denial-of-service

+::++  ZIP: Unassigned+  Title: Addressing mempool denial-of-service+  Owners: Daira Hopwood <daira@electriccoin.co>+  Status: Draft+  Category: Network+  Created: 2019-09-09+  License: MIT+++Terminology+===========++The key words "MUST", "SHOULD", and "MAY" in this document are to be interpreted+as described in RFC 2119. [#RFC2119]_+++Abstract+========++This proposal specifies a change to the behaviour of zcashd nodes intended to+mitigate denial-of-service from transaction flooding.+++Motivation+==========++Adoption of this proposal would increase robustness of Zcash nodes against+denial-of-service attack, in particular attacks that attempt to exhaust node+memory.++Bitcoin Core added size limitation for the mempool in version 0.12+[#BitcoinCore-PR6722]_, defaulting to 300 MB. This was after Zcash forked from+Bitcoin Core.+++Requirements+============++The memory usage of a node’s mempool should be bounded.++The eviction policy should as far as possible not be “gameable” by an adversary,+i.e. an adversary should not be able to cause legitimate transactions (that do not+themselves present any denial-of-service problem) to be preferentially evicted+relative to its own transactions.++Any configuration options should have reasonable defaults, i.e. without changing+relevant configuration, a node should be adequately protected from denial-of-service+via mempool memory exhaustion.+++Non-requirements+================++The current architecture of Zcash imposes fundamental limits on scaling of+transaction throughput. This proposal does not increase the aggregate transaction+capacity of the network. (The Blossom upgrade does increase transaction capacity,+by a factor of two [#zip-0208]_.)++Denial-of-service issues in the messaging layer of the peer-to-peer protocol are+out of scope for this proposal.++This proposal is focused primarily on memory exhaustion attacks. It does not+attempt to use fees to make denial-of-service economically prohibitive, since that+is unlikely to be feasible while maintaining low fees for legitimate users. It+does not preclude changes in fee policy.+++Specification+=============++This specification describes the intended behaviour of zcashd nodes. Other node+implementations MAY implement the same or similar behaviour, but this is not a+requirement of the network protocol. Thus, RFC 2119 conformance keywords below are+to be interpreted only as placing requirements on the zcashd implementation (and+potentially other implementations that have adopted this specification in full).++The mempool of a node holds a set of transactions. Each transaction has a cost,+which is an integer defined as:++  max(serialized transaction size in bytes, 4000) + fee_penalty++where ``fee_penalty`` is 16000 if the transaction pays a fee less than+10000 zatoshi, otherwise 0.++Each node also MUST hold a FIFO queue RecentlyEvicted of pairs (txid, time), where+the time indicates when the given txid was evicted. This SHOULD be empty on node+startup. The size of RecentlyEvicted MUST never exceed 10000 entries.++There MUST be a configuration option ``mempool.tx_cost_limit``, which SHOULD default+to 80000000.++There MUST be a configuration option ``mempool.eviction_memory_minutes``, which+SHOULD default to 60.++On receiving a transaction:++* If it is in RecentlyEvicted, the transaction MUST be dropped.+* Calculate its cost. If the total cost of transactions in the mempool including+  this one would exceed ``mempool.tx_cost_limit``, then the node MUST repeatedly+  call EvictTransaction (with the new transaction included as a candidate to evict)+  until the total cost does not exceed ``mempool.tx_cost_limit``.++EvictTransaction MUST do the following:++* Select a random transaction to evict, weighted by cost.+* Add the txid and the current time to RecentlyEvicted, dropping the oldest entry+  in RecentlyEvicted if necessary to keep it to at most 10000 entries.+* Remove it from the mempool.++Nodes SHOULD remove transactions from RecentlyEvicted that were evicted more than+``mempool.eviction_memory_minutes`` minutes ago. This MAY be done periodically,+and/or just before RecentlyEvicted is accessed when receiving a transaction.+++Rationale+=========++The accounting for transaction size should include some overhead per transaction,+to reflect the cost to the network of processing them (proof and signature+verification; networking overheads; size of in-memory data structures). The+implication of not including overhead is that a denial-of-service attacker would+be likely to use minimum-size transactions so that more of them would fit in a+block, increasing the unaccounted-for overhead. A possible counterargument would+be that the complexity of accounting for this overhead is unwarranted given that+the format of a transaction already imposes a minimum size. However, the proposed+cost function is almost as simple as using transaction size directly.++The threshold 4000 for the cost function is chosen so that the size in bytes of a+typical fully shielded Sapling transaction (with, say, 2 shielded outputs and up+to 5 shielded inputs) will fall below the threshold. This has the effect of+ensuring that such transactions are not evicted preferentially to typical+transparent transactions because of their size.++The proposed eviction policy differs significantly from that of Bitcoin Core+[#BitcoinCore-PR6722]_, which is primarily fee-based. This reflects differing+philosophies about the motivation for fees and the level of fee that legitimate+users can reasonably be expected to pay. The proposed cost function does involve+a penalty for transactions with a fee lower than the standard (0.0001 ZEC) value,+but since there is no further benefit to increasing the fee above the standard+value, it creates no pressure toward escalating fees. For transactions up to+4000 bytes, this penalty makes a transaction that pays less than the standard fee+value five times as likely to be chosen for eviction.++The default value of 80000000 for ``mempool.tx_cost_limit`` represents no more+than 40 blocks’ worth of transactions in the worst case, which is the default+expiration height after Blossom network upgrade [#zip-0208]_. It would serve no+purpose to make it larger.++The ``mempool.tx_cost_limit`` is a per-node configurable parameter in order to
The ``mempool.total_cost_limit`` is a per-node configurable parameter in order to
daira

comment created time in 5 months

more