@warpfork the thing's confused me about cache is that it's difficult to see it being involved in mutation. Looking at it now I think that confusion comes from the fact that there's a Put() inside modifyValue() but that's only used in the full-bucket-overflow-to-new-node case where a new node is created and it needs a CID for Link. In the other cases - delete or insert - it looks like it modifies the child and leaves it alone.

So maybe these docs should extend into modifyValue() to explain that, I'd have found that very useful when trying to grok what the cache is trying to do.

Minor nit: the rest of the docs are capped at 80 chars, these additions have a much more relaxed approach to length.

moved, but this needs to get up to sync with current multiformats, that's where it got stalled, perhaps you can help with that because I still haven't got my head around the new esm stack and am unsure how much I should pull that in to libraries like this

done

• [aaf33c3029] - build: add update-gyp script (Samuel Attard) #2167
• [3baa4e4172] - (SEMVER-MINOR) gyp: update gyp to 0.4.0 (Samuel Attard) #2165
• [f461d56c53] - (SEMVER-MINOR) build: support apple silicon (arm64 darwin) builds (Samuel Attard) #2165
• [ee6fa7d3bc] - docs: note that node-gyp@7 should solve Catalina CLT issues (Rod Vagg) #2156
• [4fc8ff179d] - doc: silence curl for macOS Catalina acid test (Chia Wei Ong) #2150
• [7857cb2eb1] - deps: increase "engines" to "node" : ">= 10.12.0" (DeeDeeG) #2153

https://github.com/nodejs/node-gyp/pull/2192

• [aaf33c3029] - build: add update-gyp script (Samuel Attard) #2167
• [3baa4e4172] - (SEMVER-MINOR) gyp: update gyp to 0.4.0 (Samuel Attard) #2165
• [f461d56c53] - (SEMVER-MINOR) build: support apple silicon (arm64 darwin) builds (Samuel Attard) #2165
• [ee6fa7d3bc] - docs: note that node-gyp@7 should solve Catalina CLT issues (Rod Vagg) #2156
• [4fc8ff179d] - doc: silence curl for macOS Catalina acid test (Chia Wei Ong) #2150
• [7857cb2eb1] - deps: increase "engines" to "node" : ">= 10.12.0" (DeeDeeG) #2153

Fixed CI bug with Go 1.11 and squashed this into just the two commits on the HEAD of this branch. Review should focus on those two, 04503fda1 particularly (the other just adds a coverage target to Makefile).

done in #60

From:

type Pointer union {
	&Node "0"
	Bucket "1"
} representation keyed

i.e. {"0": CID} or {"1": [KV...]}

To:

type Pointer union {
	&Node link
	Bucket list
} representation kinded

i.e. CID or [KV...]

Also removes redundant refmt tags.

Diff is hard to see because this bundles unmerged PRs #56 and #59. Only the last commit contains the main change here, currently that's cb07286.

Because TestMalformedHamt has some manually generated CBOR you can see the change in the block layout there and the simplification this introduces.

Closes: #53

Possibly the most controversial item in here that I want to highlight. We're validating that all links to child HAMT nodes (not links it might contain as values, only HAMT nodes) are of codec DAG-CBOR. Just another check to get some assurance we're not being fed bogus data in the structure, it doesn't go very far but we're asserting that this HAMT has homogeneity in its block generation, for now. This could be changed in future if we allowed some variability in codec or wanted to have a migration path.

I guess discussion here should focus on how close this comes to our Tier 2 definition:

• Tier 2: These platforms represent smaller segments of the Node.js user base. The Node.js Build Working Group maintains infrastructure for full test coverage. Test failures on tier 2 platforms will block releases. Infrastructure issues may delay the release of binaries for these platforms.

Given that there's an opt-in checkbox in a Jenkins subjob that gets compilation happening does it get close enough? Do we need to extend that a bit more to make a daily test job against master to get closer? I don't imagine too many people being aware of the tickbox or thinking to regularly run these jobs outside of one or two people.

It strictly meets "The Node.js Build Working Group maintains infrastructure for full test coverage." but "Test failures on tier 2 platforms will block releases" suggests that it's tied into our CI in such a way that failures would be noticed by someone preparing for releases.

That's one nasty hack in the version selector to get the checkbox to work! Probably better than building too much more of the logic into Jenkins but it might not be a good pattern to encourage unless we can find a way to make it a bit cleaner.

Is the intention to leave the checkbox only at the fanned-arm64 job and its subjobs or does it get migrated all the way up the job call stack?

And what's the path to a more permanent CI presence? Do we have any word on Windows ARM64 hosting options, do we even imagine that possibility on the horizon or are we going to be in a similar position to our Pi cluster, with Windows laptops in offices?

I've added 2 more arguments to the function so they get passed in explicitly, it gets a bit verbose but it's not terrible as it is now. We're probably going to want bucketSize at some point too so this might start to get out of hand soon and pulling up to a config object might be a good refactor.

👍 thanks, fixed

Ready for review now. This includes 2 commits from #57, the cleanChild() changes. It adds a bunch of code to LoadNode() and pulls out the loading functionality into a new loadNode() so we can toggle on root/non-root for different validation. There's a minor change that gets into the new cleanChild() because we no longer need to do one bit of validation there now. There's a new method in uhamt.go for counting the total number in the bitfield. There's a big-ol' test case named TestMalformedHamt which pushes this around with different incoming CBOR blocks. Some utilities in hamt_util_test.go plus some context-specific ones inside the test let us manually build CBOR that forces validation errors in various ways. There's also sanity-checking along the way to make sure we're not just hitting "this is all just bad CBOR" type errors.

The HAMT has (almost) zero chill now about malformed blocks. If a block smells bad then it's rejected, no pretending. One item that could be checked but is not because it's so complicated and costly (for now) is to make sure that entries are in their right positions in the graph given their keys. If that's important it could be added later and called on demand. For now this is block-local only.

5c1cfa9 with minor mods but no release cause tests aren't running .. yay

I'm attempt to create a node addon that uses the a json cpp library. I'm attempted to use libraries(referenced on #328 ) as include dir, event the including directly but the build fails on locating the package.

Here's my gyp file.

{
    "targets": [
        {
            "target_name": "bindings",
            "sources": ["analyzer.cc"]
        },
    ],
    "include_dirs": [
    	"lib/nlohmann"
    ],
    "libraries": [
    	"-lnlohmann", "-L/home/lukaswilkeer/Documents/dev/nodejs/stock-parser/lib/nlohmann"
    ]
}

What to do and how to do?

Try doing a -l<absolite path to library file>, this is not an ideal arrangement but sometimes you don't have much choice to link to an absolute location. An ideal would be to have multiple targets and link them together. See https://github.com/Level/leveldown/blob/master/binding.gyp for how this might be done if you have control over the library you're linking against - that one links against a "dependency" which has its own .gyp file, and that dependency even links against another dependency which also has its own .gyp file. So if you're bundling some C++ and publishing as a Node addon and you expect users to compile both the library and your addon wrapper then try and get it into a state where it can all be compiled with GYP and linked together by GYP rather than this manual linking arrangement.

Checklist

• [x] npm install && npm test passes
• [x] commit message follows commit guidelines

Description of change

Extends compile_commands_json generator to consider more file extensions than just c and cc. This commit adds the cpp and cxx extensions to the list of input source files.

sorry to do this but this needs to move over to https://github.com/nodejs/gyp-next which we vendor in here, we no longer touch the code in gyp/ except for pulling in new versions from over there.

Includes #57, the first two commits. The changes so far are only in LoadNode and in the tests.

Adds in some simple helper functions into the tests to help manually build CBOR blobs that'll load and trigger various cases. Only got as far as ensuring that the bitmap and the number of elements aren't out of alignment. That can get stricter with #54 if we put in precisely the size of bitmap that we need for the array of elements.

Other things I'm working on in here are listed at the bottom of hamt_test.go. The aim is that this thing should refuse to load from blocks that smell funny (i.e. aren't exactly the right form). There should only be one way of representing this data and there should be no avenues for variation. That's an ideal and not strictly possible in the extreme sense for various reasons but we can get close.

rebased to master and autosquashed

going to leave the byte/int thing for a separate PR

@nodejs/addon-api hey, the docs for node-addon-api have the @ but we don't use it for the NAN docs. Perhaps this should change? https://github.com/nodejs/node-addon-api/blob/master/doc/setup.md, I don't think it's necessary if it's just one item being returned, not an array, and in this case it seems to be to blame for doing some extra escaping on Windows, which I don't get, but maybe @nodejs/gyp understands.

Changed the title of this after playing a bit with the format. Dropping some thoughts here as I explore this.

The bitfield at the moment uses a big.Int which can spit out a type of big-endian format. (some of the guts are in https://golang.org/src/math/big/nat.go with the API in https://golang.org/src/math/big/int.go). We're using the Bytes() and SetBytes() methods for serialization and deserialization. What this seems to give us is the most compact big-endian representation of the number it holds. We're using SetBit() to set individual bits on and off to indicate the presence or absence of an element in our data array, so the number is arbitrary, it's the bits that matter.

The maximum length of the bitfield should be 2^bitWidth to hold enough bits to cover all the indexing we need for any node. So a bitWidth of 8 gives us 256 bits needed to store our bitmap data. So if we were to turn on all the bits because we have a full data array, we'd end up serializing 0xff... for 32-bytes, i.e. 256 1's. But if we only tinker with the first 8 bits, then we only need to serialize one byte. e.g. if we only had an element at index 0 then our bitfield would be a single byte, 0x01, but if we only set the 8th bit then we need to bytes so would serialize 0x0100, and so on.

Filecoin only uses a bitWidth of 5, so that's 32-bits, or 4-bytes needed to represent the bitfield.

Some thoughts about this format:

• It's somewhat Go specific. It's convenient to get these from a big.Int and set them to a big.Int but it's going to be slightly annoying for everyone else unless they have something already that works in exactly the same way. The ideal internal representation is for a node to have a bitfield of exactly 2^bitWidth ready and available to set and unset bits on. The convenience of big.Int bypasses this entirely but that's not going to be the same story across languages. I have https://github.com/rvagg/iamap/blob/master/bit-utils.js for this in JS, but to go to and from this serialization format I'd have to be trimming left-most bytes that contain zeros on the way in and padding them back on the way out.
• The randomness of the hash algorithm means that the chances of setting the 32nd bit are the same as setting the 1st. There's no bias toward smaller bits built-in here. So we buy some compaction in the serialization by using this truncated format, but it's only going to get us so far in a HAMT that's got more than a few values in it.
• In Filecoin the bitWidth is 5, so 32 bits, or a maximum of 4 bytes in the serialization format of this field. There will be some cases with only 3, less with 2 and even less with just 1. We're saving bytes, but not many, and at the cost of complexity for everyone but Go implementers.
• It's hard to validate canonical block form. The current implementation will (probably) take an arbitrarily long byte array and turn it into a valid big.Int. It'll treat as valid a block that has a byte array 1000 long in the position for the bitfield (I believe big.Int can handle this kind of arbitrary size). But then it should round-trip it back out as just-long-enough if the block was re-serialized. (So this is in a similar category to the problems suggested in https://github.com/filecoin-project/specs/issues/1045).

I don't have a strong opinion here yet, would like to hear others' thoughts. My personal preference would be for it to be stable and consistent, with the bitfield byte array in CBOR being exaclty 2^bitWidth (exactly 4 bytes, every time, for Filecoin) so serialization, validation and explanation of this spec is simpler than it currently is. I doubt that the number of bytes being saved here is very meaningful—but it's not zero.

@warpfork @Stebalien @anorth thoughts?

:thumbsup: good idea, have added that and am also adding it for the next set of tests I'm writing

re arrayWidth, it could be configurable but isn't yet, I quite like 3 as a value but it's one of those parameters that can be tuned for different results depending on what we're optimising for.

fwiw #52 has a ton more documentation on some of this stuff

But the byte thing is a good point! it also precludes being able to add support for bitWidths greater than 8 (which may have questionable benefit but I added up to 16 in mine so I could push some boundaries in the 8-10 range). I don't know why it's a byte or why that effort is involved here, we could probably back that out without any drama.

Odd, I guess the \ is being treated as escaping. What happens if you remove the @?

I plan to use these two to help with load-time validation so their utility is beyond the clean algorithm, btw

Closes: https://github.com/ipfs/go-hamt-ipld/issues/56

I've tried to push cleanChild() around in the various ways I was suspecting it might be flawed, but couldn't make it misbehave, so I'm going to call it good. Added more testing for CHAMP compaction along with a printHamt() that can show you the kinds of shapes you have.

So in this test we do things like:

‣ bafy2bzaceaabidljpjqxkb3bwz2kymapl6w6zazcaehtwh55d3solznpudfji:                                                                                     
  ⇶ [ AAAAAA11, AAAAAA12, AAAAAA21 ]                                                                                                                  

overflows when we add another colliding element to:

‣ bafy2bzacec2e7uxlmpn5bzfjfhrjbvgkgyfbi76k43epdijp626i3baaao436:                                                                                     
  ‣ bafy2bzacebgcvdzkq4glgp2kl65oyz65w2qb6ptbzjwxgbvnw2hwzs25bgeaw:                                                                                   
    ‣ bafy2bzacect7ahd4hdy24w7jfrttzgx5mh264lzobxdq5dyisqeelol6dia5o:                                                                                 
      ‣ bafy2bzacebxdmdcvhpnpsiia632eiov45yynuijl3faw7zzfzbupg4tombhnu:                                                                               
        ‣ bafy2bzacecrptawydwevr4rt3w4mbvuvh6wteh2w3kplwohkzu64zip5lslc2:                                                                             
          ‣ bafy2bzacedxu2dpomrsnqgj45pbkgyxrc4nujwfgjrwlpm3nkjvfomhkmqang:                                                                           
            ‣ bafy2bzaceadtlk6fe4pishrnyakhlz32iiakebraxudz34r4o5um3sgof3azs:                                                                         
              ⇶ [ AAAAAA11, AAAAAA12 ]                                                                                                                
              ⇶ [ AAAAAA21, AAAAAA22 ]                                                                                                                

and then back again.

It also tests a mid-way state where collisions happen at half that tree height with a collection of elements that collide in another way:

‣ bafy2bzaceacaykqvidugqydd2lskm7ajgdionqzgh22odxcmtfo6jkj6legqe:
  ‣ bafy2bzacear4ozwsojhwrjc3yroecudxdchelx74k2h6xlgzwtjsrpgo7sz3o:
    ‣ bafy2bzacec4owi2jruh53fve4vn5tyzg4yw7lfmrsuvshdrlsk4rdaa3nnbga:
      ‣ bafy2bzacedvt2qjfegib2k7bzfj4oxe4nxy46amtfwmybhhqjv4vs56xlv6gq:
        ‣ bafy2bzacecknjra3gxqdsaydwjwhgkh3pvun4kphn3xgpm6ek36mg3kzf3zrc:
          ⇶ [ AAA11AA ]
          ⇶ [ AAA12AA ]
          ⇶ [ AAA13AA ]
          ⇶ [ AAA14AA ]
        ‣ bafy2bzacecrptawydwevr4rt3w4mbvuvh6wteh2w3kplwohkzu64zip5lslc2:
          ‣ bafy2bzacedxu2dpomrsnqgj45pbkgyxrc4nujwfgjrwlpm3nkjvfomhkmqang:
            ‣ bafy2bzaceadtlk6fe4pishrnyakhlz32iiakebraxudz34r4o5um3sgof3azs:
              ⇶ [ AAAAAA11, AAAAAA12 ]
              ⇶ [ AAAAAA21, AAAAAA22 ]

and rolls that back out to see if it behaves properly. And it does.

I've refactored cleanChild() to a state that I think has more clarity about the various states it can be in and reasons it has to take action or not. I think this more closely matches the CHAMP semantics if you want to read the paper and see those semantics in this code. Maybe that's subjective, but I think I've reduced the number of branches here which should in itself clear up some confusion.

Feel free to keep on pushing your changes to this branch, at least we can have a discussion around the same code in that case.

One of the tricky things about custom streams is that you have to always be aware of calling the callback on transform() and flush() on any path through your code. Failure to do so will cause weird errors that involve back-pressure that has a delayed effect so it can be hard to track down. So the jsonStream and logTransformStream are worth studying to find codepaths that may not properly trigger a callback(), possibly even on the case of an uncaught exception which will just halt that execution path.

You do have one obvious instance of a failure to call callback though, at the bottom in the failure case, maybe it should be callback(err) after the console.log("PIPELINE HAS FAILED", err), just for completeness. But presumably you'd see that somewhere in the output (not necessarily right at the end though).

Is this change because upstream only supports Buffers and they also support arrays of Buffers?

Probably needs a const { Buffer } = require('buffer') too, right?

🤦 what a mess, so we're going to need to get a user's browser involved now for this to work. Yeah I'd look at a PR that dealt with that, there's a lot that depends on this library working and even the brownouts they're talking about are going to be annoying. This must be why the new gh tool from GitHub has a terrible auth experience. Thanks for raising this, I wasn't aware of it.

Hey, while we're doing this, if we end up with the longer description of how to get to CommP, how about we take the opportunity to insert a definition of "Fr32" into the specs because it's going to keep getting asked. @ribasushi was the latest one to ask just 2 days ago. Combining and rephrasing parts of @porcuquine's responses we could come up with something like:

The term Fr32 is derived from the name of a struct Filecoin uses to represent the elements of the arithmetic field of a pairing-friendly curve, specifically Bls12-381—which justifies use of 32 bytes. F stands for "Field", while r is simply a mathematic letter-as-variable substitution used to denote the modulus of this particular field.

👍 thanks for catching that @vtjnash, we have a $JOBS on each of our CI machines and normally use that whenever we use -j or JOBS= but that's been left off in this instance. I've inserted that back and you should see a -j 2 now on these runs I think.

https://ci.nodejs.org/job/libuv-test-commit-osx-cmake/177/ is a repeat of the last green run, https://ci.nodejs.org/job/libuv-test-commit-osx-cmake/174/, to confirm this is working properly. Please close this issue if you agree it's fixed now.

👍 good catch

I don't buy the arguments you have for retaining constructors, making this thing feel like an ArrayBuffer would probably mean we've failed to make it a nice abstraction. Just because it might be built as an abstraction over ArrayBuffer doesn't mean we need to expose that to users, there's not a good reason to leak that upward. So if you put that aside, you run out of reasons to have constructors and static factory functions. So I'd just ditch constructors entirely, or make a single-use constructor of some kind that's justifiable. I'd be more than happy with static factory methods that were explicit about what they do.

I'm a big +1 wherever we can reduce argument overloading. It's one of the biggest curses that jQuery gave JavaScript culture that we still haven't shaken and it keeps on leading to APIs that cause far more harm than the convenience they supposedly offer.

Would this work?

declare class CID {
   static from(string|ArrayBuffer|ArrayBufferView|CID):CID
   static from(buffer:ArrayBuffer, byteOffset:number=0, byteLength:number=buffer.byteLength):CID
   static create(version:number, code:number, hash:Uint8Array):CID
}

It's a big jump from js-cid, a simple one for js-multiformats in its early stage though.

CID.from(cid) could presumably serve as the formerly proposed CID.asCID(cid) wouldn't it? Just a return cid where it's deemed that the passed argument is what it needs to be, otherwise instantiate a new instance using what can be derived from it. Or would we need a form that lets us perform a full clone?

Anything above the single byte range is wide open, I see two slots that might be nice (this is mostly a subjective call given the space available!). After x11, we could probably start a block in 0x12XX for common curve pubkeys. There's also space we could work on in 0xb5XX, the preceding poseidon entries are somewhat related (in that they involve a curve).

Pulling this out of a TODO attached to cleanChild() in #52 because I think it might be the most important thing to address with the current impl:

// TODO(rvagg): I don't think the logic here is correct. A compaction should
// occur strictly when: there are no links to child nodes remaining (assuming
// we've cleaned them first and they haven't caused a cascading collapse to
// here) and the number of direct elements (actual k/v pairs) in this node is
// equal o bucketSize+1. Anything less than bucketSize+1 is invalid for a node
// other than the root node (which probably won't have cleanChild() called on
// it). e.g.
// https://github.com/rvagg/iamap/blob/fad95295b013c8b4f0faac6dd5d9be175f6e606c/iamap.js#L333
// If we perform this depth-first, then it's possible to see the collapse
// cascade upward such that we end up with some parent node with a bucket with
// only bucketSize elements. The canonical form of the HAMT requires that
// any node that could be collapsed into a parent bucket is collapsed and.

This is more than just a perf improvement and needs to be addressed - either to prove that my concerns are wrong, or get the algorithm correct so it's not pushing beyond CHAMP invariants.

But I think that it's differentiating from structures like this that's the problem with the simple compaction:

     root
    /    \
a: v   b: o 
         /
      e: o
         \
       f: vvv

The whole index needs to contribute to locating the item, otherwise you can't be sure whether you addressed the right one.

     root 
    /    \
a: v   d: vvv (height +2)

An index 011000000111 where that initial root->d selection is the first part 011 and the d->v selection is the last part 111 would also be matched by a look-up for 011111111111, so you need a way to consider the missing middle bit, or at least check whether the end you've reached is the one you want. When it's fully spanning you can assert that because you're never discarding pieces of data. With a HAMT you are discarding pieces of the hash because you can reach the end before exhausting the hash, which is why we need to store the key with the value and perform the key<>key check. We'd need a mechanism to do the same here.

Here's an algorithm which might work:

Any height>0 node that only addresses a single terminal element within its sub-tree gets that element inlined in its parent along with the index of that element.

So you'd end up with intermediate nodes that had both links to sub-trees as well as inlined values, (conceptually) like: [CID, CID, [1820422,<value>],CID]. Where the only indexes we need to store are ones at height>0. And you'd get to deal with perverse cases like a single large indexed element only needing a single node for storage (rather than all the blocks needed to form full height).

So as you navigate down through the structure and you hit one of these things that's an array rather than a CID (so these become a kinded union of link and array), you just check the index against the one you're traversing to and that tells you whether this is the entry you care about or if that entry doesn't exist. The canonical form of the data structure would require compaction whenever a deletion created a situation where there was a leaf with only one element (accounting for that makes this a little awkward because you'd need to perform some traversal to discover the full count, but because all sub-trees are like this the traversal would be minimal). Then when you perform insertions and hit one of these compacted nodes you'd need to push it maximally downward until it's on its own or is at height=0.

So there's some funky testing required to ensure that such an algorithm could remain stable and result in canonical form for any given set of data regardless of how you got there.

squashed and rebased to master

:thumbsup: thanks

Collapsing is a bit hard when we don't store the index with the elements. We can't know without being able to slice an index into all of its components (3 bits per level for this width=8 impl) which particular entry we're at. When the height gets collapsed, we lose track of which pieces of an index we need to discard.

If we stored index along with the value ([index,value] at each leaf) then we could do a check when we reach a point where the full-depth is short-circuited to see if the index we care about is the index found or not (not the same? then the one you want doesn't exist). This would add an int+array for each terminal element being stored in the AMT, which might not be too much of a cost to pay if it means ditching a bunch of intermediates.

Another option is to inline-compact, maybe a rule where if branch only has a single element (or some other number), then don't link, inline it. So you'd end up with nodes that look like (conceptually): [bmap,[CID, CID, [bmap, [entry]], CID]]. So you retain the nested array structure giving you the depth information that you can use to determine the final index. The algorithm would remain stable, you just skip the traversal and end up paying the costs of inlining the full structure but avoid the cost of a CID (or many, because you could nest a lot of these, [[[bmap,[entry]]]]).

Costs everywhere so it'd depend on the nature of the data being stored whether these are worth it.

The width provides the primary tuneable parameter, allowing for a selection along the block-size, mutation cost and traversal cost spectrums. A fixed width of 8 provides for a maximum arity of 8 which. It's also something that probably should be benchmarked to understand the true costs involved but we can't do that without the ability to change it. Some Bmap code in here suggests that variability was a TODO, but a single byte is the easy case and that's taken advantage of here, hence the 8.

Added doc.go with algorithm summary. More meat in here than the HAMT since we don't yet have a proper spec doc.

I thought about trying to pull in some ASCII art used in https://github.com/ipld/specs/blob/master/data-structures/vector.md but for sparse indexing but ran out of steam. Maybe that can be saved for a spec doc. Those are useful visualisations for what's going on here, you just have to imaging missing bits in the middle of the tree structures.

Ready for review.

@anorth @momack2 @mikeal @Stebalien

done

Did you try removing the e from the jsonStream error callback? Perhaps we're getting a line somewhere that we can't parse properly for some reason but should be able to ignore - might be worth trying. It could be that an error in either jsonStream or logTransformStream is closing the downstream streams and split2 can't push anymore in.

util.pipeline() is really worth trying here, not just for the error handling but also for the nice cleanup it does and managing these complicated events.

can we avoid this as the top-level container of the TOC?

s/an/a

@Gozala would appreciate some focused eyes on my flow and TS changes, I think you might have more of a clue than me on both of those

(this PR assumes #117, review should focus only on the HEAD commit please, c043bc5)

Ref: https://github.com/multiformats/js-cid/pull/117#issuecomment-668131658

The idea here is that js-multiformats is going to require a switch to using the multicodec integer code rather than string, which does away with the need to bundle the entire multicodec table (and if you want string and you're happy to have the table then that's fine but you get to choose that). Experience shows that this is the most painful compatibility problem when switching between this library and multiformats.CID (and vice versa). Having a constructor that can take (optionally) an integer code rather than a string makes it the same constructor as multiformats.CID and exposing the code property makes it the same interface. So we can write new code that works against CID and multiformats.CID without having to worry too much. The asCID() will be icing on the cake too.

As long as we're getting an awkward breaking change into the code, we may as well make sure that anyone who has that code also has this and there's not an in-between state where there's breaking code and then there's this feature out there and you may have one and not the other. Basically it'd be nice to have this out in the wild ASAP to ease future transition and the current breaking change release seems like a good mechanism to force that.

This is fine by me, but we're now out of space in 0xeX, I'm wondering if we should just start populating a new space for curve public keys, since there's potentially a lot of them, leaving the current ones in place. There's 4 more prime curves in the NIST set that would go along with just this one and there's no space to nicely order them if we lock this one in.

Thoughts from anyone else?

@OR13 just out of interest, do you have a current use-case for a P-384 public key?

I think you're a better judge of the audience here so should decide what level of detail to include. "in order for the file to make a binary Merkle tree" is only correct for one stage of padding, the fr32 is something else all-together. I'm not sure for this audience how much it matters that "your input padded internally and at the length such that it doesn't quite resemble your input before it is merklized". For now maybe it's fine as is, implementers and anyone else wanting to generate CommP separate from a full client (like we've had to do for these offline deals for Filecoin Discover data) will have to come to a more sophisticated understanding of the process, and this might not be the place for them to get that detail! Your call @yiannisbot, I think you have most of the details now.

links are "0", buckets are "1"

don't we have special rules for optional fields in tuple representation? like, only the last one can be optional for it to make much sense?

Yes, and in doing that there's an option to do away with expLinks and expValues entirely and just keep more careful track of index translation through Bmap. They do add a lot of convenience for some operations but it wouldn't be hard to add in convenience functions to help with indexing. I don't think they add much in the way of performance, they could even hinder performance because they're used so liberally (e.g. a get() should really just map index through Bmap and fetch directly out of Values or Links, no need to expand just to read). And you have these extra structures hanging around in memory that aren't necessary so they'll add a little to memory bloat.

oh, and its partner came in a separate email:

Your Developer ID Application Certificate will no longer be valid in 30 days. To generate a new certificate, sign in and visit Certificates, Identifiers & Profiles https://developer.apple.com/account/.

Certificate: Developer ID Application Team ID: HX7739G8FX

We got this email today:

Your Developer ID Installer Certificate will no longer be valid in 30 days. To generate a new certificate, sign in and visit Certificates, Identifiers & Profiles https://developer.apple.com/account/.

Certificate: Developer ID Installer Team ID: HX7739G8FX

We need to check if that cert is still in use or if we've moved to a newer one and this is just an unused one expiring. I think I might have made new ones when I did the notarization work and I'd be surprised if they're expiring so soon. I'm hoping this is a noop for us but it needs checking.

CUD -> CID