profile
viewpoint
Mikeal Rogers mikeal Protocol Labs San Francisco http://mikealrogers.com Creator of NodeConf and request.

apache/nano 1132

Nano is now part of Apache CouchDB. Repo moved to https://GitHub.com/apache/couchdb-nano

indutny/caine 143

Friendly butler

ipfs/blog 90

Source for the IPFS Blog

dscape/spell 88

spell is a javascript dictionary module for node.js, and the browser (including amd)

compretend/compretend-img 24

Image element that understands what is in the image, powered by ML.

jhs/couchdb 15

Mirror of Apache CouchDB

ipld/book 10

Decentralized Data Structures Book

isaacs/http-duplex-client 10

Duplex API for making an HTTP request (write the req, read the response)

ipfs/metrics 9

Regularly collect and publish metrics about the IPFS ecosystem

ipld/roadmap 8

IPLD Project Roadmap

issue commentfilecoin-project/specs

Retrieval by any CID: broadcast & usage

A decentralized storage app uses Filecoin for backup, making storage deals in units of 32GB. When one of their users needs to recover 5 minutes' worth of data (let's say 15MB), they want to serve it back relatively quickly, and don't need (or want to use bandwidth for) any other data in that sector.

This is why retrievals have selectors, so you can get less than the data in the sector. How much data depends on how precise the selector is, which depends on how precise the data structure you built and store in the CAR file is.

There’s no reason this use case must be done with arbitrary CID retrieval.

Backups are stored as diffs. It’s not hard to imagine different structures you could build the diff as that would allow for different types of queries.

mishmosh

comment created time in 11 hours

issue commentfilecoin-project/specs

Retrieval by any CID: broadcast & usage

One thing I just realized is that “indexing every CID” and “query by arbitrary CID in the CAR file” are being conflated a bit. Indexing every CID is a good idea if you want to improve all retrievals. Querying by arbitrary CID in your data doesn’t actually require an index and there’s an alternative we haven’t considered yet.

If we improved selectors to understand our HAMT spec, we could punt this to the storage side.

The way it would work is, if you want a CID index of your data you’d need to store a CID map in the CAR file. You’d prepare your graph, then create a HAMT to map every CID to every block in your data, then you’d create a root block for you CAR file that points at the HAMT. Then you can create selectors against the HAMT for any CID in your data, the only CID’s you wouldn’t be able to query would be the ones for the HAMT itself and I don’t see what use case that would be required for.

mishmosh

comment created time in 11 hours

issue commentfilecoin-project/specs

Retrieval by any CID: broadcast & usage

I’d like to understand where some of these requirements are coming from because, while I would love to have the flexibility to query for any CID, even I’m having a hard time justifying the tradeoffs that would have to be made.

The ability to do partial retrieve by deep CID is a deciding factor in choosing storage solutions. When proposing storage deals, clients want to know whether whatever they're getting themselves into will support retrieving data by deep CID.

What is this use case for this?

You still have to know the commp and/or root CID of the CAR file in order to do the retrieval. Which means that, no matter what, you’ve got a mapping somewhere of “this data is in this CAR file.” What are the use cases where you would know that but you wouldn’t be capable of creating a CAR file that is either structured in such a way that you can query it without specific CID’s or you could just store the traversal necessary to reach any CID wherever you’re storing the data about the CAR file?

In the data we’re preparing we have hundreds of millions of files and file slices in a database, each of which contains info about the CAR file the data is in and a selector to the root of the file within that CAR file. We write the CAR files with a root node that is just an array of CID’s for the roots of all the individual files, so the traversal we store is just rootCID/offset.

  1. Punt this to the retrieval market, make it required

We should not downplay how expensive this may be. Optimistically, you can max out the block size and you’re only storing large files, so you’ve got 1000 CIDs per 1GB of raw data plus a few blocks of unixfsv1 data and a root block for the CAR file, but let’s round that down to 1 CID per MB.

So, storing a PB of data will cost you 1 Billion CID indexes. That’s substantial.

That’s the most optimistic view possible. In practice, we’re seeing data sets with many many small files, so the block size is much smaller and the unixfs overhead is more visible, so 1PB of data is closer to 50 Billion CID’s.

Unless we think looking up arbitrary CID’s is a primary use case I don’t see how this overhead is justified.

  1. On-chain, as part of StorageDealProposal

Do you mean there is a boolean property in the deal for CID indexing or do you mean that the CID’s are on chain? Based on the data we have “on deck,” that’s probably 100 Billion CID’s on chain shortly after launch, so I’m going to assume this isn’t what you meant.

If they get rid of the index then they would suffer a much more expensive scan when they end up accepting a query later for an arbitrary CID.

Again, I’m not sure this needs to be a requirement at all, but if it were this makes the most sense to me. If the requirement is in the ask then the node would have to respond to this type of retrieval and if they decide not to keep around the index they are penalized plenty by having to do the scan.

mishmosh

comment created time in 12 hours

PR opened ipld/roadmap

OKR: Initial OKR’s for Q2

Apologies, I accidentally merged the first draft of this, but please modify this branch and we won’t consider these finalized until this PR gets merged.

/cc @vmx, @rvagg, @creationix, @warpfork

+0 -0

0 comment

0 changed file

pr created time in 13 hours

create barnchipld/roadmap

branch : mikeal-2020-q2-okr

created branch time in 13 hours

push eventipld/roadmap

Mikeal Rogers

commit sha 9ab51fa54998f5a6bc1d9550efe676d5ed22189c

okr: initial okr’s for Q2

view details

push time in 13 hours

PR opened ipld/roadmap

OKR: 2020 Q1 Scoring

I need to get scores in from @vmx, @rvagg @warpfork and @creationix

+1 -1

0 comment

1 changed file

pr created time in 13 hours

create barnchipld/roadmap

branch : mikeal-scoring-2020-q1

created branch time in 13 hours

push eventipld/roadmap

Mikeal Rogers

commit sha c0bc02ce65bcb6e6b1dec5389a34c2d38b049291

okr: initial Q4 scoring (#20) * okr: initial Q4 scoring * Add vmx final score to 2019/q4 Co-authored-by: Volker Mische <volker.mische@gmail.com>

view details

push time in 13 hours

PR merged ipld/roadmap

okr: initial Q4 scoring

We need to input our scores for Q4 2019.

Since 3/4 of us were pulled over to Filecoin work mid-quarter I expect most of the final scores to be pretty close to the mid-quarter scores.

+5 -5

0 comment

1 changed file

mikeal

pr closed time in 13 hours

pull request commentipld/specs

[WIP] Flexible Byte Layout

I'd like to declare exactly how/where we should return an error in the spec. If a the "part" part of a NestedByte doesn't match the given length when we resolve the part, we should return an error.

Agree, but “when we resolve the part” is a bit too ambiguous. We “resolve” the entire graph when we replicate it but because replicators work on generic IPLD graphs they won’t be able to produce errors during replication because they won’t understand this data structure or when to apply this schema.

I think “when we read the part” is less ambiguous and doable, but as @ribasushi has mentioned before, this still leaves you open to an odd situation when reading a slice of the data that occurs after a miss-alignment of the size will not cause an error and will return the data slice based on the encoded size information rather than the raw part length. This is probably fine but if we’re going to document it as part of the spec it should be this detailed, because we’re effectively saying that “when reading a slice of the data, the encoded size information is relied upon and miss-alignments will only cause errors if the miss-aligned part is being read.”

mikeal

comment created time in 17 hours

push eventipld/js-schema-validation

Mikeal Rogers

commit sha ccce4ebaa8468192e73080600fd0bc25ee53a44c

build: update pkg metadata

view details

push time in 18 hours

push eventmikeal/ipld-schema-validation

Mikeal Rogers

commit sha 09e4af6b4a7a24077a3196d1ae8cfe6e20060a1d

fix: use explicit buffer module

view details

push time in 18 hours

push eventipld/js-block

Mikeal Rogers

commit sha 93424bdd961dc1124fa118e457f07fdb3b10048e

fix: don't rely on bundler polyfill (#12)

view details

push time in 18 hours

PR merged ipld/js-block

fix: don't rely on bundler polyfill

This matches https://github.com/ipld/js-ipld-block/pull/50

+2 -0

0 comment

2 changed files

mikeal

pr closed time in 18 hours

push eventmikeal/ipld-schema-validation

Mikeal Rogers

commit sha d8cb8d17e60b8ff63ad252358806f6349b1936f1

build: browser tests

view details

push time in 19 hours

PR opened ipld/js-block

fix: don't rely on bundler polyfill

This matches https://github.com/ipld/js-ipld-block/pull/50

+2 -0

0 comment

2 changed files

pr created time in 19 hours

create barnchipld/js-block

branch : buffer

created branch time in 19 hours

pull request commentipld/js-ipld-block

fix: add buffer

sigh...

I’d love to get off of Buffer but it’s proving to be much more difficult than I had anticipated. The vast majority of the polyfill is never used but actually removing an interface that is passed around so much has proven impractical.

I’ll merge this PR now and work on getting rid of Buffer later.

hugomrdias

comment created time in 19 hours

pull request commentipld/js-ipld-block

fix: add buffer

What does this fix?

Isn’t the buffer module already pulled in to polyfill the Buffer API by bundlers? This potentially introduces a condition in which there is more than one Buffer module in a bundle due to version discrepancies.

hugomrdias

comment created time in 19 hours

Pull request review commentipld/js-datastore-car

feat: create car file from a complete graph

 async function writeStream (stream) {   return new CarDatastore(reader, writer) } +async function traverseBlock (block, get, car, concurrency = 1, seen = new Set()) {+  const cid = await block.cid()+  await car.put(cid, block.encodeUnsafe())+  seen.add(cid.toString('base58btc'))+  if (cid.codec === 'raw') return+  const reader = block.reader()+  const missing = link => !seen.has(link.toString('base58btc'))+  const links = Array.from(reader.links()).filter(missing).map(([, link]) => link)+  while (links.length) {+    const chunk = links.splice(0, concurrency)+    const blocks = chunk.map(get)

ya, i tried to think of a variable name that was more descriptive here but couldn’t come up with much other than readBatchSize and that seemed pretty ugly.

mikeal

comment created time in a day

push eventipld/specs

rvagg

commit sha d480cee07b1ccc94b1fc7025fe2ce77a851cc557

Automated deployment: Tue Mar 31 02:14:35 UTC 2020 51eb2f682f554b726021f34398843f8edf97bbd0

view details

push time in 2 days

push eventipld/specs

rvagg

commit sha 55ae555ca457dd0da1c945d1f4a646ddfba2f3d9

Automated deployment: Tue Mar 31 02:11:39 UTC 2020 4f82fc0c12b9b0d03bf35ac08ee83a3948b801f0

view details

push time in 2 days

pull request commentmikeal/bent

Add headers attribute in StatusError object

Probably we could make public getBuffer or a wrapper function for each encoding and document an example of this approach.

This would introduce a sizable divergence from the Browser.

How about we just match the browser’s fetch methods for this, so in the returned response object we add methods that return promises like .json()?

Chillaso

comment created time in 3 days

issue closedmikeal/bent

Error Response Contains Buffer, Not Content

When making a call to a REST API, it is possible for the response to contain an error status code AND contain a valid body with information as to why there was an error.

  return await get(url).catch(async e => {
    let msg = await e.responseBody

The resulting body contains a buffer, but I'm assuming it hasn't been decompressed yet?

closed time in 3 days

icpenguins

issue closedmikeal/bent

Error: Cannot find module './core'

I'm creating a Node socket server. If I require bent I get a compile error Error: Cannot find module './core':

const express = require('express');
const SocketServer = require('ws').Server;
// if I include the below line I get the compile error
// const bent = require('bent');

const port = process.env.PORT || 3333;
const server = express()
  .use((req, res) => {
    res.sendFile(INDEX);

    res.header('Access-Control-Allow-Origin', '*');
    res.header('Access-Control-Allow-Methods', 'GET, OPTIONS');
    res.header('Access-Control-Allow-Headers', 'Content-Type');

    return next();
  })
  .listen(port, () => console.log(`Listening on ${port}`));
const socketServer = new SocketServer({ server });

closed time in 3 days

Wancieho

issue commentmikeal/bent

Error: Cannot find module './core'

please upgrade, latest version is 7.1.2

Wancieho

comment created time in 3 days

issue closedmikeal/bent

Support for multiple status codes or the ability to retrieve the failed response

Use case: I want to provide context sensitive information to the end-user relating to what went wrong and why. The responses from the API might be a 200 for a post request if the result was successful, however, if the API discovered invalid input then a 400 would be returned with information on why validation failed.

Problem: Bent only seems to support one status code per request.

Investigation: However, it does look like multiple codes were a consideration at some point. core.js shows the statusCode type as a set and further code checks the set for a matching value. In-fact, changing the code in core.js:29 from:

    } else if (typeof arg === 'number') {
      statusCodes.add(arg)

to:

    } else if (typeof arg === 'number' || Array.isArray(arg)) {
      (Array.isArray(arg) ? arg : [arg]).forEach(a => statusCodes.add(a));

enables the use of multiple status codes.

closed time in 3 days

aaronchilcott

issue commentmikeal/bent

Support for multiple status codes or the ability to retrieve the failed response

multiple status codes already work, they are just multiple parameters;

const get = bent(200, 201)
aaronchilcott

comment created time in 3 days

issue commentmikeal/bent

Wrong parsing of Username

can you provide a code sample of this issue?

JonasGhost

comment created time in 3 days

Pull request review commentipld/js-datastore-car

feat: create car file from a complete graph

 async function writeStream (stream) {   return new CarDatastore(reader, writer) } +async function traverseBlock (block, get, car, concurrency = 1, seen = new Set()) {+  const cid = await block.cid()+  await car.put(cid, block.encodeUnsafe())+  seen.add(cid.toString('base58btc'))+  if (cid.codec === 'raw') return+  const reader = block.reader()+  const missing = link => !seen.has(link.toString('base58btc'))+  const links = Array.from(reader.links()).filter(missing).map((([, link]) => link))+  while (links.length) {+    const chunk = links.splice(0, concurrency)+    const blocks = chunk.map(get)+    while (chunk.length) {+      const link = chunk.shift()+      const block = blocks.shift()+      if (missing(link)) {+        await traverseBlock(await block, get, car, concurrency, seen)+      }+    }+  }+}++async function completeGraph (root, get, car) {

i ended up having to add an optional parameter for configurable concurrency.

mikeal

comment created time in 3 days

push eventipld/js-datastore-car

Mikeal Rogers

commit sha d3a244fc2a5e7a2327fe70cf0ad306880fb6c645

fix: expose concurrency option

view details

push time in 3 days

push eventipld/js-datastore-car

Mikeal Rogers

commit sha e6622d92af354a56262611469dee1f8a52d9ac71

fix: use base58 for CIDv0 support

view details

Mikeal Rogers

commit sha 242997b242fb01df952ec6d5eb9e448b893f82b7

feat: optional concurrency

view details

push time in 3 days

Pull request review commentipld/js-datastore-car

feat: create car file from a complete graph

 async function writeStream (stream) {   return new CarDatastore(reader, writer) } +async function traverseBlock (block, get, car, seen = new Set()) {+  const cid = await block.cid()+  await car.put(cid, block.encodeUnsafe())+  seen.add(cid.toString('base64'))

We can just switch to explicit ‘base58’ instead. I can’t think of anything it would break.

mikeal

comment created time in 5 days

issue commentipfs/unixfs-v2

Transfer to IPLD org

sure, we will probably archive it since the work is happening in the specs repo :)

hsanjuan

comment created time in 6 days

pull request commentipld/js-datastore-car

feat: create car file from a complete graph

I just pushed a fix. I had forgotten that raw blocks don’t have a reader (by design) so you need to filter on the codec before asking for a reader.

mikeal

comment created time in 6 days

Pull request review commentipld/js-datastore-car

feat: create car file from a complete graph

 async function writeStream (stream) {   return new CarDatastore(reader, writer) } +async function traverseBlock (block, get, car, seen = new Set()) {+  const cid = await block.cid()+  await car.put(cid, block.encodeUnsafe())+  seen.add(cid.toString('base64'))

so, the way CID.toString() works is that it caches the string type it was instantiated with and uses that as the default. this was done for perf and for some backwards compatibility, but it makes it problematic when using toString() as a cache key because it isn’t guaranteed to be consistent 😕

mikeal

comment created time in 6 days

push eventipld/js-datastore-car

Mikeal Rogers

commit sha 934ec9a528b8ece156c9dd674d0abf3e4793a427

fix: raw blocks don't have a reader

view details

push time in 6 days

push eventProtoSchool/protoschool.github.io

Deployment Bot (from Travis CI)

commit sha 6474c15a934744bb4f9e1deae059dc978bbd11b4

Deploy proto.school to github.com/ProtoSchool/protoschool.github.io.git:master

view details

push time in 7 days

issue commentrvagg/polendina

Dependency warnings lead to failed test runs

interesting, ill do the workaround for now, thanks :)

-Mikeal


From: Rod Vagg notifications@github.com Sent: Sunday, March 22, 2020 8:43:28 PM To: rvagg/polendina polendina@noreply.github.com Cc: Mikeal Rogers mikeal.rogers@gmail.com; Author author@noreply.github.com Subject: Re: [rvagg/polendina] Dependency warnings lead to failed test runs (#6)

OK, I have an easy fix for you, but it's not really a satisfactory answer and probably something that's worth trying to address.

If you do this to your repo, npm run test:browser works:

diff --git a/test/bytes.spec.js b/test/bytes.spec.js index 36415ff..a58bf9e 100644 --- a/test/bytes.spec.js +++ b/test/bytes.spec.js @@ -1,5 +1,5 @@ 'use strict' -const { it } = require('mocha') + const main = require('../') const parse = require('./parse') const bytes = require('bytesish') diff --git a/test/kinds.spec.js b/test/kinds.spec.js index e87a42a..1fb23c1 100644 --- a/test/kinds.spec.js +++ b/test/kinds.spec.js @@ -1,6 +1,5 @@ 'use strict' const assert = require('assert') -const { it } = require('mocha') const main = require('../') const parse = require('./parse')

diff --git a/test/links.spec.js b/test/links.spec.js index 2702bcf..5b665ba 100644 --- a/test/links.spec.js +++ b/test/links.spec.js @@ -1,5 +1,4 @@ 'use strict' -const { it } = require('mocha') const main = require('../') const parse = require('./parse') const Block = require('@ipld/block') diff --git a/test/structs.spec.js b/test/structs.spec.js index 320e829..c0ac0ee 100644 --- a/test/structs.spec.js +++ b/test/structs.spec.js @@ -1,6 +1,5 @@ 'use strict' const assert = require('assert') -const { it } = require('mocha') const main = require('../') const parse = require('./parse')

diff --git a/test/unions.spec.js b/test/unions.spec.js index 164ca00..cfa7132 100644 --- a/test/unions.spec.js +++ b/test/unions.spec.js @@ -1,5 +1,4 @@ 'use strict' -const { it } = require('mocha') const main = require('../') const parse = require('./parse') const Block = require('@ipld/block')

There's something about directly pulling in mocha that makes it bork. Polendina's Mocha mode requires it being able to load Mocha for you, it pulls in mocha.js rather than index.js which is what you end up with. mocha.js is a single-file compiled version of Mocha that's already safe for the browser, what you've pulled in obviously has some ES features that Webpack needs to be configured more comprehensively to handle, maybe it needs Babel too?

In theory it should be safe to do require('mocha/mocha') to get that file, but it doesn't work properly, I think it ends up with a different instance, or it messes up the start timing, or something. There might be a way for me to rewrite require('mocha') to behave properly. It would be nice to be able to have an explicit require but for now you might just have to /* globals it / or / eslint-env mocha */ to make standard happy if it complains.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/rvagg/polendina/issues/6#issuecomment-602368956, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAAAEQ3HZJIUI4PECWRG7CLRI3LGBANCNFSM4LRAVD6A.

mikeal

comment created time in 10 days

pull request commentmikeal/bent

Add headers attribute in StatusError object

first of all, if you are expecting a 301 you should probably put that status code in as valid and then just check the status code on success :) wrapping that in a try/catch on purpose will make the code perform much worse (prevents lots of VM optimizations).

that said, i have no problem with this feature, it just needs to be identical for the browser side and it needs a test.

Chillaso

comment created time in 10 days

pull request commentipld/specs

[WIP] Flexible Byte Layout

If someone writes incorrect lengths, what do we expect the behavior to be in an implementation? Understanding what the failure mode is would help a bit here.

mikeal

comment created time in 10 days

create barnchmikeal/dagdb

branch : graph-store

created branch time in 10 days

delete branch ipld/js-block

delete branch : equals

delete time in 10 days

push eventipld/js-block

Mikeal Rogers

commit sha d8a54908a698363b724ada88c9a88fc4791b1bc3

feat: equals method (#11)

view details

push time in 10 days

PR merged ipld/js-block

feat: equals method
+17 -0

0 comment

2 changed files

mikeal

pr closed time in 10 days

PR opened ipld/js-block

feat: equals method
+17 -0

0 comment

2 changed files

pr created time in 10 days

create barnchipld/js-block

branch : equals

created branch time in 10 days

create barnchipld/js-block

branch : bare

created branch time in 10 days

push eventmikeal/dagdb

mikeal

commit sha f1fdbd414ff8275015b2a0510872e814d00b9979

deploy: 64207dd46cd7aec58e6f2d3763cc3bdf0420609d

view details

push time in 11 days

push eventmikeal/dagdb

Mikeal Rogers

commit sha 64207dd46cd7aec58e6f2d3763cc3bdf0420609d

doc: more docs

view details

push time in 11 days

push eventmikeal/dagdb

mikeal

commit sha 4033fbc507c6c7a3b13f08b529768ffa13874cdb

deploy: 4d11c7627fc4167eb6c009d7cf1aacf43879d4ff

view details

push time in 11 days

push eventmikeal/dagdb

Mikeal Rogers

commit sha 4d11c7627fc4167eb6c009d7cf1aacf43879d4ff

fix: proper link to getting-started

view details

push time in 11 days

push eventmikeal/dagdb

mikeal

commit sha d78adec70503d5384182a517490471b2bad8094f

deploy: 3ae159c04297780b9ce6e6bfb2d4c976aa9d3d56

view details

push time in 11 days

push eventmikeal/dagdb

Mikeal Rogers

commit sha 3ae159c04297780b9ce6e6bfb2d4c976aa9d3d56

fix: configure doc base correctly

view details

push time in 11 days

create barnchmikeal/dagdb

branch : gh-pages

created branch time in 11 days

push eventmikeal/dagdb

Mikeal Rogers

commit sha fc1340052aef49e07f340cbef85180fc6062b8db

fix: configure new action properly

view details

Mikeal Rogers

commit sha 1ab4a775f43f106254411d26cdd8c7d1bb3920e5

debug: try without fetch checkout

view details

push time in 11 days

push eventmikeal/dagdb

Mikeal Rogers

commit sha 1b7fa553567b598028c1c14f3df57f22a246772e

fix: doc site deploy

view details

push time in 12 days

push eventmikeal/dagdb

Mikeal Rogers

commit sha 415706e43b43d92000014bcf7e20876e6c4ac63c

fix: action syntax

view details

push time in 12 days

push eventmikeal/dagdb

Mikeal Rogers

commit sha d8ac02ad12cf56d83f5ab0f19c8d186034576c71

build: better deploy method for docs

view details

push time in 12 days

push eventmikeal/dagdb

Mikeal Rogers

commit sha 5d241723e9d83b54cfb3775782f9b9529296f0cb

docs: vuepress site

view details

push time in 12 days

push eventmikeal/dagdb

Mikeal Rogers

commit sha 4cbeb213225b0812b0cef7dd3cd9eaccfde86852

doc: update readme

view details

push time in 12 days

push eventmikeal/dagdb

Mikeal Rogers

commit sha 7682661e2009e98cfe76032f6de0095fe4c9346a

build: adding browser tests and docs

view details

Mikeal Rogers

commit sha 6c7a23caf13844da40d50d03b487cec53509bb50

feat: link api

view details

push time in 12 days

issue openedrvagg/polendina

Dependency warnings lead to failed test runs

Trying to use polendina in a new project and seeing some odd behavior.

I get a ton of these warnings:

Bundling warning:  ./node_modules/mocha/lib/esm-utils.js 18:13-44
Critical dependency: the request of a dependency is an expression
    at ImportContextDependency.getWarnings (/root/ipld-schema-validation/node_modules/webpack/lib/dependencies/ContextDepende
ncy.js:40:18)
    at Compilation.reportDependencyErrorsAndWarnings (/root/ipld-schema-validation/node_modules/webpack/lib/Compilation.js:14
54:24)
    at /root/ipld-schema-validation/node_modules/webpack/lib/Compilation.js:1258:10
    at AsyncSeriesHook.eval [as callAsync] (eval at create (/root/ipld-schema-validation/node_modules/tapable/lib/HookCodeFac
tory.js:33:10), <anonymous>:15:1)
    at AsyncSeriesHook.lazyCompileHook (/root/ipld-schema-validation/node_modules/tapable/lib/Hook.js:154:20)
    at Compilation.finish (/root/ipld-schema-validation/node_modules/webpack/lib/Compilation.js:1253:28)
    at /root/ipld-schema-validation/node_modules/webpack/lib/Compiler.js:672:17
    at eval (eval at create (/root/ipld-schema-validation/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:11:
1)
    at /root/ipld-schema-validation/node_modules/webpack/lib/Compilation.js:1185:12
    at /root/ipld-schema-validation/node_modules/webpack/lib/Compilation.js:1097:9
 @ ./node_modules/mocha/lib/mocha.js
 @ ./node_modules/mocha/browser-entry.js 
 @ ./test/bytes.spec.js
 @ ./test/bytes.spec.js

Then the tests just hang and timeout, I never see any output. This is happening in like 3 projects, but one has almost no dependencies and doesn’t do anything fancy. You can reproduce by pulling down https://github.com/mikeal/ipld-schema-validation and running npm run test:browser.

webpack runs fine against the actual index.js so this must be somewhere in polendina and/or the tests themselves.

created time in 12 days

push eventmikeal/ipld-schema-validation

Mikeal Rogers

commit sha 1f5f535a080083de2afce08be541997102cba216

test: browser tests

view details

push time in 12 days

issue commentipld/js-block

Dependency injection patterns for reducing bundle size

there might be an "everything and the kitchen sink" variant or additional module that they could import?

That would still be ‘@ipld/block’, the version without any codecs would be ‘@ipld/block/bare’.

mikeal

comment created time in 12 days

push eventmikeal/ipld-schema-validation

Mikeal Rogers

commit sha 50699e0ef46d1d7c9a7eab5a7ca39ae82ce1022b

fix: removing use of static

view details

push time in 12 days

push eventmikeal/dagdb

Mikeal Rogers

commit sha cb618fe43b6b9cf2263af1e5108e42de8d2a4ee4

feat: store and retrieve arbitrary streams

view details

push time in 12 days

push eventmikeal/bent

Aral Balkan

commit sha 57e7065fab9719cfb65d08c0dae5d73023ce36ae

Only auto set content-type if it has not already been manually set (#75) * Only auto set content-type if it has not already been manually set * Add test Note: this test fails with polendina in a browser environment as it has to create its own http server. But it demonstrates that the code works. Once the echo/info servers used in the other tests are updated to return the correct content-type, the test can be rewritten to use those and browser tests will also pass. * Ensure manual content-type header override test only runs for Node.js

view details

push time in 12 days

issue closedmikeal/bent

Automatic setting of content-type header overwrites manually-specified header

Bent overwrites a manually-specified content-type header when it detects that a request has a body.

To reproduce

  1. Create a POST request
  2. Manually pass a headers object with:
    {
      'Content-Type': 'application/jose+json'
    }
    
  3. Carry out the call.

What should happen

Request’s content type should be application/jose+json.

What actually happens

Request’s content type is application/json.

Use case

RFC 8555 requires the content type of calls to be set to application/jose+json.

PS

Thanks for bent, it’s a joy to work with :)

closed time in 12 days

aral

issue openedipld/js-block

Dependency injection patterns for reducing bundle size

This issue is broader than just the Block API (it touches js-mutliformats, js-cid, js-ipfs, and js-libp2p) but this is as good a place as any to kick off the conversation.

Currently, we ship with support for a large set of:

  • IPLD codecs
  • Hashing functions
  • Multiformat table entries

This is not sustainable. Even the multiformat table, which is only metadata, has become large enough to be a bundle size concern. The number of potential codecs and hashing functions will only grow over time.

In all the libraries we’ve written for IPLD in the last year, we depend on Block API through @ipld/block. It’s how we create blocks which encode data (codecs) and CID’s (hashing and multiformats). Currently, these libraries just require(‘@ipld/block’) which includes all the default codecs and has an interface for adding more codecs. There is no way to pair the default set down and no way to remove or add hashing functions.

In considering this problem, I’ve been writing some of my newer libraries a little different. I’ve been doing the core implementation in a file called bare.js which exports a single function that accepts the Block class and the default codec. https://github.com/ipld/js-fbl/blob/master/src/bare.js#L8 Then in index.js I just return this module with the default require(‘@ipld/block’) class. The idea is, we can expose a Block class in the future that comes without any codecs, hashing functions, or multiformat table entries, and the user can add entries for each that it would like to support. Then that class can be passed into the relevant IPLD libraries that need to create or decode blocks.

const Block = require(‘@ipld/block/bare’)
Block.add(require(‘@ipld/dag-cbor’))
Block.add(require(‘@ipld/blake2’), { default: true })
const fbl = require(‘@ipld/fbl/bare’)(Block, ‘dag-cbor’)

As support for each format is added to the Block API the multiformat entries for each is added to the internal multiformat table. This means we’ll only have complete information for multiformat entries we actually have support for and this would cause a breaking change in CID because the .codec property would no longer work for codecs you didn’t explicitly add support for (the migration for this involves moving away from matching against the string and using the integer for the multiformat entry).

I’d like get feedback on this approach and everyone’s thoughts on the API before I dig in and implement it everywhere.

I also want to make sure this is going to work for the other projects once they adopt the new Block API.

/cc @alanshaw @achingbrain @vmx @rvagg @jacobheun @carsonfarmer

created time in 12 days

issue openedipld/js-ipld-dag-pb

Migrate to a pure Data Model API

This would be a breaking change.

I’d like to use the new IPLD Schema for dag-pb for this codec.

This would be a major API break, but we probably need to do it because it would:

  • make it much easier to use the codec since you’d be able to interact with regular JS objects rather Node instances
  • align with the other implementations of dag-pb
  • allow for selector engine support once the JS selector engine lands

However, this would be a pretty big break for js-ipfs. We’d want to time this along with a larger migration to the Block API.

/cc @achingbrain @vmx @alanshaw @carsonfarmer

created time in 12 days

PR opened ipld/js-datastore-car

feat: create car file from a complete graph

I tried to match the style but I’m sure there’s some differences you may want me to clean up before merging.

I’m not sure if this is your preferred API, it matches what we have in a lot of the other JS libraries but isn’t exactly aligned with the API’s that are here currently. I figured I should wait to write up the docs until you had a chance to look at the PR and potentially change the API.

+88 -0

0 comment

2 changed files

pr created time in 12 days

create barnchipld/js-datastore-car

branch : complete-graph

created branch time in 12 days

pull request commentipld/specs

[WIP] Flexible Byte Layout

Since we’re using a tuple representation the name of the field can be updated later without breaking the schema.

I think this is ready to merge now. Once there’s a consensus on that I’ll create a new issue on the naming so we can find a general term that works for everywhere we’re encountering this problem.

mikeal

comment created time in 13 days

pull request commentipld/specs

[WIP] Flexible Byte Layout

Putting non-local information into a block makes it effectively unreusable / less convergent. It also makes it harder to verify without dredging for more data

We need to document this practice a bit.

Internalizing the merkle constraint that “links only go in one direction” into a design principal that context is additive as you link to child blocks and that each thing you link to should be as useful on its own as possible.

In other words, when children contain information that is dependent on their parent they are useless independently since there is no way to link to a parent.

mikeal

comment created time in 13 days

push eventipld/js-fbl

Mikeal Rogers

commit sha 0808b2d8b1fba3115bae4423a88ed227213a3cff

doc: corrections

view details

push time in 13 days

push eventipld/js-fbl

Mikeal Rogers

commit sha 4384e2ed87c6beddac6bd3ee5fa3f1217ed01bbf

fix: simplify arguments

view details

push time in 13 days

push eventipld/js-fbl

Mikeal Rogers

commit sha f794152e982269d1314a5fcbf09a909f237e73d8

doc: read() and balanced()

view details

push time in 13 days

push eventmikeal/js-fbl

Mikeal Rogers

commit sha 8be911642a5f8d64bcc3ecf0f94bbc4bc7174930

build: automated releases

view details

push time in 13 days

push eventmikeal/js-fbl

Mikeal Rogers

commit sha 6b4c14cbf043029ddd8101cf89679fdf8d0bb353

build: full coverage

view details

push time in 13 days

push eventmikeal/js-fbl

Mikeal Rogers

commit sha aa300919d78e5031afb3250a8725a468c1dc915a

feat: read() w/ tests

view details

push time in 13 days

pull request commentipld/specs

[WIP] Flexible Byte Layout

If it's there, people will use it. My concern is actually in the other direction: if we call it a "hint", people may not validate it because it doesn't have to be correct.

I can see that too.

Still, we have this issue here and elsewhere. We need to figure out what term we want to use and apply it everywhere.

What is a concise way to describe “unverified” or “insecure” information that we can attach to arbitrary nouns?

mikeal

comment created time in 13 days

push eventipld/specs

Mikeal Rogers

commit sha 0f0e45751e5a317f2d47186dfba0a66cdd0f7f46

fix: name lengthHint since the property is insecure

view details

push time in 13 days

push eventmikeal/js-fbl

Mikeal Rogers

commit sha 858fb14cb30d16c638e515d152cfaaa10c41305c

fix: remove support for encoding buffer into CBOR while this was nice in terms of completeness it's actually rather useless and not a good practice to encourage when using the library

view details

push time in 13 days

push eventmikeal/js-fbl

Mikeal Rogers

commit sha ac61daedbf34ffeb7794b58dd1ec73c263c2a5d0

doc: add usage to readme

view details

push time in 13 days

push eventipld/specs

Mikeal Rogers

commit sha 1fda72b36d1dd015c033f1f2495138a2ffcec8ac

fix: reduce to single list tuple

view details

push time in 13 days

pull request commentipld/specs

[WIP] Flexible Byte Layout

Ya, this cleans things up nicely, I’m going to update the spec. https://github.com/mikeal/js-fbl/commit/b33659a0024f6b7d82fe9d56cea893e379a6b187

mikeal

comment created time in 13 days

push eventmikeal/js-fbl

Mikeal Rogers

commit sha b33659a0024f6b7d82fe9d56cea893e379a6b187

fix: another schema revision

view details

push time in 13 days

push eventmikeal/ipld-schema-validation

Mikeal Rogers

commit sha efaee8273fa4c410b108d7de0ee9b4066ddcc0c3

feat: tuple representation for structs

view details

push time in 13 days

pull request commentipld/specs

[WIP] Flexible Byte Layout

There was a lot of iteration so I think the separate lists were due to a prior design constraint we may no longer have. Could do something like:

type NestedByte struct {
  length Int
  part &FlexibleByteList
} representation tuple

type NestedByteList [ NestedByte ]

type FlexibleByteLayout union {
  | Bytes bytes
  | NestedByteList list
} representation kinded

In fact, this matches exactly what gets passed around in the implementation which is then converted to the separate lists.

mikeal

comment created time in 13 days

pull request commentipld/specs

Simplify recursion limit to be optional integer

Given Tim’s new issues I think it’s worth revisiting this.

If I recall, my view previously was that requiring an explicit limit was mostly meaningless because a node will want to impose its own limit anyway. I didn’t have a strong opinion about how we presented the default but now that we’re seeing actual problems in the way we did it, we might as well just go in and make it better.

Looking into the future when there’s more codegen for IPLD Schema, this issue will crop up again so we might as well fix it while we have the flexibility to do so.

Also want to ping @hannahhoward since she dealt with this last time around.

creationix

comment created time in 14 days

Pull request review commentmikeal/bent

Only auto set content-type if it has not already been manually set

 test('json based media type', async () => {   same(json.headers.accept, 'application/vnd.something.com') }) +test('manually-set content-type header when body is present', async () => {

Let’s move this test and the import statement to an else at the bottom so that we only run it on Node.js. By default, all these tests will run in both Node.js and the browser.

aral

comment created time in 14 days

issue commentipld/specs

New codecs for JOSE and COSE

Ok, let’s try to move this forward.

The low hanging fruit here is adding codecs for signed data. Those are doable today without any spec or stack changes in IPLD. Encryption support will require changes elsewhere but we can do the work to fully decode signed data today at the block level.

There’s two approaches we could take.

One is to call it dag-jose-signed? Again, we really can’t do much right now at the codec layer for encrypted data anyway. Also note that you could still encode encrypted data with dag-jose-signed, it just wouldn’t be able to decode anything at the block layer and would pass everything up like it was regular dag-json.

Another option is to call it dag-jose and when we hit encrypted data we could at least do the work of decoding the base64 on the json side so that we hand back a proper buffer. This might make a unified decryption layer in readers easier, since this should make the structure between encrypted JOSE and COSE identical.

oed

comment created time in 14 days

issue commentipld/specs

New codecs for JOSE and COSE

I get that it might not be suitable for the standard way of encrypting data in IPLD. However I still feel it would be very beneficial to support as a specific way since it already has a lot of momentum within other communities. This is also how I understood the intial post. This is not about having the ideal signing/encryption standard, but trying to see if existing standards can work when using IPLD as a intermediate layer. I think IPLD should support such use cases.

Let me clarify. My comment about suitability was more in response to Rod’s comment about potentially adopting these as the starting point for a more generic encryption standard for all of IPLD.

oed

comment created time in 14 days

push eventmikeal/js-fbl

Mikeal Rogers

commit sha b2491e2a90e4b65b6fbea16fc7730e91984dd6ba

fix: latest schema

view details

push time in 14 days

push eventipld/specs

Mikeal Rogers

commit sha 7732a28cb60424b7be4c3f6b5cb7b55503f8a6b1

fix: remove option algorithm field

view details

push time in 14 days

Pull request review commentipld/specs

[WIP] Flexible Byte Layout

+`Flexible Byte Layout` is an advanced layout for representing binary data.++It is flexible enough to support very small and very large (multi-block) binary data.++```sh+type Lengths [Int]+type ByteUnionList [&FlexibleByteLayout]++type NestedByteList struct {+  lengths Lengths+  parts ByteUnionList+  algo optional String

ya, let’s cut it from this spec and put it in a container. efficient mutations, which is the whole point of storing this info, could be considered out of scope here. this is all a lot simpler if the only role of an FBL is to represent an arbitrarily large series of order bytes.

mikeal

comment created time in 14 days

pull request commentipld/specs

[WIP] Flexible Byte Layout

The implementation of this latest revision is much cleaner than previous versions https://github.com/mikeal/js-fbl/blob/master/src/bare.js#L6-L31

mikeal

comment created time in 14 days

more