profile
viewpoint
Mikeal Rogers mikeal Protocol Labs San Francisco http://mikealrogers.com Creator of NodeConf and request.

apache/nano 1132

Nano is now part of Apache CouchDB. Repo moved to https://GitHub.com/apache/couchdb-nano

indutny/caine 144

Friendly butler

ipfs/blog 89

Source for the IPFS Blog

dscape/spell 88

spell is a javascript dictionary module for node.js, and the browser (including amd)

compretend/compretend-img 24

Image element that understands what is in the image, powered by ML.

jhs/couchdb 15

Mirror of Apache CouchDB

ipld/book 10

Decentralized Data Structures Book

isaacs/http-duplex-client 10

Duplex API for making an HTTP request (write the req, read the response)

ipfs/metrics 9

Regularly collect and publish metrics about the IPFS ecosystem

ipld/roadmap 8

IPLD Project Roadmap

push eventmikeal/merge-release

larry kirschner

commit sha e89231be867303a5d4bbc8cf69e0c88a842cbea6

fix: use env var NPM_REGISTRY_URL to determine cur version

view details

Mikeal Rogers

commit sha c05919ea73266f88c27b6ccc6001f1710050846c

Merge pull request #17 from beatthat/fixes-read-version-for-non-npm-reg fix: use env var NPM_REGISTRY_URL to determine cur version

view details

push time in 2 hours

issue commentmikeal/bent

Bent ignores "username:password" basic auth

I didn’t realize fetch already does this for you, we should get this added.

garbados

comment created time in 2 hours

issue commentnodeconf/collective

NodeConf Armenia

@nairihar I setup a CNAME pointed at nodeconf.am, it should work after it propogates.

nairihar

comment created time in 4 hours

issue commentrequest/request

Alternative libraries to request

@domenic ah yes, those are all browser bundle sizes, and since the post is pretty old many of them may be out of date although the bent figure is still close enough.

reconbot

comment created time in 4 hours

issue commentrequest/request

Alternative libraries to request

@domenic are you comparing the browser bundle size? I’m talking about the complexity of debugging these in Node.js. In the browser I would expect most of the node-fetch code to be non-existent, so I don’t really understand what you’re comparing.

reconbot

comment created time in 4 hours

issue commentrequest/request

Alternative libraries to request

@domenic the complexity of emulating the browser API’s is the main problem, it’s a lot of unnecessary code and indirection when trying to debug. You’ve got the Blob API, you’ve got a lot of marshalling for the body, you’ve got almost 400 lines of header marshalling, and that’s not even looking at the actual API that gets exposed.

Like I said, it’s impressive, but it’s also just a ton of terse, clever, and ultimately unnecessary code if you want to do anything except emulate the fetch API.

reconbot

comment created time in 4 hours

issue commentrequest/request

Alternative libraries to request

Since it has been brought up so much I should say a bit about my experience with node-fetch.

First of all, it’s quite an achievement. The amount of code and engineering effort that has gone in to it is much greater than what we ever put into request. fetch seems like a small API and I think people assume the effort to provide a compatible API in Node.js is nominal, but it really is not.

As a result, the code base is massive. It’s a sizable dependency in Node.js, which you likely won’t see at all in browser bundles, but it’s not as though dependency size is not an issue in Node.js, particularly in serverless environments.

node-fetch is indispensable when testing because it does all the work of fully emulating the browser’s APIs, but if you’re using it an application, even one that is being run in Node.js and in the browser, it’s just too much code and too much indirection to be worth it.

IMO, the right approach at this time for a library that wants to be an http client in both Node.js and browsers is to implement a uniform API with a split implementation using fetch in the browser and require(‘http’) in Node.js. Applications, and http clients, should not target fetch or require(‘http’) directly and should not rely on emulating these API’s on either side. This is actually a lot easier than you might think, as you can see in the implementation of bent which is incredibly small https://github.com/mikeal/bent/tree/master/src

reconbot

comment created time in 5 hours

push eventProtoSchool/protoschool.github.io

Deployment Bot (from Travis CI)

commit sha 9b092c8ebd8abf9c47c39f0a0e68350ddd948726

Deploy proto.school to github.com/ProtoSchool/protoschool.github.io.git:master

view details

push time in 6 hours

push eventProtoSchool/protoschool.github.io

Deployment Bot (from Travis CI)

commit sha a83015fea2116065fd263da94eed9d916b6f8f8f

Deploy proto.school to github.com/ProtoSchool/protoschool.github.io.git:master

view details

push time in 8 hours

issue commentrequest/request

Request’s Past, Present and Future

@asgetz all npm does is print that warning when installing a deprecated package, everything else works exactly as it did before.

mikeal

comment created time in a day

Pull request review commentfilecoin-project/lotus

Feat/offline deal making

 var clientDealCmd = &cli.Command{ 		if err != nil { 			return err 		}-		proposal, err := api.ClientStartDeal(ctx, data, a, miner, types.BigInt(price), uint64(dur))++		ref := &storagemarket.DataRef{+			TransferType: storagemarket.TTGraphsync,+			Root:         data,

It depends on what you mean by “input data.” The same CAR file bytes obviously need to produce the same commP, but there are CAR files that are not deterministically generated. So you can definitely have situations in which there are two commP’s for two different CAR files with the same root CID.

This isn’t an issue in the regular deal flow because two parties have to create the same CAR file, so they will always be deterministically generated. But in the offline deal flow we do have CAR files that cannot be deterministically generated from the root CID. We’re creating these for large datasets and some of the CAR files have to be created by pulling the data from source targets in parallel. Deterministic CAR file generation requires that they be created serially to preserve a consistent ordering. We should also expect that in the future people may need to create CAR files for partial graphs and you’d have a similar situation in which two different CAR files have the same root CID.

It’s also just a good idea, given the offline deal flow, to not make assumptions about the determinism of a CAR file based only on the root CID. It’ll always be possible for someone to put a CAR file in the network that is non-deterministic and the only way to validate that it is not deterministic is to parse it the entire CAR file.

whyrusleeping

comment created time in a day

push eventProtoSchool/protoschool.github.io

Deployment Bot (from Travis CI)

commit sha 4e7fd7e43707f3a3147ec14785ac3a551e6f0f52

Deploy proto.school to github.com/ProtoSchool/protoschool.github.io.git:master

view details

push time in 4 days

push eventProtoSchool/protoschool.github.io

Deployment Bot (from Travis CI)

commit sha f030c4f8d89baf1dd63c48a4ae946233dad6fc57

Deploy proto.school to github.com/ProtoSchool/protoschool.github.io.git:master

view details

push time in 5 days

PR opened ipld/roadmap

Reviewers
okr: mid quarter scoring

Filled in a few that I already know but need the other scores as well.

+2 -2

0 comment

1 changed file

pr created time in 5 days

create barnchipld/roadmap

branch : 2020-q1-mid-q-scoring

created branch time in 5 days

issue commentmikeal/merge-release

Publish to GitHub Package Registry

ah, yes! that does need to get changed :)

PR’s welcome :)

geoperez

comment created time in 5 days

issue commentmikeal/bent

Expose options for underlying http client

heres the implementation of a timeout

we need to make sure we provide an API that works in browers as well. this gets tricky because of how fetch needs to be aborted.

jgehrcke

comment created time in 5 days

push eventProtoSchool/protoschool.github.io

Deployment Bot (from Travis CI)

commit sha b915713d41c7e3c289dfae8c62e086b9553576d7

Deploy proto.school to github.com/ProtoSchool/protoschool.github.io.git:master

view details

push time in 5 days

push eventmikeal/js-fbl

Mikeal Rogers

commit sha 353166a043c3b67cf763aa0c5061fd055f7aa1f5

wip: more work on the new fbl

view details

push time in 6 days

issue commentnodejs/node

HTTP 1.1 fallback for HTTP 2 client

I would use this module 😉

mikeal

comment created time in 6 days

issue commentnodejs/node

HTTP 1.1 fallback for HTTP 2 client

first, you need to decide if your client is going to permit using plaintext HTTP/2

nope, just TLS is fine.

How common is the ALTSVC support? Are we all that likely to encounter HTTP/2 servers that only support the other discovery method?

Fairly certain they're going with the Altsvc option tho. Will try to verify.

That seems like it would be the only option for supporting all 3.

then use that to determine the location of the HTTP/2 server.

With the Altsvc method are you not able to re-use the TLS connection when it’s used for HTTP/2?

It’s a bit much to open and then throw away a connection like this.

mikeal

comment created time in 6 days

issue commentnodejs/node

HTTP 1.1 fallback for HTTP 2 client

It's definitely possible but the flow is a bit difficult.

Can we get an example, maybe as a working test, for this? Once I know the flow I can get the right abstractions in place to make it easier.

Also consider that this is going to be even more difficult with the addition of quic/http3. There's no way to upgrade or failover since it's udp instead of tcp.

Ya, that’s fine, one problem at a time 😉 What is the plan for browsers in the regard?

mikeal

comment created time in 6 days

issue commentnodejs/node

HTTP 1.1 fallback for HTTP 2 client

This might be a duplicate of https://github.com/nodejs/node/issues/16256 but it’s unclear if that was actually resolved, it certainly doesn’t seem to be documented yet.

mikeal

comment created time in 6 days

issue openednodejs/node

HTTP 1.1 fallback for HTTP 2 client

I discussed this a while back with @jasnell and I forget where it all landed and what was needed.

I’d like to make it so that bent uses HTTP2 when available and falls back to HTTP 1.1 transparently when it’s not available.

This should be possible, the standard was certainly designed that way, but I can’t find any examples of doing this in Node.js and the core http client libraries are totally separate. It could just be a documentation issue but I think there might actually be some missing API in order to make it work.

created time in 6 days

issue commentmikeal/bent

Support keepAlive in Node.js implementation

You shouldn’t need to pass the agent in. Node.js Core batches concurrent requests together to optimize for keep-alive. The browser does the same.

You’d only need to manage your own agent if you wanted to hold the connection between requests instead of having them closed out when there’s less load.

vgrichina

comment created time in 6 days

IssuesEvent

issue commentmikeal/bent

Expose options for underlying http client

I’m going to re-open. I hadn’t entirely recalled all the issues related to “cancellable promises” and aborting generally.

We may need to start returning new/special promises in order to do this properly

I’m still not sure that we’ll actually add a timeout feature, but we do need proper cancellation in order for users to be able to add a working timeout.

jgehrcke

comment created time in 6 days

issue commentrequest/request

Request’s Past, Present and Future

This is a breaking change and in my opinion unnecessary. Just leave the module as it is and we'll all move on with the next project - particularly if the alternative offers advantages. Indeed we'd be silly not to do so. But as far as I can see there is at the moment no real alternative.

Here’s the thing. This code has known bugs that won’t be fixed. This code is no longer maintained and is deprecated.

The deprecation warning is a notice that you’re relying on problematic code. If you’re fine relying on deprecated and problematic code then simply suppress the messages. Your issue seems to be the warnings and not the state of the software. If you’re fine with the state of the software then simply suppress the warnings.

We will not be altering the deprecation state and relevant warnings to be out of line with reality in order to satisfy any particular users concern over warnings they can easily suppress if they are unconcerned about relying on deprecated modules.

mikeal

comment created time in 6 days

issue commentrequest/request

Request’s Past, Present and Future

Open Source Software grants certain rights to the User

OSS licenses provide rights to re-distribute and modify, no guarantees of any kind are made on the suitability of the software for any particular use. No guarantees are ever made on future changes, including potential breaking changes.

Here’s the relevant text from the Apache 2 license. Pretty much every open source license has this.

“Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.”

mikeal

comment created time in 6 days

issue commentipfs/archive-format

Archiving this repo

+1

vmx

comment created time in 6 days

push eventProtoSchool/protoschool.github.io

Deployment Bot (from Travis CI)

commit sha 59663fe02f53a7191a4f4a8ecf8d0af9728f2f29

Deploy proto.school to github.com/ProtoSchool/protoschool.github.io.git:master

view details

push time in 6 days

pull request commentipld/ipld

Bring entry point docs inline with reality

I also want the link to IRC back.

I want to get the full discourse info in there instead of just IRC.

video

The video isn’t that out of date, but I’d just prefer not to have a video as the first thing you see. We should strive for some introductory text that can be quickly read that gives the reader all the important points they would have gotten from a 20 minute talk. The talk should still be referenced later in the resources section though.

codecs

Good points on the codec implementation list.

Maybe a better approach would be to link from here to each language’s entry point (new and old entry points when relevant) and those README’s can list the codec implementations if they think that’s valuable.

mikeal

comment created time in 6 days

Pull request review commentipld/ipld

Bring entry point docs inline with reality

 IPLD is a set of standards and implementations for creating decentralized data-s - [Contribute](#contribute) - [License](#license) -## Enter IPLD+## Docs -**Watch the Merkle Forest Talk**+- [IPLD Specs](https://github.com/ipld/specs)+- [CID spec](https://github.com/multiformats/cid) -[![](/img/enter-merkle-forest.jpg)](https://www.youtube.com/watch?v=Bqs_LzBjQyk)+## Stacks -## Docs+IPLD has a [layer model]() and is much more than "one thing." As such, implementations of IPLD aren't single libraries but stacks of libraries that vary a bit by programming language.

yup, want to work out the structure before filling in all the details.

mikeal

comment created time in 6 days

issue commentrequest/request

Request’s Past, Present and Future

What about the 151 issues and the 55 pull requests? Dump them?

Nobody has been fixing or reviewing these for some time, they were already “dumped.”

Your comments make it sound as though there is some kind of dedicated labor in this project that people are entitled to. This has never been the case, request is not a product released and backed by a company, it has always been maintained by open source developers who care and as the ecosystem has moved in a new direction we all moved with it. I recommend you move on as well.

mikeal

comment created time in 6 days

issue closedmikeal/bent

How can i use proxy?

closed time in 6 days

yviscool

issue commentmikeal/bent

How can i use proxy?

I’m going to close this as out of scope for bent.

I did this work in request and it was one of the more difficult parts to maintain and it has no real equivalent in the browser which makes it very problematic for bent.

If someone has a good idea on how to add it in painlessly I’ll consider it, but closing for now.

yviscool

comment created time in 6 days

issue closedmikeal/bent

Expose options for underlying http client

Is there a nice way here for exposing the additional configuration options for the underlying http client, such as timeout configuration?

closed time in 6 days

jgehrcke

issue commentmikeal/bent

Expose options for underlying http client

timeout isn’t actually supported in node.js and fetch. there’s a socket timeout in node.js and AFAIK there isn’t a timeout in fetch either. however, there are numerous quality timeout libraries for promises that will work generically with bent so that’s your best option.

jgehrcke

comment created time in 6 days

push eventrequest/request

Gregor Martynus

commit sha 3c0cddc7c8eb60b470e9519da85896ed7ee0081e

README: typo in deprecation message (#3268)

view details

push time in 7 days

PR merged request/request

README: typo in deprecation message

image

+1 -1

0 comment

1 changed file

gr2m

pr closed time in 7 days

issue commentrequest/request

Request’s Past, Present and Future

what about "got" as a replacement, it's sad we don't have a clear replacement while request is officially deprecated.

Maybe we should take the fact that nobody has written an API compatible replacement as an indication that adopting an API compatible replacement is undesirable once you sit down and work on it 🧐

That was certainly my experience.

mikeal

comment created time in 7 days

pull request commentmikeal/dag-cbor-sync

fix: remove promisify

v0.6.3 published.

alanshaw

comment created time in 7 days

push eventmikeal/dag-cbor-sync

Mikeal Rogers

commit sha 32526d6d2a78775de92a38978937c70f04d5f538

0.6.3

view details

push time in 7 days

pull request commentmikeal/dag-cbor-sync

fix: remove promisify

Will do.

At some point we should factor out dependence on this module since the latest dag-cbor is actually sync ;)

alanshaw

comment created time in 7 days

issue commentrequest/request

Request’s Past, Present and Future

You can call it request 3 without much issue imo. Yes the API is breaking but what's what we have semver for.

Spend some time bent and you may feel differently.

It’s not a small difference in naming or promises vs callbacks. The ergonomics are very different, the states it surfaces are very different, the way it thinks about error conditions is a radically different approach.

request is a more procedural API, you tell it to do something and it tells you what it did, it only gives an error if something irrecoverably failed. bent takes the success criteria for the entire lifecycle and returns you an API that will fail if anything but the success criteria is met.

You use these libraries very differently. There are other libraries that are closers to request’s API if that is what you want, but after almost 20 years of working on HTTP clients I’ve found a different and ultimately better approach that I’d encourage people to consider, but I’m not going to ram it down everyone’s throats by making it request 3.0.

Why is it called bent? (request was an easier name to remember.)

Because you “bend” it into a specific shape (very particular success criteria) and it provides an ideal API for the success of that shape and fails on anything but it.

The name is a bit abstract, but request is the kind of name you could never get today. I barely got request in the npm registry and I wrote the original npm registry 😜

mikeal

comment created time in 7 days

issue commentmikeal/merge-release

One time password

There isn’t a great solution to this yet.

Unfortunately, the way npm does OTP restrictions for publishing doesn’t leave a lot of room for automation.

If you disable OTP for publishes it’ll work. I know @bcoe is working on some infrastructure that puts a more limited token infrastructure in front of npm that you then trust with your OTP but I don’t think it’s available yet for people to use without setting up their own infrastructure.

bpierre

comment created time in 7 days

push eventmikeal/ipfs-for-car

Mikeal Rogers

commit sha bd20251bbb08c323b86bc57cc5e5f16814dc21a8

test: add basic linter test

view details

push time in 7 days

push eventmikeal/ipfs-for-car

Mikeal Rogers

commit sha 4c56834a80650e2d532c908858af4e52d154c46c

fix: remove global locking mechanism

view details

push time in 7 days

push eventmikeal/ipfs-for-car

Mikeal Rogers

commit sha 36d2accd8f7de79cafa225e4104b656cc5ac4c91

build: set dev version

view details

push time in 7 days

push eventmikeal/ipfs-for-car

Mikeal Rogers

commit sha 0967c972b509df2ebf19d42063b51013610f3299

fix: clear all data

view details

Mikeal Rogers

commit sha 9df9b1860c0a39f7ccbf0ca970d1bba3220f597b

build: automated releases

view details

push time in 7 days

issue commentipfs/js-ipfs

docs: how do you create an in-memory offline ipfs node for testing?

thanks! :) it works with this workaround now.

mikeal

comment created time in 7 days

push eventmikeal/ipfs-for-car

Mikeal Rogers

commit sha 9ccc487672aa791e6b923d685ad78c8749b3035b

fix: workaround bug in js-ipfs

view details

push time in 7 days

push eventmikeal/bent

Mikeal Rogers

commit sha c38d023f68c421c31b04a6f829eaba466d2fa90d

build: disabling bundle size badges

view details

push time in 7 days

issue commentrequest/request

Request’s Past, Present and Future

‘bent’ has nice errors and is designed for async/await. It’s also incredibly small and the bundle size is teeny tiny ;)

The API is nothing like request though, so I wouldn’t call it a “replacement.”

mikeal

comment created time in 7 days

issue commentrequest/request

Request’s Past, Present and Future

We are fully deprecated!

All versions on npm note the deprecation and the README notes clearly that request has been deprecated.

It’s been a great 10+ years, thanks to everyone who contributed over the last decade. Let’s all look forward to new libraries that are better suited for the changes that are occurring in the JS language and ecosystem.

mikeal

comment created time in 7 days

push eventipld/roadmap

Mikeal Rogers

commit sha 64f01cd31f0b07d213fe95f43673a75a8f87a00c

wip: 2020 q1 OKR’s (#21) * wip: 2020 q1 OKR’s * additional CAR format OKR * additional (low expectation) OKR for specs website Co-authored-by: Rod Vagg <rod@vagg.org>

view details

push time in 7 days

PR merged ipld/roadmap

wip: 2020 q1 OKR’s

Need to get OKR’s in for the rest of the quarter’s work.

Please edit and add @creationix @vmx @rvagg @warpfork

+31 -0

7 comments

1 changed file

mikeal

pr closed time in 7 days

PR opened ipld/ipld

Reviewers
Bring entry point docs inline with reality

We’re very overdue for an update here.

I’ve made some basic changes and started a bit of a new outline, would like to get input and contributions from the team before moving forward.

+12 -13

0 comment

1 changed file

pr created time in 7 days

create barnchipld/ipld

branch : new-entry-point

created branch time in 7 days

push eventrequest/request

Mikeal Rogers

commit sha aded7e4f8e57f6f33cf39d65634bfb822bfcb2c8

doc: note full deprecation (#3267)

view details

push time in 7 days

PR merged request/request

doc: note full deprecation

It’s time.

+6 -0

0 comment

1 changed file

mikeal

pr closed time in 7 days

issue commentrequest/request

Request’s Past, Present and Future

Preparing full deprecation. https://github.com/request/request/pull/3267

mikeal

comment created time in 7 days

PR opened request/request

doc: note full deprecation

It’s time.

+6 -0

0 comment

1 changed file

pr created time in 7 days

create barnchrequest/request

branch : deprecation

created branch time in 7 days

push eventmikeal/dag-cbor-sync

Alan Shaw

commit sha 6302e135726988929737a2a42aa5b7685af6a22d

fix: remove promisify No need to promisify as of v7.

view details

Mikeal Rogers

commit sha 448e540b6218f50b064882b909801c1f8b4b3299

Merge pull request #35 from alanshaw/fix/remove-promisify fix: remove promisify

view details

push time in 7 days

PR merged mikeal/dag-cbor-sync

fix: remove promisify

No need to promisify as of v7.

+1 -3

0 comment

1 changed file

alanshaw

pr closed time in 7 days

push eventProtoSchool/protoschool.github.io

Deployment Bot (from Travis CI)

commit sha 36764e6d45725daca127695b88385f87bac2fa02

Deploy proto.school to github.com/ProtoSchool/protoschool.github.io.git:master

view details

push time in 7 days

issue commentnodeschool/organizers

New chapter: East Bay (San Francisco)

Typically NodeSchool’s are named after cities rather than regions. This is a little confusing since there is at least one other NodeSchool in the East Bay already (NodeSchool Oakland).

Where do you plan on actually holding the workshops? Is there a reason you don’t want to name it after the town/city?

klequis

comment created time in 10 days

pull request commentrvagg/tmp-rust-fil-commp-generate

In memory without temporary file

This might be immovable, we can offload the padded form into memory like in this PR, we could also probably hack in a VecStore in a custom fork of the proofs or re-implementation of the bits that call merkle_light, but then we're taking on a lot of custom code but we're still stuck with roughly 3 times the amount of memory required to process a single piece.

As long as we can get a CAR file that is under 1GB to consume less than 3GB of memory and little to no disc, we’re fine, so let’s avoid forking the code beyond what is necessary to hit that target.

vmx

comment created time in 10 days

pull request commentrvagg/tmp-rust-fil-commp-generate

In memory without temporary file

you’re still seeing disc usage that high with the in-memory changes?

vmx

comment created time in 11 days

push eventmikeal/js-fbl

Mikeal Rogers

commit sha 7429e3d0484ff1a828a27f57c649f8744c9d5746

feat: prepare for bare Block API

view details

push time in 11 days

issue commentmikeal/daily

Fails to render in gmail.

@dym-sh removed that, switching to Courier New, same problem though :(

https://github.com/mikeal/daily/commit/2b2f03e744db8f3614d11ae51822ee8ef08159d7

creationix

comment created time in 11 days

push eventmikeal/daily

Mikeal Rogers

commit sha 3530ade42fae9766469b9c05da26fa70e6a08ea5

feat: add debug segment commented out

view details

push time in 11 days

push eventmikeal/daily

Mikeal Rogers

commit sha 2b2f03e744db8f3614d11ae51822ee8ef08159d7

debug: reduce css

view details

push time in 11 days

issue commentnodejs/node

enabling eslint's prefer-const rule

making sure @rvagg sees this so that he can stop frowning at my aggressive const usage 😜

misterdjules

comment created time in 11 days

issue commentnodejs/node

enabling eslint's prefer-const rule

It has been resolved for some time. Loops using let have comparable performance to loops using var.

what about const?

misterdjules

comment created time in 11 days

issue commentnodejs/node

enabling eslint's prefer-const rule

we should ping folks on the V8 team to get an idea of why this is still an issues it’s been several years now and this syntax isn’t exactly new. it might just be a matter of adding this case into some of the perf suites JS VM’s are running.

misterdjules

comment created time in 11 days

issue commentipfs/js-ipfs

docs: how do you create an in-memory offline ipfs node for testing?

Thanks, I’m sure that helps but it’s still not working.

https://github.com/mikeal/ipfs-for-car/blob/master/index.js

I get Not Found when trying to pull out a block I’ve just added. I’m assuming something is wrong with how I’m configuring the inmemory storage because I doubt basic block storage is broken 🤷‍♂️

first bafyreiec5aigscqhrvctqugyrox5hsjyar624aegy3ooq75usntmffz2hq
(node:4997) UnhandledPromiseRejectionWarning: Error: Not Found
    at Object.module.exports.notFoundError (/root/ipfs-for-car/node_modules/interface-datastore/src/errors.js:21:
16)
    at MemoryDatastore.get (/root/ipfs-for-car/node_modules/interface-datastore/src/memory.js:22:31)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async Object.get (/root/ipfs-for-car/node_modules/ipfs-repo/src/blockstore.js:49:21)
(node:4997) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwi
ng inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch
(). (rejection id: 1)
(node:4997) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
mikeal

comment created time in 11 days

push eventmikeal/ipfs-for-car

Mikeal Rogers

commit sha 5c6c6eb26ba5e9fed2e57bebd65079d62247175b

debug: new script

view details

push time in 11 days

push eventmikeal/ipfs-for-car

Mikeal Rogers

commit sha 39fcbc53a9babe61b755213920e59eb223b00c13

debug: current script

view details

push time in 11 days

pull request commentrvagg/tmp-rust-fil-commp-generate

In memory without temporary file

awesome!

I’m expecting roughly 2x the target file size to be consumed by memory. Could we get this down if we kept the files below 1GB rather than slightly over?

Can we test real quick with this CAR file https://dumbo-stage3-fast-ai-nlp.s3-us-west-2.amazonaws.com/fast-ai-nlp-4.car that is 962MB ;)

vmx

comment created time in 11 days

push eventipld/specs

vmx

commit sha 6fe3426378499eb187646966ec9ed4090b07ef0e

Automated deployment: Fri Feb 7 15:55:50 UTC 2020 a3c982518232b79123af2a2cf5e8642162c62524

view details

push time in 11 days

issue openedipfs/js-ipfs

docs: how do you create an in-memory offline ipfs node for testing

I’ve been trying to create a purely in-memory offline ipfs node and it doesn’t appear to be working. The node appears to function but blocks I’ve just added aren’t actually available. https://github.com/mikeal/ipfs-for-car/blob/master/index.js#L1-L31

created time in 12 days

push eventmikeal/ipfs-for-car

Mikeal Rogers

commit sha 84cacfb440a36f6b59652a7e9cc7974666c59f96

debug: doesnt seem to be working

view details

push time in 12 days

create barnchmikeal/ipfs-for-car

branch : master

created branch time in 12 days

created repositorymikeal/ipfs-for-car

Get an in-memory IPFS instance loaded with a car file

created time in 12 days

issue commentipfs/blog

Schedule a blog post for automatic publication

You could just run it hourly, as long as the script finishes in under an hour there’s no harm in giving you some more granularity on the time.

I have a half dozen repos with hourly actions and the only time they get into trouble is when they take longer than an hour to run.

alanshaw

comment created time in 12 days

pull request commentipld/roadmap

wip: 2020 q1 OKR’s

Immediate needs change a lot, so even a roadmap doesn't make much sense. It's more of a "make progress on the implementation, we should evaluate after the quarter if there was any progress and if not ring the alarm bells" (which doesn't fit into the OKR model).

yes, there’s not a great match between the way we need to do this and the ORK model which is why i’m relying on fuzzy language like “feature complete” ;)

mikeal

comment created time in 12 days

issue commentipfs/js-ipfs

feat: import/export a DAG from/to a CAR file

The car file contains all the blocks for the entire graph, so you have to traverse through all of the links and it’s not quite as simple as this. It could possibly still be a separate command but you’d need to output in a multi-part structure that distinguishes the blocks, which is basically a car file.

The car file library shouldn’t be too large, the POC for export may look large because the default codecs and hash functions used in the new Block API, but js-ipfs wouldn’t need to use that and already includes relevant codecs and hashing functions.

mikeal

comment created time in 13 days

push eventProtoSchool/protoschool.github.io

Deployment Bot (from Travis CI)

commit sha ab79200116d4b2b684cab7012d03ad56101eefd1

Deploy proto.school to github.com/ProtoSchool/protoschool.github.io.git:master

view details

push time in 13 days

create barnchmikeal/js-fbl

branch : master

created branch time in 14 days

created repositorymikeal/js-fbl

Flexible Byte Layout Implementation

created time in 14 days

push eventipld/specs

Mikeal Rogers

commit sha 8790cc4a90b5d08d7948bd7eb305824a522aad7e

fix: latest name change

view details

push time in 14 days

pull request commentipld/specs

feat: require deterministic map serialization

Consistent ordering requires that you prepare the keys to be serialized, either as you insert them or before you serialize. The ordering for dag-cbor would be defined by the codec.

If you’re taking an arbitrary Map-like object and serializing this means preparing the keys right before serialization, and this may involve and extra memcopy. If the API for inserting the keys is custom, and it knows the codec you’ll be using, you could keep the internal ordering consistent with the codec’s ordering rules as you insert them.

If you have a struct schema then it depends on the representation. If the representation is a map then the rules above apply.

This would actually be easier if we defined consistent ordering rules for all the data model codecs and didn’t have any variations, but I don’t know if we are prepared to do that.

mikeal

comment created time in 14 days

issue openedipfs/js-ipfs

feat: import/export a DAG from/to a CAR file

Across IPLD and Filecoin we’re starting to really zero in on CAR files being the preferred import/export format for DAGs.

It would be great if IPFS had an easy way to export the full graph for any CID to a CAR file and to load a CAR file’s blocks into its local blockstore.

It’s not very hard. I was able to write a proof of concept in less than an hour for the exporter https://github.com/mikeal/export-ipld-graph

created time in 14 days

create barnchmikeal/export-ipld-graph

branch : master

created branch time in 14 days

created repositorymikeal/export-ipld-graph

Exports and IPLD graph from the IPFS network

created time in 14 days

push eventmikeal/publish-to-github-action

Mikeal Rogers

commit sha 000c8a4f43e2a7dd4aab81e3fdf8be36d4457ed8

fix: fixing validation issues from merge-release

view details

push time in 14 days

more