profile
viewpoint
Mathias Buus mafintosh @hyperdivision Copenhagen, Denmark https://mafinto.sh Rød grød med fløde

browserify/browserify 13216

browser-side require() the node.js way

datproject/dat 7896

:floppy_disk: peer-to-peer sharing & live syncronization of files via command line

browserify/watchify 1762

watch mode for browserify builds

datproject/dat-desktop 593

Peer to peer data syncronization

bcle/fuse4js 250

FUSE bindings for Javascript and node.js

chriso/lru 120

A simple LRU cache

chrisdickinson/varint 69

use msb to create integer values of varying sizes

adventure-js/arcticjs.club 67

The http://arcticjs.club website

chm-diederichs/minisign 65

JavaScript implementation of Frank Denis' (@jedisct1) minisign tool.

andrewosh/corestore 28

A simple corestore that wraps a random-access-storage module

pull request commentsodium-friends/sodium-javascript

Add chacha20 and chacha20_ietf

@chm-diederichs sounds like a native “bug”

chm-diederichs

comment created time in 13 hours

issue commentbeakerbrowser/beaker

Can't access hyperdrives with seeds

Are you using any kind of vpn?

If you are up for it, mb you want to join the discord, chat.hypercore-protocol.org and we can do some more realtime debugging

aadilayub

comment created time in 3 days

pull request commentsodium-friends/sodium-javascript

Add chacha20 and chacha20_ietf

@emilbayes @chm-diederichs we need tests here or is this lgtm?

chm-diederichs

comment created time in 3 days

Pull request review commentdatproject/comm-comm

Added notes for meeting 53

+## Dat comm-comm 2020-05-21++## Attendees++- Mauve+- Nina+- Kevin Faaborg @zootella, from Ara+- Don Blair from edgecollective.io+- Kevin Jahns+- Krzysztof Modras @chrmod (Cliqz) +- Micah (@micahscopes)+- YOUR NAME HERE++## Intros+- Mauve+    - working on dat-sdk, who knows what it'll be called in the future?+- Bruno+    - working on dat-rs... wants to see dat everywhere!+- Don +    - working with farmers on decentralized resilient ag tech, experimenting with dat / cabal / hyper for p2p farm communications+    - more deets at [edgecollective](https://edgecollective.io/)+    - sorry my baby is sleeping right here so can't use audio haha :)+    - QUESTION I'd like to ask about: how to assess p2p connectivity with dat / hyper?  e.g. is there some fancy hyperswarm equivalent of 'dat doctor' to figure out whether one is able to connect to network / other peers?+- Kevin Faaborg+    - working on Ara https://ara.one, using hyper-stack for distributed payment platform for creators and fans++- Micah+    - p2p enthusiast+- Kevin Jahns+    - working on [Yjs](https://github.com/yjs/yjs) CRDT library+- Krzysztof+    - working on cliqz on the iOS browser+    - excited to hear more about how people are working with hypercore-protocol+    - playing with DAT on React Native https://github.com/cliqz/dat-react-native+- Martin+    - +- Mafintosh+    - creates hypercore bugs!

so true

RangerMauve

comment created time in 4 days

issue commentbeakerbrowser/beaker

HyperTrie api

Is there an API for creating new Hypertries? (prob yes) yes

Is there an API for deleteing a Hypertrie? v2 feature

Is there a browser UI for creating new Hypertries? (prob no) v3 feature

Where do users find their created Hypertries? (in the library? somehow attached to the site that created the trie?) a simple list in the library would be fine for now

How do users manage their Hypertries? (is it like a drive or is it more like localStorage values which are managed via devtools and cleared when you clear a site's storage?) they are like drives

What does Beaker render when you visit a Hypertrie? Just a page saying "this is a hypertrie with X changes"

mafintosh

comment created time in 4 days

issue closedhypercore-protocol/hyperdrive-daemon

Segfault in Node.js v14

#
# Fatal error in , line 0
# Check failed: result.second.
#
#
#
#FailureMessage Object: 0x7fff4d88eb80
 1: 0xa95a01  [hyperdrive]
 2: 0x19617d4 V8_Fatal(char const*, ...) [hyperdrive]
 3: 0xe58149 v8::internal::GlobalBackingStoreRegistry::Register(std::shared_ptr<v8::internal::BackingStore>) [hyperdrive]
 4: 0xbc5418 v8::ArrayBuffer::GetBackingStore() [hyperdrive]
 5: 0xa08589 node::Buffer::New(node::Environment*, char*, unsigned long, void (*)(char*, void*), void*) [hyperdrive]
 6: 0xa089f3 node::Buffer::New(v8::Isolate*, char*, unsigned long, void (*)(char*, void*), void*) [hyperdrive]
 7: 0x9fdbdd napi_create_external_buffer [hyperdrive]
 8: 0x7fcb29420934  [/usr/local/lib/node_modules/hyperdrive-daemon/node_modules/fuse-native/prebuilds/linux-x64/node.napi.node]
 9: 0x13203a9  [hyperdrive]
10: 0x1332810  [hyperdrive]
11: 0x1320cb8 uv_run [hyperdrive]
12: 0xa6a334 node::NodeMainInstance::Run() [hyperdrive]
13: 0x9f9651 node::Start(int, char**) [hyperdrive]
14: 0x7fcb2c52e0b3 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
15: 0x992f0c  [hyperdrive]

closed time in 4 days

bit

issue commenthypercore-protocol/hyperdrive-daemon

Segfault in Node.js v14

Should be fixed in the latest version :) (fuse version 2.2.2)

bit

comment created time in 4 days

issue commenthypercore-protocol/hyperdrive-daemon

Crashed the daemon

@okdistribute fixed in todays release if you reinstall deps/upgrade beaker to latesr

okdistribute

comment created time in 5 days

issue commenthypercore-protocol/hyperdrive-daemon

Hyperdrive directory emptied

Thanks, we’ll take a look. Just for a bit more debugging can you see if the size of ~/.hyperdrive/storage is roughly equal to your file size you put in? Assuming you didn’t add any other files

ylebre

comment created time in 5 days

Pull request review commentfuse-friends/fuse-native

malloc and track thread_local buffers from js

 before_deploy:  cache:   npm: false++deploy:+  provider: releases+  draft: false+  prerelease: true+  api_key:+    secure: "KPn3xR4LWcg/H259aSZh26XX0eapR88xSNUkBmEri/sCJSyZ0+asLZSv/HDD3KJP4HeuIKsQc0v8fcebD83fkvaSvlUzSppMQgniwuGC1cAefbrgZDwmJJ/n+lE8Wr9x4adOBTgICS5Uc8LlZ1PuJGm4mmequVs29BEw9738LzN4+3NpoCoWd0FAgGF0tDTsaYL1tJqERAyNIxHS+adUPe0F2r0d2UJ7mOrW7s8Ai6e6QryFsFvA2m0Xn/pQmNO/mcq+LPcw57pWuI3Hm3Cu3W8VPJXYp/yJaFqTAn3D9Fwz4tkmbfmca4ETwZYOS3lvL/rjLQ+69SJlRRu/QfPECkZven+wwsLX/DmyGHgEWqeGWjKj/NxYOIKUKEZZCVrF8cy4j9mac+LK6bAeDZERKSxUJ9GT5WsjvV3RNKgp3MZF7mtmj4IWXfgcuwFX49oIqhzSJsucBBXlB74J7Qua5VJPEAo/7X7Q+Y9IT9JHwxXsXVF5ZNj1PMlJicVD6oKi4XCFOVxSE9wdzlBwMOlUyBGhAIzS6lmxHOELYO9C7l8t/8Zvi4a+YGvOwn0dzLb9zuA1bzqJmEB1fkQMZXHvcEY1o5jSTQ0cNn1Wx4Ck9zyLyhnZ5KRXKzGQ1du55iVOThcbl/8j6zT218SiZMMtv8ZwPy4pJt4skMGsoOZtYlE="  file: "$ARCHIVE_NAME"

Ya it’s an encrypted key

mafintosh

comment created time in 5 days

delete tag fuse-friends/fuse-native

delete tag : v2.3.0

delete time in 5 days

release fuse-friends/fuse-native

v2.2.2

released time in 5 days

push eventfuse-friends/fuse-native

Mathias Buus

commit sha 4606e58acd4831b8360b3243e166993a13667b28

2.2.2

view details

push time in 5 days

push eventfuse-friends/fuse-native

Mathias Buus

commit sha b7f98c619235829f0001fe94a154d8a40d4cd990

fix travis file

view details

push time in 5 days

created tagfuse-friends/fuse-native

tagv2.2.2

Multithreaded FUSE bindings for Node JS.

created time in 5 days

push eventfuse-friends/fuse-native

Mathias Buus

commit sha 46aae54d5cbeabd4a41139859bf90ca48824a79c

2.2.2

view details

push time in 5 days

push eventfuse-friends/fuse-native

push time in 5 days

push eventfuse-friends/fuse-native

Mathias Buus

commit sha 3507fe09070fc580e47ab5d83ab6705fefa9c233

fix travis link

view details

Mathias Buus

commit sha 678d0f9c360a3a8f4090fffd3a579d342f16aa0f

2.3.0

view details

push time in 5 days

created tagfuse-friends/fuse-native

tagv2.3.0

Multithreaded FUSE bindings for Node JS.

created time in 5 days

delete branch fuse-friends/fuse-native

delete branch : fix-node-14

delete time in 5 days

push eventfuse-friends/fuse-native

Mathias Buus

commit sha e218d40f909d197032691ec0f22b955c7eebe188

malloc and track thread_local buffers from js (#20) * malloc and track thread_local buffers from js to avoid crashes when multiple external buffers are made from the same pointer * make tests pass on 14.3.0 * disable npm cache * debug log * detach array buffer on read free as a test * fix detatch * detach arraybuffer asap, to avoid crashes * reset test * add secure travis key * missing arraybuffer pass * bump all the flags * no linux 32

view details

push time in 5 days

PR merged fuse-friends/fuse-native

malloc and track thread_local buffers from js

to avoid crashes when multiple external buffers are made from the same pointer.

  • example currently runs but tests still crash
  • gotta verify and node 14.3 fixes the crash
+120 -46

0 comment

4 changed files

mafintosh

pr closed time in 5 days

push eventfuse-friends/fuse-native

Mathias Buus

commit sha 32aa40878fe6ff449dd365e4964ce2d47dbb7e38

no linux 32

view details

push time in 5 days

push eventfuse-friends/fuse-native

Mathias Buus

commit sha 8b8bab9f3c8e7b37eacaf0a1b1eab3f2cac9782d

bump all the flags

view details

push time in 5 days

push eventandrewosh/hyperdrive-fuse

Mathias Buus

commit sha 4b193f5445bc2a75365709c2b96e0de6f46a293c

bump corestore

view details

push time in 5 days

push eventfuse-friends/fuse-native

Mathias Buus

commit sha fcb620b614a0a0926209df9dfdb29c92fe7681a8

missing arraybuffer pass

view details

push time in 5 days

push eventfuse-friends/fuse-native

Mathias Buus

commit sha b0e760426e444b53071cdeb25a1cc55030366bf7

add secure travis key

view details

push time in 5 days

pull request commenthypercore-protocol/website

add liberate science

Nice! Will be auto deployed

juliangruber

comment created time in 5 days

push eventhypercore-protocol/website

Julian Gruber

commit sha c1506d7eba47f61abb10da40bca92e8e78b28e78

add liberate science

view details

push time in 5 days

push eventmafintosh/dht-rpc

Mathias Buus

commit sha 26e565cb731c28dc4fc2ca2061d1abaeabeb19cd

we dropped 6 and 8 in sodium

view details

push time in 5 days

push eventfuse-friends/fuse-native

Mathias Buus

commit sha 1f6ecb8bea51775bfe3f5fed8344fe29efa09581

detach arraybuffer asap, to avoid crashes

view details

Mathias Buus

commit sha ac6c5c46bb028adcdc803f801ac62b7c465a0013

reset test

view details

push time in 5 days

push eventfuse-friends/fuse-native

Mathias Buus

commit sha 570fd55297ef47266298adb7e7c54be03ea45934

fix detatch

view details

push time in 5 days

push eventfuse-friends/fuse-native

Mathias Buus

commit sha 4555cc43a0ef0a277a8b527fd8289a0bef36d50c

detach array buffer on read free as a test

view details

push time in 5 days

push eventfuse-friends/fuse-native

Mathias Buus

commit sha 258346d835f416dc923468edc7b35bc2c3571d1d

debug log

view details

push time in 6 days

push eventfuse-friends/fuse-native

Mathias Buus

commit sha d309c2d7417e851c84dfb14348cb19ff4312a3ed

disable npm cache

view details

push time in 6 days

pull request commentfuse-friends/fuse-native

fix: remove incorrect flush parameter

Thanks

piedar

comment created time in 6 days

push eventfuse-friends/fuse-native

Bennjamin Blast

commit sha 519b468995bcea43d06a082cc629a06ed9dc9282

fix: remove incorrect flush parameter

view details

push time in 6 days

PR merged fuse-friends/fuse-native

fix: remove incorrect flush parameter

I have some code like this

flush (path, fd, reply) {
  debug("flush", { path, fd })

  writer.flush(path)
    .then(() => reply(0))
    .catch(err => {
      debug({ err })
      reply(errorToFuseCode(err))
    })
}

but it produces

UnhandledPromiseRejectionWarning: TypeError: reply is not a function

because an extra datasync parameter gets in the way of the documented reply function.

This change removes the incorrect extra parameter.

+2 -2

0 comment

1 changed file

piedar

pr closed time in 6 days

push eventfuse-friends/fuse-native

Mathias Buus

commit sha 5b8c880d6c117359c6e1e5ba7919e8ae76015676

make tests pass on 14.3.0

view details

push time in 6 days

issue commentsodium-friends/sodium-native

Can't compile on Windows for Electron v9 or Node lower than v12.16 (napi_detach_arraybuffer thing)

You can work around this by simply compiling against Node latest LTS on your platform.

The n-api build works seamlessly with electron

vladimiry

comment created time in 6 days

PR opened fuse-friends/fuse-native

malloc and track thread_local buffers from js

to avoid crashes when multiple external buffers are made from the same pointer.

  • example currently runs but tests still crash
  • gotta verify and node 14.3 fixes the crash
+48 -15

0 comment

2 changed files

pr created time in 6 days

create barnchfuse-friends/fuse-native

branch : fix-node-14

created branch time in 6 days

issue commentsodium-friends/sodium-native

Can't compile on Windows for Electron v9 or Node lower than v12.16 (napi_detach_arraybuffer thing)

Unrelated to your issue, why are compiling it?

vladimiry

comment created time in 6 days

created taghypercore-protocol/hypercore

tagv9.1.0

Hypercore is a secure, distributed append-only log.

created time in 6 days

push eventhypercore-protocol/hypercore

Mathias Buus

commit sha 8db2cd49a75881ca68c81e8326a31372cb4714fe

prepare v9.1.0

view details

Mathias Buus

commit sha 339051c9ec88640892cf5d69f64b1b47e55296c1

9.1.0

view details

push time in 6 days

push eventhypercore-protocol/hypercore

Andrew Osheroff

commit sha f443a1d8f7fd62d84e86feef8388569b740863b1

Make peer._remoteOpened public

view details

push time in 6 days

PR merged hypercore-protocol/hypercore

Make peer._remoteOpened public

Need to use this property to filter peers that are added but don't have the capability.

+4 -4

0 comment

1 changed file

andrewosh

pr closed time in 6 days

push eventhypercore-protocol/hypercore

Mathias Buus

commit sha fbc670565de42d40f6a7e14af652d197f255e19f

extra 12 -> 14

view details

push time in 6 days

issue commentbeakerbrowser/beaker

Can't load any hyperdrives

This sounds a lot to me like the session mismatch bug that’ll be fixed in the next release

aadilayub

comment created time in 6 days

pull request commentnodejs/node

buffer,n-api: release external buffers from BackingStore callback

Ok, I’ll try to poke around it a bit and avoid the dual external buffer on the same pointer and see if it works

addaleax

comment created time in 6 days

issue commenthypercore-protocol/community

Proposal for Trackers

We did talk about the backoff quite a bit, and it’s def a concern

creationix

comment created time in 6 days

issue commenthypercore-protocol/community

Proposal for Trackers

I did look at quic, but I’m a sucker for the simplicity of utp. I know a friend is working on native bindings for quic right now so we can hopefully test it soon.

Have you ever messed around with the backoff configurations with utp?

creationix

comment created time in 6 days

pull request commentnodejs/node

buffer,n-api: release external buffers from BackingStore callback

@addaleax its a bit of macros on macros I know 😅. That’s a good pointer! I’ll investigate that. The actual crash is coming from the get buffer info call know. Do you know on the top of your head, if that would crash if i made two external buffers from the same pointer?

addaleax

comment created time in 6 days

issue commenthypercore-protocol/community

Provide self-contained installer for hypercore daemon

@rklaehn as a rule use latest lts, that’s the one we use.

That’s Node 12 atm

@andrewosh did we add this to the readme or just discuss it?

rklaehn

comment created time in 6 days

issue commentbeakerbrowser/beaker

HyperTrie api

@pfrazee def async iterators

mafintosh

comment created time in 6 days

issue commenthypercore-protocol/hyperdrive-daemon

.datignore equivalent

Ya, saw a lot of people reporting windows issues with it is the context I forgot to add

PepBro

comment created time in 6 days

issue commentnodejs/node

n-api: does napi_create_external_buffer copy?

Did some testing and it seems to be not memcopying it. If other people agree that this is the correct behaivour and that the docs are confusing I can volunteer a PR

mafintosh

comment created time in 6 days

issue openednodejs/node

n-api: does napi_create_external_buffer copy?

Hunting down a bug related to napi_create_external_buffer, I got a bit confused by reading the docs here:

https://nodejs.org/dist/latest-v14.x/docs/api/n-api.html#n_api_napi_create_external_buffer

[in] data: Raw pointer to the underlying buffer to copy from.

Does this mean that the buffer is created with the same pointer or a new pointer that's a memcopy?

Further down on the docs for the same method it also says:

This API allocates a node::Buffer object and initializes it with data backed by the passed in buffer.

Which I'm also not sure wheather to interpret as a memcopy

created time in 6 days

pull request commentnodejs/node

buffer,n-api: release external buffers from BackingStore callback

@addaleax I keep getting Check failed: result.second still with this commit, in our fuse-native module when making external buffers from n-api (node 12 is fine).

Any ideas? Get the crash wheather I using finalisers or not.

addaleax

comment created time in 6 days

created tagfuse-friends/fuse-shared-library-darwin

tagv1.1.0

A module containing the dylib needed to run FUSE on Mac (darwin).

created time in 6 days

created tagfuse-friends/fuse-shared-library-darwin

tagv1.0.2

A module containing the dylib needed to run FUSE on Mac (darwin).

created time in 6 days

push eventfuse-friends/fuse-shared-library-darwin

Mathias Buus

commit sha e690a63b67f57147ce222c20fd2928778e979d61

1.1.0

view details

push time in 6 days

push eventfuse-friends/fuse-shared-library-darwin

push time in 6 days

created tagfuse-friends/fuse-shared-library-darwin

tagv1.0.3

A module containing the dylib needed to run FUSE on Mac (darwin).

created time in 6 days

push eventfuse-friends/fuse-shared-library-darwin

Mathias Buus

commit sha b236899d6cc6c8b8066c3c6bb79c7a109095f80f

bundle osxfuse as tar

view details

Mathias Buus

commit sha 9a2f37f6012febfab7a645bc3b00171121938fa8

more explicit osxfuse license pointer

view details

Mathias Buus

commit sha 032d74ab217d26f1c5a4b13d7bb570895497c0b1

1.0.3

view details

push time in 6 days

issue commenthypercore-protocol/community

Proposal for Trackers

utp + tcp. I did some perf fixes to our bindings so the throughput on utp is quite good now.

If you know any good utp replacements I'm down to investigate changing that part also, as long as they are easy to multiplex with the dht traffic (utp allows that which is awesome)

creationix

comment created time in 6 days

created taghypercore-protocol/hypercore

tagv9.0.1

Hypercore is a secure, distributed append-only log.

created time in 6 days

push eventhypercore-protocol/hypercore

Mathias Buus

commit sha 2085d3b01d0ae27bee4c77d998da9bffe150fd46

bump standard

view details

Mathias Buus

commit sha 84c5b9ed82257c0a8dbf25389344e03ef0d22617

fix travis links

view details

Mathias Buus

commit sha fa511baec75bd7003c41554bc7ac20e9f6e2ed20

prepare 9.0.1

view details

Mathias Buus

commit sha bb02856f3057b4d36c87a747222f88200c0eaeec

9.0.1

view details

push time in 6 days

push eventhypercore-protocol/hypercore

Martín Acosta

commit sha e1510f13098c49a9edacff513f6b77ddff0ee38c

Add batch support in createReadStream (#261) * First commit * Added benchmark * Refactor * Fixed batchStart * Added test * Validate downloaded blocks and fixed package.json * Simplify code * Minor change * Updated readme * Updated readme

view details

push time in 6 days

PR merged hypercore-protocol/hypercore

Reviewers
Add batch support in createReadStream

It adds support for batch reading in createReadStream and improve the performance.

It's an initial idea and feedback is welcome.

Usage

feed.createReadStream({ batch: 100 }) // read batch of 100 messages

Benchmark

> node read-64kb-blocks-linear.js
685658891 bytes/s
10448 blocks/s

> node read-64kb-blocks-linear-batch.js // batch 100 
1080223162 bytes/s
16384 blocks/s

Todo

  • [x] Add test (probably I will use the same tests that we use to test createReadStream)
+153 -86

7 comments

5 changed files

tinchoz49

pr closed time in 6 days

pull request commentlgeiger/node-abi

add electron 9 support

Thanks!

develerik

comment created time in 6 days

push eventlgeiger/node-abi

Erik Bender

commit sha 8934a47f029c0f6fafc3ef1eeea26adb028f5c6b

feat: add electron 9 support (#86) * add electron 9 support * fix electron 9 abi version

view details

push time in 6 days

PR merged lgeiger/node-abi

add electron 9 support
+7 -4

0 comment

2 changed files

develerik

pr closed time in 6 days

created tagmafintosh/hypercore-multi-key

tagv4.0.0

Function that takes a hypercore and returns a new one that is signed by a new key pair but backed by the same data

created time in 6 days

push eventmafintosh/hypercore-multi-key

Mathias Buus

commit sha 15acaa2fa1d8dd1ad0a64ba03a3b90ec5dde3168

upgrade to hypercore 9

view details

Mathias Buus

commit sha 39b56be8cb3dff689b3f2e6a888dda8aa3a9e9d0

4.0.0

view details

push time in 6 days

push eventhypercore-protocol/hypercore

Mathias Buus

commit sha 899ca50801a399f7d89cc16abdf87cef4f5d6da6

bump travis

view details

push time in 6 days

push eventhypercore-protocol/hypercore

Jemma Nelson

commit sha c3c23f0e3926950b7fa61fe7f1c4649b616c35a8

Fix replication benchmarks These were broken since commit a4e5142 corresponding to the v8.0.0 release. Hypercore-protocol version was bumped to 7, which requires the addition of the `initiator` parameter for replication.

view details

push time in 6 days

PR merged hypercore-protocol/hypercore

Fix replication benchmarks

These were broken since commit a4e5142 corresponding to the v8.0.0 release. Hypercore-protocol version was bumped to 7, which requires the addition of the initiator parameter for replication.

+4 -4

1 comment

2 changed files

fwip

pr closed time in 6 days

pull request commenthypercore-protocol/hypercore

Fix replication benchmarks

Thank you!!

fwip

comment created time in 6 days

issue commenthypercore-protocol/hyperdrive-daemon

.datignore equivalent

We already have all the plumbing internally. Does anyone know of an up-to-update ignore file parser?

PepBro

comment created time in 6 days

push eventmafintosh/user-submitted-drives

Mathias Buus

commit sha d5361d1e5a3d7b82d21adbfad02f5cb50a0148e4

Update README.md

view details

push time in 7 days

push eventmafintosh/user-submitted-drives

Mathias Buus

commit sha 6851aaf12c8605a5103af947f34d97e5dcdbaa39

Update README.md

view details

push time in 7 days

issue commentbeakerbrowser/beaker

HyperTrie api

beaker.hypertrie.diff(url: String, version: Number): Promise<DiffDesc[]> should return an iterator also with <left: Node, right: Node>

beaker.hypertrie.get(url: String): Promise<String|ArrayBuffer> and the iterator returns Node

type Node {
  seq: PositiveInteger,
  key: String,
  value: Optional<String|Uint8Array>
}

In put/BatchAction the value should also be Uint8Array instead of ArrayBuffer

mafintosh

comment created time in 7 days

pull request commenthypercore-protocol/hypercore

Add batch support in createReadStream

@tinchoz49 Thanks, I'll land and release this tmw :)

If you wanna post your perf stats here we can include it in the changelog

tinchoz49

comment created time in 7 days

pull request commenthypercore-protocol/hypercore

Add batch support in createReadStream

Looks GREAT @tinchoz49 !! Seems to be much faster as well. I've added a few nits if you wanna take a look.

tinchoz49

comment created time in 7 days

Pull request review commenthypercore-protocol/hypercore

Add batch support in createReadStream

 var collect = require('stream-collector') var create = require('./helpers/create') var bufferFrom = require('buffer-from') -tape('createReadStream to createWriteStream', function (t) {-  var feed1 = create()-  var feed2 = create()--  feed1.append(['hello', 'world'], function () {-    var r = feed1.createReadStream()-    var w = feed2.createWriteStream()--    r.pipe(w).on('finish', function () {-      collect(feed2.createReadStream(), function (err, data) {-        t.error(err, 'no error')-        t.same(data, [bufferFrom('hello'), bufferFrom('world')])-        t.end()+function test (batch = 1) {+  tape('createReadStream to createWriteStream', function (t) {+    var feed1 = create()+    var feed2 = create()++    feed1.append(['hello', 'world'], function () {+      var r = feed1.createReadStream({ batch })+      var w = feed2.createWriteStream()++      r.pipe(w).on('finish', function () {+        collect(feed2.createReadStream({ batch }), function (err, data) {+          t.error(err, 'no error')+          t.same(data, [bufferFrom('hello'), bufferFrom('world')])

Unrelated to this PR this made me realise we need this https://github.com/hypercore-protocol/hypercore/issues/265

tinchoz49

comment created time in 7 days

issue openedhypercore-protocol/hypercore

Remove bufferFrom dep

We already dropped node 4

created time in 7 days

Pull request review commenthypercore-protocol/hypercore

Add batch support in createReadStream

 Feed.prototype.createReadStream = function (opts) {       read(size, cb)     })   }++  function setStart (value) {+    var _start = start

_start -> prevStart

tinchoz49

comment created time in 7 days

Pull request review commenthypercore-protocol/hypercore

Add batch support in createReadStream

 See [UPGRADE.md](UPGRADE.md) for notes on breaking changes for downstream develo - Ease of use update to signatures, https://github.com/mafintosh/hypercore/issues/260 - Updates [noise-protocol](https://github.com/emilbayes/noise-protocol) to latest, which uses chacha instead of xchacha and moves avoid from the sodium kx api for better compatability with the rest of the Noise ecosystem. - Updates [sodium-native](https://github.com/sodium-friends/sodium-native) from 2 to 3 across the board. 3 uses n-api, so no more builds needed when Node is released.+- Update the discovery key to hash in the protocol version for better migration

this is from the first commit which i force pushed out. can you remove that commit as well?

tinchoz49

comment created time in 7 days

pull request commenthypercore-protocol/hypercore

Add batch support in createReadStream

@tinchoz49 perhaps we could do a more opportunistic thing where we just check how many messages we have on disk compared to where we are and let's say we have the next 100 messages already downloaded we use the batch api to get them out

tinchoz49

comment created time in 7 days

issue commentbeakerbrowser/beaker

Host This Drive resets every time I click it

I should win coolest gif tho right?

mafintosh

comment created time in 7 days

push eventmafintosh/user-submitted-drives

Mathias Buus

commit sha 1ff3947f6736a13da7d349c7dbb238020f55b350

change summary text

view details

push time in 7 days

create barnchmafintosh/user-submitted-drives

branch : master

created branch time in 7 days

created repositorymafintosh/user-submitted-drives

An experiment to build a news aggrigator on Beaker

created time in 7 days

issue openedbeakerbrowser/beaker

HyperTrie api

When writing more dynamic sites I’m missing a p2p database api in addition to the file one. Particularly I miss:

  • multi value transactions
  • streaming iteration
  • inlined values (for perf, less important)

We already have HyperTrie which supports all those 3 things.

Would love to get access to webified version of that that exposes in some form put, del, get, iterator, batch, diff, checkout

created time in 7 days

created taghyperswarm/cli

tagv1.4.3

Series of CLI tools for Hyperswarm

created time in 7 days

push eventhyperswarm/cli

Mathias Buus

commit sha 389548906a196ea9d73e8c9abc049136f4a8a704

universal -> native

view details

Mathias Buus

commit sha c63a4bfbe2c9a9567da0119d76f9051834c21bc7

1.4.3

view details

push time in 7 days

create barnchmafintosh/hyperdomains

branch : master

created branch time in 7 days

more