profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/pvh/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
Peter van Hardenberg pvh @inkandswitch San Francisco ink and switch research formerly: heroku, songbird, etc.

orionz/judo 26

A tool for sparing with EC2

mjtognetti/pushpin-peer 15

A cloud peer for pushpin to keep data warm while your computer is sleeping.

ept/pushpin-papoc 13

PushPin: Towards Production-Quality Peer-to-Peer Collaboration

orionz/kuzushi 10

A tool used by the sumo gem for performing instance setup and management in AWS

mgc/murmuration 6

A Songbird extension

pvh/automerge-demo 4

A simple demonstration of an Automerge-based web application.

pvh/custom-xul-view 4

A Songbird Media Page View that does tree-like things without using a XUL tree

issue commentautomerge/automerge

Sync goes into infinite loop?

Just chiming in to say that if there is a defect in the API design here it's that it might be reasonable for us to conclude that we could probably return null for repeated generateSyncMessage calls, thus:

[doc, state1] = Automerge.receiveSyncMessage(doc, state0, receivedMsg);
[state2, outboundMsg1] = Automerge.generateSyncMessage(doc, state1);
[state3, outboundMsg1] = Automerge.generateSyncMessage(doc, state2); // arguably, this could be null
[state2, outboundMsg1] = Automerge.generateSyncMessage(doc, state1); // this should not be null (no side effects)

I'm not entirely sure we want to do this. I agree that the correct behaviour is to only generateSyncMessage in response to a new connection, receiving a message, or making a change. Still, it seems like it is arguably unnecessary to generate the extra messages. On the other hand, the current behaviour gives nice recovery behaviour in environments with lossy communication channels so... I don't have a strong opinion about the right path forward (aside from improving the docs.)

grrrwaaa

comment created time in a month

issue openedautomerge/automerge

Identifying newly arrived changes to save during sync (API design oversight?)

A a sync message received from another peer may contain changes. We apply those changes to the Backend as part of receiveSyncMessage, but it's not clear how a user would know which new changes have arrived to save them incrementally.

This isn't a problem for users who call save() explicitly because that function grabs all the changes for the entire document. @okdistribute solved this problem with the _receive function seen here (lightly edited for clarity, added comment is mine):

  _receive(peer, syncMsg: BinarySyncMessage): Patch {
    let oldDoc = this.doc;
    let [newDoc, newSyncState, patch] = Backend.receiveSyncMessage(
      this.doc,
      peer.state,
      syncMsg
    );
    this.doc = newDoc;
    peer.state = newSyncState;
    this._peers.set(peer.id, peer);
    if (patch) {
      let changes = Backend.getChanges(newDoc, Backend.getHeads(oldDoc)); ## whoops, why do we need this?
      this._sendToRenderer(patch, changes);
    }
    this.updatePeers();
    return patch;
  }

As you can see, she's comparing the old and new backend head states to find out what changes have arrived and then forwards those on to save elsewhere. We already know what changes have arrived inside receiveSyncMessage, and could consider including those as another return value from that function. That function signature is already getting a bit unwieldy, though, so we may want to consider other alternatives.

created time in a month

issue commentautomerge/automerge

Sync goes into infinite loop?

It only knows that it's synced if it has heard from the other side, so if you don't hear from the other side it will continue to generate a "query".

Also, if you want to rummage around inside the SyncMessage, just use decodeSyncMessage. I think it's exported from Backend?

A small note that strictly none of the sync functions have side effects*. All inputs and outputs are explicit.

The sync protocol should behave correctly regardless of what order the peers initiate contact. Both sides can reach out simultaneously or one side can wait to hear from the other. (There are even tests for this.) Still, if you call "generateSyncMessage" without changing the document or hearing back from the other peer, you're going to get the same syncMessage to send again! This is, I think, correct behaviour, and supports retries in case of lost messages. I could probably be persuaded otherwise, but we wouldn't want to return null since that (as @dan-weaver says) indicates that the local node is sure that it has everything it needs and that its peer hasn't asked for anything else (aka: fully sync'd).

One other note, which is that the unified Frontend/Backend version of the sync protocol has an annoying behaviour which is that when it receives sync messages it's hard to tell whether the document has actually changed. In the case of split frontend/backend operation it's easier -- if you don't get a patch to emit, then there's nothing to render. Unfortunately, the merged version maintains some internal state which makes this harder. I don't really have a good solution for this yet, but practically speaking it leads to a bunch of wasted render() calls in React apps during synchronization. Sorry.

grrrwaaa

comment created time in a month

issue commentautomerge/automerge

Sync goes into infinite loop?

You do need different actor IDs though it probably shouldn't enter an infinite loop if you don't have them.

Your example code is a bit complicated but doesn't have anything obviously wrong going on, so the next step would be to figure out why the sync protocol believes it needs to send another message.

If you have a look at the sync code, calling generateSyncMessage will produce a message if there are local changes we know the other peer doesn't have, or if our heads and the other peers' heads don't match (indicating they have changes we need, still.) I've not got a chance to dig into what specifically is going wrong, but I would suspect it's something like:

  • your sync states are being lost / reset between messages, so peers think the other might still have something new for them
  • the messages aren't being applied correctly, so we keep repeating ourselves

On the whole though, it looks from casual inspection like it should work, but since it isn't, I hope this gives you some improved ability to debug the problem. In general I would say that a well designed API should make it very difficult to get into infinite loops in the first place, so we probably still have some work to do here, and once we figure out the problem I certainly welcome your input into how we can either improve the error messages, the API, or failing that, improve the documentation.

grrrwaaa

comment created time in a month

issue commentautomerge/automerge

Claims-based authorization

Yes, I think this is a very interesting line of reasoning to work down and I'm excited about the promise of the approach you're exploring here.

I do want to suggest that as a foundational concept we should always allow local editing of any information and make agreement optional (though convenient and by default).

Consider, in your example, what we would do if you lost your signing keys. Now the document's time would become immutable! There would be no way for us to agree to your new key. This would be no big deal, perhaps, for a short-lived document, but could be a big problem if the document was a shared calendar for a workplace.

The extremely shorthand way I describe this change in perspective is "attention, not permission". I can scribble all over my copy of the building codes, but the inspector won't care when she comes to check if I've got the wiring right. Still, it is useful and important for me to do just that, and it's hard to predict who will participate in a system in the future: perhaps I might want to work on a proposal for a new version of that document with some collaborators. My edits may (or may not) ever be accepted upstream but there's no reason I shouldn't be able to edit, annotate, or otherwise alter my version.

MeneDev

comment created time in 2 months

issue commentautomerge/automerge

Claims-based authorization

@MeneDev I think your approach is really interesting but there are a few problems to solve. First is, as @Steve-OH points out, that you might not need to actually change Automerge to do this, but can simply unbox the changes, check their signatures, and then decide whether to pass them into applyChanges if they pass muster.

I think more interestingly the question is what to do if you get a change that is no good. Right now, every change in Automerge depends on every change that was locally visible when the change was made, and each local node has only a single linear history. That means that if you reject a change from a source you won't be able to apply any further changes from it ever, nor will you be able to accept changes from people who do. That might not be a problem in your case -- perhaps quarantine / excision of misbehaving nodes is desirable? -- but it does limit the flexibility of the approach.

MeneDev

comment created time in 2 months

issue commentpvh/automerge-demo

Browser console error: "import call expects exactly one argument"

Yea, I'm surprised! It literally "worked for me" right out of GitHub on a fresh computer. I wonder what's the matter... I suspect a local configuration issue for you but it's not much to go on.

aaaidan

comment created time in 2 months

issue commentpvh/automerge-demo

Browser console error: "import call expects exactly one argument"

Seems like an issue with bundling. I suspect it's because you (really annoyingly) can't run ES6 modules from file://. Try using yarn dev instead.

aaaidan

comment created time in 2 months

issue commentautomerge/automerge

Demo code for [generate|recieve]SyncMessage

Just to orient you, the automerge-store and shared-worker are where the meat of things are. What you'll see is a system that communicates between several tabs using BroadcastChannel and each tab gets a Worker thread to run the CRDT in. This is attempting to simulate a P2P system and you'll really want to communicate via some kind of a server but I wanted to start with something self-contained as a reference.

Each change is written to localStorage as it happens and is compacted every 100 changes to reduce storage size. (That's DB.js's job.)

Although it's written in Svelte there's basically nothing svelte-specific here if you (as I expect) are using React you should find it straightforward to just lift the bits you need. If not, let me know and we can sort it out together. Good luck and don't hesitate to ask questions and/or muse out loud about confusing bits.

aaaidan

comment created time in 2 months

issue commentautomerge/automerge

Demo code for [generate|recieve]SyncMessage

Hi @aaaidan -- happy to help. Here's a link to some demo code that I'm working on. It's rather undocumented, so please ask any and every question that comes to mind and we'll put the answers into the README / improve the code as we go. Sound good?

aaaidan

comment created time in 2 months