profile
viewpoint
Eric Hwang ericyhwang Lever San Francisco Bay Area

ericyhwang/DefinitelyTyped 0

The repository for high quality TypeScript type definitions.

ericyhwang/eric-ts-sandbox 0

Place for me to experiment with TypeScript

ericyhwang/TypeScript 0

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

lever/connect-multiparty 0

connect middleware for multiparty

lever/node-browserchannel 0

An implementation of a google browserchannel server in node.js

lever/sharedb-logger 0

Logger for pretty-printing ShareDB message streams

pull request commentshare/sharedb

⚡️ Remove server-side Presence transformations

  1. Yeah, CPU profiler data from Node would definitely be the most helpful, as that would let us identify exactly what code is hot.

Some other things that could also help: 2) On a client, you can set connection.debug = true to log out all the client's messages to/from the server. That can help identify if there's some fan-out caused by the client code. 3) Similarly, on the server, backend.use('receive', (context) => console.log('RECV', context.data)) and backend.use('reply', console.log('SEND', context.reply)) will print out the messages the server is receiving/sending.

alecgibson

comment created time in 17 hours

PullRequestReviewEvent
PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentshare/sharedb

🐛 Fix issue with null timestamp in fetchSnapshotByTimestamp

 Backend.prototype.fetchSnapshotByTimestamp = function(agent, index, id, timestam     id: id,     timestamp: timestamp   };+  timestamp = timestamp ? timestamp : Date.now();

Namely, if I pass null to getMilestoneSnapshotAtOrAfterTime() should it actually return the live snapshot (rather than the most recent milestone?).

As discussed, the milestone snapshot adapter is separate from the normal db adapter - the Mongo version uses a different collection - so we can't return the latest actual snapshot in this case.

Alternatively we could insert some short-circuit logic here for a performance gain: maybe now is finally the time to just check if we're asking for a null timestamp or version, and just directly call db.getSnapshot()?

Yeah, makes sense, for both fetchSnapshot by version and timestamp

dawidreedsy

comment created time in 18 hours

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentshare/sharedb

🐛 Prevent local presence transformation when presence created late

 LocalDocPresence.prototype._registerWithDoc = function() {  LocalDocPresence.prototype._transformAgainstOp = function(op, source) {   var presence = this;+  var docDataVersion = this._doc._dataStateVersion;+   this._pendingMessages.forEach(function(message) {+    var messageDocDataVersion = presence._docDataVersionByPresenceVersion[message.pv];

Would be nice to add a brief comment about what case this is trying to handle.

alecgibson

comment created time in 14 days

PullRequestReviewEvent
PullRequestReviewEvent
PullRequestReviewEvent

issue commentshare/sharedb-mongo

Applying ops under load

Based on that, it currently seems like the slowness is due to Mongo time and not Node time, so it's probably best to look further at Mongo for now instead of trying to get Node profiling working in Kubernetes.

The first slow part of the slow log has a bunch of o_NOTES_EDITOR_SHAREDB queries. One thing to check - Is there an appropriate index { d: 1, v: 1 } on o_NOTES_EDITOR_SHAREDB? If not, those would take a long time in Mongo and could slow down other queries.

Later on, there are also multiple queries against the NOTES_EDITOR_SHAREDB snapshot collection, which should be fast since they're id lookups. But if there are enough happening at once, that could slow things down. Or it could be residual processing on the op queries.

yaroslavputria

comment created time in 20 days

issue commentshare/sharedb-mongo

Applying ops under load

Yeah, thanks for providing the detailed timing logs, those are illustrative!

On the first attempt, the two long delays are:

### beforeSnapshotLookup middlewate 0
### snapshot is fetched 0.851
...
### OP collection is get 0.851
### ops are fetched 1.774

Which are indeed both in sharedb-mongo + Mongo.

As Alec mentions, the slowness could be coming from:

  • MongoDB itself being slow at fulfilling the read requests for the snapshot and ops
  • Or the Node process may be blocked on something, perhaps on processing other requests

It would be interesting to take a performance profile of, say, 2 Node processes handling requests from 20 clients, to see if the bottleneck is Node process CPU, time spent waiting on Mongo, or something else entirely.

With just 2 Node processes, it should be feasible to run them both with --inspect, attach Chrome Dev Tools to both, and run the Performance profiler on both while running the load test. Or use your other favorite profiling tool.

yaroslavputria

comment created time in 21 days

Pull request review commentottypes/json0

✨ Add presence support

 json.transformComponent = function(dest, c, otherC, type) {   return dest; }; +json.transformPresence = function(presence, op, isOwnOp) {+  if (!presence || !isArray(presence.p)) return null;+  if (!op) return presence;++  presence = clone(presence);+  op = clone(op);++  // Deletions should be treated as insertions: transforming+  // by them should result in a no-op (null presence), so+  // let's just pretend that deletions are oi to get the+  // desired transform behaviour+  for (var i = 0; i < op.length; i++) {+    const component = op[i];+    if ('od' in component) {+      component.oi = component.od;+      delete component.od;+    }++    if ('ld' in component) {

I'm curious if this breaks the json.transform(transformed, [component], 'right') lower down, if you, say, have presence on index 4 of an array and do a ld on index 2.

alecgibson

comment created time in 21 days

Pull request review commentottypes/json0

✨ Add presence support

 offset in a string. `TEXT` must be contained at the location specified.  --- +## Presence++`json0` has some limited support for presence information: information about+clients' transient position within a document (eg their cursor or selection).++It also supports presence in `text0`.++### Format++#### `json0`++The format of a `json0` presence object follows a similar syntax to its ops:++    [{p: ['key', 123], v: 0}]

From the implementation, this shouldn't have an array wrapping it?

alecgibson

comment created time in 21 days

Pull request review commentottypes/json0

✨ Add presence support

 json.transformComponent = function(dest, c, otherC, type) {   return dest; }; +json.transformPresence = function(presence, op, isOwnOp) {+  if (!presence || !isArray(presence.p)) return null;+  if (!op) return presence;++  presence = clone(presence);+  op = clone(op);++  // Deletions should be treated as insertions: transforming+  // by them should result in a no-op (null presence), so+  // let's just pretend that deletions are oi to get the+  // desired transform behaviour+  for (var i = 0; i < op.length; i++) {+    const component = op[i];+    if ('od' in component) {+      component.oi = component.od;+      delete component.od;+    }++    if ('ld' in component) {+      component.oi = component.ld;+      delete component.ld;+    }++    // Handle text0 ops using the subtype+    if ('si' in component || 'sd' in component) {+      convertFromText(component);+    }+  }++  // Create a fake op so we can transform the presence path using+  // existing machinery+  var transformed = [{p: presence.p, oi: ''}];++  for (var i = 0; i < op.length; i++) {+    var component = op[i];+    // Set side as 'right' because we always want the op to win ties, since+    // our transformed "op" isn't really an op

Would be worth mentioning that for json0, this is just to handle subpath changes as a result of list move, insert, delete.

alecgibson

comment created time in 21 days

Pull request review commentottypes/json0

✨ Add presence support

 json.transformComponent = function(dest, c, otherC, type) {   return dest; }; +json.transformPresence = function(presence, op, isOwnOp) {+  if (!presence || !isArray(presence.p)) return null;+  if (!op) return presence;++  presence = clone(presence);+  op = clone(op);++  // Deletions should be treated as insertions: transforming+  // by them should result in a no-op (null presence), so+  // let's just pretend that deletions are oi to get the

The "pretend that deletions are oi" comment is confusing on first read, would be worth elaborating a bit more on how that works

alecgibson

comment created time in 21 days

PullRequestReviewEvent
PullRequestReviewEvent
PullRequestReviewEvent

release share/sharedb

v2.2.1

released time in 2 months

created tagshare/sharedb

tagv2.2.1

Realtime database backend based on Operational Transformation (OT)

created time in 2 months

push eventshare/sharedb

Eric Hwang

commit sha a1a33852531432baeb3cb948ca36500bdbc60fa4

2.2.1

view details

push time in 2 months

delete branch share/sharedb

delete branch : reduce-sendPresence-deprecation

delete time in 2 months

push eventshare/sharedb

Eric Hwang

commit sha 41d13c3588d92f210c430697bceb727b12c9a918

Only display sendPresence error deprecation if presence is enabled

view details

Eric Hwang

commit sha 570509911cf41e2012c7a3710a0c7c372777e5f5

Merge pull request #525 from share/reduce-sendPresence-deprecation Only display sendPresence error deprecation if presence is enabled

view details

push time in 2 months

PR merged share/sharedb

Only display sendPresence error deprecation if presence is enabled

https://github.com/share/sharedb/pull/524 added a deprecation notice about presence error handling for all constructed backends without a new opt-in flag:

Broadcasting "sendPresence" middleware errors to clients is deprecated and will be removed in a future release. Disable this behaviour with: new Backend({doNotForwardSendPresenceErrorsToClient: true})

Since the warning is irrelevant for backends with the default of no presence, this PR updates the notice to only display for backends that were constructed with the option presence: true.

+1 -1

2 comments

1 changed file

ericyhwang

pr closed time in 2 months

PR opened share/sharedb

Reviewers
Only display sendPresence error deprecation if presence is enabled

https://github.com/share/sharedb/pull/524 added a deprecation notice about presence error handling for all constructed backends without a new opt-in flag:

Broadcasting "sendPresence" middleware errors to clients is deprecated and will be removed in a future release. Disable this behaviour with: new Backend({doNotForwardSendPresenceErrorsToClient: true})

Since the warning is irrelevant for backends with the default of no presence, this PR updates the notice to only display for backends that were constructed with the option presence: true.

+1 -1

0 comment

1 changed file

pr created time in 2 months

create barnchshare/sharedb

branch : reduce-sendPresence-deprecation

created branch time in 2 months

Pull request review commentDefinitelyTyped/DefinitelyTyped

[sharedb] Add `receivePresence` middleware hook

 const backend = new ShareDB({     milestoneDb: new MyMilestoneDB(),     suppressPublish: false,     maxSubmitRetries: 3,+    errorHandler: (error) => {

❗ The 2-arg handler should be exercised here, so that the test will fail if there's a breaking change to the error context.

alecgibson

comment created time in 2 months

Pull request review commentDefinitelyTyped/DefinitelyTyped

[sharedb] Add `receivePresence` middleware hook

 declare class sharedb extends EventEmitter {         milestoneDb?: sharedb.MilestoneDB,         suppressPublish?: boolean,         maxSubmitRetries?: number,+        doNotForwardSendPresenceErrorsToClient?: boolean,+        errorHandler?: ErrorHandler;

ℹ️ Note for future readers: This was just added to sharedb itself in https://github.com/share/sharedb/pull/524/files

alecgibson

comment created time in 2 months

more