profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/benbjohnson/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.

benbjohnson/clock 444

Clock is a small library for mocking time in Go.

benbjohnson/css 78

W3C-compliant CSS3 parser and scanner

benbjohnson/agency 57

A fast user agent string parser for Go.

benbjohnson/application-development-using-boltdb 23

Repository for my "Application Development Using BoltDB" talk

benbjohnson/burger-stack 20

Presentation for "The Burger Stack"

benbjohnson/bolt 11

A low-level key/value database for Go.

benbjohnson/boxer 10

A little app that boxes my time.

benbjohnson/bootstrap-ego 9

An ego template component library for Bootstrap 4.

benbjohnson/describe.today 6

A web site to describe today.

benbjohnson/authoritarian 2

Command line utility for authorizing Twitter users to an application.

push eventbenbjohnson/wtf

Kay S. Thiele

commit sha 50830c92f7f8072450a15e1d5c392989b004d64d

count runes instead of bytes len() counts the bytes in a string instead of runes, i.e 'len("ä")' returns 2 'utf8.RuneCountInString()' will return the actual number of runes/characters

view details

Ben Johnson

commit sha bcc6c727f98398d48fd8bfbf6e73343b18ff68c1

Merge pull request #51 from ksthiele/patch-1 count runes instead of bytes

view details

push time in 10 hours

PR merged benbjohnson/wtf

count runes instead of bytes

len() counts the bytes in a string instead of runes, i.e 'len("ä")' returns 2

'utf8.RuneCountInString()' will return the actual number of runes/characters

+2 -1

1 comment

1 changed file

ksthiele

pr closed time in 10 hours

pull request commentbenbjohnson/wtf

count runes instead of bytes

Good catch. Thanks!

ksthiele

comment created time in 10 hours

issue closedbenbjohnson/litestream

Can litestream be used for multi master replication?

Hi folks. Litestream looks great, nice work!

I'm wondering if it can be used for master master replication? For instance say I have two servers named A and B with each a local copy of the same sqlite database. Can A replicate any local change to B and B replicate any local change to A simultaneously?

I suppose this could work only if each replica makes sure all other replicas have applied a query before applying it locally. Otherwise replicas could start diverging, I think.

For instance if server A makes a local update based on a WHERE clause on the value of column C, but server B has changed the value of column C locally before the update from A is received. In that case the update from A doesn't work on B, whereas it did work on A.

Disclaimer: I know nothing about replication, just thinking out loud here.

closed time in 13 hours

DoobleD

issue commentbenbjohnson/litestream

Can litestream be used for multi master replication?

Hi @DoobleD, good question. Multi-master replication is generally a lot harder than primary-replica replication because you get into consistency issues and conflict resolution. You also have trade-offs with regards to performance if you're doing something like a two-phase commit. So, to answer your question, no, Litestream doesn't support multi-master replication. It's also not likely to be a goal of the project in the future.

DoobleD

comment created time in 13 hours

issue closedbenbjohnson/thesecretlivesofdata

May I use your visualization to teach students about raft consensus algorithms?

Hi Ben. I am preparing teaching materials as part of Samsung's effort to provide free to use education material. I came across your most excellent visualization tool and hope to use it as part of my curriculum. Unfortunatley, Samsung's policy prevents me from simply including links. This is because many of the countries where the educatiion material will be provided often have very poor or no internet connections. So, in order to use your visualization, I created a small video of the visualization. (Screen recording with me pushing the buttons).

Please let me know if I may use your materials in this manner, and if so, what type of copyright disclosure you would like me to present in the materials.

Thank you.

Henry Park

closed time in 7 days

yaksoo39

issue commentbenbjohnson/thesecretlivesofdata

May I use your visualization to teach students about raft consensus algorithms?

Hi @yaksoo39. Yes, the visualization is free to share. You can change it or do anything with it, really. You can add attribution but you don't need to. I hope your students enjoy it!

yaksoo39

comment created time in 7 days

PullRequestReviewEvent

push eventbenbjohnson/litestream

Ben Johnson

commit sha fad661c9d3eb520c93ad537f169f71c5d9f7ee57

HTTP streaming support

view details

push time in 13 days

delete branch benbjohnson/litestream

delete branch : stream

delete time in 13 days

create barnchbenbjohnson/litestream

branch : http

created branch time in 13 days

delete branch benbjohnson/litestream

delete branch : shadow-wal-segments

delete time in 13 days

push eventbenbjohnson/litestream

Ben Johnson

commit sha 77274abf81fc878fff4445c6306b9da3354b6960

Refactor shadow WAL to use segments

view details

push time in 14 days

delete branch benbjohnson/litestream

delete branch : db-wal-segments

delete time in 14 days

PR merged benbjohnson/litestream

Refactor shadow WAL to use segments

This pull request refactors the shadow WAL to store individual segments instead of simply appending on to WAL files. This provides the ability to transactionally store segments and to separate them along SQLite transaction boundaries. These characteristics are necessary to enable live replication so that the replicas do not need to compute transaction boundaries on every read.

This is a draft pull request and needs additional cleanup and testing.

Fixes #191

+1021 -832

0 comment

14 changed files

benbjohnson

pr closed time in 14 days

issue closedbenbjohnson/litestream

Store shadow WAL as a collection of compressed wal segments

Currently, the shadow WAL stores WAL files as full files. However, it would be better to store them as LZ4-compressed WAL segments that are delineated by transaction boundary. This will help when applying changes to live replicas (#8).

closed time in 14 days

benbjohnson

created tagbenbjohnson/litestream

tagv0.4.0-alpha.3

Streaming replication for SQLite.

created time in 14 days

release benbjohnson/litestream

v0.4.0-alpha.3

released time in 14 days

push eventbenbjohnson/litestream

Ben Johnson

commit sha 99fe882376e3f56b92d7a4fa81038ba2ad51dc05

Refactor shadow WAL to use segments

view details

push time in 14 days

push eventbenbjohnson/describe.today

Ben Johnson

commit sha 03adc87ae30f2f59160a350156427262634f7cb5

CNAME

view details

push time in 16 days

issue commentbenbjohnson/litestream

Missing initial WAL segment during restore

Thanks for taking the time to track that down.

I agree that having two databases writing to the same bucket is a problem but I'm not entirely sure how to detect it in a good way. One option would be to use a lock file on the bucket that essentially holds a lease. However, there's a trade-off with how often that lease needs to be renewed versus how quickly a node can be recovered. For example, if a node updates the timestamp of the file every minute to continue to hold the lease then there could be up to a minute where a node could not recover. There's also a small additional cost for those PUT requests.

benbjohnson

comment created time in 17 days

release benbjohnson/litestream

v0.4.0-alpha.2

released time in 19 days

created tagbenbjohnson/litestream

tagv0.4.0-alpha.2

Streaming replication for SQLite.

created time in 19 days

push eventbenbjohnson/litestream

Ben Johnson

commit sha 8ff784b34db45e1f2977878b00b3597432883d54

Refactor shadow WAL to use segments

view details

push time in 19 days

release benbjohnson/litestream

v0.4.0-alpha.1

released time in 20 days

created tagbenbjohnson/litestream

tagv0.4.0-alpha.1

Streaming replication for SQLite.

created time in 20 days

push eventbenbjohnson/litestream

Ben Johnson

commit sha 6eb290720b7b155960991d1117953e8ad35e7102

Refactor shadow WAL to use segments

view details

push time in 20 days

push eventbenbjohnson/litestream

Ben Johnson

commit sha 9e30408e6e4a2e5d28acdadc2a1676eaf107b50c

Refactor shadow WAL to use segments

view details

push time in 20 days

issue closedinfluxdata/influxdb

OSS 2.x TSM migration tool

What

Users of OSS 2.x alphas will have TSM data in an incompatible format once https://github.com/influxdata/influxdb/tree/sgc/tsm1 is merged.

To ease their transition to newer builds, a tool would be required to convert the existing TSM data back to the original TSM 1.x format. This tool should replace the existing implementation of the influxd migrate command.

See PR #17016 as a template for the original conversion tool, which sets expectations accordingly.

Line Protocol

It is possible for a user to export and re-import all their data as line protocol, but this approach will be intolerable for most data sets. For example, Edd noted he had 12MB of on-disk TSM data that expands to 7.5GB of line protocol.

closed time in 21 days

stuartcarnie

issue closedbenbjohnson/ego

v0.4.2 removes necessary whitespace

My HTML pages are breaking because the v0.4.2 release strips the whitespace between these ego tags:

<header class="row">
  <div class="col-5">
    <h3><%= t(req, "Batch") %> <%= st.Bid %></h3>
  </div>
</header>

The generated <h3> should contain Batch abc123 but contains Batchabc123. v0.4.1 works fine.

closed time in 23 days

mperham