profile
viewpoint

huachaohuang/awesome-dbdev 166

Awesome materials about database development.

huachaohuang/awesome-interviews 9

Awesome interviews that inspire me a lot.

huachaohuang/lockfree 2

Lockfree algorithms collection.

huachaohuang/awesome-docs 1

Awesome documentations

huachaohuang/arrow-rs 0

Official Rust implementation of Apache Arrow

huachaohuang/engula 0

Engula empowers engineers to build reliable and cost-effective databases.

huachaohuang/rocksdb 0

A library that provides an embeddable, persistent key-value store for fast storage.

issue commentengula/engula

A journal implementation based on local file system

@ztelur I think we can come back to this within two weeks.

huachaohuang

comment created time in an hour

PR opened engula/engula

kernel: an alternative implementation

This is a new implementation of Kernel, which leverages trait bounds.

I am going to use this in https://github.com/engula/engula/pull/140.

Ref: https://github.com/engula/engula/pull/136

+673 -893

0 comment

28 changed files

pr created time in an hour

create barnchhuachaohuang/engula

branch : kernel-2

created branch time in an hour

push eventhuachaohuang/engula

Richard

commit sha cc195c85a21ab0d2706999875ce4aea75b075aae

Add table builder and reader

view details

push time in 7 hours

Pull request review commentengula/engula

engine: add a memory hash engine

+// Copyright 2021 The Engula Authors.+//+// Licensed under the Apache License, Version 2.0 (the "License");+// you may not use this file except in compliance with the License.+// You may obtain a copy of the License at+//+// http://www.apache.org/licenses/LICENSE-2.0+//+// Unless required by applicable law or agreed to in writing, software+// distributed under the License is distributed on an "AS IS" BASIS,+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+// See the License for the specific language governing permissions and+// limitations under the License.++use std::{+    collections::BTreeMap,+    sync::atomic::{AtomicUsize, Ordering},+};++use tokio::sync::Mutex;++pub struct Memtable {+    map: Mutex<BTreeMap<Vec<u8>, Vec<u8>>>,+    size: AtomicUsize,+}++impl Memtable {+    pub fn new() -> Self {+        Self {+            map: Mutex::new(BTreeMap::new()),+            size: AtomicUsize::new(0),+        }+    }++    pub async fn get<K: AsRef<[u8]>>(&self, key: K) -> Option<Vec<u8>> {+        let map = self.map.lock().await;+        map.get(key.as_ref()).cloned()+    }++    pub async fn set<K: Into<Vec<u8>>, V: Into<Vec<u8>>>(&self, key: K, value: V) {+        let key = key.into();

Oh, I haven't considered MVCC at this moment. So user keys and internal keys are the same things for now.

huachaohuang

comment created time in 7 hours

PullRequestReviewEvent

PR opened engula/engula

engine: add a memory hash engine

Enough for abstractions, let's build our first engine now. This PR only handles some memory data. We will offer the hash engine an Env later to persist its state. I also remove the API part to avoid confusion for now.

+160 -176

0 comment

12 changed files

pr created time in 8 hours

create barnchhuachaohuang/engula

branch : engine

created branch time in 8 hours

issue commentengula/engula

A journal implementation based on local file system

@ztelur Your design looks fine to me. But the codebase is facing a massive change recently and I am focusing on designing other things. So I think I need some time to clean things up before coming back to this.

huachaohuang

comment created time in 9 hours

push eventhuachaohuang/engula

Richard

commit sha 775d8eb6508270696eb4e93a33273991a18176f7

Remove generic timestamp

view details

Richard

commit sha 4783ca272ce92d9f8310c50862055789008d2384

Remove generic timestamp

view details

Richard

commit sha ba8e26acd7240ae9fe10dfd0977fda5b597d0ae4

Add Send + Sync

view details

Richard

commit sha 9b2c6391e2eb8913ea436f3f5925f8546320672b

Add version and version updates

view details

push time in a day

Pull request review commentengula/engula

journal: refactor with trait objects for composition

 // See the License for the specific language governing permissions and // limitations under the License. -use super::{async_trait, stream::Stream};+use crate::{async_trait, Result, Stream};  /// An interface to manipulate a journal. #[async_trait]-pub trait Journal<S: Stream> {+pub trait Journal: Send + Sync {     /// Returns a stream.-    async fn stream(&self, name: &str) -> Result<S, S::Error>;

Yes, we can only expose a few common errors that make sense to users. I am going in that direction now, you can see how the kernel handle errors here

huachaohuang

comment created time in a day

PullRequestReviewEvent

push eventhuachaohuang/engula

Richard

commit sha ec7520bbe6714bf60ce00d71edf4587d022ee411

Add Send + Sync

view details

push time in a day

push eventhuachaohuang/engula

Richard

commit sha 62d5c896f8d6411a1264e82b9cf160b8766704b2

Remove generic timestamp

view details

push time in a day

push eventhuachaohuang/engula

Richard

commit sha f881c8e0760579c7a515d59e77953276f77e86c2

Remove generic timestamp

view details

push time in a day

pull request commentengula/engula

journal: specialize Timestamp

I will review this PR when I settle down things in https://github.com/engula/engula/pull/138.

tisonkun

comment created time in a day

pull request commentengula/engula

journal: specialize Timestamp

@tisonkun Thanks, I am thinking about a similar change. u64 is the most reasonable choice for now. And if that turns out to be not sufficient, we can simply use a bytewise comparable Vec<u8>. It is not hard to convert a u64 to Vec<u8>, for example, u64.to_be_bytes().

tisonkun

comment created time in a day

pull request commentengula/engula

kernel: add some basic structs

@tisonkun I can see two reasonable implementation so far: local and remote. The local implementation simply talks to the underlying journal/storage to do its job. The remote implementation talks to a remote kernel service that manages other things for it.

huachaohuang

comment created time in a day

pull request commentengula/engula

storage: refactor with trait objects to allow composition

@tisonkun Yeah, maybe just keep these PRs open in case someone interested in what we are doing.

huachaohuang

comment created time in 2 days

pull request commentengula/engula

journal: refactor with trait objects for composition

The generic timestamp draws me crazy in https://github.com/engula/engula/pull/136. We should consider removing it. Maybe users can embed their timestamps in the data somehow when necessary.

huachaohuang

comment created time in 2 days

push eventhuachaohuang/engula

Richard

commit sha 4f5c1878c693e31da327a6aaad7f685c53db7bda

journal: refactor with trait objects for composition

view details

Richard

commit sha 4915589b8f98e568000a529e11360f05d5ee4018

storage: refactor with trait objects to allow composition Upper-level modules like Kernel/Engine need to create a storage dynamically without knowing the specific type. Also update toolchain to address [this issue](https://github.com/rust-analyzer/rust-analyzer/issues/10772).

view details

Richard

commit sha 9a37e0d11b04debf7f4143445b82275ad1919dc7

Use journal/storage trait objects

view details

push time in 2 days

PR opened engula/engula

journal: refactor with trait objects for composition

Check https://github.com/engula/engula/pull/137 for more details.

+63 -114

0 comment

9 changed files

pr created time in 2 days

create barnchhuachaohuang/engula

branch : journal

created branch time in 2 days

Pull request review commentengula/engula

docs: rewrite the design document

  This document describes the top-level design of Engula. -This document is evolving from [the previous design document][demo-1-url]. You can check that document for more details before this document is thorough.+**The current design is still in progress.** You can also check [the previous design][demo-design] for more information. -[demo-1-url]: https://github.com/engula/engula/blob/demo-1/docs/design.md+[demo-design]: https://github.com/engula/engula/blob/demo-1/docs/design.md -# Data Model+# Overview -An Engula deployment is called a universe. A universe contains multiple databases. A database contains multiple collections.+Engula is a serverless storage engine that empowers engineers to build reliable and cost-effective databases. -A `Collection` represents a collection of data in a database. Applications use data APIs to manipulate data in collections. Engula provides some built-in data APIs for convenience. Applications can also build their data APIs outside of Engula.+Engula's design goals are as follows:++- Elastic: takes advantage of elastic resources on the cloud+- Adaptive: adapts to dynamic workloads and diverse applications+- Extensible: provides pluggable APIs and modules for customization+- Platform independent: allows flexible deployments on local hosts, on-premise servers, and cloud platforms++It is important to note that Engula is not a full-functional database. Engula is more like a framework that allows users to build their databases. However, for users that don't need customization, Engula can also be used as an out-of-the-box data service for typical applications.  # Architecture  ![Architecture](images/architecture.drawio.svg) -Engula employs *an unbundled architecture*. Engula unbundles the storage engine into different components. Each component can be run as a local component or a remote component. Local components are embedded in applications, which work like other embedded storage engines. Remote components are standalone services that interact with applications through RPC. Each remote component is served by a group of isolated resource units provisioned from a resource pool.+Engula employs *a modular and serverless architecture*.++Engula unbundles the storage engine into the following modules:++- **API** provides stateless data API services. For example, KV, SQL, or GraphQL.+- **Kernel** provides the essential storage capabilities to implement upper-level APIs.

Yeah, give me some time to prototype it before adding more precise descriptions.

huachaohuang

comment created time in 2 days

PullRequestReviewEvent

PR opened engula/engula

storage: refactor with trait objects to allow composition

Upper-level modules like Kernel/Engine need to create a storage dynamically without knowing the specific type. Check this for an example.

Also update toolchain to address this issue.

I am currently focusing on prototyping the [kernel[(https://github.com/engula/engula/pull/136). So I opt out of some implementations to avoid fixing all of them now.

+78 -551

0 comment

13 changed files

pr created time in 2 days

create barnchhuachaohuang/engula

branch : storage

created branch time in 2 days

push eventhuachaohuang/engula

Richard

commit sha d568ab159792a60c27ef197fb0d91d90e8dbe7ae

Separate Kernal and Engine

view details

push time in 2 days

PR opened engula/engula

kernel: add some basic structs

WIP, send a PR first in case someone wants to know what Kernel looks like :)

+138 -0

0 comment

6 changed files

pr created time in 3 days

create barnchhuachaohuang/engula

branch : kernel

created branch time in 3 days

more