profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/wojas/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.

wojas/docker-mac-network 183

Access your Docker for Mac's internal networks from your macOS host machine

PowerDNS/pdns-builder 9

Infrastructure for creating Dockerfiles for package building

wojas/envy 8

Shell helper that automatically sets and unsets environment variables

wojas/account_bankimport 4

OpenERP account_bankimport fork with better MT940 support

wojas/django-pgrunner 4

Create and run an independent local PostgreSQL database for your Django project

wojas/genericr 3

Go generic implementation of the logr interface

hackerbeerstest/hackerbeerstest 1

Yep, this is only for testing purposes. No fancy code. Sorry.

mpdaugherty/python-github3 1

Python wrapper for GitHub API v3

fork wojas/redigo

Go client for Redis

fork in 11 days

PR opened mna/redisc

testserver: use testing.TB interface

Changes the testserver to use the testing.TB interface instead of *testing.T to allow use in benchmarks.

This should not cause any issues for existing users of this package.

+15 -15

0 comment

1 changed file

pr created time in 12 days

create barnchwojas/redisc

branch : testserver-use-testing-tb

created branch time in 12 days

fork wojas/redisc

A Go redis cluster client built on top of redigo.

fork in 12 days

Pull request review commentPowerDNS/pdns

dnsdist: Add support for Lua per-thread FFI rules and actions

 class LuaFFIAction: public DNSAction   func_t d_func; }; +class LuaFFIPerThreadAction: public DNSAction+{+public:+  typedef std::function<int(dnsdist_ffi_dnsquestion_t* dq)> func_t;++  LuaFFIPerThreadAction(const std::string& code): d_functionCode(code), d_functionID(s_functionsCounter++)+  {+  }++  DNSAction::Action operator()(DNSQuestion* dq, std::string* ruleresult) const override+  {+    try {+      auto& state = t_perThreadStates[d_functionID];+      if (!state.d_initialized) {+        setupLuaFFIPerThreadContext(state.d_luaContext);+        state.d_func = state.d_luaContext.executeCode<func_t>(d_functionCode);+        state.d_initialized = true;+      }++      dnsdist_ffi_dnsquestion_t dqffi(dq);+      auto ret = state.d_func(&dqffi);+      if (ruleresult) {+        if (dqffi.result) {+          *ruleresult = *dqffi.result;+        }+        else {+          // default to empty string+          ruleresult->clear();+        }+      }+      return static_cast<DNSAction::Action>(ret);+    } catch (const std::exception &e) {+      warnlog("LuaFFIPerThreadAction failed inside Lua, returning ServFail: %s", e.what());+    } catch (...) {+      warnlog("LuaFFIPerthreadAction failed inside Lua, returning ServFail: [unknown exception]");+    }+    return DNSAction::Action::ServFail;+  }++  string toString() const override+  {+    return "Lua FFI per-thread script";+  }++private:+  struct PerThreadState+  {+    LuaContext d_luaContext;+    func_t d_func;+    bool d_initialized{false};+  };+  static uint64_t s_functionsCounter;

Great, then everything should be fine. Thanks!

rgacogne

comment created time in a month

PullRequestReviewEvent

Pull request review commentPowerDNS/pdns

dnsdist: Add support for Lua per-thread FFI rules and actions

 class LuaFFIAction: public DNSAction   func_t d_func; }; +class LuaFFIPerThreadAction: public DNSAction+{+public:+  typedef std::function<int(dnsdist_ffi_dnsquestion_t* dq)> func_t;++  LuaFFIPerThreadAction(const std::string& code): d_functionCode(code), d_functionID(s_functionsCounter++)+  {+  }++  DNSAction::Action operator()(DNSQuestion* dq, std::string* ruleresult) const override+  {+    try {+      auto& state = t_perThreadStates[d_functionID];+      if (!state.d_initialized) {+        setupLuaFFIPerThreadContext(state.d_luaContext);+        state.d_func = state.d_luaContext.executeCode<func_t>(d_functionCode);+        state.d_initialized = true;+      }++      dnsdist_ffi_dnsquestion_t dqffi(dq);+      auto ret = state.d_func(&dqffi);+      if (ruleresult) {+        if (dqffi.result) {+          *ruleresult = *dqffi.result;+        }+        else {+          // default to empty string+          ruleresult->clear();+        }+      }+      return static_cast<DNSAction::Action>(ret);+    } catch (const std::exception &e) {+      warnlog("LuaFFIPerThreadAction failed inside Lua, returning ServFail: %s", e.what());+    } catch (...) {+      warnlog("LuaFFIPerthreadAction failed inside Lua, returning ServFail: [unknown exception]");+    }+    return DNSAction::Action::ServFail;+  }++  string toString() const override+  {+    return "Lua FFI per-thread script";+  }++private:+  struct PerThreadState+  {+    LuaContext d_luaContext;+    func_t d_func;+    bool d_initialized{false};+  };+  static uint64_t s_functionsCounter;

Assuming the Lua state is also destroyed, I wonder if this would also allow Lua to garbage collect all the objects, in which case there should be no issue.

rgacogne

comment created time in a month

PullRequestReviewEvent

Pull request review commentPowerDNS/pdns

dnsdist: Add support for Lua per-thread FFI rules and actions

 class LuaFFIAction: public DNSAction   func_t d_func; }; +class LuaFFIPerThreadAction: public DNSAction+{+public:+  typedef std::function<int(dnsdist_ffi_dnsquestion_t* dq)> func_t;++  LuaFFIPerThreadAction(const std::string& code): d_functionCode(code), d_functionID(s_functionsCounter++)+  {+  }++  DNSAction::Action operator()(DNSQuestion* dq, std::string* ruleresult) const override+  {+    try {+      auto& state = t_perThreadStates[d_functionID];+      if (!state.d_initialized) {+        setupLuaFFIPerThreadContext(state.d_luaContext);+        state.d_func = state.d_luaContext.executeCode<func_t>(d_functionCode);+        state.d_initialized = true;+      }++      dnsdist_ffi_dnsquestion_t dqffi(dq);+      auto ret = state.d_func(&dqffi);+      if (ruleresult) {+        if (dqffi.result) {+          *ruleresult = *dqffi.result;+        }+        else {+          // default to empty string+          ruleresult->clear();+        }+      }+      return static_cast<DNSAction::Action>(ret);+    } catch (const std::exception &e) {+      warnlog("LuaFFIPerThreadAction failed inside Lua, returning ServFail: %s", e.what());+    } catch (...) {+      warnlog("LuaFFIPerthreadAction failed inside Lua, returning ServFail: [unknown exception]");+    }+    return DNSAction::Action::ServFail;+  }++  string toString() const override+  {+    return "Lua FFI per-thread script";+  }++private:+  struct PerThreadState+  {+    LuaContext d_luaContext;+    func_t d_func;+    bool d_initialized{false};+  };+  static uint64_t s_functionsCounter;

Are threads ever removed? If they are, one good solution would be to push the Lua state to a pool for later reuse by a new thread.

My main concern is that every state may open a new LMDB env with a large mapsize.

rgacogne

comment created time in a month

PullRequestReviewEvent

Pull request review commentPowerDNS/pdns

dnsdist: Add support for Lua per-thread FFI rules and actions

 class LuaFFIAction: public DNSAction   func_t d_func; }; +class LuaFFIPerThreadAction: public DNSAction+{+public:+  typedef std::function<int(dnsdist_ffi_dnsquestion_t* dq)> func_t;++  LuaFFIPerThreadAction(const std::string& code): d_functionCode(code), d_functionID(s_functionsCounter++)+  {+  }++  DNSAction::Action operator()(DNSQuestion* dq, std::string* ruleresult) const override+  {+    try {+      auto& state = t_perThreadStates[d_functionID];+      if (!state.d_initialized) {+        setupLuaFFIPerThreadContext(state.d_luaContext);+        state.d_func = state.d_luaContext.executeCode<func_t>(d_functionCode);+        state.d_initialized = true;+      }++      dnsdist_ffi_dnsquestion_t dqffi(dq);+      auto ret = state.d_func(&dqffi);+      if (ruleresult) {+        if (dqffi.result) {+          *ruleresult = *dqffi.result;+        }+        else {+          // default to empty string+          ruleresult->clear();+        }+      }+      return static_cast<DNSAction::Action>(ret);+    } catch (const std::exception &e) {+      warnlog("LuaFFIPerThreadAction failed inside Lua, returning ServFail: %s", e.what());+    } catch (...) {+      warnlog("LuaFFIPerthreadAction failed inside Lua, returning ServFail: [unknown exception]");+    }+    return DNSAction::Action::ServFail;+  }++  string toString() const override+  {+    return "Lua FFI per-thread script";+  }++private:+  struct PerThreadState+  {+    LuaContext d_luaContext;+    func_t d_func;+    bool d_initialized{false};+  };+  static uint64_t s_functionsCounter;

Related question: what is the life cycle of the created Lua states? Do I understand correctly that one is created globally for every thread and never destroyed? The reason I am asking is that we may need a way to do cleanup if they get destroyed frequently.

rgacogne

comment created time in a month

PullRequestReviewEvent

pull request commentPowerDNS/pdns

Rec: cumulative and Prometheus friendly histograms

Before I document this, I would like confirmation that the current output (see above) is as Prometheus likes. @wojas, can you confirm?

Note that for dnsdist we do some basic checking using promtool check metrics 1.

I just ran the example in the PR description through promtool check metrics after adding the following comments to them to tell promtool that these are histograms:

# HELP pdns_recursor_cumul_answers_seconds Histogram of stuff
# TYPE pdns_recursor_cumul_answers_seconds histogram
# HELP pdns_recursor_cumul_auth4answers_seconds Histogram of stuff
# TYPE pdns_recursor_cumul_auth4answers_seconds histogram
# HELP pdns_recursor_cumul_auth6answers_seconds Histogram of stuff
# TYPE pdns_recursor_cumul_auth6answers_seconds histogram

These comments are not required and I have yet to see any tool that actually depends on them, but this allowed promtool to actually check it as a histogram. I did notice that promtool does not do very strict validation.

omoerbeek

comment created time in a month

push eventPowerDNS/pdns-builder

Konrad Wojas

commit sha 008993828b290c54ece190ea12e82d32520db576

Reproducible RPM builds Add support for reproducible builds in RHEL 8+ and derived distributions: - Pass a SOURCE_DATE_EPOCH build arg to Docker. - Set the defines needed for reproducible RPMs. - Add BUILDER_SOURCE_DATE_FROM_SPEC_MTIME to enable the spec mtime as the SOURCE_DATE_EPOCH for vendor builds. - Test reproducible builds in CI.

view details

wojas

commit sha 67f5d25ae6872bae6988437e35bb8a34aa4f160b

Update usage text for -e

view details

wojas

commit sha b7d12ad7f67b058faf11d322ccdf510da9f9d5eb

Merge pull request #48 from PowerDNS/source-date-epoch Reproducible RPM builds

view details

push time in a month

PR merged PowerDNS/pdns-builder

Reproducible RPM builds

Reproducible RPM builds and SOURCE_DATE_EPOCH env var.

Note that this only works for RHEL 8+ and derived distributions, as the RPM version in RHEL 7 does not support reproducible builds.

Progress

  • [x] Pass SOURCE_DATE_EPOCH env var as a build arg (up to the Dockerfiles to explicitly use it)
  • [x] Set SOURCE_DATE_EPOCH to last commit time if not set
  • [x] Define magic variables to kindly ask rpmbuild for reproducible RPM builds
    • [ ] Discuss: is it OK to always set these? Perhaps only if SOURCE_DATE_EPOCH is set?
  • [x] Example of args to pass to tar in demo
  • [x] Add centos-8 as demo target
  • [x] Upgrade alpine
  • [x] Also make it work for the vendor RPM without breaking the Docker cache layers by using the last commit time
  • [x] CI test reproducible RPM build using centos-8 target (does not work with centos-7)
  • [x] Add a few words on this to the README
  • [x] Cleanup PR when everything works
+126 -7

1 comment

9 changed files

wojas

pr closed time in a month

PullRequestReviewEvent

push eventPowerDNS/pdns-builder

wojas

commit sha 67f5d25ae6872bae6988437e35bb8a34aa4f160b

Update usage text for -e

view details

push time in a month

Pull request review commentPowerDNS/pdns-builder

Reproducible RPM builds

 usage() {     echo "  -V VERSION      - Override version (default: run gen-version)"     echo "  -R RELEASE      - Override release tag (default: '1pdns', do not include %{dist} here)"     echo "  -m MODULES      - Build only specific components (comma separated; warning: this disables install tests)"-    echo "  -e EPOCH        - Set a specific Epoch for packages"+    echo "  -e EPOCH        - Set a specific Epoch for RPM packages"

Like this?

    echo "  -e EPOCH        - Set a specific version Epoch for RPM/DEB packages"
wojas

comment created time in a month

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentPowerDNS/pdns-builder

Reproducible RPM builds

 usage() {     echo "  -V VERSION      - Override version (default: run gen-version)"     echo "  -R RELEASE      - Override release tag (default: '1pdns', do not include %{dist} here)"     echo "  -m MODULES      - Build only specific components (comma separated; warning: this disables install tests)"-    echo "  -e EPOCH        - Set a specific Epoch for packages"+    echo "  -e EPOCH        - Set a specific Epoch for RPM packages"

OK, I will change it. I just wanted to make clear it has nothing to do with SOURCE_DATE_EPOCH.

How about something like this?

    echo "  -e EPOCH        - Set a specific Epoch for RPM/DEB packages, to reset the package versioning"
wojas

comment created time in a month

pull request commentPowerDNS/pdns-builder

Reproducible RPM builds

Updated the readme and squashed the commits. No further changes planned.

wojas

comment created time in a month

push eventPowerDNS/pdns-builder

Konrad Wojas

commit sha 008993828b290c54ece190ea12e82d32520db576

Reproducible RPM builds Add support for reproducible builds in RHEL 8+ and derived distributions: - Pass a SOURCE_DATE_EPOCH build arg to Docker. - Set the defines needed for reproducible RPMs. - Add BUILDER_SOURCE_DATE_FROM_SPEC_MTIME to enable the spec mtime as the SOURCE_DATE_EPOCH for vendor builds. - Test reproducible builds in CI.

view details

push time in a month

push eventPowerDNS/pdns-builder

Konrad Wojas

commit sha df00d2328b2437d6cde12d7161fdeb1b964f8dcb

Remove touch again, changes version

view details

push time in a month

push eventPowerDNS/pdns-builder

Konrad Wojas

commit sha aecb0d11d27d60c2503506d5f31ade38121f89a6

Tests: invalidate build context just in case

view details

push time in a month

Pull request review commentPowerDNS/pdns-builder

[WIP] Reproducible RPM builds

 for spec in "${specs[@]}"; do                  # Build the rpm and record which files are new         rpm_file_list > /tmp/rpms-before-        rpmbuild --define "_sdistdir /sdist" -ba "$spec"+        rpmbuild \+            --define "_sdistdir /sdist" \+            --define "_buildhost reproducible" \+            --define "source_date_epoch_from_changelog Y" \

The answer is here: https://fossies.org/linux/rpm/build/build.c#l_298

  298     if (rpmExpandNumeric("%{?source_date_epoch_from_changelog}") &&
  299     getenv("SOURCE_DATE_EPOCH") == NULL) {
  300     /* Use date of first (== latest) changelog entry */

The SOURCE_DATE_EPOCH env var always overrides the changelog timestamp. This is not what I expected, but still safe to keep in the script.

wojas

comment created time in a month

PullRequestReviewEvent

push eventPowerDNS/pdns-builder

Konrad Wojas

commit sha 6455890ab182b0dc8740a808ce284be4d90d42b1

Add BUILDER_SOURCE_DATE_FROM_SPEC_MTIME BUILDER_SOURCE_DATE_FROM_SPEC_MTIME can be set to use the mtime of the spec file as the SOURCE_DATE_EPOCH. This is useful for vendor packages.

view details

push time in a month

Pull request review commentPowerDNS/pdns

dnsdist: Add support for range-based lookups into a Key-Value store

 std::vector<std::string> KeyValueLookupKeySourceIP::getKeys(const ComboAddress&   std::vector<std::string> result;   ComboAddress truncated(addr); +  std::string key;   if (truncated.isIPv4()) {     truncated.truncate(d_v4Mask);-    result.emplace_back(reinterpret_cast<const char*>(&truncated.sin4.sin_addr.s_addr), sizeof(truncated.sin4.sin_addr.s_addr));+    key.reserve(sizeof(truncated.sin4.sin_addr.s_addr) + (d_includePort ? sizeof(truncated.sin4.sin_port) : 0));+    key.append(reinterpret_cast<const char*>(&truncated.sin4.sin_addr.s_addr), sizeof(truncated.sin4.sin_addr.s_addr));   }   else if (truncated.isIPv6()) {     truncated.truncate(d_v6Mask);-    result.emplace_back(reinterpret_cast<const char*>(&truncated.sin6.sin6_addr.s6_addr), sizeof(truncated.sin6.sin6_addr.s6_addr));+    key.reserve(sizeof(truncated.sin6.sin6_addr.s6_addr) + (d_includePort ? sizeof(truncated.sin4.sin_port) : 0));

The ports are only useful for IPv4 CG-NAT setups, not for IPv6. And I guess sin4.sin_port would need to use sin6?

rgacogne

comment created time in a month

PullRequestReviewEvent