profile
viewpoint

actor-framework/actor-framework 2360

An Open Source Implementation of the Actor Model in C++

mavam/stat-cookbook 1894

:orange_book: The probability and statistics cookbook

mavam/libbf 290

:dart: Bloom filters for C++11

mavam/abstract-algebra-cheatsheet 134

:green_book: A visualization of key structures in abstract algebra.

mavam/ml-driver 64

:rocket: Linux kernel driver for the DreamCheeky USB missile launcher

mavam/dotfiles 46

:desktop_computer: The very heart of an efficient work environment

mavam/gitdub 25

:outbox_tray: A github WebHook that emails detailed diffs of your commits.

mavam/brospects 20

Experimental Bro scripts with good prospects for the official bro-scripts repository.

mavam/compbench 19

:hourglass: Benchmark and visualization of various compression algorithms

pull request commenttenzir/vast

Restrict map to json conversion to string keys

I missed the failing unit test. Please address this before merging.

tobim

comment created time in 5 days

PullRequestReviewEvent

push eventtenzir/vast

Matthias Vallentin

commit sha d2e8953844553d543d630459724998b9a2b3cd15

Rename .conf to .yaml

view details

Matthias Vallentin

commit sha 74b071172291f36c22eaa76be31673dcff081871

Fix incorrect description of config file loading

view details

Matthias Vallentin

commit sha 275b05adaceb646ea0c44810a5af08514c118413

Use /etc/vast as example sysconfdir

view details

Matthias Vallentin

commit sha fd76c16b3f95749ac0a6a4a0a4274407354545f9

Do not advertise that we look for .yml Users should use vast.yaml as filename and that's the only file we officially support. However, to avoid confused users we also accept .yml.

view details

Matthias Vallentin

commit sha 1639804d93db7b4ee2f8525cab8628931ca58930

Clarify config loading order

view details

Matthias Vallentin

commit sha 018ee0971696381216a476996e01a7f3d660661d

Merge pull request #1069 Update YAML config docs

view details

push time in 5 days

delete branch tenzir/vast

delete branch : topic/yaml-docs-fixup

delete time in 5 days

PR merged tenzir/vast

Update YAML config docs maintenance :scissors:

This is an addendum to #1059.

+16 -7

0 comment

2 changed files

mavam

pr closed time in 5 days

PullRequestReviewEvent

Pull request review commenttenzir/vast

Update YAML config docs

 for `vast start`.  ## Configuration -In addition to command options, a configuration file `vast.conf` allows for-persisting option values and tweaking system parameters. Command line options-always override configuration file values.+In addition to command options, a YAML configuration file `vast.yaml` allows+for persisting option values and tweaking system parameters. Command line+options always override configuration file values. -During startup, `vast` looks for a `vast.conf` in the current directory. If-the file does not exist, `vast` then attempts to open `PREFIX/etc/vast.conf`-where `PREFIX` is the installation prefix (which defaults to `/usr/local`).+During startup, `vast` looks for a `vast.yaml` or `vast.yml` in the following+directories, in order:++1. `<sysconfdir>/vast`+2. Environment-variables (stopping when first variable found):+  1. `${XDG_CONFIG_HOME}/vast`+  2. `${HOME}/.config/vast`

I'm glad I'm the odd one out by now. 😉

I've pushed your changes with a slight simplification in the third bullet.

mavam

comment created time in 5 days

push eventtenzir/vast

Matthias Vallentin

commit sha 1639804d93db7b4ee2f8525cab8628931ca58930

Clarify config loading order

view details

push time in 5 days

push eventtenzir/vast

Matthias Vallentin

commit sha fd76c16b3f95749ac0a6a4a0a4274407354545f9

Do not advertise that we look for .yml Users should use vast.yaml as filename and that's the only file we officially support. However, to avoid confused users we also accept .yml.

view details

push time in 5 days

Pull request review commenttenzir/vast

Update YAML config docs

 for `vast start`.  ## Configuration -In addition to command options, a configuration file `vast.conf` allows for-persisting option values and tweaking system parameters. Command line options-always override configuration file values.+In addition to command options, a YAML configuration file `vast.yaml` allows+for persisting option values and tweaking system parameters. Command line+options always override configuration file values. -During startup, `vast` looks for a `vast.conf` in the current directory. If-the file does not exist, `vast` then attempts to open `PREFIX/etc/vast.conf`-where `PREFIX` is the installation prefix (which defaults to `/usr/local`).+During startup, `vast` looks for a `vast.yaml` or `vast.yml` in the following+directories, in order:++1. `<sysconfdir>/vast`+2. Environment-variables (stopping when first variable found):+  1. `${XDG_CONFIG_HOME}/vast`+  2. `${HOME}/.config/vast`
diff --git a/doc/cli/vast.md b/doc/cli/vast.md
index dbf94c2ce..065e86e55 100644
--- a/doc/cli/vast.md
+++ b/doc/cli/vast.md
@@ -36,8 +36,8 @@ In addition to command options, a YAML configuration file `vast.yaml` allows
 for persisting option values and tweaking system parameters. Command line
 options always override configuration file values.

-During startup, `vast` looks for a `vast.yaml` in the following directories, in
-order:
+VAST respects the XDG base directory specification for configuration files and
+looks for `vast.yaml` configuration at startup in the following order:

 1. `<sysconfdir>/vast`
 2. Environment-variables (stopping when first variable found):
mavam

comment created time in 5 days

PullRequestReviewEvent

Pull request review commenttenzir/vast

Update YAML config docs

 for `vast start`.  ## Configuration -In addition to command options, a configuration file `vast.conf` allows for-persisting option values and tweaking system parameters. Command line options-always override configuration file values.+In addition to command options, a YAML configuration file `vast.yaml` allows+for persisting option values and tweaking system parameters. Command line+options always override configuration file values. -During startup, `vast` looks for a `vast.conf` in the current directory. If-the file does not exist, `vast` then attempts to open `PREFIX/etc/vast.conf`-where `PREFIX` is the installation prefix (which defaults to `/usr/local`).+During startup, `vast` looks for a `vast.yaml` or `vast.yml` in the following+directories, in order:++1. `<sysconfdir>/vast`+2. Environment-variables (stopping when first variable found):+  1. `${XDG_CONFIG_HOME}/vast`+  2. `${HOME}/.config/vast`

I didn't know what that XDG thingy is unless you told me. Why not keep it explicit?

mavam

comment created time in 5 days

PullRequestReviewEvent

push eventtenzir/vast

Matthias Vallentin

commit sha 275b05adaceb646ea0c44810a5af08514c118413

Use /etc/vast as example sysconfdir

view details

push time in 5 days

pull request commenttenzir/vast

Update YAML config docs

[ch19359]

mavam

comment created time in 5 days

PR opened tenzir/vast

Update YAML config docs
+14 -7

0 comment

2 changed files

pr created time in 5 days

create barnchtenzir/vast

branch : topic/yaml-docs-fixup

created branch time in 5 days

push eventtenzir/vast

Matthias Vallentin

commit sha 6ee08cbbac74ccd28a4fd118a5266d4effde0e10

Move merge_settings into namespace detail

view details

Matthias Vallentin

commit sha 59f48271b9079aacbe676b71709312bf60fb8d44

Fail config_value conversion for null values

view details

Matthias Vallentin

commit sha d82d1bde7783e23dae2a2094d11a3fe3f8516bd6

Include counts in data conversion

view details

Matthias Vallentin

commit sha b65da277bbab8576718ae0eabc2b37db20eba1b9

Parse config files as YAML

view details

Matthias Vallentin

commit sha 728c79cd1e38eee8fcc79b126aae5d2c1c4638d7

Rewrite example config in YAML

view details

Matthias Vallentin

commit sha 7206e0c9f62d5e384bf2285d2de4acfb60b96386

Update CHANGELOG

view details

Matthias Vallentin

commit sha e9fdabe650523111b932c1a26e1bd32176fc2244

Remove double quotes where not needed

view details

Matthias Vallentin

commit sha a13de46bc69ea88275c909fd5b1691d6ed41133e

Simplify string replacement code

view details

Matthias Vallentin

commit sha 37bfdd182f34768d57bc4bbc53fea5bcb91c88d1

Make error context more user-friendly

view details

Matthias Vallentin

commit sha 4bf0b781f118ff172f824681ef63a58fab812aed

Fixup example configuration

view details

Matthias Vallentin

commit sha 6169455b9ef119d87e0aed759756f467d3051736

Improve unclear comment

view details

Matthias Vallentin

commit sha b123a98e37fb14690addf1106cfe2af97c0a05d7

Undo automatic string-to-URI conversion

view details

Matthias Vallentin

commit sha 8a1f59ef36ac10e1a3354dafcdfcdfa3379c0fac

Also look for vast.yml and vast.yaml configs

view details

Matthias Vallentin

commit sha 007cd69131c7e508ef48c1612edb2dc35c730412

Rename config_paths to config_files

view details

Matthias Vallentin

commit sha 8bef7271a516574b30f52263ae400609b331b2d0

Check for multiple config files in directories

view details

Matthias Vallentin

commit sha 4a49ed60db151baa5018e6f806aba97625e0388d

Rename vast.conf to vast.yaml

view details

Matthias Vallentin

commit sha 21d2a06d97da92b29df54a6b9d8c2bdb5136e18a

Merge config records prior to settings conversion This makes sure that later values always override previously set values.

view details

Matthias Vallentin

commit sha 9832803ad042004f93218aca5d07d95c2192504e

Do not look at .conf anymore

view details

Matthias Vallentin

commit sha aed74483f1dc6b680b63626f5454cb2c73e3dc56

Merge pull request #1059 Switch to YAML config files

view details

push time in 5 days

delete branch tenzir/vast

delete branch : story/ch19445

delete time in 5 days

PR merged tenzir/vast

Switch to YAML config files feature :gift:
  • [x] Rebase onto master after #1055 is merged.
  • [x] Rebase onto master after #1067 is merged.
+664 -526

2 comments

24 changed files

mavam

pr closed time in 5 days

push eventtenzir/vast

Matthias Vallentin

commit sha 9832803ad042004f93218aca5d07d95c2192504e

Do not look at .conf anymore

view details

push time in 5 days

startedrsmmr/justrx

started time in 6 days

push eventtenzir/vast

Dominik Lohmann

commit sha ef197a79c7fb4704f4f90a901ef033d82ea1e160

Fix release builds using gcc-8 Turns out gcc-8 is unable to explicitly capture a variable in a lambda that was implicitly captured in a surrounding lambda if that variable is then only used in an unevaluated context like decltype in the inner lambda.

view details

Dominik Lohmann

commit sha d062d16ae3ff07f6fb51352d18eff624ee4f38a5

Improve log message about partition uuids

view details

Dominik Lohmann

commit sha fc7d2fc8baf7eec2c4fc14a817bcfeeefd4ed862

Merge pull request #1068 Improve log message about partition uuids

view details

Matthias Vallentin

commit sha 3f2de3e35e74889669fb20845ee720a636aa6f1e

Merge pull request #1067 Fix release builds using gcc-8

view details

Matthias Vallentin

commit sha 6ee08cbbac74ccd28a4fd118a5266d4effde0e10

Move merge_settings into namespace detail

view details

Matthias Vallentin

commit sha 59f48271b9079aacbe676b71709312bf60fb8d44

Fail config_value conversion for null values

view details

Matthias Vallentin

commit sha d82d1bde7783e23dae2a2094d11a3fe3f8516bd6

Include counts in data conversion

view details

Matthias Vallentin

commit sha b65da277bbab8576718ae0eabc2b37db20eba1b9

Parse config files as YAML

view details

Matthias Vallentin

commit sha 728c79cd1e38eee8fcc79b126aae5d2c1c4638d7

Rewrite example config in YAML

view details

Matthias Vallentin

commit sha 7206e0c9f62d5e384bf2285d2de4acfb60b96386

Update CHANGELOG

view details

Matthias Vallentin

commit sha e9fdabe650523111b932c1a26e1bd32176fc2244

Remove double quotes where not needed

view details

Matthias Vallentin

commit sha a13de46bc69ea88275c909fd5b1691d6ed41133e

Simplify string replacement code

view details

Matthias Vallentin

commit sha 37bfdd182f34768d57bc4bbc53fea5bcb91c88d1

Make error context more user-friendly

view details

Matthias Vallentin

commit sha 4bf0b781f118ff172f824681ef63a58fab812aed

Fixup example configuration

view details

Matthias Vallentin

commit sha 6169455b9ef119d87e0aed759756f467d3051736

Improve unclear comment

view details

Matthias Vallentin

commit sha b123a98e37fb14690addf1106cfe2af97c0a05d7

Undo automatic string-to-URI conversion

view details

Matthias Vallentin

commit sha 8a1f59ef36ac10e1a3354dafcdfcdfa3379c0fac

Also look for vast.yml and vast.yaml configs

view details

Matthias Vallentin

commit sha 007cd69131c7e508ef48c1612edb2dc35c730412

Rename config_paths to config_files

view details

Matthias Vallentin

commit sha 8bef7271a516574b30f52263ae400609b331b2d0

Check for multiple config files in directories

view details

Matthias Vallentin

commit sha 4a49ed60db151baa5018e6f806aba97625e0388d

Rename vast.conf to vast.yaml

view details

push time in 7 days

push eventtenzir/vast

Dominik Lohmann

commit sha ef197a79c7fb4704f4f90a901ef033d82ea1e160

Fix release builds using gcc-8 Turns out gcc-8 is unable to explicitly capture a variable in a lambda that was implicitly captured in a surrounding lambda if that variable is then only used in an unevaluated context like decltype in the inner lambda.

view details

Matthias Vallentin

commit sha 3f2de3e35e74889669fb20845ee720a636aa6f1e

Merge pull request #1067 Fix release builds using gcc-8

view details

push time in 7 days

delete branch tenzir/vast

delete branch : story/ch19522/release-builds-gcc-8

delete time in 7 days

PR merged tenzir/vast

Fix release builds using gcc-8 bug :beetle:

Turns out gcc-8 is unable to explicitly capture a variable in a lambda that was implicitly captured in a surrounding lambda if that variable is then only used in an unevaluated context like decltype in the inner lambda.

I verified that this builds as expected by running docker build . -t vast.

+3 -3

0 comment

2 changed files

dominiklohmann

pr closed time in 7 days

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commenttenzir/vast

Fix release builds using gcc-8

 caf::behavior index(caf::stateful_actor<index_state>* self, filesystem_type fs,             self->send(client, atom::done_v);             return;           }-          auto& [query_id, query_state] = *iter;+          auto& query_state = iter->second;

Sorry, my bad. The parent context also contains an iter, but the local one shadows it.

dominiklohmann

comment created time in 7 days

pull request commenttenzir/vast

Switch to YAML config files

CI currently fails due the issue to be fixed in #1067; marking this PR as blocked until merged.

mavam

comment created time in 8 days

push eventtenzir/vast

Matthias Vallentin

commit sha 0517dd5770fe39aa1452090c2996d0e967bb333c

Merge config records prior to settings conversion This makes sure that later values always override previously set values.

view details

push time in 8 days

push eventtenzir/vast

Matthias Vallentin

commit sha c763b5f94c7d69ad9cdebf67fcbb814e516898fb

Rename vast.conf to vast.yaml

view details

push time in 8 days

push eventtenzir/vast

Matthias Vallentin

commit sha 7892927ab2ff3a994f5a561130489227812c568d

Also look for vast.yml and vast.yaml configs

view details

Matthias Vallentin

commit sha a7190a8eab90caf457f7eb1e9c939d0c6d49e2a2

Rename config_paths to config_files

view details

Matthias Vallentin

commit sha d8f64d1232a1ad23d65e53292333a2f1170f833d

Check for multiple config files in directories

view details

push time in 8 days

Pull request review commenttenzir/vast

Fix release builds using gcc-8

 caf::behavior index(caf::stateful_actor<index_state>* self, filesystem_type fs,             self->send(client, atom::done_v);             return;           }-          auto& [query_id, query_state] = *iter;+          auto& query_state = iter->second;

Independent of the syntactic change: @lava isn't capturing iter in the lambda a potential bug, since it may get invalidated until the lambda is called?

dominiklohmann

comment created time in 8 days

PullRequestReviewEvent

push eventtenzir/vast

Matthias Vallentin

commit sha 527b81f5c534d4c62efb5a9987496d77babe2d1f

Undo automatic string-to-URI conversion

view details

push time in 8 days

push eventtenzir/vast

Matthias Vallentin

commit sha 69aa7873a77c54c1aff47ba63dbce03c10078b49

Fixup example configuration

view details

Matthias Vallentin

commit sha f4f20a64acfe612ffe192995ea352aaba1b0b705

Improve unclear comment

view details

Matthias Vallentin

commit sha d7a335203d9f0b0ede38c3e466cecec54da3458e

Fix endpoint parsing (WIP)

view details

push time in 8 days

push eventtenzir/vast

Matthias Vallentin

commit sha ab75876264fe60083629708d1cd7d8d1dfe6b105

Make error context more user-friendly

view details

push time in 8 days

Pull request review commenttenzir/vast

Switch to YAML config files

 caf::error configuration::parse(int argc, char** argv) {   command_line.assign(argv + 1, argv + argc);   // Move CAF options to the end of the command line, parse them, and then   // remove them.-  auto is_vast_opt = [](auto& x) {-    return !(detail::starts_with(x, "--caf.")-             || detail::starts_with(x, "--config=")-             || detail::starts_with(x, "--config-file="));-  };+  auto is_vast_opt = [](auto& x) { return !detail::starts_with(x, "--caf."); };   auto caf_opt = std::stable_partition(command_line.begin(), command_line.end(),                                        is_vast_opt);   std::vector<std::string> caf_args;   std::move(caf_opt, command_line.end(), std::back_inserter(caf_args));   command_line.erase(caf_opt, command_line.end());-  for (auto& arg : caf_args) {-    // Remove caf. prefix for CAF parser.-    if (detail::starts_with(arg, "--caf."))-      arg.erase(2, 4);-    // Rewrite --config= option to CAF's expexted format.+  // If the user provided a config file on the command line, we attempt to+  // parse it last.+  for (auto& arg : command_line)     if (detail::starts_with(arg, "--config="))-      arg.replace(8, 0, "-file");+      config_paths.push_back(arg.substr(9));+  // Parse and merge all configuration files.+  caf::settings merged_settings;+  for (const auto& config : config_paths) {+    if (exists(config)) {+      auto contents = load_contents(config);+      if (!contents)+        return contents.error();+      auto yaml = from_yaml(*contents);+      if (!yaml)+        return yaml.error();+      auto rec = caf::get_if<record>(&*yaml);+      if (!rec)+        return caf::make_error(ec::parse_error, "config file not a YAML map");

Better?

mavam

comment created time in 8 days

PullRequestReviewEvent

push eventtenzir/vast

Matthias Vallentin

commit sha 148d9ec743b76e8c1823f28e4788429fe2c38be1

Remove double quotes where not needed

view details

Matthias Vallentin

commit sha 3a3e9f7eb53d39a771713c9d815a42d28b007897

Simplify string replacement code

view details

push time in 8 days

PullRequestReviewEvent

Pull request review commenttenzir/vast

Switch to YAML config files

 caf::error configuration::parse(int argc, char** argv) {   command_line.assign(argv + 1, argv + argc);   // Move CAF options to the end of the command line, parse them, and then   // remove them.-  auto is_vast_opt = [](auto& x) {-    return !(detail::starts_with(x, "--caf.")-             || detail::starts_with(x, "--config=")-             || detail::starts_with(x, "--config-file="));-  };+  auto is_vast_opt = [](auto& x) { return !detail::starts_with(x, "--caf."); };   auto caf_opt = std::stable_partition(command_line.begin(), command_line.end(),                                        is_vast_opt);   std::vector<std::string> caf_args;   std::move(caf_opt, command_line.end(), std::back_inserter(caf_args));   command_line.erase(caf_opt, command_line.end());-  for (auto& arg : caf_args) {-    // Remove caf. prefix for CAF parser.-    if (detail::starts_with(arg, "--caf."))-      arg.erase(2, 4);-    // Rewrite --config= option to CAF's expexted format.+  // If the user provided a config file on the command line, we attempt to+  // parse it last.+  for (auto& arg : command_line)     if (detail::starts_with(arg, "--config="))-      arg.replace(8, 0, "-file");+      config_paths.push_back(arg.substr(9));+  // Parse and merge all configuration files.+  caf::settings merged_settings;+  for (const auto& config : config_paths) {+    if (exists(config)) {+      auto contents = load_contents(config);+      if (!contents)+        return contents.error();+      auto yaml = from_yaml(*contents);+      if (!yaml)+        return yaml.error();+      auto rec = caf::get_if<record>(&*yaml);+      if (!rec)+        return caf::make_error(ec::parse_error, "config file not a YAML map");+      auto flat_yaml = flatten(*rec);+      // Erase all null values because a caf::config_value has no such notion.+      for (auto i = flat_yaml.begin(); i != flat_yaml.end();) {+        if (caf::holds_alternative<caf::none_t>(i->second))+          i = flat_yaml.erase(i);+        else+          ++i;+      }+      auto settings = to<caf::settings>(flat_yaml);+      if (!settings)+        return settings.error();+      detail::merge_settings(*settings, merged_settings);+    }   }-  for (const auto& p : config_paths) {-    if (auto err = actor_system_config::parse({}, p.str().c_str())) {-      err.context() += caf::make_message(p);-      return err;+  // TODO: Revisit this after we are on CAF 0.18.+  // Helper function to parse a config_value with the type information+  // contained in an config_option. Because our YAML config only knows about+  // strings, but a config_option may require an atom, we have to use a+  // heuristic to see whether either type works.+  auto parse_config_value+    = [](const caf::config_option& opt,+         const caf::config_value val) -> caf::expected<caf::config_value> {+    // Hackish way to get a string representation that doesn't add double+    // quotes around the value.+    auto no_quote_stringify+      = detail::overload([](const auto& x) { return caf::deep_to_string(x); },+                         [](const std::string& x) { return x; });+    auto str = caf::visit(no_quote_stringify, val);+    auto result = opt.parse(str);+    if (!result) {+      // We now try to parse strings as atom using a regex, since we get+      // recursive types like lists for free this way. A string-vs-atom type+      // clash is the only instance we currently cannot distinguish. Everything+      // else is a true type clash.+      // (With CAF 0.18, this heuristic will be obsolete.)+      str = std::regex_replace(str, std::regex("\""), "'");

Turns out we had detail::replace_all already! 🙈

mavam

comment created time in 8 days

Pull request review commenttenzir/vast

Switch to YAML config files

 caf::error configuration::parse(int argc, char** argv) {   command_line.assign(argv + 1, argv + argc);   // Move CAF options to the end of the command line, parse them, and then   // remove them.-  auto is_vast_opt = [](auto& x) {-    return !(detail::starts_with(x, "--caf.")-             || detail::starts_with(x, "--config=")-             || detail::starts_with(x, "--config-file="));-  };+  auto is_vast_opt = [](auto& x) { return !detail::starts_with(x, "--caf."); };   auto caf_opt = std::stable_partition(command_line.begin(), command_line.end(),                                        is_vast_opt);   std::vector<std::string> caf_args;   std::move(caf_opt, command_line.end(), std::back_inserter(caf_args));   command_line.erase(caf_opt, command_line.end());-  for (auto& arg : caf_args) {-    // Remove caf. prefix for CAF parser.-    if (detail::starts_with(arg, "--caf."))-      arg.erase(2, 4);-    // Rewrite --config= option to CAF's expexted format.+  // If the user provided a config file on the command line, we attempt to+  // parse it last.+  for (auto& arg : command_line)     if (detail::starts_with(arg, "--config="))-      arg.replace(8, 0, "-file");+      config_paths.push_back(arg.substr(9));+  // Parse and merge all configuration files.+  caf::settings merged_settings;+  for (const auto& config : config_paths) {+    if (exists(config)) {+      auto contents = load_contents(config);+      if (!contents)+        return contents.error();+      auto yaml = from_yaml(*contents);+      if (!yaml)+        return yaml.error();+      auto rec = caf::get_if<record>(&*yaml);+      if (!rec)+        return caf::make_error(ec::parse_error, "config file not a YAML map");+      auto flat_yaml = flatten(*rec);+      // Erase all null values because a caf::config_value has no such notion.+      for (auto i = flat_yaml.begin(); i != flat_yaml.end();) {+        if (caf::holds_alternative<caf::none_t>(i->second))+          i = flat_yaml.erase(i);+        else+          ++i;+      }

So then we need to merge all data instances first.

mavam

comment created time in 8 days

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commenttenzir/vast

Switch to YAML config files

-; This is an example configuration file for VAST, striving to show all available-; options. Lines starting with a semicolon are commented out. Options in angle-; brackets have their default value determined at runtime.--; Options that apply to VAST.-system {-  ; The host and port to listen at and connect to.-  ;endpoint = "localhost:42000"--  ; The file system path used for persistent state.-  ;db-directory = "vast.db/"--  ; The file system path used for log files.-  ;log-file = "<db-directory>/server.log"--  ; The size of an index shard.-  ;max-partition-size = 1000000--  ; The unique ID of this node.-  ;node-id = "node"--  ; List of paths to look for schema files in ascending order of priority.-  ; Note: Automatically prepended with-  ;  ["<binary_directory>/../share/vast/schema", "/etc/vast/schema"].-  ; Use the no-default-schema option to turn off this mechanism.-  ;schema-paths = []--  ; Don't load the default schema definitions.-  ;no-default-schema = false--  ; Spawn a node instead of connecting to one.-  ;node = false--  ; Don't keep track of performance metrics.-  ;disable-metrics = false--  ; Interval between two aging cycles.-  ;aging-frequency = "24h"--  ; Query for aging out obsolete data.-  ;aging-query = ""--  ; The configuration of the metrics reporting component.-  metrics {-    ;enable = true;--    ; Configures if and how metrics should be ingested back into VAST.-    self_sink {-      ;enable = true;-      ;slice_size = 100;-      ;slice_type = 'arrow';-    }--    ; Configures if and where metrics should be written to a file.-    file_sink {-      ;enable = false;-      ;path = "/tmp/vast-metrics.log"-    }--    ; Configures if and where metrics should be written to a socket.-    uds_sink {-      ;enable = false;-      ;path = "/tmp/vast-metrics.sock"-      ;type = "datagram"-    }-  }--  ; The period to wait until a shutdown sequence finishes cleanly. After the-  ; period elapses, the shutdown procedure escalates into a "hard kill".-  ; A value of "0x", where "x" is any duration unit, means an infinite grace-  ; period without escalation into a hard kill.-  ;shutdown-grace-period = "3m"-}--; The `vast count` command counts hits for a query without exporting data.-count {-  ; Estimate an upper bound by skipping candidate checks-  ;estimate = false-}--; The `vast export` command exports query results to stdout or a file.-export {-  ; Mark a query as continuous.-  ;continuous = false--  ; Mark a query as unified.-  ;unified = false--  ; The maximum number of events to export.-  ;max-events = <infinity>--  ; Path for reading the query or "-" for reading from stdin.-  ;read = "-"--  ; The `vast export ascii` command exports events formatted in a plain-text-  ; format that is internal to VAST.-  ascii {-    ; Path to write events to or "-" for writing to stdout.-    ;write = "-"--    ; Treat the write option as a UNIX domain socket to connect to.-    ;uds = false-  }--  ; The `vast export csv` command exports events formatted as CSV.-  csv {-    ; For available options, see export.ascii.-  }--  ; The `vast export json` command exports events formatted as JSONL (line--  ; delimited JSON).-  json {-    ; For additionally available options, see export.ascii.-  }--  ; The `vast export null` command exports events from a given query without-  ; printing them. Used for debugging and benchmarking only.-  null {-    ; For available options, see export.ascii.-  }--  ; The `vast export arrow` command exports events in the Apache Arrow format.-  arrow {-  }--  ; The `vast export pcap` command exports events in the PCAP format.-  pcap {-    ; Flush to disk after this many packets.-    ;flush-interval = 10000--    ; For additionally available options, see export.ascii.-  }--  ; The `vast export zeek` command exports events formatted as Zeek logs.-  zeek {-    ; For available options, see export.ascii.-  }-}--; The `vast infer` command tries to infer the schema from data.-infer {-  ; Path to read events from or "-" for reading from stdin.-  ;read = "-"--  ; Maximum number of bytes to buffer.-  ;buffer = 8192-}--; The `vast explore` command explore context around query results.-explore {-  ; The output format.-  ; format = "json"--  ; Include all records up to this much time after each result.-  ; after = ""--  ; Include all records up to this much time before each result.-  ; before = ""--  ; Perform an equijoin on the given field.-  ; by = ""--  ; Maximum number of results.-  ; max-events = <infinity>--  ; Maximum number of results for initial query.-  ; max-events-query = 100--  ; Maximum number of results per exploration.-  ; max-events-context = 100-}--; The `vast import` command imports data from stdin, files or over the network.-import {-  ; The maximum number of events to import.-  ;max-events = <infinity>+# This is an example configuration file for VAST that shows all available+# options.+#+# Options in angle brackets have their default value determined at runtime.++# Options that apply to VAST.+system:+  # The host and port to listen at and connect to.+  endpoint: "localhost:42000"++  # The file system path used for persistent state.+  db-directory: "vast.db"++  # The file system path used for log files.+  #log-file: "<db-directory>/server.log"++  # The size of an index shard.+  max-partition-size: 1000000++  # The unique ID of this node.+  node-id: "node" -  ; Timeout after which batched table slices are forwarded.-  ;batch-timeout = "10s"+  # List of paths to look for schema files in ascending order of priority.+  # Note: Automatically prepended with+  #  ["<binary_directory>/../share/vast/schema", "/etc/vast/schema"].+  # Use the no-default-schema option to turn off this mechanism.+  #schema-paths: []++  # Don't load the default schema definitions.+  no-default-schema: false -  ; Upper bound for the size of a table slice. A value of 0 causes the-  ; batch-size to be unbounded, leaving control of batching to the-  ; import.batch-timeout option only.-  ;batch-size = 100+  # Spawn a node instead of connecting to one.+  node: false -  ; Encoding type of table slices (arrow or msgpack).-  ;batch-encoding = 'arrow'+  # Don't keep track of performance metrics.+  disable-metrics: false++  # Interval between two aging cycles.+  aging-frequency: "24h"++  # Query for aging out obsolete data.+  aging-query: "" -  ; Block until the importer forwarded all data.-  ;blocking = false--  ; The `vast import csv` command imports data from CSVs with a known schema.-  csv {-    ; The endpoint to listen on ("[host]:port/type").-    ;listen = <none>--    ; Path to file to read events from or "-" for stdin.-    ;read = "-"--    ; Treat the read option as a UNIX domain socket to connect to.-    ;uds = false--    ; Path to an alternate schema.-    ;schema-file = <none>--    ; An alternate schema as a string.-    ;schema = <none>-  }+  # The configuration of the metrics reporting component.+  metrics:+    enable: true -  ; The `vast import json` command imports data from JSONLs with a known schema.-  json {-    ; For available options, see import.csv.-  }+    # Configures if and how metrics should be ingested back into VAST.+    self_sink:+      enable: true+      slice_size: 100+      slice_type: 'arrow' -  ; The `vast import pcap` command imports PCAP logs.-  pcap {-    ; Network interface to read packets from.-    ;interface = <none>+    # Configures if and where metrics should be written to a file.+    file_sink:+      #enable: false+      #path: "/tmp/vast-metrics.log" -    ; Skip flow packets after this many bytes.-    ;cutoff = <infinity>+    # Configures if and where metrics should be written to a socket.+    uds_sink:+      #enable: false+      #path: "/tmp/vast-metrics.sock"+      #type: "datagram" -    ; Number of concurrent flows to track.-    ;max-flows = 1048576+  # The period to wait until a shutdown sequence finishes cleanly. After the+  # period elapses, the shutdown procedure escalates into a "hard kill".+  # A value of "0x", where "x" is any duration unit, means an infinite grace+  # period without escalation into a hard kill.+  shutdown-grace-period: "3m" -    ; Maximum flow lifetime before eviction.-    ;max-flow-age = 60+# The `vast count` command counts hits for a query without exporting data.+count:+  # Estimate an upper bound by skipping candidate checks+  estimate: false -    ; Flow table expiration interval.-    ;flow-expiry = 10+# The `vast export` command exports query results to stdout or a file.+export:+  # Mark a query as continuous.+  continuous: false -    ; Inverse factor by which to delay packets. For example, if 5, then for two-    ; packets spaced *t* seconds apart, the source will sleep for *t/5* seconds.-    ;pseudo-realtime-factor = 0--    ; Snapshot length in bytes.-    ;snaplen = 65535+  # Mark a query as unified.+  unified: false -    ; Disable computation of community id for every packet.-    ; disable-community-id = false--    ; For additionally available options, see import.csv.-  }+  # The maximum number of events to export.+  #max-events: <infinity> -  ; The `vast import suricata` command imports Suricata eve.json logs.-  suricata {-    ; For available options, see import.csv.-  }--  ; The `vast import syslog` command imports Syslog entries.-  syslog {-    ; For available options, see import.csv.-  }--  ; The `vast import test` command imports randomly generated events. Used for-  ; debugging and benchmarking only.-  test {-    ; The PRNG seed.-    seed = 0--    ; For additionally available options, see import.csv.-  }--  ; The `vast import zeek` command imports Zeek logs.-  zeek {-    ; For available options, see import.csv.-  }-}--; The `vast pivot` command extracts related events of a given type.-pivot {-  ; The output format.-  ; format = "json"--  ; For additionally available options, see export.pcap.-}--; The `vast status` command prints a JSON-formatted status summary of the node.-status {-  ; No further configuration options are available. The system options apply.-}--; The `vast start` command spins up a new node.-start {-  ; No further configuration options are available. The system options apply.-}--; The `vast stop` command stops the node gracefully.-stop {-  ; No further configuration options are available. The system options apply.-}--; The `vast version` command prints the current version of VAST.-version {-  ; No further configuration options are available. The system options apply.-}--; The following commands are internally used either within VAST or for-; development, debugging, and benchmarking. No documentation is provided for the-; individual commands, but all options are listed.--kill {-}--peer {-}--send {-}--spawn {-  accountant {-  }--  archive {-    ;segments = 10-    ;max-segment-size = 128-  }--  consensus {-    ;id = 0-    ;store-backend = "raft"-  }--  exporter {-    ;continuous = false-    ;unified = false-    ;events = <infinity>-  }--  index {-    ;max-events = 1048576-    ;max-parts = 10-    ;taste-parts = 5-    ;max-queries = 10-  }--  source {-    ; Please consult the source code of VAST for all available options.-    ; These are mostly symmetrical with the import command.-  }--  sink {-    ; Please consult the source code of VAST for a list of available options.-    ; These are mostly symmetrical with the export command.-  }-}--; The below settings are internal to CAF, and are not checked by VAST directly.-; Please be careful when changing these options. Note that some CAF options may-; be in conflict with VAST options, and are only listed here for completeness.--logger {-  ; Format for rendering individual log file entries.-  ; Valid format specifiers are:-  ;  %c = logging category-  ;  %C = class name-  ;  %d = date-  ;  %F = source file of the log statement-  ;  %L = source line of the log statement-  ;  %m = log message-  ;  %M = source function of the log statement-  ;  %n = newline-  ;  %p = priority / severity of the message-  ;  %r = time since application start-  ;  %t = thread id-  ;  %a = actor id-  ;  %% = '%'-  ;file-format = "%r %c %p %a %t %C %M %F:%L %m%n"--  ; Configures the minimum severity of messages written to the log file.-  ; Possible values: quiet, error, warning, info, verbose, debug, trace.-  ; File logging is only available for commands that start a node (e.g.,-  ; vast start). The levels above 'verbose' are usually not available in-  ; release builds.-  ;file-verbosity = 'debug'--  ; Mode for console log output generation.-  ; Possible values: none, colored, uncolored.-  ;console = "colored"--  ; Format for printing individual log entries to the console.-  ; For a list of valid format specifiers, see file-format.-  ;console-format = "%d %m"--  ; Configures the minimum severity of messages written to the console.-  ; For a list of valid log levels, see file-verbosity.-  ;console-verbosity = 'info'--  ; Excludes listed components from logging.-  ;component-blacklist = ["caf", "caf_flow", "caf_stream"]-}--scheduler {-  ; Accepted alternative: "sharing".-  ;policy = "stealing";--  ; Configures whether the scheduler generates profiling output.-  ;enable-profiling = false--  ; Output file for profiler data (only if profiling is enabled).-  ;profiling-output-file = "/dev/null";--  ; Measurement resolution in milliseconds (only if profiling is enabled).-  ;profiling-resolution = 100ms--  ; Forces a fixed number of threads if set.-  ;max-threads = <number of cores>--  ; Maximum number of messages actors can consume in one run.-  ;max-throughput = <infinite>-}--; When using "stealing" as scheduler policy.-work-stealing {-  ; Number of zero-sleep-interval polling attempts.-  ;aggressive-poll-attempts = 100--  ; Frequency of steal attempts during aggressive polling.-  ;aggressive-steal-interval = 10--  ; Number of moderately aggressive polling attempts.-  ;moderate-poll-attempts = 500--  ; Frequency of steal attempts during moderate polling.-  ;moderate-steal-interval = 5--  ; Sleep interval between poll attempts.-  ;moderate-sleep-duration = 50us--  ; Frequency of steal attempts during relaxed polling.-  ;relaxed-steal-interval = 1--  ; Sleep interval between poll attempts.-  ;relaxed-sleep-duration = 10ms-}--stream {-  ; Processing time per batch.-  ;desired-batch-complexity = 50us--  ; Maximum delay for partial batches.-  ;max-batch-delay = 5ms--  ; Time between emitting credit.-  ;credit-round-interval = 10ms-}+  # Path for reading the query or "-" for reading from stdin.+  read: "-"++  # The `vast export ascii` command exports events formatted in a plain-text+  # format that is internal to VAST.+  ascii:+    # Path to write events to or "-" for writing to stdout.+    write: "-"++    # Treat the write option as a UNIX domain socket to connect to.+    uds: false++  # The `vast export csv` command exports events formatted as CSV.+  csv:+    # For available options, see export.ascii.++  # The `vast export json` command exports events formatted as JSONL (line-+  # delimited JSON).+  json:+    # For additionally available options, see export.ascii.++  # The `vast export null` command exports events from a given query without+  # printing them. Used for debugging and benchmarking only.+  'null':+    # For available options, see export.ascii.++  # The `vast export arrow` command exports events in the Apache Arrow format.+  arrow:++  # The `vast export pcap` command exports events in the PCAP format.+  pcap:+    # Flush to disk after this many packets.+    flush-interval: 10000++    # For additionally available options, see export.ascii.++  # The `vast export zeek` command exports events formatted as Zeek logs.+  zeek:+    # For available options, see export.ascii.++# The `vast infer` command tries to infer the schema from data.+infer:+  # Path to read events from or "-" for reading from stdin.+  read: "-"++  # Maximum number of bytes to buffer.+  buffer: 8192++# The `vast explore` command explore context around query results.+explore:+  # The output format.+  format: "json"++  # Include all records up to this much time after each result.+  # after: ""++  # Include all records up to this much time before each result.+  # before: ""++  # Perform an equijoin on the given field.+  # by: ""++  # Maximum number of results.+  # max-events: <infinity>++  # Maximum number of results for initial query.+  max-events-query: 100++  # Maximum number of results per exploration.+  max-events-context: 100++# The `vast import` command imports data from stdin, files or over the network.+import:+  # The maximum number of events to import.+  #max-events: <infinity>++  # Timeout after which batched table slices are forwarded.+  batch-timeout = "10s"++  # Read timoeut after which data is forwarded to the importer regardless of+  # batching and table slices being unfinished.+  read-timeout: "10s"++  # Block until the importer forwarded all data.+  blocking: false++  # Upper bound for the size of a table slice. A value of 0 causes the+  # batch-size to be unbounded, leaving control of batching to the+  # import.batch-timeout option only.+  batch-size = 100++  # Encoding type of table slices (arrow or msgpack).+  batch-encoding: arrow+++  # The `vast import csv` command imports data from CSVs with a known schema.+  csv:+    # The endpoint to listen on ("[host]:port/type").+    #listen: <none>++    # Path to file to read events from or "-" for stdin.+    #read: "-"++    # Treat the read option as a UNIX domain socket to connect to.+    #uds: false++    # Path to an alternate schema.+    #schema-file: <none>++    # An alternate schema as a string.+    #schema: <none>++  # The `vast import json` command imports data from JSONLs with a known schema.+  json:+    # For available options, see import.csv.++  # The `vast import pcap` command imports PCAP logs.+  pcap:+    # Network interface to read packets from.+    #interface: <none>++    # Skip flow packets after this many bytes.+    #cutoff: <infinity>++    # Number of concurrent flows to track.+    #max-flows: 1048576++    # Maximum flow lifetime before eviction.+    #max-flow-age: 60++    # Flow table expiration interval.+    #flow-expiry: 10++    # Inverse factor by which to delay packets. For example, if 5, then for two+    # packets spaced *t* seconds apart, the source will sleep for *t/5* seconds.+    #pseudo-realtime-factor: 0++    # Snapshot length in bytes.+    #snaplen: 65535++    # Disable computation of community id for every packet.+    # disable-community-id: false++    # For additionally available options, see import.csv.++  # The `vast import suricata` command imports Suricata eve.json logs.+  suricata:+    # For available options, see import.csv.++  # The `vast import syslog` command imports Syslog entries.+  syslog:+    # For available options, see import.csv.++  # The `vast import test` command imports randomly generated events. Used for+  # debugging and benchmarking only.+  test:+    # The PRNG seed.+    seed: 0++    # For additionally available options, see import.csv.++  # The `vast import zeek` command imports Zeek logs.+  zeek:+    # For available options, see import.csv.++# The `vast pivot` command extracts related events of a given type.+pivot:+  # The output format.+  # format: "json"++  # For additionally available options, see export.pcap.++# The `vast status` command prints a JSON-formatted status summary of the node.+status:+  # No further configuration options are available. The system options apply.

I just took this from the default. How about we change this as we are restructuring the YAML file and streamline the sections? Then it becomes also easier to see how the transformation from CAF to YAML config took place.

mavam

comment created time in 8 days

push eventtenzir/vast

Matthias Vallentin

commit sha 0dfc02931776aff8aa077109a83cce8822eb2b32

Update CHANGELOG

view details

push time in 8 days

pull request commenttenzir/vast

Switch to YAML config files

Note to the reviewer: this PR is best reviewed commit by commit.

mavam

comment created time in 8 days

push eventtenzir/vast

Dominik Lohmann

commit sha 3cb90934f1a08e2a699096ac47b0fbd70e4af7e3

Rename import batching parameters This commit changes the options that affect batching in the import process (and remotely spawned sources) to have a clearer, more user-facing name and description. |-------------------------|-----------------------| | Old | New | |-------------------------|-----------------------| | import.table-slice-type | import.batch-encoding | | import.table-slice-size | import.batch-size | | import.read-timeout | import.batch-timeout | |-------------------------------------------------|

view details

Dominik Lohmann

commit sha 71d19fd41982b063f13bffcf0cb8961b14aba700

Add option for unbounded batch size

view details

Dominik Lohmann

commit sha 74d3c3e5973cef8542bf941660bb9756451a5ea9

Add documentation for import batching options

view details

Dominik Lohmann

commit sha bc6cff8b2c71fc1188ebb45ed486e91dc62b0490

Shutdown stalled sources properly This fixes a very old bug: A stalled source that was unable to generate messages is called in a loop until a message is generated. While this may be considered faulty behavior in CAF, we must work around this assumption of a stream source behaving like a generator. This patch generates an invalid table slice if the reader was unable to generate events and calls quit in the reader, and discards the invalid table slice in the importer.

view details

Dominik Lohmann

commit sha 3147f36fe598c09701bf0da0be4cf923f3c34a21

Improve import helptext wording and layout

view details

Dominik Lohmann

commit sha cd762aa23d209297eb27eacd580e2e683faca8ea

Add changelog entries

view details

Dominik Lohmann

commit sha 5a3d8c245be79b428392bb7a283a474ede8674e8

Merge pull request #1058 Improve import batching options

view details

Dominik Lohmann

commit sha 25d276f8807403e290549ff316c9c59d80d10e6b

Fix display of the acccountant actor's name When pimpl-ing actor states, we must move the actor name from the state impl to the state pimpl.

view details

Dominik Lohmann

commit sha 54699c5facde69d371d9d4e23b0620651158fb67

Merge pull request #1061 Fix display of the acccountant actor's name

view details

Tobias Mayer

commit sha 4556ed8bf0e42c87a6b8913a2642309408935c34

Remove obsolete CMake option from vast expression

view details

Tobias Mayer

commit sha 827cfa2855da38006870a3aa9c56ccdfa4e3ab04

Add libyamlcpp to vast build inputs

view details

Tobias Mayer

commit sha 73fcb6971fbbe3f97ab43dc7cc4e6753bd8188e9

Mark yaml_cpp as a private dependency

view details

tobim

commit sha bbfc43e3497e126b32858eda6815431fda462227

Merge pull request #1062 Fix the static build

view details

Benno Evers

commit sha 7e8cf6499e97b64794ef836e60cdbf9b89801339

Add 'vast::path::is_absolute()' function

view details

Benno Evers

commit sha 5fe793651c1eac8ee480a81581fec5674b5e2d8d

Respect absolute paths in filesystem actor

view details

Benno Evers

commit sha 6b2c4faa1c056f11360aa50ddbb9dcadf4b9c8e6

Add .clangd/ to gitignore file

view details

Benno Evers

commit sha 0da1d1b1136e629ce6c93e86f340bd6dc80982b6

Assert evaluation order in vector_map tests

view details

Benno Evers

commit sha 10d8a7df6fe485176846334905d7771ffc678da8

Add LRU cache data structure Add a classical LRU cache. The main difference to the existing implementation in `detail::flat_lru_cache` is the ability to insert key-value pairs directly, without having to specify a generator function. Additionally, it provides the constant-time lookup and insert times that are typically associated with LRU caches.

view details

Benno Evers

commit sha 69c47403a0158dc757e6a950f3a5003590381b8f

Update vast::chunk class interface Introduce a new chunk constructor that makes a copy of the supplied memory, and ensure that the constructor taking ownership only accepts rvalue references. Add a new `inspect()` function to make it possible to send chunks in messages.

view details

Benno Evers

commit sha 2b316859b9fc8c13243a2590f2252218d2ad01b7

Update notifying stream manager API Add `Self` as template argument and a convenience function to attach a stream stage with a notifying stream manager in order to be able to use it in combination with the new v2 index.

view details

push time in 8 days

push eventtenzir/vast

Matthias Vallentin

commit sha 18293fea2b06fce6b0e30504486aae1d0270ac41

Fail config_value conversion for null values

view details

Matthias Vallentin

commit sha c930b2b9ca447c0decfc17e8ff6b4289b6ae9b00

Include counts in data conversion

view details

Matthias Vallentin

commit sha 3401eaef516cd0f2a7713740728d0afd2cc7372e

Parse config files as YAML

view details

Matthias Vallentin

commit sha 7671b7894ea7ddee5714aed5119bf62eea10b7bc

Rewrite example config in YAML

view details

push time in 8 days

Pull request review commenttenzir/vast

Spawn INDEXERs lazily from partition

 passive_partition(caf::stateful_actor<passive_partition_state>* self, uuid id,   // to queries. The `skip` default handler is used to buffer all messages   // arriving until then.   self->set_default_handler(skip);-  self->send(caf::actor_cast<caf::actor>(fs), atom::read_v, path);+  self->send(caf::actor_cast<caf::actor>(fs), atom::mmap_v, path);

It's great to see that we literally only need a one-line change to switch to mmap'ing now!

lava

comment created time in 9 days

Pull request review commenttenzir/vast

Spawn INDEXERs lazily from partition

 using namespace caf;  namespace vast::system { -namespace {--// The functions in this namespace take PartitionState as template argument-// because the impelementation is the same for passive and active partitions.+caf::actor active_partition_state::indexer_at(size_t position) const {+  VAST_ASSERT(position < indexers.size());+  return as_vector(indexers)[position].second;+}  /// Gets the INDEXER at a certain position in the `indexers` stable map.

I think the comment no longer holds, right?

lava

comment created time in 9 days

PullRequestReviewEvent
PullRequestReviewEvent
PullRequestReviewEvent

startedcorelight/zerologon

started time in 9 days

PullRequestReviewEvent

startedOTRF/mordor

started time in 10 days

startedsans-blue-team/DeepBlueCLI

started time in 10 days

Pull request review commenttenzir/threatbus

Add RabbitMQ Backbone Plugin

 plugins:     rabbitmq:       host: localhost       port: 5672+      username: guest+      password: guest+      vhost: /+      naming_join_pattern: . # symbol to concatenate names with. Example queue-name: threatbus.intel."hostname"+      queue:+        name_suffix: "my_suffix" # optional. remove property / set empty to use 'hostname'+        durable: true+        auto_delete: false+        lazy: true+        exclusive: false+        max_items: 100000 # optional. remove property / set to 0 to allow infinite length

Unless these are clear to every RabbitMQ user, it would be nice to see one comment per option. For example, I wouldn't know what auto_delete or exclusive does.

0ortmann

comment created time in 10 days

PullRequestReviewEvent
PullRequestReviewEvent
PullRequestReviewEvent

push eventtenzir/vast

Dominik Lohmann

commit sha c4c587dc29ff70fb1ce30e309b3185f4db911878

Remove falsely advertised Arrow sink options

view details

Dominik Lohmann

commit sha b42ff7cc8b2c902909bd6d6d9b1ebb10665f492c

Merge pull request #1057 Remove falsely advertised Arrow sink options

view details

Matthias Vallentin

commit sha 7157e8f22d26cb58e213a0abed5413deeb1ddf89

Merge pull request #1055 Enable settings conversion to CAF

view details

Matthias Vallentin

commit sha d577532b8ba90ed9e6dbd6fb298789fe0fcbc7aa

Move merge_settings into namespace detail

view details

Matthias Vallentin

commit sha 077341d3aeb3c3cd336cd36bed853cf2c826ba08

Parse YAML config files (WIP)

view details

push time in 10 days

push eventtenzir/vast

Matthias Vallentin

commit sha 72068d603aec6df752ce2fb6eddfd5e2cdee38d9

Allow convert to return caf::error

view details

Matthias Vallentin

commit sha 5f7ba019c0957a1ea5a7a0c5b02c82c5cb93ad0f

Make data convertible to caf::settings

view details

Matthias Vallentin

commit sha 59a803822734f7fef6927157b6960c98955542d7

Forward options properly

view details

Matthias Vallentin

commit sha 0a1b3baba456d1095116c0e85298db1b3f037cf7

Broaden test coverage and streamline overloads

view details

Matthias Vallentin

commit sha 7157e8f22d26cb58e213a0abed5413deeb1ddf89

Merge pull request #1055 Enable settings conversion to CAF

view details

push time in 10 days

delete branch tenzir/vast

delete branch : story/ch19315

delete time in 10 days

PR merged tenzir/vast

Enable settings conversion to CAF enhancement :sparkles:
+142 -5

1 comment

4 changed files

mavam

pr closed time in 10 days

pull request commenttenzir/vast

Enable settings conversion to CAF

Why did you move from caf::settings to caf::dictionary<caf::config_value>?

The reason is that caf::settings is just a top-level alias. The real recursive structure is a config_value and this is what I'd like the user of the API to remember: a config_value can contain a dictionary of config_values. Since the recursion occurs in the implementation, I find it cleaning to stick to a single set of vocabulary.

mavam

comment created time in 10 days

PullRequestReviewEvent

push eventtenzir/vast

Matthias Vallentin

commit sha 59a803822734f7fef6927157b6960c98955542d7

Forward options properly

view details

Matthias Vallentin

commit sha 0a1b3baba456d1095116c0e85298db1b3f037cf7

Broaden test coverage and streamline overloads

view details

Matthias Vallentin

commit sha 29b38a0c10c86b67ba5b4daed5ad93cfea5892dd

Move merge_settings into namespace detail

view details

Matthias Vallentin

commit sha 3c0b82c9a3c08b3c86d95b976524fc92ca5c50c6

Parse YAML config files (WIP)

view details

push time in 11 days

Pull request review commenttenzir/vast

Improve import batching options

 The above command signals the running node to ingest (i.e., to archive and index for later export) all Suricata events from the Eve JSON file passed via standard input. +The import command batches parsed events in table slices. To control the+batching, the following options are available:

Looks like you went with the previous wording in your new commit?

dominiklohmann

comment created time in 11 days

PullRequestReviewEvent

push eventtenzir/vast

Matthias Vallentin

commit sha 0a1b3baba456d1095116c0e85298db1b3f037cf7

Broaden test coverage and streamline overloads

view details

push time in 11 days

push eventtenzir/vast

Matthias Vallentin

commit sha f517e5b3561c8f23ae2be3a5892245c33f9031c8

Broaden test coverage and streamline overloads

view details

push time in 11 days

push eventtenzir/vast

Matthias Vallentin

commit sha 59a803822734f7fef6927157b6960c98955542d7

Forward options properly

view details

push time in 11 days

Pull request review commenttenzir/vast

Improve import batching options

 The above command signals the running node to ingest (i.e., to archive and index for later export) all Suricata events from the Eve JSON file passed via standard input. +The import command batches parsed events in table slices. To control the+batching, the following options are available:++- `import.batch-encoding`: Controls the encoding of table slices. Available+  options are `msgpack` (row-based) and `arrow` (column-based).+- `import.batch-size`: Sets an upper bound for the size of every table slice.+  A table slice is the unit that all components work on, causing this to be a+  high impact tuning parameter. Decreasing the table slice size causes reduced+  latency up to the point where messaging overhead becomes important, and+  increasing it may improve overall performance. Note that this setting does not+  mean that events are forwarded to the index and archive immediately upon+  exceeding the given number of events, as the table slices themselves are+  buffered again. Setting this option to 0 causes the table slice size to be+  unbounded, leaving control of the batch size to other parameters.+- `import.batch-timeout`: Sets a timeout for forwarding buffered table slices to+  the index and archive, and can cause the `import.batch-size` parameter to be+  underrun.+ An optional filter expression allows for importing the relevant subset of

This paragraph now needs a bit gentler introduction.

dominiklohmann

comment created time in 11 days

Pull request review commenttenzir/vast

Improve import batching options

 The above command signals the running node to ingest (i.e., to archive and index for later export) all Suricata events from the Eve JSON file passed via standard input. +The import command batches parsed events in table slices. To control the+batching, the following options are available:++- `import.batch-encoding`: Controls the encoding of table slices. Available+  options are `msgpack` (row-based) and `arrow` (column-based).+- `import.batch-size`: Sets an upper bound for the size of every table slice.+  A table slice is the unit that all components work on, causing this to be a+  high impact tuning parameter. Decreasing the table slice size causes reduced+  latency up to the point where messaging overhead becomes important, and+  increasing it may improve overall performance. Note that this setting does not+  mean that events are forwarded to the index and archive immediately upon+  exceeding the given number of events, as the table slices themselves are+  buffered again. Setting this option to 0 causes the table slice size to be+  unbounded, leaving control of the batch size to other parameters.
  Most components in VAST operate on table slices, which makes the table slice size a fundamental tuning knob on the spectrum of throughput and latency. Small table slices allow for shorter processing times, resulting in more scheduler context switches and a more balanced workload. However, the increased pressure on the scheduler comes at the cost of throughput. A large table slice size allows actors to spend more time processing a block of memory, but makes them yield less frequently to the scheduler. As a result, other actors scheduled on the same thread may have to wait a little longer. 
  
  The `batch-size` merely controls number of events per table slice, but not necessarily the number of events until a component forwards a batch to the next stage in a stream. The [CAF streaming framework](https://actor-framework.readthedocs.io/en/latest/Streaming.html) uses a credit-based flow-control mechanism to determine buffering of tables slices. Setting `batch-size` to 0 causes the table slice size to be "unbounded" and leaves it to other parameters to determine the actual table slice size.
dominiklohmann

comment created time in 11 days

Pull request review commenttenzir/vast

Improve import batching options

 The above command signals the running node to ingest (i.e., to archive and index for later export) all Suricata events from the Eve JSON file passed via standard input. +The import command batches parsed events in table slices. To control the+batching, the following options are available:++- `import.batch-encoding`: Controls the encoding of table slices. Available+  options are `msgpack` (row-based) and `arrow` (column-based).+- `import.batch-size`: Sets an upper bound for the size of every table slice.+  A table slice is the unit that all components work on, causing this to be a+  high impact tuning parameter. Decreasing the table slice size causes reduced+  latency up to the point where messaging overhead becomes important, and+  increasing it may improve overall performance. Note that this setting does not+  mean that events are forwarded to the index and archive immediately upon+  exceeding the given number of events, as the table slices themselves are+  buffered again. Setting this option to 0 causes the table slice size to be+  unbounded, leaving control of the batch size to other parameters.+- `import.batch-timeout`: Sets a timeout for forwarding buffered table slices to+  the index and archive, and can cause the `import.batch-size` parameter to be+  underrun.
- `import.batch-timeout`: Sets a timeout for forwarding buffered table slices to
  the importer. If the timeout fires before a table slice reaches `import.batch-size`, then the table slice will contain fewer events but ship immediately.

🗣️💣

dominiklohmann

comment created time in 11 days

Pull request review commenttenzir/vast

Improve import batching options

 The above command signals the running node to ingest (i.e., to archive and index for later export) all Suricata events from the Eve JSON file passed via standard input. +The import command batches parsed events in table slices. To control the+batching, the following options are available:++- `import.batch-encoding`: Controls the encoding of table slices. Available
- `import.batch-encoding`: Selects the encoding of table slices. Available
dominiklohmann

comment created time in 11 days

Pull request review commenttenzir/vast

Improve import batching options

 The above command signals the running node to ingest (i.e., to archive and index for later export) all Suricata events from the Eve JSON file passed via standard input. +The import command batches parsed events in table slices. To control the+batching, the following options are available:++- `import.batch-encoding`: Controls the encoding of table slices. Available+  options are `msgpack` (row-based) and `arrow` (column-based).+- `import.batch-size`: Sets an upper bound for the size of every table slice.
- `import.batch-size`: Sets an upper bound for the number of events per table slice.
dominiklohmann

comment created time in 11 days

Pull request review commenttenzir/vast

Improve import batching options

 The above command signals the running node to ingest (i.e., to archive and index for later export) all Suricata events from the Eve JSON file passed via standard input. +The import command batches parsed events in table slices. To control the+batching, the following options are available:
The import command parses events into table slices or *batches*.
The following options control the batching:
dominiklohmann

comment created time in 11 days

PullRequestReviewEvent
PullRequestReviewEvent

startedzvelo/cmph

started time in 11 days

PR opened tenzir/vast

Switch to YAML config files
+161 -38

0 comment

12 changed files

pr created time in 11 days

create barnchtenzir/vast

branch : story/ch19445

created branch time in 11 days

push eventtenzir/vast

Matthias Vallentin

commit sha 5f7ba019c0957a1ea5a7a0c5b02c82c5cb93ad0f

Make data convertible to caf::settings

view details

push time in 11 days

PullRequestReviewEvent

startedcorelight/c-community-id

started time in 12 days

Pull request review commenttenzir/threatbus

Add RabbitMQ Backbone Plugin

 Chat with us on [Matrix][chat-url].  ## Getting Started +The `config.yaml.example` file provides a working configuration for Threat Bus+with all existing application plugins enabled together with the RabbitMQ+backbone.+ The following example shows how to connect [MISP][misp], [Zeek][zeek] via Threat Bus. There are more integrations available, so make sure to check out all [Threat Bus projects on PyPI](https://pypi.org/search/?q=threatbus).  *Start Threat Bus*  ```sh-venv/bin/threatbus -c config.yaml+venv/bin/threatbus -c config.yaml.example

Is this the procedure users should go through? I'd imagine it's typically (1) copy the example conf, and then (2) run with my own conf.

0ortmann

comment created time in 12 days

PullRequestReviewEvent

startedbiojppm/rapidyaml

started time in 15 days

Pull request review commenttenzir/threatbus

Add RabbitMQ Backbone Plugin

 class Sighting:     ts: datetime     intel: str     context: dict+++@dataclass()+class SnapshotEnvelope:+    snapshot_type: MessageType+    snapshot_id: str+    body: Intel or Sighting+++@dataclass+class SnapshotRequest:+    snapshot_type: MessageType+    snapshot_id: str+    snapshot: timedelta

Could we have some documentation for these data classes?

0ortmann

comment created time in 15 days

Pull request review commenttenzir/threatbus

Add RabbitMQ Backbone Plugin

 def RunZeek():                 "/trace.pcap",                 "/opt/zeek/share/zeek/site/threatbus.zeek",                 "--",-                "Tenzir::log_operations=T",+                "Tenzir::log_operations=F",             ]         )     except subprocess.CalledProcessError:         return False  -class TestRoundtrips(unittest.TestCase):-    def test_zeek_plugin_message_roundtrip(self):-        """-        Backend agnostic message passing screnario. Sends a fixed amount of-        messages via the threatbus Zeek plugin, subscribes to threatbus, and-        checks if the initially sent messages can be retrieved back.-        """-        result_q = queue.Queue()-        items = 5-        rec = threading.Thread(-            target=zeek_receiver.forward, args=(items, result_q), daemon=False+def StopZeek():+    try:+        return subprocess.Popen(+            [+                "docker",+                "kill",+                "zeek-int",+            ]         )-        rec.start()-        zeek_sender.send_generic("threatbus/intel", items)-        rec.join()+    except subprocess.CalledProcessError:+        return False -        self.assertEqual(result_q.qsize(), items)-        for _ in range(items):-            event = result_q.get()-            self.assertIsNotNone(event)-            result_q.task_done()-        self.assertEqual(0, result_q.qsize())-        result_q.join()++class TestZeekSightingReports(unittest.TestCase):+    def setUp(self):+        config = confuse.Configuration("threatbus")+        config.set_file("config_integration_test.yaml")+        self.threatbus = threading.Thread(+            target=start,+            args=(config,),+            daemon=True,+        )+        self.threatbus.start()+        time.sleep(1)      def test_intel_sighting_roundtrip(self):         """         Backend agnostic routrip screnario, that starts a Zeek
        Backend-agnostic roundtrip scenario, that starts a Zeek
0ortmann

comment created time in 15 days

Pull request review commenttenzir/threatbus

Add RabbitMQ Backbone Plugin

 def run(config, logging, inq):         validate_config(config)     except Exception as e:         logger.fatal("Invalid config for plugin {}: {}".format(plugin_name, str(e)))-    threading.Thread(target=provision, args=(logger, inq), daemon=True).start()+    threading.Thread(target=provision, args=(inq,), daemon=True).start()

Is that valid syntax?

0ortmann

comment created time in 15 days

Pull request review commenttenzir/threatbus

Add RabbitMQ Backbone Plugin

+import confuse+import queue+import threading+from threatbus import start+import time+import unittest++from tests.utils import zeek_receiver, zeek_sender+++class TestMessageRoundtrip(unittest.TestCase):+    def setUp(self):+        config = confuse.Configuration("threatbus")+        config.set_file("config_integration_test.yaml")+        self.threatbus = threading.Thread(+            target=start,+            args=(config,),+            daemon=True,+        )+        self.threatbus.start()+        time.sleep(1)++    def test_zeek_plugin_message_roundtrip(self):+        """+        Backend agnostic message passing screnario. Sends a fixed amount of
        Backend-agnostic message passing scenario. Sends a fixed amount of
0ortmann

comment created time in 15 days

Pull request review commenttenzir/threatbus

Add RabbitMQ Backbone Plugin

 logging:  plugins:   backbones:-    inmem:+    #inmem:

Why comment this one?

0ortmann

comment created time in 15 days

Pull request review commenttenzir/threatbus

Add RabbitMQ Backbone Plugin

 plugins:         manage: 13370         pub: 13371         sub: 13372-    cif3:-      api:-        host: http://localhost:5000-        ssl: false-        token: CIF_TOKEN-      group: everyone-      confidence: 7.5-      tlp: amber-      tags:-        - test-        - malicious+    #cif3:+    #  api:+    #    host: http://localhost:5000+    #    ssl: false+    #    token: CIF_TOKEN+    #  group: everyone+    #  confidence: 7.5+    #  tlp: amber+    #  tags:+    #    - test+    #    - malicious

Maybe move to a config.yaml.example instead of commenting?

0ortmann

comment created time in 15 days

PullRequestReviewEvent
more