profile
viewpoint
Jean Mertz JeanMertz Ethical Engineering The Netherlands, Remote https://jeanmertz.com Freelance Software Engineer at @EthicalEngineering, Game developer in my spare time at @rustic-games.

JeanMertz/chruby-fish 89

Thin wrapper around chruby to make it work with the Fish shell

JeanMertz/consulctl 7

Easily get and manage key/value pairs in a Consul cluster.

JeanMertz/docker-aws-cli 5

AWS CLI inside a small'ish Docker container

JeanMertz/automaat 1

🤖 Automate repeatable tasks for everyone within your organisation.

JeanMertz/chef-homebrew 1

Convincing Chef to use Homebrew as its native package management tool

JeanMertz/chef-sugar-dip 1

An extension to the awesome "chef-sugar" gem produced by Seth Vargo.

JeanMertz/chef-zip_app 1

Chef LWRP to install Mac OS X applications from zip archives

JeanMertz/dmg 1

Development repository for Opscode Cookbook dmg

automaat-it/worker 0

Automaat Worker

JeanMertz/addressabler 0

Addressabler extends the Addressable::URI class to provide information about, and manipulation of, URI strings.

PullRequestReviewEvent

issue commentvectordotdev/vector

[Topology] New reroute dropped feature creates `.dropped` stream regardless of `reroute_dropped=false` config

So that would confirm the issue then. Vector is creating the .dropped stream regardless of reroute_dropped=false.

This is a known limitation of the current implementation of the "dropped" stream. We're leveraging the existing topology feature set, which allowed us to bring this feature into a stable release more quickly, but it does have drawbacks (this is one of them, the other being that we have to require operators to explicitly set reroute_dropped, since we can't — yet — detect if the dropped stream is being used in a pipeline).

There are improvements planned for this feature going forward, and you running into this does prove to us that this is one of the issues we need to tackle 👍

For now, the best thing to do here is to explicitly exclude the streams you aren't interested in, in your glob pattern.

hhromic

comment created time in 7 days

pull request commentvectordotdev/vector

enhancement(vrl): Explore allowing access to metric values

With the shift to only using vrl for unit test conditions, being able to access the metric values with vrl is particularly important for unit testing log_to_metric transforms.

I don't think we ever supported testing metrics using the old-style unit tests, did we?

Regardless, I agree, this is something we definitely want to support for both unit-testing, and using remap to transform metrics.

jszwedko

comment created time in 11 days

PullRequestReviewEvent

Pull request review commentvectordotdev/vector

chore: add VRL enum VM (VenuM) RFC

+# RFC 9811 - 2021-11-12 - VRL enum VM (VenuM)++This RFC proposes implementing a enum VM. VRL will be compiled to a list of+instructions and executed by this VM, with the aim to significantly improve the+performance of executing VRL programs.++## Context++This RFC is a follow on from+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Scope++### In scope++This RFC is purely about developing a enum VM for VRL. It will discuss the+risks involved in running VRL with a VM and the measures we will take to+mitigate those risks.++### Out of scope++Any other performance issues relating to VRL will be discussed in+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Pain++Vrl compiles to an AST that is then walked during resolution. Each node in that+tree is boxed and stored in disparate regions of memory. As a result walking+the tree means that the CPU caches must be constantly swapped.++## Proposal++### Implementation++Instead we can create an enum VM to store the execution of the Vrl program.++The enum is essentially a big enum of instructions:++```rust+#[derive(Copy, Clone, Debug, PartialEq, Eq)]+pub enum OpCode {+    Return,+    Constant,+    Negate,+    Add,+    Subtract,+    Multiply,+    Divide,+    Print,+    Not,+    Greater,+    GreaterEqual,+    Less,+    LessEqual,+    NotEqual,+    Equal,+    Pop,+    JumpIfFalse,+    Jump,+    SetPath,+    GetPath,+    Call,+    ...+}++enum Instruction {+    Opcode(Opcode),+    Literal(LiteralIndex),+}++pub struct LiteralIndex(usize);+```++The Vm is a struct comprising of the following fields:++```rust+#[derive(Clone, Debug, Default)]+pub struct Vm {+    instructions: Vec<Instruction>,+    constants: Vec<Literal>,+    targets: Vec<Variable>,+    stack: Vec<Value>,+    parameter_stack: Vec<Option<Value>>,+    error: Option<Error>,+    instruction_pointer: usize,+}+```++- instructions++The instructions field is a `Vec` of `Instruction`. An instruction can be+either an OpCode or some data for that OpCode. For example, the instructions+`[.., OpCode(Constant), Literal(12), ..]` when evaluated will load the constant+stored in the `values` `Vec` that is found in position 12 onto the stack.++- constants++A list of constant values found in the program. Since the bytecode only+contains integers any actual values must be stored here. This also allows+literals to be deduped.++- targets++A list of paths used in the program, similar to `values`.++- stack++The Vm is a stack based Vm. Every expression that is evaluated pushes the+result on the stack. Every operation pulls the values it uses from the stack.++- parameter_stack++For function calls, the parameters need to be evaluated and passed to the+function. The stack is a `Vec` of parameter name and value. We need an OpCode+that will copy a value from the main stack to the parameter stack tagged with+the parameter name. The `FunctionArgs` compiler will dump this OpCode after the+code to evaluate the given arg is dumped.++- error++If an operation errors, it needs to populate this field. Operations such as+the error assignment (`result, error = thing()`) will need to check if this+field has been populated and action accordingly.++- instruction_pointer++The instruction pointer points to the next instruction to evaluate.++#### Calling functions++Calling functions in the stdlib will be a case of evaluating each parameter+with the results pushed onto the parameter stack.++Since VRL allows for named parameters, parameters need to be pushed on to the+parameter stack _in the order that they are specified in the `Function`+implementation_.

Got it, that makes perfect sense 👍

StephenWakely

comment created time in 12 days

Pull request review commentvectordotdev/vector

chore: add VRL enum VM (VenuM) RFC

+# RFC 9811 - 2021-11-12 - VRL bytecode VM++This RFC proposes implementing a bytecode VM. VRL will be compiled to the bytecode+and executed by this VM, with the aim to significantly improve the performance of+executing VRL scripts.++## Context++This RFC is a follow on from+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Scope++### In scope++This RFC is purely about developing a bytecode VM for VRL.++### Out of scope++Any other performance issues relating to VRL will be discussed in+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Proposal++### Implementation++Vrl compiles to an AST that is then walked during resolution. Each node in that+tree is boxed and stored in disparate regions of memory. As a result walking+the tree means that the CPU caches must be constantly swapped.++Instead we can create a bytecode VM to store the execution of the Vrl program.++Bytecode is essentially a big enum of instructions (that are represented by an integer+using the [num-derive](https://crates.io/crates/num-derive),+[num-traits](https://crates.io/crates/num-traits) crates):++```rust+#[derive(FromPrimitive, ToPrimitive, Copy, Clone, Debug, PartialEq, Eq)]+pub enum OpCode {+    Return = 255,+    Constant,+    Negate,+    Add,+    Subtract,+    Multiply,+    Divide,+    Print,+    Not,+    Greater,+    GreaterEqual,+    Less,+    LessEqual,+    NotEqual,+    Equal,+    Pop,+    JumpIfFalse,+    Jump,+    SetPath,+    GetPath,+    Call,+    ...+}+```++The Vm is a struct comprising of the following fields:++```rust+#[derive(Clone, Debug, Default)]+pub struct Vm {+    instructions: Vec<usize>,+    values: Vec<Literal>,+    targets: Vec<Variable>,+    stack: Vec<Value>,+    ip: usize,+}+```++- instructions++The instructions field is a `Vec` of `OpCode` cast to a usize. The reason for+the cast is because not all instructions are `OpCode`. For example the+instructions `[.., Constant, 12, ..]` when evaluated will load the constant+stored in the `values` `Vec` that is found in position 12 onto the stack.

We could also keep the index in the OpCode itself like:

I don't see how it gives more type safety, given that the index for one opcode can point to a different store than the index for another opcode. With a wrapper type, you can ensure that the index for one store cannot be mixed with the index into another store.

Also, it might be worth looking into repr(C) and NonZeroUsize and the like, that allow the Rust compiler to add optimizations during compilation to reduce the word count of these wrapper types. I'm not that familiar with how this works, but for example Option<NonZeroUsize> is equal in size to usize because the compiler can use the first bit (the 0 value of the usize) to encode whether the value is a Some or None.

StephenWakely

comment created time in 12 days

PullRequestReviewEvent

pull request commentvectordotdev/vector

chore: add VRL enum VM (VenuM) RFC

I would strongly encourage looking into dynamically sized bytecode. This would require the instructions to be a pure Vec<u8>, which means you do lose some of the typing that an enum gives you, but it is almost certainly going to be more performant.

I'm in the opposite camp on this. I agree it might increase the performance gain, but I'd rather take it step-by-step, and start with a more safe (and easier to grok) implementation that leverages Rust's types/type-safety as much as possible.

All other optimizations shouldn't require an extensive rewrite in the future, and can thus be split up into issues that we can tackle as the need arises, similar to how we started with a simple tree-based runtime implementation in the first version because it was "fast enough" for most use-cases at the time, and meant it was easier for us to iterate on the language in the beginning stages.

We aren't at the "the language is done" stage yet, so while I concur that we need this change to get us closer to the ideal performance, we still need to strike a balance, and that balance for me lies at going with a VM as described in this RFC, but leave some performance gains on the table for later, in favour of making it as easy/safe as possible to use/iterate on.

StephenWakely

comment created time in 12 days

Pull request review commentvectordotdev/vector

chore: add VRL enum VM (VenuM) RFC

+# RFC 9811 - 2021-11-12 - VRL enum VM (VenuM)++This RFC proposes implementing a enum VM. VRL will be compiled to a list of+instructions and executed by this VM, with the aim to significantly improve the+performance of executing VRL programs.++## Context++This RFC is a follow on from+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Scope++### In scope++This RFC is purely about developing a enum VM for VRL. It will discuss the+risks involved in running VRL with a VM and the measures we will take to+mitigate those risks.++### Out of scope++Any other performance issues relating to VRL will be discussed in+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Pain++Vrl compiles to an AST that is then walked during resolution. Each node in that+tree is boxed and stored in disparate regions of memory. As a result walking+the tree means that the CPU caches must be constantly swapped.++## Proposal++### Implementation++Instead we can create an enum VM to store the execution of the Vrl program.++The enum is essentially a big enum of instructions:++```rust+#[derive(Copy, Clone, Debug, PartialEq, Eq)]+pub enum OpCode {+    Return,+    Constant,+    Negate,+    Add,+    Subtract,+    Multiply,+    Divide,+    Print,+    Not,+    Greater,+    GreaterEqual,+    Less,+    LessEqual,+    NotEqual,+    Equal,+    Pop,+    JumpIfFalse,+    Jump,+    SetPath,+    GetPath,+    Call,+    ...+}++enum Instruction {+    Opcode(Opcode),+    Literal(LiteralIndex),+}++pub struct LiteralIndex(usize);+```++The Vm is a struct comprising of the following fields:++```rust+#[derive(Clone, Debug, Default)]+pub struct Vm {+    instructions: Vec<Instruction>,+    constants: Vec<Literal>,+    targets: Vec<Variable>,+    stack: Vec<Value>,+    parameter_stack: Vec<Option<Value>>,+    error: Option<Error>,+    instruction_pointer: usize,+}+```++- instructions++The instructions field is a `Vec` of `Instruction`. An instruction can be+either an OpCode or some data for that OpCode. For example, the instructions+`[.., OpCode(Constant), Literal(12), ..]` when evaluated will load the constant+stored in the `values` `Vec` that is found in position 12 onto the stack.++- constants++A list of constant values found in the program. Since the bytecode only+contains integers any actual values must be stored here. This also allows+literals to be deduped.++- targets++A list of paths used in the program, similar to `values`.++- stack++The Vm is a stack based Vm. Every expression that is evaluated pushes the+result on the stack. Every operation pulls the values it uses from the stack.++- parameter_stack++For function calls, the parameters need to be evaluated and passed to the+function. The stack is a `Vec` of parameter name and value. We need an OpCode+that will copy a value from the main stack to the parameter stack tagged with+the parameter name. The `FunctionArgs` compiler will dump this OpCode after the+code to evaluate the given arg is dumped.++- error++If an operation errors, it needs to populate this field. Operations such as+the error assignment (`result, error = thing()`) will need to check if this+field has been populated and action accordingly.++- instruction_pointer++The instruction pointer points to the next instruction to evaluate.++#### Calling functions++Calling functions in the stdlib will be a case of evaluating each parameter+with the results pushed onto the parameter stack.++Since VRL allows for named parameters, parameters need to be pushed on to the+parameter stack _in the order that they are specified in the `Function`+implementation_.++Parameters are optional and may not be specified. However, to avoid passing+parameters to the wrong function, an OpCode still needs to be emitted to move+a placeholder value - `None` - to the parameter stack.++Take the following hypothetical situation:++```coffeescript+thing(surname: "nong", name: thang(name: "nork"))+```++Supposing function `thang` also took an optional parameter `surname`, when+`thang` is called, we will have a `surname` parameter on the parameter stack.+Without the placeholder, `thang` would thing that `surname` was being passed to+it. If the placeholder is pushed, it would consume this instead.++We do not want `stdlib` functions to have access to the VM since that risks a+rogue function destroying the VM's state.++Current `stdlib` functions are composed of a combination of `compile` and+`resolve` functions. These function will need to be combined into a single+function `call`.++The `ArgumentList` parameter that is passed into `call` will have access to the+parameter stack and the parameter list exposed by the `Function`. This can use+these to return the appropriate values for the `required`, `optional` etc..+functions.

I'd love to see a stripped-down example of one of the existing function implementations (f.e. upcase), to get a better sense of how this would work, and if this pattern fits all existing use-cases.

StephenWakely

comment created time in 12 days

Pull request review commentvectordotdev/vector

chore: add VRL enum VM (VenuM) RFC

+# RFC 9811 - 2021-11-12 - VRL enum VM (VenuM)++This RFC proposes implementing a enum VM. VRL will be compiled to a list of+instructions and executed by this VM, with the aim to significantly improve the+performance of executing VRL programs.++## Context++This RFC is a follow on from+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Scope++### In scope++This RFC is purely about developing a enum VM for VRL. It will discuss the+risks involved in running VRL with a VM and the measures we will take to+mitigate those risks.++### Out of scope++Any other performance issues relating to VRL will be discussed in+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Pain++Vrl compiles to an AST that is then walked during resolution. Each node in that+tree is boxed and stored in disparate regions of memory. As a result walking+the tree means that the CPU caches must be constantly swapped.++## Proposal++### Implementation++Instead we can create an enum VM to store the execution of the Vrl program.++The enum is essentially a big enum of instructions:++```rust+#[derive(Copy, Clone, Debug, PartialEq, Eq)]+pub enum OpCode {+    Return,+    Constant,+    Negate,+    Add,+    Subtract,+    Multiply,+    Divide,+    Print,+    Not,+    Greater,+    GreaterEqual,+    Less,+    LessEqual,+    NotEqual,+    Equal,+    Pop,+    JumpIfFalse,+    Jump,+    SetPath,+    GetPath,+    Call,+    ...+}++enum Instruction {+    Opcode(Opcode),+    Literal(LiteralIndex),+}++pub struct LiteralIndex(usize);+```++The Vm is a struct comprising of the following fields:++```rust+#[derive(Clone, Debug, Default)]+pub struct Vm {+    instructions: Vec<Instruction>,+    constants: Vec<Literal>,+    targets: Vec<Variable>,+    stack: Vec<Value>,+    parameter_stack: Vec<Option<Value>>,+    error: Option<Error>,+    instruction_pointer: usize,+}+```++- instructions++The instructions field is a `Vec` of `Instruction`. An instruction can be+either an OpCode or some data for that OpCode. For example, the instructions+`[.., OpCode(Constant), Literal(12), ..]` when evaluated will load the constant+stored in the `values` `Vec` that is found in position 12 onto the stack.++- constants++A list of constant values found in the program. Since the bytecode only+contains integers any actual values must be stored here. This also allows+literals to be deduped.++- targets++A list of paths used in the program, similar to `values`.++- stack++The Vm is a stack based Vm. Every expression that is evaluated pushes the+result on the stack. Every operation pulls the values it uses from the stack.++- parameter_stack++For function calls, the parameters need to be evaluated and passed to the+function. The stack is a `Vec` of parameter name and value. We need an OpCode+that will copy a value from the main stack to the parameter stack tagged with+the parameter name. The `FunctionArgs` compiler will dump this OpCode after the+code to evaluate the given arg is dumped.++- error++If an operation errors, it needs to populate this field. Operations such as+the error assignment (`result, error = thing()`) will need to check if this+field has been populated and action accordingly.++- instruction_pointer++The instruction pointer points to the next instruction to evaluate.++#### Calling functions++Calling functions in the stdlib will be a case of evaluating each parameter+with the results pushed onto the parameter stack.++Since VRL allows for named parameters, parameters need to be pushed on to the+parameter stack _in the order that they are specified in the `Function`+implementation_.

Is this correct? Or, is this the complete story?

I'm still unsure how this solves the problem of VRL programs allowing function calls to mix-and-match named and positional arguments and allowing named arguments to be anywhere in the function call, i.e. there is no pre-defined order for named parameters.

StephenWakely

comment created time in 12 days

PullRequestReviewEvent

Pull request review commentvectordotdev/vector

chore: add VRL enum VM (VenuM) RFC

+# RFC 9811 - 2021-11-12 - VRL enum VM (VenuM)++This RFC proposes implementing a enum VM. VRL will be compiled to a list of+instructions and executed by this VM, with the aim to significantly improve the+performance of executing VRL programs.++## Context++This RFC is a follow on from+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Scope++### In scope++This RFC is purely about developing a enum VM for VRL. It will discuss the+risks involved in running VRL with a VM and the measures we will take to+mitigate those risks.++### Out of scope

It came to mind that I think it would be good to add "user-facing breaking changes" as out of scope for this RFC. That is, whatever implementation we end up with, there shouldn't be any difference to the user, other than a faster runtime.

If we do want to do optimizations that require a breaking change, I suggest pulling those out into a separate issue, and discuss them separately, so that we don't block any work listed in this RFC.

StephenWakely

comment created time in 12 days

PullRequestReviewEvent

Pull request review commentvectordotdev/vector

chore: add VRL enum VM (VenuM) RFC

+# RFC 9811 - 2021-11-12 - VRL enum VM (VenuM)++This RFC proposes implementing a enum VM. VRL will be compiled to a list of+instructions and executed by this VM, with the aim to significantly improve the+performance of executing VRL programs.++## Context++This RFC is a follow on from+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Scope++### In scope++This RFC is purely about developing a enum VM for VRL. It will discuss the+risks involved in running VRL with a VM and the measures we will take to+mitigate those risks.++### Out of scope++Any other performance issues relating to VRL will be discussed in+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Pain++Vrl compiles to an AST that is then walked during resolution. Each node in that+tree is boxed and stored in disparate regions of memory. As a result walking+the tree means that the CPU caches must be constantly swapped.++## Proposal++### Implementation++Instead we can create an enum VM to store the execution of the Vrl program.++The enum is essentially a big enum of instructions:++```rust+#[derive(Copy, Clone, Debug, PartialEq, Eq)]+pub enum OpCode {+    Return,+    Constant,+    Negate,+    Add,+    Subtract,+    Multiply,+    Divide,+    Print,+    Not,+    Greater,+    GreaterEqual,+    Less,+    LessEqual,+    NotEqual,+    Equal,+    Pop,+    JumpIfFalse,+    Jump,+    SetPath,+    GetPath,+    Call,+    ...+}++enum Instruction {+    Opcode(Opcode),+    Literal(LiteralIndex),+}++pub struct LiteralIndex(usize);+```++The Vm is a struct comprising of the following fields:++```rust+#[derive(Clone, Debug, Default)]+pub struct Vm {+    instructions: Vec<Instruction>,+    constants: Vec<Literal>,+    targets: Vec<Variable>,+    stack: Vec<Value>,+    parameter_stack: Vec<Option<Value>>,+    error: Option<Error>,+    instruction_pointer: usize,+}+```++- instructions++The instructions field is a `Vec` of `Instruction`. An instruction can be+either an OpCode or some data for that OpCode. For example, the instructions+`[.., OpCode(Constant), Literal(12), ..]` when evaluated will load the constant+stored in the `values` `Vec` that is found in position 12 onto the stack.++- constants++A list of constant values found in the program. Since the bytecode only+contains integers any actual values must be stored here. This also allows+literals to be deduped.++- targets++A list of paths used in the program, similar to `values`.++- stack++The Vm is a stack based Vm. Every expression that is evaluated pushes the+result on the stack. Every operation pulls the values it uses from the stack.++- parameter_stack++For function calls, the parameters need to be evaluated and passed to the+function. The stack is a `Vec` of parameter name and value. We need an OpCode+that will copy a value from the main stack to the parameter stack tagged with+the parameter name. The `FunctionArgs` compiler will dump this OpCode after the+code to evaluate the given arg is dumped.++- error++If an operation errors, it needs to populate this field. Operations such as+the error assignment (`result, error = thing()`) will need to check if this+field has been populated and action accordingly.++- instruction_pointer++The instruction pointer points to the next instruction to evaluate.++#### Calling functions++Calling functions in the stdlib will be a case of evaluating each parameter+with the results pushed onto the parameter stack.++Since VRL allows for named parameters, parameters need to be pushed on to the+parameter stack _in the order that they are specified in the `Function`+implementation_.++Parameters are optional and may not be specified. However, to avoid passing+parameters to the wrong function, an OpCode still needs to be emitted to move+a placeholder value - `None` - to the parameter stack.++Take the following hypothetical situation:++```coffeescript+thing(surname: "nong", name: thang(name: "nork"))+```++Supposing function `thang` also took an optional parameter `surname`, when+`thang` is called, we will have a `surname` parameter on the parameter stack.+Without the placeholder, `thang` would thing that `surname` was being passed to+it. If the placeholder is pushed, it would consume this instead.++We do not want `stdlib` functions to have access to the VM since that risks a+rogue function destroying the VM's state.++Current `stdlib` functions are composed of a combination of `compile` and+`resolve` functions. These function will need to be combined into a single+function `call`.++The `ArgumentList` parameter that is passed into `call` will have access to the+parameter stack and the parameter list exposed by the `Function`. This can use+these to return the appropriate values for the `required`, `optional` etc..+functions.++Since the `compile` function was also used by some `stdlib` functions to+validate the parameters we may alse need a function `validate_parameters` that+these functions can use to perform extra validation on the parameters.++### Debugging++We need debugging tools to help us introspect and debug the compiled bytecode.++#### Dissassembler++Output the compiled code in readable format to the console.++#### Step debugger++Output each step of the VM to the console, print the Stack and other state+variables in the VM. Allow the user to press a key to move to the next step.++#### Linter++Check each instruction to ensure:++- each Jump Opcode jumps to a valid location in the bytecode.+- each constant references a valid index in the constants list+- each path index references a valid path.++It should be possible to run the linter after every compile (at Vector boot+time), so the linter should run relatively fast.++### Testing++The VM will need thorough testing since it is possible to generate invalid+bytecode. For example, the following bytecode is invalid:++```rust+OpCode(CONSTANT)+Primitive(1)+OpCode(ADD)+```++The `ADD` OpCode expects there to be two values on the stack to add+together. We should never get into the situation where the VRL compiler can+generate this.++With property testing we can generate a vast combination of valid VRL ASTs,+compile this to bytecode and then run the VM to ensure we get no buffer+underflows, invalid constant locations or other invalid bytecode.++Given that implementing the VM involves adding methods to the Expression trait,+it should be possible to run the existing AST alongside the VM. We can then+ensure that we get exactly the same results from each approach.++## Rationale++With each node of the AST compiled down to just a few bytes and all+instructions held in contiguous memory evaluation of the program should be able+to take full advantage of the CPU cache which should result in much faster+execution.++## Drawbacks++Downsides to using a Vm:++- The code is a bit more complex. With an AST that is walked it is fairly+  apparent what the code will be doing at any point. With a Vm, this is not the+  case, it is harder to look at the instructions in the Vm and follow back to+  what part of the VRL code is being evaluated. We will need to write some+  extensive debugging tools to allow for decent introspection into the Vm.++  However, VRL is a very simple language, which does also mean the VM will be+  simple.++- We lose some safety that we get from the Rust compiler. There will need+  to be significant fuzz testing to ensure that the code runs correctly under+  all circumstances.++  It should be noted however that the entire VM will not require any _unsafe_+  Rust code.++- Currently each stdlib function is responsible for evaluating their own+  parameters. This allows parameters to be lazily evaluated. With the Vm, the+  parameters will need to be evaluated up front and the stack passed into the+  function. This could impact performance.

This could potentially be a big downside, given that almost all VRL programs are function-call-heavy.

I'd love to see this fleshed out a bit more, with some real-world examples of functions that would most likely be impacted by this, and how that impact changes their behaviour.

StephenWakely

comment created time in 12 days

Pull request review commentvectordotdev/vector

chore: add VRL enum VM (VenuM) RFC

+# RFC 9811 - 2021-11-12 - VRL enum VM (VenuM)++This RFC proposes implementing a enum VM. VRL will be compiled to a list of+instructions and executed by this VM, with the aim to significantly improve the+performance of executing VRL programs.++## Context++This RFC is a follow on from+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Scope++### In scope++This RFC is purely about developing a enum VM for VRL. It will discuss the+risks involved in running VRL with a VM and the measures we will take to+mitigate those risks.++### Out of scope++Any other performance issues relating to VRL will be discussed in+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Pain++Vrl compiles to an AST that is then walked during resolution. Each node in that+tree is boxed and stored in disparate regions of memory. As a result walking+the tree means that the CPU caches must be constantly swapped.++## Proposal++### Implementation++Instead we can create an enum VM to store the execution of the Vrl program.++The enum is essentially a big enum of instructions:++```rust+#[derive(Copy, Clone, Debug, PartialEq, Eq)]+pub enum OpCode {+    Return,+    Constant,+    Negate,+    Add,+    Subtract,+    Multiply,+    Divide,+    Print,+    Not,+    Greater,+    GreaterEqual,+    Less,+    LessEqual,+    NotEqual,+    Equal,+    Pop,+    JumpIfFalse,+    Jump,+    SetPath,+    GetPath,+    Call,+    ...+}++enum Instruction {+    Opcode(Opcode),+    Literal(LiteralIndex),+}++pub struct LiteralIndex(usize);+```++The Vm is a struct comprising of the following fields:++```rust+#[derive(Clone, Debug, Default)]+pub struct Vm {+    instructions: Vec<Instruction>,+    constants: Vec<Literal>,+    targets: Vec<Variable>,+    stack: Vec<Value>,+    parameter_stack: Vec<Option<Value>>,+    error: Option<Error>,+    instruction_pointer: usize,+}+```++- instructions++The instructions field is a `Vec` of `Instruction`. An instruction can be+either an OpCode or some data for that OpCode. For example, the instructions+`[.., OpCode(Constant), Literal(12), ..]` when evaluated will load the constant+stored in the `values` `Vec` that is found in position 12 onto the stack.++- constants++A list of constant values found in the program. Since the bytecode only+contains integers any actual values must be stored here. This also allows+literals to be deduped.++- targets++A list of paths used in the program, similar to `values`.++- stack++The Vm is a stack based Vm. Every expression that is evaluated pushes the+result on the stack. Every operation pulls the values it uses from the stack.++- parameter_stack++For function calls, the parameters need to be evaluated and passed to the+function. The stack is a `Vec` of parameter name and value. We need an OpCode+that will copy a value from the main stack to the parameter stack tagged with+the parameter name. The `FunctionArgs` compiler will dump this OpCode after the+code to evaluate the given arg is dumped.++- error++If an operation errors, it needs to populate this field. Operations such as+the error assignment (`result, error = thing()`) will need to check if this+field has been populated and action accordingly.++- instruction_pointer++The instruction pointer points to the next instruction to evaluate.++#### Calling functions++Calling functions in the stdlib will be a case of evaluating each parameter+with the results pushed onto the parameter stack.++Since VRL allows for named parameters, parameters need to be pushed on to the+parameter stack _in the order that they are specified in the `Function`+implementation_.++Parameters are optional and may not be specified. However, to avoid passing+parameters to the wrong function, an OpCode still needs to be emitted to move+a placeholder value - `None` - to the parameter stack.++Take the following hypothetical situation:++```coffeescript+thing(surname: "nong", name: thang(name: "nork"))+```++Supposing function `thang` also took an optional parameter `surname`, when+`thang` is called, we will have a `surname` parameter on the parameter stack.+Without the placeholder, `thang` would thing that `surname` was being passed to+it. If the placeholder is pushed, it would consume this instead.++We do not want `stdlib` functions to have access to the VM since that risks a+rogue function destroying the VM's state.++Current `stdlib` functions are composed of a combination of `compile` and+`resolve` functions. These function will need to be combined into a single+function `call`.++The `ArgumentList` parameter that is passed into `call` will have access to the+parameter stack and the parameter list exposed by the `Function`. This can use+these to return the appropriate values for the `required`, `optional` etc..+functions.++Since the `compile` function was also used by some `stdlib` functions to+validate the parameters we may alse need a function `validate_parameters` that+these functions can use to perform extra validation on the parameters.++### Debugging++We need debugging tools to help us introspect and debug the compiled bytecode.++#### Dissassembler++Output the compiled code in readable format to the console.++#### Step debugger++Output each step of the VM to the console, print the Stack and other state+variables in the VM. Allow the user to press a key to move to the next step.++#### Linter++Check each instruction to ensure:++- each Jump Opcode jumps to a valid location in the bytecode.+- each constant references a valid index in the constants list+- each path index references a valid path.++It should be possible to run the linter after every compile (at Vector boot+time), so the linter should run relatively fast.++### Testing++The VM will need thorough testing since it is possible to generate invalid+bytecode. For example, the following bytecode is invalid:++```rust+OpCode(CONSTANT)+Primitive(1)+OpCode(ADD)+```++The `ADD` OpCode expects there to be two values on the stack to add+together. We should never get into the situation where the VRL compiler can+generate this.++With property testing we can generate a vast combination of valid VRL ASTs,+compile this to bytecode and then run the VM to ensure we get no buffer+underflows, invalid constant locations or other invalid bytecode.++Given that implementing the VM involves adding methods to the Expression trait,+it should be possible to run the existing AST alongside the VM. We can then+ensure that we get exactly the same results from each approach.++## Rationale++With each node of the AST compiled down to just a few bytes and all+instructions held in contiguous memory evaluation of the program should be able+to take full advantage of the CPU cache which should result in much faster+execution.++## Drawbacks++Downsides to using a Vm:++- The code is a bit more complex. With an AST that is walked it is fairly+  apparent what the code will be doing at any point. With a Vm, this is not the+  case, it is harder to look at the instructions in the Vm and follow back to+  what part of the VRL code is being evaluated. We will need to write some+  extensive debugging tools to allow for decent introspection into the Vm.++  However, VRL is a very simple language, which does also mean the VM will be+  simple.++- We lose some safety that we get from the Rust compiler. There will need+  to be significant fuzz testing to ensure that the code runs correctly under+  all circumstances.++  It should be noted however that the entire VM will not require any _unsafe_+  Rust code.++- Currently each stdlib function is responsible for evaluating their own+  parameters. This allows parameters to be lazily evaluated. With the Vm, the+  parameters will need to be evaluated up front and the stack passed into the+  function. This could impact performance.++## Prior Art++- GoScript - An implementation of Go using a bytecode Vm,+  https://github.com/oxfeeefeee/goscript++- CPython - https://github.com/python/cpython++## Alternatives++### WASM++Instead of writing our own VM we could transpile VRL code to WASM. WASM is a+mature and performant VM that could give us near native speeds.++The issues with using WASM are that it would require data to be serialized in+order to move between Vector and WASM. This would incur a significant+performance penalty.++WASM provides a lot more functionality and complexity than is required for VRL.+By developing our own VM we will be able to customize it purely for the usecase+required by VRL. This allows us to implement very VRL specific OpCodes which+should allow the bytecode stay simple.++### Represent additional data in the OpCode++The current suggestion represents each OpCode as a single instruction in the+instruction list. An OpCode with it's data occupies two instructions.++For example, with the current suggestion to load a constant three things have+to happen:++1. Add the Constant OpCode to the instructions.+2. Add the actual constant value to the list of constants.+3. Add the index of the constant in the constant list to the list of constants.++#### Include value in the OpCode++These three things could be represented as a single instruction if we included+the constant value in the OpCode:++```rust+enum OpCode {+    ...+    Constant(Box<Value>)+    ...+}+```++This will increase `size_of::<OpCode>` from 8 to 16, potentially doubling the+memory size of the instruction list.++It also means we do not need to store a separate list of constants. On the plus+side, this is one less lookup at runtime to load the constant. On the down side,+we wouldn't be able to dedupe the constant, so for example, if the code uses the+same string twice, it would be represented twice in the bytecode.++#### Bitmask the OpCode data

This is an interesting proposal, and worth exploring in the future, but my initial preference would be to leverage Rust's types and type-safety as much as possible. The suggestion here isn't necessarily "unsafe", but it adds another point where we can mess up (and requires a closed-down API to avoid others from doing the same).

I would probably suggest moving this (and "Include value in the OpCode") out of the "alternatives" section, and into "future work", as I don't think it's either this or the other, but rather we can start safer but potentially a bit slower, and then move towards more "unsafe" solutions that give us an extra few percentages of performance gains.

StephenWakely

comment created time in 12 days

Pull request review commentvectordotdev/vector

chore: add VRL enum VM (VenuM) RFC

+# RFC 9811 - 2021-11-12 - VRL enum VM (VenuM)++This RFC proposes implementing a enum VM. VRL will be compiled to a list of+instructions and executed by this VM, with the aim to significantly improve the+performance of executing VRL programs.++## Context++This RFC is a follow on from+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Scope++### In scope++This RFC is purely about developing a enum VM for VRL. It will discuss the+risks involved in running VRL with a VM and the measures we will take to+mitigate those risks.++### Out of scope++Any other performance issues relating to VRL will be discussed in+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Pain++Vrl compiles to an AST that is then walked during resolution. Each node in that+tree is boxed and stored in disparate regions of memory. As a result walking+the tree means that the CPU caches must be constantly swapped.++## Proposal++### Implementation++Instead we can create an enum VM to store the execution of the Vrl program.++The enum is essentially a big enum of instructions:++```rust+#[derive(Copy, Clone, Debug, PartialEq, Eq)]+pub enum OpCode {+    Return,+    Constant,+    Negate,+    Add,+    Subtract,+    Multiply,+    Divide,+    Print,+    Not,+    Greater,+    GreaterEqual,+    Less,+    LessEqual,+    NotEqual,+    Equal,+    Pop,+    JumpIfFalse,+    Jump,+    SetPath,+    GetPath,+    Call,+    ...+}++enum Instruction {+    Opcode(Opcode),+    Literal(LiteralIndex),+}++pub struct LiteralIndex(usize);+```++The Vm is a struct comprising of the following fields:++```rust+#[derive(Clone, Debug, Default)]+pub struct Vm {+    instructions: Vec<Instruction>,+    constants: Vec<Literal>,+    targets: Vec<Variable>,+    stack: Vec<Value>,+    parameter_stack: Vec<Option<Value>>,+    error: Option<Error>,+    instruction_pointer: usize,+}+```++- instructions++The instructions field is a `Vec` of `Instruction`. An instruction can be+either an OpCode or some data for that OpCode. For example, the instructions+`[.., OpCode(Constant), Literal(12), ..]` when evaluated will load the constant+stored in the `values` `Vec` that is found in position 12 onto the stack.++- constants++A list of constant values found in the program. Since the bytecode only+contains integers any actual values must be stored here. This also allows+literals to be deduped.++- targets++A list of paths used in the program, similar to `values`.++- stack++The Vm is a stack based Vm. Every expression that is evaluated pushes the+result on the stack. Every operation pulls the values it uses from the stack.++- parameter_stack++For function calls, the parameters need to be evaluated and passed to the+function. The stack is a `Vec` of parameter name and value. We need an OpCode+that will copy a value from the main stack to the parameter stack tagged with+the parameter name. The `FunctionArgs` compiler will dump this OpCode after the+code to evaluate the given arg is dumped.++- error++If an operation errors, it needs to populate this field. Operations such as+the error assignment (`result, error = thing()`) will need to check if this+field has been populated and action accordingly.++- instruction_pointer++The instruction pointer points to the next instruction to evaluate.++#### Calling functions++Calling functions in the stdlib will be a case of evaluating each parameter+with the results pushed onto the parameter stack.++Since VRL allows for named parameters, parameters need to be pushed on to the+parameter stack _in the order that they are specified in the `Function`+implementation_.++Parameters are optional and may not be specified. However, to avoid passing+parameters to the wrong function, an OpCode still needs to be emitted to move+a placeholder value - `None` - to the parameter stack.++Take the following hypothetical situation:++```coffeescript+thing(surname: "nong", name: thang(name: "nork"))+```++Supposing function `thang` also took an optional parameter `surname`, when+`thang` is called, we will have a `surname` parameter on the parameter stack.+Without the placeholder, `thang` would thing that `surname` was being passed to+it. If the placeholder is pushed, it would consume this instead.++We do not want `stdlib` functions to have access to the VM since that risks a+rogue function destroying the VM's state.++Current `stdlib` functions are composed of a combination of `compile` and+`resolve` functions. These function will need to be combined into a single+function `call`.++The `ArgumentList` parameter that is passed into `call` will have access to the+parameter stack and the parameter list exposed by the `Function`. This can use+these to return the appropriate values for the `required`, `optional` etc..+functions.++Since the `compile` function was also used by some `stdlib` functions to+validate the parameters we may alse need a function `validate_parameters` that+these functions can use to perform extra validation on the parameters.++### Debugging++We need debugging tools to help us introspect and debug the compiled bytecode.++#### Dissassembler++Output the compiled code in readable format to the console.++#### Step debugger++Output each step of the VM to the console, print the Stack and other state+variables in the VM. Allow the user to press a key to move to the next step.++#### Linter++Check each instruction to ensure:++- each Jump Opcode jumps to a valid location in the bytecode.+- each constant references a valid index in the constants list+- each path index references a valid path.++It should be possible to run the linter after every compile (at Vector boot+time), so the linter should run relatively fast.++### Testing++The VM will need thorough testing since it is possible to generate invalid+bytecode. For example, the following bytecode is invalid:++```rust+OpCode(CONSTANT)+Primitive(1)+OpCode(ADD)+```++The `ADD` OpCode expects there to be two values on the stack to add+together. We should never get into the situation where the VRL compiler can+generate this.++With property testing we can generate a vast combination of valid VRL ASTs,+compile this to bytecode and then run the VM to ensure we get no buffer+underflows, invalid constant locations or other invalid bytecode.++Given that implementing the VM involves adding methods to the Expression trait,+it should be possible to run the existing AST alongside the VM. We can then+ensure that we get exactly the same results from each approach.++## Rationale++With each node of the AST compiled down to just a few bytes and all+instructions held in contiguous memory evaluation of the program should be able+to take full advantage of the CPU cache which should result in much faster+execution.++## Drawbacks++Downsides to using a Vm:++- The code is a bit more complex. With an AST that is walked it is fairly+  apparent what the code will be doing at any point. With a Vm, this is not the+  case, it is harder to look at the instructions in the Vm and follow back to+  what part of the VRL code is being evaluated. We will need to write some+  extensive debugging tools to allow for decent introspection into the Vm.++  However, VRL is a very simple language, which does also mean the VM will be+  simple.++- We lose some safety that we get from the Rust compiler. There will need+  to be significant fuzz testing to ensure that the code runs correctly under+  all circumstances.++  It should be noted however that the entire VM will not require any _unsafe_+  Rust code.++- Currently each stdlib function is responsible for evaluating their own+  parameters. This allows parameters to be lazily evaluated. With the Vm, the+  parameters will need to be evaluated up front and the stack passed into the+  function. This could impact performance.++## Prior Art++- GoScript - An implementation of Go using a bytecode Vm,+  https://github.com/oxfeeefeee/goscript++- CPython - https://github.com/python/cpython++## Alternatives++### WASM++Instead of writing our own VM we could transpile VRL code to WASM. WASM is a+mature and performant VM that could give us near native speeds.++The issues with using WASM are that it would require data to be serialized in+order to move between Vector and WASM. This would incur a significant+performance penalty.

I mentioned this in our meeting, but this isn't actually accurate. Wasm allows both the host and the Wasm module to mutably access the same chunk of memory.

See, for example, the wasmer crate.

It's true, though, that we'll need wrappers to represent our types, but there's also a lot of ongoing work with Wasm interface types to make this simpler.

Still, I agree with the conclusion that this is a step too far for a first iteration, but I want to avoid that conclusion being based on the wrong assumptions.

StephenWakely

comment created time in 12 days

issue commentvectordotdev/vector

VRL: add `parse_json_object` as a convenient way to parse JSON objects

I think you meant this?

ok, err = parse_json(.message)
# "function call error for \"parse_json\" at (10:30): expected \"string\", got \"null\""

You were using the ! modifier, thus indeed making the call infallible.

No, I meant it as I wrote it. The ! in parse_json! is there to catch the error at runtime when the field isn't a JSON-encoded string. After that, you can get a non-object back, which is why I proposed catching that with ok, err |= ....

JeanMertz

comment created time in 13 days

issue commentvectordotdev/vector

VRL: add `parse_json_object` as a convenient way to parse JSON objects

Ah, actually, the above wouldn't work because the error-assign operation assigns the "empty" value to the ok target if the rhs expression fails.

So in this case, after failing a parse, if parse_json where to result in a boolean, then . would be assigned the false value, which isn't allowed to be assigned to ..

So yeah, you'd have to do it in two steps:

value, _ |= parse_json!(.message)
. = object!(value)

Which wins you exactly... nothing 😅

I wonder if we should prevent the err-assign operation to assign a value to the ok target if that target already exists. It would be a breaking change, and it might be confusing to people now that I think about it, but it would at least solve the above issue.

JeanMertz

comment created time in 13 days

issue commentvectordotdev/vector

VRL: add `parse_json_object` as a convenient way to parse JSON objects

To be clear, you can still use array!(parse_json(.)), which is a bit less ergonomic, but its usage is probably not high enough that parse_json_array should be needed.

I'm also not entirely on board with parse_json_object yet, there might be a higher level solution to solve this problem.

One potential solution I was thinking about is to have the error-assign operation catch invalid merge assignments, that is:

ok, err |= parse_json!(.message)

Currently, this isn't valid:

error[E104]: unnecessary error assignment
  ┌─ :1:5
  │
1 │ ok, err |= parse_json!(.message)
  │ --  ^^^    --------------------- because this expression can't fail
  │ │   │
  │ │   this error assignment is unnecessary
  │
  = see documentation about error handling at https://errors.vrl.dev/#handling
  = learn more about error code 104 at https://errors.vrl.dev/104
  = see language documentation at https://vrl.dev

But we could make it so that the error assignment catches an invalid merge-assign operation if the rhs type isn't an object at runtime.

So instead of this:

. |= parse_json!(.message)

You could write:

# do something with `err` at runtime
., err |= parse_json!(.message)

Or event:

# ignore `err` at runtime
., _ |= parse_json!(.message)

This is one potential solution that came to mind, there might be others as well.

JeanMertz

comment created time in 13 days

Pull request review commentvectordotdev/vector

chore: add VRL bytecode VM RFC

+# RFC 9811 - 2021-11-12 - VRL bytecode VM++This RFC proposes implementing a bytecode VM. VRL will be compiled to the bytecode+and executed by this VM, with the aim to significantly improve the performance of+executing VRL scripts.++## Context++This RFC is a follow on from+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Scope++### In scope++This RFC is purely about developing a bytecode VM for VRL.++### Out of scope++Any other performance issues relating to VRL will be discussed in+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Proposal++### Implementation++Vrl compiles to an AST that is then walked during resolution. Each node in that+tree is boxed and stored in disparate regions of memory. As a result walking+the tree means that the CPU caches must be constantly swapped.++Instead we can create a bytecode VM to store the execution of the Vrl program.++Bytecode is essentially a big enum of instructions (that are represented by an integer+using the [num-derive](https://crates.io/crates/num-derive),+[num-traits](https://crates.io/crates/num-traits) crates):++```rust+#[derive(FromPrimitive, ToPrimitive, Copy, Clone, Debug, PartialEq, Eq)]+pub enum OpCode {+    Return = 255,+    Constant,+    Negate,+    Add,+    Subtract,+    Multiply,+    Divide,+    Print,+    Not,+    Greater,+    GreaterEqual,+    Less,+    LessEqual,+    NotEqual,+    Equal,+    Pop,+    JumpIfFalse,+    Jump,+    SetPath,+    GetPath,+    Call,+    ...+}+```++The Vm is a struct comprising of the following fields:++```rust+#[derive(Clone, Debug, Default)]+pub struct Vm {+    instructions: Vec<usize>,+    values: Vec<Literal>,+    targets: Vec<Variable>,+    stack: Vec<Value>,+    ip: usize,+}+```++- instructions++The instructions field is a `Vec` of `OpCode` cast to a usize. The reason for+the cast is because not all instructions are `OpCode`. For example the+instructions `[.., Constant, 12, ..]` when evaluated will load the constant+stored in the `values` `Vec` that is found in position 12 onto the stack.++- values++A list of constant values found in the program. Since the bytecode only+contains integers any actual values must be stored here. This also allows+literals to be deduped.++- targets++A list of paths used in the program, similar to `values`.++- stack++The Vm is a stack based Vm. Every expression that is evaluated pushes the+result on the stack. Every operation pulls the values it uses from the stack.++- ip++The instruction pointer points to the next instruction to evaluate.++With each node of the AST compiled down to just a few bytes and all+instructions held in contiguous memory evaluation of the program should be able+to take full advantage of the CPU cache which should result in much faster+execution.+++#### Calling functions

I'd like this to be fleshed out more, so we can get a sense of what would be required for function implementors to use this new implementation. The goal would be to have minimal/no changes be needed, but whatever changes are needed should be listed here, and we should describe how we envision making the API exposed to function authors a safe one.

StephenWakely

comment created time in 17 days

Pull request review commentvectordotdev/vector

chore: add VRL bytecode VM RFC

+# RFC 9811 - 2021-11-12 - VRL bytecode VM++This RFC proposes implementing a bytecode VM. VRL will be compiled to the bytecode+and executed by this VM, with the aim to significantly improve the performance of+executing VRL scripts.++## Context++This RFC is a follow on from+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Scope++### In scope++This RFC is purely about developing a bytecode VM for VRL.

I'd like this RFC to be more explicit about the goals around making this as safe and easy to use. That includes:

  1. Documentation/code comments
  2. Testing (unit/fuzzing/...)
  3. Preventing misuse through a safe API
StephenWakely

comment created time in 17 days

Pull request review commentvectordotdev/vector

chore: add VRL bytecode VM RFC

+# RFC 9811 - 2021-11-12 - VRL bytecode VM++This RFC proposes implementing a bytecode VM. VRL will be compiled to the bytecode+and executed by this VM, with the aim to significantly improve the performance of

We should add a section to this RFC discussing (and proving) the performance gains we've seen from the early spike, as those gains are what warrants the effort needed to implement this RFC.

StephenWakely

comment created time in 17 days

Pull request review commentvectordotdev/vector

chore: add VRL bytecode VM RFC

+# RFC 9811 - 2021-11-12 - VRL bytecode VM++This RFC proposes implementing a bytecode VM. VRL will be compiled to the bytecode+and executed by this VM, with the aim to significantly improve the performance of+executing VRL scripts.++## Context++This RFC is a follow on from+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Scope++### In scope++This RFC is purely about developing a bytecode VM for VRL.++### Out of scope++Any other performance issues relating to VRL will be discussed in+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Proposal++### Implementation++Vrl compiles to an AST that is then walked during resolution. Each node in that+tree is boxed and stored in disparate regions of memory. As a result walking+the tree means that the CPU caches must be constantly swapped.++Instead we can create a bytecode VM to store the execution of the Vrl program.++Bytecode is essentially a big enum of instructions (that are represented by an integer+using the [num-derive](https://crates.io/crates/num-derive),+[num-traits](https://crates.io/crates/num-traits) crates):++```rust+#[derive(FromPrimitive, ToPrimitive, Copy, Clone, Debug, PartialEq, Eq)]+pub enum OpCode {+    Return = 255,+    Constant,+    Negate,+    Add,+    Subtract,+    Multiply,+    Divide,+    Print,+    Not,+    Greater,+    GreaterEqual,+    Less,+    LessEqual,+    NotEqual,+    Equal,+    Pop,+    JumpIfFalse,+    Jump,+    SetPath,+    GetPath,+    Call,+    ...+}+```++The Vm is a struct comprising of the following fields:++```rust+#[derive(Clone, Debug, Default)]+pub struct Vm {+    instructions: Vec<usize>,+    values: Vec<Literal>,+    targets: Vec<Variable>,+    stack: Vec<Value>,+    ip: usize,+}+```++- instructions++The instructions field is a `Vec` of `OpCode` cast to a usize. The reason for+the cast is because not all instructions are `OpCode`. For example the+instructions `[.., Constant, 12, ..]` when evaluated will load the constant+stored in the `values` `Vec` that is found in position 12 onto the stack.++- values++A list of constant values found in the program. Since the bytecode only+contains integers any actual values must be stored here. This also allows+literals to be deduped.++- targets++A list of paths used in the program, similar to `values`.++- stack++The Vm is a stack based Vm. Every expression that is evaluated pushes the+result on the stack. Every operation pulls the values it uses from the stack.++- ip++The instruction pointer points to the next instruction to evaluate.++With each node of the AST compiled down to just a few bytes and all+instructions held in contiguous memory evaluation of the program should be able+to take full advantage of the CPU cache which should result in much faster+execution.+++#### Calling functions++Calling functions in the stdlib will be a case of evaluating each parameter+with the results pushed onto the stack.++Since Vrl allows for named parameters and optional parameters, the compiler+will need to ensure the bytecode evaluates each parameter in a specific order+(likely the order declared in the function). Bytecode will need to be emmitted+for parameters that are not specified in the Vrl script to add a default value+to the stack.+++### Optimization++With the code as a single dimension array of Bytecode, it could be possible to+scan the code for patterns and reorganise the Bytecode so it can run in a more+optimal way.++A lot more thought and research needs to go into this before we can consider+implementing these changes.++## Drawbacks++Downsides to using a Vm:++- The code is a bit more complex. With an AST that is walked it is fairly apparent+  what the code will be doing at any point. With a Vm, this is not the case, it+  is harder to look at the instructions in the Vm and follow back to what part+  of the Vrl code is being evaluated. We will need to write some extensive+  debugging tools to allow for decent introspection into the Vm.++  However, VRL is a very simple language, which does also mean the VM will be+  simple.

VRL is also still evolving. We're adding iteration soon, and there's no way to know what else we might introduce in the future, so this is somewhat of a moot point, I think.

StephenWakely

comment created time in 17 days

Pull request review commentvectordotdev/vector

chore: add VRL bytecode VM RFC

+# RFC 9811 - 2021-11-12 - VRL bytecode VM++This RFC proposes implementing a bytecode VM. VRL will be compiled to the bytecode+and executed by this VM, with the aim to significantly improve the performance of+executing VRL scripts.++## Context++This RFC is a follow on from+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Scope++### In scope++This RFC is purely about developing a bytecode VM for VRL.++### Out of scope++Any other performance issues relating to VRL will be discussed in+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Proposal++### Implementation++Vrl compiles to an AST that is then walked during resolution. Each node in that+tree is boxed and stored in disparate regions of memory. As a result walking+the tree means that the CPU caches must be constantly swapped.++Instead we can create a bytecode VM to store the execution of the Vrl program.++Bytecode is essentially a big enum of instructions (that are represented by an integer+using the [num-derive](https://crates.io/crates/num-derive),+[num-traits](https://crates.io/crates/num-traits) crates):++```rust+#[derive(FromPrimitive, ToPrimitive, Copy, Clone, Debug, PartialEq, Eq)]+pub enum OpCode {+    Return = 255,+    Constant,+    Negate,+    Add,+    Subtract,+    Multiply,+    Divide,+    Print,+    Not,+    Greater,+    GreaterEqual,+    Less,+    LessEqual,+    NotEqual,+    Equal,+    Pop,+    JumpIfFalse,+    Jump,+    SetPath,+    GetPath,+    Call,+    ...+}+```++The Vm is a struct comprising of the following fields:++```rust+#[derive(Clone, Debug, Default)]+pub struct Vm {+    instructions: Vec<usize>,+    values: Vec<Literal>,+    targets: Vec<Variable>,+    stack: Vec<Value>,+    ip: usize,+}+```++- instructions++The instructions field is a `Vec` of `OpCode` cast to a usize. The reason for+the cast is because not all instructions are `OpCode`. For example the+instructions `[.., Constant, 12, ..]` when evaluated will load the constant+stored in the `values` `Vec` that is found in position 12 onto the stack.++- values++A list of constant values found in the program. Since the bytecode only+contains integers any actual values must be stored here. This also allows+literals to be deduped.++- targets++A list of paths used in the program, similar to `values`.++- stack++The Vm is a stack based Vm. Every expression that is evaluated pushes the+result on the stack. Every operation pulls the values it uses from the stack.++- ip++The instruction pointer points to the next instruction to evaluate.++With each node of the AST compiled down to just a few bytes and all+instructions held in contiguous memory evaluation of the program should be able+to take full advantage of the CPU cache which should result in much faster+execution.+++#### Calling functions++Calling functions in the stdlib will be a case of evaluating each parameter+with the results pushed onto the stack.++Since Vrl allows for named parameters and optional parameters, the compiler+will need to ensure the bytecode evaluates each parameter in a specific order+(likely the order declared in the function). Bytecode will need to be emmitted+for parameters that are not specified in the Vrl script to add a default value+to the stack.+++### Optimization++With the code as a single dimension array of Bytecode, it could be possible to+scan the code for patterns and reorganise the Bytecode so it can run in a more+optimal way.++A lot more thought and research needs to go into this before we can consider+implementing these changes.++## Drawbacks++Downsides to using a Vm:++- The code is a bit more complex. With an AST that is walked it is fairly apparent+  what the code will be doing at any point. With a Vm, this is not the case, it+  is harder to look at the instructions in the Vm and follow back to what part+  of the Vrl code is being evaluated. We will need to write some extensive+  debugging tools to allow for decent introspection into the Vm.

There's no mention of these debugging tools anywhere else in the RFC. It's probably worth elaborating on what you have in mind for this.

StephenWakely

comment created time in 17 days

Pull request review commentvectordotdev/vector

chore: add VRL bytecode VM RFC

+# RFC 9811 - 2021-11-12 - VRL bytecode VM++This RFC proposes implementing a bytecode VM. VRL will be compiled to the bytecode+and executed by this VM, with the aim to significantly improve the performance of+executing VRL scripts.++## Context++This RFC is a follow on from+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Scope++### In scope++This RFC is purely about developing a bytecode VM for VRL.++### Out of scope++Any other performance issues relating to VRL will be discussed in+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Proposal++### Implementation++Vrl compiles to an AST that is then walked during resolution. Each node in that+tree is boxed and stored in disparate regions of memory. As a result walking+the tree means that the CPU caches must be constantly swapped.++Instead we can create a bytecode VM to store the execution of the Vrl program.++Bytecode is essentially a big enum of instructions (that are represented by an integer+using the [num-derive](https://crates.io/crates/num-derive),+[num-traits](https://crates.io/crates/num-traits) crates):++```rust+#[derive(FromPrimitive, ToPrimitive, Copy, Clone, Debug, PartialEq, Eq)]+pub enum OpCode {+    Return = 255,+    Constant,+    Negate,+    Add,+    Subtract,+    Multiply,+    Divide,+    Print,+    Not,+    Greater,+    GreaterEqual,+    Less,+    LessEqual,+    NotEqual,+    Equal,+    Pop,+    JumpIfFalse,+    Jump,+    SetPath,+    GetPath,+    Call,+    ...+}+```++The Vm is a struct comprising of the following fields:++```rust+#[derive(Clone, Debug, Default)]+pub struct Vm {+    instructions: Vec<usize>,+    values: Vec<Literal>,+    targets: Vec<Variable>,+    stack: Vec<Value>,+    ip: usize,+}+```++- instructions++The instructions field is a `Vec` of `OpCode` cast to a usize. The reason for+the cast is because not all instructions are `OpCode`. For example the+instructions `[.., Constant, 12, ..]` when evaluated will load the constant+stored in the `values` `Vec` that is found in position 12 onto the stack.++- values++A list of constant values found in the program. Since the bytecode only+contains integers any actual values must be stored here. This also allows+literals to be deduped.++- targets++A list of paths used in the program, similar to `values`.++- stack++The Vm is a stack based Vm. Every expression that is evaluated pushes the+result on the stack. Every operation pulls the values it uses from the stack.++- ip++The instruction pointer points to the next instruction to evaluate.++With each node of the AST compiled down to just a few bytes and all+instructions held in contiguous memory evaluation of the program should be able+to take full advantage of the CPU cache which should result in much faster+execution.

This should be part of the "Rationale" chapter (it's probably worth taking another look at the template as I've added a few other comments on rejigging a few paragraphs.

StephenWakely

comment created time in 17 days

Pull request review commentvectordotdev/vector

chore: add VRL bytecode VM RFC

+# RFC 9811 - 2021-11-12 - VRL bytecode VM++This RFC proposes implementing a bytecode VM. VRL will be compiled to the bytecode+and executed by this VM, with the aim to significantly improve the performance of+executing VRL scripts.++## Context++This RFC is a follow on from+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Scope++### In scope++This RFC is purely about developing a bytecode VM for VRL.++### Out of scope++Any other performance issues relating to VRL will be discussed in+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Proposal++### Implementation++Vrl compiles to an AST that is then walked during resolution. Each node in that+tree is boxed and stored in disparate regions of memory. As a result walking+the tree means that the CPU caches must be constantly swapped.++Instead we can create a bytecode VM to store the execution of the Vrl program.++Bytecode is essentially a big enum of instructions (that are represented by an integer+using the [num-derive](https://crates.io/crates/num-derive),+[num-traits](https://crates.io/crates/num-traits) crates):++```rust+#[derive(FromPrimitive, ToPrimitive, Copy, Clone, Debug, PartialEq, Eq)]+pub enum OpCode {+    Return = 255,+    Constant,+    Negate,+    Add,+    Subtract,+    Multiply,+    Divide,+    Print,+    Not,+    Greater,+    GreaterEqual,+    Less,+    LessEqual,+    NotEqual,+    Equal,+    Pop,+    JumpIfFalse,+    Jump,+    SetPath,+    GetPath,+    Call,+    ...+}+```++The Vm is a struct comprising of the following fields:++```rust+#[derive(Clone, Debug, Default)]+pub struct Vm {

As discussed in our meeting, we'll need to update this to be able to keep track of errors.

StephenWakely

comment created time in 17 days

Pull request review commentvectordotdev/vector

chore: add VRL bytecode VM RFC

+# RFC 9811 - 2021-11-12 - VRL bytecode VM++This RFC proposes implementing a bytecode VM. VRL will be compiled to the bytecode+and executed by this VM, with the aim to significantly improve the performance of+executing VRL scripts.++## Context++This RFC is a follow on from+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Scope++### In scope++This RFC is purely about developing a bytecode VM for VRL.++### Out of scope++Any other performance issues relating to VRL will be discussed in+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Proposal++### Implementation++Vrl compiles to an AST that is then walked during resolution. Each node in that+tree is boxed and stored in disparate regions of memory. As a result walking+the tree means that the CPU caches must be constantly swapped.++Instead we can create a bytecode VM to store the execution of the Vrl program.++Bytecode is essentially a big enum of instructions (that are represented by an integer+using the [num-derive](https://crates.io/crates/num-derive),+[num-traits](https://crates.io/crates/num-traits) crates):++```rust+#[derive(FromPrimitive, ToPrimitive, Copy, Clone, Debug, PartialEq, Eq)]+pub enum OpCode {+    Return = 255,+    Constant,+    Negate,+    Add,+    Subtract,+    Multiply,+    Divide,+    Print,+    Not,+    Greater,+    GreaterEqual,+    Less,+    LessEqual,+    NotEqual,+    Equal,+    Pop,+    JumpIfFalse,+    Jump,+    SetPath,+    GetPath,+    Call,+    ...+}+```++The Vm is a struct comprising of the following fields:++```rust+#[derive(Clone, Debug, Default)]+pub struct Vm {+    instructions: Vec<usize>,+    values: Vec<Literal>,+    targets: Vec<Variable>,+    stack: Vec<Value>,+    ip: usize,+}+```++- instructions++The instructions field is a `Vec` of `OpCode` cast to a usize. The reason for+the cast is because not all instructions are `OpCode`. For example the+instructions `[.., Constant, 12, ..]` when evaluated will load the constant+stored in the `values` `Vec` that is found in position 12 onto the stack.

In the vein of what we discussed during our get-together, I'd like to get as much safety as possible through Rust's type system, including introducing wrapper types to avoid misusing the VM API.

For example, for instructions, we could/should codify the invariants a bit more in the type system using something like:

enum Instruction {
	Opcode(Opcode),
	Literal(LiteralIndex),
}

pub struct LiteralIndex(usize);

There are more examples like that, we don't have to have consensus on all of them in this RFC, but we should have a section that explicitly mentions this as a requirement to successfully implementing the RFC, we can then iterate on the exact shape of those APIs in the implementation PRs.

StephenWakely

comment created time in 17 days

Pull request review commentvectordotdev/vector

chore: add VRL bytecode VM RFC

+# RFC 9811 - 2021-11-12 - VRL bytecode VM++This RFC proposes implementing a bytecode VM. VRL will be compiled to the bytecode+and executed by this VM, with the aim to significantly improve the performance of+executing VRL scripts.++## Context++This RFC is a follow on from+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Scope++### In scope++This RFC is purely about developing a bytecode VM for VRL.++### Out of scope++Any other performance issues relating to VRL will be discussed in+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Proposal++### Implementation++Vrl compiles to an AST that is then walked during resolution. Each node in that+tree is boxed and stored in disparate regions of memory. As a result walking+the tree means that the CPU caches must be constantly swapped.++Instead we can create a bytecode VM to store the execution of the Vrl program.++Bytecode is essentially a big enum of instructions (that are represented by an integer+using the [num-derive](https://crates.io/crates/num-derive),+[num-traits](https://crates.io/crates/num-traits) crates):++```rust+#[derive(FromPrimitive, ToPrimitive, Copy, Clone, Debug, PartialEq, Eq)]+pub enum OpCode {+    Return = 255,+    Constant,+    Negate,+    Add,+    Subtract,+    Multiply,+    Divide,+    Print,+    Not,+    Greater,+    GreaterEqual,+    Less,+    LessEqual,+    NotEqual,+    Equal,+    Pop,+    JumpIfFalse,+    Jump,+    SetPath,+    GetPath,+    Call,+    ...+}+```++The Vm is a struct comprising of the following fields:++```rust+#[derive(Clone, Debug, Default)]+pub struct Vm {+    instructions: Vec<usize>,+    values: Vec<Literal>,+    targets: Vec<Variable>,+    stack: Vec<Value>,+    ip: usize,+}+```++- instructions++The instructions field is a `Vec` of `OpCode` cast to a usize. The reason for+the cast is because not all instructions are `OpCode`. For example the+instructions `[.., Constant, 12, ..]` when evaluated will load the constant+stored in the `values` `Vec` that is found in position 12 onto the stack.++- values++A list of constant values found in the program. Since the bytecode only+contains integers any actual values must be stored here. This also allows+literals to be deduped.++- targets++A list of paths used in the program, similar to `values`.++- stack++The Vm is a stack based Vm. Every expression that is evaluated pushes the+result on the stack. Every operation pulls the values it uses from the stack.++- ip++The instruction pointer points to the next instruction to evaluate.++With each node of the AST compiled down to just a few bytes and all+instructions held in contiguous memory evaluation of the program should be able+to take full advantage of the CPU cache which should result in much faster+execution.+++#### Calling functions++Calling functions in the stdlib will be a case of evaluating each parameter+with the results pushed onto the stack.++Since Vrl allows for named parameters and optional parameters, the compiler+will need to ensure the bytecode evaluates each parameter in a specific order+(likely the order declared in the function). Bytecode will need to be emmitted+for parameters that are not specified in the Vrl script to add a default value+to the stack.+++### Optimization++With the code as a single dimension array of Bytecode, it could be possible to+scan the code for patterns and reorganise the Bytecode so it can run in a more+optimal way.++A lot more thought and research needs to go into this before we can consider+implementing these changes.++## Drawbacks++Downsides to using a Vm:++- The code is a bit more complex. With an AST that is walked it is fairly apparent+  what the code will be doing at any point. With a Vm, this is not the case, it+  is harder to look at the instructions in the Vm and follow back to what part+  of the Vrl code is being evaluated. We will need to write some extensive+  debugging tools to allow for decent introspection into the Vm.++  However, VRL is a very simple language, which does also mean the VM will be+  simple.++- We lose some safety that we get from the Rust compiler. There will need+  to be significant fuzz testing to ensure that the code runs correctly under+  all circumstances. It should be noted however that the entire VM will not+  require any _unsafe_ code.++- Currently each stdlib function is responsible for evaluating their own+  parameters. This allows parameters to be lazily evaluated. Most likely with a+  Vm, the parameters will need to be evaluated up front and the stack passed+  into the function. This could impact performance.++## Prior Art++- Goscript - An implementation of Go using a bytecode Vm,+  https://github.com/oxfeeefeee/goscript++- CPython - https://github.com/python/cpython++## Plan Of Attack++- [ ] Submit a PR with spike-level code _roughly_ demonstrating the change for+      the VM.

This needs to be fleshed out more.

Specifically, I'd like to see the approach we discussed listed here:

  1. Introduce/implement the VM itself, without updating any of the expressions. This includes tests (unit/property), documentation/code comments, and an as-safe-as-possible API surrounding the VM operations.
  2. Update one or two expressions to set the tone for future work of updating all the other expressions and functions (if needed)
  3. Divvy up the work to update all other expressions(/functions).
StephenWakely

comment created time in 17 days

Pull request review commentvectordotdev/vector

chore: add VRL bytecode VM RFC

+# RFC 9811 - 2021-11-12 - VRL bytecode VM++This RFC proposes implementing a bytecode VM. VRL will be compiled to the bytecode+and executed by this VM, with the aim to significantly improve the performance of+executing VRL scripts.
executing VRL programs.
StephenWakely

comment created time in 17 days

Pull request review commentvectordotdev/vector

chore: add VRL bytecode VM RFC

+# RFC 9811 - 2021-11-12 - VRL bytecode VM++This RFC proposes implementing a bytecode VM. VRL will be compiled to the bytecode+and executed by this VM, with the aim to significantly improve the performance of+executing VRL scripts.++## Context++This RFC is a follow on from+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Scope++### In scope++This RFC is purely about developing a bytecode VM for VRL.++### Out of scope++Any other performance issues relating to VRL will be discussed in+[rfcs/2021-10-14-9811-vrl-performance.md](https://github.com/vectordotdev/vector/pull/9812)++## Proposal++### Implementation++Vrl compiles to an AST that is then walked during resolution. Each node in that+tree is boxed and stored in disparate regions of memory. As a result walking+the tree means that the CPU caches must be constantly swapped.++Instead we can create a bytecode VM to store the execution of the Vrl program.++Bytecode is essentially a big enum of instructions (that are represented by an integer+using the [num-derive](https://crates.io/crates/num-derive),+[num-traits](https://crates.io/crates/num-traits) crates):++```rust+#[derive(FromPrimitive, ToPrimitive, Copy, Clone, Debug, PartialEq, Eq)]+pub enum OpCode {+    Return = 255,+    Constant,+    Negate,+    Add,+    Subtract,+    Multiply,+    Divide,+    Print,+    Not,+    Greater,+    GreaterEqual,+    Less,+    LessEqual,+    NotEqual,+    Equal,+    Pop,+    JumpIfFalse,+    Jump,+    SetPath,+    GetPath,+    Call,+    ...+}+```++The Vm is a struct comprising of the following fields:++```rust+#[derive(Clone, Debug, Default)]+pub struct Vm {+    instructions: Vec<usize>,+    values: Vec<Literal>,+    targets: Vec<Variable>,+    stack: Vec<Value>,+    ip: usize,

I know this is just an example for this RFC, but I strongly recommend using full names instead of acronyms, since not many people might/will be familiar with the terminology used.

StephenWakely

comment created time in 17 days

more