Add constant folding and lookup function to transform plugin

plugin/action/transform/README.idoc.md

@@ -5,7 +5,178 @@
 @config-params|description
 
 ## Language
-@language
+A program is a list of expressions executed top to bottom for every event.
+Expressions are separated by newlines, or by `;` when written on one line.
+Comments start with `#` and last to the end of the line.
+
+### Event fields
+
+Paths address fields of the current event and always start with a dot:
+
+```
+.level                    # top-level field
+.user.name                # nested field
+."key with spaces"        # quoted field name
+.items[0]                 # array element
+.items[-1]                # array element, counted from the end
+.items[i]                 # index from a variable
+```
+
+Reading a missing path returns `null` — it is never an error, so checks like
+`if .user.name == null { ... }` are safe on any event.
+
+Assigning to a path writes the field and creates missing parent objects on the way:
+
+```
+.a.b.c = 1                # {} -> {"a":{"b":{"c":1}}}
+```
+
+`del` removes a field (no-op when the field does not exist):
+
+```
+del .user.password
+```
+
+Path dots must be written tightly: `.a.b` is one path, while `.a .b` is a compile error.
+
+### Values and literals
+
+| type | literals |
+|------|----------|
+| integer | `42`, `0` |
+| float | `3.14`, `1e10`, `1.5e-3` |
+| string | `"hello"` with `\n`, `\t`, `\"` escapes |
+| raw string | `s'C:\new\path'` — backslashes kept as is |
+| regex | `r'\d+'` — compiled at startup |
+| timestamp | `t'2024-01-15T10:30:00Z'` — RFC3339, `2006-01-02T15:04:05` or `2006-01-02` |
+| bool | `true`, `false` |
+| null | `null` |
+| array | `[1, "two", true]` |
+| object | `{level: "info", "other key": 2}` |
+
+In conditions `null` and `false` are falsy; every other value is truthy.
+
+### Variables
+
+Variables hold intermediate values and live for one event:
+
+```
+name = .user.name             # read a field into a variable
+parts = capture(.log, r'...') # keep a function result
+.out = name                   # write it back to the event
+```
+
+Fields of object values are accessed with a dot or an index; both forms are
+assignable, and arrays grow with `null`s when assigned past their end:
+
+```
+parts.level                   # same as parts["level"]
+arr[0] = 1
+obj.key = "value"
+```
+
+Member access dots follow the same rule as paths: `m.level` is member access,
+`m .level` is a compile error.
+
+### Operators
+
+In order of increasing precedence:
+
+| operators | meaning |
+|-----------|---------|
+| `=` | assignment, right-associative: `a = b = 1` |
+| <code>&#124;&#124;</code> | logical or, short-circuit |
+| `&&` | logical and, short-circuit |
+| `==` `!=` | equality (integers and floats compare numerically) |
+| `<` `<=` `>` `>=` | comparison of numbers, strings or timestamps |
+| `+` `-` | addition, subtraction; `+` also concatenates strings |
+| `*` `/` `%` | multiplication, division, modulo; division by zero is a runtime error |
+| `!` `-` (unary) | negation |
+| `f()` `a[i]` `a.b` | call, index, member access |
+
+`+` concatenates only strings with strings — convert other values first:
+`"code " + string(.code)`.
+
+### Control flow
+
+```
+if .status >= 500 {
+  .severity = "crit"
+} else if .status >= 400 {
+  .severity = "warn"
+} else {
+  .severity = "ok"
+}
+
+for i, item in .items {      # iterate an array; use _ to skip a variable
+  .items[i] = item
+}
+
+if .level == "DEBUG" {
+  abort                      # stop the program for this event
+}
+```
+
+`abort` only stops the transform program — the event itself continues down the pipeline.
 
 ## Functions
-@functions
+Functions are called with positional and named arguments. Positional arguments
+are required; named arguments are optional and fall back to their defaults:
+`fn(value, option: "x")`.
+
++ `upcase(value)` — returns the string in upper case.
+  ```
+  .level = upcase(.level)    # "info" -> "INFO"
+  ```
+
++ `string(value)` — converts any value to its string representation; `null`
+  becomes an empty string. Use it to build strings from non-string fields:
+  ```
+  .msg = "code is " + string(.code)
+  ```
+
++ `capture(value, pattern)` — matches the string against a regular expression
+  and returns an object of its named groups `(?P<name>...)`, or `null` when the
+  value does not match (unnamed groups are ignored):
+  ```
+  m = capture(.log, r'^(?P<level>\S+)\s+(?P<message>.+)$')
+  if m != null {
+    .level = m.level
+    .message = m.message
+  }
+  ```
+
++ `after(value, separator)` — returns everything after the first occurrence of
+  `separator`; the value is returned unchanged when the separator is not found.
+  ```
+  .message = after(.log, " - ")
+  ```
+
++ `before(value, separator)` — returns everything before the first occurrence of
+  `separator`; unchanged when not found.
+  ```
+  .level = before(.log, " ")
+  ```
+
++ `between(value, open, close)` — returns the text between the first `open` and
+  the following `close`; unchanged when either delimiter is not found.
+  ```
+  .shard = between(.log, "[", "]")
+  ```
+
++ `lookup(value, table, default: <unchanged>)` — translates a value through a
+  table of replacements. It turns enumeration codes into readable names without
+  a chain of `if`s:
+  ```
+  api_key = {"0": "produce", "1": "fetch", "2": "offsets"}
+  .kafka_request_api_key = lookup(.kafka_request_api_key, api_key)
+  ```
+  Keys are matched by their string form, so the number `0` and the string `"0"`
+  are the same key — JSON writes codes both ways. A value that is not in the
+  table is returned unchanged; pass `default:` to replace it instead:
+  ```
+  .severity = lookup(.status, {"500": "crit", "400": "warn"}, default: "ok")
+  ```
+  A table written as a literal is built once at startup, not per event, so a
+  large table costs no more than a small one. Keep it in a variable when the
+  same table is used more than once.

plugin/action/transform/README.md

@@ -209,4 +209,21 @@ are required; named arguments are optional and fall back to their defaults:
   .shard = between(.log, "[", "]")
   ```
 
++ `lookup(value, table, default: <unchanged>)` — translates a value through a
+  table of replacements. It turns enumeration codes into readable names without
+  a chain of `if`s:
+  ```
+  api_key = {"0": "produce", "1": "fetch", "2": "offsets"}
+  .kafka_request_api_key = lookup(.kafka_request_api_key, api_key)
+  ```
+  Keys are matched by their string form, so the number `0` and the string `"0"`
+  are the same key — JSON writes codes both ways. A value that is not in the
+  table is returned unchanged; pass `default:` to replace it instead:
+  ```
+  .severity = lookup(.status, {"500": "crit", "400": "warn"}, default: "ok")
+  ```
+  A table written as a literal is built once at startup, not per event, so a
+  large table costs no more than a small one. Keep it in a variable when the
+  same table is used more than once.
+
 <br>*Generated using [__insane-doc__](https://github.com/vitkovskii/insane-doc)*

plugin/action/transform/compiler/fold.go

@@ -0,0 +1,141 @@
+package compiler
+
+import (
+	"github.com/ozontech/file.d/plugin/action/transform/core"
+)
+
+// Constant folding computes the value of constant sub-expressions once, at
+// startup, and stores it in a core.ConstExpr that replaces them in the AST.
+//
+// It exists so that data prepared for a function - a lookup table written as an
+// object literal, most of all - is built a single time instead of on every
+// event: core.ObjectExpr.Eval allocates a fresh map on every evaluation.
+//
+// Folding runs inside the validation walk (see validateExpr) rather than in a
+// pass of its own, because that walk already visits every node and already
+// prepares nodes in place: it compiles regex literals and parses timestamp
+// literals. Order matters - a node is folded only after the walk has visited
+// it, so RegexLit.Compiled and TimestampLit.Parsed are populated by then.
+//
+// An expression whose evaluation fails is never folded. It stays in the AST and
+// fails per event exactly as it does today, so folding cannot turn a runtime
+// error into a startup error.
+
+// tryFold returns a core.ConstExpr holding the value of expr when that value
+// can be computed at startup, and expr itself otherwise.
+func tryFold(expr core.Expr) core.Expr {
+	if _, ok := expr.(*core.ConstExpr); ok {
+		return expr
+	}
+	v, ok := foldConst(expr)
+	if !ok {
+		return expr
+	}
+	return &core.ConstExpr{Node: core.NewNode(expr.Pos()), V: v}
+}
+
+// foldConst computes the value of expr, reporting whether it is constant.
+// A composite is constant only when every part of it is.
+func foldConst(expr core.Expr) (core.Value, bool) {
+	switch e := expr.(type) {
+	case *core.ConstExpr:
+		return e.V, true
+
+	case *core.IntLit:
+		return core.IntegerValue{V: e.Value}, true
+	case *core.FloatLit:
+		return core.FloatValue{V: e.Value}, true
+	case *core.StringLit:
+		return core.StringValue{V: e.Value}, true
+	case *core.BoolLit:
+		return core.BoolValue{V: e.Value}, true
+	case *core.NullLit:
+		return core.NullValue{}, true
+
+	case *core.RegexLit:
+		// Compiled by the validation walk; nil means this node has not been
+		// visited yet, so leave it to be evaluated at runtime.
+		if e.Compiled == nil {
+			return nil, false
+		}
+		return core.RegexValue{V: e.Compiled}, true
+
+	case *core.TimestampLit:
+		// Parsed by the validation walk. A zero time means the node has not
+		// been visited; t'0001-01-01T00:00:00Z' simply misses the optimization.
+		if e.Parsed.IsZero() {
+			return nil, false
+		}
+		return core.TimestampValue{V: e.Parsed}, true
+
+	case *core.ArrayExpr:
+		elements := make([]core.Value, len(e.Elements))
+		for i, el := range e.Elements {
+			v, ok := foldConst(el)
+			if !ok {
+				return nil, false
+			}
+			elements[i] = v
+		}
+		return core.ArrayValue{V: elements}, true
+
+	case *core.ObjectExpr:
+		// Duplicate keys are rejected by validateExpr before we get here.
+		pairs := make(map[string]core.Value, len(e.Pairs))
+		for _, kv := range e.Pairs {
+			v, ok := foldConst(kv.Value)
+			if !ok {
+				return nil, false
+			}
+			pairs[kv.Key] = v
+		}
+		return core.ObjectValue{V: pairs}, true
+
+	case *core.UnaryExpr:
+		operand, ok := foldConst(e.Operand)
+		if !ok {
+			return nil, false
+		}
+		return evalConst(&core.UnaryExpr{
+			Node:    core.NewNode(e.Pos()),
+			Op:      e.Op,
+			Operand: constNode(e.Operand, operand),
+		})
+
+	case *core.BinaryExpr:
+		left, ok := foldConst(e.Left)
+		if !ok {
+			return nil, false
+		}
+		right, ok := foldConst(e.Right)
+		if !ok {
+			return nil, false
+		}
+		return evalConst(&core.BinaryExpr{
+			Node:  core.NewNode(e.Pos()),
+			Op:    e.Op,
+			Left:  constNode(e.Left, left),
+			Right: constNode(e.Right, right),
+		})
+	}
+
+	return nil, false
+}
+
+func constNode(from core.Expr, v core.Value) core.Expr {
+	return &core.ConstExpr{Node: core.NewNode(from.Pos()), V: v}
+}
+
+// evalConst evaluates an operator node whose operands are already core.ConstExpr.
+// Those ignore the evaluation context, and UnaryExpr/BinaryExpr touch the context
+// only to evaluate their operands, so a nil context is never dereferenced.
+//
+// A failing expression - `1 / 0`, `"a" + 1` - is reported as not constant so it
+// keeps failing at runtime instead of breaking pipeline startup.
+func evalConst(expr core.Expr) (core.Value, bool) {
+	v, err := expr.Eval(nil)
+	if err != nil {
+		return nil, false
+	}
+	return v, true
+}