profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/mlrtime/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.

mlrtime/cloud-init 0

unofficial mirror of Ubuntu's cloud-init

mlrtime/docker-formula 0

Install and set up Docker

mlrtime/porsche-vin-lookup 0

Lookup porsche listings for sale, compare options on vin

push eventsaltstack-formulas/openssh-formula

Imran Iqbal

commit sha 79321be76fa91234414dd53ea81ee0327276bafe

ci(kitchen+ci): use latest pre-salted images (after CVE) [skip ci] * Automated using https://github.com/myii/ssf-formula/pull/299

view details

push time in 11 hours

push eventsaltstack-formulas/openssh-formula

Imran Iqbal

commit sha ba640fc0c3055cb10e8e096a9ef8a9b74a02e0ca

chore(yamllint): add `.git/` to ignores [skip ci] * Automated using https://github.com/myii/ssf-formula/pull/297

view details

push time in 3 days

fork romaincabassot/process-metrics-collector

A Bash Script to monitor, collect, and visualize metrics of a given Linux process

fork in 6 days

push eventsaltstack-formulas/openssh-formula

Daniel Dehennin

commit sha eafc780866e7fca46aa3aee2096ddd46af41fe89

chore(rubocop): allow use of `YAML.load` for `_mapdata.rb` [skip ci] * Automated using https://github.com/myii/ssf-formula/pull/296

view details

push time in 12 days

push eventsaltstack-formulas/openssh-formula

Dafydd Jones

commit sha d21579fdae7ee64f18f62cc2b9f03a482c186631

chore(shellcheck): switch hook for `shellcheck` [skip ci] * Automated using https://github.com/myii/ssf-formula/pull/295

view details

push time in 15 days

push eventsaltstack-formulas/openssh-formula

Imran Iqbal

commit sha 01512a0ec47b42ea41fcc949f59372b7e95e817c

ci(gemfile+lock): use `ssf` customised `kitchen-docker` repo [skip ci] * Automated using https://github.com/myii/ssf-formula/pull/294

view details

push time in 19 days

push eventsaltstack-formulas/openssh-formula

Imran Iqbal

commit sha 748ededc7af79b792cac8fa01abcd20c8c27d8ed

test(_mapdata): add verification files for new platforms * `fedora-33` * `opensuse-tumbleweed` * `oraclelinux-7` * `oraclelinux-8` * `gentoo-2-sysd` * `gentoo-2-sysv`

view details

Imran Iqbal

commit sha c2a366f9c721fc0956cd08c5e3f239a751be7a10

ci(kitchen+gitlab-ci): use latest pre-salted images [skip ci] * Automated using https://github.com/myii/ssf-formula/pull/293

view details

push time in 22 days

push eventsaltstack-formulas/openssh-formula

Imran Iqbal

commit sha a0d3e3ed8c212a040fd902d0c936211cb8545cfe

chore: standardise structure (`.gitignore` & `_mapdata.rb`) [skip ci] * Automated using https://github.com/myii/ssf-formula/pull/292

view details

push time in 25 days

startedmikefarah/yq

started time in a month

push eventsaltstack-formulas/openssh-formula

Imran Iqbal

commit sha ccb6a4487580eb75b3d735e7cfb398f2b8ebb316

ci(pre-commit): update hook for `rubocop` [skip ci] * Automated using https://github.com/myii/ssf-formula/pull/290

view details

push time in a month

push eventsaltstack-formulas/openssh-formula

Imran Iqbal

commit sha de969f10f1b22a86491f1b33d1d06eb7d721a980

test(share): standardise with latest changes [skip ci] * Automated using https://github.com/myii/ssf-formula/pull/288

view details

push time in a month

pull request commentsaltstack-formulas/openssh-formula

feat(map): use targeting like syntax for configuration

@baby-gnu It took us only 6 months (nearly) but we got there in the end! Thanks for your patience and perseverance.

A big thanks also goes to @n-rodriguez @dafyddj and @vutny for their involvement with improving this implementation.

baby-gnu

comment created time in 2 months

pull request commentsaltstack-formulas/openssh-formula

feat(map): use targeting like syntax for configuration

:tada: This PR is included in version 3.0.0 :tada:

The release is available on GitHub release

Your semantic-release bot :package::rocket:

baby-gnu

comment created time in 2 months

created tagsaltstack-formulas/openssh-formula

tagv3.0.0

created time in 2 months

release saltstack-formulas/openssh-formula

v3.0.0

released time in 2 months

push eventsaltstack-formulas/openssh-formula

semantic-release-bot

commit sha e04536913dc531e8755fa2444ed6a56ec0dd3f49

chore(release): 3.0.0 [skip ci] # [3.0.0](https://github.com/saltstack-formulas/openssh-formula/compare/v2.0.6...v3.0.0) (2021-01-12) ### Code Refactoring * **map:** compound matchers like parsing with `libmatchers` ([925c86e](https://github.com/saltstack-formulas/openssh-formula/commit/925c86ea698c68f684ba1645a58c88d688e6acc5)) * **map:** load `defaults.jinja` configuration with `libmapstack` ([174bb68](https://github.com/saltstack-formulas/openssh-formula/commit/174bb68432366a449a8327a9dbb648271f123224)) * **map:** load `map.jinja` configuration with `libmapstack` ([568bb7c](https://github.com/saltstack-formulas/openssh-formula/commit/568bb7ce4075ee376e8c49a45a1470d252f82ab9)) * **map:** load formula configuration with `libmatchers` ([ff6b56c](https://github.com/saltstack-formulas/openssh-formula/commit/ff6b56c4a4e282f41ddfc8f379f95096fea0553f)) ### Documentation * **map:** document the new `map.jinja` with targeting like syntax ([7ecb24b](https://github.com/saltstack-formulas/openssh-formula/commit/7ecb24bdc1ff84ddac4c7c3e5d8d70c7512f4fb5)) ### Features * **map:** use targeting like syntax for configuration ([1be0d87](https://github.com/saltstack-formulas/openssh-formula/commit/1be0d8725ad933034f4e87cc9636bcc5100bd55c)) * **matchers:** add delimiter option for source definitions ([d69556d](https://github.com/saltstack-formulas/openssh-formula/commit/d69556d5ae79a907d79351d4b9775e0ce2970b39)) ### Styles * **mapstack:** variables in macro can't be exported ([7de2d6f](https://github.com/saltstack-formulas/openssh-formula/commit/7de2d6fd756b3e4b7154e660b639d7ce6edb8cfe)) ### BREAKING CHANGES * **map:** the configuration `map_jinja:sources` is only configurable with `salt://parameters/map_jinja.yaml` and `salt://{{ tplroot }}/parameters/map_jinja.yaml` * **map:** the `map_jinja:config_get_roots` is replaced by compound like `map_jinja:sources` * **map:** the two `config_get_lookup` and `config_get` are replaced by `C@<tplroot>:lookup` and `C@<tplroot>` sources

view details

push time in 2 months

push eventsaltstack-formulas/openssh-formula

Daniel Dehennin

commit sha 1be0d8725ad933034f4e87cc9636bcc5100bd55c

feat(map): use targeting like syntax for configuration The `config_get_lookup` and `config_get` sources lack flexibility. It's not easy to query several pillars and/or grains keys with the actual system. And the query method is forced to `config.get` without being configurable by the user. We define a mechanism to select `map.jinja` sources with similar notation as the salt targeting system. The `map.jinja` file uses several sources where to lookup parameter values. The list of sources can be modified by two files: 1. a global salt://parameters/map_jinja.yaml 2. a per formula salt://{{ tplroot }}/parameters/map_jinja.yaml. Each source definition has the form `<TYPE>:<OPTION>@<KEY>` where `<TYPE>` can be one of: - `Y` to load values from YAML files, this is the default when no type is defined - `C` to lookup values with `config.get` - `G` to lookup values with `grains.get` - `I` to lookup values with `pillar.get` The YAML type option can define the query method to lookup the key value to build the file name: - `C` to query with `config.get`, this is the default when to query method is defined - `G` to query with `grains.get` - `I` to query with `pillar.get` The `C`, `G` or `I` types can define the `SUB` option to store values in the sub key `mapdata.<key>` instead of directly in `mapdata`. Finally, the `<KEY>` describe what to lookup to either build the YAML filename or gather values using one of the query method. BREAKING CHANGE: the configuration `map_jinja:sources` is only configurable with `salt://parameters/map_jinja.yaml` and `salt://{{ tplroot }}/parameters/map_jinja.yaml` BREAKING CHANGE: the `map_jinja:config_get_roots` is replaced by compound like `map_jinja:sources` BREAKING CHANGE: the two `config_get_lookup` and `config_get` are replaced by `C@<tplroot>:lookup` and `C@<tplroot>` sources

view details

Daniel Dehennin

commit sha 7ecb24bdc1ff84ddac4c7c3e5d8d70c7512f4fb5

docs(map): document the new `map.jinja` with targeting like syntax

view details

Daniel Dehennin

commit sha 925c86ea698c68f684ba1645a58c88d688e6acc5

refactor(map): compound matchers like parsing with `libmatchers`

view details

Daniel Dehennin

commit sha d69556d5ae79a907d79351d4b9775e0ce2970b39

feat(matchers): add delimiter option for source definitions

view details

Daniel Dehennin

commit sha 568bb7ce4075ee376e8c49a45a1470d252f82ab9

refactor(map): load `map.jinja` configuration with `libmapstack`

view details

Daniel Dehennin

commit sha 174bb68432366a449a8327a9dbb648271f123224

refactor(map): load `defaults.jinja` configuration with `libmapstack`

view details

Daniel Dehennin

commit sha ff6b56c4a4e282f41ddfc8f379f95096fea0553f

refactor(map): load formula configuration with `libmatchers`

view details

Daniel Dehennin

commit sha 7de2d6fd756b3e4b7154e660b639d7ce6edb8cfe

style(mapstack): variables in macro can't be exported No need to prefix some of them with underscore `_`.

view details

Imran Iqbal

commit sha 9955923682b19f330408e06849a4901138cba72e

Merge pull request #191 from baby-gnu/feature/map-config-targeting-like-syntax feat(map): use targeting like syntax for configuration

view details

push time in 2 months

PR merged saltstack-formulas/openssh-formula

feat(map): use targeting like syntax for configuration

<!-- Please fill in this PR template to make it easier to review and merge.

It has been designed so that a lot of it can be completed after submitting it, e.g. filling in the checklists.

Notes:

  1. Please keep the PR as small as is practicable; the larger a PR gets, the harder it becomes to review and the more time it requires to get it merged.
  2. Similarly, please avoid PRs that cover more than one type; it should be a bug fix OR a new feature OR a refactor, etc.
  3. Please direct questions to the #formulas channel on Slack, which is bridged to #saltstack-formulas on Freenode.
  4. Feel free to suggest improvements to this template by reporting an issue or submitting a PR. The source of this template is:
  • https://github.com/saltstack-formulas/.github/blob/master/.github/pull_request_template.md -->

PR progress checklist (to be filled in by reviewers)

<!-- Please leave this checklist for reviewers to tick as they work through the PR. -->

  • [ ] Changes to documentation are appropriate (or tick if not required)
  • [ ] Changes to tests are appropriate (or tick if not required)
  • [ ] Reviews completed

What type of PR is this?

<!-- Please tick each box that is relevant (after creating the PR). -->

Primary type

<!-- There really should be only one of these types ticked for each PR. -->

  • [ ] [build] Changes related to the build system
  • [ ] [chore] Changes to the build process or auxiliary tools and libraries such as documentation generation
  • [ ] [ci] Changes to the continuous integration configuration
  • [x] [feat] A new feature
  • [ ] [fix] A bug fix
  • [ ] [perf] A code change that improves performance
  • [ ] [refactor] A code change that neither fixes a bug nor adds a feature
  • [ ] [revert] A change used to revert a previous commit
  • [ ] [style] Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc.)

Secondary type

<!-- Most PRs should include all of the following types as well. -->

  • [x] [docs] Documentation changes
  • [ ] [test] Adding missing or correcting existing tests

Does this PR introduce a BREAKING CHANGE?

<!-- If so, change the following to a Yes and explain what the breaking changes are. --> <!-- If there are multiple breaking changes, list them all. -->

Yes.

BREAKING CHANGE: the configuration map_jinja:sources is only configurable with salt://parameters/map_jinja.yaml and salt://{{ tplroot }}/parameters/map_jinja.yaml

BREAKING CHANGE: the map_jinja:config_get_roots is replaced by compound like map_jinja:sources

BREAKING CHANGE: the two map_jinja:sources config_get_lookup and config_get are replaced by C@<tplroot>:lookup and C@<tplroot> sources

Related issues and/or pull requests

<!-- Please link any related issues/PRs here, especially any issues that are closed by this PR. -->

#186

Describe the changes you're proposing

<!-- A clear and concise description of what you have implemented. --> <!-- Consider explaining each commit if they cover different aspects of the proposed changes. -->

The config_get_lookup and config_get sources lack flexibility.

It's not easy to query several pillars and/or grains keys with the actual system. And the query method is forced to config.get without being configurable by the user.

We define a mechanism to select map.jinja sources with similar notation as the salt targeting system.

Each source has a type:

  • Y to load values from YAML files, this is the default when no type is defined
  • C to lookup values with config.get
  • G to lookup values with grains.get
  • I to lookup values with pillar.get

The YAML type can define the query method to lookup the key value to build the file name:

  • C to query with config.get, this is the default when to query method is defined
  • G to query with grains.get
  • I to query with pillar.get

The C, G or I types can define the SUB attribute to merge values in the sub key mapdata.<key> instead of directly in mapdata.

BREAKING CHANGE: the configuration map_jinja:sources is only configurable with salt://parameters/map_jinja.yaml and salt://{{ tplroot }}/parameters/map_jinja.yaml

BREAKING CHANGE: the map_jinja:config_get_roots is replaced by compound like map_jinja:sources

BREAKING CHANGE: the two map_jinja:sources config_get_lookup and config_get are replaced by C@<tplroot>:lookup and C@<tplroot> sources

Users can now define map.jinja sources like:

values:
  map_jinja:
    sources:
      # By default, when no `@` sign is present, it's `Y:C@` for YAML files with key lookup using `config.get`
      - "osarch"
      - "os_family"
      - "os"
      - "osfinger"

      # Use `:SUB` to merge values from `config.get` under `mapdata.<key>` to keep
      # compatibility with user pillars.
      # The `<key>` and `<key>:lookup` are merged together
      - "C:SUB@openssh:lookup"
      - "C:SUB@openssh"
      - "C:SUB@sshd_config:lookup"
      - "C:SUB@sshd_config"
      - "C:SUB@ssh_config:lookup"
      - "C:SUB@ssh_config"

      # Lookup role based YAML files with role names from pillars only
      - "Y:I@roles"

      # Lookup DNS domain name based YAML files with DNS domain names from grains only
      - "Y:G@dns:domain"

      - "id"

The map.jinja configuration are looked-up from:

  • salt://parameters/map_jinja.yaml to define it globally for all formulas
  • salt://{{ tplroot }}/parameters/map_jinja.yaml to define it per formula

The map_jinja.yaml file can contains Jinja constructs like the use of {{ tplroot }}:

values:
  map_jinja:
    sources:
      # default values
      - "osarch"
      - "os_family"
      - "os"
      - "osfinger"
      - "C@{{ tplroot ~ ':lookup' }}"
      - "C@{{ tplroot }}"
  
      # role based configuration
      - "roles"

      # DNS domain configured (DHCP or resolv.conf)
      - "dns:domain"
  
      # Based on minion ID
      - "domain"
  
      # default values
      - "id"

Pillar / config required to test the proposed changes

<!-- Provide links to the SLS files and/or relevant configs (be sure to remove sensitive info). -->

Included in the commit.

Debug log showing how the proposed changes work

<!-- Include a debug log showing how these changes work, e.g. using salt-minion -l debug. --> <!-- Alternatively, linking to Kitchen debug logs is useful, e.g. via. Travis CI. --> <!-- Most useful is providing a passing InSpec test, which can be used to verify any proposed changes. -->

[DEBUG   ] In saltenv 'base', looking at rel_path 'openssh/map.jinja' to resolve 'salt://openssh/map.jinja'
[DEBUG   ] In saltenv 'base', ** considering ** path '/tmp/kitchen/var/cache/salt/minion/files/base/openssh/map.jinja' to resolve 'salt://openssh/map.jinja'
[DEBUG   ] In saltenv 'base', looking at rel_path 'openssh/libsaltcli.jinja' to resolve 'salt://openssh/libsaltcli.jinja'
[DEBUG   ] In saltenv 'base', ** considering ** path '/tmp/kitchen/var/cache/salt/minion/files/base/openssh/libsaltcli.jinja' to resolve 'salt://openssh/libsaltcli.jinja'
[DEBUG   ] [libsaltcli] the salt command type has been identified to be: local
[DEBUG   ] map.jinja: built-in configuration sources:
values:
  map_jinja:
    sources:
    - osarch
    - os_family
    - os
    - osfinger
    - C@openssh:lookup
    - C@openssh
    - id
[DEBUG   ] map.jinja: load global map.jinja values from parameters/map_jinja.yaml
[DEBUG   ] Could not find file 'salt://parameters/map_jinja.yaml' in saltenv 'base'
[DEBUG   ] map.jinja: load per formula map.jinja values from openssh/parameters/map_jinja.yaml
[DEBUG   ] In saltenv 'base', looking at rel_path 'openssh/parameters/map_jinja.yaml' to resolve 'salt://openssh/parameters/map_jinja.yaml'
[DEBUG   ] In saltenv 'base', ** considering ** path '/tmp/kitchen/var/cache/salt/minion/files/base/openssh/parameters/map_jinja.yaml' to resolve 'salt://openssh/parameters/map_jinja.yaml'
[DEBUG   ] map.jinja: configure sources from formula map.jinja configuration openssh/parameters/map_jinja.yaml:
values:
  map_jinja:
    sources:
    - osarch
    - os_family
    - os
    - osfinger
    - C:SUB@openssh:lookup
    - C:SUB@openssh
    - C:SUB@sshd_config:lookup
    - C:SUB@sshd_config
    - C:SUB@ssh_config:lookup
    - C:SUB@ssh_config
    - id
[DEBUG   ] map.jinja: load parameters from sources:
- osarch
- os_family
- os
- osfinger
- C:SUB@openssh:lookup
- C:SUB@openssh
- C:SUB@sshd_config:lookup
- C:SUB@sshd_config
- C:SUB@ssh_config:lookup
- C:SUB@ssh_config
- id
[DEBUG   ] map.jinja: load per formula default values from openssh/parameters/defaults.yaml
[DEBUG   ] In saltenv 'base', looking at rel_path 'openssh/parameters/defaults.yaml' to resolve 'salt://openssh/parameters/defaults.yaml'
[DEBUG   ] In saltenv 'base', ** considering ** path '/tmp/kitchen/var/cache/salt/minion/files/base/openssh/parameters/defaults.yaml' to resolve 'salt://openssh/parameters/defaults.yaml'
[DEBUG   ] map.jinja: lookup 'osarch' with 'config.get'
[DEBUG   ] map.jinja: load parameters from file openssh/parameters/osarch/amd64.yaml
[DEBUG   ] Could not find file 'salt://openssh/parameters/osarch/amd64.yaml' in saltenv 'base'
[DEBUG   ] map.jinja: lookup 'os_family' with 'config.get'
[DEBUG   ] map.jinja: load parameters from file openssh/parameters/os_family/Debian.yaml
[DEBUG   ] In saltenv 'base', looking at rel_path 'openssh/parameters/os_family/Debian.yaml' to resolve 'salt://openssh/parameters/os_family/Debian.yaml'
[DEBUG   ] In saltenv 'base', ** considering ** path '/tmp/kitchen/var/cache/salt/minion/files/base/openssh/parameters/os_family/Debian.yaml' to resolve 'salt://openssh/parameters/os_family/Debian.yaml'
[DEBUG   ] map.jinja: merge parameters from openssh/parameters/os_family/Debian.yaml
[DEBUG   ] map.jinja: lookup 'os' with 'config.get'
[DEBUG   ] map.jinja: load parameters from file openssh/parameters/os/Debian.yaml
[DEBUG   ] Could not find file 'salt://openssh/parameters/os/Debian.yaml' in saltenv 'base'
[DEBUG   ] map.jinja: lookup 'osfinger' with 'config.get'
[DEBUG   ] map.jinja: load parameters from file openssh/parameters/osfinger/Debian-10.yaml
[DEBUG   ] Could not find file 'salt://openssh/parameters/osfinger/Debian-10.yaml' in saltenv 'base'
[DEBUG   ] map.jinja: retrieve 'openssh:lookup' with 'config.get', merge: strategy='None'
[DEBUG   ] map.jinja: merge sub key 'openssh:lookup' retrieved with 'config.get', merge: strategy='smart', lists='False'
[DEBUG   ] map.jinja: retrieve 'openssh' with 'config.get', merge: strategy='None'
[DEBUG   ] map.jinja: merge sub key 'openssh' retrieved with 'config.get', merge: strategy='smart', lists='False'
[DEBUG   ] map.jinja: retrieve 'sshd_config:lookup' with 'config.get', merge: strategy='None'
[DEBUG   ] map.jinja: merge sub key 'sshd_config:lookup' retrieved with 'config.get', merge: strategy='smart', lists='False'
[DEBUG   ] map.jinja: retrieve 'sshd_config' with 'config.get', merge: strategy='None'
[DEBUG   ] map.jinja: merge sub key 'sshd_config' retrieved with 'config.get', merge: strategy='smart', lists='False'
[DEBUG   ] map.jinja: retrieve 'ssh_config:lookup' with 'config.get', merge: strategy='None'
[DEBUG   ] map.jinja: merge sub key 'ssh_config:lookup' retrieved with 'config.get', merge: strategy='smart', lists='False'
[DEBUG   ] map.jinja: retrieve 'ssh_config' with 'config.get', merge: strategy='None'
[DEBUG   ] map.jinja: merge sub key 'ssh_config' retrieved with 'config.get', merge: strategy='smart', lists='False'
[DEBUG   ] map.jinja: lookup 'id' with 'config.get'
[DEBUG   ] map.jinja: load parameters from file openssh/parameters/id/example.net.yaml
[DEBUG   ] Could not find file 'salt://openssh/parameters/id/example.net.yaml' in saltenv 'base'
[DEBUG   ] map.jinja: save parameters in variable 'mapdata'

Documentation checklist

<!-- Please tick each box that is relevant (after creating the PR). -->

  • [ ] Updated the README (e.g. Available states).
  • [ ] Updated pillar.example.

Testing checklist

<!-- Please tick each box that is relevant (after creating the PR). -->

  • [x] Included in Kitchen (i.e. under state_top).
  • [x] Covered by new/existing tests (e.g. InSpec, Serverspec, etc.).
  • [ ] Updated the relevant test pillar.

Additional context

<!-- Add any other context about the proposed changes here. -->

+1251 -250

37 comments

21 changed files

baby-gnu

pr closed time in 2 months

pull request commentsaltstack-formulas/openssh-formula

feat(map): use targeting like syntax for configuration

Thanks a lot @myii for your scrutting eyes. The branch and the code should be in proper state to be merged now. It pass on my test bed for all kitchen platforms except centos-6 which does not build at all.

baby-gnu

comment created time in 2 months

Pull request review commentsaltstack-formulas/openssh-formula

feat(map): use targeting like syntax for configuration

  {#- Get the `tplroot` from `tpldir` #} {%- set tplroot = tpldir.split("/")[0] %}-{%- from tplroot ~ "/libsaltcli.jinja" import cli with context %}+{%- from tplroot ~ "/libmapstack.jinja" import mapstack %}  {#- Where to lookup parameters source files #}-{%- set map_sources_dir = tplroot ~ "/parameters" %}--{#- Load defaults first to allow per formula default map.jinja configuration #}-{%- set _defaults_filename = map_sources_dir ~ "/defaults.yaml" %}-{%- do salt["log.debug"](-      "map.jinja: initialise parameters from "-      ~ _defaults_filename-    ) %}-{%- import_yaml _defaults_filename as default_settings %}+{%- set formula_param_dir = tplroot ~ "/parameters" %}  {#- List of sources to lookup for parameters #}-{%- do salt["log.debug"]("map.jinja: lookup 'map_jinja' configuration sources") %} {#- Fallback to previously used grains plus minion `id` #} {%- set map_sources = [-      "osarch",-      "os_family",-      "os",-      "osfinger",-      "config_get_lookup",-      "config_get",-      "id",+      "Y:G@osarch",+      "Y:G@os_family",+      "Y:G@os",+      "Y:G@osfinger",+      "C@" ~ tplroot ~ ":lookup",+      "C@" ~ tplroot,+      "Y:G@id",     ] %}-{#- Configure map.jinja from defaults.yaml #}-{%- set map_sources = default_settings | traverse(-      "values:map_jinja:sources",-      map_sources,-    ) %} -{#- Lookup global sources #}-{%- set map_sources = salt["config.get"]("map_jinja:sources", map_sources) %}-{#- Lookup per formula sources #}-{%- set map_sources = salt["config.get"](-      tplroot ~ ":map_jinja:sources",-      map_sources,-    ) %}--{%- do salt["log.debug"](-      "map.jinja: load parameters with sources from "-      ~ map_sources-    ) %}--{#- Lookup with `config.get` from configurable roots #}-{%- do salt["log.debug"](-      "map.jinja: initialise 'config.get' roots with 'tplroot' "-      ~ tplroot-    ) %}-{%- set config_get_roots = [tplroot] %}-{#- Configure `config.get` from defaults.yaml #}-{%- set config_get_roots = default_settings | traverse(-      "values:map_jinja:config_get_roots",-      config_get_roots-    ) %}-{#- Lookup global `config.get` roots #}-{%- set config_get_roots = salt["config.get"](-      "map_jinja:config_get_roots",-      config_get_roots-    ) %}-{#- Lookup per formula `config.get` roots #}-{%- set config_get_roots = salt["config.get"](-      tplroot ~ ":map_jinja:config_get_roots",-      config_get_roots,-    ) %}+{%- set _map_settings = mapstack(+      matchers=["map_jinja.yaml"],+      defaults={"values":+                 {"sources": map_sources}+               },

Thanks, I always wonder what is the best. I should do more python to get used to black style.

baby-gnu

comment created time in 2 months

Pull request review commentsaltstack-formulas/openssh-formula

feat(map): use targeting like syntax for configuration

+{#- -*- coding: utf-8 -*- #}+{#- vim: ft=jinja #}++{#- Get the `tplroot` from `tpldir` #}+{%- set tplroot = tpldir.split("/")[0] %}+{%- from tplroot ~ "/libsaltcli.jinja" import cli %}++{%- set query_map = {+      "C": "config.get",+      "G": "grains.get",+      "I": "pillar.get",+    } %}++{#- When no part before `@` is provided: #}+{#-   - define a filename path, noted `F` #}+{#-   - use `salt["config.get"]`, noted `C` #}+{#-   - use colon `:` delimiter for querying #}+{%- set _defaults = {+      "type": "F",+      "query_type": "C",+      "query_delimiter": ":"+    } %}++{%- macro parse_matchers(+      matchers,+      config_get_strategy=None,+      log_prefix="libmatchers: "+    ) %}+{#-   matcher format is `[<TYPE>[:<OPTION>[:DELIMITER]]@]<KEY>` #}+{#-   each matcher has a type: #}+{#-   - `F` to build a file name (the default when no type is set) #}+{#-   - `C` to lookup values with `config.get` #}+{#-   - `G` to lookup values with `grains.get` #}+{#-   - `I` to lookup values with `pillar.get` #}+{#-   The `FILE` type option can define query type to build the file name: #}+{#-   - `C` for query with `config.get` (the default when to query type is set) #}+{#-   - `G` for query with `grains.get` #}+{#-   - `I` for query with `pillar.get` #}+{#-   With `DELIMITER`, you can choose a different delimiter when doing queries #}+{%-   set parsed_matchers = [] %}+{%-   for matcher in matchers %}+{%-     do salt["log.debug"](+          log_prefix+          ~ "process matcher: "+          ~ "'" ~ matcher ~ "'"+        ) %}++{%-     set parsed = {} %}+{%-     set matcher_parts = matcher.split('@') %}+{%-     if matcher_parts | length == 1 %}+{#-       By default we load YAML files for config looked up by `config.get` #}+{%-       do parsed.update(+            {+              "type": _defaults["type"],+              "option": None,+              "query_method": query_map[_defaults["query_type"]],+              "query_delimiter": _defaults["query_delimiter"],+              "query": matcher+            }+          ) %}+{%-       do salt["log.debug"](+            log_prefix+            ~ "use built-in defaults for matcher:\n"+            ~ parsed | yaml(False) | indent(4, True)+          ) %}+{%-     else %}+{%-       do salt["log.debug"](+            log_prefix+            ~ "parse matcher: "+            ~ "'" ~ matcher ~ "'"+          ) %}+{%-       set metadatas = matcher_parts[0].split(":") %}+{%-       do parsed.update(+            {+              "query": matcher_parts[1]+            }+          ) %}+{%-       if metadatas | length == 1 %}+{%-         do parsed.update(+              {+                "type": metadatas[0],+                "option": "C",+                "query_delimiter": ":"+              }+            ) %}+{%-         do salt["log.debug"](+              log_prefix+              ~ "parse as 1 metadata matcher:\n"+              ~ parsed | yaml(False) | indent(4, True)+            ) %}+{%-       elif metadatas | length == 2 %}+{%-         do parsed.update(+              {+                "type": metadatas[0],+                "option": metadatas[1],+                "query_delimiter": ":"+              }+            ) %}+{%-         do salt["log.debug"](+              log_prefix+              ~ "parse as 2 metadata matcher:\n"+              ~ parsed | yaml(False) | indent(4, True)+            ) %}+{%-       elif metadatas | length == 3 %}+{%-         do parsed.update(+              {+                "type": metadatas[0],+                "option": metadatas[1],+                "query_delimiter": metadatas[2] | default(":", boolean=True)+              }+            ) %}+{%-         do salt["log.debug"](+              log_prefix+              ~ "parse as 3 metadata matcher:\n"+              ~ parsed | yaml(False) | indent(4, True)+            ) %}+{%-       elif metadatas | length == 4 %}+{#-         The delimiter is `:` #}+{%-         do parsed.update(+              {+                "type": metadatas[0],+                "option": metadatas[1],+                "query_delimiter": ":"+              }+            ) %}+{%-         do salt["log.debug"](+              log_prefix+              ~ "parse as 4 metadata matcher:\n"+              ~ parsed | yaml(False) | indent(4, True)+            ) %}+{%-       endif %}+{%-     endif %}++{#-     The `<OPTION>` has different meaning based on type #}+{%-     if query_map.get(parsed.type, False) %}+{%-       do parsed.update(+            {+              "query_method": query_map[parsed.type]+            }+          ) %}+{%-     else %}+{%-       do parsed.update(+            {+              "query_method": query_map[+                parsed.option+                | default("C", boolean=True)+              ]+            }+          ) %}+{%-     endif %}++{#-     Add `merge:` option to `salt["config.get"]` if configured #}+{%-     if cli in ["minion", "local"] and parsed.query_method == "config.get" and config_get_strategy %}+{%-       set query_opts = {"merge": config_get_strategy,+                            "delimiter": parsed.query_delimiter,+          } %}+{%-       set query_opts_msg = ", delimiter='" ~ parsed.query_delimiter ~ "'"+            ~ ", merge: strategy='" ~ config_get_strategy ~ "'"+          %}

I even merge some "'" with previous or next string. I did the same for some lonely ":\n" which could be included in the previous string.

baby-gnu

comment created time in 2 months

Pull request review commentsaltstack-formulas/openssh-formula

feat(map): use targeting like syntax for configuration

+{#- -*- coding: utf-8 -*- #}+{#- vim: ft=jinja #}++{#- Get the `tplroot` from `tpldir` #}+{%- set tplroot = tpldir.split("/")[0] %}+{%- from tplroot ~ "/libsaltcli.jinja" import cli %}++{%- set query_map = {+      "C": "config.get",+      "G": "grains.get",+      "I": "pillar.get",+    } %}++{#- When no part before `@` is provided: #}+{#-   - define a filename path, noted `F` #}+{#-   - use `salt["config.get"]`, noted `C` #}+{#-   - use colon `:` delimiter for querying #}+{%- set _defaults = {+      "type": "F",+      "query_type": "C",+      "query_delimiter": ":"+    } %}++{%- macro parse_matchers(+      matchers,+      config_get_strategy=None,+      log_prefix="libmatchers: "+    ) %}+{#-   matcher format is `[<TYPE>[:<OPTION>[:DELIMITER]]@]<KEY>` #}+{#-   each matcher has a type: #}+{#-   - `F` to build a file name (the default when no type is set) #}+{#-   - `C` to lookup values with `config.get` #}+{#-   - `G` to lookup values with `grains.get` #}+{#-   - `I` to lookup values with `pillar.get` #}+{#-   The `FILE` type option can define query type to build the file name: #}+{#-   - `C` for query with `config.get` (the default when to query type is set) #}+{#-   - `G` for query with `grains.get` #}+{#-   - `I` for query with `pillar.get` #}+{#-   With `DELIMITER`, you can choose a different delimiter when doing queries #}+{%-   set parsed_matchers = [] %}+{%-   for matcher in matchers %}+{%-     do salt["log.debug"](+          log_prefix+          ~ "process matcher: "+          ~ "'" ~ matcher ~ "'"+        ) %}++{%-     set parsed = {} %}+{%-     set matcher_parts = matcher.split('@') %}+{%-     if matcher_parts | length == 1 %}+{#-       By default we load YAML files for config looked up by `config.get` #}+{%-       do parsed.update(+            {+              "type": _defaults["type"],+              "option": None,+              "query_method": query_map[_defaults["query_type"]],+              "query_delimiter": _defaults["query_delimiter"],+              "query": matcher+            }+          ) %}+{%-       do salt["log.debug"](+            log_prefix+            ~ "use built-in defaults for matcher:\n"+            ~ parsed | yaml(False) | indent(4, True)+          ) %}+{%-     else %}+{%-       do salt["log.debug"](+            log_prefix+            ~ "parse matcher: "+            ~ "'" ~ matcher ~ "'"+          ) %}+{%-       set metadatas = matcher_parts[0].split(":") %}+{%-       do parsed.update(+            {+              "query": matcher_parts[1]+            }+          ) %}+{%-       if metadatas | length == 1 %}+{%-         do parsed.update(+              {+                "type": metadatas[0],+                "option": "C",+                "query_delimiter": ":"+              }+            ) %}+{%-         do salt["log.debug"](+              log_prefix+              ~ "parse as 1 metadata matcher:\n"+              ~ parsed | yaml(False) | indent(4, True)+            ) %}+{%-       elif metadatas | length == 2 %}+{%-         do parsed.update(+              {+                "type": metadatas[0],+                "option": metadatas[1],+                "query_delimiter": ":"+              }+            ) %}+{%-         do salt["log.debug"](+              log_prefix+              ~ "parse as 2 metadata matcher:\n"+              ~ parsed | yaml(False) | indent(4, True)+            ) %}+{%-       elif metadatas | length == 3 %}+{%-         do parsed.update(+              {+                "type": metadatas[0],+                "option": metadatas[1],+                "query_delimiter": metadatas[2] | default(":", boolean=True)+              }+            ) %}+{%-         do salt["log.debug"](+              log_prefix+              ~ "parse as 3 metadata matcher:\n"+              ~ parsed | yaml(False) | indent(4, True)+            ) %}+{%-       elif metadatas | length == 4 %}+{#-         The delimiter is `:` #}+{%-         do parsed.update(+              {+                "type": metadatas[0],+                "option": metadatas[1],+                "query_delimiter": ":"+              }+            ) %}+{%-         do salt["log.debug"](+              log_prefix+              ~ "parse as 4 metadata matcher:\n"+              ~ parsed | yaml(False) | indent(4, True)+            ) %}+{%-       endif %}+{%-     endif %}++{#-     The `<OPTION>` has different meaning based on type #}+{%-     if query_map.get(parsed.type, False) %}+{%-       do parsed.update(+            {+              "query_method": query_map[parsed.type]+            }+          ) %}+{%-     else %}+{%-       do parsed.update(+            {+              "query_method": query_map[+                parsed.option+                | default("C", boolean=True)+              ]+            }+          ) %}+{%-     endif %}++{#-     Add `merge:` option to `salt["config.get"]` if configured #}+{%-     if cli in ["minion", "local"] and parsed.query_method == "config.get" and config_get_strategy %}+{%-       set query_opts = {"merge": config_get_strategy,+                            "delimiter": parsed.query_delimiter,+          } %}

This is much more elegant, and ways easier to cut&past. Thanks.

baby-gnu

comment created time in 2 months

Pull request review commentsaltstack-formulas/openssh-formula

feat(map): use targeting like syntax for configuration

+{#- -*- coding: utf-8 -*- #}+{#- vim: ft=jinja #}++{#- Get the `tplroot` from `tpldir` #}+{%- set tplroot = tpldir.split("/")[0] %}+{%- from tplroot ~ "/libsaltcli.jinja" import cli %}++{%- set query_map = {+      "C": "config.get",+      "G": "grains.get",+      "I": "pillar.get",+    } %}++{#- When no part before `@` is provided: #}+{#-   - define a filename path, noted `F` #}+{#-   - use `salt["config.get"]`, noted `C` #}+{#-   - use colon `:` delimiter for querying #}+{%- set _defaults = {+      "type": "F",+      "query_type": "C",+      "query_delimiter": ":"+    } %}++{%- macro parse_matchers(+      matchers,+      config_get_strategy=None,+      log_prefix="libmatchers: "+    ) %}+{#-   matcher format is `[<TYPE>[:<OPTION>[:DELIMITER]]@]<KEY>` #}+{#-   each matcher has a type: #}+{#-   - `F` to build a file name (the default when no type is set) #}+{#-   - `C` to lookup values with `config.get` #}+{#-   - `G` to lookup values with `grains.get` #}+{#-   - `I` to lookup values with `pillar.get` #}+{#-   The `FILE` type option can define query type to build the file name: #}+{#-   - `C` for query with `config.get` (the default when to query type is set) #}+{#-   - `G` for query with `grains.get` #}+{#-   - `I` for query with `pillar.get` #}+{#-   With `DELIMITER`, you can choose a different delimiter when doing queries #}+{%-   set parsed_matchers = [] %}+{%-   for matcher in matchers %}+{%-     do salt["log.debug"](+          log_prefix+          ~ "process matcher: "+          ~ "'" ~ matcher ~ "'"+        ) %}++{%-     set parsed = {} %}+{%-     set matcher_parts = matcher.split('@') %}+{%-     if matcher_parts | length == 1 %}+{#-       By default we load YAML files for config looked up by `config.get` #}+{%-       do parsed.update(+            {+              "type": _defaults["type"],+              "option": None,+              "query_method": query_map[_defaults["query_type"]],+              "query_delimiter": _defaults["query_delimiter"],+              "query": matcher+            }+          ) %}+{%-       do salt["log.debug"](+            log_prefix+            ~ "use built-in defaults for matcher:\n"+            ~ parsed | yaml(False) | indent(4, True)

I did it and even changed some alignment of some | <filter> at other places which were indented by 2 more spaces than the previous line. Thanks.

baby-gnu

comment created time in 2 months

Pull request review commentsaltstack-formulas/openssh-formula

feat(map): use targeting like syntax for configuration

+{#- -*- coding: utf-8 -*- #}+{#- vim: ft=jinja #}++{#- Get the `tplroot` from `tpldir` #}+{%- set tplroot = tpldir.split("/")[0] %}+{%- from tplroot ~ "/libmatchers.jinja" import parse_matchers, query_map %}++{%- set _default_config_dirs = [+      "parameters/",+      tplroot ~ "/parameters"+    ] %}++{%- macro mapstack(+      matchers,+      defaults=None,+      dirs=_default_config_dirs,+      log_prefix="libmapstack: "+    ) %}+{#-+    Load configuration in the order of `matchers` and merge+    successively the values with `defaults`.++    The `matchers` are processed using `libmatchers.jinja` to select+    the configuration sources from where the values are loaded.++    Parameters:++      - `matchers`: list of matchers in the form+        `[<TYPE>[:<OPTION>[:<DELIMITER>]]@]<QUERY>`++      - `defaults`: dictionary of default values to start the merging,+        they are considered built-ins. It must conform to the same+        layout as the YAML files: a mandatory `values` key and two+        optional `strategy` and `merge_lists` keys.++      - `dirs`: list of directory where to look-up the configuration+        file matching the matchers, by default a global `salt://parameters/`+        and a per formula `salt://<tplroot>/parameters`++      - `log_prefix`: prefix used in the log outputs, by default it is+        `libmapstack: `++    Example: On a Debian system with `roles=["nginx/server", "telegraf"]`++      {%- set settings = mapstack(+            matchers=[+              "Y:G@os_family",+              "I@" ~ tplroot,+              "Y:C@roles",+            ],+            dirs=["defaults", tplroot ~ "/parameters"],+          )+          | load_yaml+          %}++      This will merge the values:++        - starting with the default empty dictionary `{}` (no+          `defaults` parameter)++        - from the YAML files++          - `salt://defaults/os_family/Debian.yaml`++          - `salt://{{ tplroot }}/parameters/os_family/Debian.yaml`++        - from the pillar `salt["pillar.get"](tplroot)`++        - from the `nginx/server` YAML files:++          - `salt://defaults/roles/nginx/server.yaml`++          - `salt://{{ tplroot }}/parameters/roles/nginx/server.yaml`++        - from the `telegraf` YAML files:++          - `salt://defaults/roles/telegraf.yaml`++          - `salt://{{ tplroot }}/parameters/roles/telegraf.yaml`++    Each YAML file and the `defaults` parameters must conform to the+    following layout:++      - a mandatory `values` key to store the configuration values++      - two optional keys to configure the use of `salt.slsutil.merge`++        - an optional `strategy` key to configure the merging+          strategy, for example `strategy: 'recurse'`, the default is+          `smart`++        - an optional `merge_lists` key to configure if lists should+          be merged or overridden for the `recurse` and `overwrite`+          strategies, for example `merge_lists: 'true'`+#}+{%-   set stack = defaults | default({"values": {} }, boolean=True) %}++{#-   Build configuration file names based on matchers #}+{%-   set matchers = parse_matchers(+        matchers,+        log_prefix=log_prefix+      )+      | load_yaml+      %}++{%-   do salt["log.debug"](+        log_prefix+        ~ "built-in configuration:\n"+        ~ {"values": defaults | traverse("values")}+          | yaml(False)+      ) %}++{%-   for param_dir in dirs %}+{%-     for matcher in matchers %}+{#-       `slsutil.merge` options from #}+{#-       1. the `value` #}+{#-       3. the `defaults` #}+{#-       4. the built-in #}+{%-       set strategy = matcher.value+          | traverse(+            "strategy",+            defaults+            | traverse(+              "strategy",+              "smart"+            )+          ) %}+{%-       set merge_lists = matcher.value+          | traverse(+            "merge_lists",+            defaults+            | traverse(+              "merge_lists",+              False+            )+          )+          | to_bool+          %}++{%-       if matcher.type in query_map.keys() %}+{#-         No value is an empty list, must be a dict for `stack.update` #}+{%-         set normalized_value = matcher.value | default({}, boolean=True) %}++{#-         Merge in `mapdata.<query>` instead of directcly in `mapdata` #}+{%-         set is_sub_key = matcher.option | default(False) == "SUB" %}+{%-         if is_sub_key %}+{#-           Merge values with `mapdata.<key>`, `<key>` and `<key>:lookup` are merged together #}+{%-           set value = { matcher.query | regex_replace(":lookup$", ""): normalized_value } %}+{%-         else %}+{%-           set value = normalized_value %}+{%-         endif %}++{%-         do salt["log.debug"](+              log_prefix+              ~ "merge "+              ~ "sub key " * is_sub_key+              ~ "'" ~ matcher.query+              ~ "' retrieved with '"+              ~ matcher.query_method ~ "'"+              ~ ", merge: strategy='"+              ~ strategy+              ~ "', lists='"+              ~ merge_lists+              ~ "'"+              ~ ":\n"+              ~ value | yaml(False)+            ) %}++{%-         do stack.update(+              {+                "values": salt["slsutil.merge"](+                  stack["values"],+                  value,+                  strategy=strategy,+                  merge_lists=merge_lists,+                )+              }+            ) %}++{%-       else %}+{#-         Load YAML file matching the grain/pillar/... #}+{#-         Fallback to use the source name as a direct filename #}++{%-         if matcher.value | length == 0 %}+{#-           Mangle `matcher.value` to use it as literal path #}+{%-           set query_parts = matcher.query.split("/") %}+{%-           set yaml_dirname = query_parts[0:-1] | join("/") %}+{%-           set yaml_names = query_parts[-1] %}+{%-         else %}+{%-           set yaml_dirname = matcher.query %}+{%-           set yaml_names = matcher.value %}+{%-         endif %}++{#-         Some configuration return list #}+{%-         if yaml_names is string %}+{%-           set yaml_names = [yaml_names] %}+{%-         endif %}++{#-         `yaml_dirname` can be an empty string with literal path like `myconf.yaml` #}+{%-         set yaml_dir = [+              param_dir,+              yaml_dirname+            ]+            | select+            | join("/")+            %}++{%-         for yaml_name in yaml_names %}+{#-           Make sure to have a `.yaml` extension #}+{#-           Use `.rpatition` to strip last `.yaml` in `dir.yaml/file.yaml` #}+{%-           set yaml_filename = [+                yaml_dir.rstrip("/"),+                yaml_name.rpartition(".yaml")+                | reject("equalto", ".yaml")+                | join+                ~ ".yaml"+              ]+              | select+              | join("/")+              %}++{%-           do salt["log.debug"](+                log_prefix+                ~ "load configuration values from "+                ~ yaml_filename+              ) %}+{%-           load_yaml as yaml_values %}+{%-             include  yaml_filename ignore missing %}+{%-           endload %}++{%-           if yaml_values %}+{%-             do salt["log.debug"](+                  log_prefix+                  ~ "loaded configuration values from "+                  ~ yaml_name+                  ~ ":\n"+                  ~ yaml_values+                    | yaml(False)+                ) %}++{#-             `slsutil.merge` options from #}+{#-             1. the `value` #}+{#-             3. the `defaults` #}+{#-             4. the built-in #}+{%-             set strategy = yaml_values+                  | traverse(+                    "strategy",+                    defaults | traverse("strategy", "smart")+                  ) %}+{%-             set merge_lists = yaml_values+                  | traverse(+                    "merge_lists",+                    defaults | traverse("merge_lists", False) | to_bool+                  ) %}+{%-             do stack.update(+                  {+                    "values": salt["slsutil.merge"](+                      stack["values"],+                      yaml_values+                      | traverse("values", {}),+                      strategy=strategy,+                      merge_lists=merge_lists,+                    )+                  }+                ) %}+{%-             do salt["log.debug"](+                  log_prefix+                  ~ "merged configuration values from "+                  ~ yaml_name+                  ~ ", merge: strategy='"+                  ~ strategy+                  ~ "', merge_lists='"+                  ~ merge_lists+                  ~ "':\n"+                  ~ {"values": stack["values"]}+                  | yaml(False)+                ) %}+{%-           endif %}+{%-         endfor %}+{%-       endif %}+{%-     endfor %}+{%-   endfor %}++{%-   do salt["log.debug"](+        log_prefix+        ~ "final configuration values:\n"+        ~ {"values": stack | traverse("values")}+        | yaml(False)+      ) %}++{#- Output stack as YAML, caller should use with something like #}+{#- `{%- set config = mapstack(matchers=["foo"]) | load_yaml %}` #}+{{ stack | yaml }}++{%- endmacro -%}

I was not aware of problem, I just wanted to have YAML without any extra empty lines at the end.

But as this macro should be passed to the load_yaml filter it does not matter if there is a last empty line.

Thanks.

baby-gnu

comment created time in 2 months

Pull request review commentsaltstack-formulas/openssh-formula

feat(map): use targeting like syntax for configuration

+{#- -*- coding: utf-8 -*- #}+{#- vim: ft=jinja #}++{#- Get the `tplroot` from `tpldir` #}+{%- set tplroot = tpldir.split("/")[0] %}+{%- from tplroot ~ "/libmatchers.jinja" import parse_matchers, query_map %}++{%- set _default_config_dirs = [+      "parameters/",+      tplroot ~ "/parameters"+    ] %}++{%- macro mapstack(+      matchers,+      defaults=None,+      dirs=_default_config_dirs,+      log_prefix="libmapstack: "+    ) %}+{#-+    Load configuration in the order of `matchers` and merge+    successively the values with `defaults`.++    The `matchers` are processed using `libmatchers.jinja` to select+    the configuration sources from where the values are loaded.++    Parameters:++      - `matchers`: list of matchers in the form+        `[<TYPE>[:<OPTION>[:<DELIMITER>]]@]<QUERY>`++      - `defaults`: dictionary of default values to start the merging,+        they are considered built-ins. It must conform to the same+        layout as the YAML files: a mandatory `values` key and two+        optional `strategy` and `merge_lists` keys.++      - `dirs`: list of directory where to look-up the configuration+        file matching the matchers, by default a global `salt://parameters/`+        and a per formula `salt://<tplroot>/parameters`++      - `log_prefix`: prefix used in the log outputs, by default it is+        `libmapstack: `++    Example: On a Debian system with `roles=["nginx/server", "telegraf"]`++      {%- set settings = mapstack(+            matchers=[+              "Y:G@os_family",+              "I@" ~ tplroot,+              "Y:C@roles",+            ],+            dirs=["defaults", tplroot ~ "/parameters"],+          )+          | load_yaml+          %}++      This will merge the values:++        - starting with the default empty dictionary `{}` (no+          `defaults` parameter)++        - from the YAML files++          - `salt://defaults/os_family/Debian.yaml`++          - `salt://{{ tplroot }}/parameters/os_family/Debian.yaml`++        - from the pillar `salt["pillar.get"](tplroot)`++        - from the `nginx/server` YAML files:++          - `salt://defaults/roles/nginx/server.yaml`++          - `salt://{{ tplroot }}/parameters/roles/nginx/server.yaml`++        - from the `telegraf` YAML files:++          - `salt://defaults/roles/telegraf.yaml`++          - `salt://{{ tplroot }}/parameters/roles/telegraf.yaml`++    Each YAML file and the `defaults` parameters must conform to the+    following layout:++      - a mandatory `values` key to store the configuration values++      - two optional keys to configure the use of `salt.slsutil.merge`++        - an optional `strategy` key to configure the merging+          strategy, for example `strategy: 'recurse'`, the default is+          `smart`++        - an optional `merge_lists` key to configure if lists should+          be merged or overridden for the `recurse` and `overwrite`+          strategies, for example `merge_lists: 'true'`+#}+{%-   set stack = defaults | default({"values": {} }, boolean=True) %}++{#-   Build configuration file names based on matchers #}+{%-   set matchers = parse_matchers(+        matchers,+        log_prefix=log_prefix+      )+      | load_yaml+      %}++{%-   do salt["log.debug"](+        log_prefix+        ~ "built-in configuration:\n"+        ~ {"values": defaults | traverse("values")}+          | yaml(False)+      ) %}++{%-   for param_dir in dirs %}+{%-     for matcher in matchers %}+{#-       `slsutil.merge` options from #}+{#-       1. the `value` #}+{#-       3. the `defaults` #}+{#-       4. the built-in #}+{%-       set strategy = matcher.value+          | traverse(+            "strategy",+            defaults+            | traverse(+              "strategy",+              "smart"+            )+          ) %}+{%-       set merge_lists = matcher.value+          | traverse(+            "merge_lists",+            defaults+            | traverse(+              "merge_lists",+              False+            )+          )+          | to_bool+          %}++{%-       if matcher.type in query_map.keys() %}+{#-         No value is an empty list, must be a dict for `stack.update` #}+{%-         set normalized_value = matcher.value | default({}, boolean=True) %}++{#-         Merge in `mapdata.<query>` instead of directcly in `mapdata` #}+{%-         set is_sub_key = matcher.option | default(False) == "SUB" %}+{%-         if is_sub_key %}+{#-           Merge values with `mapdata.<key>`, `<key>` and `<key>:lookup` are merged together #}+{%-           set value = { matcher.query | regex_replace(":lookup$", ""): normalized_value } %}+{%-         else %}+{%-           set value = normalized_value %}+{%-         endif %}++{%-         do salt["log.debug"](+              log_prefix+              ~ "merge "+              ~ "sub key " * is_sub_key+              ~ "'" ~ matcher.query+              ~ "' retrieved with '"+              ~ matcher.query_method ~ "'"+              ~ ", merge: strategy='"+              ~ strategy+              ~ "', lists='"+              ~ merge_lists+              ~ "'"+              ~ ":\n"+              ~ value | yaml(False)+            ) %}++{%-         do stack.update(+              {+                "values": salt["slsutil.merge"](+                  stack["values"],+                  value,+                  strategy=strategy,+                  merge_lists=merge_lists,+                )+              }+            ) %}++{%-       else %}+{#-         Load YAML file matching the grain/pillar/... #}+{#-         Fallback to use the source name as a direct filename #}++{%-         if matcher.value | length == 0 %}+{#-           Mangle `matcher.value` to use it as literal path #}+{%-           set query_parts = matcher.query.split("/") %}+{%-           set yaml_dirname = query_parts[0:-1] | join("/") %}+{%-           set yaml_names = query_parts[-1] %}+{%-         else %}+{%-           set yaml_dirname = matcher.query %}+{%-           set yaml_names = matcher.value %}+{%-         endif %}++{#-         Some configuration return list #}+{%-         if yaml_names is string %}+{%-           set yaml_names = [yaml_names] %}+{%-         endif %}++{#-         `yaml_dirname` can be an empty string with literal path like `myconf.yaml` #}+{%-         set yaml_dir = [+              param_dir,+              yaml_dirname+            ]+            | select+            | join("/")+            %}++{%-         for yaml_name in yaml_names %}+{#-           Make sure to have a `.yaml` extension #}+{#-           Use `.rpatition` to strip last `.yaml` in `dir.yaml/file.yaml` #}+{%-           set yaml_filename = [+                yaml_dir.rstrip("/"),+                yaml_name.rpartition(".yaml")+                | reject("equalto", ".yaml")+                | join+                ~ ".yaml"+              ]+              | select+              | join("/")+              %}++{%-           do salt["log.debug"](+                log_prefix+                ~ "load configuration values from "+                ~ yaml_filename+              ) %}+{%-           load_yaml as yaml_values %}+{%-             include  yaml_filename ignore missing %}+{%-           endload %}++{%-           if yaml_values %}+{%-             do salt["log.debug"](+                  log_prefix+                  ~ "loaded configuration values from "+                  ~ yaml_name+                  ~ ":\n"+                  ~ yaml_values+                    | yaml(False)+                ) %}++{#-             `slsutil.merge` options from #}+{#-             1. the `value` #}+{#-             3. the `defaults` #}+{#-             4. the built-in #}+{%-             set strategy = yaml_values+                  | traverse(+                    "strategy",+                    defaults | traverse("strategy", "smart")+                  ) %}+{%-             set merge_lists = yaml_values+                  | traverse(+                    "merge_lists",+                    defaults | traverse("merge_lists", False) | to_bool+                  ) %}

You are right, there was a possible bug where only merge_lists from defaults where converted and not from the value itself. I applied the suggested style too. Thanks.

baby-gnu

comment created time in 2 months

Pull request review commentsaltstack-formulas/openssh-formula

feat(map): use targeting like syntax for configuration

+{#- -*- coding: utf-8 -*- #}+{#- vim: ft=jinja #}++{#- Get the `tplroot` from `tpldir` #}+{%- set tplroot = tpldir.split("/")[0] %}+{%- from tplroot ~ "/libmatchers.jinja" import parse_matchers, query_map %}++{%- set _default_config_dirs = [+      "parameters/",+      tplroot ~ "/parameters"+    ] %}++{%- macro mapstack(+      matchers,+      defaults=None,+      dirs=_default_config_dirs,+      log_prefix="libmapstack: "+    ) %}+{#-+    Load configuration in the order of `matchers` and merge+    successively the values with `defaults`.++    The `matchers` are processed using `libmatchers.jinja` to select+    the configuration sources from where the values are loaded.++    Parameters:++      - `matchers`: list of matchers in the form+        `[<TYPE>[:<OPTION>[:<DELIMITER>]]@]<QUERY>`++      - `defaults`: dictionary of default values to start the merging,+        they are considered built-ins. It must conform to the same+        layout as the YAML files: a mandatory `values` key and two+        optional `strategy` and `merge_lists` keys.++      - `dirs`: list of directory where to look-up the configuration+        file matching the matchers, by default a global `salt://parameters/`+        and a per formula `salt://<tplroot>/parameters`++      - `log_prefix`: prefix used in the log outputs, by default it is+        `libmapstack: `++    Example: On a Debian system with `roles=["nginx/server", "telegraf"]`++      {%- set settings = mapstack(+            matchers=[+              "Y:G@os_family",+              "I@" ~ tplroot,+              "Y:C@roles",+            ],+            dirs=["defaults", tplroot ~ "/parameters"],+          )+          | load_yaml+          %}

I selected the first one to stay consistent with other files.

baby-gnu

comment created time in 2 months

Pull request review commentsaltstack-formulas/openssh-formula

feat(map): use targeting like syntax for configuration

+{#- -*- coding: utf-8 -*- #}+{#- vim: ft=jinja #}++{#- Get the `tplroot` from `tpldir` #}+{%- set tplroot = tpldir.split("/")[0] %}+{%- from tplroot ~ "/libmatchers.jinja" import parse_matchers, query_map %}++{%- set _default_config_dirs = [+      "parameters/",+      tplroot ~ "/parameters"+    ] %}++{%- macro mapstack(+      matchers,+      defaults=None,+      dirs=_default_config_dirs,+      log_prefix="libmapstack: "+    ) %}+{#-+    Load configuration in the order of `matchers` and merge+    successively the values with `defaults`.++    The `matchers` are processed using `libmatchers.jinja` to select+    the configuration sources from where the values are loaded.++    Parameters:++      - `matchers`: list of matchers in the form+        `[<TYPE>[:<OPTION>[:<DELIMITER>]]@]<QUERY>`++      - `defaults`: dictionary of default values to start the merging,+        they are considered built-ins. It must conform to the same+        layout as the YAML files: a mandatory `values` key and two+        optional `strategy` and `merge_lists` keys.++      - `dirs`: list of directory where to look-up the configuration+        file matching the matchers, by default a global `salt://parameters/`+        and a per formula `salt://<tplroot>/parameters`++      - `log_prefix`: prefix used in the log outputs, by default it is+        `libmapstack: `++    Example: On a Debian system with `roles=["nginx/server", "telegraf"]`++      {%- set settings = mapstack(+            matchers=[+              "Y:G@os_family",+              "I@" ~ tplroot,+              "Y:C@roles",+            ],+            dirs=["defaults", tplroot ~ "/parameters"],+          )+          | load_yaml+          %}++      This will merge the values:++        - starting with the default empty dictionary `{}` (no+          `defaults` parameter)++        - from the YAML files++          - `salt://defaults/os_family/Debian.yaml`++          - `salt://{{ tplroot }}/parameters/os_family/Debian.yaml`++        - from the pillar `salt["pillar.get"](tplroot)`++        - from the `nginx/server` YAML files:++          - `salt://defaults/roles/nginx/server.yaml`++          - `salt://{{ tplroot }}/parameters/roles/nginx/server.yaml`++        - from the `telegraf` YAML files:++          - `salt://defaults/roles/telegraf.yaml`++          - `salt://{{ tplroot }}/parameters/roles/telegraf.yaml`++    Each YAML file and the `defaults` parameters must conform to the+    following layout:++      - a mandatory `values` key to store the configuration values++      - two optional keys to configure the use of `salt.slsutil.merge`++        - an optional `strategy` key to configure the merging+          strategy, for example `strategy: 'recurse'`, the default is+          `smart`++        - an optional `merge_lists` key to configure if lists should+          be merged or overridden for the `recurse` and `overwrite`+          strategies, for example `merge_lists: 'true'`+#}+{%-   set stack = defaults | default({"values": {} }, boolean=True) %}++{#-   Build configuration file names based on matchers #}+{%-   set matchers = parse_matchers(+        matchers,+        log_prefix=log_prefix+      )+      | load_yaml+      %}++{%-   do salt["log.debug"](+        log_prefix+        ~ "built-in configuration:\n"+        ~ {"values": defaults | traverse("values")}+          | yaml(False)+      ) %}++{%-   for param_dir in dirs %}+{%-     for matcher in matchers %}+{#-       `slsutil.merge` options from #}+{#-       1. the `value` #}+{#-       3. the `defaults` #}+{#-       4. the built-in #}+{%-       set strategy = matcher.value+          | traverse(+            "strategy",+            defaults+            | traverse(+              "strategy",+              "smart"+            )+          ) %}+{%-       set merge_lists = matcher.value+          | traverse(+            "merge_lists",+            defaults+            | traverse(+              "merge_lists",+              False+            )+          )+          | to_bool+          %}++{%-       if matcher.type in query_map.keys() %}+{#-         No value is an empty list, must be a dict for `stack.update` #}+{%-         set normalized_value = matcher.value | default({}, boolean=True) %}++{#-         Merge in `mapdata.<query>` instead of directcly in `mapdata` #}+{%-         set is_sub_key = matcher.option | default(False) == "SUB" %}+{%-         if is_sub_key %}+{#-           Merge values with `mapdata.<key>`, `<key>` and `<key>:lookup` are merged together #}+{%-           set value = { matcher.query | regex_replace(":lookup$", ""): normalized_value } %}+{%-         else %}+{%-           set value = normalized_value %}+{%-         endif %}++{%-         do salt["log.debug"](+              log_prefix+              ~ "merge "+              ~ "sub key " * is_sub_key+              ~ "'" ~ matcher.query+              ~ "' retrieved with '"+              ~ matcher.query_method ~ "'"+              ~ ", merge: strategy='"+              ~ strategy+              ~ "', lists='"+              ~ merge_lists+              ~ "'"+              ~ ":\n"+              ~ value | yaml(False)+            ) %}++{%-         do stack.update(+              {+                "values": salt["slsutil.merge"](+                  stack["values"],+                  value,+                  strategy=strategy,+                  merge_lists=merge_lists,+                )+              }+            ) %}++{%-       else %}+{#-         Load YAML file matching the grain/pillar/... #}+{#-         Fallback to use the source name as a direct filename #}++{%-         if matcher.value | length == 0 %}+{#-           Mangle `matcher.value` to use it as literal path #}+{%-           set query_parts = matcher.query.split("/") %}+{%-           set yaml_dirname = query_parts[0:-1] | join("/") %}+{%-           set yaml_names = query_parts[-1] %}+{%-         else %}+{%-           set yaml_dirname = matcher.query %}+{%-           set yaml_names = matcher.value %}+{%-         endif %}++{#-         Some configuration return list #}+{%-         if yaml_names is string %}+{%-           set yaml_names = [yaml_names] %}+{%-         endif %}++{#-         `yaml_dirname` can be an empty string with literal path like `myconf.yaml` #}+{%-         set yaml_dir = [+              param_dir,+              yaml_dirname+            ]+            | select+            | join("/")+            %}++{%-         for yaml_name in yaml_names %}+{#-           Make sure to have a `.yaml` extension #}+{#-           Use `.rpatition` to strip last `.yaml` in `dir.yaml/file.yaml` #}+{%-           set yaml_filename = [+                yaml_dir.rstrip("/"),+                yaml_name.rpartition(".yaml")+                | reject("equalto", ".yaml")+                | join+                ~ ".yaml"+              ]+              | select+              | join("/")+              %}++{%-           do salt["log.debug"](+                log_prefix+                ~ "load configuration values from "+                ~ yaml_filename+              ) %}+{%-           load_yaml as yaml_values %}+{%-             include  yaml_filename ignore missing %}+{%-           endload %}++{%-           if yaml_values %}+{%-             do salt["log.debug"](+                  log_prefix+                  ~ "loaded configuration values from "+                  ~ yaml_name+                  ~ ":\n"+                  ~ yaml_values+                    | yaml(False)+                ) %}++{#-             `slsutil.merge` options from #}+{#-             1. the `value` #}+{#-             3. the `defaults` #}+{#-             4. the built-in #}

same as above.

baby-gnu

comment created time in 2 months

Pull request review commentsaltstack-formulas/openssh-formula

feat(map): use targeting like syntax for configuration

+{#- -*- coding: utf-8 -*- #}+{#- vim: ft=jinja #}++{#- Get the `tplroot` from `tpldir` #}+{%- set tplroot = tpldir.split("/")[0] %}+{%- from tplroot ~ "/libmatchers.jinja" import parse_matchers, query_map %}++{%- set _default_config_dirs = [+      "parameters/",+      tplroot ~ "/parameters"+    ] %}++{%- macro mapstack(+      matchers,+      defaults=None,+      dirs=_default_config_dirs,+      log_prefix="libmapstack: "+    ) %}+{#-+    Load configuration in the order of `matchers` and merge+    successively the values with `defaults`.++    The `matchers` are processed using `libmatchers.jinja` to select+    the configuration sources from where the values are loaded.++    Parameters:++      - `matchers`: list of matchers in the form+        `[<TYPE>[:<OPTION>[:<DELIMITER>]]@]<QUERY>`++      - `defaults`: dictionary of default values to start the merging,+        they are considered built-ins. It must conform to the same+        layout as the YAML files: a mandatory `values` key and two+        optional `strategy` and `merge_lists` keys.++      - `dirs`: list of directory where to look-up the configuration+        file matching the matchers, by default a global `salt://parameters/`+        and a per formula `salt://<tplroot>/parameters`++      - `log_prefix`: prefix used in the log outputs, by default it is+        `libmapstack: `++    Example: On a Debian system with `roles=["nginx/server", "telegraf"]`++      {%- set settings = mapstack(+            matchers=[+              "Y:G@os_family",+              "I@" ~ tplroot,+              "Y:C@roles",+            ],+            dirs=["defaults", tplroot ~ "/parameters"],+          )+          | load_yaml+          %}++      This will merge the values:++        - starting with the default empty dictionary `{}` (no+          `defaults` parameter)++        - from the YAML files++          - `salt://defaults/os_family/Debian.yaml`++          - `salt://{{ tplroot }}/parameters/os_family/Debian.yaml`++        - from the pillar `salt["pillar.get"](tplroot)`++        - from the `nginx/server` YAML files:++          - `salt://defaults/roles/nginx/server.yaml`++          - `salt://{{ tplroot }}/parameters/roles/nginx/server.yaml`++        - from the `telegraf` YAML files:++          - `salt://defaults/roles/telegraf.yaml`++          - `salt://{{ tplroot }}/parameters/roles/telegraf.yaml`++    Each YAML file and the `defaults` parameters must conform to the+    following layout:++      - a mandatory `values` key to store the configuration values++      - two optional keys to configure the use of `salt.slsutil.merge`++        - an optional `strategy` key to configure the merging+          strategy, for example `strategy: 'recurse'`, the default is+          `smart`++        - an optional `merge_lists` key to configure if lists should+          be merged or overridden for the `recurse` and `overwrite`+          strategies, for example `merge_lists: 'true'`+#}+{%-   set stack = defaults | default({"values": {} }, boolean=True) %}++{#-   Build configuration file names based on matchers #}+{%-   set matchers = parse_matchers(+        matchers,+        log_prefix=log_prefix+      )+      | load_yaml+      %}++{%-   do salt["log.debug"](+        log_prefix+        ~ "built-in configuration:\n"+        ~ {"values": defaults | traverse("values")}+          | yaml(False)+      ) %}++{%-   for param_dir in dirs %}+{%-     for matcher in matchers %}+{#-       `slsutil.merge` options from #}+{#-       1. the `value` #}+{#-       3. the `defaults` #}+{#-       4. the built-in #}+{%-       set strategy = matcher.value+          | traverse(+            "strategy",+            defaults+            | traverse(+              "strategy",+              "smart"+            )+          ) %}+{%-       set merge_lists = matcher.value+          | traverse(+            "merge_lists",+            defaults+            | traverse(+              "merge_lists",+              False+            )+          )+          | to_bool+          %}++{%-       if matcher.type in query_map.keys() %}+{#-         No value is an empty list, must be a dict for `stack.update` #}+{%-         set normalized_value = matcher.value | default({}, boolean=True) %}++{#-         Merge in `mapdata.<query>` instead of directcly in `mapdata` #}

Thanks.

baby-gnu

comment created time in 2 months

Pull request review commentsaltstack-formulas/openssh-formula

feat(map): use targeting like syntax for configuration

+{#- -*- coding: utf-8 -*- #}+{#- vim: ft=jinja #}++{#- Get the `tplroot` from `tpldir` #}+{%- set tplroot = tpldir.split("/")[0] %}+{%- from tplroot ~ "/libmatchers.jinja" import parse_matchers, query_map %}++{%- set _default_config_dirs = [+      "parameters/",+      tplroot ~ "/parameters"+    ] %}++{%- macro mapstack(+      matchers,+      defaults=None,+      dirs=_default_config_dirs,+      log_prefix="libmapstack: "+    ) %}+{#-+    Load configuration in the order of `matchers` and merge+    successively the values with `defaults`.++    The `matchers` are processed using `libmatchers.jinja` to select+    the configuration sources from where the values are loaded.++    Parameters:++      - `matchers`: list of matchers in the form+        `[<TYPE>[:<OPTION>[:<DELIMITER>]]@]<QUERY>`++      - `defaults`: dictionary of default values to start the merging,+        they are considered built-ins. It must conform to the same+        layout as the YAML files: a mandatory `values` key and two+        optional `strategy` and `merge_lists` keys.++      - `dirs`: list of directory where to look-up the configuration+        file matching the matchers, by default a global `salt://parameters/`+        and a per formula `salt://<tplroot>/parameters`++      - `log_prefix`: prefix used in the log outputs, by default it is+        `libmapstack: `++    Example: On a Debian system with `roles=["nginx/server", "telegraf"]`++      {%- set settings = mapstack(+            matchers=[+              "Y:G@os_family",+              "I@" ~ tplroot,+              "Y:C@roles",+            ],+            dirs=["defaults", tplroot ~ "/parameters"],+          )+          | load_yaml+          %}++      This will merge the values:++        - starting with the default empty dictionary `{}` (no+          `defaults` parameter)++        - from the YAML files++          - `salt://defaults/os_family/Debian.yaml`++          - `salt://{{ tplroot }}/parameters/os_family/Debian.yaml`++        - from the pillar `salt["pillar.get"](tplroot)`++        - from the `nginx/server` YAML files:++          - `salt://defaults/roles/nginx/server.yaml`++          - `salt://{{ tplroot }}/parameters/roles/nginx/server.yaml`++        - from the `telegraf` YAML files:++          - `salt://defaults/roles/telegraf.yaml`++          - `salt://{{ tplroot }}/parameters/roles/telegraf.yaml`++    Each YAML file and the `defaults` parameters must conform to the+    following layout:++      - a mandatory `values` key to store the configuration values++      - two optional keys to configure the use of `salt.slsutil.merge`++        - an optional `strategy` key to configure the merging+          strategy, for example `strategy: 'recurse'`, the default is+          `smart`++        - an optional `merge_lists` key to configure if lists should+          be merged or overridden for the `recurse` and `overwrite`+          strategies, for example `merge_lists: 'true'`+#}+{%-   set stack = defaults | default({"values": {} }, boolean=True) %}++{#-   Build configuration file names based on matchers #}+{%-   set matchers = parse_matchers(+        matchers,+        log_prefix=log_prefix+      )+      | load_yaml+      %}++{%-   do salt["log.debug"](+        log_prefix+        ~ "built-in configuration:\n"+        ~ {"values": defaults | traverse("values")}+          | yaml(False)+      ) %}++{%-   for param_dir in dirs %}+{%-     for matcher in matchers %}+{#-       `slsutil.merge` options from #}+{#-       1. the `value` #}+{#-       3. the `defaults` #}+{#-       4. the built-in #}

Thanks, this comes from different refactor of the code without changing the comment.

baby-gnu

comment created time in 2 months

Pull request review commentsaltstack-formulas/openssh-formula

feat(map): use targeting like syntax for configuration

  {#- Get the `tplroot` from `tpldir` #} {%- set tplroot = tpldir.split("/")[0] %}-{%- from tplroot ~ "/libsaltcli.jinja" import cli with context %}+{%- from tplroot ~ "/libmapstack.jinja" import mapstack %}  {#- Where to lookup parameters source files #}-{%- set map_sources_dir = tplroot ~ "/parameters" %}--{#- Load defaults first to allow per formula default map.jinja configuration #}-{%- set _defaults_filename = map_sources_dir ~ "/defaults.yaml" %}-{%- do salt["log.debug"](-      "map.jinja: initialise parameters from "-      ~ _defaults_filename-    ) %}-{%- import_yaml _defaults_filename as default_settings %}+{%- set formula_param_dir = tplroot ~ "/parameters" %}  {#- List of sources to lookup for parameters #}-{%- do salt["log.debug"]("map.jinja: lookup 'map_jinja' configuration sources") %} {#- Fallback to previously used grains plus minion `id` #} {%- set map_sources = [-      "osarch",-      "os_family",-      "os",-      "osfinger",-      "config_get_lookup",-      "config_get",-      "id",+      "Y:G@osarch",+      "Y:G@os_family",+      "Y:G@os",+      "Y:G@osfinger",+      "C@" ~ tplroot ~ ":lookup",+      "C@" ~ tplroot,+      "Y:G@id",     ] %}-{#- Configure map.jinja from defaults.yaml #}-{%- set map_sources = default_settings | traverse(-      "values:map_jinja:sources",-      map_sources,-    ) %} -{#- Lookup global sources #}-{%- set map_sources = salt["config.get"]("map_jinja:sources", map_sources) %}-{#- Lookup per formula sources #}-{%- set map_sources = salt["config.get"](-      tplroot ~ ":map_jinja:sources",-      map_sources,-    ) %}--{%- do salt["log.debug"](-      "map.jinja: load parameters with sources from "-      ~ map_sources-    ) %}--{#- Lookup with `config.get` from configurable roots #}-{%- do salt["log.debug"](-      "map.jinja: initialise 'config.get' roots with 'tplroot' "-      ~ tplroot-    ) %}-{%- set config_get_roots = [tplroot] %}-{#- Configure `config.get` from defaults.yaml #}-{%- set config_get_roots = default_settings | traverse(-      "values:map_jinja:config_get_roots",-      config_get_roots-    ) %}-{#- Lookup global `config.get` roots #}-{%- set config_get_roots = salt["config.get"](-      "map_jinja:config_get_roots",-      config_get_roots-    ) %}-{#- Lookup per formula `config.get` roots #}-{%- set config_get_roots = salt["config.get"](-      tplroot ~ ":map_jinja:config_get_roots",-      config_get_roots,-    ) %}+{%- set _map_settings = mapstack(+      matchers=["map_jinja.yaml"],+      defaults={"values":+                 {"sources": map_sources}+               },+      log_prefix="map.jinja configuration: ",+    )+    | load_yaml+    %}++{%- set map_sources = _map_settings | traverse("values:sources") %} {%- do salt["log.debug"](-      "map.jinja: load parameters with 'config.get' from roots "-      ~ config_get_roots-    ) %}--{#- Work around assignment inside for loop #}-{#- load configuration values used in `config.get` merging strategies #}-{%- set _config = {-      "stack": default_settings.get("values", {}),-      "merge_strategy": salt["config.get"](tplroot ~ ":strategy", None),-      "merge_lists": salt["config.get"](tplroot ~ ":merge_lists", False),-    } %}--{#- the `config.get` merge option only works for `minion` or `local` salt command types #}-{%- if cli in ["minion", "local"] %}-{%-   do _config.update(-        {-          "merge_opt": {"merge": _config["merge_strategy"]},-          "merge_msg": ", merge: strategy='" ~ _config["merge_strategy"] ~ "'",-        }-      ) %}-{#- the `config.get` merge option is not available for `ssh` or `unknown` salt command types #}-{%- else %}-{%-   if _config["merge_strategy"] %}-{%-     do salt["log.error"](-          "map.jinja: the 'merge' option of 'config.get' is skipped when the salt command type is '"-          ~ cli-          ~ "'"-        ) %}-{%-   endif %}-{%-   do _config.update(-        {-          "merge_opt": {},-          "merge_msg": "",-        }-      ) %}-{%- endif %}---{#- process each `map.jinja` source #}-{%- for map_source in map_sources %}-{%-   if map_source  in ["config_get", "config_get_lookup"] %}-{%-     for _config_root in config_get_roots %}-{%-       set _config_key = {-            "config_get": _config_root,-            "config_get_lookup": _config_root ~ ":lookup",-          }.get(map_source) %}--{%-       do salt["log.debug"](-            "map.jinja: retrieve '"-            ~ _config_key-            ~ "' with 'config.get'"-            ~ _config["merge_msg"]-          ) %}-{%-       set _config_get = salt["config.get"](-            _config_key, default={}, **_config["merge_opt"]-          ) %}--{#-       `slsutil.merge` defaults to `smart` instead of `None` for `config.get` #}-{%-       set _strategy = _config["merge_strategy"] | default("smart", boolean=True) %}-{%-       do salt["log.debug"](-            "map.jinja: merge '"-            ~ _config_key-            ~ "' retrieved with 'config.get'"-            ~ ", merge: strategy='"-            ~ _strategy-            ~ "', lists='"-            ~ _config["merge_lists"]-            ~ "'"-          ) %}--{#-       Keep values under each root key when there are more than one #}-{%-       if config_get_roots|length > 1 %}-{%-         set _config_get = { _config_root: _config_get } %}-{%-       endif %}-{%-       do _config.update(-            {-              "stack": salt["slsutil.merge"](-                _config["stack"],-                _config_get,-                strategy=_strategy,-                merge_lists=_config["merge_lists"],-              )-            }-          ) %}-{%-     endfor %}-{%-   else %}-{#-     Lookup the grain/pillar/... #}-{#-     Fallback to use the source name as a direct filename #}-{%-     set map_values = salt["config.get"](map_source, []) %}--{#-     Mangle `map_source` to use it as literal path #}-{%-     if map_values | length == 0 %}-{%-       set map_source_parts = map_source.split("/") %}-{%-       set map_source = map_source_parts[0:-1] | join("/") %}-{%-       set map_values = map_source_parts[-1].rstrip(".yaml") %}-{%-     endif %}--{#-     Some configuration return list #}-{%-     if map_values is string %}-{%-       set map_values = [map_values] %}-{%-     endif %}--{%-     for map_value in map_values %}-{%-       set yamlfile = [-            map_sources_dir,-            map_source,-            map_value ~ ".yaml",-          ]-          | join("/")-          %}-{%-       do salt["log.debug"]("map.jinja: load parameters from file " ~ yamlfile) %}-{%-       load_yaml as loaded_values %}-{%-         include yamlfile ignore missing %}-{%-       endload %}--{%-       if loaded_values %}-{#-         Merge loaded values on the stack #}-{%-         do salt["log.debug"]("map.jinja: merge parameters from " ~ yamlfile) %}-{%-         do _config.update(-              {-                "stack": salt["slsutil.merge"](-                  _config["stack"],-                  loaded_values.get("values", {}),-                  strategy=loaded_values.get("strategy", "smart"),-                  merge_lists=loaded_values.get("merge_lists", False)-                  | to_bool,-                )-              }-            ) %}-{%-       endif %}-{%-     endfor %}-{%-   endif %}-{%- endfor %}+      "map.jinja: load parameters from sources"+      ~ ":\n" ~ map_sources+      | yaml(False)+    ) %}++{#- Load formula parameters values #}+{%- set _formula_matchers = ["defaults.yaml"] + map_sources %}+{%- set _formula_settings = mapstack(+      matchers=_formula_matchers,+      dirs=[formula_param_dir],+      defaults={"values": {},+                "merge_strategy": salt["config.get"](tplroot ~ ":strategy", None),+                "merge_lists": salt["config.get"](tplroot ~ ":merge_lists", False),+               },+      log_prefix="map.jinja: ",+    )+    | load_yaml+    %}

Minor style consistency issue, we're closing Jinja tags on the last line for the rest of the lines, so let's maintain that here:

    | load_yaml %}

If it needs to be maintained on a separate line, I'd propose moving it to the same indentation as the starting tag instead:

    | load_yaml
%}
baby-gnu

comment created time in 2 months