profile
viewpoint

jpreese/advent-2019 1

problems from https://adventofcode.com/2019

jpreese/about 0

Sourcegraph blog, feature announcements, and website (about.sourcegraph.com)

jpreese/AbstractFactory 0

Design Patterns Project #1

jpreese/adr-docker 0

Dockerfile for ADR Tools

jpreese/afero 0

A FileSystem Abstraction System for Go

jpreese/alertmanager 0

Prometheus Alertmanager

jpreese/allReady 0

This repo contains the code for allReady, an open-source solution focused on increasing awareness, efficiency and impact of preparedness campaigns as they are delivered by humanitarian and disaster response organizations in local communities.

jpreese/angular 0

One framework. Mobile & desktop.

issue commentopen-policy-agent/conftest

Exception is outputting "No policies found" instead of registering an exception

This has been resolved in the latest, unreleased version of Conftest.

If you fetch latest, build, and use that version against the example exception-- it should work for you.

06kellyjac

comment created time in 9 hours

issue commentopen-policy-agent/conftest

`warn_something` can't be ignored with an exception (with caveat)

Resolved via https://github.com/open-policy-agent/conftest/pull/416

06kellyjac

comment created time in 9 hours

push eventjpreese/conftest

John Reese

commit sha f2cb3473d29787ee9d4af451438ad0d5555d40bf

Remove warn prefix when evaluating exceptions Signed-off-by: John Reese <john@reese.dev>

view details

push time in 9 hours

PR opened open-policy-agent/conftest

Remove warn prefix when evaluating exceptions

Resolves #413

+7 -7

0 comment

1 changed file

pr created time in 9 hours

create barnchjpreese/conftest

branch : warn-exception

created branch time in 9 hours

issue commentopen-policy-agent/conftest

--combine Feature Not Working

The docs were updated to reflect the new --combine behavior which hasn't been released yet, but will be in 0.22.0.

The previous docs for 0.21.0 that show the combine behavior can be found here

package main


deny[msg] {
  deployment := input["deployment.yaml"]["spec"]["selector"]["matchLabels"]["app"]
  service := input["service.yaml"]["spec"]["selector"]["app"]

  deployment != service

  msg = sprintf("Expected these values to be the same but received %v for deployment and %v for service", [deployment, service])
}
alexgaganashvili

comment created time in 9 hours

issue commentopen-policy-agent/conftest

Inconsistent rule evaluation

That is what that policy should do. It will print out the annotations that exist but do not have an associated nginx.ingress.kubernetes.io annotation. If the rule has a non-empty msg, the policy has failed.

grzesuav

comment created time in 9 hours

push eventjpreese/conftest

John Reese

commit sha a40248a95afad01ebbc55ad3c98d821a3758bca5

Localhost pathing Signed-off-by: John Reese <john@reese.dev>

view details

push time in 9 hours

push eventjpreese/conftest

John Reese

commit sha 5748d63e273a2f1a3a45560ef0c03ce51da80452

Localhost pathing Signed-off-by: John Reese <john@reese.dev>

view details

push time in 9 hours

push eventjpreese/conftest

John Reese

commit sha e10ac19bd98940f01ce738d64c39c9602489721e

Localhost pathing Signed-off-by: John Reese <john@reese.dev>

view details

push time in 9 hours

push eventjpreese/conftest

John Reese

commit sha 126e0f6307710c72f6f7719bf0061e08b2efeb3a

Localhost pathing Signed-off-by: John Reese <john@reese.dev>

view details

push time in 9 hours

push eventjpreese/conftest

John Reese

commit sha 19aff67282e1b818ac03f2db539ca3aa3c5c544b

Localhost pathing Signed-off-by: John Reese <john@reese.dev>

view details

push time in 9 hours

push eventjpreese/conftest

John Reese

commit sha 0ef90e0287ed7698bbdf9df11fc65a31ee544a6a

Localhost pathing Signed-off-by: John Reese <john@reese.dev>

view details

push time in 10 hours

push eventjpreese/conftest

John Reese

commit sha 3879a1ec92323c281856a2c678c9e5bdaafa8388

Localhost pathing Signed-off-by: John Reese <john@reese.dev>

view details

push time in 10 hours

push eventjpreese/conftest

John Reese

commit sha 63cb814ba037e97a0f53dbce93edd94cd6e1cefc

Localhost pathing Signed-off-by: John Reese <john@reese.dev>

view details

push time in 10 hours

PR opened open-policy-agent/conftest

Fix push command on Windows

Changes of note:

  • Fix more pathing issues on Windows for policies/data docs during push and pull.

  • Fix an inconsistency where it was possible to push to localhost, but could not pull from localhost.

  • Fix an issue where pushing to a URL that did not have a tag, but had a port, would not push as latest (e.g. localhost:5000/repo)

  • Add a --policy flag to the push command to be consistent with the other commands. Previously the path to push to the registry was supplied as an argument.

  • Remove the Loader{} type in favor of just Load(), not real reason to have to create a struct. Additionally split Load() into two. One for loading just policies, and one for loading policies and data. Noticed this when updating the tests and was being forced to pass in empty slices for the data path.

  • Added a test for push/pull using the registry image

+232 -241

0 comment

13 changed files

pr created time in 10 hours

push eventjpreese/conftest

John Reese

commit sha fab84b70f4b3edc4e2b5b0ed09116e7eb91f982d

Localhost pathing Signed-off-by: John Reese <john@reese.dev>

view details

push time in 10 hours

issue openedhashicorp/go-getter

localhost not considered a valid scheme

When the URL is localhost:80, the detector errors with "localhost is not a valid scheme". However, 127.0.0.1:80 is considered valid.

created time in 10 hours

push eventjpreese/conftest

John Reese

commit sha bb53a2e3f52eaa3280e726a92e7b5c071e77bc78

Localhost pathing Signed-off-by: John Reese <john@reese.dev>

view details

push time in 10 hours

push eventjpreese/conftest

John Reese

commit sha e0f686ae59d14fdbb4994569d5936c10a1d33d8e

Localhost pathing Signed-off-by: John Reese <john@reese.dev>

view details

push time in 11 hours

push eventjpreese/conftest

John Reese

commit sha da47bc3141934af024daeaed78c02bd9daa9d388

Localhost pathing Signed-off-by: John Reese <john@reese.dev>

view details

push time in 11 hours

push eventjpreese/conftest

John Reese

commit sha 1ac8d2c2ecd13edd233f639d00eedecc5069f2ed

Localhost pathing Signed-off-by: John Reese <john@reese.dev>

view details

push time in 12 hours

push eventjpreese/conftest

John Reese

commit sha 5c16e93cd07e21555d36c6b9065194569f1956f2

Localhost pathing Signed-off-by: John Reese <john@reese.dev>

view details

push time in 12 hours

push eventplexsystems/sinker

Yan Grunenberger

commit sha 59218eed2e2686c3102fb2ee239a9276dc7c29c0

pull.go: log on stderr instead of returning error on a per image basis (#37)

view details

push time in 13 hours

PR merged plexsystems/sinker

pull.go: log on stderr instead of returning error on a per image basis

This PR makes the pull image loop less sensitive to registry connection error - basically returns a message on stderr but does not exit sinker when the pull operation on one image is failing, which is more adapted for CI usage. Example:

INFO[0025] Pulling docker.elastic.co/kibana/kibana-oss:7.4.2-SNAPSHOT 
INFO[0026] Unable to pull docker.elastic.co/kibana/kibana-oss:7.4.2-SNAPSHOT (Retrying #1) 
INFO[0032] Unable to pull docker.elastic.co/kibana/kibana-oss:7.4.2-SNAPSHOT (Retrying #2) 
ERRO[0032] pull image and wait: retry: All attempts fail:
#1: try pull image: pull image: Error response from daemon: Get https://docker.elastic.co/v2/kibana/kibana-oss/manifests/7.4.2-SNAPSHOT: unauthorized: authentication required
#2: try pull image: pull image: Error response from daemon: Get https://docker.elastic.co/v2/kibana/kibana-oss/manifests/7.4.2-SNAPSHOT: unauthorized: authentication required 
INFO[0032] All images have been pulled!  
+1 -1

1 comment

1 changed file

ravens

pr closed time in 13 hours

pull request commentplexsystems/sinker

pull.go: log on stderr instead of returning error on a per image basis

From #38, going to check for the existence of the image and validate auth before starting to pull with sinker pull.

Approving and merging this, and we'll work off of that state! Thanks so much @ravens !

ravens

comment created time in 13 hours

issue commentplexsystems/sinker

Don't block on image pulling

That is a fair point, theres no way to fully guarantee success due to network errors. That is the hope with Retry, to mitigate those (and they would hopefully be rare!).

But this would prevent issues relating to authorization, which looks like was the problem from the error message in the PR

7.4.2-SNAPSHOT: unauthorized: authentication required
ravens

comment created time in 14 hours

push eventjpreese/conftest

John Reese

commit sha 53fe2fa6aff749739231f415886c74b81bfa4a11

initial commit Signed-off-by: John Reese <john@reese.dev>

view details

push time in a day

create barnchjpreese/conftest

branch : push-windows

created branch time in a day

issue commentplexsystems/sinker

Don't block on image pulling

Thanks for opening this issue @ravens. The scenario definitely isn't ideal.

Rather than logging an error during the pull operation, what are your thoughts on first verifying that you can successfully pull all images in the manifest? We could just call ImageExistsAtRemote on all of the images, and only pull them if all of the images are there. This would cover the case of invalid auth as well as if the image just didn't exist.

ravens

comment created time in a day

push eventjpreese/conftest

John Reese

commit sha 14bb7af5553ba2f2ce2abd1e8d60733d39e57469

Push image on release Signed-off-by: John Reese <john@reese.dev>

view details

push time in a day

push eventjpreese/conftest

John Reese

commit sha 4d199aa22ccffcfcd213e0eac7d3d8a7dd6b46a5

Push image on release Signed-off-by: John Reese <john@reese.dev>

view details

push time in a day

push eventjpreese/conftest

John Reese

commit sha 163ee5b5a577ae4e8929824fe691c0ae02b6264b

Push image on release Signed-off-by: John Reese <john@reese.dev>

view details

push time in a day

issue commentopen-policy-agent/conftest

blocks with same name no longer testable

@sarcasticadmin Awesome, much appreciated!

drewmullen

comment created time in a day

push eventjpreese/conftest

John Reese

commit sha 0a8d957f6d51a006b695a5d9ab3d1b742b5d158b

Push image on release Signed-off-by: John Reese <john@reese.dev>

view details

push time in a day

push eventjpreese/conftest

John Reese

commit sha b59d60a094fede0544d580d205c939f392af4d1b

Push image on release Signed-off-by: John Reese <john@reese.dev>

view details

push time in a day

push eventjpreese/conftest

John Reese

commit sha 827849ac41f802cddc6ae7f6cfcd98812b975349

Push image on release Signed-off-by: John Reese <john@reese.dev>

view details

push time in a day

push eventjpreese/conftest

John Reese

commit sha 450751557d12cd9fc7ae44def01183885d97fb8f

Push image on release Signed-off-by: John Reese <john@reese.dev>

view details

push time in a day

push eventjpreese/conftest

John Reese

commit sha 37db8c87ded253b08713feafa7894cc707a4e395

Push image on release Signed-off-by: John Reese <john@reese.dev>

view details

push time in a day

push eventjpreese/conftest

John Reese

commit sha fd1fc283dc5cd6e0235a2d51c562c0015c0dea6b

Push image on release Signed-off-by: John Reese <john@reese.dev>

view details

push time in a day

push eventjpreese/conftest

John Reese

commit sha 46b55d5ff7b509be725b09bb0c6ae6cb9cdf44ba

Push image on release Signed-off-by: John Reese <john@reese.dev>

view details

push time in a day

push eventjpreese/conftest

John Reese

commit sha 118fd3fb49ab4a38b2252d954430c334ea172cd5

Push image on release Signed-off-by: John Reese <john@reese.dev>

view details

push time in a day

push eventjpreese/conftest

John Reese

commit sha 3299fea86af10ce49b0175f2c4171191852c2f5f

Push image on release Signed-off-by: John Reese <john@reese.dev>

view details

push time in a day

push eventjpreese/conftest

John Reese

commit sha 27af9f43952f08287e99cdbeebc2690c4bd742e8

Push image on release Signed-off-by: John Reese <john@reese.dev>

view details

push time in a day

PR opened open-policy-agent/conftest

Push image on release

Resolves #324

+22 -19

0 comment

1 changed file

pr created time in a day

create barnchjpreese/conftest

branch : auto-docker

created branch time in a day

push eventjpreese/conftest

John Reese

commit sha b5bad18fdd56c2e8e27908609d0337bd8218ec9c

Change input flag to parser (#410) Signed-off-by: John Reese <john@reese.dev>

view details

push time in 2 days

push eventopen-policy-agent/conftest

John Reese

commit sha b5bad18fdd56c2e8e27908609d0337bd8218ec9c

Change input flag to parser (#410) Signed-off-by: John Reese <john@reese.dev>

view details

push time in 2 days

PR merged open-policy-agent/conftest

Change input flag to parser

This PR aims to change the --input flag to instead, be named --parser.

Reasoning:

The parser package exposes a number of different parsers, and these parsers are responsible for parsing different file formats. I feel its currently a little confusing to have conftest test --input tf, under the hood this just means to use the hcl2 parser, which we've done a mapping for. Instead, I think it makes more sense to say conftest test --parser hcl1 or conftest test --parser hcl2.

I do think theres value in being able to figure out which Parser to use, based on a file, which NewFromPath accomplishes.

This also keeps the language more consistent with what is really happening under the hood, figuring out which parser to use with the given input. As well as keeps the parser package focused on its parsers rather than specific implementations that use the language.

+163 -180

1 comment

7 changed files

jpreese

pr closed time in 2 days

pull request commentopen-policy-agent/conftest

Change input flag to parser

Thanks for the review! I had the same concern as well, but being that I suspect this is a low-usage flag and being pre-v1, I didn't think it worth it to maintain backwards compatibility.

jpreese

comment created time in 2 days

push eventjpreese/conftest

John Reese

commit sha 01ab5c8cbb931b77fbe874f0c7769ca185cd1ff0

Fix plugins on Windows (#400) * Fix plugins on Windows Signed-off-by: John Reese <john@reese.dev> * Run tidy Signed-off-by: John Reese <john@reese.dev>

view details

push time in 2 days

pull request commentopen-policy-agent/conftest

Combine configurations into struct when using combine

There is not currently plans to include this in non-combined mode. To get path data, #65 has been suggested. I'm not a fan of always needing index[] for policies.

jpreese

comment created time in 2 days

pull request commentopen-policy-agent/conftest

Combine configurations into struct when using combine

@benjamin-bergia we knew this would be a breaking change, but feel it's the better approach moving forward. That said, I'll look into this to see if there's anything that can be done to support both approaches in a way that makes sense.

Do you have any specific concerns with the new syntax? I definitely understand fear of rewriting policies.

jpreese

comment created time in 2 days

issue commentopen-policy-agent/conftest

Create a Windows test environment

Shouldn't be too big of a hurdle. CircleCI has matrix jobs which would let us run CI against our current environment as well as a Windows-based one.

Blokje5

comment created time in 2 days

push eventjpreese/conftest

John Reese

commit sha f2723729789776640a79c8994864df0ec8a8d957

Conflicts Signed-off-by: John Reese <john@reese.dev>

view details

push time in 2 days

push eventjpreese/conftest

John Reese

commit sha 67f5f4e54bc3196a11b618e144d27c7fc1598697

Add maintainers field (#408) Signed-off-by: John Reese <john@reese.dev>

view details

push time in 2 days

issue commentopen-policy-agent/conftest

Support XDG Specification for Plugin Directory

#1 sounds good to me. I initially hesitated to add it into the same command, because always printing the path seemed noisy to me. The plugin dir should never really change unless you're messing around with variables.

Adding some sort of flag to list to show more information about the found plugins, including its location, sounds like a good approach.

06kellyjac

comment created time in 2 days

push eventjpreese/conftest

John Reese

commit sha 037ecfae5d3f4c330e674faa7ecd2ce170d895dc

Conflicts Signed-off-by: John Reese <john@reese.dev>

view details

push time in 2 days

push eventjpreese/conftest

John Reese

commit sha d31f0c8961fd4d1bc554dc2e0a48fc42323c7f95

Initial commit (#402) Signed-off-by: John Reese <john@reese.dev>

view details

John Reese

commit sha b314e48cb146be31d6927d10349bd06fd13a4062

Conflicts Signed-off-by: John Reese <john@reese.dev>

view details

push time in 2 days

push eventjpreese/conftest

John Reese

commit sha d31f0c8961fd4d1bc554dc2e0a48fc42323c7f95

Initial commit (#402) Signed-off-by: John Reese <john@reese.dev>

view details

push time in 2 days

push eventjpreese/conftest

John Reese

commit sha c148932abeba56426884736ce1e0854e99f10e05

Change input flag to parser Signed-off-by: John Reese <john@reese.dev>

view details

push time in 2 days

Pull request review commentopen-policy-agent/conftest

Fix plugins on Windows

 cloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxK cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc= cloud.google.com/go v0.46.3 h1:AVXDdKsrtX33oR9fbCMu/+c1o8Ofjq6Ku/MInaLVg5Y= cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0=+cloud.google.com/go/bigquery v1.0.1 h1:hL+ycaJpVE9M7nLoiXb/Pn10ENE2u+oddxbD8uu0ZVU=

Not really sure, I've always considered go.sum more or less an artifact of the build and usually don't pay much attention to the results. Lots of drift can occur just based on upstream dependency changes and such!

I ran go mod tidy on the branch and pushed the resulting commit.

jpreese

comment created time in 2 days

PullRequestReviewEvent

push eventjpreese/conftest

John Reese

commit sha 9d1e428b92f384f5c42b07123fb22e7bde646587

Run tidy Signed-off-by: John Reese <john@reese.dev>

view details

push time in 2 days

PR opened open-policy-agent/conftest

Change input flag to parser

This PR aims to change the --input flag to instead, be named --parser.

Reasoning:

The parser package exposes a number of different parsers, and these parsers are responsible for parsing different file formats. I feel its currently a little confusing to have conftest test --input tf, under the hood this just means to use the hcl2 parser, which we've done a mapping for. Instead, I think it makes more sense to say conftest test --parser hcl1 or conftest test --parser hcl2.

I do think theres value in being able to figure out which Parser to use, based on a file, which NewFromPath accomplishes.

This also keeps the language more consistent with what is really happening under the hood, figuring out which parser to use with the given input. As well as keeps the parser package focused on its parsers rather than specific implementations that use the language.

+158 -175

0 comment

7 changed files

pr created time in 2 days

push eventjpreese/conftest

John Reese

commit sha 5f4be6af10319ca4e92c33e28783f995768f7361

Change input flag to parser Signed-off-by: John Reese <john@reese.dev>

view details

push time in 2 days

create barnchjpreese/conftest

branch : change-input-parser

created branch time in 2 days

push eventjpreese/conftest

John Reese

commit sha 3a772ae02ccb4f288fe75effff7c10566ee54670

Update docs to reflect newest releases (#407) * Update documentation to reflect newest changes Signed-off-by: John Reese <john@reese.dev> * Add more details around reading about policies Signed-off-by: John Reese <john@reese.dev>

view details

John Reese

commit sha 3e5e03c9371768cb96233e84581008ae5dd70ccc

Remove custom HCL2 conversion (#403) Signed-off-by: John Reese <john@reese.dev>

view details

push time in 2 days

issue commentopen-policy-agent/conftest

Support XDG Specification for Plugin Directory

I'd add that with the priority order and behavior if $XDG_XYZ is set but the directory doesn't actually exist it's probably best to log an error for the user & fall back to the next one?

In this context, is the directory the plugin directory or the directory defined in $XDG_XYZ? In the event that the plugin directory does not exist, to me, that would just mean no plugins are installed. plugin install would add the directory.

If the $XDG_XYZ directory itself does not exist, I would vote for hard erroring. If the data in that variable isn't good, then I would pressure the user to remove or fix it.

I'd say maybe it'd be better named as conftest plugin location as it avoids the confusion between $XDG_CACHE_HOME and $XDG_DATA_HOME & also reading it without context could look like a command to somehow cache your plugins?

I think that's a fair argument. Location does make sense to me. I'll look at some other tools that support this behavior and see what they end up calling it.

06kellyjac

comment created time in 2 days

issue commentopen-policy-agent/conftest

New `.conftest` dir for plugins doesn't follow XDG Base Dir spec

@Blokje5 @06kellyjac

After https://github.com/open-policy-agent/conftest/pull/400 is completed, was going to take a look at this next.

I think we should be able to detect if $XDG_DATA_HOME or $XDG_DATA_DIRS, and if so, use that as the location to store the plugins.

$XDG_DATA_HOME seems the most appropriate for where plugins should be downloaded to. While we refer to the plugin dir as a cache, it feels more like an installation folder. [ Reference discussion on $XDG_DATA_HOME ] : https://github.com/helm/helm/issues/7206

Additional consideration would be the $XDG_DATA_DIRS variable. If $XDG_DATA_HOME is not set, but XDG_DATA_DIRS is, we would parse that variable. [ Reference discussion on $XDG_DATA_DIRS and priority ] : https://github.com/KhronosGroup/Vulkan-Loader/issues/245

This would result in the following priority order and behavior:

  1. If $XDG_DATA_HOME is set, the plugins are put in $XDG_DATA_HOME/conftest/plugins

  2. If $XDG_DATA_HOME is not set, but $XDG_DATA_DIRS is set, iterate through $XDG_DATA_DIRS in the order they appear for $dir/conftest/plugins

  3. If neither $XDG_DATA_HOME or $XDG_DATA_DIRS is set, plugins will be at $HOME/conftest/plugins

I would also like to introduce new commands:

  • conftest plugin cache - Print the location of the plugin cache that is currently being looked at. This value would change based on how the XDG_ variables are set and would be incredibly useful for troubleshooting.

  • conftest plugin list - List the currently installed plugins.

I also think it makes sense to update our plugin configuration YAML to include the source URL (or path on disk) it came from for when we start talking about needing to re-download plugins automatically and checking for version updates.

Does all of this sound OK? Before this issue I have honestly not heard of the XDG_ spec so this is all how I currently understand the spec to work, but I may be misunderstanding.

06kellyjac

comment created time in 2 days

issue closedopen-policy-agent/conftest

dpkg-query: warning: parsing file '/var/lib/dpkg/status' .... missing 'Maintainer' field

When installing conftest from deb file, it puts the following content into /var/lib/dpkg/status:

Package: conftest
Status: install ok installed
Installed-Size: 33708
Architecture: amd64
Version: 0.21.0
Description: Test your configuration using Open Policy Agent
Homepage: https://github.com/open-policy-agent/conftest

On any subsequent apt operation, the following warning is displayed:

dpkg-query: warning: parsing file '/var/lib/dpkg/status' near line 2067 package 'conftest':
 missing 'Maintainer' field

closed time in 3 days

eyalzek

issue commentopen-policy-agent/conftest

dpkg-query: warning: parsing file '/var/lib/dpkg/status' .... missing 'Maintainer' field

Thanks for reporting this @eyalzek! Resolved in #408

eyalzek

comment created time in 3 days

push eventjpreese/conftest

John Reese

commit sha 6bdab3bd8c348d02268b6821f315273e6dbfec5c

Add maintainers field Signed-off-by: John Reese <john@reese.dev>

view details

push time in 3 days

PR opened open-policy-agent/conftest

Add maintainers field

Resolves #399

This field appears to be required by dpkg and .. does not support multiple aliases?

+1 -0

0 comment

1 changed file

pr created time in 3 days

create barnchjpreese/conftest

branch : maintainer-release

created branch time in 3 days

pull request commentopen-policy-agent/conftest

[DRAFT] feat(conftest): policy results as github comments and status checks

Looks like activity has stalled out on this, and @jalseth has created this functionality over at https://github.com/YubicoLabs/action-conftest if we want to route users looking for this functionality there.

naiduarvind

comment created time in 3 days

PR closed open-policy-agent/conftest

[HOLD] refactor hcl2

Fixes: #266

But I'm not sure that we should merge that one because it's not backward-compatible. Folks, who have already tests that written for conftest, will have to refactor the tests

Need thoughts on that

+45 -33

3 comments

4 changed files

boranx

pr closed time in 3 days

pull request commentopen-policy-agent/conftest

[HOLD] refactor hcl2

Closing in favor of the PR in the upstream HCL2JSON implementation https://github.com/tmccombs/hcl2json/pull/20

boranx

comment created time in 3 days

pull request commentopen-policy-agent/conftest

Refactor/separate fixtures

I think the examples will stay in bats, those have to be files on disk so it makes sense for those to live there.

There might be testdata files that are also used in bats, if the case that were testing doesn't make sense to have the scenario with inline Go. But the objective would be to reduce the reliance on it.

For steps forward, I don't mind just merging this in and we can adjust as we go. Just waiting on the other PRs to see what kind of state the repository is in after that.

xchapter7x

comment created time in 3 days

Pull request review commentopen-policy-agent/conftest

Remove custom HCL2 conversion

 func TestConversion(t *testing.T) { 	} 	for name, tc := range testTable { 		bytes := []byte(tc.input)-		conf, diags := hclsyntax.ParseConfig(bytes, "test", hcl.Pos{Byte: 0, Line: 1, Column: 1})-		if diags.HasErrors() {-			t.Errorf("Failed to parse config: %v", diags)-		}-		converted, err := convertFile(conf)--		if err != nil {-			t.Errorf("Unable to convert from hcl: %v", err)-		} -		jb, err := json.MarshalIndent(converted, "", "\t")+		json, err := convert.Bytes(bytes, "", convert.Options{})

It probably makes sense to do so. My intent here was to not change the tests to prove the upstream was backwards compatible with what we already had.

I'll be updating this again soon after we figure out https://github.com/tmccombs/hcl2json/pull/20 which will introduce breaking changes anyway. At that point, these tests matter less and we can call the Parser directly.

Thoughts?

jpreese

comment created time in 3 days

PullRequestReviewEvent

Pull request review commenttmccombs/hcl2json

Always use sets for block values

 func File(file *hcl.File, options Options) ([]byte, error) {  type jsonObj map[string]interface{} -func convertFile(file *hcl.File, options Options) (jsonObj, error) {-	c := converter{bytes: file.Bytes, options: options}-	body := file.Body.(*hclsyntax.Body)--	return c.convertBody(body)-}- type converter struct { 	bytes   []byte 	options Options } -func (c *converter) rangeSource(r hcl.Range) string {-	// for some reason the range doesn't include the ending paren, so-	// check if the next character is an ending paren, and include it if it is.-	end := r.End.Byte-	if c.bytes[end] == ')' {-		end+++func convertFile(file *hcl.File, options Options) (jsonObj, error) {+	body, ok := file.Body.(*hclsyntax.Body)+	if !ok {+		return nil, fmt.Errorf("convert file body to body type") 	}-	return string(c.bytes[r.Start.Byte:end])++	c := converter{+		bytes:   file.Bytes,+		options: options,+	}++	out, err := c.convertBody(body)+	if err != nil {+		return nil, fmt.Errorf("convert body: %w", err)+	}++	return out, nil }  func (c *converter) convertBody(body *hclsyntax.Body) (jsonObj, error) {-	var err error 	out := make(jsonObj)-	for key, value := range body.Attributes {-		out[key], err = c.convertExpression(value.Expr)-		if err != nil {-			return nil, err++	for _, block := range body.Blocks {+		if err := c.convertBlock(block, out); err != nil {+			return nil, fmt.Errorf("convert block: %w", err) 		} 	} -	for _, block := range body.Blocks {-		err = c.convertBlock(block, out)+	var err error+	for key, value := range body.Attributes {+		out[key], err = c.convertExpression(value.Expr) 		if err != nil {-			return nil, err+			return nil, fmt.Errorf("convert expression: %w", err) 		} 	}  	return out, nil } -func (c *converter) convertBlock(block *hclsyntax.Block, out jsonObj) error {-	var key string = block.Type--	value, err := c.convertBody(block.Body)-	if err != nil {-		return err+func (c *converter) rangeSource(r hcl.Range) string {+	// for some reason the range doesn't include the ending paren, so+	// check if the next character is an ending paren, and include it if it is.+	end := r.End.Byte+	if c.bytes[end] == ')' {+		end++ 	}+	return string(c.bytes[r.Start.Byte:end])+} +func (c *converter) convertBlock(block *hclsyntax.Block, out jsonObj) error {+	key := block.Type 	for _, label := range block.Labels {-		if inner, exists := out[key]; exists {++		// Labels represented in HCL are defined as quoted strings after the name of the block:+		// block "label_one" "label_two"+		//+		// Labels represtend in JSON are nested one after the other:+		// "label_one": {+		//   "label_two": {}+		// }+		//+		// To create the JSON representation, check to see if the label exists in the current output:+		//+		// When the label exists, move onto the next label reference.+		// When a label does not exist, create the label in the output and set that as the next label reference+		// in order to append (potential) labels to it.+		if _, exists := out[key]; exists { 			var ok bool-			out, ok = inner.(jsonObj)+			out, ok = out[key].(jsonObj) 			if !ok {-				// TODO: better diagnostics 				return fmt.Errorf("Unable to convert Block to JSON: %v.%v", block.Type, strings.Join(block.Labels, ".")) 			} 		} else {-			obj := make(jsonObj)-			out[key] = obj-			out = obj+			out[key] = make(jsonObj)+			out = out[key].(jsonObj)

For both of these, this tripped me up for quite a bit.

When using out[key] =, we're actually changing the value passed in by the user. When using out =, we're just setting the "next" out value to get the next label.

All of this makes sense in hindsight, but creating the single obj and setting it to both out[key] as well as out wasn't immediately clear.

This feels more natural in that we're creating a new value at out[key] and then setting out to that value that we just created.

I'm more than happy to revert any of this. It really was just trying to organize things in my head, but I don't want to make it confusing for you if the old way made more sense to you.

jpreese

comment created time in 3 days

PullRequestReviewEvent

Pull request review commenttmccombs/hcl2json

Always use sets for block values

 func File(file *hcl.File, options Options) ([]byte, error) {  type jsonObj map[string]interface{} -func convertFile(file *hcl.File, options Options) (jsonObj, error) {-	c := converter{bytes: file.Bytes, options: options}-	body := file.Body.(*hclsyntax.Body)--	return c.convertBody(body)-}- type converter struct { 	bytes   []byte 	options Options } -func (c *converter) rangeSource(r hcl.Range) string {-	// for some reason the range doesn't include the ending paren, so-	// check if the next character is an ending paren, and include it if it is.-	end := r.End.Byte-	if c.bytes[end] == ')' {-		end+++func convertFile(file *hcl.File, options Options) (jsonObj, error) {+	body, ok := file.Body.(*hclsyntax.Body)+	if !ok {+		return nil, fmt.Errorf("convert file body to body type") 	}-	return string(c.bytes[r.Start.Byte:end])++	c := converter{+		bytes:   file.Bytes,+		options: options,+	}++	out, err := c.convertBody(body)+	if err != nil {+		return nil, fmt.Errorf("convert body: %w", err)+	}++	return out, nil }  func (c *converter) convertBody(body *hclsyntax.Body) (jsonObj, error) {-	var err error 	out := make(jsonObj)-	for key, value := range body.Attributes {-		out[key], err = c.convertExpression(value.Expr)-		if err != nil {-			return nil, err++	for _, block := range body.Blocks {

It does not, no. This must have just gotten moved as I was trying to make sense of the codebase and how the HCL parser bucketed each part.

jpreese

comment created time in 3 days

PullRequestReviewEvent

issue commentopen-policy-agent/conftest

blocks with same name no longer testable

@drewmullen @sarcasticadmin

PR to set-i-fy all blocks here. Additional discussions on the related issue.

https://github.com/tmccombs/hcl2json/pull/20

drewmullen

comment created time in 3 days

PR opened tmccombs/hcl2json

Always use sets for block values

For https://github.com/tmccombs/hcl2json/issues/18

Disclaimer: 99.9% of the change is just always wrapping value on line 125 (https://github.com/tmccombs/hcl2json/blob/master/convert/convert.go#L125) inside of []interface{}.

The other diffs are mostly from me just trying to understand and document the behavior.

Without going into Terraform-specific checks, this feels like the path forward. For Terraform, it might be a little silly for some blocks, like variables:

variable "my_var" {
  type = string
}

will produce

"variable": {
  "my_var": [
    {
       "type": "${string}"
    }
   ]
}

Even though its impossible for it to have more than one. But in HCL, a block is a block.

+207 -91

0 comment

2 changed files

pr created time in 3 days

push eventjpreese/hcl2json

John Reese

commit sha c269182a1356d3e85b1f6b64250c8a4a77cc6364

Always use sets for block types

view details

push time in 3 days

delete branch plexsystems/plexsystems.github.io

delete branch : versionless-graphql

delete time in 3 days

Pull request review commentopen-policy-agent/conftest

Update docs to reflect newest releases

 # Conftest -Conftest is a utility to help you write tests against structured configuration data. For instance you could write tests for your Kubernetes configurations, or Tekton pipeline definitions, Terraform code, Serverless configs or any other structured data.+Conftest is a utility to help you write tests against structured configuration data. For instance, you could write tests for your Kubernetes configurations, Tekton pipeline definitions, Terraform code, Serverless configs or any other structured data.  Conftest relies on the Rego language from [Open Policy Agent](https://www.openpolicyagent.org/) for writing the assertions. You can read more about Rego in [How do I write policies](https://www.openpolicyagent.org/docs/how-do-i-write-policies.html) in the Open Policy Agent documentation.  ## Usage -Conftest allows you to write policies using Open Policy Agent/rego and apply them to one or-more configuration files. Policies by default should be placed in a directory called `policy` but this can be overridden.+Policies by default should be placed in a directory called `policy`, but this can be overridden with the `--policy` flag.

Good catch, I do prefer the consistency to keep calling them policies. I also updated the link to the documentation site.

jpreese

comment created time in 3 days

PullRequestReviewEvent

push eventjpreese/conftest

John Reese

commit sha 2127984f981813cf7c271ef063d23b30e13d3fdc

Add more details around reading about policies Signed-off-by: John Reese <john@reese.dev>

view details

push time in 3 days

Pull request review commentopen-policy-agent/conftest

Update docs to reflect newest releases

 Before submitting large changes, please open an issue on GitHub outlining: - Detailed description of what your changes would entail. - Alternative solutions or approaches if applicable. -Use your judgement about what constitutes a large change. If you aren't sure, send a message to the `#conftest` channel in the OPA slack or submit an issue on GitHub.+Use your judgment about what constitutes a large change. If you aren't sure, send a message to the `#conftest` channel in the OPA slack or submit an issue on GitHub.

LOL I actually Google'd this last night. Judgment appears to be the "more correct" approach according to what I'm seeing, but in the end both are acceptable and didn't necessarily warrant a change. I just didn't bother going back and undoing it.

jpreese

comment created time in 3 days

PullRequestReviewEvent

push eventjpreese/conftest

John Reese

commit sha 336af58e51c52c9555d1c1f99b595b80cb21fe81

Add git to Dockerfile (#404) Signed-off-by: John Reese <john@reese.dev>

view details

John Reese

commit sha cf13dffb7061aba3ef46b4d8811f4e67c7fc006d

Remove runtime check on YAML parser (#406) Signed-off-by: John Reese <john@reese.dev>

view details

John Reese

commit sha 1d94894027dbde1d813862a4402fc6311cad8f44

Sort combined results by file path (#405) Signed-off-by: John Reese <john@reese.dev>

view details

push time in 3 days

create barnchjpreese/hcl2json

branch : provider-array

created branch time in 4 days

push eventjpreese/hcl2json

John Reese

commit sha cc821717a9997100183e8b3ff85f984f0faea191

Remove indentation from package

view details

Thayne McCombs

commit sha 92a7ef6bb3b9acedcb9e46a0bb3601866e0b1fd6

Merge pull request #19 from jpreese/remove-indent Remove indentation from package

view details

push time in 4 days

pull request commentopen-policy-agent/conftest

Refactor/separate fixtures

I agree with the idea of splitting up examples from testdata, but I'm not sure which files actually need to be moved and which could be removed in favor of inline Rego in a Go test. I have a lot of other PRs that touch on tests, and don't necessarily want to cause conflicts when its unknown if the data needs to move.

I planned on going through and combing through the current test suite and making sure that everything is covered appropriately, but my hope is that testdata would be incredibly small, maybe not even needed.

xchapter7x

comment created time in 4 days

push eventjpreese/conftest

John Reese

commit sha 0a5d15321a37ca17e33d8008bd4fa846dfa4e2d8

Update documentation to reflect newest changes Signed-off-by: John Reese <john@reese.dev>

view details

push time in 4 days

more