Here is how formatting works. https://catalog.kpt.dev/format/v0.1/?id=functionconfig

This falls into the category of unrecognized fields and are being sorted. I will try to figure out a way to handle this, probably not sort the unrecognized fields and retain their original order. Need to see if there is any downside due to that approach.

How big of a problem is it ? If you run render again, the order is not changed correct ? I see the problem with this comment though # You can leave these defaults. You want the order to be preserved for this. Is there a work around?

Sounds good. I am fine with it. Should we make --save functionality a default behavior then ? This makes the model consistent with kpt pkg update

kpt pkg update @v0.8 = Declare v0.8 in Kptfile + kpt pkg update kpt fn eval --image apply-setters:v0.1 -- foo=bar = declare apply-setters in the pipeline(if not present) + declare param foo=bar + kpt fn render

cc @frankfarzan @droot

And it was just my opinion. We need to come up with a design doc, weigh the pros and cons and discuss the feasibility. Then we can move forward if we get consensus.

Right, they're still tied to the immediately following line. The problem is the whole map being rearranged.

If you look at the yaml lines with comment, the comments are still tied to the same yaml line after the fields reordering.

  # This should be the name of your Config Controller instance
  cluster-name: cluster-name

  # This should be the project where you deployed Config Controller
  project-id: project-id

  # You can leave these defaults
  namespace: config-control

The comments preservation works as expected. It seems what causes the problem here is field reordering by formatting yaml.

I'm supportive of --save being the default (if a Kptfile is present).

Sounds good. I am fine with it. Should we make --save functionality a default behavior then ? This makes the model consistent with kpt pkg update

kpt pkg update @v0.8 = Declare v0.8 in Kptfile + kpt pkg update kpt fn eval --image apply-setters:v0.1 -- foo=bar = declare apply-setters in the pipeline(if not present) + declare param foo=bar + kpt fn render

cc @frankfarzan @droot

Filed https://github.com/GoogleContainerTools/kpt/issues/2333 to track function discovery

See https://github.com/GoogleContainerTools/kpt/issues/2009#issuecomment-867259380

I suggest filing a separate issue for function discovery. This issue is specifically about the doc command that shows the help message for a given function which is orthogonal to this.

This should be added to kioutil package in kyaml for functions to consume.

Here are the series of steps:

1. Add this to kioutil package and release kyaml
2. Reference it from this PR.
3. Update upsert-resource function to retain this annotation as well.
4. Release kpt and upsert-resource.

It is a bit tedious process.

Right. We could create a separate Kptfile editing command for this, as discussed in https://github.com/GoogleContainerTools/kpt/issues/2284#issuecomment-867263570. However, I could easily imagine the user wanting to actually test the operation before adding it to the Kptfile, and having to manually translate the eval command to another syntax seems unnecessary.

Hi @phanimarupaka, thanks for chiming in.

I don't agree that a hard distinction between "eval" and "render" serves our users well. We should focus on CUJs.

1. One CUJ is applying all functions which have previously been defined (often by the original package author). This is done via kpt fn render.
2. Another CUJ is adding an additional customization to a package which has not been predefined. kpt fn eval accommodates this CUJ.

A natural extension of CUJ2 is saving this customization so it can be automatically applied again in the future.

If we compare with other tools, terraform also has 2 models similar to eval and render. I don't think comparison is meaningful here. Terraform has no concept of sequential transforms.

would it make more sense to throw an error in this case or just discard it like we would a comment?

let's also add a test with a comment, --- #foo, as a regression test

@KnVerey

Eval and render serve 2 different purposes.

Eval is meant to be used for non-repeatable one-time transformations in an imperative fashion. Render is meant to be used for repetitive transformations and the functions are declared in the pipeline.

By design, eval is not the porcelain for render, but it serves a different purpose. If we introduce this flag, we are essentially making eval a porcelain for render which means

kpt fn eval --image apply-setters:v0.1 --save -- foo=bar = declare apply-setters in the pipeline + declare param foo=bar + kpt fn render

This can be done but it is beyond adding just a flag. We will be having only the declarative model.

If we compare with other tools, terraform also has 2 models similar to eval and render.

Eval equivalent: If you want to apply a tf module with adhoc values once, you can use terraform apply -var="image_id=ami-abc123". This value is not saved to tfvars making it a one time non-repetitive imperative transaction. Render equivalent: Declare vars in tfvars file and run terraform apply

I agree, the following should not be mutating the local filesystem:

kpt pkg get [email protected]:GoogleContainertools/kpt.git/bibbidi-bobbidi-boo

It should fail with an error message that the Git directory doesn't exist (not the cryptic local tmp directory).

The mdtogo is an interesting one though. kpt can have heuristics (no KRM resource found) to detect when it's likely not what the user intends, although the utility is questionable at this point.

@borismw Does the strict mode described above (to mean there should be a Kptfile in the Git directory) solve this general problem for you?

Function parameter descriptions and declarative validation are needed.

This is something in our radar already. We are trying to figure out the right UX for package publishers to provide the validation rules for function parameters without loosing the simplicity of model. These rules can also be consumed by Cloud Code to provide authoring assist to users.

For required vs optional, I like delegating that to the Kptfile rather than to the function metadata. The user could toggle the property as necessary.

There are 2 layers here.

1. Function author sets few obvious rules on function parameters. For e.g. set-namespace function needs namespace parameter to be passed. Package publisher need not worry about that.
2. Package publisher can set additional rules. For e.g. The namespace parameter should have my-org prefix and should be of maximum length of 10 chars.

Another way we could possibly treat it is as a priority spectrum: P0 must be set, P1 would be recommended, P2 would be optional, P3 would be unlikely to be changed. Or we could use words for that: an enum rather than a bool: required, recommended, optional, advanced. Or something like that. To facilitate progressive disclosure.

let's add a comment for why we are doing this

what's the motivation for declaring this outside of the function?

I thought a simple approach for kustomize might be to just match that same behavior using a regexp to allow splitting on \n---.*\n instead of \n---\n

This is the solution I was thinking of initially too. Thanks for the PR - we'll take a look when we can!

https://github.com/go-yaml/yaml/pull/753

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: <a href="https://github.com/kubernetes-sigs/kustomize/pull/3655#" title="Author self-approved">maknihamdi</a> To complete the pull request process, please assign justinsb after the PR has been reviewed. You can assign the PR to them by writing /assign @justinsb in a comment when ready.

The full list of commands accepted by this bot can be found here.

<details open> Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":["justinsb"]} -->

@maknihamdi: This PR has multiple commits, and the default merge method is: merge. You can request commits to be squashed using the label: tide/merge-method-squash

<details>

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. </details>

So quite a time passed in inactivity.

@monopole I guess this should be marked as kind/bug.

I as a argocd user, used argocd 1.6 which bundled with Kustomize v3.6.1. I upgraded argocd 2.0 which bundled with Kustomize v3.9.4, then I surprised from the multibyte string error issue. So I configured argocd to use the latest Kustomize v4.1.3, the multibyte issue fixed, but this issue happened. The order of container array changed after patch. Currently, I stick to Kustomize v3.6.1 because of this issue.

I as a Kustomize v3 user, this is a huge breaking change, I cannot upgrade to Kustomize v4. Any array order should not be changed after patch.

Can someone show me how to run the starlark function with an eval statement?

This is as part of a build pipeline, so I don't want to depend on any files existing.

v1 doesn't work for us, see #2302. Design process was behind Google auth instead of out in the open on GitHub...

Example YAML is literally any YAML with a leading comment. Should take a maintainer 30 seconds to put together

Using that k/k has done seemed like a good idea at first, but on second look, it would take quite a bit of refactoring. The approach in k/k is one of a scanning of the content, identifying each document as it is encountered. Whereas in kustomize, the entire content is split into an array of documents upfront and then processed after that.

Given that the effect of the k/k approach is to allow (and discard) any content that comes after ---, I thought a simple approach for kustomize might be to just match that same behavior using a regexp to allow splitting on \n---.*\n instead of \n---\n.

I opened a PR for this in case you want to take a look.

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: <a href="https://github.com/kubernetes-sigs/kustomize/pull/4014#" title="Author self-approved">brianpursley</a> To complete the pull request process, please assign natasha41575 after the PR has been reviewed. You can assign the PR to them by writing /assign @natasha41575 in a comment when ready.

The full list of commands accepted by this bot can be found here.

<details open> Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":["natasha41575"]} -->

Instead of splitting on \n---\n, split on \n---.*\n regular expression.

This will allow (and discard) any spaces, comments, or other content that exists on the same line as ---

Fixes #3876

Seems like starkark can help here and we are no longer blocked. Let's wait on more signal to add grep.

@ash2k: The following test failed, say /retest to rerun all failed tests:

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. </details> <!-- test report -->