profile
viewpoint

meori/kubespray 0

Setup a kubernetes cluster

mildebrandt/CASSANDRA-13991 0

https://issues.apache.org/jira/browse/CASSANDRA-13991

mildebrandt/cFIREsim-open 0

An attempt to re-write the popular retirement calculator (cFIREsim.com) in mainly javascript, and begin the journey toward open-source.

mildebrandt/confluent-kafka-go-191 0

Project to recreate https://github.com/confluentinc/confluent-kafka-go/issues/191

mildebrandt/fish-kubectl-completions 0

kubectl completions for fish shell

mildebrandt/influxdb 0

Scalable datastore for metrics, events, and real-time analytics

mildebrandt/telegraf 0

The plugin-driven server agent for collecting & reporting metrics.

issue closed23andMe/Yamale

Validate struct of dict

How dow I validate this yaml structure:

people:
  personA:
    name: Some Name
    age: Some Age
  personB:
    name: Some Other Name
    age: Some Other Age

The problem is that this is not a list, it will be parsed into a dict structure. In python this would then be ["people"]["personA"]["age"] for instance.

closed time in 5 days

mflage

issue comment23andMe/Yamale

Validate struct of dict

Hi, thanks for your interest in Yamale. You can use the following schema:

people: map(include('person'))
---
person:
  name: str()
  age: int()

Have fun!

mflage

comment created time in 5 days

issue comment23andMe/Yamale

Custom validators require `eval`

In yaml, timestamp is an actual type: https://yaml.org/type/timestamp.html

So when the yaml parser reads in the file, it converts 2020-03-23 into a Python datetime object.

Yaml doesn't know about Python objects like Range(). I'd avoid using yaml like this, and I'd caution against using eval anywhere in your code....but especially in a place that processes user-provided strings that aren't cleaned before running the eval. You don't want the string coming in to end up being something like: "__import__('os').system('rm -rf /')"

Instead, I would suggest using a string for the range, like "1-20", and parse that in your validator. Or use the built-in regex validator.

McCroden

comment created time in 7 days

issue comment23andMe/Yamale

Custom validators require `eval`

Hi, thanks for using Yamale!

Can you provide the schema file, data file, and actual code you're testing with?

Here's mine that works on MacOS 10.14.6, Python 3.7.6, Yamale 2.0.1:

schema.yaml:

start_time: include('date_validator')
---
date_validator: date()

data.yaml:

start_time: 2020-03-23

test.py:

import yamale
import datetime
from yamale.validators import DefaultValidators, Validator

class Date(Validator):
    """ Custom Date validator """
    tag = 'date'
    def _is_valid(self, value):
        return isinstance(value, datetime.date)

validators = DefaultValidators.copy()  # This is a dictionary
validators[Date.tag] = Date
schema = yamale.make_schema('./schema.yaml', validators=validators)
data = yamale.make_data('./data.yaml')
yamale.validate(schema, data)
McCroden

comment created time in 8 days

issue openedaws-cloudformation/aws-cloudformation-coverage-roadmap

AWS::EFS::AccessPoint

Instructions for CloudFormation Coverage New Issues Template

Quick Sample Summary:

  1. Title -> AWS::Service::Resource-Attribute-Existing Attribute
  2. Scope of request -> AWS::Service::ResourceType::TheResource-ExistingAttribute supports A and B today, but not new C feature.
  3. Expected behavior -> In Create, it should include this and this, like the console. Also include tag creation, like the API allows.
  4. Test case recommendation (optional) -> it should help me do X given existing Y (if applicable); it should/shouldn't change Z.
  5. Links to existing API doc (optional) -> Updated API docs at https://example.com/api-new.html
  6. Category tag (optional) -> Compute
  7. Any additional context (optional)

Detailed Instructions to create a new issue here: (delete this text before you save your issue, to keep them as short as possible :) ).

1. Title

AWS::EFS::AccessPoint

2. Scope of request

AWS::EFS::AccessPoint - Can create an access point from the API, but not via CF

6. Category (required) - Will help with tagging and be easier to find by other users to +1

  1. Storage (S3, EFS, Backup...)

created time in a month

issue commentinfluxdata/telegraf

Create a debian/rpm package with a telegraf version that has plugin support

I'm beginning to think there's no good way to support this with Go's limitations right now.

https://github.com/golang/go/issues/27751

https://github.com/golang/go/issues/26759

https://www.reddit.com/r/golang/comments/b6h8qq/is_anyone_actually_using_go_plugins/ejkxd2k/

mildebrandt

comment created time in a month

issue commentinfluxdata/influxdata-docker

Create a docker container with plugin support

I'm beginning to think there's no good way to support this with Go's limitations right now.

https://github.com/golang/go/issues/27751

https://github.com/golang/go/issues/26759

https://www.reddit.com/r/golang/comments/b6h8qq/is_anyone_actually_using_go_plugins/ejkxd2k/

mildebrandt

comment created time in a month

issue openedinfluxdata/telegraf

Docker container for telegraf 1.13.3 is not available

Please kick-off the packaging process to build and publish a telegraf 1.13.3 container. Thanks!

created time in a month

issue closedinfluxdata/telegraf

Unable to load external plugins -- plugin: not implemented

When attempting to load an external plugin, I get the following error message:

2020-02-14T23:23:54Z I! Loading external plugins from: /usr/lib/telegraf-plugins/
2020-02-14T23:23:54Z E! error loading /usr/lib/telegraf-plugins/telegraf-plugins.so: plugin: not implemented

I built telegraf with the goplugin build tag and built a sample external plugin based on https://github.com/danielnelson/telegraf-plugins.

Here are the instructions to reproduce the error, also outlined at https://github.com/mildebrandt/telegraf-plugins/blob/master/README.md:

build telegraf

go get github.com/influxdata/telegraf/
cd ~/go/src/github.com/influxdata/telegraf/
git checkout 1.13.3
make
./scripts/build.py --package --platform=linux --arch=amd64 --build-tags=goplugin

package telegraf in a container

go get github.com/mildebrandt/telegraf-plugins/
cd ~/go/src/github.com/mildebrandt/telegraf-plugins/
cp ~/go/src/github.com/influxdata/telegraf/build/telegraf_1.13.3-1_amd64.deb ./telegraf
make telegraf

run the plugin

make
docker run -it --rm --name=telegraf -v $PWD/telegraf.conf:/etc/telegraf/telegraf.conf:ro -v $PWD/telegraf-plugins.so:/usr/lib/telegraf-plugins/telegraf-plugins.so:ro telegraf:plugin-support --test

closed time in 2 months

mildebrandt

issue commentinfluxdata/telegraf

Unable to load external plugins -- plugin: not implemented

Thanks @danielnelson , enabling cgo did work. Here's the updated process that works:

mkdir ~/golinux
docker run -it -e CGO_ENABLED=1 -v ~/golinux:/go --rm golang:1.13.8 /bin/bash
apt-get update
apt-get install go-dep ruby-dev build-essential rpm -y
gem install --no-document fpm
go get github.com/influxdata/telegraf/
cd /go/src/github.com/influxdata/telegraf/
git checkout 1.13.3
./scripts/build.py --package --platform=linux --arch=amd64 --build-tags=goplugin
go get github.com/mildebrandt/telegraf-plugins/
cd /go/src/github.com/mildebrandt/telegraf-plugins/
make
exit

cd ~/golinux/src/github.com/mildebrandt/telegraf-plugins/
cp ~/golinux/src/github.com/influxdata/telegraf/build/telegraf_1.13.3-1_amd64.deb ./telegraf
make telegraf
docker run -it --rm --name=telegraf -v $PWD/telegraf.conf:/etc/telegraf/telegraf.conf:ro -v $PWD/telegraf-plugins.so:/usr/lib/telegraf-plugins/telegraf-plugins.so:ro telegraf:plugin-support --test

Which gives the following output:

2020-02-15T18:42:51Z I! Loading external plugins from: /usr/lib/telegraf-plugins/
2020-02-15T18:42:52Z I! Starting Telegraf 1.13.3
2020-02-15T18:42:52Z I! Using config file: /etc/telegraf/telegraf.conf
> example,host=844c7979cfd8,source=844c7979cfd8 value=123i 1581792173000000000

Thanks for your help!

mildebrandt

comment created time in 2 months

issue commentinfluxdata/telegraf

Unable to load external plugins -- plugin: not implemented

Sorry, it's probably not clear that I forked your repo and added a few things to make for easy reproduction. My repo is: https://github.com/mildebrandt/telegraf-plugins

mildebrandt

comment created time in 2 months

issue commentinfluxdata/telegraf

Unable to load external plugins -- plugin: not implemented

The make target, telegraf, uses a Dockerfile to install the .deb. The Dockerfile was taken from https://github.com/influxdata/influxdata-docker/tree/master/telegraf/1.13 and modified to use the local .deb copied from the output of the build telegraf step. I also modified the entrypoint.sh to specify the --plugin-directory:

#!/bin/bash
set -e

if [ "${1:0:1}" = '-' ]; then
    set -- telegraf --plugin-directory /usr/lib/telegraf-plugins/ "$@"
fi

exec "$@"

Hopefully that makes sense, and thanks for your help.

mildebrandt

comment created time in 2 months

issue commentinfluxdata/telegraf

Unable to load external plugins -- plugin: not implemented

Hi @danielnelson , I tried to use your sample plugin code to try out external plugin support and ran into issues. Is it something I'm doing wrong?

mildebrandt

comment created time in 2 months

issue openedinfluxdata/telegraf

Unable to load external plugins -- plugin: not implemented

When attempting to load an external plugin, I get the following error message:

2020-02-14T23:23:54Z I! Loading external plugins from: /usr/lib/telegraf-plugins/
2020-02-14T23:23:54Z E! error loading /usr/lib/telegraf-plugins/telegraf-plugins.so: plugin: not implemented

I built telegraf with the goplugin build tag and built a sample external plugin based on https://github.com/danielnelson/telegraf-plugins.

Here are the instructions to reproduce the error, also outlined at https://github.com/mildebrandt/telegraf-plugins/blob/master/README.md:

build telegraf

go get github.com/influxdata/telegraf/
cd ~/go/src/github.com/influxdata/telegraf/
git checkout 1.13.3
make
./scripts/build.py --package --platform=linux --arch=amd64 --build-tags=goplugin

package telegraf in a container

go get github.com/mildebrandt/telegraf-plugins/
cd ~/go/src/github.com/mildebrandt/telegraf-plugins/
cp ~/go/src/github.com/influxdata/telegraf/build/telegraf_1.13.3-1_amd64.deb ./telegraf
make telegraf

run the plugin

make
docker run -it --rm --name=telegraf -v $PWD/telegraf.conf:/etc/telegraf/telegraf.conf:ro -v $PWD/telegraf-plugins.so:/usr/lib/telegraf-plugins/telegraf-plugins.so:ro telegraf:plugin-support --test

created time in 2 months

push eventmildebrandt/telegraf-plugins

Chris Mildebrandt

commit sha bf9f808e8b81043d31ba44cf835cb5cbc3bd82b4

Demo files for plugin support

view details

push time in 2 months

issue commentdrone-plugins/drone-docker

DNS should default to local Docker host settings

That's great, thanks for being open to suggestions. :)

Just to be clear, the plugin itself can still be in a user-defined network. It's the build step inside the container that would be given the host network setting. Something like:

docker build --network host -t myrepo:1.1 .

So it may affect even less than you think, and could be a set in the plugin itself rather than on Drone.

jpds

comment created time in 2 months

issue commentdrone-plugins/drone-docker

DNS should default to local Docker host settings

Yeah, I understand. I didn't think about the dnsmasq issue earlier. I agree using the host machine's docker daemon could be a good solution too and will be following the linked issue.

I just tested something locally that seems to work. How feasible would it be to use host networking for the dind comtainer, e.g. --network host from the command line. Using that, I'm able to resolve all our internal DNS entries.

Thoughts?

jpds

comment created time in 2 months

issue openedinfluxdata/influxdb

go module for v2 needs to end in "v2".

When attempting to go get the influxdb v2 code, I get the following error:

$ go get github.com/influxdata/influxdb@v2.0.0-beta.3
go: finding github.com/influxdata/influxdb v2.0.0-beta.3
go: finding github.com v2.0.0-beta.3
go: finding github.com/influxdata v2.0.0-beta.3
go: finding github.com/influxdata/influxdb v2.0.0-beta.3
go get github.com/influxdata/influxdb@v2.0.0-beta.3: github.com/influxdata/influxdb@v2.0.0-beta.3: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v2

This is because the go.mod file is missing a v2 at the end of the module name. It should be:

module github.com/mildebrandt/influxdb/v2

See for additional information: https://golang.org/cmd/go/#hdr-Module_compatibility_and_semantic_versioning

created time in 2 months

created tagmildebrandt/influxdb

tagv2.0.0-beta.5

Scalable datastore for metrics, events, and real-time analytics

created time in 2 months

push eventmildebrandt/influxdb

Chris Mildebrandt

commit sha 65072f199c12b653e4090cbb8172f3304a53d4ae

update module

view details

push time in 2 months

created tagmildebrandt/influxdb

tagv2.0.0-beta.4

Scalable datastore for metrics, events, and real-time analytics

created time in 2 months

push eventmildebrandt/influxdb

Chris Mildebrandt

commit sha 783f6121f4b6c666c50bd876beb3100ab18d77e8

update module

view details

push time in 2 months

fork mildebrandt/influxdb

Scalable datastore for metrics, events, and real-time analytics

https://influxdata.com

fork in 2 months

issue commentdrone-plugins/drone-docker

DNS should default to local Docker host settings

The issue stems from the fact that drone uses a custom network which kicks-in docker's DNS service which isn't available to dind containers.

I also see that Drone treats certain containers as special already, including this one: https://github.com/drone/drone/blob/d84b79f027247284d26066041b1521414f68c631/operator/runner/runner.go#L255

So, how about creating another special category which passes the host's DNS settings into a plugin via --dns and related switches? That way the child containers will get the proper settings since the nameserver wouldn't be a local address.

jpds

comment created time in 2 months

issue commentdrone-plugins/drone-docker

DNS should default to local Docker host settings

While passing the issue upstream is the easy thing to do, I don't think it's the right thing in this case. People writing build scripts and using this plugin shouldn't need to be concerned about which DNS is being used. They expect each plugin to behave the same....if a network call works in one plugin, it should work in another.

The fact that this is performing a docker-in-docker is an implementation detail that the user should be isolated from knowing. The build engineer just wants to build a container.

If the solution is to set the PLUGIN_CUSTOM_DNS environment variable, that should be front and center in big bold letters as a prerequisite to using this plugin. If not, every new user that runs into this issue has to hop on Google and figure out why their container fails to build due to a server not being found.

Ideally, the plugin would pass its DNS settings into the child container every time.

jpds

comment created time in 2 months

issue commentinfluxdata/influxdata-docker

Create a docker container with plugin support

Thanks for looking into it. From the linked issue, I thought it was just adding the goplugin tag to the build command. I'll be attempting to build with that tag and writing external plugins...I'll be sure to post any issues I come across.

mildebrandt

comment created time in 2 months

push eventmildebrandt/telegraf

Daniel Nelson

commit sha adb156c185f893ac1abae7e61e882dd3b8262b25

Remove quantile example from aggregators docs Avoid confusion since we don't yet have an aggregator for quantiles.

view details

Daniel Nelson

commit sha 40ef7fe4d12a925258e2f1045a0b67d80653b380

Update changelog

view details

Daniel Nelson

commit sha c9a38875012ea96f4c9586736de5077ca570a28c

Update to github.com/safchain/ethtool@latest (#6975)

view details

Daniel Nelson

commit sha bc09c21f6ea836161dbc7a7eb25b082ae9bb5852

Update changelog

view details

Antonio Garcia

commit sha bbb8858c1dba7142369e995506b12ddc96d59d3d

Delete .README.md.swp (#6980)

view details

Daniel Nelson

commit sha 8792a5fd5ebe7f8e48d9f8cc1ce4a95a67a12314

Exclude processors and aggregators from minimal config example

view details

Daniel Nelson

commit sha e8d9add2d15a743d28cc48ccff976b02c76026ee

Expire metrics on query in addition to on add (#6981) Ensures that expired metrics are removed even when no new data is sent to the output.

view details

David Reimschussel

commit sha 4991fcb28134993360c4f59f841c21f6a6d3d627

Update changelog

view details

David Reimschussel

commit sha e51d9a35754576bdcb7d8eeb556e8c15a6675474

Set 1.13.3 release date

view details

Daniel Nelson

commit sha 0cc71dbd510319400934a82825d4a2a50c2697e0

Document the behavior of an unset routing key (#6983)

view details

Manthan Sharma

commit sha ae22db4b8117384ba1aef50f81ef863fe3b09f0e

Add support for titlecase transformation to strings processor (#6982)

view details

Daniel Nelson

commit sha 62ffd7172fa4e929309cbbb6300a049fd43e55eb

Update changelog

view details

RobMalvern

commit sha 15d016692287aaf6fb5a95e27406ad9610a3a21c

Add template processor (#6494)

view details

Daniel Nelson

commit sha 5f2f2ff33d5c6bc810f8b66dcbee515cc03a3759

Update readme and changelog for template processor

view details

Daniel Nelson

commit sha d46f94112cccdf69480c7124ccd2b980c6915035

Use require in cisco mdt tests to avoid follow on errors (#6984)

view details

Daniel Nelson

commit sha 0710cc74880919af6ae470d968fdc8ab0d84cd5a

Parse NaN values from summary types in prometheus input (#6997)

view details

Daniel Nelson

commit sha d3b89ec51ff322983655cd195a907897edbfc4c0

Update changelog

view details

Daniel Nelson

commit sha 5b8c71e61a0499edc00137bb6ed36998af957d40

Search for chronyc only when chrony input plugin is enabled (#7005)

view details

Daniel Nelson

commit sha c1eb0c8501ff619e718250b99a8863ceadda68e4

Update changelog

view details

Daniel Nelson

commit sha c681eb3524127e3c61422e91d96f4d70fd6981aa

Fix issue number in changelog

view details

push time in 2 months

pull request commentinfluxdata/telegraf

Require goplugin build flag to enable go plugin support

And I've opened this for the docker container: https://github.com/influxdata/influxdata-docker/issues/369

danielnelson

comment created time in 2 months

issue openedinfluxdata/influxdata-docker

Create a docker container with plugin support

Create a docker container of telegraf that has plugin support. Enabling plugin support is outlined here: https://github.com/influxdata/telegraf/pull/6393

The plugin support in telegraf lets us include plugins without needing to compile our own version of telegraf. However, since a version of telegraf with plugin support isn't built as part of the official release process, we need to compile our own version of telegraf.

What we'd like to do is, create a docker container based on an officially released telegraf container and package our plugins and conf files along side.

created time in 2 months

pull request commentinfluxdata/telegraf

Require goplugin build flag to enable go plugin support

Thanks, a package that enables plugin support would also be welcome. I've opened this to request it: https://github.com/influxdata/telegraf/issues/7003

danielnelson

comment created time in 2 months

issue openedinfluxdata/telegraf

Create a debian/rpm package with a telegraf version that has plugin support

Feature Request

Create an installable package of telegraf that has plugin support. Enabling plugin support is outlined here: https://github.com/influxdata/telegraf/pull/6393

Use case

The plugin support in telegraf lets us include plugins without needing to compile our own version of telegraf. However, since a version of telegraf with plugin support isn't built as part of the official release process, we need to compile our own version of telegraf.

What we'd like to do is, create a docker container with an officially released telegraf build and package our plugins and conf files along side.

created time in 2 months

pull request commentinfluxdata/telegraf

Require goplugin build flag to enable go plugin support

@danielnelson I'm working on a way to build our internal plugins and package them into the telegraf docker container. Correct me if I'm wrong, but it sounds like the docker container won't include a build of telegraf that supports plugins. I was hoping to build on top of the existing container, insert my plugins/conf files, and repackage. With this PR, it sounds like I need to separately build/host/support a telegraf binary/container if I want plugin support.

Is that correct? If so, any chance of Influx making a separate docker container with a telegraf binary that supports plugins as part of the release cycle?

danielnelson

comment created time in 2 months

pull request comment23andMe/Yamale

New schema example added to README.md

Thanks!

europ

comment created time in 2 months

push event23andMe/Yamale

Adrián Tóth

commit sha 618289c07424ca34892f367d356cb993af69c406

New schema example added to README.md (#85) Example describes if the data is a list of items without a key at the top level.

view details

push time in 2 months

PR merged 23andMe/Yamale

Reviewers
New schema example added to README.md

Example describes if the data is a list of items without a key at the top level.

+25 -0

0 comment

1 changed file

europ

pr closed time in 2 months

push event23andMe/Yamale

Chris Mildebrandt

commit sha fed3b73d6a407346b794438b796ffe89ecffd897

Catch more speicfic exception types

view details

push time in 2 months

pull request comment23andMe/Yamale

Bugfix/clean output

Perhaps I didn't explain myself clear enough, I wasn't suggesting to continue printing the stacktrace to the console.....the whole purpose of this PR is to stop doing that. To avoid misunderstand even more, here are the changes I would expect to solve the issue: https://github.com/23andMe/Yamale/compare/temp/better_output

Please take a look and let me know if you have thoughts on why that wouldn't work. Thanks!

abourree

comment created time in 2 months

create barnch23andMe/Yamale

branch : temp/better_output

created branch time in 2 months

Pull request review comment23andMe/Yamale

New schema example added to README.md

 person2:     age: 20 ``` +If the data is a list of items without a key at the top level:

Thanks for helping our docs. I think this may be better in the examples section.

europ

comment created time in 2 months

push event23andMe/Yamale

Adrián Tóth

commit sha 75ff8a944730f466dd2eaa69cc0acce0d6afd94c

Fixed example for 'Custom validators' (#86) * missing comma fixed in README.md example * missing module fixed in README.md example

view details

push time in 2 months

PR merged 23andMe/Yamale

Fixed example for 'Custom validators'
  • added missing comma
  • added missing module - datetime
+2 -1

1 comment

1 changed file

europ

pr closed time in 2 months

pull request comment23andMe/Yamale

Fixed example for 'Custom validators'

Thanks for finding that!

europ

comment created time in 2 months

pull request comment23andMe/Yamale

Bugfix/clean output

Thank you for your PR. I have some concerns that it does a little too much wrapping of the errors. I think the only file that needs to change (other than the tests of course) is the command_line.py. Please let me know if there's something I'm not taking into consideration.

abourree

comment created time in 2 months

Pull request review comment23andMe/Yamale

Bugfix/clean output

 def main():     parser.add_argument('--strict', action='store_true',                         help='Enable strict mode, unexpected elements in the data will not be accepted.')     args = parser.parse_args()-    _router(args.path, args.schema, args.cpu_num, args.parser, args.strict)-    print('Validation success! 👍')+    if _router(args.path, args.schema, args.cpu_num, args.parser, args.strict) == 0:

Here, you could catch the exceptions and print out the result....then you don't need the 0/1 status.

abourree

comment created time in 2 months

Pull request review comment23andMe/Yamale

Bugfix/clean output

 def _validate_dir(root, schema_name, cpus, parser, strict):      print('Found %s yaml files.' % len(res))     print('Validating...')+    results = []

Instead of collating and returning results, you could raise an exception with all the errors. Then the _router method doesn't have to change and the exception would get passed directly to main where it can be handled.

abourree

comment created time in 2 months

Pull request review comment23andMe/Yamale

Bugfix/clean output

  schemas = {} +status = 0  def _validate(schema_path, data_path, parser, strict):     schema = schemas.get(schema_path)-    try:-        if not schema:-            schema = yamale.make_schema(schema_path, parser)-            schemas[schema_path] = schema-        data = yamale.make_data(data_path, parser)-        yamale.validate(schema, data, strict)-    except Exception as e:-        error = '\nError!\n'-        error += 'Schema: %s\n' % schema_path-        error += 'Data file: %s\n' % data_path-        error += traceback.format_exc()-        print(error)-        raise ValueError('Validation failed!')+    if not schema:+        schema = yamale.make_schema(schema_path, parser)+        schemas[schema_path] = schema+    data = yamale.make_data(data_path, parser)+    return yamale.validate(schema, data, strict)

This is a good change, except for the return....can just let that call raise an exception.

abourree

comment created time in 2 months

Pull request review comment23andMe/Yamale

Bugfix/clean output

+import sys++PY2 = sys.version_info[0] == 2++class ValidationResult:++    def __init__(self, data, schema, errors):+        self.data = data+        self.schema = schema+        self.errors = errors++    def __str__(self):+        head_line = "Error validating data '%s' with '%s'\n\t" % (self.data, self.schema)+        error_str = head_line + '\n\t'.join(self.errors)+        if PY2:+            error_str = error_str.encode('utf-8')+        return error_str++    def isValid(self):+        return len(self.errors) == 0++    def hasError(self, expected):

This doesn't seem to be used anywhere.

abourree

comment created time in 2 months

Pull request review comment23andMe/Yamale

Bugfix/clean output

 def _parse_schema_item(self, path, expression, validators):     def validate(self, data, data_name, strict):         path = DataPath()         errors = self._validate(self._schema, data, path, strict)-

It seems there is either a validation error or there isn't. Can you explain why we need a result object rather than just raising an exception here?

abourree

comment created time in 2 months

Pull request review comment23andMe/Yamale

Bugfix/clean output

  schemas = {} +status = 0

This doesn't seem to be used.

abourree

comment created time in 2 months

push eventmildebrandt/telegraf

Chris Mildebrandt

commit sha 68b387061d0d34e939dae0fea3ce07bcda92a906

Fix fmt issues

view details

push time in 2 months

PR opened influxdata/telegraf

Support obtaining all builds within the max_build_age

This fixes #6966

This changes the job query to include a depth parameter so the build information for the job is obtained in the same result as the job details. I also limit the size of the body to only the fields we're interested in using the tree parameter.

Before the change, we would get metrics of 20 nodes and 100 builds in 23.56 seconds on average. After this change, we get metrics of 20 nodes and 2006 builds in 7.76 seconds on average.

We were able to get a 3x decrease in time while increasing the number of metrics by over 16x.

Required for all PRs:

  • [x] Signed CLA.
  • [NA] Associated README.md updated.
  • [x] Has appropriate unit tests.
+130 -149

0 comment

3 changed files

pr created time in 2 months

create barnchmildebrandt/telegraf

branch : bugfix/6966

created branch time in 2 months

fork mildebrandt/telegraf

The plugin-driven server agent for collecting & reporting metrics.

fork in 2 months

issue openedinfluxdata/telegraf

Jenkins plugin doesn't collect all builds for a job

The plugin only collects the metrics of builds that have finished. That's fine, but it also only collects the metrics of the last build to run within a job. So if there is a new build that starts before the previous has finished, the metrics of the previous build will not be collected.

System info:

telegraf 1.13.2 Docker image

Steps to reproduce:

  1. Configure Jenkins plugin
  2. Run two concurrent builds in Jenkins

Expected behavior:

Both builds will be recorded. Even better, I want all builds within the max_build_age setting of the conf file to be recorded.

Actual behavior:

Only the last build to finish will be recorded.

created time in 2 months

Pull request review comment23andMe/Yamale

Simplify manual installation instructions

 $ pip install yamale ```  ### Manual-1. Download Yamale from: https://github.com/23andMe/Yamale/archive/master.zip-2. Unzip somewhere temporary-3. Run `python setup.py install` (may have to prepend `sudo`)+Run `pip install https://github.com/23andMe/Yamale/archive/master.zip` (may have to prepend `sudo`)

Thanks for your interest in Yamale and your PR. In this case, manual installation refers to installing without pip.

If you'd like to add this instruction under the pip section as a way to install the latest master branch, it would probably fit better there. Thanks.

basnijholt

comment created time in 2 months

fork mildebrandt/twitter-block-chain

Chrome extension to block all users on a followers/following page.

fork in 2 months

pull request comment23andMe/Yamale

Bugfix/clean output

I'm getting mixed signals here. :)

$ cat 54.schema 
log_file: str(required=True)
$ cat 54.yaml 
$ yamale -s 54.schema 54.yaml
Validating /Users/chrism/workspace/abourree/Yamale/54.yaml...
Error validating data '/Users/chrism/workspace/abourree/Yamale/54.yaml' with '54.schema'
        log_file: Required field missing
Validation success! �
abourree

comment created time in 2 months

issue comment23andMe/Yamale

Cleaner command line error output

I agree, thanks!

televi

comment created time in 3 months

PR closed 23andMe/Yamale

New feature: includeIf(<path>, <include to test on path>, <then include> [,<else include>)

I propose new keyword includeIf for conditional sub-schema. The goal is to validate with the following schema:

a: bool()
b: includeIf('a', 'isTrue', 'bDefined')
---
isTrue: enum('True')
---
bDefined:
  c: int()

This data files where 'b' is not optional but depends of 'a':

a: False

or

a: True
b:
  c: 1

Because 'b' could have two formats depends of 'a', the following schema include also else sub-schema:

a: bool()
b: includeIf('a', 'isTrue', 'bIfTrue', 'bIfFalse')
---
isTrue: enum('True')
---
bifTrue:
  c: int()
---
bIfFalse: enum('Disabled')

validates:

a: False
b: Disabled

or

a: True
b:
  c: 1

This new feature add dependency on dpath-python module.

+276 -12

3 comments

14 changed files

abourree

pr closed time in 3 months

pull request comment23andMe/Yamale

New feature: includeIf(<path>, <include to test on path>, <then include> [,<else include>)

Thanks for the PR and the detailed README updates. Unfortunately, we don't feel this is a good fit for Yamale right now. We'd like to avoid conditional logic within the schema. Please feel free to fork this project if this feature is something you really need for your work. Let me know if you have any questions. Thanks!

abourree

comment created time in 3 months

push event23andMe/Yamale

Arnaud Bourrée

commit sha 8d8e4b809c9e313795ac5c3721697ac853bfada5

resolves #54: empty data should failed if schema requires something (#81) * resolves #54 by replacing [] in make_data by [(None, path)] * Correct fix with empty map and not None * replace assert count_exception_lines by match_exception_lines in test_bad_nested_issue_54()

view details

push time in 3 months

PR merged 23andMe/Yamale

resolves #54: empty data should failed if schema requires something

resolves #54 by replacing [] in make_data by [(None, path)]

+41 -2

5 comments

5 changed files

abourree

pr closed time in 3 months

issue closed23andMe/Yamale

empty yml to validate not detected

hello, assume the simple block of code below (may be not a bug)

schema = yamale.make_schema('schema.yml')
cfg = yamale.make_data('config.yml')

try:
    yamale.validate(schema, cfg)
except ValueError as e:
    log.critical(e)
    sys.exit(1)

where: schema.yml is the yaml schema used to validate a config.yml file

Now let's assume in the schema.yml there a line like: log_file: str(required=True)

Now if I provide an empty config.yml file, the validation succeed with no exception. So in this way any config.yml is considered valid.

I think would make sense to add a simple check to a non-empty yml file to validate.

if you think this make sense, let me know if you will accept a pull request for it. I can help to fix it.

The tool is great and I enjoy it

thnx Antonio

closed time in 3 months

mancusoa74

pull request comment23andMe/Yamale

resolves #54: empty data should failed if schema requires something

Awesome, thanks!

abourree

comment created time in 3 months

Pull request review comment23andMe/Yamale

resolves #54: empty data should failed if schema requires something

 def test_bad_nested():     assert count_exception_lines(nested['schema'], nested['bad']) == 4  +def test_bad_nested_issue_54():

I'd rather not add more things that I'll have to change later. Can you change this to match the format of the tests below that use match_exception_lines()? You can create a separate schema file instead of reusing nested.yaml if you want to limit the number of errors in your output. Thanks!

abourree

comment created time in 3 months

push event23andMe/Yamale

Arnaud Bourrée

commit sha 750a8e80a4840072fea16c0592793d742c2e33cc

replace try:/expect: by 'with pytest.raise(ValueError):' (#80) * replace try:/expect: by 'with pytest.raise(ValueError):' * fix test_empty_schema_file * restore test_bad_dir test execution * add non empty data.yaml file

view details

push time in 3 months

PR merged 23andMe/Yamale

replace try:/expect: by 'with pytest.raise(ValueError):'

On my Linux box, running test_command_line.py never end. I search in Google and found nothing. I disable all test inside and add them one by one, only tests with try/expect: stuck pytest

I Google how to expect exception with pytest: asserting exceptions with pytest

I replace all try:/expect: with pytest.raise() and then:

  • excepted test_bad_dir witch I mark skip, none get stuck
  • test_empty_schema_file now failed because it didn't raise ValueError
+16 -31

7 comments

2 changed files

abourree

pr closed time in 3 months

pull request comment23andMe/Yamale

replace try:/expect: by 'with pytest.raise(ValueError):'

Can you try running in an virtual environment:

python3 -m venv ./venv
source ./venv/bin/activate
pip install -e .
make test
abourree

comment created time in 3 months

pull request comment23andMe/Yamale

replace try:/expect: by 'with pytest.raise(ValueError):'

On my Linux box, running test_command_line.py never end.

I'm curious why the tests never completed. How do you run the tests? Can you run make test to see if that works with the original code?

I agree with this PR, I would just like to know why it was hanging. Thanks!

abourree

comment created time in 3 months

Pull request review comment23andMe/Yamale

replace try:/expect: by 'with pytest.raise(ValueError):'

 def test_external_glob_schema(parser):         os.path.join(dir_path, 'command_line_fixtures/schema_dir/ex*.yaml'), 1, parser)  -def test_empty_schema_file():-    try:+def test_empty_schema_file(capsys):+    with pytest.raises(ValueError, match='Validation failed!'):         command_line._router(-            'yamale/tests/command_line_fixtures/empty_schema',+            'yamale/tests/command_line_fixtures/empty_schema/data.yaml',

Nice find! This test was missing the final assert False. Please add the data.yaml file to git. Thanks!

abourree

comment created time in 3 months

pull request comment23andMe/Yamale

resolves #54: empty data should failed if schema requires something

Sorry, I had the wrong branch. I see your validation error is the following:

Error validating data /Users/chrism/workspace/abourree/Yamale/54.yaml with schema schema.yaml
         : 'None' is not a map

I would expect this instead:

Error validating data /Users/chrism/workspace/abourree/Yamale/54.yaml with schema schema.yaml
        log_file: Required field missing
abourree

comment created time in 3 months

pull request comment23andMe/Yamale

resolves #54: empty data should failed if schema requires something

I cloned your repo and couldn't validate your fix. For example:

(venv) ~/w/a/Yamale (master|…) $ cat schema.yaml 
log_file: str(required=True)
(venv) ~/w/a/Yamale (master|…) $ cat 54.yaml 
(venv) ~/w/a/Yamale (master|…) $ yamale -s schema.yaml 54.yaml
Validating /Users/chrism/workspace/abourree/Yamale/54.yaml...
Validation success!
abourree

comment created time in 3 months

Pull request review comment23andMe/Yamale

resolves #54: empty data should failed if schema requires something

 def test_bad_nested():     assert count_exception_lines(nested['schema'], nested['bad']) == 4  +def test_bad_nested_issue_54():

I'm not a fan of this format of checking the number of lines in the exception message and would like to change the existing tests. For example, you're missing the nested_issue_54.yaml file, but the test passes. This means that the "file not found" exception message has the same number of lines as what you're raising.

abourree

comment created time in 3 months

Pull request review comment23andMe/Yamale

resolves #54: empty data should failed if schema requires something

     'bad': 'static_list_bad.yaml' } +nested_issue_54 = {

I don't see nested_issue_54.yaml, can you commit that too?

abourree

comment created time in 3 months

pull request comment23andMe/Yamale

resolves #54: empty data should failed if schema requires something

Thanks for your PR! I'll take a look.

abourree

comment created time in 3 months

PR closed 23andMe/Yamale

minor improvement in Regex.__init__
+2 -3

2 comments

1 changed file

SergBobrovsky

pr closed time in 3 months

pull request comment23andMe/Yamale

minor improvement in Regex.__init__

This PR doesn't add any obvious improvements.

SergBobrovsky

comment created time in 3 months

issue commentazavea/django-amazon-ses

Region should not default to us-east-1

Hi @hectcastro!

So, what I did was stand-up an ECS container in us-west-2 without any settings pointing to a specific region, and it attempted to send e-mail through us-east-1. If SES is not available in the region my container is in, then I would expect the API call to fail. Making the default us-east-1 will fail for anyone using other regions since their e-mail addresses aren't verified in us-east-1.

This issue is about changing the default from us-east-1 to the region of the instance. I think that would be a more logical default.

mildebrandt

comment created time in 4 months

issue openedazavea/django-amazon-ses

Region should not default to us-east-1

When running in an ECS container, I would expect the package to inherit the AWS settings from the container instead of defaulting to us-east-1. I would rather not specify the region since the container knows where it is.

created time in 4 months

pull request comment23andMe/Yamale

Custom Validator with Include and Schema Validation

Hi, thanks for your interest in Yamle. Instead of validating keys within a custom validator, you can use the --strict option to deny elements that aren't defined in the schema. Using your example files, it would result in this:

$ yamale -s pr78.schema pr78.fail --strict
Validating /Users/chrism/workspace/Yamale/pr78.fail...

Error!
Schema: pr78.schema
Data file: /Users/chrism/workspace/Yamale/pr78.fail
Traceback (most recent call last):
  File "/Users/chrism/workspace/Yamale/yamale/command_line.py", line 29, in _validate
    yamale.validate(schema, data, strict)
  File "/Users/chrism/workspace/Yamale/yamale/yamale.py", line 40, in validate
    schema.validate(d, path, strict)
  File "/Users/chrism/workspace/Yamale/yamale/schema/schema.py", line 65, in validate
    raise ValueError(error_str)
ValueError: 
Error validating data /Users/chrism/workspace/Yamale/pr78.fail with schema pr78.schema
        test_includes.my_custom_env: Unexpected element

Would that work for you?

nascimento

comment created time in 4 months

push event23andMe/Yamale

dpasichnyi

commit sha ffa9b8b97159e340a993d91f02a9698b465d0c63

Fix Deprecation Warning 'invalid escape sequence \*' by using raw string for regular expressions (#76)

view details

push time in 4 months

PR merged 23andMe/Yamale

Fix Deprecation Warning 'invalid escape sequence \*'

There were 3 deprecation warnings while running tests against Py36:

yamale/validators/tests/test_validate.py:40
  /Users/dpasichnyi/ttam/Yamale/yamale/validators/tests/test_validate.py:40: DeprecationWarning: invalid escape sequence \s
    v = val.Regex('[a-z0-9]{3,}s\s$', ignore_case=True)

yamale/validators/tests/test_validate.py:47
  /Users/dpasichnyi/ttam/Yamale/yamale/validators/tests/test_validate.py:47: DeprecationWarning: invalid escape sequence \d
    v = val.Regex('A.+\d$', ignore_case=False, multiline=True)

yamale/validators/tests/test_validate.py:51
  /Users/dpasichnyi/ttam/Yamale/yamale/validators/tests/test_validate.py:51: DeprecationWarning: invalid escape sequence \.
    v = val.Regex('.*^Ye.*s\.', ignore_case=True, multiline=True, dotall=True)

Fixed them by using raw string for regular expressions.

+3 -3

0 comment

1 changed file

dpasichnyi

pr closed time in 4 months

push event23andMe/Yamale

dpasichnyi

commit sha 2ae33b409f075629cbefd6bc92d0094d56e0f8e3

Remove duplicated word 'than' from LengthMin class (#75)

view details

push time in 4 months

PR merged 23andMe/Yamale

Remove duplicated word 'than' from LengthMin class
  • Remove the duplicated word than from LengthMin class
  • Add .python-version into .gitignore
+4 -1

0 comment

2 changed files

dpasichnyi

pr closed time in 4 months

issue comment23andMe/Yamale

Add support for 'rec' core type

Thanks for your interest in Yamale. I don't know exactly what you mean by the "rec" type. The only reference I could find is from the Rx tool: http://rx.codesimply.com/coretypes.html

If that's what you're referring to, Yamale already supports this in the schema itself using the include validator: https://github.com/23andMe/Yamale#includes

If you want something else, please post an example of the yaml and the suggested schema and we'll see if there's a solution.

slw07g

comment created time in 4 months

Pull request review comment23andMe/Yamale

minor improvement in Regex.__init__

 class Regex(Validator):     def __init__(self, *args, **kwargs):         self.regex_name = kwargs.pop('name', None) -        flags = 0-        for k, v in util.get_iter(self._regex_flags):-            flags |= v if kwargs.pop(k, False) else 0+        flags = sum(v for k, v in util.get_iter(self._regex_flags)

Thanks for watching @drmull :) In this case, it should be fine since they're defined as ints anyway, and the flags won't overlap:

SRE_FLAG_TEMPLATE = 1 # template mode (disable backtracking)
SRE_FLAG_IGNORECASE = 2 # case insensitive
SRE_FLAG_LOCALE = 4 # honour system locale
SRE_FLAG_MULTILINE = 8 # treat target as multiline string
SRE_FLAG_DOTALL = 16 # treat target as a single string
SRE_FLAG_UNICODE = 32 # use unicode "locale"
SRE_FLAG_VERBOSE = 64 # ignore whitespace and comments
SRE_FLAG_DEBUG = 128 # debugging
SRE_FLAG_ASCII = 256 # use ascii "locale"

But I'm not a fan of change for the sake of change, so I was curious if this solved a problem I wasn't aware of before.

SergBobrovsky

comment created time in 5 months

pull request comment23andMe/Yamale

minor improvement in Regex.__init__

Hi, thanks for your interest in Yamale. Can you describe what improvement this makes? If it's a performance improvement, can you provide some numbers to go along with this? Thanks!

SergBobrovsky

comment created time in 5 months

PR closed 23andMe/Yamale

minor improvement in Regex.__init__
+1 -3

0 comment

1 changed file

SergBobrovsky

pr closed time in 5 months

pull request commentevanlucas/fish-kubectl-completions

Add context support

Thanks, sorry I didn’t get that last piece you asked for. I switched jobs around that time and was getting up to speed there. I’m also not using kubernetes in my new role, and I forgot about this PR. Hopefully it’s useful for others.

mildebrandt

comment created time in 5 months

pull request comment23andMe/Yamale

Add a check for schema.yml

Thank you for your contribution!

tapaswenipathak

comment created time in 5 months

push event23andMe/Yamale

Tapasweni Pathak

commit sha 7a705e52fcc10d4b2da0ff1c89dc18bbcd86cd3b

Add a check for schema.yml (#70) * Add a check for schema.yml * Travis fixes * Add test and fix error clause * test fixes * add api implementation remove command_line implementation * test fixes * add removed line * review enhancements * test fixes * Remove try catch * Fix value error

view details

push time in 5 months

PR merged 23andMe/Yamale

Add a check for schema.yml

Fixes https://github.com/23andMe/Yamale/issues/55.

Hi! I like the open source projects of the org, just understanding the culture of the open source contributors in the org by writing a PR. Hope the pr helps. Few more interesting projects have no tickets. :)

+18 -1

5 comments

5 changed files

tapaswenipathak

pr closed time in 5 months

issue closed23andMe/Yamale

empty schema generated IndexError exception

hello, assume the simple block of code below:

schema = yamale.make_schema('schema.yml')

where: schema.yml is an empty file

this generates the following exception

File "....../lib/python3.7/site-packages/yamale/yamale.py", line 17, in make_schema s = Schema(raw_schemas[0], path, validators=validators) IndexError: list index out of range

I think would make sense to add a simple check to a non-empty yml schema.

if you think this make sense, let me know if you will accept a pull request for it. I can help to fix it.

The tool is great and I enjoy it

thnx Antonio

closed time in 5 months

mancusoa74

issue comment23andMe/Yamale

[Feature request] "in" validator

Hi, thanks for using Yamale!

I'm usually not a fan of schema contained within the file being validated. One, it requires multiple passes through each structure, the schema and the target files, which complicates the code and requires special handling just for that validator. Two, more opinion based, it's better to keep validation values within the schema. If there's a different structure that's valid, then it should require a different schema.

I would suggest you use the enum validator for this case.

LoveIsGrief

comment created time in 5 months

issue comment23andMe/Yamale

schema file in current directory overrules schemas in deeper directories

Thanks for using Yamale and raising this issue. I'll try to find time to address this. If you think you have a good solution, we do welcome PRs.

dvandok

comment created time in 5 months

issue comment23andMe/Yamale

Cleaner command line error output

Hi, thanks for using Yamale. I think those proposed changes are fine. I'm a little concerned why ValueError exceptions are being passed, can you explain that? In any case, we'd love a PR for this. Thanks!

televi

comment created time in 5 months

Pull request review comment23andMe/Yamale

Add a check for schema.yml

 #!/usr/bin/env python import sys+import os from .schema import Schema  PY2 = sys.version_info[0] == 2   def make_schema(path, parser='PyYAML', validators=None):     # validators = None means use default.+    # test if the schema.yaml file is empty

I've been thinking a little about this. How about checking raw_schemas instead of the size of the file. For example:

    if not raw_schemas:
        raise ValueError('{} is an empty file!'.format(path))

Or you can even place it within the current try block for something like this:

    try:
        if not raw_schemas:
            raise ValueError('No schema documents found!')
        s = Schema(raw_schemas[0], path, validators=validators)
        # Additional documents contain Includes.
        for raw_schema in raw_schemas[1:]:
            s.add_include(raw_schema)
    except (TypeError, SyntaxError, ValueError) as e:
        error = 'Schema error in file %s\n' % path
        error += str(e)
        if PY2:
            error.encode('utf-8')
        raise SyntaxError(error)
tapaswenipathak

comment created time in 5 months

Pull request review comment23andMe/Yamale

Add a check for schema.yml

 #!/usr/bin/env python import sys+import os from .schema import Schema  PY2 = sys.version_info[0] == 2   def make_schema(path, parser='PyYAML', validators=None):     # validators = None means use default.+    # test if the schema.yaml file is empty+    try:

Hi! Sorry for the delay, I was on another vacation. :) It would be better to just remove the try block so it's just the following:

if not os.path.getsize(path):
    error_message = os.path.basename(path) + ' is an empty file!'
    raise ValueError(error_message)

Let me know if you have any questions about that.

tapaswenipathak

comment created time in 5 months

Pull request review comment23andMe/Yamale

Add a check for schema.yml

 #!/usr/bin/env python import sys+import os from .schema import Schema  PY2 = sys.version_info[0] == 2   def make_schema(path, parser='PyYAML', validators=None):     # validators = None means use default.+    # test if the schema.yaml file is empty+    try:

Hi, I'm back. Have you thought about a better way to write the following code?

    try:
        if not os.path.getsize(path):
            error_message = os.path.basename(path) + ' is an empty file!'
            raise ValueError(error_message)
    except ValueError as e:
        raise ValueError(e)
tapaswenipathak

comment created time in 6 months

more