profile
viewpoint
Srdjan Grubor sgnn7 DataDog St. Louis, Missouri, US http://www.sgnn7.org Software Engineer - Agent Core @DataDog. OSS enthusiast. Maintainer of many projects.

infamousjoeg/pyaim 14

@CyberArk Application Access Manager Client Library for Python 3

sgnn7/deploying_with_docker 13

Code samples from my Deployment with Docker book

sgnn7/Android2Linux 2

Android app that can be used to install Linux and run it in parallel with Android

sgnn7/android-screengrab 1

Basic utility to continuously grab screenshots of a debug-enabled Android device. Requires just the adb executable in project directory.

sgnn7/BuildPolice 1

Arduino board-based code that activates the selected digital pin/s given Hudson/Jenkins CI build status

sgnn7/copy_assistant 1

Attempt at getting a replacement copy hook for windows (possibly even *nix), trying to make it more robust and better for large file copy operations

sgnn7/crtool 1

Helper tooling for certificate management

PullRequestReviewEvent

push eventDataDog/datadog-agent

Srdjan Grubor

commit sha 56ba5f1bcfe13b908f70d77865b926f12028d0a8

[tools] Add copyright lint tooling for Golang files (#10083) * [tools] Add copyright linting tool for Golang files This addition can be used via `inv lint-copyright` which traverses the repo tree and fails if it finds any Golang files with an unexpected copyright header. For now the tool is intended to be invoked manually but once all the files get fixed, we can add it to our internal CI/CD pipeline as an automated check.

view details

push time in 3 days

delete branch DataDog/datadog-agent

delete branch : sgnn7/add_copyright_checker

delete time in 3 days

PR merged DataDog/datadog-agent

[tools] Add copyright lint tooling for Golang files kind/enhancement dev/tooling changelog/no-changelog severity/minor team/agent-platform qa/skip-qa python category/improvement

What does this PR do?

This tool addition can be used via inv lint-copyrights which traverses the repo tree and fails if it finds any Golang files with an unexpected copyright header.

For now the tool is intended to be invoked manually but once all the files get fixed, we can add it to our internal CI/CD pipeline as an automated check.

Motivation

Legal compliance needs

Additional Notes

  • Eventually we would like to enable this check in CI/CD but we need all the failures cleaned up first.

Possible Drawbacks / Trade-offs

The tool could ideally have the ability to auto-fix problems but that's outside of the scope of this task right now.

Describe how to test/QA your changes

N/A but if test is needed, running inv lint-copyrights should show the output.

Reviewer's Checklist

  • [x] If known, an appropriate milestone has been selected; otherwise the Triage milestone is set.
  • [x] The appropriate team/.. label has been applied, if known.
  • [x] A release note has been added or the changelog/no-changelog label has been applied.
  • [ ] Changed code has automated tests for its functionality.
  • [ ] Adequate QA/testing plan information is provided if the qa/skip-qa label is not applied.
  • [ ] If applicable, docs team has been notified or an issue has been opened on the documentation repo.
  • [ ] If applicable, the need-change/operator and need-change/helm labels have been applied.
  • [ ] If applicable, the config template has been updated.
+198 -0

0 comment

3 changed files

sgnn7

pr closed time in 3 days

Pull request review commentDataDog/datadog-agent

[tools] Add copyright lint tooling for Golang files

+#!/usr/bin/env python3+# -*- coding: utf-8 -*-++import os+import re+import subprocess+import sys+from pathlib import Path, PurePosixPath++GLOB_PATTERN = "**/*.go"++COPYRIGHT_REGEX = [+    r'^// Unless explicitly stated otherwise all files in this repository are licensed$',+    r'^// under the Apache License Version 2.0\.$',+    r'^// This product includes software developed at Datadog \(https://www\.[Dd]atadoghq\.com/\)\.$',+    r'^// Copyright 20[1-3][0-9]-([Pp]resent|20[1-3][0-9]) Datadog, (Inc|Inmetrics)\.$',+]++# These path patterns are excluded from checks+PATH_EXCLUSION_REGEX = [+    '/third_party/golang/',+    '/third_party/kubernetes/',+]++# These header matchers skip enforcement of the rules if found in the first+# line of the file+HEADER_EXCLUSION_REGEX = [+    '^// Code generated ',+    '^//go:generate ',+    '^// Copyright.* OpenTelemetry Authors',+    '^// Copyright.* The Go Authors',+]+++COMPILED_COPYRIGHT_REGEX = [re.compile(regex, re.UNICODE) for regex in COPYRIGHT_REGEX]+COMPILED_PATH_EXCLUSION_REGEX = [re.compile(regex, re.UNICODE) for regex in PATH_EXCLUSION_REGEX]+COMPILED_HEADER_EXCLUSION_REGEX = [re.compile(regex, re.UNICODE) for regex in HEADER_EXCLUSION_REGEX]+++class CopyrightLinter:+    """+    This class is used to enforce copyright headers on specified file patterns+    """++    @staticmethod+    def _get_repo_dir():+        script_dir = PurePosixPath(__file__).parent+        current_dir = Path.cwd()++        os.chdir(script_dir)+        repo_dir = subprocess.check_output(['git', 'rev-parse', '--show-toplevel']).decode(sys.stdout.encoding).strip()+        os.chdir(current_dir)++        return PurePosixPath(repo_dir)++    @staticmethod+    def _get_matching_files(root_dir, glob_pattern, exclude=None):+        if exclude is None:+            exclude = []++        all_matching_files = Path(root_dir).glob(glob_pattern)++        # `all_matching_files` is a generator so we have to do the counting ourselves+        all_matching_files_cnt = 0++        filtered_files = []

@mx-psi I made the inside loop be predicate-based but due to the generator being the iterable over which we evaluate this block and the fact that we need the total count for debugging, I left the outer loop as-is. I know the generator can be turned into a list easy for the length calculation but I think at that point we would be skyrocketing in mem usage so I opted against it.

sgnn7

comment created time in 3 days

PullRequestReviewEvent

push eventDataDog/datadog-agent

Srdjan Grubor

commit sha eee0dca3d77545261f20d7c9cb88c9f2d583bcb7

[tools] Fix minor nits from code review There were some inefficiencies and nits in the original commit that needed fixes so this change fixes them.

view details

Srdjan Grubor

commit sha d64ee9e62d1952e323282a809b954e216bdbb161

[tools] Add loglevel prefix to most copyright linter printouts This should make it visually easier to parse the output of this tooling.

view details

push time in 3 days

Pull request review commentDataDog/datadog-agent

[tools] Add copyright lint tooling for Golang files

 def lint_python(ctx):     ctx.run("vulture --ignore-decorators @task --ignore-names 'test_*,Test*' tasks")  +@task+def lint_copyrights(_):+    """+    Checks that all files matching the search pattern (default: '**/*.go')+    contain the appropriate copyright header.

It's the other way around. For now this only checks Golang files but we may expand it to do the others types like Python and C. I'll apply your suggestion for now though.

sgnn7

comment created time in 3 days

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentDataDog/datadog-agent

[tools] Add copyright lint tooling for Golang files

+#!/usr/bin/env python3+# -*- coding: utf-8 -*-++import os+import re+import subprocess+import sys+from pathlib import Path, PurePosixPath++GLOB_PATTERN = "**/*.go"++COPYRIGHT_REGEX = [+    r'^// Unless explicitly stated otherwise all files in this repository are licensed$',+    r'^// under the Apache License Version 2.0\.$',+    r'^// This product includes software developed at Datadog \(https://www\.[Dd]atadoghq\.com/\)\.$',+    r'^// Copyright 20[1-3][0-9]-([Pp]resent|20[1-3][0-9]) Datadog, (Inc|Inmetrics)\.$',+]++# These path patterns are excluded from checks+PATH_EXCLUSION_REGEX = [+    '/third_party/golang/',+    '/third_party/kubernetes/',+]++# These header matchers skip enforcement of the rules if found in the first+# line of the file+HEADER_EXCLUSION_REGEX = [+    '^// Code generated ',+    '^//go:generate ',+    '^// Copyright.* OpenTelemetry Authors',+    '^// Copyright.* The Go Authors',+]+++COMPILED_COPYRIGHT_REGEX = [re.compile(regex, re.UNICODE) for regex in COPYRIGHT_REGEX]+COMPILED_PATH_EXCLUSION_REGEX = [re.compile(regex, re.UNICODE) for regex in PATH_EXCLUSION_REGEX]+COMPILED_HEADER_EXCLUSION_REGEX = [re.compile(regex, re.UNICODE) for regex in HEADER_EXCLUSION_REGEX]+++class CopyrightLinter:+    """+    This class is used to enforce copyright headers on specified file patterns+    """++    @staticmethod+    def _get_repo_dir():+        script_dir = PurePosixPath(__file__).parent+        current_dir = Path.cwd()++        os.chdir(script_dir)+        repo_dir = subprocess.check_output(['git', 'rev-parse', '--show-toplevel']).decode(sys.stdout.encoding).strip()+        os.chdir(current_dir)++        return PurePosixPath(repo_dir)++    @staticmethod+    def _get_matching_files(root_dir, glob_pattern, exclude=None):+        if exclude is None:+            exclude = []++        all_matching_files = Path(root_dir).glob(glob_pattern)++        # `all_matching_files` is a generator so we have to do the counting ourselves+        all_matching_files_cnt = 0++        filtered_files = []+        for filepath in all_matching_files:+            all_matching_files_cnt += 1++            excluded = False+            for matcher in exclude:+                if re.search(matcher, filepath.as_posix()):+                    excluded = True+                    break++            if excluded:+                continue++            filtered_files.append(filepath)++        excluded_files_cnt = all_matching_files_cnt - len(filtered_files)+        print(f"[WARN] Excluding {excluded_files_cnt} files based on path filters!")++        return sorted(filtered_files)++    @staticmethod+    def _get_header(filepath):+        header = []+        with open(filepath, "r") as file_obj:+            # We expect a specific header format which should be 4 lines+            for _ in range(4):+                header.append(file_obj.readline().strip())++        return header++    @staticmethod+    def _is_excluded_header(header, exclude=None):+        if exclude is None:+            exclude = []++        for matcher in exclude:+            if re.search(matcher, header[0]):+                return True++        return False++    @staticmethod+    def _has_copyright(filepath, debug=False):+        header = CopyrightLinter._get_header(filepath)+        if header is None:+            print("Mismatch found! Could not find any content in file!")+            return False++        if len(header) > 0 and CopyrightLinter._is_excluded_header(header, exclude=COMPILED_HEADER_EXCLUSION_REGEX):+            if debug:+                print(f"[WARN] Excluding {filepath} based on header '{header[0]}'")+            return True++        if len(header) <= 3:+            print("Mismatch found! File too small for header stanza!")+            return False++        for line_idx, matcher in enumerate(COMPILED_COPYRIGHT_REGEX):+            if not re.match(matcher, header[line_idx]):+                print(f"Mismatch found! Expected '{COPYRIGHT_REGEX[line_idx]}' pattern but got '{header[line_idx]}'")+                return False++        return True++    @staticmethod+    def _assert_copyrights(files, debug=False):+        failing_files = []+        for filepath in files:+            if CopyrightLinter._has_copyright(filepath, debug=debug):+                if debug:+                    print(f"[ OK ] {filepath}")++                continue++            print(f"[FAIL] {filepath}")+            failing_files.append(filepath)++        total_files = len(files)+        if failing_files != 0:

Yup - good catch!

sgnn7

comment created time in 3 days

PR opened DataDog/datadog-agent

[tools] Add copyright lint tooling for Golang files kind/enhancement dev/tooling severity/minor team/agent-platform qa/skip-qa python category/improvement

What does this PR do?

This tool addition can be used via inv lint-copyrights which traverses the repo tree and fails if it finds any Golang files with an unexpected copyright header.

For now the tool is intended to be invoked manually but once all the files get fixed, we can add it to our internal CI/CD pipeline as an automated check.

Motivation

Legal compliance needs

Additional Notes

  • Eventually we would like to enable this check in CI/CD but we need all the failures cleaned up first.

Possible Drawbacks / Trade-offs

The tool could ideally have the ability to auto-fix problems but that's outside of the scope of this task right now.

Describe how to test/QA your changes

N/A but if test is needed, running inv lint-copyrights should show the output.

Reviewer's Checklist

  • [ ] If known, an appropriate milestone has been selected; otherwise the Triage milestone is set.
  • [ ] The appropriate team/.. label has been applied, if known.
  • [ ] A release note has been added or the changelog/no-changelog label has been applied.
  • [ ] Changed code has automated tests for its functionality.
  • [ ] Adequate QA/testing plan information is provided if the qa/skip-qa label is not applied.
  • [ ] If applicable, docs team has been notified or an issue has been opened on the documentation repo.
  • [ ] If applicable, the need-change/operator and need-change/helm labels have been applied.
  • [ ] If applicable, the config template has been updated.
+197 -0

0 comment

3 changed files

pr created time in 3 days

push eventDataDog/datadog-agent

Srdjan Grubor

commit sha 4d875fe61662a526d4617cd162550ac0f6c30aa8

[tools] Add copyright linting tool for Golang files This addition can be used via `inv lint-copyright` which traverses the repo tree and fails if it finds any Golang files with an unexpected copyright header. For now the tool is intended to be invoked manually but once all the files get fixed, we can add it to our internal CI/CD pipeline as an automated check.

view details

push time in 3 days

create barnchDataDog/datadog-agent

branch : sgnn7/add_copyright_checker

created branch time in 3 days

PullRequestReviewEvent

delete branch sgnn7/openssl-fips

delete branch : update_readme

delete time in 4 days

PR merged sgnn7/openssl-fips

[Docs] Update README

This should provide a bit more info on the project and how to build it.

+35 -0

0 comment

1 changed file

sgnn7

pr closed time in 4 days

push eventsgnn7/openssl-fips

Srdjan Grubor

commit sha 87cbe05cc2cab1bf0617a7272c405d025e932de3

[Docs] Update README This should provide a bit more info on the project and how to build it.

view details

Srdjan Grubor

commit sha d19cd90f320da909bc3bcafe58b05f9ee0e51a76

Merge pull request #2 from sgnn7/update_readme [Docs] Update README

view details

push time in 4 days

PR opened sgnn7/openssl-fips

[Docs] Update README

This should provide a bit more info on the project and how to build it.

+35 -0

0 comment

1 changed file

pr created time in 4 days

create barnchsgnn7/openssl-fips

branch : update_readme

created branch time in 4 days

push eventsgnn7/openssl-fips

Srdjan Grubor

commit sha d9989d9e741fc7223a66d105ac81f489e4f7bec8

[Build] Create GitHub Actions to build/test the Linux containers Automations for testing the changes is needed at this point and this should now be done automatically.

view details

Srdjan Grubor

commit sha eef6ae096f429f812744578f5a9a463837205481

[Meta] Add CODEOWNERS

view details

push time in 4 days

push eventsgnn7/openssl-fips

Srdjan Grubor

commit sha d9989d9e741fc7223a66d105ac81f489e4f7bec8

[Build] Create GitHub Actions to build/test the Linux containers Automations for testing the changes is needed at this point and this should now be done automatically.

view details

Srdjan Grubor

commit sha eef6ae096f429f812744578f5a9a463837205481

[Meta] Add CODEOWNERS

view details

push time in 4 days

push eventsgnn7/openssl-fips

Srdjan Grubor

commit sha 18d05a47a21aeda7cf3c3accc0072ddf72147771

[Build] Create GitHub Actions to build/test the Linux containers Automations for testing the changes is needed at this point and this should now be done automatically.

view details

Srdjan Grubor

commit sha 047540424e05d90181f50bb8c4f12337750f1849

[Meta] Add CODEOWNERS

view details

push time in 4 days

push eventsgnn7/openssl-fips

Srdjan Grubor

commit sha 18d05a47a21aeda7cf3c3accc0072ddf72147771

[Build] Create GitHub Actions to build/test the Linux containers Automations for testing the changes is needed at this point and this should now be done automatically.

view details

Srdjan Grubor

commit sha 047540424e05d90181f50bb8c4f12337750f1849

[Meta] Add CODEOWNERS

view details

push time in 4 days

create barnchsgnn7/openssl-fips

branch : add_linux_github_actions

created branch time in 4 days

push eventsgnn7/openssl-fips

Srdjan Grubor

commit sha 36871e806b288977a80ecf2965716ef38b8431c9

[Build] Create GitHub Actions to build/test the Linux containers Automations for testing the changes is needed at this point and this should now be done automatically.

view details

Srdjan Grubor

commit sha 34fae839151f43354c747ba291beee46babbde9a

[Meta] Add CODEOWNERS

view details

push time in 4 days

create barnchsgnn7/openssl-fips

branch : test_actions

created branch time in 4 days

push eventsgnn7/openssl-fips

Srdjan Grubor

commit sha 754f2cb8483949c79d6637f9488321bb2f305c0c

[Windows] Fix various issues in installing Cygwin in the container Cygwin had various issues getting installed in the Windows container so this commit should fix most of those: - Path addition - Bash error ignoring - Package installation - Compliance with security policy

view details

push time in 5 days

push eventsgnn7/openssl-fips

Srdjan Grubor

commit sha 860b37770321504aef283cbd884bfeee79ae50de

[Build] Made build consider non-sudo setups On some platforms (like MacOS), builds do not require sudo as the docker invocation prefix so this change ensures that we handle this condition.

view details

Srdjan Grubor

commit sha 6930f9868866dc0df5c8f4d71980a8ef1b38fb96

[Windows] Change default image to 20H2 Since the test image is running agains Windows 20H2, we don't necessaerly want the ltsc tag.

view details

Srdjan Grubor

commit sha be8a32c074c50fd323f8ff83346af676734266f1

[Build] Allow for auto-detected multi-platform build We can now tell what platform we are on and accordiingly build the right Dockerfile.

view details

push time in 6 days

push eventsgnn7/openssl-fips

Srdjan Grubor

commit sha 8f4d3afdec5836bbac95377ee1424b5f4b3ea13c

[WIP] Added Windows canister image Dcokerfile This is still just a skeleton but so far it can download the right file, install cygwin, and verify checksum.

view details

push time in 7 days

more