@jpreese thanks for picking this back up! Ill take a look at those PR changes shortly

Description

• Adding some changes that have been lingering locally for IRC and RSS
• Adding clipboard copying in tmux and bash alias to pbcopy

Description

• Adding some changes that have been lingering locally for IRC and RSS
• Adding clipboard copying in tmux and bash alias to pbcopy

@ckauhaus The aws provider is package by nixpkgs: https://github.com/NixOS/nixpkgs/blob/241eef1ec360ea3a578d996cee71051ee563c3c6/pkgs/applications/networking/cluster/terraform-providers/data.nix#L75-L82

This issue should still be closed since according to NVD report it looks like its referencing a pretty old version of the aws provider:

aws/resource_aws_iam_user_login_profile.go in the HashiCorp Terraform Amazon Web Services (AWS) provider through v1.12.0

Minimal trivial changes, I think everything looks fine otherwise.

@samueldr Implemented those changes and ran the same set of tests mentioned in the description. Everything checked out fine

Definitely makes sense

Thank you for the context, Ill keep that in mind for the future

@cemeyer Thanks for the clarification, thats makes sense since xrandr is under /usr/local/bin in FreeBSD.

Regardless, I think it still makes some sense cover both cases and check _PATH_STDPATH if for whatever reason the user was to botch their PATH.

Description

Depends on: https://github.com/Ventto/libshlist/pull/2 Closes: #30

It does not seem that "command -pv xrandr" is able to find xrandr in my PATH on FreeBSD. Im not sure if this is due to there just not being a default /etc/environment set or what but I think we should be checking the current PATH as well as the default to make sure we search all possibilities.

Let me know what you think.

Tests

Building with my PR from libshlist: https://github.com/Ventto/libshlist/pull/2 nets me the expect results:

mkdir build
make install PREFIX=$(realpath ./build) LIB=$(realpath ../libshlist/liblist.sh)
...
$ build/bin/mons
Monitors:        2
Mode: primary
0:*  LVDS-1   (enabled)

Description

This was related to the FreeBSD port for mons and the issue opened up in the portstree bugzilla: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248632

BSD sed was unable to correctly leverage i command resulting in the following output when running mons on master 375bbba3aa700c8b3b33645a7fb70605c8b0ff0c:

 $ ./mons
sed: 1: "1iLVDS-1": command i expects \ followed by text
Monitors: 0
Mode: primary

The following change was initially suggested by @kevans91 here: https://github.com/Ventto/mons/issues/30#issuecomment-674335403 but I did some more testing to get the following to be complaint for both GNU and BSD versions of sed.

Tests

I have added appropriate tests for both the libshlist and libshlist_unsafe variations.

$ sh tests/test_freebsd.sh 
sed is /usr/bin/sed
test: insert pass
--------------------------
sed is a shell function
test: insert pass

$ sh tests/test_unsafe_freebsd.sh 
sed is /usr/bin/sed
test: insert pass
--------------------------
sed is a shell function

As for mons this also fixes the original error. I do have an issue with how command -pv xrandr is working and have to modify it to command -v xrandr but Ill open that issue in mons for discussion:

$ ./mons
Monitors:        2
Mode: primary
0:*  LVDS-1   (enabled) 

<!-- To help with the large amounts of pull requests, we would appreciate your reviews of other pull requests, especially simple package updates. Just leave a comment describing what you have tested in the relevant package/service. Reviewing helps to reduce the average time-to-merge for everyone. Thanks a lot if you do! List of open PRs: https://github.com/NixOS/nixpkgs/pulls Reviewing guidelines: https://hydra.nixos.org/job/nixpkgs/trunk/manual/latest/download/1/nixpkgs/manual.html#chap-reviewing-contributions -->

Motivation for this change

This adds sd-mux-ctrl which is a tool for controlling multiple sd-mux devices for automated testing of single board computers: https://wiki.tizen.org/Laboratory

I figured this might be useful to others who are working on single-board computers (for example nixos: https://nixos.wiki/wiki/NixOS_on_ARM/Raspberry_Pi) and require automated testing around these systems.

I have been using sd-mux-ctrl with SDWire hardware. This board is based on sd-mux devices minus any power switch or USB switching. Essentially it enables you to flash the sdcard without having to take the card out of the board. The workflow ends up being something along the lines of:

sudo sd-mux-ctrl --device-serial=odroid_u3_1 --ts
... - do the flashing things (dd)
sudo sd-mux-ctrl --device-serial=odroid_u3_1 --dut

ts = test server (i.e. laptop, workstation) dut = device under test (i.e. rpi, Odroid, etc.)

Things done

<!-- Please check what applies. Note that these are not hard requirements but merely serve as information for reviewers. -->

• [ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • [ ] NixOS
  • [ ] macOS
  • [x] other Linux distributions
• [ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• [ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• [x] Tested execution of all binary files (usually in ./result/bin/)
• [ ] Determined the impact on package closure size (by running nix path-info -S before and after)
• [ ] Ensured that relevant documentation is up to date
• [x] Fits CONTRIBUTING.md.

@dlangille saw your tweet: https://twitter.com/DLangille/status/1285750793711292416 sounds like youve made some good progress!

[freshports@devgit-ingress01 ~/ports-jail/var/db/repos/PORTS-head-git]$ git status HEAD detached at 44d4d38cf77e nothing to commit, working tree clean

This seems to be due to the fact that when git-to-freshports-xml.py exits with an error that it leaves the repo in a less than ideal state and in this case detached HEAD. Im not sure why git-to-freshports-xml.py actually needs to try to checkout each of these commits vs just leverage something like git show <SHA> but it still looks like its having issues with being able to leverage the type of HEAD, again this seems like its due to a mismatch of the actual type being used in python.

Anyway it sounds like youve got a path forward and thats good 😄

Description

This PR adds in another Docker image to the repo, pki-validator. This image is designed to be used to run test cases with openssl and make.

Tests

• [x] Validated image can be created via nix-shell
• [x] Validated that all needed packages are placed in configuration

@maishiroma awesome! thanks for adding the last comment about the owners group too

nice README 😎

Looks like what we would need

Description

This PR continues to improve on the publish-imgs GitHub Action workflow by now auto-creating any new Docker Hub repository when a new image is being published for the first time.

In order to facilitate this workflow, the following new items are added to the repository:

1. A templates/ directory which contains the following:
   • A README_imgs.md, which outlines a standardized README for all images made in this repository
   • A README.md, explaining the purpose and usage of the templates in that directory
2. A new script, create-repo, which facilitates the creation of a new Docker Hub repo via API calls
   • This script also does the following:
     • Adds in the engineering group with read/write permissions
     • Adds in both a short_description and a long_description, with them being based on the README_imgs.md that will be in all imgs/

Benefits

The biggest win for this PR is the ability for new images that are not yet existing in Nebulaworks Docker Hub to be pushed up without manual intervention on a Nebulaworks user to create a repository first. That small limitation slows down the publishing process, and with the current workflow we have in this repository, new images will have to get approved by a verified user, which helps prevent any unnecessary or malicious images to be published under Nebulaworks.

Workflow changes

Other than having a soft requirement of needing all imgs/ to have a README.md under the template as the README_imgs.md, from a CI perspective, nothing has changed.

• The repo creation will be invoked during the CI build of publish-imgs, and will only actually create a repo if there does not exist one already.
• For manually publishing images, the documentation for that has been modified accordingly.

Special Notes:

• The reason for parts of the publish-imgs now having set +x and set -x being encapsulated around code parts that are using the DOCKER_HUB_TOKEN is to prevent that value from being visible to anyone (since this is a public repository). While one alternative could be to modify the create-repo script to calculate the token, this script will be invoked X times (X = numb of images), which could lead to the script slowing down + being inefficient with its data. As such, only doing it once and passing in the token as a CLI argument was the most ideal way I found.
• With the standardization of the image READMEs, I also added in READMEs for all existing images that did not have one, making sure all of them abid by the new format.

Tests

• Verified workflow in own environment
• Specific GitHub Action run that yielded the desired results.
• Verified that the org permissions work (WIP)

@maishiroma this looks good, lets :shipit:

If we are going to write out this data.json can we just put it in a temp directory just in case? Id hate for this to end up clobbering something else in the future.

tmpdata=$(mktemp /tmp/tempdata.XXXXXX)
cat << EOF > $tmpdata
...
EOF

Thanks for masking the credentials

We should exit 0 here if the repo already exists. I dont think we need to worry enforcing the group or readme for every publish run (Im worried about API limits as the number repos grows). Maybe we come back and add for -f (force flag) that does this but once we reconcile the delta with the 3 existing docker repos all the others should come into line.

Thanks for adding some checks for args

Itll be so much nicer to have these descriptions 👍

Can we make this a README_imgs.tpl or something along those lines just to make it a bit more obvious?

The saga continues for these small devices. This could effect our our tplinks too:

• https://bugs.openwrt.org/index.php?do=details&task_id=2651#comment8515

@davidelang just found this on the mailing lists, going to standby to see how things progress: https://bugs.openwrt.org/index.php?do=details&task_id=1714#comment8500

@skozlov404

I'd say it doesn't even matter if the tree is dirty or not - the script processes the commits that already happened, so git reset shouldn't be needed at all. The only thing that matters is that you do git fetch origin beforehand so your tree is up to date

Oh, I get what's been suggested in #3 - since git-to-freshports-xml.py used to only process all the commits from the specified one to the HEAD - if we move the HEAD to right above the commit we're specifying - this would give us the effect of processing the single commit.

Thing is, with --single-commit and --commit-range flags now implemented - we don't need to jump around the git tree anymore like that - by just using git fetch origin and the proper flags to git-to-freshports-xml.py we're now able to achieve everything required.

The idea of reset --hard HEAD was just to make sure for any reason that the local master was clean so that we could git rebase since the rebase needs to have a clean tree before it can proceed.

git-delta.sh was assuming that git-to-freshports-xml.py just needed a starting point, so I figured we could leverage the remote local master branch compared to the local master from when we last synced to get our starting point and just pass that along to git-to-freshports-xml.py to process the commits from the starting point to HEAD.

We arent jumping around the tree, just leveraging the lag from the last time we synced and then just fetching last copy of master and comparing it to local master and bring local master up to date.

@dlangille it looks like your error is just a TypeError due to how that git library processes the commits. Pulling from your output above https://github.com/FreshPorts/git_proc_commit/issues/7#issuecomment-655797104:

TypeError: HEAD is a detached symbolic reference as it points to 'f2bfe60090b840b6d99a3288c0b745843cefcfe1'

It would seem that we need to handle HEAD in a special way (python type is different) or just get the commit hash that HEAD currently points to so we can make the ranging in git-to-freshports-xml.py work correctly.

One issue I keep thinking about: getting out of sync. We want to make sure the latest commit in our repo is the latest commit in the database. If it is not, we need to process what's in the repo before doing a fetch.

In regards to this, I think this is a separate issue compared to whats being discussed here. I would think just checking the database for the last commit stored then comparing that to the starting point in git-delta.sh. The starting point should just be 1 commit ahead of whats in the database, if not move the starting point to the the commit right after the database entry and let git-to-freshports-xml.py process it all. We are guaranteed that the history will be in the right order as a no one in the portstree is rewriting history on master (this is not the case)

@dlangille thanks, Ill take a look at this tonight and give you some feedback. At the moment Im not near my machine.

Description

It would appear that ath79 is landing in upstream openwrt: http://lists.openwrt.org/pipermail/openwrt-devel/2020-July/029879.html

Update openwrt ath79 to kernel 5.4 using the automatic build process. We might have to patch the existing version if its not yet landed upstream in master

Acceptance Criteria

• Confirm that openwrt builds on ath79
• Run through automatic build process

Summary

This PR adds in a new docker image configuration that automatically sets up a self-hosted rendezvous server for a magic-wormhole service.

Benefit

Since there are little open sourced solutions to building consistent solutions in running self-hosted solutions, I think this can greatly help others see a working example of this and how they can implement this in their own solutions.

Tests

• Build derivation successfully and imported docker image
• Ran container and tested service locally

@maishiroma awesome, looking forward to using this! :shipit:

Can we make this more uniform? /var/lib/wormhole or something like that would be more appropriate I think. This way we can standardize on the hierarchy of the filesystem: https://www.man7.org/linux/man-pages/man7/hier.7.html vs just having a bunch of things hanging off /.

Also lets make the something a tighter, 750 seems more appropriate

Absolutely @dlangille you can the following:

# Make sure we are on the master branch
git checkout master
# Go back to a specific hash
git reset --hard COMMITHASH

Just substitute the COMMITHASH for the hash you originally started from.

@maishiroma thanks for all the help on this, this is now ready to go! 🚀

Adding some context around why helmsman-aws change. @maishiroma dug into why the derivation changed due to #10 and his findings were:

I just did a build of the old helmsman-aws image prior to the fmt and I was able to get the existing image hash. The change did
arose from the fmt, but it was in the deps.nix in the pkgs/helm-diff . By fmt that, it caused the hash to change in the 
nix-build, creating a new image for helmsman-aws. The odd thing though is that there is no heredoc that is being modified in
that nix config. In fact, the heredoc  doesn’t seem to matter in this case. awsutils has a heredoc that got modified in the
default.nix , but that was not enough of a difference to generate a brand new image (looking at the last job for publishing
images showed this).

Output taken from: https://github.com/Nebulaworks/nix-garage/runs/828756395?check_suite_focus=true

/nix/store/pbj9q2qyw511wfy18g8lz9jw2mpc5qvw-docker-image-helmsman-aws.tar.gz
+ cp /__w/nix-garage/nix-garage/imgs/helmsman-aws//result /__w/nix-garage/nix-garage/result
+ nix-shell --run './publish-imgs helmsman-aws'
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
time="2020-07-02T00:06:45Z" level=fatal msg="Error parsing image name \"docker://docker.io/nebulaworks/helmsman-aws:pbj9q2qyw511wfy18g8lz9jw2mpc5qvw\": Error reading manifest pbj9q2qyw511wfy18g8lz9jw2mpc5qvw in docker.io/nebulaworks/helmsman-aws: manifest unknown: manifest unknown"
Getting image source signatures
Copying blob sha256:4864e02891e4b109b1a8399507183f666a1a04340ac51a4c7203281836254d14
Copying config sha256:252c03ebc4b2f7c4bd79baa8488a6d4a96d1ca9414ff2d4a4765b6701291b8cb
Writing manifest to image destination
Storing signatures

Closing and going with #11

Description

Copy of #12

Trying out a branch in this repo vs fork

publish

publish

Description

After noticing publish failures it looks like we were expecting the wrong secret credential. These was an oversight while testing some new features of our actions workflow. This should fix things.

publish

publish

publish

Description

Copy of #12

Trying out a branch in this repo vs fork

publish

Summary

This PR fixes an issue with the workflow files in that they were missing a key parameter to make sure that they were pulling code from the PR, not from the default branch. Also, some minor clean up was done on the workflow files in order to keep consistency as well as clarity.

Note: The lint-nixpkgs job still worked as intended because it was using a PR event, which the default action for actions/checkout in that case is to use the PR's head commit. This isn't the case for the publish_imgs, which uses an issue_comment event. As such, to keep consistency, the lint_nixpkgs is also modified to reflect the specific usage of a branch SHA.

Authentication Fix Explanation

The issue of the authentication error comes from the context that the event was running from. Because of the default behavior of actions/checkout@v2 as well as the event being used coming from an issue_comment, the context used was not the base repository, but instead, the context of the forked branch. As such, secrets are not passed in that fork context, despite the workflow job running in the base repository.

The fix for this was to extract the SHA of the PR's head commit, which will take the work that was made from the PR, but allowing for the use of secrets declared in the base repository. This behavior can be seen in this PR and in this successful job. Note that the workflow in that repository is using a SHA ref to the PR.

Tests

• Verified in my own environment of changes:
  • PR used
  • Successful Job, noting the steps in Verify Commentor is Approver and Authenticate and Execute Publish Script

Thanks for adding this conditional just in case

@maishiroma thanks for the work on this. Looking forward to see how the checkout behavior changes with this new version of the github action. I think it should get us toward fixing #11.

As we discussed outside of this thread there seems to be something brought about by the nixpkg-fmt from #10 causing the derivation change on helmsman-aws

Description

@dlangille following up from our conversation earlier today: https://twitter.com/sarcasticadmin/status/1278075104417603584

I know in your initial examples you have the commit hash that youll be updating each time you run this script. Wanted to also point out that you could alternatively leverage some of the built-in functionality in git to do this without having to embedded a commit SHA in the code.

Below is a script for updating freesbd-ports. It assumes that the repo is already cloned down although that could be handled here too if desired. But the idea is that since you have a local copy of the remote master branch and a local master you can diff them to see where you last left on the git log to xml conversion and then just run the conversion as you typically would.

git-delta.sh:

# Assumes you already cloned down the repo once
# Leverage the fact that we already have the new starting
# point since we know the position of the local master branch
# the last time we ran this

REMOTE='origin'

# Update local copies of remote branches
git fetch $REMOTE

# Make sure branch is clean and on master
git reset --hard HEAD
git checkout master

# Get the first commit for our starting point
STARTPOINT=$(git log master..$REMOTE/master --oneline --reverse | head -n 1 | cut -d' ' -f1)

# Bring local branch up-to-date with the local remote
git rebase $REMOTE/master

# Call xml conversion with starting point
#python3 git-to-freshports.py -c $STARTPOINT
echo "Would have ran git-to-freshport.py starting at: $STARTPOINT"

Example

The following is an example I had from a local ports tree. I have git-delta.sh in the same dir as the rest of the ports tree. Set ports tree master so some old commit. This commits from Jan:

$ git checkout master && git reset --hard d3b56095dd0e

Run the delta:

$ sh git-delta.sh  
HEAD is now at d3b56095dd0e Update to 0.11.2.
Already on 'master'
Your branch is behind 'origin/master' by 47065 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)
Successfully rebased and updated refs/heads/master.
Would have ran git-to-freshport.py starting at: d475beb9e713

Looking at d475beb9e713 its the next commit in the tree after d3b56095dd0e:

commit d475beb9e7130c8bc2ccf4108229932285f52f84
Author: jbeich <jbeich@FreeBSD.org>
Date:   Wed Jan 30 05:28:56 2019 +0000

    multimedia/libva-intel-media-driver: update to 18.4.0 (release)
    
    Changes:        https://github.com/intel/media-driver/compare/7e966d06...intel-media-18.4.0
    Changes:        https://github.com/intel/media-driver/releases/tag/intel-media-18.4.0
    MFH:            2019Q1 (stabilization)

Hope this is helpful!

@dlangille awesome, happy to help!

BTW, you mentioned "without having to embedded a commit SHA in the code " - git-show-commit.sh was a proof of concept to verify iteration could happen. When expanded, it would not have had a hardcoded hash.

Ya, understood. I just wanted to point out that the repo has all the info you need.

Thanks for all the work you've done with Freshports, I use it daily!

Description

@dlangille following up from our conversation earlier today: https://twitter.com/sarcasticadmin/status/1278075104417603584

I know in your initial examples you have the commit hash that youll be updating each time you run this script. Wanted to also point out that you could alternatively leverage some of the built-in functionality in git to do this without having to embedded a commit SHA in the code.

Below is a script for updating freesbd-ports. It assumes that the repo is already cloned down although that could be handled here too if desired. But the idea is that since you have a local copy of the remote master branch and a local master you can diff them to see where you last left on the git log to xml conversion and then just run the conversion as you typically would.

git-delta.sh:

# Assumes you already cloned down the repo once
# Leverage the fact that we already have the new starting
# point since we know the position of the local master branch
# the last time we ran this

REMOTE='origin'

# Update local copies of remote branches
git fetch $REMOTE

# Make sure branch is clean and on master
git reset --hard HEAD
git checkout master

# Get the first commit for our starting point
STARTPOINT=$(git log master..$REMOTE/master --oneline --reverse | head -n 1 | cut -d' ' -f1)

# Bring local branch up-to-date with the local remote
git rebase $REMOTE/master

# Call xml conversion with starting point
#python3 git-to-freshports.py -c $STARTPOINT
echo "Would have ran git-to-freshport.py starting at: $STARTPOINT"

Example

The following is an example I had from a local ports tree. I have git-delta.sh in the same dir as the rest of the ports tree. Set ports tree master so some old commit. This commits from Jan:

$ git checkout master && git reset --hard d3b56095dd0e

Run the delta:

$ sh git-delta.sh  
HEAD is now at d3b56095dd0e Update to 0.11.2.
Already on 'master'
Your branch is behind 'origin/master' by 47065 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)
Successfully rebased and updated refs/heads/master.
Would have ran git-to-freshport.py starting at: d475beb9e713

Looking at d475beb9e713 its the next commit in the tree after d3b56095dd0e:

commit d475beb9e7130c8bc2ccf4108229932285f52f84
Author: jbeich <jbeich@FreeBSD.org>
Date:   Wed Jan 30 05:28:56 2019 +0000

    multimedia/libva-intel-media-driver: update to 18.4.0 (release)
    
    Changes:        https://github.com/intel/media-driver/compare/7e966d06...intel-media-18.4.0
    Changes:        https://github.com/intel/media-driver/releases/tag/intel-media-18.4.0
    MFH:            2019Q1 (stabilization)

Hope this is helpful!

publish