profile
viewpoint

beehive-lab/TornadoVM 409

🌪️ TornadoVM: A practical and efficient heterogeneous programming framework for managed languages

graalvm/mandrel 166

Mandrel is a downstream distribution of the GraalVM community edition. Mandrel's main goal is to provide a native-image release specifically to support Quarkus.

zakkak/android-emacs-toolkit 5

Automatically exported from code.google.com/p/android-emacs-toolkit

beehive-lab/Maxine-Dockerfile 4

A Dockerfile to build a docker image that is capable of compiling and running the Maxine VM

DaKnOb/OpenHost 4

A host monitoring tool for the Computer Science Dept. of the University of Crete

acticloud/neo4j 2

Graphs for Everyone

acticloud/tpch-monetdblite 2

Run TPC-H benchmarks on MonetDBLite

zakkak/c1visualizer 2

A mirror of the c1visualizer source code http://lafo.ssw.uni-linz.ac.at/c1visualizer/

DaKnOb/Doorlock 1

The Doorlock project of ToLABaki Hackerspace / www.tolabaki.gr

pull request commentgraalvm/mandrel-packaging

Jbang the release process!

I can add support for it though.

@Karm Please let me know if you need this :)

zakkak

comment created time in 4 days

issue commentgraalvm/mandrel

Native image, executable debugging, GDB: `list` does not show current source, "No such file or directory"

@Karm does mvn package generate an archive with the sources along with target/debug-symbols-smoke.jar? If not, that's the issue here.

When using jars Graal is looking for target/debug-symbols-smoke-sources.jar or target/debug-symbols-smoke.zip to get the source files from. See https://github.com/graalvm/mandrel/blob/mandrel/20.1/substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/image/sources/SourceCache.java#L198-L217

Karm

comment created time in 4 days

pull request commentgraalvm/mandrel-packaging

Jbang the release process!

@zakkak I cleared .git/config, updated to your latest code and run it again. It went further than before, failing on the below mark now:

...
[INFO] Changes pushed to remote mandrel-release-fork
[INFO] Checked out mandrel/20.2
[ERROR] {"message":"Validation Failed","errors":[{"resource":"PullRequest","field":"head","code":"invalid"}],"documentation_url":"https://docs.github.com/rest/reference/pulls#create-a-pull-request"}

Note the branch was actually created: https://github.com/Karm/mandrel-demo-release/tree/release/mandrel-20.2.0.0.Final ...but it seems opening PR failed?

Hmmm this is probably because you are trying to open a PR in the same repository (both your base and target are Karm/mandrel-demo-release). I didn't think we need to support this since we always work on forks of graalvm/mandrel. I can add support for it though.

zakkak

comment created time in 4 days

pull request commentgraalvm/mandrel-packaging

Jbang the release process!

I was more worried about you, picturing you at the other end of the PR getting like "OMG, why Karm ain't merging yet!?". I will cycle back to it in some minutes / soon and try again.

Nuh, everything you pointed out makes sense and should be fixed. That's the purpose of the review after all :) Thanks

zakkak

comment created time in 4 days

push eventzakkak/mandrel-packaging

Foivos Zakkak

commit sha 1dd6c5e98aa7b7aace8154b9dbefa5b2c20a487a

Jbang the release process!

view details

push time in 4 days

pull request commentgraalvm/mandrel-packaging

Jbang the release process!

I think this PR is too much dependent on what you already have configured and set on your own system and that's why it runs just fine for you....

I wouldn't say so. Till now one issue was the wrong version in branch 20.2 and the others were related to unclear documentation:

  • one was about the location from where you should run the script
  • and the other about the --fork-name whose default value is my fork (which I am happy to change, and make it a required parameter).

The latest issue is the only one that seems to be related to my environment but for it to appear I would have to run the sequence:

$ release-scripts/./mandrel-release.java prepare
$ release-scripts/./mandrel-release.java prepare --fork-name myaccount/mandrel-fork

So even with a clean env and a demo/bot account it would be hard to detect :) (since I would know that --fork-name is needed and would probably not execute that sequence)

The problem was that https://github.com/zakkak/mandrel-packaging/blob/3ee9149edfd47b3e436f0c3cc4f874a8da2ce266/release-scripts/mandrel-release.java#L123 did not force-add/update the remote if it already existed. So in your case it was created in the first run pointing to zakkak/mandrel and in the second run it was not updated to point to Karm/mandrel-demo-release (but was not detected so it failed silently).

WDYT?

I can give the new version a "clean room" spin, but it will take me some time to prepare the env and set up a new GH account. So if it's not too frustrating could you please give it another spin?

zakkak

comment created time in 4 days

push eventzakkak/mandrel-packaging

Foivos Zakkak

commit sha bfbb21930afb7870d66163f3dcc3960dd8840c1f

Jbang the release process!

view details

push time in 4 days

issue commentbeehive-lab/Maxine-VM

[RISCV64]Fedora(on QEMU) run test,FatalError: stack reset position two

No unfortunately I don't. Maybe @gigiblender, @bakaliosdim, or @evankar can help you here. Does any of you remember if xalan was functional? According to https://github.com/beehive-lab/Maxine-VM/blob/develop/docs/Status.rst#dacapo-912-bach-mr1 it was, but I recall @bakaliosdim and @evankar reporting some differences between their results and what's listed in https://github.com/beehive-lab/Maxine-VM/blob/develop/docs/Status.rst#dacapo-912-bach-mr1

zdlgv5

comment created time in 4 days

delete branch zakkak/mandrel

delete branch : fix-quarkus-CI

delete time in 4 days

PR closed oracle/graal

Don't use a container image to produce the native-images oca-signed

Currently the workflow uses -Dquarkus.native.container-build=true which uses a container image from https://quay.io/repository/quarkus/ubi-quarkus-native-image?tab=tags instead of using the build we just created.

+2 -2

1 comment

1 changed file

zakkak

pr closed time in 4 days

pull request commentoracle/graal

Don't use a container image to produce the native-images

This was merged in https://github.com/oracle/graal/commit/39416cb9b358bc6ac12601b15e294ef4905144cd

Thanks @dougxc

zakkak

comment created time in 4 days

push eventzakkak/mandrel-packaging

Foivos Zakkak

commit sha 3ee9149edfd47b3e436f0c3cc4f874a8da2ce266

Jbang the release process!

view details

push time in 5 days

pull request commentgraalvm/mandrel-packaging

Jbang the release process!

@Karm what's the error in the second case?

zakkak

comment created time in 5 days

PR opened graalvm/mandrel-packaging

Reviewers
Remove dependencies instead of building with --only enhancement

The idea of this change is to remove the dependencies we don't need instead of defining which ones we want and having to figure out the dependency tree ourselves.

Changes tested in: master build: https://ci.modcluster.io/view/Mandrel/job/mandrel-20.1-linux-build/105/ integration-test: https://ci.modcluster.io/job/mandrel-20.1-integration-test/OS_AXIS=centos8%7C%7Crhel8/14/ quarkus-tests: https://ci.modcluster.io/view/Mandrel/job/mandrel-20.1-linux-test/OS_AXIS=centos7/122/ and https://ci.modcluster.io/view/Mandrel/job/mandrel-20.1-linux-test/122/OS_AXIS=centos8%7C%7Crhel8/ (still in progress)

+41 -30

0 comment

1 changed file

pr created time in 5 days

pull request commentgraalvm/mandrel-packaging

Jbang the release process!

@Karm I think I tackled all issues. Please try again and let me know.

zakkak

comment created time in 5 days

push eventzakkak/mandrel-packaging

Foivos Zakkak

commit sha 67aed57760e16dd74515f79c860c78dcd8dfeb7e

Jbang the release process!

view details

push time in 5 days

delete branch zakkak/mandrel

delete branch : fix-suites

delete time in 5 days

push eventgraalvm/mandrel

Foivos Zakkak

commit sha 46e0285650b0affeca6ca2f296fd71a106637d3e

Fix version and release fields in suite.py files

view details

push time in 5 days

PR merged graalvm/mandrel

Fix version and release fields in suite.py files oca-signed
+15 -15

0 comment

8 changed files

zakkak

pr closed time in 5 days

PR opened graalvm/mandrel

Reviewers
Fix version and release fields in suite.py files
+15 -15

0 comment

8 changed files

pr created time in 5 days

create barnchzakkak/mandrel

branch : fix-suites

created branch time in 5 days

pull request commentgraalvm/mandrel-packaging

Jbang the release process!

It's supposed to be run inside the mandrel-repo so 3 was expected (by me) to fail. I think running it from the mandrel-packaging directory might make more sense indeed, I'll make it ask for the mandrel-repo path as an argument.

Step 4 failed because of https://github.com/graalvm/mandrel/blob/052fce581c89fef8f75dd831464f8bca34828e68/substratevm/mx.substratevm/suite.py#L4

The version there should be 20.2.0.0 instead of 20.2.0 (mandrel versioning vs graal versioning). I will open a PR to fix this on 20.2 and will try to improve the error message of the release script

Thanks for the feedback!

zakkak

comment created time in 5 days

issue closedgraalvm/mandrel

Mandrel 20.1 branch is broken: missing implementation for runtime call: indexOf1Byte(byte[],int,int,byte)int

14:47:06 + javac Hello.java
14:47:07 + native-image Hello
14:47:10 [hello:30957]    classlist:   1,864.97 ms,  0.96 GB
14:47:11 [hello:30957]        (cap):     733.80 ms,  0.96 GB
14:47:12 [hello:30957]        setup:   2,353.48 ms,  1.21 GB
14:47:21 [hello:30957]     (clinit):     157.44 ms,  1.69 GB
14:47:21 [hello:30957]   (typeflow):   4,025.43 ms,  1.69 GB
14:47:21 [hello:30957]    (objects):   4,373.29 ms,  1.69 GB
14:47:21 [hello:30957]   (features):     219.18 ms,  1.69 GB
14:47:21 [hello:30957]     analysis:   8,954.62 ms,  1.69 GB
14:47:21 [hello:30957]     universe:     294.50 ms,  1.69 GB
14:47:22 [hello:30957]      (parse):     939.89 ms,  1.71 GB
14:47:24 [hello:30957]     (inline):   1,054.62 ms,  1.71 GB
14:47:26 [hello:30957]    (compile):   2,752.19 ms,  2.65 GB
14:47:26 [hello:30957]      compile:   4,831.85 ms,  2.65 GB
14:47:26 Fatal error:org.graalvm.compiler.debug.GraalError: com.oracle.svm.core.util.VMError$HostedError: missing implementation for runtime call: indexOf1Byte(byte[],int,int,byte)int
14:47:26 	at method: int java.lang.String.indexOf(int, int)  [Direct call from boolean RuntimePropertyParser.parseProperty(String)]
14:47:26 	at com.oracle.svm.hosted.code.CompileQueue.defaultCompileFunction(CompileQueue.java:1014)
14:47:26 	at com.oracle.svm.hosted.code.CompileQueue.doCompile(CompileQueue.java:945)
14:47:26 	at com.oracle.svm.hosted.code.CompileQueue$CompileTask.run(CompileQueue.java:263)
14:47:26 	at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:173)
14:47:26 	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
14:47:26 	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
14:47:26 	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
14:47:26 	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
14:47:26 	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
14:47:26 	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
14:47:26 Caused by: com.oracle.svm.core.util.VMError$HostedError: missing implementation for runtime call: indexOf1Byte(byte[],int,int,byte)int
14:47:26 	at com.oracle.svm.core.util.VMError.shouldNotReachHere(VMError.java:68)
14:47:26 	at com.oracle.svm.core.graal.meta.SubstrateForeignCallsProvider.lookupForeignCall(SubstrateForeignCallsProvider.java:63)
14:47:26 	at com.oracle.svm.core.graal.meta.SubstrateForeignCallsProvider.getKilledLocations(SubstrateForeignCallsProvider.java:80)
14:47:26 	at jdk.internal.vm.compiler/org.graalvm.compiler.nodes.extended.ForeignCall.getKilledLocationIdentities(ForeignCall.java:60)
14:47:26 	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.common.FloatingReadPhase$FloatingReadClosure.processCheckpoint(FloatingReadPhase.java:379)
14:47:26 	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.common.FloatingReadPhase$FloatingReadClosure.processNode(FloatingReadPhase.java:335)
14:47:26 	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.common.FloatingReadPhase$FloatingReadClosure.processNode(FloatingReadPhase.java:296)
14:47:26 	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.graph.ReentrantNodeIterator.apply(ReentrantNodeIterator.java:119)
14:47:26 	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.graph.ReentrantNodeIterator.apply(ReentrantNodeIterator.java:102)
14:47:26 	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.common.FloatingReadPhase.run(FloatingReadPhase.java:228)
14:47:26 	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.run(Phase.java:49)
14:47:26 	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:214)
14:47:26 	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:147)
14:47:26 	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.PhaseSuite.run(PhaseSuite.java:209)
14:47:26 	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.common.IncrementalCanonicalizerPhase.run(IncrementalCanonicalizerPhase.java:56)
14:47:26 	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.common.IncrementalCanonicalizerPhase.run(IncrementalCanonicalizerPhase.java:38)
14:47:26 	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:214)
14:47:26 	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:147)
14:47:26 	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.PhaseSuite.run(PhaseSuite.java:209)
14:47:26 	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:214)
14:47:26 	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:147)
14:47:26 	at jdk.internal.vm.compiler/org.graalvm.compiler.core.GraalCompiler.emitFrontEnd(GraalCompiler.java:226)
14:47:26 	at jdk.internal.vm.compiler/org.graalvm.compiler.core.GraalCompiler.compile(GraalCompiler.java:145)
14:47:26 	at jdk.internal.vm.compiler/org.graalvm.compiler.core.GraalCompiler.compileGraph(GraalCompiler.java:130)
14:47:26 	at com.oracle.svm.hosted.code.CompileQueue.defaultCompileFunction(CompileQueue.java:983)
14:47:26 	... 9 more

Uh uh....#ThisShouldNotBeHappeningNow

ideas?

closed time in 5 days

Karm

push eventzakkak/mandrel-packaging

Foivos Zakkak

commit sha a81b16964c315184da6f873a009db514a403e0b1

Remove dependencies instead of building with --only

view details

push time in 5 days

create barnchzakkak/mandrel-packaging

branch : remove-deps-instead

created branch time in 5 days

push eventzakkak/mandrel-packaging

Foivos Zakkak

commit sha f5b5e86b523178f6d567b1fff32f3f5ea5426017

Fix and simplify maven parameter processing

view details

push time in 5 days

create barnchzakkak/mandrel-packaging

branch : simplify-maven-parameters

created branch time in 5 days

issue commentbeehive-lab/Maxine-VM

[RISCV64]Fedora(on QEMU) run test,FatalError: stack reset position two

@zdlgv5 I would use https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/tag/jdk8u212-b04 instead of jdk8u222 to make sure the versions match.

zdlgv5

comment created time in 5 days

issue commentbeehive-lab/Maxine-VM

[RISCV64]Fedora(on QEMU) run test,FatalError: stack reset position two

Oh sorry, I think I miss-read your comment, it looks like you are cross-compiling the image not the JDK itself. This should work @kotselidis, see https://github.com/beehive-lab/Maxine-VM-internal/blob/develop/docs/Working-with-RISCV-on-QEMU.rst#building-maxine-vm-for-risc-v.

zdlgv5

comment created time in 5 days

issue commentbeehive-lab/Maxine-VM

[RISCV64]Fedora(on QEMU) run test,FatalError: stack reset position two

Hi @zdlgv5, you don't need to cross-compile the JDK. Fedora comes with 8u212 in its repositories and this should work with maxine.

See http://fedora.riscv.rocks/koji/buildinfo?buildID=87633

https://github.com/beehive-lab/Maxine-VM-internal/blob/develop/docs/Status.rst looks wrong to me, it should say 212 for RISC-V not 222.

zdlgv5

comment created time in 5 days

push eventzakkak/mandrel

Foivos Zakkak

commit sha f07346366ca26fbe03b156eb35504016b4dd9b2c

Introduce --system-libffi to use system-provided libffi instead of building libffi from source

view details

push time in 6 days

pull request commentoracle/graal

Introduce --system-libffi to use system-provided libffi instead of building libffi from source

@gilles-duboscq can you please review this PR?

zakkak

comment created time in 6 days

PR opened oracle/graal

Introduce --system-libffi to use system-provided libffi instead of building libffi from source

Adding a --system-libffi flag as discussed in https://github.com/oracle/graal/issues/2502#issuecomment-680764625 (Re 2.)

+71 -64

0 comment

4 changed files

pr created time in 6 days

push eventzakkak/mandrel

Foivos Zakkak

commit sha 04eaf9b247157e40926952728f02beecf8668365

Introduce --system-libffi to use system-provided libffi instead of building libffi from source

view details

push time in 6 days

create barnchzakkak/mandrel

branch : system-libffi-flag

created branch time in 6 days

create barnchzakkak/jdk

branch : graal-shenandoah

created branch time in 6 days

fork zakkak/jdk

JDK main-line development

https://openjdk.java.net/projects/jdk/

fork in 6 days

issue openedgraalvm/mandrel-packaging

Find a way to get the correct JDK to build mandrel

Graal as well as Mandrel are tightly coupled to the OpenJDK used to build them. As we push out new releases it would be nice to be able to keep track of a map from Mandrel versions to OpenJDK versions so that in the future we can come back and build an older version using the OpenJDK that was available at the time.

Graal uses https://github.com/graalvm/mandrel/blob/graal/master/common.json to keep such info in the sources. This file gets updated as new JDK's become available etc.

Following a similar approach would enable us to checkout at any time any version and using mandrel-packaging build mandrel with the correct JDK version, e.g. if you try to build 20.1.0.2 it would use 11.0.8-GA but if you try to build say 20.1.0.5 it would use 11.0.9-GA etc.

We can even have something similar to graalvm with JAVA_HOME and EXTRA_JAVA_HOMES to define more than one JDK versions that are known to work with the corresponding mandrel version (or that we would like to test, although this won't work right in GH actions, but maybe Jenkins can take advantage of it)

created time in 6 days

Pull request review commentgraalvm/mandrel-packaging

Jbang the release process!

+//usr/bin/env jbang "$0" "$@" ; exit $?+//DEPS org.eclipse.jgit:org.eclipse.jgit:5.9.0.202009080501-r+//DEPS org.eclipse.jgit:org.eclipse.jgit.pgm:5.9.0.202009080501-r+//DEPS org.eclipse.jgit:org.eclipse.jgit.gpg.bc:5.9.0.202009080501-r+//DEPS info.picocli:picocli:4.5.0+//DEPS org.kohsuke:github-api:1.116++import org.eclipse.jgit.api.Git;+import org.eclipse.jgit.api.ResetCommand;+import org.eclipse.jgit.api.Status;+import org.eclipse.jgit.api.errors.GitAPIException;+import org.eclipse.jgit.api.errors.RefAlreadyExistsException;+import org.eclipse.jgit.console.ConsoleCredentialsProvider;+import org.eclipse.jgit.revwalk.RevCommit;+import org.eclipse.jgit.transport.URIish;+import org.kohsuke.github.*;+import picocli.CommandLine;+import picocli.CommandLine.Command;+import picocli.CommandLine.Help.Ansi;++import java.io.File;+import java.io.IOException;+import java.net.URISyntaxException;+import java.nio.charset.StandardCharsets;+import java.nio.file.Files;+import java.nio.file.Path;+import java.util.*;+import java.util.concurrent.Callable;+import java.util.function.Consumer;+import java.util.function.Function;+import java.util.regex.Matcher;+import java.util.regex.Pattern;+import java.util.stream.Collectors;+import java.util.stream.Stream;++@Command(name = "mandrel-release", mixinStandardHelpOptions = true,+        description = "Scipt automating part of the Mandrel release process")+class MandrelRelease implements Callable<Integer> {++    @CommandLine.Parameters(index = "0", description = "The kind of steps to execute, must be one of \"prepare\" or \"release\"")+    private String phase;++    @CommandLine.Option(names = {"-s", "--suffix"}, description = "The release suffix, e.g, Final, Alpha2, Beta1, etc. (default: \"${DEFAULT-VALUE}\")", defaultValue = "Final")+    private String suffix;++    @CommandLine.Option(names = {"-f", "--fork-name"}, description = "The repository name of the github fork to push the changes to (default: \"${DEFAULT-VALUE}\")", defaultValue = "zakkak/mandrel")+    private String forkName;++    @CommandLine.Option(names = {"-S", "--sign-commits"}, description = "Sign commits")+    private boolean signCommits;++    @CommandLine.Option(names = {"-D", "--dry-run"}, description = "Perform a dry run (no remote pushes and PRs)")+    private boolean dryRun;++    private static final String REMOTE_NAME = "mandrel-release-fork";+    private String version = null;+    private String fullVersion = null;+    private String releaseBranch = null;+    private String baseBranch = null;+    private String newVersion = null;+    private String newFullVersion = null;+    private String developBranch = null;++    public static void main(String... args) {+        int exitCode = new CommandLine(new MandrelRelease()).execute(args);+        System.exit(exitCode);+    }++    @Override+    public Integer call() {+        if (!suffix.equals("Final") && !Pattern.compile("^(Alpha|Beta)\\d*$").matcher(suffix).find()) {+            error("Invalid version suffix : " + suffix);+        }++        version = getCurrentVersion();+        fullVersion = version + "." + suffix;+        newVersion = getNewVersion(version);+        newFullVersion = newVersion + "." + suffix;+        releaseBranch = "release/mandrel-" + fullVersion;+        developBranch = "develop/mandrel-" + newVersion;+        final Matcher majorMinorMatcher = Pattern.compile("(\\d+\\.\\d+).*").matcher(version);+        if (!majorMinorMatcher.find()) {+            throw new RuntimeException("Version is in wrong format : " + version);+        }+        final String majorMinor = majorMinorMatcher.group(1);+        baseBranch = "mandrel/" + majorMinor;++        if (!phase.equals("prepare") && !phase.equals("release")) {+            throw new IllegalArgumentException(phase + " is not a valid phase. Please use \"prepare\" or \"release\".");+        }++        if (phase.equals("release")) {+            createGHRelease(fullVersion);+        }+        checkAndPrepareRepository();+        if (phase.equals("release")) {+            updateSuites(this::updateReleaseAndBumpVersionInSuite);+        } else {+            updateSuites(this::markSuiteAsRelease);+        }+        final String authorEmail = commitAndPushChanges();+        openPR(authorEmail);+        return 0;+    }++    private void checkAndPrepareRepository() {+        try (Git git = Git.open(new File("."))) {+            if (!git.getRepository().getBranch().equals(baseBranch)) {+                error("Please checkout " + baseBranch + " and try again!");+            }+            final Status status = git.status().call();+            if (status.hasUncommittedChanges() || !status.getChanged().isEmpty()) {+                error("Status of branch " + baseBranch + " is not clean, aborting!");+            }++            final String forkURL = "git@github.com:" + forkName;+            git.remoteAdd().setName(REMOTE_NAME).setUri(new URIish(forkURL)).call();+            info("Git remote " + REMOTE_NAME + " added and points to " + forkURL);+            final String newBranch = phase.equals("release") ? developBranch : releaseBranch;+            try {+                git.checkout().setCreateBranch(true).setName(newBranch).setStartPoint(newBranch).call();+                info("Created new branch " + newBranch + " based on " + baseBranch);+            } catch (RefAlreadyExistsException e) {+                warn(e.getMessage());+                gitCheckout(git, newBranch);+                git.reset().setRef(baseBranch).setMode(ResetCommand.ResetType.HARD).call();+                warn(newBranch + " reset (hard) to " + baseBranch);+            }+        } catch (IOException | GitAPIException | URISyntaxException e) {+            e.printStackTrace();+        }+    }++    /**+     * Visit all suite.py files and change the "release" and "version" fields+     *+     */+    private void updateSuites(Consumer<File> updater) {+        File cwd = new File(".");+        Stream.of(Objects.requireNonNull(cwd.listFiles()))+                .filter(File::isDirectory)+                .flatMap(path -> Stream.of(Objects.requireNonNull(path.listFiles())).filter(child -> child.getName().startsWith("mx.")))+                .map(path -> new File(path, "suite.py"))+                .filter(File::exists)+                .forEach(updater);+        info("Updated suites");+    }++    /**+     * Visit {@code suite} file and change the "release" value to {@code asRelease}+     *  @param suite+     *+     */+    private void markSuiteAsRelease(File suite) {+        try {+            info("Marking " + suite.getPath());+            List<String> lines = Files.readAllLines(suite.toPath());+            final String pattern = "(.*\"release\" : )False(.*)";+            final Pattern releasePattern = Pattern.compile(pattern);+            for (int i = 0; i < lines.size(); i++) {+                final Matcher releaseMatcher = releasePattern.matcher(lines.get(i));+                if (releaseMatcher.find()) {+                    String newLine = releaseMatcher.group(1) + "True" + releaseMatcher.group(2);+                    lines.set(i, newLine);+                    break;+                }+            }+            Files.write(suite.toPath(), lines, StandardCharsets.UTF_8);+        } catch (IOException e) {+            e.printStackTrace();+        }+    }++    private void updateReleaseAndBumpVersionInSuite(File suite) {+        try {+            info("Updating " + suite.getPath());+            List<String> lines = Files.readAllLines(suite.toPath());+            final Pattern releasePattern = Pattern.compile("(.*\"release\" : )True(.*)");+            final Pattern versionPattern = Pattern.compile("(.*\"version\" : \")"+version+"(\".*)");+            for (int i = 0; i < lines.size(); i++) {+                final Matcher releaseMatcher = releasePattern.matcher(lines.get(i));+                if (releaseMatcher.find()) {+                    String newLine = releaseMatcher.group(1) + "False" + releaseMatcher.group(2);+                    lines.set(i, newLine);+                }+                final Matcher versionMatcher = versionPattern.matcher(lines.get(i));+                if (versionMatcher.find()) {+                    String newLine = versionMatcher.group(1) + newVersion + versionMatcher.group(2);+                    lines.set(i, newLine);+                }+            }+            Files.write(suite.toPath(), lines, StandardCharsets.UTF_8);+        } catch (IOException e) {+            e.printStackTrace();+        }+    }++    private String commitAndPushChanges() {+        try (Git git = Git.open(new File("."))) {+            ConsoleCredentialsProvider.install(); // Needed for gpg signing+            final String message;+            if (phase.equals("release")) {+                message = "Unmark suites and bump version to " + newVersion;+            } else {+                message = "Mark suites for " + fullVersion + " release";+            }+            final RevCommit commit = git.commit().setAll(true).setMessage(message).setSign(signCommits).call();+            final String author = commit.getAuthorIdent().getEmailAddress();+            info("Changes commited");+            git.push().setForce(true).setRemote(REMOTE_NAME).setDryRun(dryRun).call();+            if (dryRun) {+                warn("Changes not pushed to remote due to --dry-run being present");+            } else {+                info("Changes pushed to remote " + REMOTE_NAME);+            }+            gitCheckout(git, baseBranch);+            return author;+        } catch (IOException | GitAPIException e) {+            e.printStackTrace();+            error(e.getMessage());+        }+        return null;+    }++    private void gitCheckout(Git git, String baseBranch) throws GitAPIException {+        git.checkout().setName(baseBranch).call();+        info("Checked out " + baseBranch);+    }++    private void openPR(String authorEmail) {+        GitHub github = connectToGitHub();+        try {+            final GHRepository repository = github.getRepository("graalvm/mandrel");+            if (dryRun) {+                warn("Pull request creation skipped due to --dry-run being present");+                return;+            }++            final String title;+            final String head;+            final String body;+            if (phase.equals("release")) {+                title = "Mark suites for " + fullVersion + " release";+                head = forkName.split("/")[0] + ":" + releaseBranch;+                body = "This PR was automatically generated by `mandrel-release.java` of graalvm/mandrel-packaging!\n\n" ++                        "Please tag branch `" + baseBranch + "` after merging, and push the tag.\n" ++                        "```\n" ++                        "git checkout mandrel/20.1\n" ++                        "git pull upstream mandrel/20.1\n" ++                        "git tag -a mandrel-" + fullVersion + " -m \"mandrel-" + fullVersion + "\" -s\n" ++                        "git push upstream mandrel-" + fullVersion + "\n" ++                        "```\n" ++                        "where `upstream` is a the git remote showing to https://github.com/graalvm/mandrel";+            } else {+                title = "Unmark suites and dump version to " + newVersion;+                head = forkName.split("/")[0] + ":" + developBranch;+                body = "This PR was automatically generated by `mandrel-release.java` of graalvm/mandrel-packaging!";+            }++            final GHPullRequest pullRequest = repository.createPullRequest(title, head, baseBranch, body, true, true);++            final GHUser galderz = github.getUser("galderz");+            final GHUser jerboaa = github.getUser("jerboaa");+            final GHUser karm = github.getUser("Karm");+            final GHUser zakkak = github.getUser("zakkak");++            ArrayList<GHUser> reviewers = new ArrayList<>();+            if (!authorEmail.contains("galder")) {+                reviewers.add(galderz);+            } else {+                pullRequest.addAssignees(Collections.singletonList(galderz));+            }+            if (!authorEmail.contains("sgehwolf") && !authorEmail.contains("jerboaa")) {+                reviewers.add(jerboaa);+            } else {+                pullRequest.addAssignees(Collections.singletonList(jerboaa));+            }+            if (!authorEmail.contains("karm")) {+                reviewers.add(karm);+            } else {+                pullRequest.addAssignees(Collections.singletonList(karm));+            }+            if (!authorEmail.contains("zakkak")) {+                reviewers.add(zakkak);+            } else {+                pullRequest.addAssignees(Collections.singletonList(zakkak));+            }+            pullRequest.requestReviewers(reviewers);++            info("Pull request " + pullRequest.getHtmlUrl() + " created");+        } catch (IOException e) {+            error(e.getMessage());+        }+    }++    /**+     * Returns current version of substratevm+     *+     * @return+     */+    private static String getCurrentVersion() {+        final Path substrateSuite = Path.of("substratevm", "mx.substratevm", "suite.py");+        String version = null;+        try {+            final List<String> lines = Files.readAllLines(substrateSuite);+            final Pattern versionPattern = Pattern.compile("\"version\" : \"([\\d.]+)\"");+            for (String line: lines) {+                final Matcher versionMatcher = versionPattern.matcher(line);+                if (versionMatcher.find()) {+                    version = versionMatcher.group(1);+                    break;+                }+            }+        } catch (IOException e) {+            e.printStackTrace();+        }+        info("Current version is " + version);+        return version;+    }++    /**+     * Calculates the new version based on {@code version} by bumping pico in major.minor.micro.pico+     *+     * @param version The current version+     * @return The new version+     */+    private static String getNewVersion(String version) {+        final Pattern versionPattern = Pattern.compile("(\\d+\\.\\d+\\.\\d+\\.)(\\d+)");+        final Matcher versionMatcher = versionPattern.matcher(version);+        boolean found = versionMatcher.find();+        assert found;+        int pico = Integer.parseInt(versionMatcher.group(2)) + 1;+        final String newVersion = versionMatcher.group(1) + pico;+        info("New version will be " + newVersion);+        return newVersion;+    }++    private void createGHRelease(String fullVersion) {+        GitHub github = connectToGitHub();+        try {+            final GHRepository repository = github.getRepository("graalvm/mandrel");+            final PagedIterable<GHMilestone> ghMilestones = repository.listMilestones(GHIssueState.OPEN);+            final GHMilestone milestone = ghMilestones.toList().stream().filter(m -> m.getTitle().equals(fullVersion)).findAny().orElse(null);+            String changelog = createChangelog(fullVersion, repository, milestone);+            if (dryRun) {+                warn("Skipping release due to --dry-run");+                info("Release body would look like");+                System.out.println(releaseMainBody(fullVersion, changelog));+                return;+            }+            final GHRelease ghRelease = repository.createRelease("mandrel-" + fullVersion)+                    .name("Mandrel " + fullVersion)+                    .prerelease(!suffix.equals("Final"))+                    .body(releaseMainBody(fullVersion, changelog))+                    .draft(true)+                    .create();+            uploadAssets(fullVersion, ghRelease);+            info("Created new draft release: " + ghRelease.getHtmlUrl());+            info("Please review and publish!");+            manageMilestones(repository, ghMilestones, milestone);+        } catch (IOException e) {+            e.printStackTrace();+        }+    }++    private void uploadAssets(String fullVersion, GHRelease ghRelease) throws IOException {+        final File archive = new File("mandrel-java11-linux-amd64-" + fullVersion + ".tar.gz");+        final File archiveSHA1 = new File("mandrel-java11-linux-amd64-" + fullVersion + ".tar.gz.sha1");+        if (!archive.exists()) {+            warn("Archive \"" + archive.getName() + "\" was not found. Skipping asset upload.");+            warn("Please upload asset manually.");+        } else if (!archiveSHA1.exists()) {+            warn("Archive sha1 \"" + archive.getName() + "\" was not found. Skipping asset upload.");+            warn("Please upload asset manually.");+        } else {+            info("Uploading " + archive.getName());+            ghRelease.uploadAsset(archive, "application/gzip");+            info("Uploaded " + archive.getName());+            info("Uploading " + archiveSHA1.getName());+            ghRelease.uploadAsset(archiveSHA1, "text/plain");+            info("Uploaded " + archiveSHA1.getName());+        }+    }++    private String releaseMainBody(String version, String changelog) {+        final Matcher majorMinorMicroMatcher = Pattern.compile("(\\d+\\.\\d+\\.\\d+).*").matcher(version);+        if (!majorMinorMicroMatcher.find()) {+            throw new RuntimeException("Version is in wrong format : " + version);+        }+        final String majorMinorMicro = majorMinorMicroMatcher.group(1);+        return "# Mandrel\n" ++                "\n" ++                "Mandrel " + version + " is a downstream distribution of the [GraalVM community edition " + majorMinorMicro + "](https://github.com/graalvm/graalvm-ce-builds/releases/tag/vm-" + majorMinorMicro + ").\n" ++                "Mandrel's main goal is to provide a `native-image` release specifically to support [Quarkus](https://quarkus.io).\n" ++                "The aim is to align the `native-image` capabilities from GraalVM with OpenJDK and Red Hat Enterprise Linux libraries to improve maintainability for native Quarkus applications.\n" ++                "\n" ++                "## How Does Mandrel Differ From Graal\n" ++                "\n" ++                "Mandrel releases are built from a code base derived from the upstream GraalVM code base, with only minor changes but some significant exclusions.\n" ++                "They support the same native image capability as GraalVM with no significant changes to functionality.\n" ++                "They do not include support for Polyglot programming via the Truffle interpreter and compiler framework.\n" ++                "In consequence, it is not possible to extend Mandrel by downloading languages from the Truffle language catalogue.\n" ++                "\n" ++                "Mandrel is also built slightly differently to GraalVM, using the standard OpenJDK project release of jdk11u.\n" ++                "This means it does not profit from a few small enhancements that Oracle have added to the version of OpenJDK used to build their own GraalVM downloads.\n" ++                "Most of these enhancements are to the JVMCI module that allows the Graal compiler to be run inside OpenJDK.\n" ++                "The others are small cosmetic changes to behaviour.\n" ++                "These enhancements may in some cases cause minor differences in the progress of native image generation.\n" ++                "They should not cause the resulting images themselves to execute in a noticeably different manner.\n" ++                "\n" ++                "### Prerequisites\n" ++                "\n" ++                "Mandrel's `native-image` depends on the following packages:\n" ++                "* glibc-devel\n" ++                "* zlib-devel\n" ++                "* gcc\n" ++                "* libffi-devel\n" ++                "\n" ++                "On Fedora/CentOS/RHEL they can be installed with:\n" ++                "```bash\n" ++                "dnf install glibc-devel zlib-devel gcc libffi-devel libstdc++-static\n" ++                "```\n" ++                "\n" ++                "Note the package might be called `glibc-static` instead of `libstdc++-static`.\n" ++                "\n" ++                "On Ubuntu-like systems with:\n" ++                "```bash\n" ++                "apt install gcc zlib1g-dev libffi-dev build-essential\n" ++                "```\n" ++                "\n" ++                "## Quick start\n" ++                "\n" ++                "```\n" ++                "$ tar -xf mandrel-java11-linux-amd64-" + version + ".tar.gz\n" ++                "$ export JAVA_HOME=\"$( pwd )/mandrel-java11-" + version + "\"\n" ++                "$ export GRAALVM_HOME=\"${JAVA_HOME}\"\n" ++                "$ export PATH=\"${JAVA_HOME}/bin:${PATH}\"\n" ++                "$ curl -O -J  https://code.quarkus.io/api/download\n" ++                "$ unzip code-with-quarkus.zip\n" ++                "$ cd code-with-quarkus/\n" ++                "$ ./mvnw package -Pnative\n" ++                "$ ./target/code-with-quarkus-1.0.0-SNAPSHOT-runner\n" ++                "```\n" ++                "\n" ++                "### Quarkus builder image\n" ++                "\n" ++                "The Quarkus builder image for this release is still being prepared, please try again later.\n" ++                "<!--\n" ++                "Mandrel Quarkus builder image can be used to build a Quarkus native Linux executable right away without any GRAALVM_HOME setup.\n" ++                "\n" ++                "```bash\n" ++                "curl -O -J  https://code.quarkus.io/api/download\n" ++                "unzip code-with-quarkus.zip\n" ++                "cd code-with-quarkus\n" ++                "        ./mvnw package -Pnative -Dquarkus.native.container-build=true -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-mandrel:" + version + "-java11\n" ++                "        ./target/code-with-quarkus-1.0.0-SNAPSHOT-runner\n" ++                "```\n" ++                "\n" ++                "One can use the builder image on Windows with Docker Desktop (mind `Resources-> File sharing` settings so as Quarkus project directory is mountable).\n" ++                "\n" ++                "```batchfile\n" ++                "powershell -c \"Invoke-WebRequest -OutFile quarkus.zip -Uri https://code.quarkus.io/api/download\"\n" ++                "powershell -c \"Expand-Archive -Path quarkus.zip -DestinationPath . -Force\n" ++                "cd code-with-quarkus\n" ++                "mvnw package -Pnative -Dquarkus.native.container-build=true -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-mandrel:" + version + "-java11\n" ++                "docker build -f src/main/docker/Dockerfile.native -t my-quarkus-mandrel-app .\n" ++                "        docker run -i --rm -p 8080:8080 my-quarkus-mandrel-app\n" ++                "```\n" ++                "-->\n" ++                changelog ++                "\n---\n" ++                "Mandrel " + version + "\n" ++                "OpenJDK used: 11.0.8+10\n";

I implemented it to get the version from the underlying runtime for now and we can discuss the "file approach" in another issue.

zakkak

comment created time in 6 days

PullRequestReviewEvent

push eventzakkak/mandrel-packaging

Foivos Zakkak

commit sha d10b10f46ed23ea13c933bac31118ec4d6e50e0f

Jbang the release process!

view details

push time in 6 days

Pull request review commentgraalvm/mandrel-packaging

Jbang the release process!

+//usr/bin/env jbang "$0" "$@" ; exit $?+//DEPS org.eclipse.jgit:org.eclipse.jgit:5.9.0.202009080501-r+//DEPS org.eclipse.jgit:org.eclipse.jgit.pgm:5.9.0.202009080501-r+//DEPS org.eclipse.jgit:org.eclipse.jgit.gpg.bc:5.9.0.202009080501-r+//DEPS info.picocli:picocli:4.5.0+//DEPS org.kohsuke:github-api:1.116++import org.eclipse.jgit.api.Git;+import org.eclipse.jgit.api.ResetCommand;+import org.eclipse.jgit.api.Status;+import org.eclipse.jgit.api.errors.GitAPIException;+import org.eclipse.jgit.api.errors.RefAlreadyExistsException;+import org.eclipse.jgit.console.ConsoleCredentialsProvider;+import org.eclipse.jgit.revwalk.RevCommit;+import org.eclipse.jgit.transport.URIish;+import org.kohsuke.github.*;+import picocli.CommandLine;+import picocli.CommandLine.Command;+import picocli.CommandLine.Help.Ansi;++import java.io.File;+import java.io.IOException;+import java.net.URISyntaxException;+import java.nio.charset.StandardCharsets;+import java.nio.file.Files;+import java.nio.file.Path;+import java.util.*;+import java.util.concurrent.Callable;+import java.util.function.Consumer;+import java.util.function.Function;+import java.util.regex.Matcher;+import java.util.regex.Pattern;+import java.util.stream.Collectors;+import java.util.stream.Stream;++@Command(name = "mandrel-release", mixinStandardHelpOptions = true,+        description = "Scipt automating part of the Mandrel release process")+class MandrelRelease implements Callable<Integer> {++    @CommandLine.Parameters(index = "0", description = "The kind of steps to execute, must be one of \"prepare\" or \"release\"")+    private String phase;++    @CommandLine.Option(names = {"-s", "--suffix"}, description = "The release suffix, e.g, Final, Alpha2, Beta1, etc. (default: \"${DEFAULT-VALUE}\")", defaultValue = "Final")+    private String suffix;++    @CommandLine.Option(names = {"-f", "--fork-name"}, description = "The repository name of the github fork to push the changes to (default: \"${DEFAULT-VALUE}\")", defaultValue = "zakkak/mandrel")+    private String forkName;++    @CommandLine.Option(names = {"-S", "--sign-commits"}, description = "Sign commits")+    private boolean signCommits;++    @CommandLine.Option(names = {"-D", "--dry-run"}, description = "Perform a dry run (no remote pushes and PRs)")+    private boolean dryRun;++    private static final String REMOTE_NAME = "mandrel-release-fork";+    private String version = null;+    private String fullVersion = null;+    private String releaseBranch = null;+    private String baseBranch = null;+    private String newVersion = null;+    private String newFullVersion = null;+    private String developBranch = null;++    public static void main(String... args) {+        int exitCode = new CommandLine(new MandrelRelease()).execute(args);+        System.exit(exitCode);+    }++    @Override+    public Integer call() {+        if (!suffix.equals("Final") && !Pattern.compile("^(Alpha|Beta)\\d*$").matcher(suffix).find()) {+            error("Invalid version suffix : " + suffix);+        }++        version = getCurrentVersion();+        fullVersion = version + "." + suffix;+        newVersion = getNewVersion(version);+        newFullVersion = newVersion + "." + suffix;+        releaseBranch = "release/mandrel-" + fullVersion;+        developBranch = "develop/mandrel-" + newVersion;+        final Matcher majorMinorMatcher = Pattern.compile("(\\d+\\.\\d+).*").matcher(version);+        if (!majorMinorMatcher.find()) {+            throw new RuntimeException("Version is in wrong format : " + version);+        }+        final String majorMinor = majorMinorMatcher.group(1);+        baseBranch = "mandrel/" + majorMinor;++        if (!phase.equals("prepare") && !phase.equals("release")) {+            throw new IllegalArgumentException(phase + " is not a valid phase. Please use \"prepare\" or \"release\".");+        }++        if (phase.equals("release")) {+            createGHRelease(fullVersion);+        }+        checkAndPrepareRepository();+        if (phase.equals("release")) {+            updateSuites(this::updateReleaseAndBumpVersionInSuite);+        } else {+            updateSuites(this::markSuiteAsRelease);+        }+        final String authorEmail = commitAndPushChanges();+        openPR(authorEmail);+        return 0;+    }++    private void checkAndPrepareRepository() {+        try (Git git = Git.open(new File("."))) {+            if (!git.getRepository().getBranch().equals(baseBranch)) {+                error("Please checkout " + baseBranch + " and try again!");+            }+            final Status status = git.status().call();+            if (status.hasUncommittedChanges() || !status.getChanged().isEmpty()) {+                error("Status of branch " + baseBranch + " is not clean, aborting!");+            }++            final String forkURL = "git@github.com:" + forkName;+            git.remoteAdd().setName(REMOTE_NAME).setUri(new URIish(forkURL)).call();+            info("Git remote " + REMOTE_NAME + " added and points to " + forkURL);+            final String newBranch = phase.equals("release") ? developBranch : releaseBranch;+            try {+                git.checkout().setCreateBranch(true).setName(newBranch).setStartPoint(newBranch).call();+                info("Created new branch " + newBranch + " based on " + baseBranch);+            } catch (RefAlreadyExistsException e) {+                warn(e.getMessage());+                gitCheckout(git, newBranch);+                git.reset().setRef(baseBranch).setMode(ResetCommand.ResetType.HARD).call();+                warn(newBranch + " reset (hard) to " + baseBranch);+            }+        } catch (IOException | GitAPIException | URISyntaxException e) {+            e.printStackTrace();+        }+    }++    /**+     * Visit all suite.py files and change the "release" and "version" fields+     *+     */+    private void updateSuites(Consumer<File> updater) {+        File cwd = new File(".");+        Stream.of(Objects.requireNonNull(cwd.listFiles()))+                .filter(File::isDirectory)+                .flatMap(path -> Stream.of(Objects.requireNonNull(path.listFiles())).filter(child -> child.getName().startsWith("mx.")))+                .map(path -> new File(path, "suite.py"))+                .filter(File::exists)+                .forEach(updater);+        info("Updated suites");+    }++    /**+     * Visit {@code suite} file and change the "release" value to {@code asRelease}+     *  @param suite+     *+     */+    private void markSuiteAsRelease(File suite) {+        try {+            info("Marking " + suite.getPath());+            List<String> lines = Files.readAllLines(suite.toPath());+            final String pattern = "(.*\"release\" : )False(.*)";+            final Pattern releasePattern = Pattern.compile(pattern);+            for (int i = 0; i < lines.size(); i++) {+                final Matcher releaseMatcher = releasePattern.matcher(lines.get(i));+                if (releaseMatcher.find()) {+                    String newLine = releaseMatcher.group(1) + "True" + releaseMatcher.group(2);+                    lines.set(i, newLine);+                    break;+                }+            }+            Files.write(suite.toPath(), lines, StandardCharsets.UTF_8);+        } catch (IOException e) {+            e.printStackTrace();+        }+    }++    private void updateReleaseAndBumpVersionInSuite(File suite) {+        try {+            info("Updating " + suite.getPath());+            List<String> lines = Files.readAllLines(suite.toPath());+            final Pattern releasePattern = Pattern.compile("(.*\"release\" : )True(.*)");+            final Pattern versionPattern = Pattern.compile("(.*\"version\" : \")"+version+"(\".*)");+            for (int i = 0; i < lines.size(); i++) {+                final Matcher releaseMatcher = releasePattern.matcher(lines.get(i));+                if (releaseMatcher.find()) {+                    String newLine = releaseMatcher.group(1) + "False" + releaseMatcher.group(2);+                    lines.set(i, newLine);+                }+                final Matcher versionMatcher = versionPattern.matcher(lines.get(i));+                if (versionMatcher.find()) {+                    String newLine = versionMatcher.group(1) + newVersion + versionMatcher.group(2);+                    lines.set(i, newLine);+                }+            }+            Files.write(suite.toPath(), lines, StandardCharsets.UTF_8);+        } catch (IOException e) {+            e.printStackTrace();+        }+    }++    private String commitAndPushChanges() {+        try (Git git = Git.open(new File("."))) {+            ConsoleCredentialsProvider.install(); // Needed for gpg signing+            final String message;+            if (phase.equals("release")) {+                message = "Unmark suites and bump version to " + newVersion;+            } else {+                message = "Mark suites for " + fullVersion + " release";+            }+            final RevCommit commit = git.commit().setAll(true).setMessage(message).setSign(signCommits).call();+            final String author = commit.getAuthorIdent().getEmailAddress();+            info("Changes commited");+            git.push().setForce(true).setRemote(REMOTE_NAME).setDryRun(dryRun).call();+            if (dryRun) {+                warn("Changes not pushed to remote due to --dry-run being present");+            } else {+                info("Changes pushed to remote " + REMOTE_NAME);+            }+            gitCheckout(git, baseBranch);+            return author;+        } catch (IOException | GitAPIException e) {+            e.printStackTrace();+            error(e.getMessage());+        }+        return null;+    }++    private void gitCheckout(Git git, String baseBranch) throws GitAPIException {+        git.checkout().setName(baseBranch).call();+        info("Checked out " + baseBranch);+    }++    private void openPR(String authorEmail) {+        GitHub github = connectToGitHub();+        try {+            final GHRepository repository = github.getRepository("graalvm/mandrel");+            if (dryRun) {+                warn("Pull request creation skipped due to --dry-run being present");+                return;+            }++            final String title;+            final String head;+            final String body;+            if (phase.equals("release")) {+                title = "Mark suites for " + fullVersion + " release";+                head = forkName.split("/")[0] + ":" + releaseBranch;+                body = "This PR was automatically generated by `mandrel-release.java` of graalvm/mandrel-packaging!\n\n" ++                        "Please tag branch `" + baseBranch + "` after merging, and push the tag.\n" ++                        "```\n" ++                        "git checkout mandrel/20.1\n" ++                        "git pull upstream mandrel/20.1\n" ++                        "git tag -a mandrel-" + fullVersion + " -m \"mandrel-" + fullVersion + "\" -s\n" ++                        "git push upstream mandrel-" + fullVersion + "\n" ++                        "```\n" ++                        "where `upstream` is a the git remote showing to https://github.com/graalvm/mandrel";+            } else {+                title = "Unmark suites and dump version to " + newVersion;+                head = forkName.split("/")[0] + ":" + developBranch;+                body = "This PR was automatically generated by `mandrel-release.java` of graalvm/mandrel-packaging!";+            }++            final GHPullRequest pullRequest = repository.createPullRequest(title, head, baseBranch, body, true, true);++            final GHUser galderz = github.getUser("galderz");+            final GHUser jerboaa = github.getUser("jerboaa");+            final GHUser karm = github.getUser("Karm");+            final GHUser zakkak = github.getUser("zakkak");++            ArrayList<GHUser> reviewers = new ArrayList<>();+            if (!authorEmail.contains("galder")) {+                reviewers.add(galderz);+            } else {+                pullRequest.addAssignees(Collections.singletonList(galderz));+            }+            if (!authorEmail.contains("sgehwolf") && !authorEmail.contains("jerboaa")) {+                reviewers.add(jerboaa);+            } else {+                pullRequest.addAssignees(Collections.singletonList(jerboaa));+            }+            if (!authorEmail.contains("karm")) {+                reviewers.add(karm);+            } else {+                pullRequest.addAssignees(Collections.singletonList(karm));+            }+            if (!authorEmail.contains("zakkak")) {+                reviewers.add(zakkak);+            } else {+                pullRequest.addAssignees(Collections.singletonList(zakkak));+            }+            pullRequest.requestReviewers(reviewers);++            info("Pull request " + pullRequest.getHtmlUrl() + " created");+        } catch (IOException e) {+            error(e.getMessage());+        }+    }++    /**+     * Returns current version of substratevm+     *+     * @return+     */+    private static String getCurrentVersion() {+        final Path substrateSuite = Path.of("substratevm", "mx.substratevm", "suite.py");+        String version = null;+        try {+            final List<String> lines = Files.readAllLines(substrateSuite);+            final Pattern versionPattern = Pattern.compile("\"version\" : \"([\\d.]+)\"");+            for (String line: lines) {+                final Matcher versionMatcher = versionPattern.matcher(line);+                if (versionMatcher.find()) {+                    version = versionMatcher.group(1);+                    break;+                }+            }+        } catch (IOException e) {+            e.printStackTrace();+        }+        info("Current version is " + version);+        return version;+    }++    /**+     * Calculates the new version based on {@code version} by bumping pico in major.minor.micro.pico+     *+     * @param version The current version+     * @return The new version+     */+    private static String getNewVersion(String version) {+        final Pattern versionPattern = Pattern.compile("(\\d+\\.\\d+\\.\\d+\\.)(\\d+)");+        final Matcher versionMatcher = versionPattern.matcher(version);+        boolean found = versionMatcher.find();+        assert found;+        int pico = Integer.parseInt(versionMatcher.group(2)) + 1;+        final String newVersion = versionMatcher.group(1) + pico;+        info("New version will be " + newVersion);+        return newVersion;+    }++    private void createGHRelease(String fullVersion) {+        GitHub github = connectToGitHub();+        try {+            final GHRepository repository = github.getRepository("graalvm/mandrel");+            final PagedIterable<GHMilestone> ghMilestones = repository.listMilestones(GHIssueState.OPEN);+            final GHMilestone milestone = ghMilestones.toList().stream().filter(m -> m.getTitle().equals(fullVersion)).findAny().orElse(null);+            String changelog = createChangelog(fullVersion, repository, milestone);+            if (dryRun) {+                warn("Skipping release due to --dry-run");+                info("Release body would look like");+                System.out.println(releaseMainBody(fullVersion, changelog));+                return;+            }+            final GHRelease ghRelease = repository.createRelease("mandrel-" + fullVersion)+                    .name("Mandrel " + fullVersion)+                    .prerelease(!suffix.equals("Final"))+                    .body(releaseMainBody(fullVersion, changelog))+                    .draft(true)+                    .create();+            uploadAssets(fullVersion, ghRelease);+            info("Created new draft release: " + ghRelease.getHtmlUrl());+            info("Please review and publish!");+            manageMilestones(repository, ghMilestones, milestone);+        } catch (IOException e) {+            e.printStackTrace();+        }+    }++    private void uploadAssets(String fullVersion, GHRelease ghRelease) throws IOException {+        final File archive = new File("mandrel-java11-linux-amd64-" + fullVersion + ".tar.gz");+        final File archiveSHA1 = new File("mandrel-java11-linux-amd64-" + fullVersion + ".tar.gz.sha1");+        if (!archive.exists()) {+            warn("Archive \"" + archive.getName() + "\" was not found. Skipping asset upload.");+            warn("Please upload asset manually.");+        } else if (!archiveSHA1.exists()) {+            warn("Archive sha1 \"" + archive.getName() + "\" was not found. Skipping asset upload.");+            warn("Please upload asset manually.");+        } else {+            info("Uploading " + archive.getName());+            ghRelease.uploadAsset(archive, "application/gzip");+            info("Uploaded " + archive.getName());+            info("Uploading " + archiveSHA1.getName());+            ghRelease.uploadAsset(archiveSHA1, "text/plain");+            info("Uploaded " + archiveSHA1.getName());+        }+    }++    private String releaseMainBody(String version, String changelog) {+        final Matcher majorMinorMicroMatcher = Pattern.compile("(\\d+\\.\\d+\\.\\d+).*").matcher(version);+        if (!majorMinorMicroMatcher.find()) {+            throw new RuntimeException("Version is in wrong format : " + version);+        }+        final String majorMinorMicro = majorMinorMicroMatcher.group(1);+        return "# Mandrel\n" ++                "\n" ++                "Mandrel " + version + " is a downstream distribution of the [GraalVM community edition " + majorMinorMicro + "](https://github.com/graalvm/graalvm-ce-builds/releases/tag/vm-" + majorMinorMicro + ").\n" ++                "Mandrel's main goal is to provide a `native-image` release specifically to support [Quarkus](https://quarkus.io).\n" ++                "The aim is to align the `native-image` capabilities from GraalVM with OpenJDK and Red Hat Enterprise Linux libraries to improve maintainability for native Quarkus applications.\n" ++                "\n" ++                "## How Does Mandrel Differ From Graal\n" ++                "\n" ++                "Mandrel releases are built from a code base derived from the upstream GraalVM code base, with only minor changes but some significant exclusions.\n" ++                "They support the same native image capability as GraalVM with no significant changes to functionality.\n" ++                "They do not include support for Polyglot programming via the Truffle interpreter and compiler framework.\n" ++                "In consequence, it is not possible to extend Mandrel by downloading languages from the Truffle language catalogue.\n" ++                "\n" ++                "Mandrel is also built slightly differently to GraalVM, using the standard OpenJDK project release of jdk11u.\n" ++                "This means it does not profit from a few small enhancements that Oracle have added to the version of OpenJDK used to build their own GraalVM downloads.\n" ++                "Most of these enhancements are to the JVMCI module that allows the Graal compiler to be run inside OpenJDK.\n" ++                "The others are small cosmetic changes to behaviour.\n" ++                "These enhancements may in some cases cause minor differences in the progress of native image generation.\n" ++                "They should not cause the resulting images themselves to execute in a noticeably different manner.\n" ++                "\n" ++                "### Prerequisites\n" ++                "\n" ++                "Mandrel's `native-image` depends on the following packages:\n" ++                "* glibc-devel\n" ++                "* zlib-devel\n" ++                "* gcc\n" ++                "* libffi-devel\n" ++                "\n" ++                "On Fedora/CentOS/RHEL they can be installed with:\n" ++                "```bash\n" ++                "dnf install glibc-devel zlib-devel gcc libffi-devel libstdc++-static\n" ++                "```\n" ++                "\n" ++                "Note the package might be called `glibc-static` instead of `libstdc++-static`.\n" ++                "\n" ++                "On Ubuntu-like systems with:\n" ++                "```bash\n" ++                "apt install gcc zlib1g-dev libffi-dev build-essential\n" ++                "```\n" ++                "\n" ++                "## Quick start\n" ++                "\n" ++                "```\n" ++                "$ tar -xf mandrel-java11-linux-amd64-" + version + ".tar.gz\n" ++                "$ export JAVA_HOME=\"$( pwd )/mandrel-java11-" + version + "\"\n" ++                "$ export GRAALVM_HOME=\"${JAVA_HOME}\"\n" ++                "$ export PATH=\"${JAVA_HOME}/bin:${PATH}\"\n" ++                "$ curl -O -J  https://code.quarkus.io/api/download\n" ++                "$ unzip code-with-quarkus.zip\n" ++                "$ cd code-with-quarkus/\n" ++                "$ ./mvnw package -Pnative\n" ++                "$ ./target/code-with-quarkus-1.0.0-SNAPSHOT-runner\n" ++                "```\n" ++                "\n" ++                "### Quarkus builder image\n" ++                "\n" ++                "The Quarkus builder image for this release is still being prepared, please try again later.\n" ++                "<!--\n" ++                "Mandrel Quarkus builder image can be used to build a Quarkus native Linux executable right away without any GRAALVM_HOME setup.\n" ++                "\n" ++                "```bash\n" ++                "curl -O -J  https://code.quarkus.io/api/download\n" ++                "unzip code-with-quarkus.zip\n" ++                "cd code-with-quarkus\n" ++                "        ./mvnw package -Pnative -Dquarkus.native.container-build=true -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-mandrel:" + version + "-java11\n" ++                "        ./target/code-with-quarkus-1.0.0-SNAPSHOT-runner\n" ++                "```\n" ++                "\n" ++                "One can use the builder image on Windows with Docker Desktop (mind `Resources-> File sharing` settings so as Quarkus project directory is mountable).\n" ++                "\n" ++                "```batchfile\n" ++                "powershell -c \"Invoke-WebRequest -OutFile quarkus.zip -Uri https://code.quarkus.io/api/download\"\n" ++                "powershell -c \"Expand-Archive -Path quarkus.zip -DestinationPath . -Force\n" ++                "cd code-with-quarkus\n" ++                "mvnw package -Pnative -Dquarkus.native.container-build=true -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-mandrel:" + version + "-java11\n" ++                "docker build -f src/main/docker/Dockerfile.native -t my-quarkus-mandrel-app .\n" ++                "        docker run -i --rm -p 8080:8080 my-quarkus-mandrel-app\n" ++                "```\n" ++                "-->\n" ++                changelog ++                "\n---\n" ++                "Mandrel " + version + "\n" ++                "OpenJDK used: 11.0.8+10\n";

I am more afraid about issues like "I built mandrel with JDK-A and then ran jbang with JDK-B" and the release readme ends up wrong.

Also what are your thoughts about the file/configuration specifying the correct JDK to use?

This would allow you to checkout at any time versionA and using mandrel-packaging build mandrel with the correct JDK version, e.g. if you try to build 20.1.0.2 it uses 11.0.8-GA but if you try to build say 20.1.0.5 it would use 11.0.9-GA etc. And we can even have something similar to graalvm with JAVA_HOME and EXTRA_JAVA_HOMES to define more than one JDK versions that are known to work with the corresponding mandrel version (or that we would like to test, allthough this won't work right in GH actions, but maybe Jenkins can take advantage of it)

zakkak

comment created time in 6 days

PullRequestReviewEvent

Pull request review commentgraalvm/mandrel-packaging

Jbang the release process!

+//usr/bin/env jbang "$0" "$@" ; exit $?+//DEPS org.eclipse.jgit:org.eclipse.jgit:5.9.0.202009080501-r+//DEPS org.eclipse.jgit:org.eclipse.jgit.pgm:5.9.0.202009080501-r+//DEPS org.eclipse.jgit:org.eclipse.jgit.gpg.bc:5.9.0.202009080501-r+//DEPS info.picocli:picocli:4.5.0+//DEPS org.kohsuke:github-api:1.116++import org.eclipse.jgit.api.Git;+import org.eclipse.jgit.api.ResetCommand;+import org.eclipse.jgit.api.Status;+import org.eclipse.jgit.api.errors.GitAPIException;+import org.eclipse.jgit.api.errors.RefAlreadyExistsException;+import org.eclipse.jgit.console.ConsoleCredentialsProvider;+import org.eclipse.jgit.revwalk.RevCommit;+import org.eclipse.jgit.transport.URIish;+import org.kohsuke.github.*;+import picocli.CommandLine;+import picocli.CommandLine.Command;+import picocli.CommandLine.Help.Ansi;++import java.io.File;+import java.io.IOException;+import java.net.URISyntaxException;+import java.nio.charset.StandardCharsets;+import java.nio.file.Files;+import java.nio.file.Path;+import java.util.*;+import java.util.concurrent.Callable;+import java.util.function.Consumer;+import java.util.function.Function;+import java.util.regex.Matcher;+import java.util.regex.Pattern;+import java.util.stream.Collectors;+import java.util.stream.Stream;++@Command(name = "mandrel-release", mixinStandardHelpOptions = true,+        description = "Scipt automating part of the Mandrel release process")+class MandrelRelease implements Callable<Integer> {++    @CommandLine.Parameters(index = "0", description = "The kind of steps to execute, must be one of \"prepare\" or \"release\"")+    private String phase;++    @CommandLine.Option(names = {"-s", "--suffix"}, description = "The release suffix, e.g, Final, Alpha2, Beta1, etc. (default: \"${DEFAULT-VALUE}\")", defaultValue = "Final")+    private String suffix;++    @CommandLine.Option(names = {"-f", "--fork-name"}, description = "The repository name of the github fork to push the changes to (default: \"${DEFAULT-VALUE}\")", defaultValue = "zakkak/mandrel")+    private String forkName;++    @CommandLine.Option(names = {"-S", "--sign-commits"}, description = "Sign commits")+    private boolean signCommits;++    @CommandLine.Option(names = {"-D", "--dry-run"}, description = "Perform a dry run (no remote pushes and PRs)")+    private boolean dryRun;++    private static final String REMOTE_NAME = "mandrel-release-fork";+    private String version = null;+    private String fullVersion = null;+    private String releaseBranch = null;+    private String baseBranch = null;+    private String newVersion = null;+    private String newFullVersion = null;+    private String developBranch = null;++    public static void main(String... args) {+        int exitCode = new CommandLine(new MandrelRelease()).execute(args);+        System.exit(exitCode);+    }++    @Override+    public Integer call() {+        if (!suffix.equals("Final") && !Pattern.compile("^(Alpha|Beta)\\d*$").matcher(suffix).find()) {+            error("Invalid version suffix : " + suffix);+        }++        version = getCurrentVersion();+        fullVersion = version + "." + suffix;+        newVersion = getNewVersion(version);+        newFullVersion = newVersion + "." + suffix;+        releaseBranch = "release/mandrel-" + fullVersion;+        developBranch = "develop/mandrel-" + newVersion;+        final Matcher majorMinorMatcher = Pattern.compile("(\\d+\\.\\d+).*").matcher(version);+        if (!majorMinorMatcher.find()) {+            throw new RuntimeException("Version is in wrong format : " + version);+        }+        final String majorMinor = majorMinorMatcher.group(1);+        baseBranch = "mandrel/" + majorMinor;++        if (!phase.equals("prepare") && !phase.equals("release")) {+            throw new IllegalArgumentException(phase + " is not a valid phase. Please use \"prepare\" or \"release\".");+        }++        if (phase.equals("release")) {+            createGHRelease(fullVersion);+        }+        checkAndPrepareRepository();+        if (phase.equals("release")) {+            updateSuites(this::updateReleaseAndBumpVersionInSuite);+        } else {+            updateSuites(this::markSuiteAsRelease);+        }+        final String authorEmail = commitAndPushChanges();+        openPR(authorEmail);+        return 0;+    }++    private void checkAndPrepareRepository() {+        try (Git git = Git.open(new File("."))) {+            if (!git.getRepository().getBranch().equals(baseBranch)) {+                error("Please checkout " + baseBranch + " and try again!");+            }+            final Status status = git.status().call();+            if (status.hasUncommittedChanges() || !status.getChanged().isEmpty()) {+                error("Status of branch " + baseBranch + " is not clean, aborting!");+            }++            final String forkURL = "git@github.com:" + forkName;+            git.remoteAdd().setName(REMOTE_NAME).setUri(new URIish(forkURL)).call();+            info("Git remote " + REMOTE_NAME + " added and points to " + forkURL);+            final String newBranch = phase.equals("release") ? developBranch : releaseBranch;+            try {+                git.checkout().setCreateBranch(true).setName(newBranch).setStartPoint(newBranch).call();+                info("Created new branch " + newBranch + " based on " + baseBranch);+            } catch (RefAlreadyExistsException e) {+                warn(e.getMessage());+                gitCheckout(git, newBranch);+                git.reset().setRef(baseBranch).setMode(ResetCommand.ResetType.HARD).call();+                warn(newBranch + " reset (hard) to " + baseBranch);+            }+        } catch (IOException | GitAPIException | URISyntaxException e) {+            e.printStackTrace();+        }+    }++    /**+     * Visit all suite.py files and change the "release" and "version" fields+     *+     */+    private void updateSuites(Consumer<File> updater) {+        File cwd = new File(".");+        Stream.of(Objects.requireNonNull(cwd.listFiles()))+                .filter(File::isDirectory)+                .flatMap(path -> Stream.of(Objects.requireNonNull(path.listFiles())).filter(child -> child.getName().startsWith("mx.")))+                .map(path -> new File(path, "suite.py"))+                .filter(File::exists)+                .forEach(updater);+        info("Updated suites");+    }++    /**+     * Visit {@code suite} file and change the "release" value to {@code asRelease}+     *  @param suite+     *+     */+    private void markSuiteAsRelease(File suite) {+        try {+            info("Marking " + suite.getPath());+            List<String> lines = Files.readAllLines(suite.toPath());+            final String pattern = "(.*\"release\" : )False(.*)";+            final Pattern releasePattern = Pattern.compile(pattern);+            for (int i = 0; i < lines.size(); i++) {+                final Matcher releaseMatcher = releasePattern.matcher(lines.get(i));+                if (releaseMatcher.find()) {+                    String newLine = releaseMatcher.group(1) + "True" + releaseMatcher.group(2);+                    lines.set(i, newLine);+                    break;+                }+            }+            Files.write(suite.toPath(), lines, StandardCharsets.UTF_8);+        } catch (IOException e) {+            e.printStackTrace();+        }+    }++    private void updateReleaseAndBumpVersionInSuite(File suite) {+        try {+            info("Updating " + suite.getPath());+            List<String> lines = Files.readAllLines(suite.toPath());+            final Pattern releasePattern = Pattern.compile("(.*\"release\" : )True(.*)");+            final Pattern versionPattern = Pattern.compile("(.*\"version\" : \")"+version+"(\".*)");+            for (int i = 0; i < lines.size(); i++) {+                final Matcher releaseMatcher = releasePattern.matcher(lines.get(i));+                if (releaseMatcher.find()) {+                    String newLine = releaseMatcher.group(1) + "False" + releaseMatcher.group(2);+                    lines.set(i, newLine);+                }+                final Matcher versionMatcher = versionPattern.matcher(lines.get(i));+                if (versionMatcher.find()) {+                    String newLine = versionMatcher.group(1) + newVersion + versionMatcher.group(2);+                    lines.set(i, newLine);+                }+            }+            Files.write(suite.toPath(), lines, StandardCharsets.UTF_8);+        } catch (IOException e) {+            e.printStackTrace();+        }+    }++    private String commitAndPushChanges() {+        try (Git git = Git.open(new File("."))) {+            ConsoleCredentialsProvider.install(); // Needed for gpg signing+            final String message;+            if (phase.equals("release")) {+                message = "Unmark suites and bump version to " + newVersion;+            } else {+                message = "Mark suites for " + fullVersion + " release";+            }+            final RevCommit commit = git.commit().setAll(true).setMessage(message).setSign(signCommits).call();+            final String author = commit.getAuthorIdent().getEmailAddress();+            info("Changes commited");+            git.push().setForce(true).setRemote(REMOTE_NAME).setDryRun(dryRun).call();+            if (dryRun) {+                warn("Changes not pushed to remote due to --dry-run being present");+            } else {+                info("Changes pushed to remote " + REMOTE_NAME);+            }+            gitCheckout(git, baseBranch);+            return author;+        } catch (IOException | GitAPIException e) {+            e.printStackTrace();+            error(e.getMessage());+        }+        return null;+    }++    private void gitCheckout(Git git, String baseBranch) throws GitAPIException {+        git.checkout().setName(baseBranch).call();+        info("Checked out " + baseBranch);+    }++    private void openPR(String authorEmail) {+        GitHub github = connectToGitHub();+        try {+            final GHRepository repository = github.getRepository("graalvm/mandrel");+            if (dryRun) {+                warn("Pull request creation skipped due to --dry-run being present");+                return;+            }++            final String title;+            final String head;+            final String body;+            if (phase.equals("release")) {+                title = "Mark suites for " + fullVersion + " release";+                head = forkName.split("/")[0] + ":" + releaseBranch;+                body = "This PR was automatically generated by `mandrel-release.java` of graalvm/mandrel-packaging!\n\n" ++                        "Please tag branch `" + baseBranch + "` after merging, and push the tag.\n" ++                        "```\n" ++                        "git checkout mandrel/20.1\n" ++                        "git pull upstream mandrel/20.1\n" ++                        "git tag -a mandrel-" + fullVersion + " -m \"mandrel-" + fullVersion + "\" -s\n" ++                        "git push upstream mandrel-" + fullVersion + "\n" ++                        "```\n" ++                        "where `upstream` is a the git remote showing to https://github.com/graalvm/mandrel";+            } else {+                title = "Unmark suites and dump version to " + newVersion;+                head = forkName.split("/")[0] + ":" + developBranch;+                body = "This PR was automatically generated by `mandrel-release.java` of graalvm/mandrel-packaging!";+            }++            final GHPullRequest pullRequest = repository.createPullRequest(title, head, baseBranch, body, true, true);++            final GHUser galderz = github.getUser("galderz");+            final GHUser jerboaa = github.getUser("jerboaa");+            final GHUser karm = github.getUser("Karm");+            final GHUser zakkak = github.getUser("zakkak");++            ArrayList<GHUser> reviewers = new ArrayList<>();+            if (!authorEmail.contains("galder")) {+                reviewers.add(galderz);+            } else {+                pullRequest.addAssignees(Collections.singletonList(galderz));+            }+            if (!authorEmail.contains("sgehwolf") && !authorEmail.contains("jerboaa")) {+                reviewers.add(jerboaa);+            } else {+                pullRequest.addAssignees(Collections.singletonList(jerboaa));+            }+            if (!authorEmail.contains("karm")) {+                reviewers.add(karm);+            } else {+                pullRequest.addAssignees(Collections.singletonList(karm));+            }+            if (!authorEmail.contains("zakkak")) {+                reviewers.add(zakkak);+            } else {+                pullRequest.addAssignees(Collections.singletonList(zakkak));+            }+            pullRequest.requestReviewers(reviewers);++            info("Pull request " + pullRequest.getHtmlUrl() + " created");+        } catch (IOException e) {+            error(e.getMessage());+        }+    }++    /**+     * Returns current version of substratevm+     *+     * @return+     */+    private static String getCurrentVersion() {+        final Path substrateSuite = Path.of("substratevm", "mx.substratevm", "suite.py");+        String version = null;+        try {+            final List<String> lines = Files.readAllLines(substrateSuite);+            final Pattern versionPattern = Pattern.compile("\"version\" : \"([\\d.]+)\"");+            for (String line: lines) {+                final Matcher versionMatcher = versionPattern.matcher(line);+                if (versionMatcher.find()) {+                    version = versionMatcher.group(1);+                    break;+                }+            }+        } catch (IOException e) {+            e.printStackTrace();+        }+        info("Current version is " + version);+        return version;+    }++    /**+     * Calculates the new version based on {@code version} by bumping pico in major.minor.micro.pico+     *+     * @param version The current version+     * @return The new version+     */+    private static String getNewVersion(String version) {+        final Pattern versionPattern = Pattern.compile("(\\d+\\.\\d+\\.\\d+\\.)(\\d+)");+        final Matcher versionMatcher = versionPattern.matcher(version);+        boolean found = versionMatcher.find();+        assert found;+        int pico = Integer.parseInt(versionMatcher.group(2)) + 1;+        final String newVersion = versionMatcher.group(1) + pico;+        info("New version will be " + newVersion);+        return newVersion;+    }++    private void createGHRelease(String fullVersion) {+        GitHub github = connectToGitHub();+        try {+            final GHRepository repository = github.getRepository("graalvm/mandrel");+            final PagedIterable<GHMilestone> ghMilestones = repository.listMilestones(GHIssueState.OPEN);+            final GHMilestone milestone = ghMilestones.toList().stream().filter(m -> m.getTitle().equals(fullVersion)).findAny().orElse(null);+            String changelog = createChangelog(fullVersion, repository, milestone);+            if (dryRun) {+                warn("Skipping release due to --dry-run");+                info("Release body would look like");+                System.out.println(releaseMainBody(fullVersion, changelog));+                return;+            }+            final GHRelease ghRelease = repository.createRelease("mandrel-" + fullVersion)+                    .name("Mandrel " + fullVersion)+                    .prerelease(!suffix.equals("Final"))+                    .body(releaseMainBody(fullVersion, changelog))+                    .draft(true)+                    .create();+            uploadAssets(fullVersion, ghRelease);+            info("Created new draft release: " + ghRelease.getHtmlUrl());+            info("Please review and publish!");+            manageMilestones(repository, ghMilestones, milestone);+        } catch (IOException e) {+            e.printStackTrace();+        }+    }++    private void uploadAssets(String fullVersion, GHRelease ghRelease) throws IOException {+        final File archive = new File("mandrel-java11-linux-amd64-" + fullVersion + ".tar.gz");+        final File archiveSHA1 = new File("mandrel-java11-linux-amd64-" + fullVersion + ".tar.gz.sha1");+        if (!archive.exists()) {+            warn("Archive \"" + archive.getName() + "\" was not found. Skipping asset upload.");+            warn("Please upload asset manually.");+        } else if (!archiveSHA1.exists()) {+            warn("Archive sha1 \"" + archive.getName() + "\" was not found. Skipping asset upload.");+            warn("Please upload asset manually.");+        } else {+            info("Uploading " + archive.getName());+            ghRelease.uploadAsset(archive, "application/gzip");+            info("Uploaded " + archive.getName());+            info("Uploading " + archiveSHA1.getName());+            ghRelease.uploadAsset(archiveSHA1, "text/plain");+            info("Uploaded " + archiveSHA1.getName());+        }+    }++    private String releaseMainBody(String version, String changelog) {+        final Matcher majorMinorMicroMatcher = Pattern.compile("(\\d+\\.\\d+\\.\\d+).*").matcher(version);+        if (!majorMinorMicroMatcher.find()) {+            throw new RuntimeException("Version is in wrong format : " + version);+        }+        final String majorMinorMicro = majorMinorMicroMatcher.group(1);+        return "# Mandrel\n" ++                "\n" ++                "Mandrel " + version + " is a downstream distribution of the [GraalVM community edition " + majorMinorMicro + "](https://github.com/graalvm/graalvm-ce-builds/releases/tag/vm-" + majorMinorMicro + ").\n" ++                "Mandrel's main goal is to provide a `native-image` release specifically to support [Quarkus](https://quarkus.io).\n" ++                "The aim is to align the `native-image` capabilities from GraalVM with OpenJDK and Red Hat Enterprise Linux libraries to improve maintainability for native Quarkus applications.\n" ++                "\n" ++                "## How Does Mandrel Differ From Graal\n" ++                "\n" ++                "Mandrel releases are built from a code base derived from the upstream GraalVM code base, with only minor changes but some significant exclusions.\n" ++                "They support the same native image capability as GraalVM with no significant changes to functionality.\n" ++                "They do not include support for Polyglot programming via the Truffle interpreter and compiler framework.\n" ++                "In consequence, it is not possible to extend Mandrel by downloading languages from the Truffle language catalogue.\n" ++                "\n" ++                "Mandrel is also built slightly differently to GraalVM, using the standard OpenJDK project release of jdk11u.\n" ++                "This means it does not profit from a few small enhancements that Oracle have added to the version of OpenJDK used to build their own GraalVM downloads.\n" ++                "Most of these enhancements are to the JVMCI module that allows the Graal compiler to be run inside OpenJDK.\n" ++                "The others are small cosmetic changes to behaviour.\n" ++                "These enhancements may in some cases cause minor differences in the progress of native image generation.\n" ++                "They should not cause the resulting images themselves to execute in a noticeably different manner.\n" ++                "\n" ++                "### Prerequisites\n" ++                "\n" ++                "Mandrel's `native-image` depends on the following packages:\n" ++                "* glibc-devel\n" ++                "* zlib-devel\n" ++                "* gcc\n" ++                "* libffi-devel\n" ++                "\n" ++                "On Fedora/CentOS/RHEL they can be installed with:\n" ++                "```bash\n" ++                "dnf install glibc-devel zlib-devel gcc libffi-devel libstdc++-static\n" ++                "```\n" ++                "\n" ++                "Note the package might be called `glibc-static` instead of `libstdc++-static`.\n" ++                "\n" ++                "On Ubuntu-like systems with:\n" ++                "```bash\n" ++                "apt install gcc zlib1g-dev libffi-dev build-essential\n" ++                "```\n" ++                "\n" ++                "## Quick start\n" ++                "\n" ++                "```\n" ++                "$ tar -xf mandrel-java11-linux-amd64-" + version + ".tar.gz\n" ++                "$ export JAVA_HOME=\"$( pwd )/mandrel-java11-" + version + "\"\n" ++                "$ export GRAALVM_HOME=\"${JAVA_HOME}\"\n" ++                "$ export PATH=\"${JAVA_HOME}/bin:${PATH}\"\n" ++                "$ curl -O -J  https://code.quarkus.io/api/download\n" ++                "$ unzip code-with-quarkus.zip\n" ++                "$ cd code-with-quarkus/\n" ++                "$ ./mvnw package -Pnative\n" ++                "$ ./target/code-with-quarkus-1.0.0-SNAPSHOT-runner\n" ++                "```\n" ++                "\n" ++                "### Quarkus builder image\n" ++                "\n" ++                "The Quarkus builder image for this release is still being prepared, please try again later.\n" ++                "<!--\n" ++                "Mandrel Quarkus builder image can be used to build a Quarkus native Linux executable right away without any GRAALVM_HOME setup.\n" ++                "\n" ++                "```bash\n" ++                "curl -O -J  https://code.quarkus.io/api/download\n" ++                "unzip code-with-quarkus.zip\n" ++                "cd code-with-quarkus\n" ++                "        ./mvnw package -Pnative -Dquarkus.native.container-build=true -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-mandrel:" + version + "-java11\n" ++                "        ./target/code-with-quarkus-1.0.0-SNAPSHOT-runner\n" ++                "```\n" ++                "\n" ++                "One can use the builder image on Windows with Docker Desktop (mind `Resources-> File sharing` settings so as Quarkus project directory is mountable).\n" ++                "\n" ++                "```batchfile\n" ++                "powershell -c \"Invoke-WebRequest -OutFile quarkus.zip -Uri https://code.quarkus.io/api/download\"\n" ++                "powershell -c \"Expand-Archive -Path quarkus.zip -DestinationPath . -Force\n" ++                "cd code-with-quarkus\n" ++                "mvnw package -Pnative -Dquarkus.native.container-build=true -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-mandrel:" + version + "-java11\n" ++                "docker build -f src/main/docker/Dockerfile.native -t my-quarkus-mandrel-app .\n" ++                "        docker run -i --rm -p 8080:8080 my-quarkus-mandrel-app\n" ++                "```\n" ++                "-->\n" ++                changelog ++                "\n---\n" ++                "Mandrel " + version + "\n" ++                "OpenJDK used: 11.0.8+10\n";

Graal uses https://github.com/graalvm/mandrel/blob/graal/master/common.json to achieve this

zakkak

comment created time in 6 days

PullRequestReviewEvent

Pull request review commentgraalvm/mandrel-packaging

Jbang the release process!

+//usr/bin/env jbang "$0" "$@" ; exit $?+//DEPS org.eclipse.jgit:org.eclipse.jgit:5.9.0.202009080501-r+//DEPS org.eclipse.jgit:org.eclipse.jgit.pgm:5.9.0.202009080501-r+//DEPS org.eclipse.jgit:org.eclipse.jgit.gpg.bc:5.9.0.202009080501-r+//DEPS info.picocli:picocli:4.5.0+//DEPS org.kohsuke:github-api:1.116++import org.eclipse.jgit.api.Git;+import org.eclipse.jgit.api.ResetCommand;+import org.eclipse.jgit.api.Status;+import org.eclipse.jgit.api.errors.GitAPIException;+import org.eclipse.jgit.api.errors.RefAlreadyExistsException;+import org.eclipse.jgit.console.ConsoleCredentialsProvider;+import org.eclipse.jgit.revwalk.RevCommit;+import org.eclipse.jgit.transport.URIish;+import org.kohsuke.github.*;+import picocli.CommandLine;+import picocli.CommandLine.Command;+import picocli.CommandLine.Help.Ansi;++import java.io.File;+import java.io.IOException;+import java.net.URISyntaxException;+import java.nio.charset.StandardCharsets;+import java.nio.file.Files;+import java.nio.file.Path;+import java.util.*;+import java.util.concurrent.Callable;+import java.util.function.Consumer;+import java.util.function.Function;+import java.util.regex.Matcher;+import java.util.regex.Pattern;+import java.util.stream.Collectors;+import java.util.stream.Stream;++@Command(name = "mandrel-release", mixinStandardHelpOptions = true,+        description = "Scipt automating part of the Mandrel release process")+class MandrelRelease implements Callable<Integer> {++    @CommandLine.Parameters(index = "0", description = "The kind of steps to execute, must be one of \"prepare\" or \"release\"")+    private String phase;++    @CommandLine.Option(names = {"-s", "--suffix"}, description = "The release suffix, e.g, Final, Alpha2, Beta1, etc. (default: \"${DEFAULT-VALUE}\")", defaultValue = "Final")+    private String suffix;++    @CommandLine.Option(names = {"-f", "--fork-name"}, description = "The repository name of the github fork to push the changes to (default: \"${DEFAULT-VALUE}\")", defaultValue = "zakkak/mandrel")+    private String forkName;++    @CommandLine.Option(names = {"-S", "--sign-commits"}, description = "Sign commits")+    private boolean signCommits;++    @CommandLine.Option(names = {"-D", "--dry-run"}, description = "Perform a dry run (no remote pushes and PRs)")+    private boolean dryRun;++    private static final String REMOTE_NAME = "mandrel-release-fork";+    private String version = null;+    private String fullVersion = null;+    private String releaseBranch = null;+    private String baseBranch = null;+    private String newVersion = null;+    private String newFullVersion = null;+    private String developBranch = null;++    public static void main(String... args) {+        int exitCode = new CommandLine(new MandrelRelease()).execute(args);+        System.exit(exitCode);+    }++    @Override+    public Integer call() {+        if (!suffix.equals("Final") && !Pattern.compile("^(Alpha|Beta)\\d*$").matcher(suffix).find()) {+            error("Invalid version suffix : " + suffix);+        }++        version = getCurrentVersion();+        fullVersion = version + "." + suffix;+        newVersion = getNewVersion(version);+        newFullVersion = newVersion + "." + suffix;+        releaseBranch = "release/mandrel-" + fullVersion;+        developBranch = "develop/mandrel-" + newVersion;+        final Matcher majorMinorMatcher = Pattern.compile("(\\d+\\.\\d+).*").matcher(version);+        if (!majorMinorMatcher.find()) {+            throw new RuntimeException("Version is in wrong format : " + version);+        }+        final String majorMinor = majorMinorMatcher.group(1);+        baseBranch = "mandrel/" + majorMinor;++        if (!phase.equals("prepare") && !phase.equals("release")) {+            throw new IllegalArgumentException(phase + " is not a valid phase. Please use \"prepare\" or \"release\".");+        }++        if (phase.equals("release")) {+            createGHRelease(fullVersion);+        }+        checkAndPrepareRepository();+        if (phase.equals("release")) {+            updateSuites(this::updateReleaseAndBumpVersionInSuite);+        } else {+            updateSuites(this::markSuiteAsRelease);+        }+        final String authorEmail = commitAndPushChanges();+        openPR(authorEmail);+        return 0;+    }++    private void checkAndPrepareRepository() {+        try (Git git = Git.open(new File("."))) {+            if (!git.getRepository().getBranch().equals(baseBranch)) {+                error("Please checkout " + baseBranch + " and try again!");+            }+            final Status status = git.status().call();+            if (status.hasUncommittedChanges() || !status.getChanged().isEmpty()) {+                error("Status of branch " + baseBranch + " is not clean, aborting!");+            }++            final String forkURL = "git@github.com:" + forkName;+            git.remoteAdd().setName(REMOTE_NAME).setUri(new URIish(forkURL)).call();+            info("Git remote " + REMOTE_NAME + " added and points to " + forkURL);+            final String newBranch = phase.equals("release") ? developBranch : releaseBranch;+            try {+                git.checkout().setCreateBranch(true).setName(newBranch).setStartPoint(newBranch).call();+                info("Created new branch " + newBranch + " based on " + baseBranch);+            } catch (RefAlreadyExistsException e) {+                warn(e.getMessage());+                gitCheckout(git, newBranch);+                git.reset().setRef(baseBranch).setMode(ResetCommand.ResetType.HARD).call();+                warn(newBranch + " reset (hard) to " + baseBranch);+            }+        } catch (IOException | GitAPIException | URISyntaxException e) {+            e.printStackTrace();+        }+    }++    /**+     * Visit all suite.py files and change the "release" and "version" fields+     *+     */+    private void updateSuites(Consumer<File> updater) {+        File cwd = new File(".");+        Stream.of(Objects.requireNonNull(cwd.listFiles()))+                .filter(File::isDirectory)+                .flatMap(path -> Stream.of(Objects.requireNonNull(path.listFiles())).filter(child -> child.getName().startsWith("mx.")))+                .map(path -> new File(path, "suite.py"))+                .filter(File::exists)+                .forEach(updater);+        info("Updated suites");+    }++    /**+     * Visit {@code suite} file and change the "release" value to {@code asRelease}+     *  @param suite+     *+     */+    private void markSuiteAsRelease(File suite) {+        try {+            info("Marking " + suite.getPath());+            List<String> lines = Files.readAllLines(suite.toPath());+            final String pattern = "(.*\"release\" : )False(.*)";+            final Pattern releasePattern = Pattern.compile(pattern);+            for (int i = 0; i < lines.size(); i++) {+                final Matcher releaseMatcher = releasePattern.matcher(lines.get(i));+                if (releaseMatcher.find()) {+                    String newLine = releaseMatcher.group(1) + "True" + releaseMatcher.group(2);+                    lines.set(i, newLine);+                    break;+                }+            }+            Files.write(suite.toPath(), lines, StandardCharsets.UTF_8);+        } catch (IOException e) {+            e.printStackTrace();+        }+    }++    private void updateReleaseAndBumpVersionInSuite(File suite) {+        try {+            info("Updating " + suite.getPath());+            List<String> lines = Files.readAllLines(suite.toPath());+            final Pattern releasePattern = Pattern.compile("(.*\"release\" : )True(.*)");+            final Pattern versionPattern = Pattern.compile("(.*\"version\" : \")"+version+"(\".*)");+            for (int i = 0; i < lines.size(); i++) {+                final Matcher releaseMatcher = releasePattern.matcher(lines.get(i));+                if (releaseMatcher.find()) {+                    String newLine = releaseMatcher.group(1) + "False" + releaseMatcher.group(2);+                    lines.set(i, newLine);+                }+                final Matcher versionMatcher = versionPattern.matcher(lines.get(i));+                if (versionMatcher.find()) {+                    String newLine = versionMatcher.group(1) + newVersion + versionMatcher.group(2);+                    lines.set(i, newLine);+                }+            }+            Files.write(suite.toPath(), lines, StandardCharsets.UTF_8);+        } catch (IOException e) {+            e.printStackTrace();+        }+    }++    private String commitAndPushChanges() {+        try (Git git = Git.open(new File("."))) {+            ConsoleCredentialsProvider.install(); // Needed for gpg signing+            final String message;+            if (phase.equals("release")) {+                message = "Unmark suites and bump version to " + newVersion;+            } else {+                message = "Mark suites for " + fullVersion + " release";+            }+            final RevCommit commit = git.commit().setAll(true).setMessage(message).setSign(signCommits).call();+            final String author = commit.getAuthorIdent().getEmailAddress();+            info("Changes commited");+            git.push().setForce(true).setRemote(REMOTE_NAME).setDryRun(dryRun).call();+            if (dryRun) {+                warn("Changes not pushed to remote due to --dry-run being present");+            } else {+                info("Changes pushed to remote " + REMOTE_NAME);+            }+            gitCheckout(git, baseBranch);+            return author;+        } catch (IOException | GitAPIException e) {+            e.printStackTrace();+            error(e.getMessage());+        }+        return null;+    }++    private void gitCheckout(Git git, String baseBranch) throws GitAPIException {+        git.checkout().setName(baseBranch).call();+        info("Checked out " + baseBranch);+    }++    private void openPR(String authorEmail) {+        GitHub github = connectToGitHub();+        try {+            final GHRepository repository = github.getRepository("graalvm/mandrel");+            if (dryRun) {+                warn("Pull request creation skipped due to --dry-run being present");+                return;+            }++            final String title;+            final String head;+            final String body;+            if (phase.equals("release")) {+                title = "Mark suites for " + fullVersion + " release";+                head = forkName.split("/")[0] + ":" + releaseBranch;+                body = "This PR was automatically generated by `mandrel-release.java` of graalvm/mandrel-packaging!\n\n" ++                        "Please tag branch `" + baseBranch + "` after merging, and push the tag.\n" ++                        "```\n" ++                        "git checkout mandrel/20.1\n" ++                        "git pull upstream mandrel/20.1\n" ++                        "git tag -a mandrel-" + fullVersion + " -m \"mandrel-" + fullVersion + "\" -s\n" ++                        "git push upstream mandrel-" + fullVersion + "\n" ++                        "```\n" ++                        "where `upstream` is a the git remote showing to https://github.com/graalvm/mandrel";+            } else {+                title = "Unmark suites and dump version to " + newVersion;+                head = forkName.split("/")[0] + ":" + developBranch;+                body = "This PR was automatically generated by `mandrel-release.java` of graalvm/mandrel-packaging!";+            }++            final GHPullRequest pullRequest = repository.createPullRequest(title, head, baseBranch, body, true, true);++            final GHUser galderz = github.getUser("galderz");+            final GHUser jerboaa = github.getUser("jerboaa");+            final GHUser karm = github.getUser("Karm");+            final GHUser zakkak = github.getUser("zakkak");++            ArrayList<GHUser> reviewers = new ArrayList<>();+            if (!authorEmail.contains("galder")) {+                reviewers.add(galderz);+            } else {+                pullRequest.addAssignees(Collections.singletonList(galderz));+            }+            if (!authorEmail.contains("sgehwolf") && !authorEmail.contains("jerboaa")) {+                reviewers.add(jerboaa);+            } else {+                pullRequest.addAssignees(Collections.singletonList(jerboaa));+            }+            if (!authorEmail.contains("karm")) {+                reviewers.add(karm);+            } else {+                pullRequest.addAssignees(Collections.singletonList(karm));+            }+            if (!authorEmail.contains("zakkak")) {+                reviewers.add(zakkak);+            } else {+                pullRequest.addAssignees(Collections.singletonList(zakkak));+            }+            pullRequest.requestReviewers(reviewers);++            info("Pull request " + pullRequest.getHtmlUrl() + " created");+        } catch (IOException e) {+            error(e.getMessage());+        }+    }++    /**+     * Returns current version of substratevm+     *+     * @return+     */+    private static String getCurrentVersion() {+        final Path substrateSuite = Path.of("substratevm", "mx.substratevm", "suite.py");+        String version = null;+        try {+            final List<String> lines = Files.readAllLines(substrateSuite);+            final Pattern versionPattern = Pattern.compile("\"version\" : \"([\\d.]+)\"");+            for (String line: lines) {+                final Matcher versionMatcher = versionPattern.matcher(line);+                if (versionMatcher.find()) {+                    version = versionMatcher.group(1);+                    break;+                }+            }+        } catch (IOException e) {+            e.printStackTrace();+        }+        info("Current version is " + version);+        return version;+    }++    /**+     * Calculates the new version based on {@code version} by bumping pico in major.minor.micro.pico+     *+     * @param version The current version+     * @return The new version+     */+    private static String getNewVersion(String version) {+        final Pattern versionPattern = Pattern.compile("(\\d+\\.\\d+\\.\\d+\\.)(\\d+)");+        final Matcher versionMatcher = versionPattern.matcher(version);+        boolean found = versionMatcher.find();+        assert found;+        int pico = Integer.parseInt(versionMatcher.group(2)) + 1;+        final String newVersion = versionMatcher.group(1) + pico;+        info("New version will be " + newVersion);+        return newVersion;+    }++    private void createGHRelease(String fullVersion) {+        GitHub github = connectToGitHub();+        try {+            final GHRepository repository = github.getRepository("graalvm/mandrel");+            final PagedIterable<GHMilestone> ghMilestones = repository.listMilestones(GHIssueState.OPEN);+            final GHMilestone milestone = ghMilestones.toList().stream().filter(m -> m.getTitle().equals(fullVersion)).findAny().orElse(null);+            String changelog = createChangelog(fullVersion, repository, milestone);+            if (dryRun) {+                warn("Skipping release due to --dry-run");+                info("Release body would look like");+                System.out.println(releaseMainBody(fullVersion, changelog));+                return;+            }+            final GHRelease ghRelease = repository.createRelease("mandrel-" + fullVersion)+                    .name("Mandrel " + fullVersion)+                    .prerelease(!suffix.equals("Final"))+                    .body(releaseMainBody(fullVersion, changelog))+                    .draft(true)+                    .create();+            uploadAssets(fullVersion, ghRelease);+            info("Created new draft release: " + ghRelease.getHtmlUrl());+            info("Please review and publish!");+            manageMilestones(repository, ghMilestones, milestone);+        } catch (IOException e) {+            e.printStackTrace();+        }+    }++    private void uploadAssets(String fullVersion, GHRelease ghRelease) throws IOException {+        final File archive = new File("mandrel-java11-linux-amd64-" + fullVersion + ".tar.gz");+        final File archiveSHA1 = new File("mandrel-java11-linux-amd64-" + fullVersion + ".tar.gz.sha1");+        if (!archive.exists()) {+            warn("Archive \"" + archive.getName() + "\" was not found. Skipping asset upload.");+            warn("Please upload asset manually.");+        } else if (!archiveSHA1.exists()) {+            warn("Archive sha1 \"" + archive.getName() + "\" was not found. Skipping asset upload.");+            warn("Please upload asset manually.");+        } else {+            info("Uploading " + archive.getName());+            ghRelease.uploadAsset(archive, "application/gzip");+            info("Uploaded " + archive.getName());+            info("Uploading " + archiveSHA1.getName());+            ghRelease.uploadAsset(archiveSHA1, "text/plain");+            info("Uploaded " + archiveSHA1.getName());+        }+    }++    private String releaseMainBody(String version, String changelog) {+        final Matcher majorMinorMicroMatcher = Pattern.compile("(\\d+\\.\\d+\\.\\d+).*").matcher(version);+        if (!majorMinorMicroMatcher.find()) {+            throw new RuntimeException("Version is in wrong format : " + version);+        }+        final String majorMinorMicro = majorMinorMicroMatcher.group(1);+        return "# Mandrel\n" ++                "\n" ++                "Mandrel " + version + " is a downstream distribution of the [GraalVM community edition " + majorMinorMicro + "](https://github.com/graalvm/graalvm-ce-builds/releases/tag/vm-" + majorMinorMicro + ").\n" ++                "Mandrel's main goal is to provide a `native-image` release specifically to support [Quarkus](https://quarkus.io).\n" ++                "The aim is to align the `native-image` capabilities from GraalVM with OpenJDK and Red Hat Enterprise Linux libraries to improve maintainability for native Quarkus applications.\n" ++                "\n" ++                "## How Does Mandrel Differ From Graal\n" ++                "\n" ++                "Mandrel releases are built from a code base derived from the upstream GraalVM code base, with only minor changes but some significant exclusions.\n" ++                "They support the same native image capability as GraalVM with no significant changes to functionality.\n" ++                "They do not include support for Polyglot programming via the Truffle interpreter and compiler framework.\n" ++                "In consequence, it is not possible to extend Mandrel by downloading languages from the Truffle language catalogue.\n" ++                "\n" ++                "Mandrel is also built slightly differently to GraalVM, using the standard OpenJDK project release of jdk11u.\n" ++                "This means it does not profit from a few small enhancements that Oracle have added to the version of OpenJDK used to build their own GraalVM downloads.\n" ++                "Most of these enhancements are to the JVMCI module that allows the Graal compiler to be run inside OpenJDK.\n" ++                "The others are small cosmetic changes to behaviour.\n" ++                "These enhancements may in some cases cause minor differences in the progress of native image generation.\n" ++                "They should not cause the resulting images themselves to execute in a noticeably different manner.\n" ++                "\n" ++                "### Prerequisites\n" ++                "\n" ++                "Mandrel's `native-image` depends on the following packages:\n" ++                "* glibc-devel\n" ++                "* zlib-devel\n" ++                "* gcc\n" ++                "* libffi-devel\n" ++                "\n" ++                "On Fedora/CentOS/RHEL they can be installed with:\n" ++                "```bash\n" ++                "dnf install glibc-devel zlib-devel gcc libffi-devel libstdc++-static\n" ++                "```\n" ++                "\n" ++                "Note the package might be called `glibc-static` instead of `libstdc++-static`.\n" ++                "\n" ++                "On Ubuntu-like systems with:\n" ++                "```bash\n" ++                "apt install gcc zlib1g-dev libffi-dev build-essential\n" ++                "```\n" ++                "\n" ++                "## Quick start\n" ++                "\n" ++                "```\n" ++                "$ tar -xf mandrel-java11-linux-amd64-" + version + ".tar.gz\n" ++                "$ export JAVA_HOME=\"$( pwd )/mandrel-java11-" + version + "\"\n" ++                "$ export GRAALVM_HOME=\"${JAVA_HOME}\"\n" ++                "$ export PATH=\"${JAVA_HOME}/bin:${PATH}\"\n" ++                "$ curl -O -J  https://code.quarkus.io/api/download\n" ++                "$ unzip code-with-quarkus.zip\n" ++                "$ cd code-with-quarkus/\n" ++                "$ ./mvnw package -Pnative\n" ++                "$ ./target/code-with-quarkus-1.0.0-SNAPSHOT-runner\n" ++                "```\n" ++                "\n" ++                "### Quarkus builder image\n" ++                "\n" ++                "The Quarkus builder image for this release is still being prepared, please try again later.\n" ++                "<!--\n" ++                "Mandrel Quarkus builder image can be used to build a Quarkus native Linux executable right away without any GRAALVM_HOME setup.\n" ++                "\n" ++                "```bash\n" ++                "curl -O -J  https://code.quarkus.io/api/download\n" ++                "unzip code-with-quarkus.zip\n" ++                "cd code-with-quarkus\n" ++                "        ./mvnw package -Pnative -Dquarkus.native.container-build=true -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-mandrel:" + version + "-java11\n" ++                "        ./target/code-with-quarkus-1.0.0-SNAPSHOT-runner\n" ++                "```\n" ++                "\n" ++                "One can use the builder image on Windows with Docker Desktop (mind `Resources-> File sharing` settings so as Quarkus project directory is mountable).\n" ++                "\n" ++                "```batchfile\n" ++                "powershell -c \"Invoke-WebRequest -OutFile quarkus.zip -Uri https://code.quarkus.io/api/download\"\n" ++                "powershell -c \"Expand-Archive -Path quarkus.zip -DestinationPath . -Force\n" ++                "cd code-with-quarkus\n" ++                "mvnw package -Pnative -Dquarkus.native.container-build=true -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-mandrel:" + version + "-java11\n" ++                "docker build -f src/main/docker/Dockerfile.native -t my-quarkus-mandrel-app .\n" ++                "        docker run -i --rm -p 8080:8080 my-quarkus-mandrel-app\n" ++                "```\n" ++                "-->\n" ++                changelog ++                "\n---\n" ++                "Mandrel " + version + "\n" ++                "OpenJDK used: 11.0.8+10\n";

Another way would be to include a file in mandrel sources (or mandrel-packaging sources) that maps mandrel versions to openJDK versions (this will make it easier to reproduce builds in the future as well).

zakkak

comment created time in 6 days

PullRequestReviewEvent

Pull request review commentgraalvm/mandrel-packaging

Jbang the release process!

+//usr/bin/env jbang "$0" "$@" ; exit $?+//DEPS org.eclipse.jgit:org.eclipse.jgit:5.9.0.202009080501-r+//DEPS org.eclipse.jgit:org.eclipse.jgit.pgm:5.9.0.202009080501-r+//DEPS org.eclipse.jgit:org.eclipse.jgit.gpg.bc:5.9.0.202009080501-r+//DEPS info.picocli:picocli:4.5.0+//DEPS org.kohsuke:github-api:1.116++import org.eclipse.jgit.api.Git;+import org.eclipse.jgit.api.ResetCommand;+import org.eclipse.jgit.api.Status;+import org.eclipse.jgit.api.errors.GitAPIException;+import org.eclipse.jgit.api.errors.RefAlreadyExistsException;+import org.eclipse.jgit.console.ConsoleCredentialsProvider;+import org.eclipse.jgit.revwalk.RevCommit;+import org.eclipse.jgit.transport.URIish;+import org.kohsuke.github.*;+import picocli.CommandLine;+import picocli.CommandLine.Command;+import picocli.CommandLine.Help.Ansi;++import java.io.File;+import java.io.IOException;+import java.net.URISyntaxException;+import java.nio.charset.StandardCharsets;+import java.nio.file.Files;+import java.nio.file.Path;+import java.util.*;+import java.util.concurrent.Callable;+import java.util.function.Consumer;+import java.util.function.Function;+import java.util.regex.Matcher;+import java.util.regex.Pattern;+import java.util.stream.Collectors;+import java.util.stream.Stream;++@Command(name = "mandrel-release", mixinStandardHelpOptions = true,+        description = "Scipt automating part of the Mandrel release process")+class MandrelRelease implements Callable<Integer> {++    @CommandLine.Parameters(index = "0", description = "The kind of steps to execute, must be one of \"prepare\" or \"release\"")+    private String phase;++    @CommandLine.Option(names = {"-s", "--suffix"}, description = "The release suffix, e.g, Final, Alpha2, Beta1, etc. (default: \"${DEFAULT-VALUE}\")", defaultValue = "Final")+    private String suffix;++    @CommandLine.Option(names = {"-f", "--fork-name"}, description = "The repository name of the github fork to push the changes to (default: \"${DEFAULT-VALUE}\")", defaultValue = "zakkak/mandrel")+    private String forkName;++    @CommandLine.Option(names = {"-S", "--sign-commits"}, description = "Sign commits")+    private boolean signCommits;++    @CommandLine.Option(names = {"-D", "--dry-run"}, description = "Perform a dry run (no remote pushes and PRs)")+    private boolean dryRun;++    private static final String REMOTE_NAME = "mandrel-release-fork";+    private String version = null;+    private String fullVersion = null;+    private String releaseBranch = null;+    private String baseBranch = null;+    private String newVersion = null;+    private String newFullVersion = null;+    private String developBranch = null;++    public static void main(String... args) {+        int exitCode = new CommandLine(new MandrelRelease()).execute(args);+        System.exit(exitCode);+    }++    @Override+    public Integer call() {+        if (!suffix.equals("Final") && !Pattern.compile("^(Alpha|Beta)\\d*$").matcher(suffix).find()) {+            error("Invalid version suffix : " + suffix);+        }++        version = getCurrentVersion();+        fullVersion = version + "." + suffix;+        newVersion = getNewVersion(version);+        newFullVersion = newVersion + "." + suffix;+        releaseBranch = "release/mandrel-" + fullVersion;+        developBranch = "develop/mandrel-" + newVersion;+        final Matcher majorMinorMatcher = Pattern.compile("(\\d+\\.\\d+).*").matcher(version);+        if (!majorMinorMatcher.find()) {+            throw new RuntimeException("Version is in wrong format : " + version);+        }+        final String majorMinor = majorMinorMatcher.group(1);+        baseBranch = "mandrel/" + majorMinor;++        if (!phase.equals("prepare") && !phase.equals("release")) {+            throw new IllegalArgumentException(phase + " is not a valid phase. Please use \"prepare\" or \"release\".");+        }++        if (phase.equals("release")) {+            createGHRelease(fullVersion);+        }+        checkAndPrepareRepository();+        if (phase.equals("release")) {+            updateSuites(this::updateReleaseAndBumpVersionInSuite);+        } else {+            updateSuites(this::markSuiteAsRelease);+        }+        final String authorEmail = commitAndPushChanges();+        openPR(authorEmail);+        return 0;+    }++    private void checkAndPrepareRepository() {+        try (Git git = Git.open(new File("."))) {+            if (!git.getRepository().getBranch().equals(baseBranch)) {+                error("Please checkout " + baseBranch + " and try again!");+            }+            final Status status = git.status().call();+            if (status.hasUncommittedChanges() || !status.getChanged().isEmpty()) {+                error("Status of branch " + baseBranch + " is not clean, aborting!");+            }++            final String forkURL = "git@github.com:" + forkName;+            git.remoteAdd().setName(REMOTE_NAME).setUri(new URIish(forkURL)).call();+            info("Git remote " + REMOTE_NAME + " added and points to " + forkURL);+            final String newBranch = phase.equals("release") ? developBranch : releaseBranch;+            try {+                git.checkout().setCreateBranch(true).setName(newBranch).setStartPoint(newBranch).call();+                info("Created new branch " + newBranch + " based on " + baseBranch);+            } catch (RefAlreadyExistsException e) {+                warn(e.getMessage());+                gitCheckout(git, newBranch);+                git.reset().setRef(baseBranch).setMode(ResetCommand.ResetType.HARD).call();+                warn(newBranch + " reset (hard) to " + baseBranch);+            }+        } catch (IOException | GitAPIException | URISyntaxException e) {+            e.printStackTrace();+        }+    }++    /**+     * Visit all suite.py files and change the "release" and "version" fields+     *+     */+    private void updateSuites(Consumer<File> updater) {+        File cwd = new File(".");+        Stream.of(Objects.requireNonNull(cwd.listFiles()))+                .filter(File::isDirectory)+                .flatMap(path -> Stream.of(Objects.requireNonNull(path.listFiles())).filter(child -> child.getName().startsWith("mx.")))+                .map(path -> new File(path, "suite.py"))+                .filter(File::exists)+                .forEach(updater);+        info("Updated suites");+    }++    /**+     * Visit {@code suite} file and change the "release" value to {@code asRelease}+     *  @param suite+     *+     */+    private void markSuiteAsRelease(File suite) {+        try {+            info("Marking " + suite.getPath());+            List<String> lines = Files.readAllLines(suite.toPath());+            final String pattern = "(.*\"release\" : )False(.*)";+            final Pattern releasePattern = Pattern.compile(pattern);+            for (int i = 0; i < lines.size(); i++) {+                final Matcher releaseMatcher = releasePattern.matcher(lines.get(i));+                if (releaseMatcher.find()) {+                    String newLine = releaseMatcher.group(1) + "True" + releaseMatcher.group(2);+                    lines.set(i, newLine);+                    break;+                }+            }+            Files.write(suite.toPath(), lines, StandardCharsets.UTF_8);+        } catch (IOException e) {+            e.printStackTrace();+        }+    }++    private void updateReleaseAndBumpVersionInSuite(File suite) {+        try {+            info("Updating " + suite.getPath());+            List<String> lines = Files.readAllLines(suite.toPath());+            final Pattern releasePattern = Pattern.compile("(.*\"release\" : )True(.*)");+            final Pattern versionPattern = Pattern.compile("(.*\"version\" : \")"+version+"(\".*)");+            for (int i = 0; i < lines.size(); i++) {+                final Matcher releaseMatcher = releasePattern.matcher(lines.get(i));+                if (releaseMatcher.find()) {+                    String newLine = releaseMatcher.group(1) + "False" + releaseMatcher.group(2);+                    lines.set(i, newLine);+                }+                final Matcher versionMatcher = versionPattern.matcher(lines.get(i));+                if (versionMatcher.find()) {+                    String newLine = versionMatcher.group(1) + newVersion + versionMatcher.group(2);+                    lines.set(i, newLine);+                }+            }+            Files.write(suite.toPath(), lines, StandardCharsets.UTF_8);+        } catch (IOException e) {+            e.printStackTrace();+        }+    }++    private String commitAndPushChanges() {+        try (Git git = Git.open(new File("."))) {+            ConsoleCredentialsProvider.install(); // Needed for gpg signing+            final String message;+            if (phase.equals("release")) {+                message = "Unmark suites and bump version to " + newVersion;+            } else {+                message = "Mark suites for " + fullVersion + " release";+            }+            final RevCommit commit = git.commit().setAll(true).setMessage(message).setSign(signCommits).call();+            final String author = commit.getAuthorIdent().getEmailAddress();+            info("Changes commited");+            git.push().setForce(true).setRemote(REMOTE_NAME).setDryRun(dryRun).call();+            if (dryRun) {+                warn("Changes not pushed to remote due to --dry-run being present");+            } else {+                info("Changes pushed to remote " + REMOTE_NAME);+            }+            gitCheckout(git, baseBranch);+            return author;+        } catch (IOException | GitAPIException e) {+            e.printStackTrace();+            error(e.getMessage());+        }+        return null;+    }++    private void gitCheckout(Git git, String baseBranch) throws GitAPIException {+        git.checkout().setName(baseBranch).call();+        info("Checked out " + baseBranch);+    }++    private void openPR(String authorEmail) {+        GitHub github = connectToGitHub();+        try {+            final GHRepository repository = github.getRepository("graalvm/mandrel");+            if (dryRun) {+                warn("Pull request creation skipped due to --dry-run being present");+                return;+            }++            final String title;+            final String head;+            final String body;+            if (phase.equals("release")) {+                title = "Mark suites for " + fullVersion + " release";+                head = forkName.split("/")[0] + ":" + releaseBranch;+                body = "This PR was automatically generated by `mandrel-release.java` of graalvm/mandrel-packaging!\n\n" ++                        "Please tag branch `" + baseBranch + "` after merging, and push the tag.\n" ++                        "```\n" ++                        "git checkout mandrel/20.1\n" ++                        "git pull upstream mandrel/20.1\n" ++                        "git tag -a mandrel-" + fullVersion + " -m \"mandrel-" + fullVersion + "\" -s\n" ++                        "git push upstream mandrel-" + fullVersion + "\n" ++                        "```\n" ++                        "where `upstream` is a the git remote showing to https://github.com/graalvm/mandrel";+            } else {+                title = "Unmark suites and dump version to " + newVersion;+                head = forkName.split("/")[0] + ":" + developBranch;+                body = "This PR was automatically generated by `mandrel-release.java` of graalvm/mandrel-packaging!";+            }++            final GHPullRequest pullRequest = repository.createPullRequest(title, head, baseBranch, body, true, true);++            final GHUser galderz = github.getUser("galderz");+            final GHUser jerboaa = github.getUser("jerboaa");+            final GHUser karm = github.getUser("Karm");+            final GHUser zakkak = github.getUser("zakkak");++            ArrayList<GHUser> reviewers = new ArrayList<>();+            if (!authorEmail.contains("galder")) {+                reviewers.add(galderz);+            } else {+                pullRequest.addAssignees(Collections.singletonList(galderz));+            }+            if (!authorEmail.contains("sgehwolf") && !authorEmail.contains("jerboaa")) {+                reviewers.add(jerboaa);+            } else {+                pullRequest.addAssignees(Collections.singletonList(jerboaa));+            }+            if (!authorEmail.contains("karm")) {+                reviewers.add(karm);+            } else {+                pullRequest.addAssignees(Collections.singletonList(karm));+            }+            if (!authorEmail.contains("zakkak")) {+                reviewers.add(zakkak);+            } else {+                pullRequest.addAssignees(Collections.singletonList(zakkak));+            }+            pullRequest.requestReviewers(reviewers);++            info("Pull request " + pullRequest.getHtmlUrl() + " created");+        } catch (IOException e) {+            error(e.getMessage());+        }+    }++    /**+     * Returns current version of substratevm+     *+     * @return+     */+    private static String getCurrentVersion() {+        final Path substrateSuite = Path.of("substratevm", "mx.substratevm", "suite.py");+        String version = null;+        try {+            final List<String> lines = Files.readAllLines(substrateSuite);+            final Pattern versionPattern = Pattern.compile("\"version\" : \"([\\d.]+)\"");+            for (String line: lines) {+                final Matcher versionMatcher = versionPattern.matcher(line);+                if (versionMatcher.find()) {+                    version = versionMatcher.group(1);+                    break;+                }+            }+        } catch (IOException e) {+            e.printStackTrace();+        }+        info("Current version is " + version);+        return version;+    }++    /**+     * Calculates the new version based on {@code version} by bumping pico in major.minor.micro.pico+     *+     * @param version The current version+     * @return The new version+     */+    private static String getNewVersion(String version) {+        final Pattern versionPattern = Pattern.compile("(\\d+\\.\\d+\\.\\d+\\.)(\\d+)");+        final Matcher versionMatcher = versionPattern.matcher(version);+        boolean found = versionMatcher.find();+        assert found;+        int pico = Integer.parseInt(versionMatcher.group(2)) + 1;+        final String newVersion = versionMatcher.group(1) + pico;+        info("New version will be " + newVersion);+        return newVersion;+    }++    private void createGHRelease(String fullVersion) {+        GitHub github = connectToGitHub();+        try {+            final GHRepository repository = github.getRepository("graalvm/mandrel");+            final PagedIterable<GHMilestone> ghMilestones = repository.listMilestones(GHIssueState.OPEN);+            final GHMilestone milestone = ghMilestones.toList().stream().filter(m -> m.getTitle().equals(fullVersion)).findAny().orElse(null);+            String changelog = createChangelog(fullVersion, repository, milestone);+            if (dryRun) {+                warn("Skipping release due to --dry-run");+                info("Release body would look like");+                System.out.println(releaseMainBody(fullVersion, changelog));+                return;+            }+            final GHRelease ghRelease = repository.createRelease("mandrel-" + fullVersion)+                    .name("Mandrel " + fullVersion)+                    .prerelease(!suffix.equals("Final"))+                    .body(releaseMainBody(fullVersion, changelog))+                    .draft(true)+                    .create();+            uploadAssets(fullVersion, ghRelease);+            info("Created new draft release: " + ghRelease.getHtmlUrl());+            info("Please review and publish!");+            manageMilestones(repository, ghMilestones, milestone);+        } catch (IOException e) {+            e.printStackTrace();+        }+    }++    private void uploadAssets(String fullVersion, GHRelease ghRelease) throws IOException {+        final File archive = new File("mandrel-java11-linux-amd64-" + fullVersion + ".tar.gz");+        final File archiveSHA1 = new File("mandrel-java11-linux-amd64-" + fullVersion + ".tar.gz.sha1");+        if (!archive.exists()) {+            warn("Archive \"" + archive.getName() + "\" was not found. Skipping asset upload.");+            warn("Please upload asset manually.");+        } else if (!archiveSHA1.exists()) {+            warn("Archive sha1 \"" + archive.getName() + "\" was not found. Skipping asset upload.");+            warn("Please upload asset manually.");+        } else {+            info("Uploading " + archive.getName());+            ghRelease.uploadAsset(archive, "application/gzip");+            info("Uploaded " + archive.getName());+            info("Uploading " + archiveSHA1.getName());+            ghRelease.uploadAsset(archiveSHA1, "text/plain");+            info("Uploaded " + archiveSHA1.getName());+        }+    }++    private String releaseMainBody(String version, String changelog) {+        final Matcher majorMinorMicroMatcher = Pattern.compile("(\\d+\\.\\d+\\.\\d+).*").matcher(version);+        if (!majorMinorMicroMatcher.find()) {+            throw new RuntimeException("Version is in wrong format : " + version);+        }+        final String majorMinorMicro = majorMinorMicroMatcher.group(1);+        return "# Mandrel\n" ++                "\n" ++                "Mandrel " + version + " is a downstream distribution of the [GraalVM community edition " + majorMinorMicro + "](https://github.com/graalvm/graalvm-ce-builds/releases/tag/vm-" + majorMinorMicro + ").\n" ++                "Mandrel's main goal is to provide a `native-image` release specifically to support [Quarkus](https://quarkus.io).\n" ++                "The aim is to align the `native-image` capabilities from GraalVM with OpenJDK and Red Hat Enterprise Linux libraries to improve maintainability for native Quarkus applications.\n" ++                "\n" ++                "## How Does Mandrel Differ From Graal\n" ++                "\n" ++                "Mandrel releases are built from a code base derived from the upstream GraalVM code base, with only minor changes but some significant exclusions.\n" ++                "They support the same native image capability as GraalVM with no significant changes to functionality.\n" ++                "They do not include support for Polyglot programming via the Truffle interpreter and compiler framework.\n" ++                "In consequence, it is not possible to extend Mandrel by downloading languages from the Truffle language catalogue.\n" ++                "\n" ++                "Mandrel is also built slightly differently to GraalVM, using the standard OpenJDK project release of jdk11u.\n" ++                "This means it does not profit from a few small enhancements that Oracle have added to the version of OpenJDK used to build their own GraalVM downloads.\n" ++                "Most of these enhancements are to the JVMCI module that allows the Graal compiler to be run inside OpenJDK.\n" ++                "The others are small cosmetic changes to behaviour.\n" ++                "These enhancements may in some cases cause minor differences in the progress of native image generation.\n" ++                "They should not cause the resulting images themselves to execute in a noticeably different manner.\n" ++                "\n" ++                "### Prerequisites\n" ++                "\n" ++                "Mandrel's `native-image` depends on the following packages:\n" ++                "* glibc-devel\n" ++                "* zlib-devel\n" ++                "* gcc\n" ++                "* libffi-devel\n" ++                "\n" ++                "On Fedora/CentOS/RHEL they can be installed with:\n" ++                "```bash\n" ++                "dnf install glibc-devel zlib-devel gcc libffi-devel libstdc++-static\n" ++                "```\n" ++                "\n" ++                "Note the package might be called `glibc-static` instead of `libstdc++-static`.\n" ++                "\n" ++                "On Ubuntu-like systems with:\n" ++                "```bash\n" ++                "apt install gcc zlib1g-dev libffi-dev build-essential\n" ++                "```\n" ++                "\n" ++                "## Quick start\n" ++                "\n" ++                "```\n" ++                "$ tar -xf mandrel-java11-linux-amd64-" + version + ".tar.gz\n" ++                "$ export JAVA_HOME=\"$( pwd )/mandrel-java11-" + version + "\"\n" ++                "$ export GRAALVM_HOME=\"${JAVA_HOME}\"\n" ++                "$ export PATH=\"${JAVA_HOME}/bin:${PATH}\"\n" ++                "$ curl -O -J  https://code.quarkus.io/api/download\n" ++                "$ unzip code-with-quarkus.zip\n" ++                "$ cd code-with-quarkus/\n" ++                "$ ./mvnw package -Pnative\n" ++                "$ ./target/code-with-quarkus-1.0.0-SNAPSHOT-runner\n" ++                "```\n" ++                "\n" ++                "### Quarkus builder image\n" ++                "\n" ++                "The Quarkus builder image for this release is still being prepared, please try again later.\n" ++                "<!--\n" ++                "Mandrel Quarkus builder image can be used to build a Quarkus native Linux executable right away without any GRAALVM_HOME setup.\n" ++                "\n" ++                "```bash\n" ++                "curl -O -J  https://code.quarkus.io/api/download\n" ++                "unzip code-with-quarkus.zip\n" ++                "cd code-with-quarkus\n" ++                "        ./mvnw package -Pnative -Dquarkus.native.container-build=true -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-mandrel:" + version + "-java11\n" ++                "        ./target/code-with-quarkus-1.0.0-SNAPSHOT-runner\n" ++                "```\n" ++                "\n" ++                "One can use the builder image on Windows with Docker Desktop (mind `Resources-> File sharing` settings so as Quarkus project directory is mountable).\n" ++                "\n" ++                "```batchfile\n" ++                "powershell -c \"Invoke-WebRequest -OutFile quarkus.zip -Uri https://code.quarkus.io/api/download\"\n" ++                "powershell -c \"Expand-Archive -Path quarkus.zip -DestinationPath . -Force\n" ++                "cd code-with-quarkus\n" ++                "mvnw package -Pnative -Dquarkus.native.container-build=true -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-mandrel:" + version + "-java11\n" ++                "docker build -f src/main/docker/Dockerfile.native -t my-quarkus-mandrel-app .\n" ++                "        docker run -i --rm -p 8080:8080 my-quarkus-mandrel-app\n" ++                "```\n" ++                "-->\n" ++                changelog ++                "\n---\n" ++                "Mandrel " + version + "\n" ++                "OpenJDK used: 11.0.8+10\n";

So we need to either ensure that jbang will be run using the same OpenJDK used to build the release or we need to just add it as a parameter to the jbang script. Which one do you prefer? I find the latter safer.

zakkak

comment created time in 6 days

PullRequestReviewEvent

issue commentoracle/graal

native-image shell script requires a modded JDK to work.

HI @gilles-duboscq sorry for the late reply

Re 1.: So would a flag like --no-jlinking be welcome?

Re 2.:

So I need to remove all distributions and dependencies that depend on libraries built from source from the corresponding suite.py files and dynamically create them by defining mx_register_dynamic_suite_constituents in the corresponding suite modules. For LIBFFI those would be dist substratevm:SVM_HOSTED_NATIVE and dep truffle:libffi. I think I will start with this :)

Re 3.:

  • com.oracle.svm.junit is needed to run junit tests in a native image right? BTW why is --macro:junit missing from the releases?
graalvm-ce-java11-20.1.0/bin/native-image --macro:junit test
Error: Unknown name in option specification: macro:junit
Available macro options are:
    --macro:polyglot-launcher
    --macro:gu-launcher
    --macro:native-image-agent-library
    --macro:graalvm-native-binutil-launcher
    --macro:graalvm-native-clang-launcher
    --macro:truffle
    --macro:graalvm-native-ld-launcher
    --macro:js-launcher
    --macro:jvmcicompiler-library
    --macro:graalvm-native-clang++-launcher
    --macro:lli-launcher
    --macro:polyglot-library
    --macro:native-image-launcher
  • Do you think there is any way to remove the JLINE dependency without breaking things?
zakkak

comment created time in 7 days

delete branch zakkak/mx

delete branch : fix-ci

delete time in 7 days

pull request commentoracle/graal

CI: Use docker image for Elasticsearch ITs

Thanks to whoever did the re-run :) The failing test group was Data 6 and it looks fine now https://github.com/oracle/graal/pull/2797/checks?check_run_id=1118210794

zakkak

comment created time in 7 days

pull request commentoracle/graal

CI: Use docker image for Elasticsearch ITs

Quarkus 1.8.0 is now released, could someone trigger a re-run of the action please?

zakkak

comment created time in 7 days

pull request commentgraalvm/mx

Replace travis CI with GH actions

Yup, please see #227 :)

zakkak

comment created time in 7 days

delete branch zakkak/mx

delete branch : no-repo-test

delete time in 7 days

PullRequestReviewEvent

PR opened graalvm/mx

Fix pylint issues
+13 -12

0 comment

3 changed files

pr created time in 7 days

push eventzakkak/mx

Foivos Zakkak

commit sha 5be5c02c516c30b772316d446594e88ffcfcd0ed

Fix pylint issues

view details

push time in 7 days

pull request commentgraalvm/mx

Skip dependencies of native tasks with --no-native

Hi @gilles-duboscq sorry for the late reply.

The following quote from Doug Simon (https://github.com/oracle/graal/issues/2221#issuecomment-595208732) seems to indicate that this is not "much simpler in terms of code" though:

When creating the build schedule, all projects, dists, library and custom dependencies are considered as roots. The schedule then just sorts them such that dependencies are built before things that depend on them. There is currently no easy way to see that a dependency is a native-only dependency.

It's not clear to me how the use of a pseudo dependency would solve the issue. As Doug points out "all projects, dists, library and custom dependencies are considered as roots", thus even with a JARS pseudo-dependency I am ending up fetching LIBFFI_SOURCES for instance.

zakkak

comment created time in 7 days

push eventzakkak/mx

Foivos Zakkak

commit sha 21f9695872bbc499d2e7e2af409b9b2ed59b1b8a

Fix pylint issues

view details

push time in 7 days

push eventzakkak/mx

Foivos Zakkak

commit sha 8d721aa40270bf654d2aa6f0e49b6e15ea26401e

Fix pylint issues

view details

push time in 7 days

push eventzakkak/mx

Foivos Zakkak

commit sha a93418d898b0cc00bb866f170d14e5df8961db08

Fix pyling issues

view details

push time in 7 days

issue closedgraalvm/mx

Travis CI not running on master branch

Last run seems to be 3 years ago https://travis-ci.org/github/graalvm/mx/builds

Why is that? Could someone please re-enable it?

PS: I am volunteering to port the existing configuration to github-actions.

closed time in 7 days

zakkak

issue commentgraalvm/mx

Travis CI not running on master branch

mx is now using GH actions.

zakkak

comment created time in 7 days

create barnchzakkak/mx

branch : fix-ci

created branch time in 7 days

delete branch zakkak/mx

delete branch : gh-action

delete time in 7 days

pull request commentgraalvm/mx

Travis test running mx in a non-version-controlled directory

@dougxc @gilles-duboscq shall we close this?

zakkak

comment created time in 7 days

pull request commentgraalvm/mx

Replace travis CI with GH actions

@dougxc @RoxaBee any updates on this?

zakkak

comment created time in 7 days

delete branch zakkak/quarkus-images

delete branch : print-version-of-images

delete time in 7 days

push eventzakkak/mandrel-packaging

Foivos Zakkak

commit sha 7c5d9a6e24b6dbfaa84e76764eb6a14e0e597aee

Jbang the release process!

view details

push time in 7 days

Pull request review commentgraalvm/mandrel-packaging

Jbang the release process!

+# Mandrel release scripts++This folder contains a number of scripts targetting to ease the process of+creating new Mandrel releases.++## Step 0 (Prerequisities)++1. [Install jbang](https://github.com/jbangdev/jbang#installation)+2. Create a `~/.github` [property files](https://github-api.kohsuke.org/#Property_file) or set [environmental variables](https://github-api.kohsuke.org/#Environmental_variables)++We suggest [creating a personal access token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) and using the [property file approach](https://github-api.kohsuke.org/#Property_file) by creating a file `~/.github` with content `oauth=YOURTOKEN` ++## Step 1++First we invoke `./mandrel-release.java prepare` which is responsible for:++1. Marking the suite.py files as releases+2. Creating a new branch and commiting the changes there+3. Pushing the changes to a fork+4. Openning a PR++For usage information please run `./mandrel-release.java -h`++## Step 2++After the changes get approved and merged someone needs to tag the new commit+like this:++```+git checkout mandrel/20.1+git pull upstream mandrel/20.1+git tag -a mandrel-20.1.0.2.Final -m "mandrel-20.1.0.2.Final" -s+git push upstream mandrel-20.1.0.2.Final+```++NOTE: This can't be integrated into `post-mandrel-release.java` yet due to

Good catch, thanks!

zakkak

comment created time in 7 days

PullRequestReviewEvent

Pull request review commentgraalvm/mandrel-packaging

Jbang the release process!

+# Mandrel release scripts++This folder contains a number of scripts targetting to ease the process of+creating new Mandrel releases.++## Step 0 (Prerequisities)++1. [Install jbang](https://github.com/jbangdev/jbang#installation)+2. Create a `~/.github` [property files](https://github-api.kohsuke.org/#Property_file) or set [environmental variables](https://github-api.kohsuke.org/#Environmental_variables)++We suggest [creating a personal access token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) and using the [property file approach](https://github-api.kohsuke.org/#Property_file) by creating a file `~/.github` with content `oauth=YOURTOKEN` ++## Step 1++First we invoke `./mandrel-release.java prepare` which is responsible for:++1. Marking the suite.py files as releases+2. Creating a new branch and commiting the changes there+3. Pushing the changes to a fork+4. Openning a PR++For usage information please run `./mandrel-release.java -h`++## Step 2++After the changes get approved and merged someone needs to tag the new commit+like this:++```+git checkout mandrel/20.1+git pull upstream mandrel/20.1+git tag -a mandrel-20.1.0.2.Final -m "mandrel-20.1.0.2.Final" -s+git push upstream mandrel-20.1.0.2.Final+```++NOTE: This can't be integrated into `post-mandrel-release.java` yet due to+https://bugs.eclipse.org/bugs/show_bug.cgi?id=386908++## Step 3++Then `./mandrel-release.java release` must be invoked which is responsible for:++1. Creating a new GitHub release for the new tag including a changelog+2. Marking the suite.py files as non-releases+3. Bumping the version in suite.py files+4. Creating a new branch and commiting the changes there+5. Pushing the changes to a fork+6. Openning a PR+6. TODO Open PR for a new image on quarkus-images based on the new release+7. TODO send out emails to relevant lists +8. TODO? Send a Slack message++For usage information please run `./post-mandrel-release.java -h`

Good catch, thanks!

zakkak

comment created time in 7 days

PullRequestReviewEvent

push eventzakkak/mandrel-packaging

Foivos Zakkak

commit sha e3de5014c03fe79e47423acbd3482eccd1b039a0

Jbang the release process!

view details

push time in 7 days

pull request commentquarkusio/quarkus-images

Run image native-image --version

@cescoffier please have a look. This should help us avoid issues like https://github.com/quarkusio/quarkus-images/pull/100#discussion_r484978115 in the future

zakkak

comment created time in 8 days

push eventzakkak/quarkus-images

Foivos Zakkak

commit sha 33d98c169cebbcaf78a8974d3a089f8537766b1f

Run image native-image --version

view details

push time in 8 days

push eventzakkak/quarkus-images

Foivos Zakkak

commit sha 5de1ecdf9f80e421a679a20ac7f9e7753d141073

Run image native-image --version

view details

push time in 8 days

push eventzakkak/quarkus-images

Foivos Zakkak

commit sha f5a883c43e661ae449e09ecddd976fe9af64f96f

Run image native-image --version

view details

push time in 8 days

push eventzakkak/quarkus-images

Foivos Zakkak

commit sha 95eb179df4a5efd7699c1a9d5a25e20340a465f2

Run image native-image --version

view details

push time in 8 days

push eventzakkak/quarkus-images

Foivos Zakkak

commit sha d1468c6a40910c887d07a7775eade240755f2674

Run image native-image --version

view details

push time in 8 days

push eventzakkak/quarkus-images

Foivos Zakkak

commit sha 6e855f253b61ed05fb3a7327b767b5a3c3456bf9

Run image native-image --version

view details

push time in 8 days

push eventzakkak/quarkus-images

Foivos Zakkak

commit sha 0c824d9fffdf51eeed126407c0bd1f9904b23d5e

Run image native-image --version

view details

push time in 8 days

pull request commentgraalvm/mandrel-packaging

Jbang the release process!

Closes https://github.com/graalvm/mandrel/issues/55

zakkak

comment created time in 8 days

PR opened graalvm/mandrel-packaging

Reviewers
Jbang the release process! enhancement

This aims to ease the release process of Mandrel. @Karm please review and if it looks OK let's try it out with 20.1.0.3.Final :)

+636 -0

0 comment

2 changed files

pr created time in 8 days

push eventzakkak/mandrel-packaging

Foivos Zakkak

commit sha 6729fbee92d71b6d700c09ccf0f26c1e69f2081d

Jbang the release process!

view details

push time in 8 days

push eventzakkak/quarkus-images

Foivos Zakkak

commit sha ac389417d53b522a0d60c7591bcf6c1a1587af74

Run image native-image --version

view details

push time in 8 days

issue commentgraalvm/mandrel

Mandrel 20.2 (and master) Warning: Using an older version of the labsjdk-11

@zakkak FYI: I'm working on a backport of oracle@d7b5904

Nice, thanks for the heads up!

Karm

comment created time in 8 days

push eventzakkak/quarkus-images

Foivos Zakkak

commit sha 22efca7c58c76ec85d6de8ede7b479490a0e1cd8

Run image native-image --version

view details

push time in 8 days

create barnchzakkak/quarkus-images

branch : print-version-of-images

created branch time in 8 days

delete branch zakkak/mandrel-packaging

delete branch : alignment-script

delete time in 8 days

create barnchzakkak/mandrel-packaging

branch : alignment-script

created branch time in 8 days

pull request commentjbangdev/jbang

Fixes some typos in readme.adoc

@maxandersen are you sure you merged this? It doesn't look so :)

zakkak

comment created time in 8 days

create barnchzakkak/jbang

branch : fix-typos

created branch time in 8 days

delete branch zakkak/jbang

delete branch : fix-typos

delete time in 8 days

Pull request review commentjbangdev/jbang

Implemented local catalogs

  public class AliasUtil { 	public static final String JBANG_CATALOG_JSON = "jbang-catalog.json";+	public static final String JBANG_DOT_DIR = ".jbang";  	private static final String GITHUB_URL = "https://github.com/"; 	private static final String GITLAB_URL = "https://gitlab.com/"; 	private static final String BITBUCKET_URL = "https://bitbucket.org/";  	private static final String JBANG_CATALOG_REPO = "jbang-catalog"; +	static Map<Path, Aliases> catalogCache = new HashMap<>();+ 	public static class Alias { 		@SerializedName(value = "script-ref", alternate = { "scriptRef" }) 		public final String scriptRef; 		public final String description; 		public final List<String> arguments; 		public final Map<String, String> properties;+		public transient Aliases aliases; -		public Alias(String scriptRef, String description, List<String> arguments, Map<String, String> properties) {+		public Alias(String scriptRef, String description, List<String> arguments, Map<String, String> properties,+				Aliases aliases) { 			this.scriptRef = scriptRef; 			this.description = description; 			this.arguments = arguments; 			this.properties = properties;+			this.aliases = aliases;+		}++		/**+		 * This method returns the scriptRef of the Alias with all contextual modifiers+		 * like baseRefs and current working directories applied.+		 */+		public String resolve(Path cwd) {+			if (cwd == null) {+				cwd = getCwd();+			}+			String baseRef = aliases.baseRef;+			String ref = scriptRef;+			if (baseRef != null && !isAbsoluteRef(ref)) {+				if (!baseRef.endsWith("/")) {+					baseRef += "/";+				}+				if (ref.startsWith("./")) {+					baseRef += ref.substring(2);+				} else {+					baseRef += ref;+				}+				ref = baseRef;+			}+			if (!isAbsoluteRef(ref)) {+				Path script = scriptParent(aliases.catalogFile).resolve(ref);+				if (script.startsWith(cwd) || cwd.startsWith(script)+						|| !cwd.relativize(script).startsWith("../../..")) {

Nice, thanks for the explanation @quintesse :)

quintesse

comment created time in 8 days

PullRequestReviewEvent
more