profile
viewpoint
Danny Thomas DanielThomas @Netflix Los Gatos, California Developer Productivity @ Netflix

DanielThomas/ec2gaming 100

EC2 Gaming on macOS with Steam In-Home Streaming

DanielThomas/dotfiles 13

Opinionated dotfiles for Mac OS with zsh

DanielThomas/groundhog 8

Simple, high performance HTTP capture and replay

DanielThomas/advent 0

Advent of Code

DanielThomas/artifactory-user-plugins 0

Sample Artifactory User Plugins

DanielThomas/clouddriver 0

read and write operations across cloud providers

DanielThomas/deck 0

Management UI for Spinnaker

DanielThomas/frigga 0

Utilities for working with Asgard named objects

DanielThomas/genie 0

Federated Job Execution Engine

DanielThomas/go-left-pad 0

String left pad

push eventnebula-plugins/nebula-kotlin-plugin

Danny Thomas

commit sha 97a8ff4acca6f3d98a07f3274a82402ab6aa759d

Update README.md

view details

push time in 2 days

issue closednebula-plugins/nebula-release-plugin

Tag support for v2020-05-07-1 format

Currently we are trying to adopt the v2020-05-07-1 format (last digit stands for build number for the build day)for tag, and by reading through the documentation and trying, looks like it's not supportable:

  1. v2020.5.7 is workable, but it strips out all the leading zero. Behind the scene, everything is still treated as a number.
  2. v2020.5.7-rc.1 is kind of a work-around, but not quite what we want, and it only works with candidate task, not final task.

Is it configurable to add the tag format of v2020-05-07-1? If yes, will it be treated as valid date (year, month, day), or still just number?

Thanks

closed time in 15 days

kzhu2012

issue commentnebula-plugins/nebula-release-plugin

Tag support for v2020-05-07-1 format

The plugin supports semver and is about managing release lines and release promotion, which isn't what you'd be doing with that kind of scheme. If you want <timestamp>.<build> that feels something suited for continuous releases - use your own logic to set the version and always publish, and treat every build as a release candidate.

kzhu2012

comment created time in 15 days

issue commentjetty-project/jetty-alpn

jetty-alpn incompatible with OpenJDK 8u251

Certainly is in the Zulu distribution:

/Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home $ unzip -l src.zip | grep sun.security.ssl
     8529  04-07-2020 00:49   sun/security/ssl/Alerts.java
     6059  04-07-2020 00:49   sun/security/ssl/ALPNExtension.java
     4952  09-19-2018 10:02   sun/security/ssl/AppInputStream.java
     5413  09-19-2018 10:02   sun/security/ssl/AppOutputStream.java
     5824  09-19-2018 10:02   sun/security/ssl/Authenticator.java
    18181  09-19-2018 10:02   sun/security/ssl/BaseSSLSocketImpl.java
     5080  09-19-2018 10:02   sun/security/ssl/ByteBufferInputStream.java
    40283  09-19-2018 10:02   sun/security/ssl/CipherBox.java
    83282  04-07-2020 00:49   sun/security/ssl/CipherSuite.java
     5750  09-19-2018 10:02   sun/security/ssl/CipherSuiteList.java
    75957  04-07-2020 00:49   sun/security/ssl/ClientHandshaker.java
     6779  09-19-2018 10:02   sun/security/ssl/Debug.java
     3437  09-19-2018 10:02   sun/security/ssl/DHClientKeyExchange.java
    25857  03-02-2020 03:31   sun/security/ssl/DHCrypt.java
     2729  09-19-2018 10:02   sun/security/ssl/ECDHClientKeyExchange.java
     5800  09-19-2018 10:02   sun/security/ssl/ECDHCrypt.java
    14518  03-02-2020 03:31   sun/security/ssl/EllipticCurvesExtension.java
     3483  09-19-2018 10:02   sun/security/ssl/EllipticPointFormatsExtension.java
     7608  09-19-2018 10:02   sun/security/ssl/EngineArgs.java
    14947  09-19-2018 10:02   sun/security/ssl/EngineInputRecord.java
    11453  09-19-2018 10:02   sun/security/ssl/EngineOutputRecord.java
     7751  09-19-2018 10:02   sun/security/ssl/EngineWriter.java
     3958  09-19-2018 10:02   sun/security/ssl/EphemeralKeyManager.java
     2443  04-07-2020 00:19   sun/security/ssl/ExtendedMasterSecretExtension.java
     4395  04-07-2020 00:49   sun/security/ssl/ExtensionType.java
    14555  09-19-2018 10:02   sun/security/ssl/HandshakeHash.java
     6748  09-19-2018 10:02   sun/security/ssl/HandshakeInStream.java
    71091  04-07-2020 00:49   sun/security/ssl/HandshakeMessage.java
     7346  09-19-2018 10:02   sun/security/ssl/HandshakeOutStream.java
    58810  04-07-2020 00:49   sun/security/ssl/Handshaker.java
    33905  10-03-2018 12:35   sun/security/ssl/HandshakeStateManager.java
     1635  09-19-2018 10:02   sun/security/ssl/HelloExtension.java
     5794  04-07-2020 00:49   sun/security/ssl/HelloExtensions.java
    30620  09-19-2018 10:02   sun/security/ssl/InputRecord.java
    15554  09-19-2018 10:02   sun/security/ssl/JsseJce.java
     5249  09-19-2018 10:02   sun/security/ssl/KerberosClientKeyExchange.java
     4897  09-19-2018 10:02   sun/security/ssl/KeyManagerFactoryImpl.java
    18219  09-19-2018 10:02   sun/security/ssl/krb5/KerberosClientKeyExchangeImpl.java
    10672  09-19-2018 10:02   sun/security/ssl/krb5/KerberosPreMasterSecret.java
     4081  09-19-2018 10:02   sun/security/ssl/krb5/Krb5ProxyImpl.java
     4578  09-19-2018 10:02   sun/security/ssl/Krb5Helper.java
     2724  09-19-2018 10:02   sun/security/ssl/Krb5Proxy.java
     5242  09-19-2018 10:02   sun/security/ssl/MAC.java
    21353  09-19-2018 10:02   sun/security/ssl/OutputRecord.java
     5261  09-19-2018 10:02   sun/security/ssl/ProtocolList.java
     6762  09-19-2018 10:02   sun/security/ssl/ProtocolVersion.java
     2952  09-19-2018 10:02   sun/security/ssl/RandomCookie.java
     5391  09-19-2018 10:02   sun/security/ssl/Record.java
     4188  09-19-2018 10:02   sun/security/ssl/RenegotiationInfoExtension.java
    10679  09-19-2018 10:02   sun/security/ssl/RSAClientKeyExchange.java
     7263  04-07-2020 00:49   sun/security/ssl/RSASignature.java
    85665  04-07-2020 00:49   sun/security/ssl/ServerHandshaker.java
    10921  09-19-2018 10:02   sun/security/ssl/ServerNameExtension.java
     4324  07-10-2019 10:59   sun/security/ssl/SessionId.java
     4751  09-19-2018 10:02   sun/security/ssl/SignatureAlgorithmsExtension.java
    18714  03-02-2020 03:31   sun/security/ssl/SignatureAndHashAlgorithm.java
     9494  09-19-2018 10:02   sun/security/ssl/SSLAlgorithmConstraints.java
     8625  06-28-2019 07:16   sun/security/ssl/SSLAlgorithmDecomposer.java
    54791  07-10-2019 10:59   sun/security/ssl/SSLContextImpl.java
    78305  04-07-2020 00:49   sun/security/ssl/SSLEngineImpl.java
     4212  09-19-2018 10:02   sun/security/ssl/SSLServerSocketFactoryImpl.java
    12533  04-07-2020 00:49   sun/security/ssl/SSLServerSocketImpl.java
     8125  07-10-2019 10:59   sun/security/ssl/SSLSessionContextImpl.java
    29295  04-04-2020 01:12   sun/security/ssl/SSLSessionImpl.java
     7317  09-19-2018 10:02   sun/security/ssl/SSLSocketFactoryImpl.java
   104269  04-07-2020 00:49   sun/security/ssl/SSLSocketImpl.java
     9316  04-07-2020 00:49   sun/security/ssl/SunJSSE.java
    15501  09-19-2018 10:02   sun/security/ssl/SunX509KeyManagerImpl.java
     6731  07-10-2019 10:59   sun/security/ssl/TrustManagerFactoryImpl.java
    14664  07-10-2019 10:59   sun/security/ssl/TrustStoreManager.java
     1988  09-19-2018 10:02   sun/security/ssl/UnknownExtension.java
     4363  09-19-2018 10:02   sun/security/ssl/Utilities.java
    34094  09-19-2018 10:02   sun/security/ssl/X509KeyManagerImpl.java
    18148  03-02-2020 03:31   sun/security/ssl/X509TrustManagerImpl.java
DanielThomas

comment created time in 16 days

push eventnebula-plugins/nebula-kotlin-plugin

Danny Thomas

commit sha eb3f5de55f60b97f22474f5639f2285a36d08347

Update README.md

view details

push time in 21 days

push eventnebula-plugins/nebula-kotlin-plugin

Danny Thomas

commit sha d35d85ad9aa67b420a9539031089508c63f88075

Update README.md

view details

push time in 21 days

push eventnebula-plugins/nebula-kotlin-plugin

Danny Thomas

commit sha 1c6aaba5cd62515a8ab4847f86b4f5e35fd24be8

Update README.md

view details

push time in 21 days

push eventnebula-plugins/nebula-kotlin-plugin

Danny Thomas

commit sha b6b64667b0d2978f2d495a7a8d6e76c2601782ab

Update README.md

view details

push time in 21 days

push eventnebula-plugins/nebula-kotlin-plugin

Danny Thomas

commit sha b86a89797cc6624c1c7268a7092b41aaf9f52c3a

Update README.md

view details

push time in 21 days

push eventnebula-plugins/nebula-kotlin-plugin

Danny Thomas

commit sha 1d12e07b591878a73c4df78480ad083e018a3a7f

Update README.md

view details

push time in 21 days

issue closednebula-plugins/nebula-kotlin-plugin

Warning on kotlin 1.3.70

plugins { id 'nebula.kotlin' version '1.3.70'

The 'org.jetbrains.kotlin.platform.*' plugins are deprecated and will no longer be available in Kotlin 1.4. Please migrate the project to the 'org.jetbrains.kotlin.multiplatform' plugin. See: https://kotlinlang.org/docs/reference/building-mpp-with-gradle.html

closed time in 21 days

mattbroekhuis

issue commentnebula-plugins/nebula-kotlin-plugin

Warning on kotlin 1.3.70

I finally got a chance to take a look at the multiplatform plugin. It supports all of the ergonomic features we had such as JVM targets and library version management and is a complete migration from the existing plugin. We'll continue to release 1.x releases of this plugin but will retire it otherwise and recommend migrating to the multiplatform plugin

mattbroekhuis

comment created time in 21 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

 private void fingerprintZipEntries(String parentName, List<FileSystemLocationFin             }             String fullName = parentName.isEmpty() ? zipEntry.getName() : parentName + "/" + zipEntry.getName();             if (isZipFile(zipEntry.getName())) {-                fingerprintZipEntries(fullName, fingerprints, new StreamZipInput(zipEntry.getInputStream()));+                fingerprintZipEntries(fullName, rootParentName, fingerprints, new StreamZipInput(zipEntry.getInputStream()));+            } else if (isManifestFile(zipEntry.getName())) {+                fingerprintZipEntryContentWithFallback(rootParentName, fingerprints, zipEntry, fullName, this::hashManifest);+            } else if (isManifestPropertyFile(zipEntry.getName())) {+                fingerprintZipEntryContentWithFallback(rootParentName, fingerprints, zipEntry, fullName, this::hashProperties);             } else {-                HashCode hash = resourceHasher.hash(zipEntry);-                if (hash != null) {-                    fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+                fingerprintZipEntry(zipEntry, fullName, fingerprints);+            }+        }+    }++    private void fingerprintZipEntry(ZipEntry zipEntry, String fullName, List<FileSystemLocationFingerprint> fingerprints) throws IOException {+        HashCode hash = resourceHasher.hash(zipEntry);+        if (hash != null) {+            fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+        }+    }++    private void fingerprintZipEntryContentWithFallback(String rootParentName, List<FileSystemLocationFingerprint> fingerprints, ZipEntry zipEntry, String fullName, ByteContentHasher hasher) throws IOException {+        // JdkZipEntry can be backed by a stream, so we assume that getInputStream is a single shot and read the manifest to a byte array so we can fallback should content hashing fail+        byte[] entryBytes = ByteStreams.toByteArray(zipEntry.getInputStream());+        try {+            HashCode hash = hasher.hash(entryBytes);+            if (hash != null) {+                fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+            }+        } catch (Exception e) {+            LOGGER.warn("Could not load fingerprint " + rootParentName + ". Falling back to regular fingerprinting", e);

See https://github.com/gradle/gradle/pull/11937/commits/271389aedd8bd52be788f9b15d465061d0eafa93

DanielThomas

comment created time in 21 days

push eventDanielThomas/gradle

Danny Thomas

commit sha 271389aedd8bd52be788f9b15d465061d0eafa93

Make fallback handling more granular This handles fallback for attributes and properties only if an IOException is thrown. It also loosens the outer zip fallback to the same to avoid swallowing bugs in normalization or fingerprinting code (FileZipInput throws UncheckedIOExceptions). Signed-off-by: Danny Thomas <dannyt@netflix.com>

view details

push time in 21 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

 private void fingerprintZipEntries(String parentName, List<FileSystemLocationFin             }             String fullName = parentName.isEmpty() ? zipEntry.getName() : parentName + "/" + zipEntry.getName();             if (isZipFile(zipEntry.getName())) {-                fingerprintZipEntries(fullName, fingerprints, new StreamZipInput(zipEntry.getInputStream()));+                fingerprintZipEntries(fullName, rootParentName, fingerprints, new StreamZipInput(zipEntry.getInputStream()));+            } else if (isManifestFile(zipEntry.getName())) {+                fingerprintZipEntryContentWithFallback(rootParentName, fingerprints, zipEntry, fullName, this::hashManifest);+            } else if (isManifestPropertyFile(zipEntry.getName())) {+                fingerprintZipEntryContentWithFallback(rootParentName, fingerprints, zipEntry, fullName, this::hashProperties);             } else {-                HashCode hash = resourceHasher.hash(zipEntry);-                if (hash != null) {-                    fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+                fingerprintZipEntry(zipEntry, fullName, fingerprints);+            }+        }+    }++    private void fingerprintZipEntry(ZipEntry zipEntry, String fullName, List<FileSystemLocationFingerprint> fingerprints) throws IOException {+        HashCode hash = resourceHasher.hash(zipEntry);+        if (hash != null) {+            fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+        }+    }++    private void fingerprintZipEntryContentWithFallback(String rootParentName, List<FileSystemLocationFingerprint> fingerprints, ZipEntry zipEntry, String fullName, ByteContentHasher hasher) throws IOException {+        // JdkZipEntry can be backed by a stream, so we assume that getInputStream is a single shot and read the manifest to a byte array so we can fallback should content hashing fail+        byte[] entryBytes = ByteStreams.toByteArray(zipEntry.getInputStream());+        try {+            HashCode hash = hasher.hash(entryBytes);+            if (hash != null) {+                fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+            }+        } catch (Exception e) {+            LOGGER.warn("Could not load fingerprint " + rootParentName + ". Falling back to regular fingerprinting", e);

To make all of this code safer it likely makes sense to make the internal exception handling far more granular so a malformed properties/manifest/zip file throws a specific exception that allows a fallback, with all other failures bubbling up. I'll look at that now.

DanielThomas

comment created time in 21 days

push eventDanielThomas/gradle

Danny Thomas

commit sha 60ab05f5442e426413014ec94837f45c4a9cae1e

Decode properties files as UTF-8, falling back to ISO 8859-1 on failure Allows UTF-8 property keys to be ignored. Implemented with a backport of PropertyResourceBundleCharset for Java 8 compatibility. Signed-off-by: Danny Thomas <dannyt@netflix.com>

view details

push time in 22 days

push eventDanielThomas/gradle

Danny Thomas

commit sha b232548c192d842b7dc6e6f6e36e6e83953c6cd6

Decode properties files as UTF-8, falling back to ISO 8859-1 on failure Allows UTF-8 property keys to be ignored. Implemented with a backport of PropertyResourceBundleCharset for Java 8 compatibility.

view details

push time in 22 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

 class ConfigureRuntimeClasspathNormalizationIntegrationTest extends AbstractInte             return this         } +        ProjectWithRuntimeClasspathNormalization withManifestAttributesIgnored() {+            root.file('build.gradle') << """+                normalization {+                    runtimeClasspath {+                        ignoreManifestAttribute "Implementation-Version"+                    }

Good to know - in that case I'll do what PropertyResourceBundle does - wire up an {{CharsetDecoder}} that attempts UTF-8 decoding by default and with a fallback to ISO 8859-1.

DanielThomas

comment created time in 22 days

push eventDanielThomas/gradle

Danny Thomas

commit sha d1210eea3a4a3cb0ea1053ae157654ec4192de41

Add missing incubating annotation Signed-off-by: Danny Thomas <dannyt@netflix.com>

view details

push time in 22 days

push eventDanielThomas/gradle

Danny Thomas

commit sha 55b312f74dfc73e3adc9a896adde7f42d3b633aa

Latest changes for second code review feedback Will rebase later once the changes are complete. Signed-off-by: Danny Thomas <dannyt@netflix.com>

view details

push time in 22 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

 private ZipEntryRelativePath(ZipEntry zipEntry) {         }     } -    private HashCode hashMalformedZip(RegularFileSnapshot zipFileSnapshot, Exception e) {-        LOGGER.debug("Malformed archive '{}'. Falling back to full content hash instead of entry hashing.", zipFileSnapshot.getName(), e);-        return zipFileSnapshot.getHash();+    private static class MapEntry implements Factory<String[]>, Comparable<MapEntry> {

Nope, looks fine. Dropped.

DanielThomas

comment created time in 22 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

 import org.gradle.api.internal.changedetection.state.IgnoringResourceFilter; import org.gradle.api.internal.changedetection.state.ResourceFilter; +import java.util.Locale;+import java.util.concurrent.atomic.AtomicBoolean;+ public class DefaultRuntimeClasspathNormalization implements RuntimeClasspathNormalizationInternal {     private final ImmutableSet.Builder<String> ignoresBuilder = ImmutableSet.builder();+    private final ImmutableSet.Builder<String> attributeIgnoresBuilder = ImmutableSet.builder();+    private final ImmutableSet.Builder<String> propertiesIgnoresBuilder = ImmutableSet.builder();+    private AtomicBoolean evaluated = new AtomicBoolean();     private ResourceFilter resourceFilter;+    private ResourceFilter attributeResourceFilter;+    private ResourceFilter propertyResourceFilter;      @Override     public void ignore(String pattern) {-        if (resourceFilter != null) {-            throw new GradleException("Cannot configure runtime classpath normalization after execution started.");-        }+        checkNotEvaluated();         ignoresBuilder.add(pattern);     }      @Override-    public ResourceFilter getResourceFilter() {-        if (resourceFilter == null) {+    public void ignoreManifestAttribute(String pattern) {+        checkNotEvaluated();+        attributeIgnoresBuilder.add(pattern.toLowerCase(Locale.ROOT));+    }++    @Override+    public void ignoreManifestProperty(String pattern) {+        checkNotEvaluated();+        propertiesIgnoresBuilder.add(pattern);+    }++    @Override+    public RuntimeClasspathResourceFilters getResourceFilters() {

It allows one constructor argument instead of three for values that should be carried together - there's at least three classes along that path that already have very long method/constructor argument lists.

DanielThomas

comment created time in 22 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

     private final StringInterner stringInterner;     private final HashCode zipHasherConfigurationHash; -    private ClasspathFingerprintingStrategy(String identifier, NonJarFingerprintingStrategy nonZipFingerprintingStrategy, ResourceHasher classpathResourceHasher, ResourceFilter classpathResourceFilter, ResourceSnapshotterCacheService cacheService, StringInterner stringInterner) {+    private ClasspathFingerprintingStrategy(String identifier, NonJarFingerprintingStrategy nonZipFingerprintingStrategy, ResourceHasher classpathResourceHasher, ClasspathResourceFilters classpathResourceFilters, ResourceSnapshotterCacheService cacheService, StringInterner stringInterner) {         super(identifier);         this.nonZipFingerprintingStrategy = nonZipFingerprintingStrategy;-        this.classpathResourceFilter = classpathResourceFilter;+        this.classpathResourceFilter = classpathResourceFilters.getResourceFilter();         this.classpathResourceHasher = classpathResourceHasher;         this.cacheService = cacheService;         this.stringInterner = stringInterner;-        this.zipHasher = new ZipHasher(classpathResourceHasher, classpathResourceFilter);+        if (classpathResourceFilters instanceof RuntimeClasspathResourceFilters) {+            RuntimeClasspathResourceFilters runtimeClasspathResourceFilters = (RuntimeClasspathResourceFilters) classpathResourceFilters;+            this.zipHasher = new ZipHasher(classpathResourceHasher, classpathResourceFilter, runtimeClasspathResourceFilters.getManifestAttributeResourceFilter(), runtimeClasspathResourceFilters.getManifestPropertyResourceFilter());+        } else {+            this.zipHasher = new ZipHasher(classpathResourceHasher, classpathResourceFilter);+        }

Or are you suggesting RuntimeClasspathResourceFilters is unnecessary and ClasspathResourceFilters should carry it with the FILTER_NOTHING filters for classpath normalization?

DanielThomas

comment created time in 22 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

     private final StringInterner stringInterner;     private final HashCode zipHasherConfigurationHash; -    private ClasspathFingerprintingStrategy(String identifier, NonJarFingerprintingStrategy nonZipFingerprintingStrategy, ResourceHasher classpathResourceHasher, ResourceFilter classpathResourceFilter, ResourceSnapshotterCacheService cacheService, StringInterner stringInterner) {+    private ClasspathFingerprintingStrategy(String identifier, NonJarFingerprintingStrategy nonZipFingerprintingStrategy, ResourceHasher classpathResourceHasher, ClasspathResourceFilters classpathResourceFilters, ResourceSnapshotterCacheService cacheService, StringInterner stringInterner) {         super(identifier);         this.nonZipFingerprintingStrategy = nonZipFingerprintingStrategy;-        this.classpathResourceFilter = classpathResourceFilter;+        this.classpathResourceFilter = classpathResourceFilters.getResourceFilter();         this.classpathResourceHasher = classpathResourceHasher;         this.cacheService = cacheService;         this.stringInterner = stringInterner;-        this.zipHasher = new ZipHasher(classpathResourceHasher, classpathResourceFilter);+        if (classpathResourceFilters instanceof RuntimeClasspathResourceFilters) {+            RuntimeClasspathResourceFilters runtimeClasspathResourceFilters = (RuntimeClasspathResourceFilters) classpathResourceFilters;+            this.zipHasher = new ZipHasher(classpathResourceHasher, classpathResourceFilter, runtimeClasspathResourceFilters.getManifestAttributeResourceFilter(), runtimeClasspathResourceFilters.getManifestPropertyResourceFilter());+        } else {+            this.zipHasher = new ZipHasher(classpathResourceHasher, classpathResourceFilter);+        }

The ClasspathResourceFilters wrapper is to avoid constructor argument blow out where this is passed through as a depenency - VirtualFileSystemServices and DefaultClasspathFingerprinter specifically. Would you prefer we add two additional constructor parameters (and later more should normalization be further enhanced)?

DanielThomas

comment created time in 22 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

 private ZipEntryRelativePath(ZipEntry zipEntry) {         }     } -    private HashCode hashMalformedZip(RegularFileSnapshot zipFileSnapshot, Exception e) {-        LOGGER.debug("Malformed archive '{}'. Falling back to full content hash instead of entry hashing.", zipFileSnapshot.getName(), e);-        return zipFileSnapshot.getHash();+    private static class MapEntry implements Factory<String[]>, Comparable<MapEntry> {

Oops, yeah forgot to drop that Factory.

I'm really sure there's a another reason this was necessary, let me take a look.

DanielThomas

comment created time in 22 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

 private void fingerprintZipEntries(String parentName, List<FileSystemLocationFin             }             String fullName = parentName.isEmpty() ? zipEntry.getName() : parentName + "/" + zipEntry.getName();             if (isZipFile(zipEntry.getName())) {-                fingerprintZipEntries(fullName, fingerprints, new StreamZipInput(zipEntry.getInputStream()));+                fingerprintZipEntries(fullName, rootParentName, fingerprints, new StreamZipInput(zipEntry.getInputStream()));+            } else if (isManifestFile(zipEntry.getName())) {+                fingerprintZipEntryContentWithFallback(rootParentName, fingerprints, zipEntry, fullName, this::hashManifest);+            } else if (isManifestPropertyFile(zipEntry.getName())) {+                fingerprintZipEntryContentWithFallback(rootParentName, fingerprints, zipEntry, fullName, this::hashProperties);             } else {-                HashCode hash = resourceHasher.hash(zipEntry);-                if (hash != null) {-                    fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+                fingerprintZipEntry(zipEntry, fullName, fingerprints);+            }+        }+    }++    private void fingerprintZipEntry(ZipEntry zipEntry, String fullName, List<FileSystemLocationFingerprint> fingerprints) throws IOException {+        HashCode hash = resourceHasher.hash(zipEntry);+        if (hash != null) {+            fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+        }+    }++    private void fingerprintZipEntryContentWithFallback(String rootParentName, List<FileSystemLocationFingerprint> fingerprints, ZipEntry zipEntry, String fullName, ByteContentHasher hasher) throws IOException {+        // JdkZipEntry can be backed by a stream, so we assume that getInputStream is a single shot and read the manifest to a byte array so we can fallback should content hashing fail+        byte[] entryBytes = ByteStreams.toByteArray(zipEntry.getInputStream());+        try {+            HashCode hash = hasher.hash(entryBytes);+            if (hash != null) {+                fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+            }+        } catch (Exception e) {+            LOGGER.warn("Could not load fingerprint " + rootParentName + ". Falling back to regular fingerprinting", e);

Except you do - you have no control over the incoming archives on the classpath, you're not just normalizing the jars your projects are producing. This accomodates the very real possibility of a malformed manifest or properties file in an archive that otherwise would not be a problem at all either in the build or at runtime. ZipHasher has an identical fallback for malformed zip archives, I'm doing exactly the same thing here:

https://github.com/gradle/gradle/blob/fd00959c27bdb4e940a0066eb462c87a2868a69e/subprojects/core/src/main/java/org/gradle/api/internal/changedetection/state/ZipHasher.java#L91-L93

If the fallback was to ignore the file completely I'd agree but we fallback to the current behaviour and hash the file contents.

DanielThomas

comment created time in 22 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

 class ConfigureRuntimeClasspathNormalizationIntegrationTest extends AbstractInte             return this         } +        ProjectWithRuntimeClasspathNormalization withManifestAttributesIgnored() {+            root.file('build.gradle') << """+                normalization {+                    runtimeClasspath {+                        ignoreManifestAttribute "Implementation-Version"+                    }

The reason for this pull request is to allow us to adopt the build cache - without properties support this the normalization doesn't help us. Happy to leave out those convenience methods, but this is more about META-INF, not just the manifest attributes.

I believe properties files only support ISO 8859-1 encoding. Unicode characters are only supported by escapes?

DanielThomas

comment created time in 22 days

pull request commentgradle/gradle

Normalize META-INF manifest attributes and properties files

@lptr this is ready for another once over. I've resolved the comments I either addressed with changes or with feedback. Let me know if there's anything else you'd still like looked at.

Left one comment open for discussion.

DanielThomas

comment created time in 23 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

     private final StringInterner stringInterner;     private final HashCode zipHasherConfigurationHash; -    private ClasspathFingerprintingStrategy(String identifier, NonJarFingerprintingStrategy nonZipFingerprintingStrategy, ResourceHasher classpathResourceHasher, ResourceFilter classpathResourceFilter, ResourceSnapshotterCacheService cacheService, StringInterner stringInterner) {+    private ClasspathFingerprintingStrategy(String identifier, NonJarFingerprintingStrategy nonZipFingerprintingStrategy, ResourceHasher classpathResourceHasher, ClasspathResourceFilters classpathResourceFilters, ResourceSnapshotterCacheService cacheService, StringInterner stringInterner) {         super(identifier);         this.nonZipFingerprintingStrategy = nonZipFingerprintingStrategy;-        this.classpathResourceFilter = classpathResourceFilter;+        this.classpathResourceFilter = classpathResourceFilters.getResourceFilter();         this.classpathResourceHasher = classpathResourceHasher;         this.cacheService = cacheService;         this.stringInterner = stringInterner;-        this.zipHasher = new ZipHasher(classpathResourceHasher, classpathResourceFilter);+        if (classpathResourceFilters instanceof RuntimeClasspathResourceFilters) {+            RuntimeClasspathResourceFilters runtimeClasspathResourceFilters = (RuntimeClasspathResourceFilters) classpathResourceFilters;+            this.zipHasher = new ZipHasher(classpathResourceHasher, classpathResourceFilter, runtimeClasspathResourceFilters.getManifestAttributeResourceFilter(), runtimeClasspathResourceFilters.getManifestPropertyResourceFilter());+        } else {+            this.zipHasher = new ZipHasher(classpathResourceHasher, classpathResourceFilter);+        }

I think ClasspathResourceFilters is still useful, given the other classes in that object graph that needs to pass these around and avoids needing currently three, and in future possibly more filter types.

DanielThomas

comment created time in 23 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

     private final StringInterner stringInterner;     private final HashCode zipHasherConfigurationHash; -    private ClasspathFingerprintingStrategy(String identifier, NonJarFingerprintingStrategy nonZipFingerprintingStrategy, ResourceHasher classpathResourceHasher, ResourceFilter classpathResourceFilter, ResourceSnapshotterCacheService cacheService, StringInterner stringInterner) {+    private ClasspathFingerprintingStrategy(String identifier, NonJarFingerprintingStrategy nonZipFingerprintingStrategy, ResourceHasher classpathResourceHasher, ClasspathResourceFilters classpathResourceFilters, ResourceSnapshotterCacheService cacheService, StringInterner stringInterner) {         super(identifier);         this.nonZipFingerprintingStrategy = nonZipFingerprintingStrategy;-        this.classpathResourceFilter = classpathResourceFilter;+        this.classpathResourceFilter = classpathResourceFilters.getResourceFilter();         this.classpathResourceHasher = classpathResourceHasher;         this.cacheService = cacheService;         this.stringInterner = stringInterner;-        this.zipHasher = new ZipHasher(classpathResourceHasher, classpathResourceFilter);+        if (classpathResourceFilters instanceof RuntimeClasspathResourceFilters) {+            RuntimeClasspathResourceFilters runtimeClasspathResourceFilters = (RuntimeClasspathResourceFilters) classpathResourceFilters;+            this.zipHasher = new ZipHasher(classpathResourceHasher, classpathResourceFilter, runtimeClasspathResourceFilters.getManifestAttributeResourceFilter(), runtimeClasspathResourceFilters.getManifestPropertyResourceFilter());+        } else {+            this.zipHasher = new ZipHasher(classpathResourceHasher, classpathResourceFilter);+        }

Got rid of the overloaded constructor to make the FILTER_NOTHING more obvious.

DanielThomas

comment created time in 23 days

push eventDanielThomas/gradle

Danny Thomas

commit sha aad8e17dc69f83f30131bee7b6da6fd65ba06f35

Normalize metaInf entries Jar file manifests can't be normalized via the current normalization method due to their potential impact on runtime behaviour, so this makes their normalization a feature. In addition to allowing attributes to be ignored by key, for changing attributes such as Created-By (typically includes the exact JDK version) attributes are sorted before they're filtered and hashed ensuring a stable result. This also normalizes property files with in META-INF/ and optionally allows values to be ignored by key. Signed-off-by: Danny Thomas <dannyt@netflix.com>

view details

push time in 23 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

 private void fingerprintZipEntries(String parentName, List<FileSystemLocationFin             }             String fullName = parentName.isEmpty() ? zipEntry.getName() : parentName + "/" + zipEntry.getName();             if (isZipFile(zipEntry.getName())) {-                fingerprintZipEntries(fullName, fingerprints, new StreamZipInput(zipEntry.getInputStream()));+                fingerprintZipEntries(fullName, rootParentName, fingerprints, new StreamZipInput(zipEntry.getInputStream()));+            } else if (isManifestFile(zipEntry.getName())) {+                fingerprintZipEntryContentWithFallback(rootParentName, fingerprints, zipEntry, fullName, this::hashManifest);+            } else if (isManifestPropertyFile(zipEntry.getName())) {+                fingerprintZipEntryContentWithFallback(rootParentName, fingerprints, zipEntry, fullName, this::hashProperties);             } else {-                HashCode hash = resourceHasher.hash(zipEntry);-                if (hash != null) {-                    fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+                fingerprintZipEntry(zipEntry, fullName, fingerprints);+            }+        }+    }++    private void fingerprintZipEntry(ZipEntry zipEntry, String fullName, List<FileSystemLocationFingerprint> fingerprints) throws IOException {+        HashCode hash = resourceHasher.hash(zipEntry);+        if (hash != null) {+            fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+        }+    }++    private void fingerprintZipEntryContentWithFallback(String rootParentName, List<FileSystemLocationFingerprint> fingerprints, ZipEntry zipEntry, String fullName, ByteContentHasher hasher) throws IOException {+        // JdkZipEntry can be backed by a stream, so we assume that getInputStream is a single shot and read the manifest to a byte array so we can fallback should content hashing fail+        byte[] entryBytes = ByteStreams.toByteArray(zipEntry.getInputStream());+        try {+            HashCode hash = hasher.hash(entryBytes);+            if (hash != null) {+                fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+            }+        } catch (Exception e) {+            LOGGER.warn("Could not load fingerprint " + rootParentName + ". Falling back to regular fingerprinting", e);+            ZipEntry manifestZipEntry = new ZipEntry() {+                @Override+                public boolean isDirectory() {+                    return false;+                }++                @Override+                public String getName() {+                    return zipEntry.getName();+                }++                @Override+                public InputStream getInputStream() {+                    return new ByteArrayInputStream(entryBytes);                 }++                @Override+                public int size() {+                    return entryBytes.length;+                }+            };+            fingerprintZipEntry(manifestZipEntry, fullName, fingerprints);+        }+    }++    private HashCode hashManifest(byte[] entryBytes) throws IOException {+        Manifest manifest = new Manifest(new ByteArrayInputStream(entryBytes));+        Hasher hasher = Hashing.newHasher();+        Attributes mainAttributes = manifest.getMainAttributes();+        hashAttributes(mainAttributes, "main", hasher);+        Map<String, Attributes> entries = manifest.getEntries();+        Set<String> names = new TreeSet<>(manifest.getEntries().keySet());+        for (String name : names) {+            hashAttributes(entries.get(name), name, hasher);+        }+        return hasher.hash();+    }++    private void hashAttributes(Attributes attributes, String name, Hasher hasher) {+        List<MapEntry> entries = attributes+            .entrySet()+            .stream()+            .map(MapEntry::new)+            .map(MapEntry::withLowercaseKey)

Yep, case sensitivity of the ignore/normalization reflects the underlying case sensitivity of the file format. I'll update the javadoc to reflect that. I went with exact matching for now.

DanielThomas

comment created time in 23 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

 private void fingerprintZipEntries(String parentName, List<FileSystemLocationFin             }             String fullName = parentName.isEmpty() ? zipEntry.getName() : parentName + "/" + zipEntry.getName();             if (isZipFile(zipEntry.getName())) {-                fingerprintZipEntries(fullName, fingerprints, new StreamZipInput(zipEntry.getInputStream()));+                fingerprintZipEntries(fullName, rootParentName, fingerprints, new StreamZipInput(zipEntry.getInputStream()));+            } else if (isManifestFile(zipEntry.getName())) {+                fingerprintZipEntryContentWithFallback(rootParentName, fingerprints, zipEntry, fullName, this::hashManifest);+            } else if (isManifestPropertyFile(zipEntry.getName())) {+                fingerprintZipEntryContentWithFallback(rootParentName, fingerprints, zipEntry, fullName, this::hashProperties);             } else {-                HashCode hash = resourceHasher.hash(zipEntry);-                if (hash != null) {-                    fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+                fingerprintZipEntry(zipEntry, fullName, fingerprints);+            }+        }+    }++    private void fingerprintZipEntry(ZipEntry zipEntry, String fullName, List<FileSystemLocationFingerprint> fingerprints) throws IOException {+        HashCode hash = resourceHasher.hash(zipEntry);+        if (hash != null) {+            fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+        }+    }++    private void fingerprintZipEntryContentWithFallback(String rootParentName, List<FileSystemLocationFingerprint> fingerprints, ZipEntry zipEntry, String fullName, ByteContentHasher hasher) throws IOException {+        // JdkZipEntry can be backed by a stream, so we assume that getInputStream is a single shot and read the manifest to a byte array so we can fallback should content hashing fail+        byte[] entryBytes = ByteStreams.toByteArray(zipEntry.getInputStream());+        try {+            HashCode hash = hasher.hash(entryBytes);+            if (hash != null) {+                fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+            }+        } catch (Exception e) {+            LOGGER.warn("Could not load fingerprint " + rootParentName + ". Falling back to regular fingerprinting", e);+            ZipEntry manifestZipEntry = new ZipEntry() {+                @Override+                public boolean isDirectory() {+                    return false;+                }++                @Override+                public String getName() {+                    return zipEntry.getName();+                }++                @Override+                public InputStream getInputStream() {+                    return new ByteArrayInputStream(entryBytes);                 }++                @Override+                public int size() {+                    return entryBytes.length;+                }+            };+            fingerprintZipEntry(manifestZipEntry, fullName, fingerprints);+        }+    }++    private HashCode hashManifest(byte[] entryBytes) throws IOException {+        Manifest manifest = new Manifest(new ByteArrayInputStream(entryBytes));+        Hasher hasher = Hashing.newHasher();+        Attributes mainAttributes = manifest.getMainAttributes();+        hashAttributes(mainAttributes, "main", hasher);+        Map<String, Attributes> entries = manifest.getEntries();+        Set<String> names = new TreeSet<>(manifest.getEntries().keySet());+        for (String name : names) {+            hashAttributes(entries.get(name), name, hasher);+        }+        return hasher.hash();+    }++    private void hashAttributes(Attributes attributes, String name, Hasher hasher) {+        List<MapEntry> entries = attributes+            .entrySet()+            .stream()+            .map(MapEntry::new)+            .map(MapEntry::withLowercaseKey)+            .filter(entry -> !attributeResourceFilter.shouldBeIgnored(entry))+            .sorted()+            .collect(Collectors.toList());

Addressed on the other thread.

DanielThomas

comment created time in 23 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

 private void fingerprintZipEntries(String parentName, List<FileSystemLocationFin             }             String fullName = parentName.isEmpty() ? zipEntry.getName() : parentName + "/" + zipEntry.getName();             if (isZipFile(zipEntry.getName())) {-                fingerprintZipEntries(fullName, fingerprints, new StreamZipInput(zipEntry.getInputStream()));+                fingerprintZipEntries(fullName, rootParentName, fingerprints, new StreamZipInput(zipEntry.getInputStream()));+            } else if (isManifestFile(zipEntry.getName())) {+                fingerprintZipEntryContentWithFallback(rootParentName, fingerprints, zipEntry, fullName, this::hashManifest);+            } else if (isManifestPropertyFile(zipEntry.getName())) {+                fingerprintZipEntryContentWithFallback(rootParentName, fingerprints, zipEntry, fullName, this::hashProperties);             } else {-                HashCode hash = resourceHasher.hash(zipEntry);-                if (hash != null) {-                    fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+                fingerprintZipEntry(zipEntry, fullName, fingerprints);+            }+        }+    }++    private void fingerprintZipEntry(ZipEntry zipEntry, String fullName, List<FileSystemLocationFingerprint> fingerprints) throws IOException {+        HashCode hash = resourceHasher.hash(zipEntry);+        if (hash != null) {+            fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+        }+    }++    private void fingerprintZipEntryContentWithFallback(String rootParentName, List<FileSystemLocationFingerprint> fingerprints, ZipEntry zipEntry, String fullName, ByteContentHasher hasher) throws IOException {+        // JdkZipEntry can be backed by a stream, so we assume that getInputStream is a single shot and read the manifest to a byte array so we can fallback should content hashing fail+        byte[] entryBytes = ByteStreams.toByteArray(zipEntry.getInputStream());+        try {+            HashCode hash = hasher.hash(entryBytes);+            if (hash != null) {+                fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+            }+        } catch (Exception e) {+            LOGGER.warn("Could not load fingerprint " + rootParentName + ". Falling back to regular fingerprinting", e);+            ZipEntry manifestZipEntry = new ZipEntry() {+                @Override+                public boolean isDirectory() {+                    return false;+                }++                @Override+                public String getName() {+                    return zipEntry.getName();+                }++                @Override+                public InputStream getInputStream() {+                    return new ByteArrayInputStream(entryBytes);                 }++                @Override+                public int size() {+                    return entryBytes.length;+                }+            };+            fingerprintZipEntry(manifestZipEntry, fullName, fingerprints);+        }+    }++    private HashCode hashManifest(byte[] entryBytes) throws IOException {+        Manifest manifest = new Manifest(new ByteArrayInputStream(entryBytes));+        Hasher hasher = Hashing.newHasher();+        Attributes mainAttributes = manifest.getMainAttributes();+        hashAttributes(mainAttributes, "main", hasher);+        Map<String, Attributes> entries = manifest.getEntries();+        Set<String> names = new TreeSet<>(manifest.getEntries().keySet());+        for (String name : names) {+            hashAttributes(entries.get(name), name, hasher);+        }+        return hasher.hash();+    }++    private void hashAttributes(Attributes attributes, String name, Hasher hasher) {+        List<MapEntry> entries = attributes+            .entrySet()+            .stream()+            .map(MapEntry::new)+            .map(MapEntry::withLowercaseKey)+            .filter(entry -> !attributeResourceFilter.shouldBeIgnored(entry))+            .sorted()+            .collect(Collectors.toList());+        if (!entries.isEmpty()) {+            hasher.putString(name);+            for (MapEntry entry : entries) {+                hasher.putString(entry.key);+                hasher.putString(entry.value);             }         }     } +    private @Nullable HashCode hashProperties(byte[] entryBytes) throws IOException {+        Properties properties = new Properties();+        properties.load(new ByteArrayInputStream(entryBytes));+        List<MapEntry> entries = properties.entrySet()+            .stream()+            .map(MapEntry::new)+            .filter(entry -> !propertyResourceFilter.shouldBeIgnored(entry))+            .sorted()+            .collect(Collectors.toList());+        if (entries.isEmpty()) {+            return null;

Aha found it, I had a test for this - the reason this exists for manifest attributes is if we ignore all entries in a manifest section, we want to ignore the entire section. Properties now return the hash regardless and use {{forEach}}. Attributes remain to be ignored but I've added a comment explaining why.

DanielThomas

comment created time in 23 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

      * Ignore resources in classpath entries matching {@code pattern}.      */     void ignore(String pattern);++    /**+     * Ignore attributes in {@code MANIFEST-INF/MANIFEST.MF} within archives matching {@code pattern}.

Looks like all of the best glob compilers are either legacy (oro - incidentally that's missing as a dependency for Ivy so you guys can't support Ivy glob matching) or file system specific. I'm not going to have time to implement and test an alternative so I'll go with exact matching for now.

DanielThomas

comment created time in 23 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

 public static boolean isZipFile(final String name) {         return false;     } +    public static boolean isManifestFile(final String name) {+        return name.equals("META-INF/MANIFEST.MF");+    }++    public static boolean isManifestPropertyFile(final String name) {+        return name.startsWith("META-INF") && name.endsWith(".properties");+    }+     private final ResourceHasher resourceHasher;     private final ResourceFilter resourceFilter;+    private final ResourceFilter attributeResourceFilter;+    private final ResourceFilter propertyResourceFilter;

Those are ByteContentHasher because manifests and properties files aren't just filtered, but normalized regardless of the configured filters. MapEntry exists for sorting primarily and IIRC there were duplicate key cases we need to avoid so sorted maps were a bad fit.

Agree it's shoehorning a little here, but ResourceFilter/PathMatcher felt reasonably general purpose, in that it's simple matching against one or more string parts. Per my other comment I think we want at least simple wildcard matches. ResourceEntryFilter is a good name, I'll start there.

DanielThomas

comment created time in 23 days

push eventDanielThomas/gradle

Danny Thomas

commit sha 2bcdedf1005cb4c25e2285edcd3ec95c3e44b524

Normalize metaInf entries Jar file manifests can't be normalized via the current normalization method due to their potential impact on runtime behaviour, so this makes their normalization a feature. In addition to allowing attributes to be ignored by key, for changing attributes such as Created-By (typically includes the exact JDK version) attributes are sorted before they're filtered and hashed ensuring a stable result. This also normalizes property files with in META-INF/ and optionally allows values to be ignored by key.

view details

push time in 23 days

push eventDanielThomas/dotfiles

Danny Thomas

commit sha 7ac9852d166b6ae0a6741631a8c8baf0f7708628

Add Zulu 13

view details

push time in 23 days

push eventDanielThomas/gradle

Danny Thomas

commit sha 7e3cfce4f79d41c390972cdddd04ff9f894d8e17

Normalize metaInf entries Jar file manifests can't be normalized via the current normalization method due to their potential impact on runtime behaviour, so this makes their normalization a feature. In addition to allowing attributes to be ignored by key, for changing attributes such as Created-By (typically includes the exact JDK version) attributes are sorted before they're filtered and hashed ensuring a stable result. This also normalizes property files with in META-INF/ and optionally allows values to be ignored by key.

view details

push time in 23 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

 private void fingerprintZipEntries(String parentName, List<FileSystemLocationFin             }             String fullName = parentName.isEmpty() ? zipEntry.getName() : parentName + "/" + zipEntry.getName();             if (isZipFile(zipEntry.getName())) {-                fingerprintZipEntries(fullName, fingerprints, new StreamZipInput(zipEntry.getInputStream()));+                fingerprintZipEntries(fullName, rootParentName, fingerprints, new StreamZipInput(zipEntry.getInputStream()));+            } else if (isManifestFile(zipEntry.getName())) {+                fingerprintZipEntryContentWithFallback(rootParentName, fingerprints, zipEntry, fullName, this::hashManifest);+            } else if (isManifestPropertyFile(zipEntry.getName())) {+                fingerprintZipEntryContentWithFallback(rootParentName, fingerprints, zipEntry, fullName, this::hashProperties);             } else {-                HashCode hash = resourceHasher.hash(zipEntry);-                if (hash != null) {-                    fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+                fingerprintZipEntry(zipEntry, fullName, fingerprints);+            }+        }+    }++    private void fingerprintZipEntry(ZipEntry zipEntry, String fullName, List<FileSystemLocationFingerprint> fingerprints) throws IOException {+        HashCode hash = resourceHasher.hash(zipEntry);+        if (hash != null) {+            fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+        }+    }++    private void fingerprintZipEntryContentWithFallback(String rootParentName, List<FileSystemLocationFingerprint> fingerprints, ZipEntry zipEntry, String fullName, ByteContentHasher hasher) throws IOException {+        // JdkZipEntry can be backed by a stream, so we assume that getInputStream is a single shot and read the manifest to a byte array so we can fallback should content hashing fail+        byte[] entryBytes = ByteStreams.toByteArray(zipEntry.getInputStream());+        try {+            HashCode hash = hasher.hash(entryBytes);+            if (hash != null) {+                fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+            }+        } catch (Exception e) {+            LOGGER.warn("Could not load fingerprint " + rootParentName + ". Falling back to regular fingerprinting", e);+            ZipEntry manifestZipEntry = new ZipEntry() {+                @Override+                public boolean isDirectory() {+                    return false;+                }++                @Override+                public String getName() {+                    return zipEntry.getName();+                }++                @Override+                public InputStream getInputStream() {+                    return new ByteArrayInputStream(entryBytes);                 }++                @Override+                public int size() {+                    return entryBytes.length;+                }+            };+            fingerprintZipEntry(manifestZipEntry, fullName, fingerprints);+        }+    }++    private HashCode hashManifest(byte[] entryBytes) throws IOException {+        Manifest manifest = new Manifest(new ByteArrayInputStream(entryBytes));+        Hasher hasher = Hashing.newHasher();+        Attributes mainAttributes = manifest.getMainAttributes();+        hashAttributes(mainAttributes, "main", hasher);+        Map<String, Attributes> entries = manifest.getEntries();+        Set<String> names = new TreeSet<>(manifest.getEntries().keySet());+        for (String name : names) {+            hashAttributes(entries.get(name), name, hasher);+        }+        return hasher.hash();+    }++    private void hashAttributes(Attributes attributes, String name, Hasher hasher) {+        List<MapEntry> entries = attributes+            .entrySet()+            .stream()+            .map(MapEntry::new)+            .map(MapEntry::withLowercaseKey)+            .filter(entry -> !attributeResourceFilter.shouldBeIgnored(entry))+            .sorted()+            .collect(Collectors.toList());

Same here as for the properties - if no entries are matched then the whole file is implicitly ignored. Would you be happier for these both to always output a fingerprint?

DanielThomas

comment created time in 23 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

 private void fingerprintZipEntries(String parentName, List<FileSystemLocationFin             }             String fullName = parentName.isEmpty() ? zipEntry.getName() : parentName + "/" + zipEntry.getName();             if (isZipFile(zipEntry.getName())) {-                fingerprintZipEntries(fullName, fingerprints, new StreamZipInput(zipEntry.getInputStream()));+                fingerprintZipEntries(fullName, rootParentName, fingerprints, new StreamZipInput(zipEntry.getInputStream()));+            } else if (isManifestFile(zipEntry.getName())) {+                fingerprintZipEntryContentWithFallback(rootParentName, fingerprints, zipEntry, fullName, this::hashManifest);+            } else if (isManifestPropertyFile(zipEntry.getName())) {+                fingerprintZipEntryContentWithFallback(rootParentName, fingerprints, zipEntry, fullName, this::hashProperties);             } else {-                HashCode hash = resourceHasher.hash(zipEntry);-                if (hash != null) {-                    fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+                fingerprintZipEntry(zipEntry, fullName, fingerprints);+            }+        }+    }++    private void fingerprintZipEntry(ZipEntry zipEntry, String fullName, List<FileSystemLocationFingerprint> fingerprints) throws IOException {+        HashCode hash = resourceHasher.hash(zipEntry);+        if (hash != null) {+            fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+        }+    }++    private void fingerprintZipEntryContentWithFallback(String rootParentName, List<FileSystemLocationFingerprint> fingerprints, ZipEntry zipEntry, String fullName, ByteContentHasher hasher) throws IOException {+        // JdkZipEntry can be backed by a stream, so we assume that getInputStream is a single shot and read the manifest to a byte array so we can fallback should content hashing fail+        byte[] entryBytes = ByteStreams.toByteArray(zipEntry.getInputStream());+        try {+            HashCode hash = hasher.hash(entryBytes);+            if (hash != null) {+                fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+            }+        } catch (Exception e) {+            LOGGER.warn("Could not load fingerprint " + rootParentName + ". Falling back to regular fingerprinting", e);+            ZipEntry manifestZipEntry = new ZipEntry() {+                @Override+                public boolean isDirectory() {+                    return false;+                }++                @Override+                public String getName() {+                    return zipEntry.getName();+                }++                @Override+                public InputStream getInputStream() {+                    return new ByteArrayInputStream(entryBytes);                 }++                @Override+                public int size() {+                    return entryBytes.length;+                }+            };+            fingerprintZipEntry(manifestZipEntry, fullName, fingerprints);+        }+    }++    private HashCode hashManifest(byte[] entryBytes) throws IOException {+        Manifest manifest = new Manifest(new ByteArrayInputStream(entryBytes));+        Hasher hasher = Hashing.newHasher();+        Attributes mainAttributes = manifest.getMainAttributes();+        hashAttributes(mainAttributes, "main", hasher);+        Map<String, Attributes> entries = manifest.getEntries();+        Set<String> names = new TreeSet<>(manifest.getEntries().keySet());+        for (String name : names) {+            hashAttributes(entries.get(name), name, hasher);+        }+        return hasher.hash();+    }++    private void hashAttributes(Attributes attributes, String name, Hasher hasher) {+        List<MapEntry> entries = attributes+            .entrySet()+            .stream()+            .map(MapEntry::new)+            .map(MapEntry::withLowercaseKey)+            .filter(entry -> !attributeResourceFilter.shouldBeIgnored(entry))+            .sorted()+            .collect(Collectors.toList());+        if (!entries.isEmpty()) {+            hasher.putString(name);+            for (MapEntry entry : entries) {+                hasher.putString(entry.key);+                hasher.putString(entry.value);             }         }     } +    private @Nullable HashCode hashProperties(byte[] entryBytes) throws IOException {+        Properties properties = new Properties();+        properties.load(new ByteArrayInputStream(entryBytes));+        List<MapEntry> entries = properties.entrySet()+            .stream()+            .map(MapEntry::new)+            .filter(entry -> !propertyResourceFilter.shouldBeIgnored(entry))+            .sorted()+            .collect(Collectors.toList());+        if (entries.isEmpty()) {+            return null;

If I remember correctly, I was was thinking I'd treat the entire file as if it'd been ignored entirely if no entries matched so they didn't appear in the fingerprints at all.

DanielThomas

comment created time in 23 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

 import org.slf4j.LoggerFactory;  import javax.annotation.Nullable;+import java.io.ByteArrayInputStream; import java.io.File; import java.io.IOException;+import java.io.InputStream; import java.util.List;+import java.util.Locale;+import java.util.Map;+import java.util.Properties; import java.util.Set;+import java.util.TreeSet;+import java.util.jar.Attributes;+import java.util.jar.Manifest;

It's just wrapping java.util.Manifest for the reading of the manifest anyway, not sure it adds any value?

DanielThomas

comment created time in 23 days

pull request commentgradle/gradle

Normalize META-INF manifest attributes and properties files

Rebased and squashed the existing work before I apply the remaining review feedback.

DanielThomas

comment created time in 23 days

push eventDanielThomas/gradle

Louis Jacomet

commit sha 2becfdfed9a44eea2644c35f5c3a137c876c0aa8

Fix handling of configuration level excludes For realised Ivy metadata, excludes at the level of configurations added by rules need to be explicitly serialized as we cannot rely on the usual derivation from the base Ivy metadata since the mapping between the added configuration and the base one is lost. The test has also been adapted to show that one derived variant excludes while another does not.

view details

Donat Csikos

commit sha 86da9ac5d8987f340a08f53f6cf4705d7a718110

Remove comments

view details

Donat Csikos

commit sha 994f868485de0f6c5a40b0e1cbd693681161c31f

Fix test

view details

Jendrik Johannes

commit sha 1b497c1025f9c572c9d86f954923d496399dfc75

Use new 'mainClass' property of application plugin in samples and docs

view details

Donat Csikos

commit sha 63769a625546e5055176f5c772310c1259b1251f

Move disconnect() method to API

view details

Louis Jacomet

commit sha ff432a7bb4d54442a56525befcf25fb6b28c61d5

Merge branch 'release'

view details

Vaidotas Valuckas

commit sha 12ff239f528cb3c770e8cd99ba66ba50c7b0ffe7

Merge branch 'release' into blindpirate/remove-old-registry-entry

view details

Rodrigo B. de Oliveira

commit sha 4e4ecefe0c46876cd8d4f2d3a83d16e5d8ffda0e

Remove `@ToBeFixedForInstantExecution` from `TaskTypeUpToDateIntegrationTest`

view details

Stefan Wolf

commit sha fbe3f0a8bdb56943a817e80a9bcf9c8af99af5eb

Do not use finalizedBy in test since it is not yet supported by instant execution.

view details

Jendrik Johannes

commit sha caf12b0bf9f190cbeaf2896751c9b3b17880860f

Simplify or move snippet tests that only run on Java 9+

view details

Paul Merlin

commit sha 53c000f67db337c9ac099df7ccd6a8dd0e3451a8

Add @ToBeFixedForInstantExecution to BomSupportPluginsSmokeTest because of the GenerateGraphTask in ResolveTestFixture that has a Configuration field Signed-off-by: Paul Merlin <paul@gradle.com>

view details

Paul Merlin

commit sha c45540059cef1e72254188c636e8ca68aba7a369

Skip errorprone plugin smoke test with instant execution on Java 8 because the errorprone plugin is using a task that has a Configuration field when running with Java 8 it passes with Java 14 Signed-off-by: Paul Merlin <paul@gradle.com>

view details

Paul Merlin

commit sha 03e181bb174a654a023c1e7db9958777f65c4193

Introduce internal annotation to opt-out of field type check Signed-off-by: Paul Merlin <paul@gradle.com>

view details

Stefan Wolf

commit sha d8bdd36bbbeb2cabcf4377e2cb7de406709927da

Handle watchFilter in base class

view details

Paul Merlin

commit sha f35a975e9fac115a957a8b04e23a6821aa485687

Let GenerateGraphTask fixture skip instant execution field type check Signed-off-by: Paul Merlin <paul@gradle.com>

view details

Paul Merlin

commit sha 48a7a2c47dec58e44511325a6e89b747e0672128

Merge pull request #12721 from gradle/eskatos/ie/report-user-listeners Report user build listeners registered on Gradle as instant execution problems

view details

Louis Jacomet

commit sha 0bdc853dea56511e4e80c3f436b70c61a205cf33

Fix CacheLayout version for metadata

view details

Louis Jacomet

commit sha fb98be2b2e659ba97bbf44f779cd1a337840c381

Merge branch 'release'

view details

Louis Jacomet

commit sha 11c15f78ac03ce600f8f265fc1ac6c119f5f971d

Ignore test to unblock master This is temporary and will be reverted soon

view details

Paul Merlin

commit sha 642cabffe410a8d4483debf981081ee8be4b77fb

nebula plugin plugin smoke test @ToBeFixedForInstantExecution Signed-off-by: Paul Merlin <paul@gradle.com>

view details

push time in 23 days

push eventDanielThomas/gradle

Danny Thomas

commit sha b3fbfc6fb5c0c51e46a3f62fc8604ab151deb643

Rename manifest normalization to metaInf and add shorthand configurations Adds ignoreCompletely and ignoreManifest to simplify handling META-INF ignores.

view details

push time in 23 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

 class ConfigureRuntimeClasspathNormalizationIntegrationTest extends AbstractInte             return this         } +        ProjectWithRuntimeClasspathNormalization withManifestAttributesIgnored() {+            root.file('build.gradle') << """+                normalization {+                    runtimeClasspath {+                        ignoreManifestAttribute "Implementation-Version"+                    }

Thinking about it - I think this makes more sense for this to be called metaInf instead of manifest. This started as being originally only about META-INF/MANIFEST.MF but properties filtering applies to META-INF/*.properties and you'd want ignoreCompletely() to ignore META-INF/* and ignoreManifest() to ignore META-INF/MANIFEST.MF.

DanielThomas

comment created time in 23 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

 public static boolean isZipFile(final String name) {         return false;     } +    public static boolean isManifestFile(final String name) {+        return name.equals("META-INF/MANIFEST.MF");+    }++    public static boolean isManifestPropertyFile(final String name) {+        return name.startsWith("META-INF") && name.endsWith(".properties");

Fixed.

DanielThomas

comment created time in 23 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

 public void appendConfigurationToHasher(Hasher hasher) {         hasher.putString(getClass().getName());         resourceHasher.appendConfigurationToHasher(hasher);         resourceFilter.appendConfigurationToHasher(hasher);+        attributeResourceFilter.appendConfigurationToHasher(hasher);

Added.

DanielThomas

comment created time in 23 days

push eventDanielThomas/gradle

Vaidotas Valuckas

commit sha c966dc38cc90c42779c4b8127d0a895e8bf7cd7f

Use specific deprecateMethod for deprecating methods in StartParameter

view details

Vaidotas Valuckas

commit sha c2b28f661c3778e0748f1550baf8907b46ea155e

Add a test for SourceDirectorySetFactory deprecations

view details

Vaidotas Valuckas

commit sha 46c10701bd80199f1e2a75e727b83ba370102240

Use specific deprecateInternalApi for deprecating methods in DefaultSourceDirectorySetFactory

view details

Vaidotas Valuckas

commit sha 1c748ae973f886c5cde12779caeab5b8bd76b751

Use specific deprecateMethod for deprecating methods in DefaultGradle

view details

Vaidotas Valuckas

commit sha 8942254af71228815e4c47076ba2bc502eb9b690

Use specific deprecateTask deprecation method in BasePlugin

view details

Vaidotas Valuckas

commit sha 27800b593aa0f6166954d1fe3e48f6d1b9f75761

Make incubation and deprecation feature handlers final

view details

Vaidotas Valuckas

commit sha e5af04575ee67dc0f156d5f65f0b230152568e6d

Convert deprecation output comments to javadoc style

view details

Vaidotas Valuckas

commit sha 6abc3fff0bcc6f186fa2942a098b26cab1e76b49

Add @CheckReturnValue to DeprecationMessageBuilder

view details

Rodrigo B. de Oliveira

commit sha bfd901f6a1df6a63725bb4d4624d6b7a75423c1f

Compute `buildTimestamp` using a value source to capture the relevant build logic inputs

view details

Rodrigo B. de Oliveira

commit sha 5ed26a269a50542c0e6fa9596bfa4cee8fad51bf

Don't bother checking if the build is running on CI when running an install task One less build logic input to capture.

view details

Rodrigo B. de Oliveira

commit sha c209e2cf1746060e1714b09b03c79c2facecae5e

Remove `CI` environment var from instant execution smoke test environment So the test is not affected by `buildTimestamp` differences and better reproduces what happens in developer machine.

view details

Rodrigo B. de Oliveira

commit sha 80ee685af115359177b51f2aab9dae1681209f0c

Reduce visibility of extension methods

view details

Rodrigo B. de Oliveira

commit sha 630627cd45b16a26283e31e901b0e87d6fe4b3cd

Add note about the relevance of the CI env var to the ie smoke test

view details

Rodrigo B. de Oliveira

commit sha 7579b73207b18b912942a8923350ef82b06eb52f

Refine `BuildVersionPlugin` - Extract `BuildTimestampFromBuildReceipt` - Improve names

view details

Rodrigo B. de Oliveira

commit sha 60072acfc6e6b3c512e27857c40bef7ff8256e95

Remove no longer used method `BuildReceipt.readBuildReceipt`

view details

Gradleware Git Bot

commit sha fcce339e7b28795ca99a166fb817fb6da6449948

Publish 6.1-20200119000039+0000

view details

Bo Zhang

commit sha 987e83ded2bce7874aa4608d5dbd0b263e77ef64

Automate build buckets generation This commit automate the build buckets generation by using historical build time data, so that we can have many evenly distributed build buckets.

view details

Bo Zhang

commit sha 845ebbd305a604c1a6e5dfc381cc7f1ef94c368b

Fix ktlint

view details

Gradleware Git Bot

commit sha adabae1bb48c2bcace9b3b2cb311f14afc48e3c2

Publish 6.1-20200120000037+0000

view details

Bo Zhang

commit sha 3e0892350f3cb8808e70c6dba06b569004294b2f

Split allVersionsCrossVersionTest and allVersionsMultiIntegTest

view details

push time in 23 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

 class ZipHasherTest extends Specification {         hash1 != hash2     } +    def "changing manifest attributes changes the hashcode"() {+        given:+        def jarfile = tmpDir.file("test.jar")+        def manifest = new Manifest()+        def attributes = manifest.getMainAttributes()+        attributes.put(Attributes.Name.MANIFEST_VERSION, "1.0")

Required, because manifest.write is called by the JarOutputStream:

https://docs.oracle.com/javase/7/docs/api/java/util/jar/Manifest.html#write(java.io.OutputStream)

Writes the Manifest to the specified OutputStream. Attributes.Name.MANIFEST_VERSION must be set in MainAttributes prior to invoking this method.

DanielThomas

comment created time in 23 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

     private final StringInterner stringInterner;     private final HashCode zipHasherConfigurationHash; -    private ClasspathFingerprintingStrategy(String identifier, NonJarFingerprintingStrategy nonZipFingerprintingStrategy, ResourceHasher classpathResourceHasher, ResourceFilter classpathResourceFilter, ResourceSnapshotterCacheService cacheService, StringInterner stringInterner) {+    private ClasspathFingerprintingStrategy(String identifier, NonJarFingerprintingStrategy nonZipFingerprintingStrategy, ResourceHasher classpathResourceHasher, ClasspathResourceFilters classpathResourceFilters, ResourceSnapshotterCacheService cacheService, StringInterner stringInterner) {         super(identifier);         this.nonZipFingerprintingStrategy = nonZipFingerprintingStrategy;-        this.classpathResourceFilter = classpathResourceFilter;+        this.classpathResourceFilter = classpathResourceFilters.getResourceFilter();         this.classpathResourceHasher = classpathResourceHasher;         this.cacheService = cacheService;         this.stringInterner = stringInterner;-        this.zipHasher = new ZipHasher(classpathResourceHasher, classpathResourceFilter);+        if (classpathResourceFilters instanceof RuntimeClasspathResourceFilters) {+            RuntimeClasspathResourceFilters runtimeClasspathResourceFilters = (RuntimeClasspathResourceFilters) classpathResourceFilters;+            this.zipHasher = new ZipHasher(classpathResourceHasher, classpathResourceFilter, runtimeClasspathResourceFilters.getManifestAttributeResourceFilter(), runtimeClasspathResourceFilters.getManifestPropertyResourceFilter());+        } else {+            this.zipHasher = new ZipHasher(classpathResourceHasher, classpathResourceFilter);+        }

I wasn't concerned about this being upside down because it's a private constructor and the callees are the static constructor methods in the same class. Though I should at least be asserting that classpathResourceFilters is RuntimeClasspathResourceFilters in that static method I guess?

I went with this approach because there are other classes such as DefaultClasspathFingerprinter where overloading constructors, blowing out fields, etc. doesn't make sense, the strategy/hasher is the only place we need that along that part of the object graph.

I'll tinker with it and see if I can come up with something a little cleaner.

DanielThomas

comment created time in 23 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

 import org.gradle.api.internal.changedetection.state.IgnoringResourceFilter; import org.gradle.api.internal.changedetection.state.ResourceFilter; +import java.util.Locale;+import java.util.concurrent.atomic.AtomicBoolean;+ public class DefaultRuntimeClasspathNormalization implements RuntimeClasspathNormalizationInternal {     private final ImmutableSet.Builder<String> ignoresBuilder = ImmutableSet.builder();+    private final ImmutableSet.Builder<String> attributeIgnoresBuilder = ImmutableSet.builder();+    private final ImmutableSet.Builder<String> propertiesIgnoresBuilder = ImmutableSet.builder();+    private AtomicBoolean evaluated = new AtomicBoolean();     private ResourceFilter resourceFilter;+    private ResourceFilter attributeResourceFilter;+    private ResourceFilter propertyResourceFilter;      @Override     public void ignore(String pattern) {-        if (resourceFilter != null) {-            throw new GradleException("Cannot configure runtime classpath normalization after execution started.");-        }+        checkNotEvaluated();         ignoresBuilder.add(pattern);     }      @Override-    public ResourceFilter getResourceFilter() {-        if (resourceFilter == null) {+    public void ignoreManifestAttribute(String pattern) {+        checkNotEvaluated();+        attributeIgnoresBuilder.add(pattern.toLowerCase(Locale.ROOT));+    }++    @Override+    public void ignoreManifestProperty(String pattern) {+        checkNotEvaluated();+        propertiesIgnoresBuilder.add(pattern);+    }++    @Override+    public RuntimeClasspathResourceFilters getResourceFilters() {

Grouping them is for the downstream use of the class - it avoids needing three (and maybe later more) constructor arguments for the filters.

The evaluated state seems necessary to me, and it's atomic and thread safe? We don't want people adding normalization configuration at runtime and not realizing it's not taking affect?

DanielThomas

comment created time in 23 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

      * Ignore resources in classpath entries matching {@code pattern}.      */     void ignore(String pattern);++    /**+     * Ignore attributes in {@code MANIFEST-INF/MANIFEST.MF} within archives matching {@code pattern}.

Glob patterns for property names and attributes felt reasonable to me: the most common use case for ignoring properties is values added by build tooling, like commit and date, which means you'll want to have Build-* properties/attributes for instance that you want to ignore without having to enumerate all of them.

DanielThomas

comment created time in 23 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

 private void fingerprintZipEntries(String parentName, List<FileSystemLocationFin             }             String fullName = parentName.isEmpty() ? zipEntry.getName() : parentName + "/" + zipEntry.getName();             if (isZipFile(zipEntry.getName())) {-                fingerprintZipEntries(fullName, fingerprints, new StreamZipInput(zipEntry.getInputStream()));+                fingerprintZipEntries(fullName, rootParentName, fingerprints, new StreamZipInput(zipEntry.getInputStream()));+            } else if (isManifestFile(zipEntry.getName())) {+                fingerprintZipEntryContentWithFallback(rootParentName, fingerprints, zipEntry, fullName, this::hashManifest);+            } else if (isManifestPropertyFile(zipEntry.getName())) {+                fingerprintZipEntryContentWithFallback(rootParentName, fingerprints, zipEntry, fullName, this::hashProperties);             } else {-                HashCode hash = resourceHasher.hash(zipEntry);-                if (hash != null) {-                    fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+                fingerprintZipEntry(zipEntry, fullName, fingerprints);+            }+        }+    }++    private void fingerprintZipEntry(ZipEntry zipEntry, String fullName, List<FileSystemLocationFingerprint> fingerprints) throws IOException {+        HashCode hash = resourceHasher.hash(zipEntry);+        if (hash != null) {+            fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+        }+    }++    private void fingerprintZipEntryContentWithFallback(String rootParentName, List<FileSystemLocationFingerprint> fingerprints, ZipEntry zipEntry, String fullName, ByteContentHasher hasher) throws IOException {+        // JdkZipEntry can be backed by a stream, so we assume that getInputStream is a single shot and read the manifest to a byte array so we can fallback should content hashing fail+        byte[] entryBytes = ByteStreams.toByteArray(zipEntry.getInputStream());+        try {+            HashCode hash = hasher.hash(entryBytes);+            if (hash != null) {+                fingerprints.add(new DefaultFileSystemLocationFingerprint(fullName, FileType.RegularFile, hash));+            }+        } catch (Exception e) {+            LOGGER.warn("Could not load fingerprint " + rootParentName + ". Falling back to regular fingerprinting", e);

Both java.util.jar.Manifest and java.util.Properties will throw on malformed files and instead of duplicating a fallback in hashManifest and hashAttributes I added the fallback in this outer method.

I personally want the warning, or at least very obvious output from the build cache debug logging about what's doing on, because problems with fingerprinting have been impossible to diagnose without attaching a debugger in my experience.

IIRC rootParentName is required because this recursively hashes zips within zips and the filename becomes part of the path.

The method takes a ByteContentHasher, falling back to fingerprintZipEntry on an exception which is why I called itfingerprintZipEntryContentWithFallback. hashAttributes and hashProperties normalize the file contents, and happens to filter entries while it's at it.

DanielThomas

comment created time in 23 days

Pull request review commentgradle/gradle

Normalize META-INF manifest attributes and properties files

 class ConfigureRuntimeClasspathNormalizationIntegrationTest extends AbstractInte             return this         } +        ProjectWithRuntimeClasspathNormalization withManifestAttributesIgnored() {+            root.file('build.gradle') << """+                normalization {+                    runtimeClasspath {+                        ignoreManifestAttribute "Implementation-Version"+                    }

That would be a useful shorthand. Let me see about that.

DanielThomas

comment created time in 23 days

issue commentjetty-project/jetty-alpn

jetty-alpn incompatible with OpenJDK 8u251

The sources are readily available. They're bundled in the JDK in src.zip. It's definitely not the same as 8u252, it has the ALPN backport, 8u251 does not.

DanielThomas

comment created time in a month

issue openedjetty-project/jetty-alpn

jetty-alpn incompatible with OpenJDK 8u251

The readme for this project appears to be mistaken - 8u251 does not backport the ALPN APIs, instead it's broken by the removal of SSLSessionImpl.nullSession:

Caused by: java.lang.NoSuchFieldError: nullSession
	at sun.security.ssl.ClientHandshaker.getKickstartMessage(ClientHandshaker.java:1350)
	at sun.security.ssl.Handshaker.kickstart(Handshaker.java:1074)
	at sun.security.ssl.SSLSocketImpl.kickstartHandshake(SSLSocketImpl.java:1479)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1346)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
	at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:142)

See:

  • https://github.com/jetty-project/jetty-alpn/blob/279b4aedd32f7dcffd183f98230c79cbcea51e23/alpn-boot/src/main/java/sun/security/ssl/ClientHandshaker.java#L1350
  • http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/comparison/f54e9b7c1036/src/share/classes/sun/security/ssl/SSLSessionImpl.java
  • https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-2781

created time in a month

issue commentsquare/okhttp

OkHttp 3.x fails with OpenJDK Zulu 8.0.252

Looks great! JDK 9 platform activates and the fallback for {{UnsupportedOperationException}} for our incompatible SslSocket wrapper successfully falls back to HTTP/1.

rpalcolea

comment created time in a month

issue commentsquare/okhttp

OkHttp 3.x fails with OpenJDK Zulu 8.0.252

Sure can - let us know where to pickup your snapshots and we'll give it a try.

Our wrapper is fixed, we switched to a javaassist generated proxy, so we should be good to go there.

rpalcolea

comment created time in a month

issue commentsquare/okhttp

OkHttp 3.x fails with OpenJDK Zulu 8.0.252

Ah! It's the combination of the JDK 9 platform activation our internal mTLS wrapper that delegates to the underlying impl that's breaking this on our end - it doesn't implement those methods causing those calls to fall through to SslSocket and hit the UnsupportedOperationException. Guess we'll have to conditionally drop in the right wrapper depending on the method availability.

Guess u252 and upwards still needs a leg up on OkHttp 3 and 4 so the built-in ALPN support does activate there.

rpalcolea

comment created time in a month

issue commentsquare/okhttp

OkHttp 3.x fails with OpenJDK Zulu 8.0.252

We're not opting in, and {{org.openjsse.sun.security.ssl.SSLSocketImpl}} implements {{getApplicationProtocol}}. Going to try and get an easily reproducible case here to see what impl we're actually getting.

rpalcolea

comment created time in a month

issue commentsquare/okhttp

OkHttp 3.x fails with OpenJDK Zulu 8.0.252

Oh yeah, interesting. I wonder what SSLSocketImpl we're getting at runtime...

rpalcolea

comment created time in a month

issue commentsquare/okhttp

OkHttp 3.x fails with OpenJDK Zulu 8.0.252

Yeah, we have a few applications that install the native modules required for JDK 8 ALPN support.

rpalcolea

comment created time in a month

issue commentsquare/okhttp

OkHttp 3.x fails with OpenJDK Zulu 8.0.252

It's part of the contract for that method:

https://docs.oracle.com/javase/9/docs/api/javax/net/ssl/SSLSocket.html#getApplicationProtocol--

FWIW it's exactly how SSLSocket looks on Java 11 too.

rpalcolea

comment created time in a month

issue commentsquare/okhttp

OkHttp 3.x fails with OpenJDK Zulu 8.0.252

Here's the upstream change https://hg.openjdk.java.net/jdk8u/jdk8u41/jdk/rev/b26b096d4c89

Full stack trace:

java.lang.AssertionError: failed to get ALPN selected protocol
	at okhttp3.internal.platform.Jdk9Platform.getSelectedProtocol(Jdk9Platform.java:68) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:363) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:300) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:185) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:223) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[carson-0.461.0-all.jar:0.461.0]
	at com.netflix.engtools.carson.JenkinsClient$userAgentUpdate$1.invoke(JenkinsClient.kt:32) ~[carson-0.461.0-all.jar:0.461.0]
	at com.netflix.engtools.carson.JenkinsClient$userAgentUpdate$1.invoke(JenkinsClient.kt:19) ~[carson-0.461.0-all.jar:0.461.0]
	at com.netflix.engtools.carson.JenkinsClient$sam$okhttp3_Interceptor$0.intercept(JenkinsClient.kt) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229) ~[carson-0.461.0-all.jar:0.461.0]
	at okhttp3.RealCall.execute(RealCall.java:81) ~[carson-0.461.0-all.jar:0.461.0]
	at retrofit2.OkHttpCall.execute(OkHttpCall.java:190) ~[carson-0.461.0-all.jar:0.461.0]
	at com.netflix.engtools.carson.JenkinsUtil$createJnlpAgent$token$1.invoke(JenkinsUtil.kt:55) ~[carson-0.461.0-all.jar:0.461.0]
	at com.netflix.engtools.carson.JenkinsUtil$createJnlpAgent$token$1.invoke(JenkinsUtil.kt:5) ~[carson-0.461.0-all.jar:0.461.0]
	at com.netflix.engtools.carson.BackOffManager.runWithBackOff(BackOffManager.kt:35) [carson-0.461.0-all.jar:0.461.0]
	at com.netflix.engtools.carson.BackOffManager.runWithBackOff$default(BackOffManager.kt:33) [carson-0.461.0-all.jar:0.461.0]
	at com.netflix.engtools.carson.JenkinsUtil.createJnlpAgent(JenkinsUtil.kt:54) [carson-0.461.0-all.jar:0.461.0]
	at com.netflix.engtools.carson.Carson.run(Carson.kt:144) [carson-0.461.0-all.jar:0.461.0]
	at com.netflix.engtools.carson.Carson$Companion.main(Carson.kt:52) [carson-0.461.0-all.jar:0.461.0]
	at com.netflix.engtools.carson.Carson.main(Carson.kt) [carson-0.461.0-all.jar:0.461.0]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_252]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_252]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
	at okhttp3.internal.platform.Jdk9Platform.getSelectedProtocol(Jdk9Platform.java:58) ~[carson-0.461.0-all.jar:0.461.0]
	... 37 more
Caused by: java.lang.UnsupportedOperationException
	at javax.net.ssl.SSLSocket.getApplicationProtocol(SSLSocket.java:691) ~[?:1.8.0_252]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_252]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_252]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
	at okhttp3.internal.platform.Jdk9Platform.getSelectedProtocol(Jdk9Platform.java:58) ~[carson-0.461.0-all.jar:0.461.0]
	... 37 more
rpalcolea

comment created time in a month

issue commentsquare/okhttp

OkHttp 3.x fails with OpenJDK Zulu 8.0.252

Those methods are being used to detect Java 9 ALPN support however they're now available on Java 8, but ALPN support is not, so cause runtime failures due to the Jdk9Platform being used:

java.lang.AssertionError: failed to get ALPN selected protocol
	at okhttp3.internal.platform.Jdk9Platform.getSelectedProtocol(Jdk9Platform.java:68) ~[carson-0.461.0-all.jar:0.461.0]
rpalcolea

comment created time in a month

issue openednpm/cli

[BUG] Frequent CloudFlare 502/503 responses from audit endpoints

What / Why

We're seeing fairly regular timeouts for quick audits. It's not terminal for npm install but surfaces in metrics and contract tests we have for our registry caches.

When

Looks pretty frequent: Screen Shot 2020-04-14 at 10 19 25 AM

Where

Can reproduce with this dummy audit request. Failed several times this morning:

curl -v https://registry.npmjs.org/-/npm/v1/security/audits -d '{"requires":{"test":"^0.6.0"},"dependencies":{"ansi-font":{"version":"0.0.2","integrity":"sha1-iQMBvVhBRi/TnAt3Ca/R9SUUMzE="},"test":{"version":"0.6.0","integrity":"sha1-WYasRF7Bd1QyJRLRBLoyyKY+k44=","requires":{"ansi-font":"0.0.2"}}},"install":["test@0.6.0"],"remove":[],"metadata":{"npm_version":"6.4.1","node_version":"v10.11.0","platform":"darwin"}}'

Surfaces as both 502: Bad gateway:

< HTTP/2 502
< date: Tue, 14 Apr 2020 17:12:58 GMT
< content-type: text/html; charset=UTF-8
< set-cookie: __cfduid=d97a97fe40a5a0a5abc6751edde4be22c1586884370; expires=Thu, 14-May-20 17:12:50 GMT; path=/; domain=.npmjs.org; HttpOnly; SameSite=Lax
< cf-ray: 583f0dd4cccced0f-SJC
< cache-control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< set-cookie: cf_ob_info=502:583f0dd4d47aed0f:SJC; path=/; expires=Tue, 14-Apr-20 17:13:28 GMT
< pragma: no-cache
< set-cookie: cf_use_ob=443; path=/; expires=Tue, 14-Apr-20 17:13:28 GMT
< x-frame-options: SAMEORIGIN
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< server: cloudflare
<
<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<meta http-equiv="refresh" content="0">

<title>prod-frontdoor-lb-west.internal.npmjs.com | 502: Bad gateway</title>

created time in 2 months

issue openednebula-plugins/gradle-ospackage-plugin

Undefined versions result in invalid packages

Debian package versions must start with a number, so undefined should probably be mapped to 0.

created time in 2 months

fork DanielThomas/hitch

A scalable TLS proxy by Varnish Software.

https://www.varnish-software.com/

fork in 2 months

issue commentnebula-plugins/nebula-kotlin-plugin

Warning on kotlin 1.3.70

Weird to deprecate something in favor of something still considered experimental in 1.2 and 1.3. We could maybe have nebula.kotlin apply the multi-project plugin with a jvm target.

I'd hoped to be able to retire this plugin eventually, having it replaced with core features in the upstream plugin. I'll take a look at see how we're doing on parity now.

mattbroekhuis

comment created time in 3 months

push eventDanielThomas/dotfiles

Danny Thomas

commit sha 54dc9f646697f2fdb8a2f7a1213eaac6db55bd90

Remove md5sha1sum - it's covered by coreutils

view details

push time in 3 months

created tagnebula-plugins/nebula-kotlin-plugin

tagv1.3.70

Provides the Kotlin plugin via the Gradle plugin portal, automatically depends on the standard library, and allows Kotlin library versions to be omitted

created time in 3 months

release nebula-plugins/nebula-kotlin-plugin

v1.3.70

released time in 3 months

push eventnebula-plugins/nebula-kotlin-plugin

Danny Thomas

commit sha 16319361fff8f32c5ae336544f4826bd52b0ac64

Kotlin 1.3.70

view details

push time in 3 months

startedgnachman/iTerm2

started time in 3 months

more