profile
viewpoint
Jeremy David Giesbrecht SDGGiesbrecht

SDGGiesbrecht/Workspace 75

Automated Management of Swift Projects

SDGGiesbrecht/SDGSwift 31

The Swift Compiler As a Package

SDGGiesbrecht/SDGCornerstone 18

The Foundation of the SDG Module Family

SDGGiesbrecht/SDGCommandLine 3

Tools for Implementing a Command Line Interface

SDGGiesbrecht/SDGInterface 3

Tools for Implementing a Graphical User Interface

NCrusher74/AudiobookTagger 2

An audio(book)-centric Swift wrapper combining MP4Foundation with ID3TagEditor for tagging .mp3, .m4b, and .m4a files

SDGGiesbrecht/SDGMathematics 2

Extended Support for Mathematics

NCrusher74/SwiftTagger 1

A simple library for reading and editing metadata and chapters of Mp4 and Mp3 audio files

NCrusher74/SwiftTaggerMP4 1

A simple library for tagging and chaptering Mp4 audio files

SDGGiesbrecht/SDGCaching 1

Automated Caching of Function Results

Pull request review commentapple/swift-package-manager

Fix an internal error resulting from reusing a stale in-memory cached manifest when switching dependencies between forks

 public final class ManifestLoader: ManifestLoaderProtocol {                                                 swiftpmVersion: Versioning.currentVersion.displayString,                                                 fileSystem: fileSystem) -            if self.useInMemoryCache, let manifest = self.memoryCache[cacheKey] {+            if self.useInMemoryCache, let manifest = self.memoryCache[cacheKey], manifest.url == baseURL {

I think this is fine, we can also just remove the in-memory cache if we are worried that URL is not the only attribute that could potentially be changes (tho right now its the only issue afaict)

abertelrud

comment created time in an hour

PR opened apple/swift-package-manager

Fix an internal error resulting from reusing a stale in-memory cached manifest when switching dependencies between forks

Fix an internal that would occur when switching a dependency between two repositories that had the same last path component and identical package manifest contents (happens frequently with forks). This was due to the in-memory manifest cache, and thus only seen in libSwiftPM clients and not in the CLI.

rdar://73462555

Need to figure out a good way to unit test this without actually accessing network.

+1 -1

0 comment

1 changed file

pr created time in 2 hours

pull request commentapple/swift-package-manager

SR-12050: lldb error: could not build C module 'CNIOBoringSSLShims' (Linux 5.1.3) when debugging

Once the tests pass, this can be merged to main. I have already opened a 5.4 PR for SourceKit-LSP, which a 5.4 PR of this fix will then require.

abertelrud

comment created time in 2 hours

pull request commentapple/swift-package-manager

SR-12050: lldb error: could not build C module 'CNIOBoringSSLShims' (Linux 5.1.3) when debugging

SourceKit-LSP fix has now landed. Restarting tests.

abertelrud

comment created time in 2 hours

pull request commentapple/swift-package-manager

Refactor global verbosity value to parameters

@tomerd back to you. After I removed the default value for verbosity, it surfaced even more places where verbosity parameter is added.

krzyzanowskim

comment created time in 3 hours

Pull request review commentapple/swift-package-manager

Refactor global verbosity value to parameters

 public final class BuildOperation: PackageStructureDelegate, SPMBuildCore.BuildS      /// The loaded package graph.     private var packageGraph: PackageGraph?-     /// The stdout stream for the build delegate.     let stdoutStream: OutputByteStream +    /// Wether the build is in a verbose mode.+    private let isVerbose: Bool

it's akin to what's in ToolWorkspaceDelegate, would you prefer to change there as well?

krzyzanowskim

comment created time in 4 hours

push eventapple/swift-package-manager

Yim Lee

commit sha 95e5c3f6ec31fc3612ba7cc5df495726a740216a

Package collections: improve search performance (#3213) * Package collections: improve search performance This is an alternate to https://github.com/apple/swift-package-manager/pull/3090 but is a complete solution. Motivation: Currently to support search for package collections API we read and deserialize collection blobs from SQLite then perform string matchings on individual properties in memory (e.g., `package.summary.contains("foobar")`). This can be optimized. Modifications: Use SQLite FTS--define FTS virtual tables for packages and targets, and update implementation for `findPackage`, `searchPackages`, and `searchTargets` methods of `SQLitePackageCollectionsStorage`. Without optimization, `PackageCollectionsTests.testPackageSearchPerformance` and `testTargetsSearchPerformance` take about ~400ms to run on my local machine. With FTS, `testPackageSearchPerformance` takes ~40ms and `testTargetsSearchPerformance` ~50ms. The `testSearchTargetsPerformance` in `InMemoryPackageCollectionsSearchTests` (https://github.com/apple/swift-package-manager/pull/3090) yields result in ~10ms, though it queries the trie directly without going through the PackageCollections API layer. Since target search is either exact or prefix match and doesn't tokenize the query, and given the good results I saw in `InMemoryPackageCollectionsSearchTests`, this implementation includes a trie on top of SQLite FTS for target search. The trie is in-memory and loads from SQLite FTS during initialization. `put`/`remove` updates both the SQLite FTS and trie. The improvement with using trie varies--`testTargetsSearchPerformance` takes between ~15-50ms to complete. `put` now takes longer to complete because of the search index updates. Result: Better search performance. * Fallback to slow path, which does not use FTS, in case FTS4 is not available

view details

push time in 4 hours

PR merged apple/swift-package-manager

Reviewers
Package collections: improve search performance

This is an alternative to https://github.com/apple/swift-package-manager/pull/3090 but is a complete solution.

Motivation: Currently to support search for package collections API we read and deserialize collection blobs from SQLite then perform string matchings on individual properties in memory (e.g., package.summary.contains("foobar")). This can be optimized.

Modifications: Use SQLite FTS--define FTS virtual tables for packages and targets, and update implementation for findPackage, searchPackages, and searchTargets methods of SQLitePackageCollectionsStorage.

Without optimization, PackageCollectionsTests.testPackageSearchPerformance and testTargetsSearchPerformance take about ~400ms to run on my local machine.

With FTS, testPackageSearchPerformance takes ~40ms and testTargetsSearchPerformance ~50ms.

The testSearchTargetsPerformance in InMemoryPackageCollectionsSearchTests (https://github.com/apple/swift-package-manager/pull/3090) yields result in ~10ms, though it queries the trie directly without going through the PackageCollections API layer.

Since target search is either exact or prefix match and doesn't tokenize the query, and given the good results I saw in InMemoryPackageCollectionsSearchTests, this implementation includes a trie on top of SQLite FTS for target search. The trie is in-memory and loads from SQLite FTS during initialization. put/remove updates both the SQLite FTS and trie. The improvement with using trie varies--testTargetsSearchPerformance takes between ~15-50ms to complete.

put now takes longer to complete because of the search index updates.

Result: Better search performance.

+936 -93

4 comments

8 changed files

yim-lee

pr closed time in 4 hours

pull request commentapple/swift-package-manager

SR-12050: lldb error: could not build C module 'CNIOBoringSSLShims' (Linux 5.1.3) when debugging

Talked with @benlangmuir and opened https://github.com/apple/sourcekit-lsp/pull/362 to make SourceKit-LSP's unit test accept both forms of command line until this PR has been merged.

abertelrud

comment created time in 4 hours

Pull request review commentapple/swift-package-manager

api cleanup in preperation for identity improvements

 public enum DependencyResolutionNode {         // Don’t create a version lock for anything but a product.         guard specificProduct != nil else { return nil }         return PackageContainerConstraint(-            container: package,+            package: package,

Can we call this packageReference?

tomerd

comment created time in 5 hours

Pull request review commentapple/swift-package-manager

api cleanup in preperation for identity improvements

 import TSCUtility /// should be used as-is. Infact, they might not even have a git repository. /// Examples: Root packages, local dependencies, edited packages. public final class LocalPackageContainer: PackageContainer {-    public let identifier: PackageReference+    public let package: PackageReference

Can we call this packageReference?

tomerd

comment created time in 5 hours

pull request commentapple/swift-package-manager

SR-12050: lldb error: could not build C module 'CNIOBoringSSLShims' (Linux 5.1.3) when debugging

Looks like I missed a Linux-specific unit test. Investigating.

So, 1) it's not Linux-specific, and 2) it's a SourceKit-LSP test (which is why local unit tests all passed and the self-hosted test passed). Looks as if we have a revlock issue here on the SourceKit-LSP tests. @benlangmuir what's your view on how we should address this, since these are in two different repos and neither change will succeed on its own. Could the SourceKit-LSP tests become a bit less specific about whether we're passing -Xcc?

abertelrud

comment created time in 5 hours

Pull request review commentapple/swift-package-manager

Package collections: improve search performance

 final class SQLitePackageCollectionsStorage: PackageCollectionsStorage, Closable         """         try db.exec(query: table) -        let ftsPackages = """-            CREATE VIRTUAL TABLE IF NOT EXISTS \(Self.packagesFTSName) USING fts4(-                collection_id_blob_base64, id, version, name, repository_url, summary, keywords, products, targets,-                notindexed=collection_id_blob_base64,-                tokenize=unicode61-            );-        """-        try db.exec(query: ftsPackages)--        let ftsTargets = """-            CREATE VIRTUAL TABLE IF NOT EXISTS \(Self.targetsFTSName) USING fts4(-                collection_id_blob_base64, package_repository_url, name,-                notindexed=collection_id_blob_base64,-                tokenize=unicode61-            );-        """-        try db.exec(query: ftsTargets)+        do {+            let ftsPackages = """+                CREATE VIRTUAL TABLE IF NOT EXISTS \(Self.packagesFTSName) USING fts4(+                    collection_id_blob_base64, id, version, name, repository_url, summary, keywords, products, targets,+                    notindexed=collection_id_blob_base64,+                    tokenize=unicode61+                );+            """+            try db.exec(query: ftsPackages)++            let ftsTargets = """+                CREATE VIRTUAL TABLE IF NOT EXISTS \(Self.targetsFTSName) USING fts4(+                    collection_id_blob_base64, package_repository_url, name,+                    notindexed=collection_id_blob_base64,+                    tokenize=unicode61+                );+            """+            try db.exec(query: ftsTargets)++            useSearchIndices.put(true)+        } catch {+            // These DDL statements work but queries yield different results when run on different+            // platforms. This could be because of SQLite version perhaps? But since we can't get+            // consistent results we will not fallback to FTS3 and just give up if FTS4 is not available.+            /*+             let ftsPackages = """+                 CREATE VIRTUAL TABLE IF NOT EXISTS \(Self.packagesFTSName) USING fts3(+                     collection_id_blob_base64, id, version, name, repository_url, summary, keywords, products, targets,+                     tokenize=simple+                 );+             """+             try db.exec(query: ftsPackages)++             let ftsTargets = """+                 CREATE VIRTUAL TABLE IF NOT EXISTS \(Self.targetsFTSName) USING fts3(+                     collection_id_blob_base64, package_repository_url, name,+                     tokenize=porter+                 );+             """+             try db.exec(query: ftsTargets)

removed https://github.com/apple/swift-package-manager/pull/3213/commits/a84e786e1d2fab0254c3d21327a0e79eb8f4e642

yim-lee

comment created time in 5 hours

pull request commentapple/swift-package-manager

Package collections: improve search performance

@swift-ci please smoke test

yim-lee

comment created time in 5 hours

pull request commentapple/swift-package-manager

SR-12050: lldb error: could not build C module 'CNIOBoringSSLShims' (Linux 5.1.3) when debugging

Looks like I missed a Linux-specific unit test. Investigating.

abertelrud

comment created time in 6 hours

Pull request review commentapple/swift-package-manager

Refactor global verbosity value to parameters

 public final class XcodeBuildSystem: BuildSystem {     public init(         buildParameters: BuildParameters,         packageGraphLoader: @escaping () throws -> PackageGraph,-        isVerbose: Bool,+        isVerbose: Bool = false,

as in the other spot, I prefer passing the full Verbosity value and not having a default so call-sites make conscious decision

krzyzanowskim

comment created time in 6 hours

Pull request review commentapple/swift-package-manager

Refactor global verbosity value to parameters

 public final class ManifestLoader: ManifestLoaderProtocol {                           packageKind: packageKind,                           fileSystem: fileSystem,                           diagnostics: diagnostics,-                          on: queue) { result in+                          on: queue,+                          verbosity: verbosity+                          ) { result in

not: align this to the back

krzyzanowskim

comment created time in 6 hours

Pull request review commentapple/swift-package-manager

Refactor global verbosity value to parameters

 public final class ManifestLoader: ManifestLoaderProtocol {         toolsVersion: ToolsVersion,         packageKind: PackageReference.Kind,         fileSystem: FileSystem? = nil,-        diagnostics: DiagnosticsEngine? = nil+        diagnostics: DiagnosticsEngine? = nil,+        verbosity: Verbosity = TSCUtility.verbosity

this API is deprecated so we should not add argument here, just pass in TSCUtility.verbosity as default in L282

krzyzanowskim

comment created time in 6 hours

Pull request review commentapple/swift-package-manager

Refactor global verbosity value to parameters

 extension ManifestLoaderProtocol {         fileSystem: FileSystem? = nil,         diagnostics: DiagnosticsEngine? = nil,         on queue: DispatchQueue,+        verbosity: Verbosity = TSCUtility.verbosity,

I'd rather not have a default set here to make sure the call-site make a conscious decision. For the unit test we can have an short-form via extension

krzyzanowskim

comment created time in 6 hours

Pull request review commentapple/swift-package-manager

Refactor global verbosity value to parameters

 public final class BuildOperation: PackageStructureDelegate, SPMBuildCore.BuildS      /// The loaded package graph.     private var packageGraph: PackageGraph?-     /// The stdout stream for the build delegate.     let stdoutStream: OutputByteStream +    /// Wether the build is in a verbose mode.+    private let isVerbose: Bool

wdyt about storing Verbosity instead of Bool as the member, and adding a "isVerbose" computed property that is calculated off of that enum value?

krzyzanowskim

comment created time in 6 hours

Pull request review commentapple/swift-package-manager

Move package collection JSON models to a separate module

 struct JSONPackageCollectionProvider: PackageCollectionProvider { extension Model.Product {     fileprivate init(from: JSONModel.Product, packageTargets: [Model.Target]) {         let targets = packageTargets.filter { from.targets.map { $0.lowercased() }.contains($0.name.lowercased()) }-        self = .init(name: from.name, type: from.type, targets: targets)+        self = .init(name: from.name, type: .init(from: from.type), targets: targets)+    }+}++extension PackageModel.ProductType {+    fileprivate init(from: JSONModel.ProductType) {+        switch from {+        case .library(let libraryType):+            self = .library(.init(from: libraryType))+        case .executable:+            self = .executable+        case .test:+            self = .test+        }+    }+}++extension PackageModel.ProductType.LibraryType {+    fileprivate init(from: JSONModel.ProductType.LibraryType) {+        switch from {+        case .static:+            self = .static+        case .dynamic:+            self = .dynamic+        case .automatic:+            self = .automatic+        }     }

I assume we dont re-user the package model types for separation of concerns?

yim-lee

comment created time in 6 hours

Pull request review commentapple/swift-package-manager

Move package collection JSON models to a separate module

 /*  This source file is part of the Swift.org open source project - Copyright (c) 2020 Apple Inc. and the Swift project authors+ Copyright (c) 2020-2021 Apple Inc. and the Swift project authors  Licensed under Apache License v2.0 with Runtime Library Exception   See http://swift.org/LICENSE.txt for license information  See http://swift.org/CONTRIBUTORS.txt for Swift project authors  */ +import Foundation+ public enum JSONPackageCollectionModel {}

is the JSON prefix needed, since we are using Codable it could theoretically some different format? not a strong feeling, more of a question really

yim-lee

comment created time in 7 hours

Pull request review commentapple/swift-package-manager

Package collections: improve search performance

 final class SQLitePackageCollectionsStorage: PackageCollectionsStorage, Closable         """         try db.exec(query: table) -        let ftsPackages = """-            CREATE VIRTUAL TABLE IF NOT EXISTS \(Self.packagesFTSName) USING fts4(-                collection_id_blob_base64, id, version, name, repository_url, summary, keywords, products, targets,-                notindexed=collection_id_blob_base64,-                tokenize=unicode61-            );-        """-        try db.exec(query: ftsPackages)--        let ftsTargets = """-            CREATE VIRTUAL TABLE IF NOT EXISTS \(Self.targetsFTSName) USING fts4(-                collection_id_blob_base64, package_repository_url, name,-                notindexed=collection_id_blob_base64,-                tokenize=unicode61-            );-        """-        try db.exec(query: ftsTargets)+        do {+            let ftsPackages = """+                CREATE VIRTUAL TABLE IF NOT EXISTS \(Self.packagesFTSName) USING fts4(+                    collection_id_blob_base64, id, version, name, repository_url, summary, keywords, products, targets,+                    notindexed=collection_id_blob_base64,+                    tokenize=unicode61+                );+            """+            try db.exec(query: ftsPackages)++            let ftsTargets = """+                CREATE VIRTUAL TABLE IF NOT EXISTS \(Self.targetsFTSName) USING fts4(+                    collection_id_blob_base64, package_repository_url, name,+                    notindexed=collection_id_blob_base64,+                    tokenize=unicode61+                );+            """+            try db.exec(query: ftsTargets)++            useSearchIndices.put(true)+        } catch {+            // These DDL statements work but queries yield different results when run on different+            // platforms. This could be because of SQLite version perhaps? But since we can't get+            // consistent results we will not fallback to FTS3 and just give up if FTS4 is not available.+            /*+             let ftsPackages = """+                 CREATE VIRTUAL TABLE IF NOT EXISTS \(Self.packagesFTSName) USING fts3(+                     collection_id_blob_base64, id, version, name, repository_url, summary, keywords, products, targets,+                     tokenize=simple+                 );+             """+             try db.exec(query: ftsPackages)++             let ftsTargets = """+                 CREATE VIRTUAL TABLE IF NOT EXISTS \(Self.targetsFTSName) USING fts3(+                     collection_id_blob_base64, package_repository_url, name,+                     tokenize=porter+                 );+             """+             try db.exec(query: ftsTargets)

should we remove this? or keep around so we can fix later?

yim-lee

comment created time in 7 hours

pull request commentapple/swift-package-manager

Add swift-crypto dependency

Unless you plan on using these new crypto features when building SPM with itself, which shouldn't be necessary since build-script should have already checked out SPM's dependencies, I think it's fine to not add this swift-crypto dependency for the CMake bootstrap and avoid adding a CMake config for swift-crypto.

yim-lee

comment created time in 7 hours

PR opened apple/swift-package-manager

Refactor global verbosity value to parameters

Refactor global TSCUtils.verbosity value to functions parameter

Motivation:

While it's okayish to maintain global TSCUtils.verbosity for CLI tool, it's not really for library API. I believe it's better to provide a configurable verbosity parameter for library public API.

Modifications:

replace TSCUtils.verbosity with function parameter that default to global TSCUtils.verbosity

Result:

API verbosity setup is more granulated.

+47 -19

0 comment

10 changed files

pr created time in 9 hours

push eventapple/swift-syntax

Alex Hoppen

commit sha b227790692b73cb494d9f47a00fac2f4b824b88a

Remove advancedToEndOfChildren These methods are not used anywhere and are potentially incorrect (indexInParent should probably be raw.numberOfChildren - 1). Remove the until we really need them

view details

Alex Hoppen

commit sha 1e998e808e3ff7c28c6f7d33c4480dbac0a7040a

Rename advancedBy/reversedBySibling to advancedBy/reversedBy We are not actually advancing by a sibling but just by a node, simplify the method name to reflect this.

view details

Alex Hoppen

commit sha d8aa974f0532d7f0ce42b87afe476d5e4e9211ab

Merge pull request #255 from ahoppen/api-improvals Minor internal API improvements

view details

push time in 17 hours

PR merged apple/swift-syntax

Minor internal API improvements

Two minor internal API improvements:

  1. Remove advancedToEndOfChildren These methods are not used anywhere and are potentially incorrect (indexInParent should probably be initialized with raw.numberOfChildren - 1). Remove the until we really need them
  2. Rename advancedBySibling/reversedBySibling to advancedBy/reversedBy We are not actually advancing by a sibling but just by a node, simplify the method name to reflect this.
+10 -29

1 comment

2 changed files

ahoppen

pr closed time in 17 hours

pull request commentapple/swift-package-manager

Package collections: improve search performance

@swift-ci please smoke test

yim-lee

comment created time in 17 hours

more