It would be great to have a list of html pages, that XCLogParser parser with html report creates with links to it.

I guess that's the next destination of current public release, without any additional analytics over a bunch of builds. Probably we need a fork now :)

Same problem here :(

Hey, I have a question regarding the duration double value for builds, which time unit is it.. microseconds or milliseconds?

Thank you

Seems like it's megaseconds :) i.e. actual duration = duration * 1000000 You can simply check it calculating the difference between end_timestamp and start_timestamp columns

Thank you :)

I found the xcmetrics working gif pretty useful to understand the tool. I was curious to learn how that gif was created? What tools were used to design/create it?

https://github.com/spotify/XCMetrics/blob/gh-pages/images/xcmetrics-open-source-xcode-tool.gif

A interactive designer designed it for us, and I think some more complex program such as Adobe Animate was used.

Is there any plans for releasing web interface? Even experimental UI is better than RAW data)

Hey @tinder-kaijing did you create metrics on grafana?

It's clearly stating that the crash is caused by an optional with nil value

they can be both in successful or failed results logs this fixes #131

Seems like it's megaseconds :) i.e. actual duration = duration * 1000000 You can simply check it calculating the difference between end_timestamp and start_timestamp columns

I was using 0.2.16 and just saw that I was out of date. Using 0.2.16 I was seeing Swift Type check times reported in the HTML report. I updated to 0.2.26 and re-ran the report and see Did you compile your project with the flags -Xfrontend -debug-time-expression-type-checking

Yes, I did compile with those flags. I ran a report with 0.2.16, it looked fine. Updated to 0.2.26 and ran the report on the exact same project and nothing. I double-checked, just in case something strange happened, and the flag is absolutely present.

The debug-time-function-bodies report is displaying properly.

Hey, I have a question regarding the duration double value for builds, which time unit is it.. microseconds or milliseconds?

Thank you

I have a log that contains the next line

0.08ms	<redacted-folder>/src/pages/headers-list/cells-content/mail-bill/analytic/PaymentAnalyticHandlerInReadView.swift:45:-0#0#0#-7@118"file://<redacted-folder>/src/pages/headers-list/cells-content/mail-bill/analytic/PaymentAnalyticHandler.swift0000000000000000^8797"CompileSwift normal x86_64 <redacted-folder>/src/pages/headers-list/cells-content/mail-bill/analytic/PaymentAnalyticHandler.swift (in target '<redacted-appname>' from project '<redacted-app>')

This line causes the crash Thread 1: Fatal error: Unexpectedly found nil while unwrapping an Optional value at SwiftCompilerTimeOptionParser.swift:59. (PaymentAnalyticHandlerInReadView.swift:45 contains simple method with a single function call.)

Seems like we should remove force optionals unwrapping

We use the latest docker image and Xcode 12.1.

The crash log:

[ NOTICE ] 22/03/2021 17:06:05 [ProcessMetricsJob] fetching log from file:///tmp/7277A4A2-7D1D-4FE5-BDA3-BCFCD68157DC/D6F72682-AEE4-466F-8DCE-FCA4A027B08F.xcactivitylog [job_id: 4DE44B28-C790-462D-A6C9-1AA8E556C4C7]
[ NOTICE ] 22/03/2021 17:06:05 [ProcessMetricsJob] log fetched to file:///tmp/7277A4A2-7D1D-4FE5-BDA3-BCFCD68157DC/D6F72682-AEE4-466F-8DCE-FCA4A027B08F.xcactivitylog [job_id: 4DE44B28-C790-462D-A6C9-1AA8E556C4C7]
0x7fd68045d97f
0x5598e503ee3f, Swift runtime failure: Unexpectedly found nil while unwrapping an Optional value at .build/checkouts/xclogparser/Sources/XCLogParser/parser/SwiftCompilerTimeOptionParser.swift:0
0x5598e503ee3f, function signature specialization <Arg[1] = Dead> of generic specialization <XCLogParser.SwiftCompilerTypeCheckOptionParser> of (extension in XCLogParser):XCLogParser.SwiftCompilerTimeOptionParser.parseNameAndLocation(from: Swift.String) -> Swift.Optional<(Swift.String, Swift.Int, Swift.Int)> at .build/checkouts/xclogparser/Sources/XCLogParser/parser/SwiftCompilerTimeOptionParser.swift:59
0x5598e503ee3f, generic specialization <XCLogParser.SwiftCompilerTypeCheckOptionParser> of (extension in XCLogParser):XCLogParser.SwiftCompilerTimeOptionParser.parseNameAndLocation(from: Swift.String) -> Swift.Optional<(Swift.String, Swift.Int, Swift.Int)> at /build/<compiler-generated>:0
0x5598e503ee3f, closure #1 (Swift.String) -> Swift.Optional<XCLogParser.SwiftTypeCheck> in XCLogParser.SwiftCompilerTypeCheckOptionParser.(parse in _EDF18D047378C5B6247F84B294158F58)(command: Swift.String, occurrences: Swift.Int) -> Swift.Optional<Swift.Array<XCLogParser.SwiftTypeCheck>> at /build/.build/checkouts/xclogparser/Sources/XCLogParser/parser/SwiftCompilerTypeCheckOptionParser.swift:61
0x5598e503ee3f, reabstraction thunk helper from @callee_guaranteed (@guaranteed Swift.String) -> (@owned Swift.Optional<XCLogParser.SwiftTypeCheck>, @error @owned Swift.Error) to @escaping @callee_guaranteed (@in_guaranteed Swift.String) -> (@out Swift.Optional<XCLogParser.SwiftTypeCheck>, @error @owned Swift.Error) at /build/<compiler-generated>:0
0x5598e503ee3f, generic specialization <Swift.Array<Swift.String>, XCLogParser.SwiftTypeCheck> of (extension in Swift):Swift.Sequence.compactMap<A>((A.Element) throws -> Swift.Optional<A1>) throws -> Swift.Array<A1> at /build/<compiler-generated>:0
0x5598e503ee3f, XCLogParser.SwiftCompilerTypeCheckOptionParser.(parse in _EDF18D047378C5B6247F84B294158F58)(command: Swift.String, occurrences: Swift.Int) -> Swift.Optional<Swift.Array<XCLogParser.SwiftTypeCheck>> at /build/.build/checkouts/xclogparser/Sources/XCLogParser/parser/SwiftCompilerTypeCheckOptionParser.swift:48
0x5598e503ee3f, closure #1 ((key: Swift.String, value: Swift.Int)) -> Swift.Optional<Swift.Array<XCLogParser.SwiftTypeCheck>> in XCLogParser.SwiftCompilerTypeCheckOptionParser.parse(from: Swift.Dictionary<Swift.String, Swift.Int>) -> Swift.Dictionary<Swift.String, Swift.Array<XCLogParser.SwiftTypeCheck>> at /build/.build/checkouts/xclogparser/Sources/XCLogParser/parser/SwiftCompilerTypeCheckOptionParser.swift:33
0x5598e503ee3f, reabstraction thunk helper from @callee_guaranteed (@guaranteed Swift.String, @unowned Swift.Int) -> (@owned Swift.Optional<Swift.Array<XCLogParser.SwiftTypeCheck>>, @error @owned Swift.Error) to @escaping @callee_guaranteed (@in_guaranteed (key: Swift.String, value: Swift.Int)) -> (@out Swift.Optional<Swift.Array<XCLogParser.SwiftTypeCheck>>, @error @owned Swift.Error) at /build/<compiler-generated>:0
0x5598e503ee3f, generic specialization <Swift.Dictionary<Swift.String, Swift.Int>, Swift.Array<XCLogParser.SwiftTypeCheck>> of (extension in Swift):Swift.Sequence.compactMap<A>((A.Element) throws -> Swift.Optional<A1>) throws -> Swift.Array<A1> at /build/<compiler-generated>:0
0x5598e503ee3f, XCLogParser.SwiftCompilerTypeCheckOptionParser.parse(from: Swift.Dictionary<Swift.String, Swift.Int>) -> Swift.Dictionary<Swift.String, Swift.Array<XCLogParser.SwiftTypeCheck>> at /build/.build/checkouts/xclogparser/Sources/XCLogParser/parser/SwiftCompilerTypeCheckOptionParser.swift:33
0x5598e503d9a4, XCLogParser.SwiftCompilerParser.parse() -> () at /build/.build/checkouts/xclogparser/Sources/XCLogParser/parser/SwiftCompilerParser.swift:73
0x5598e502d872, XCLogParser.ParserBuildSteps.(decorateWithSwiftcTimes in _3469993AC311903DEC578A49106939F6)(XCLogParser.BuildStep) -> XCLogParser.BuildStep at /build/.build/checkouts/xclogparser/Sources/XCLogParser/parser/ParserBuildSteps.swift:298
0x5598e5028456, XCLogParser.ParserBuildSteps.parse(activityLog: XCLogParser.IDEActivityLog) throws -> XCLogParser.BuildStep at /build/.build/checkouts/xclogparser/Sources/XCLogParser/parser/ParserBuildSteps.swift:95
0x5598e50b7258, function signature specialization <Arg[7] = Dead> of static XCMetricsBackendLib.LogParser.parseFromURL(_: Foundation.URL, machineName: Swift.String, projectName: Swift.String, userId: Swift.String, userIdSHA256: Swift.String, isCI: Swift.Bool, sleepTime: Swift.Optional<Swift.Int>) throws -> XCMetricsBackendLib.BuildMetrics at /build/Sources/XCMetricsBackendLib/UploadMetrics/LogProcessing/LogParser.swift:54
0x5598e50b7dd0, static XCMetricsBackendLib.LogParser.parseFromURL(_: Foundation.URL, machineName: Swift.String, projectName: Swift.String, userId: Swift.String, userIdSHA256: Swift.String, isCI: Swift.Bool, sleepTime: Swift.Optional<Swift.Int>) throws -> XCMetricsBackendLib.BuildMetrics at /build/<compiler-generated>:0
0x5598e50b7dd0, function signature specialization <Arg[3] = Dead> of static XCMetricsBackendLib.MetricsProcessor.process(metricsRequest: XCMetricsBackendLib.UploadMetricsRequest, logURL: Foundation.URL, redactUserData: Swift.Bool) throws -> XCMetricsBackendLib.BuildMetrics at /build/Sources/XCMetricsBackendLib/UploadMetrics/LogProcessing/MetricsProcessor.swift:13
0x5598e50a7812, static XCMetricsBackendLib.MetricsProcessor.process(metricsRequest: XCMetricsBackendLib.UploadMetricsRequest, logURL: Foundation.URL, redactUserData: Swift.Bool) throws -> XCMetricsBackendLib.BuildMetrics at /build/<compiler-generated>:0
0x5598e50a7812, closure #1 () -> () in XCMetricsBackendLib.ProcessMetricsJob.dequeue(Queues.QueueContext, XCMetricsBackendLib.UploadMetricsRequest) -> NIO.EventLoopFuture<()> at /build/Sources/XCMetricsBackendLib/UploadMetrics/Jobs/ProcessMetricsJob.swift:68
0x5598e444d68d, reabstraction thunk helper from @escaping @callee_guaranteed () -> () to @escaping @callee_unowned @convention(block) () -> () at /build/<compiler-generated>:0
0x7fd67fa68406
0x7fd67fa73c28
0x7fd67fa748cd
0x7fd67fa7c521
0x7fd6804526da
0x7fd67e65d71e
0xffffffffffffffff

I'm triggering the xcmetric binary through scripts and didn't use the launcher. I notice that occasionally xcmetric runs forever when I trigger it without any new xcactivitylog.

I did a very brief skim on the source code, please kindly correct me if I'm wrong:

In MetricUploaderLogic.swift, it seems both cleanedUpLogs and savedUploadRequests need to be received before we can send mobiusLoopCompleted? For my cases, I never received savedUploadRequests.

Then I trace it back to UploadMetricsEffectHandler.swift, line 42, there's an empty check

if !failedURLs.isEmpty {
    effects.append(.logsUploadFailed(logs: failedURLs))
}

So it seems when nothing fails, the logsUploadFailed will be skipped, and consequently persistNonUploadedLogs will not be triggered, so in the end no savedUploadRequests events will be emitted.

Currently I removed the empty check and it seems to work for my cases now. Have I understood the problem correctly? And will there be any other unwanted side effects if I remove the empty check?

The issue still reproduces on Xcode 12.1. Output logs from Xcode's Build command are parseable, but from xcodebuild -archive isn't. Archive's logs are smaller than those were build with Xcode (1 MB vs 13 MB). I validate logs withgzip -t ...

I haven't tested this change directly (there seems to be an issue with Azure Redis servers that I haven't solved), but I did test it with RediStack directly in a sample and it worked exactly as expected.

The existing implementation does not support passing in a password. This change allows an envirornment variable REDIS_PASSWORD to be set which will be propagated through.

Here's the catch: This is totally untested. It looks like a straightforward enough change, so I don't foresee any issues, but I haven't yet been able to get things up and running to test. I can probably test at some point, but it will likely be a day or two at minimum before I do.

So because we will be adding the database username and password when deploying the backend, is there another way to build this with a configuration file so that we could encrypt the password/username.

Thank you :)

For static images, I use draw.io However for such gifs, what tools are used? Is it a matter of overlaying multiple static images to form a gif? Are there some tools that make this easier?

I found the xcmetrics working gif pretty useful to understand. I was curious to learn how that gif was created? What tools were used to design/create it?

https://github.com/spotify/XCMetrics/blob/gh-pages/images/xcmetrics-open-source-xcode-tool.gif

Motivation:

We'd like to send build data to a gRPC endpoint based on Bazel's Build Event Stream. I assume this might be of value to other folks either migrating to build iOS projects with Bazel, or DevX teams supporting multiple build systems and interested in a unified solution.

This implementation is workable as-is for us and we'll be launching the alpha immediately. That said, this lives on our fork, and upstreaming isn't really a blocker for us, but we are happy to contribute it back if there is interest.

Implementation:

Based on the presence of XCMETRICS_BES_* env vars, the server now supports booting into "BES Proxy" mode in which it will send parsed xcactivity logs to the specified BES server instead of persisting to the database; this has limited the surface area of the changes to the addition of a new implementation of the MetricsRepository protocol.

As this PR also contains generated gRPC code, I'll update this PR message with a list of the items to look at during review once the links are available.

The command execution does not finish, and the memory usage is increasing rapidly. I check the generated HTML, I see the same situation as in StephenCheung's image

just like the pic, cmd is xclogparser parse --project xxx --reporter html --output xxx

my Xcode version is 12.3 xclogparser version: XCLogParser 0.2.26

I'm using XCLogParser in a command-line tool that filters out some build steps. I would still like to be able to use the HTML reporter for the edited builds steps but HtmlReporter does not have a public initializer. A quick way to achieve what I want would be to make Reporter.makeLogReporter() public. This way all reporters could be initialized if needed

It probably about 6Mb. As for warnings, about 120+ Warnings. Well bump the specs a bit and keep you posted.

@ecamacho Thank you for quick fixing. I confirmed that --omit_notes worked fine. If @jiaxw32 can get the same good result, I think we can close this issue. 🙏 How about you?

work fine for me now, nice work! @sgr-ksmt @ecamacho

test log as follows:

# xclogparser version
% ./xclogparser version
XCLogParser 0.2.26

# parse with containing the notes
% ./xclogparser parse --reporter json --output report.json --file  C5A4CC00-2064-4733-BC2C-AF6A599917CF.xcactivitylog
File written to ~/XCBuildLogParser/tools/report.json

# parse with ignoring the nodes
% ./xclogparser parse --reporter json --output report_without_notes.json --file  C5A4CC00-2064-4733-BC2C-AF6A599917CF.xcactivitylog --omit_notes
File written to ~/XCBuildLogParser/tools/report_omit_notes.json

# list the report file size
% ls -lh *.json
-rw-r--r--  1 user  staff   2.1G  3  3 16:22 report.json
-rw-r--r--  1 user  staff   236M  3  3 16:24 report_omit_notes.json