Improvements to logging in the app using swift-log and structured logging #168

xtremekforever · 2025-01-17T15:26:28Z

This is a draft since I've only done the initial set of changes. The output currently looks like this with the logger being used almost everywhere, and with structured logging being used:

$ swift run swift-sdk-generator make-linux-sdk --swift-version 6.0.3-RELEASE --host x86_64-unknown-linux-gnu --target aarch64-unknown-linux-gnu
warning: 'swift-algorithms': found 1 file(s) which are unhandled; explicitly declare them as resources or exclude from the target
    ~/swift-sdk-generator/.build/checkouts/swift-algorithms/Sources/Algorithms/Documentation.docc
Building for debugging...
[79/79] Linking swift-sdk-generator
Build of product 'swift-sdk-generator' complete! (14.74s)
2025-01-17T10:19:19-0500 info org.swift.swift-sdk-generator : [SwiftSDKGenerator] 
Downloading required toolchain packages...
2025-01-17T10:19:21-0500 info org.swift.swift-sdk-generator : [SwiftSDKGenerator] Using downloaded artifacts in these locations:
2025-01-17T10:19:21-0500 info org.swift.swift-sdk-generator : path=~/swift-sdk-generator/Artifacts/target_swift_6.0.3-RELEASE_aarch64-unknown-linux-gnu.tar.gz [SwiftSDKGenerator] -
2025-01-17T10:19:21-0500 info org.swift.swift-sdk-generator : path=~/swift-sdk-generator/Artifacts/host_swift_6.0.3-RELEASE_x86_64-unknown-linux-gnu.tar.gz [SwiftSDKGenerator] -
2025-01-17T10:19:21-0500 info org.swift.swift-sdk-generator : [SwiftSDKGenerator] 
Parsing Ubuntu packages list...
2025-01-17T10:19:23-0500 info org.swift.swift-sdk-generator : packageCount=11 [SwiftSDKGenerator] Downloading Ubuntu packages...
2025-01-17T10:19:25-0500 info org.swift.swift-sdk-generator : size=40 KB url=http://ports.ubuntu.com/ubuntu-ports/pool/main/g/gcc-12/libgcc-s1_12.3.0-1ubuntu1~22.04_arm64.deb [SwiftSDKGenerator] Downloaded
2025-01-17T10:19:25-0500 info org.swift.swift-sdk-generator : size=57 KB url=http://ports.ubuntu.com/ubuntu-ports/pool/main/z/zlib/zlib1g_1.2.11.dfsg-2ubuntu9.2_arm64.deb [SwiftSDKGenerator] Downloaded
2025-01-17T10:19:25-0500 info org.swift.swift-sdk-generator : size=1.5 MB url=http://ports.ubuntu.com/ubuntu-ports/pool/main/g/glibc/libc6-dev_2.35-0ubuntu3.8_arm64.deb [SwiftSDKGenerator] Downloaded
2025-01-17T10:19:25-0500 info org.swift.swift-sdk-generator : size=163 KB url=http://ports.ubuntu.com/ubuntu-ports/pool/main/z/zlib/zlib1g-dev_1.2.11.dfsg-2ubuntu9.2_arm64.deb [SwiftSDKGenerator] Downloaded
2025-01-17T10:19:25-0500 info org.swift.swift-sdk-generator : size=2.7 MB url=http://ports.ubuntu.com/ubuntu-ports/pool/main/g/glibc/libc6_2.35-0ubuntu3.8_arm64.deb [SwiftSDKGenerator] Downloaded
2025-01-17T10:19:25-0500 info org.swift.swift-sdk-generator : size=661 KB url=http://ports.ubuntu.com/ubuntu-ports/pool/main/g/gcc-12/libstdc++6_12.3.0-1ubuntu1~22.04_arm64.deb [SwiftSDKGenerator] Downloaded
2025-01-17T10:19:25-0500 info org.swift.swift-sdk-generator : size=1.2 MB url=http://ports.ubuntu.com/ubuntu-ports/pool/main/g/gcc-12/libgcc-12-dev_12.3.0-1ubuntu1~22.04_arm64.deb [SwiftSDKGenerator] Downloaded
2025-01-17T10:19:25-0500 info org.swift.swift-sdk-generator : size=1.3 MB url=http://ports.ubuntu.com/ubuntu-ports/pool/main/l/linux/linux-libc-dev_5.15.0-130.140_arm64.deb [SwiftSDKGenerator] Downloaded
2025-01-17T10:19:25-0500 info org.swift.swift-sdk-generator : size=11.6 MB url=http://ports.ubuntu.com/ubuntu-ports/pool/main/i/icu/libicu-dev_70.1-2_arm64.deb [SwiftSDKGenerator] Downloaded
2025-01-17T10:19:25-0500 info org.swift.swift-sdk-generator : size=2.2 MB url=http://ports.ubuntu.com/ubuntu-ports/pool/universe/g/gcc-12/libstdc++-12-dev_12.3.0-1ubuntu1~22.04_arm64.deb [SwiftSDKGenerator] Downloaded
2025-01-17T10:19:25-0500 info org.swift.swift-sdk-generator : size=10.5 MB url=http://ports.ubuntu.com/ubuntu-ports/pool/main/i/icu/libicu70_70.1-2_arm64.deb [SwiftSDKGenerator] Downloaded
2025-01-17T10:19:25-0500 info org.swift.swift-sdk-generator : fileName=libc6_2.35-0ubuntu3.8_arm64.deb [SwiftSDKGenerator] Extracting deb package...
2025-01-17T10:19:25-0500 info org.swift.swift-sdk-generator : fileName=libc6-dev_2.35-0ubuntu3.8_arm64.deb [SwiftSDKGenerator] Extracting deb package...
2025-01-17T10:19:25-0500 info org.swift.swift-sdk-generator : fileName=libgcc-s1_12.3.0-1ubuntu1~22.04_arm64.deb [SwiftSDKGenerator] Extracting deb package...
2025-01-17T10:19:25-0500 info org.swift.swift-sdk-generator : fileName=libgcc-12-dev_12.3.0-1ubuntu1~22.04_arm64.deb [SwiftSDKGenerator] Extracting deb package...
2025-01-17T10:19:25-0500 info org.swift.swift-sdk-generator : fileName=libicu70_70.1-2_arm64.deb [SwiftSDKGenerator] Extracting deb package...
2025-01-17T10:19:25-0500 info org.swift.swift-sdk-generator : fileName=libicu-dev_70.1-2_arm64.deb [SwiftSDKGenerator] Extracting deb package...
2025-01-17T10:19:25-0500 info org.swift.swift-sdk-generator : fileName=libstdc++-12-dev_12.3.0-1ubuntu1~22.04_arm64.deb [SwiftSDKGenerator] Extracting deb package...
2025-01-17T10:19:25-0500 info org.swift.swift-sdk-generator : fileName=libstdc++6_12.3.0-1ubuntu1~22.04_arm64.deb [SwiftSDKGenerator] Extracting deb package...
2025-01-17T10:19:25-0500 info org.swift.swift-sdk-generator : fileName=linux-libc-dev_5.15.0-130.140_arm64.deb [SwiftSDKGenerator] Extracting deb package...
2025-01-17T10:19:25-0500 info org.swift.swift-sdk-generator : fileName=zlib1g_1.2.11.dfsg-2ubuntu9.2_arm64.deb [SwiftSDKGenerator] Extracting deb package...
2025-01-17T10:19:25-0500 info org.swift.swift-sdk-generator : fileName=zlib1g-dev_1.2.11.dfsg-2ubuntu9.2_arm64.deb [SwiftSDKGenerator] Extracting deb package...
2025-01-17T10:19:25-0500 info org.swift.swift-sdk-generator : [SwiftSDKGenerator] 
Unpacking and copying Swift binaries for the host triple...
2025-01-17T10:19:37-0500 info org.swift.swift-sdk-generator : [SwiftSDKGenerator] 
Unpacking Swift distribution for the target triple...
2025-01-17T10:19:49-0500 info org.swift.swift-sdk-generator : [SwiftSDKGenerator] 
Copying Swift core libraries for the target triple into Swift SDK bundle...
2025-01-17T10:19:50-0500 info org.swift.swift-sdk-generator : [SwiftSDKGenerator] 
Fixing up absolute symlinks...
2025-01-17T10:19:51-0500 info org.swift.swift-sdk-generator : [SwiftSDKGenerator] 
Generating toolset JSON file...
2025-01-17T10:19:51-0500 info org.swift.swift-sdk-generator : [SwiftSDKGenerator] 
Generating destination JSON file...
2025-01-17T10:19:51-0500 info org.swift.swift-sdk-generator : [SwiftSDKGenerator] 
Generating .artifactbundle manifest file...
2025-01-17T10:19:51-0500 info org.swift.swift-sdk-generator : [SwiftSDKGenerator] 
All done! Install the newly generated SDK with this command:
swift experimental-sdk install ~/swift-sdk-generator/Bundles/6.0.3-RELEASE_ubuntu_jammy_aarch64.artifactbundle

After that, use the newly installed SDK when building with this command:
swift build --experimental-swift-sdk 6.0.3-RELEASE_ubuntu_jammy_aarch64
2025-01-17T10:19:51-0500 info org.swift.swift-sdk-generator : elapsedTime=31 seconds [GeneratorCLI] 
Generator run finished successfully.

So, questions:

For human readable output, ideally we'd remove the timestamp, label, and module name from the log output, right @euanh ?
Should we maybe look at switching the order of the metadata so that it appears after the log message? e.g. "Generator run finished successfully. elapsedTime=31 seconds" @MaxDesiatov

I haven't figured out how to change the logger output format, but I can have a look a bit later.

- Easier for structured logging.

- And pass a logger to the SwiftSDKRecipe and implementation to be used as well.

MaxDesiatov · 2025-01-17T15:38:21Z

I wonder if we should introduce a --verbose/-v flag that sets the log level to info or change the level for existing messages to something like debug? Or keep levels of individual messages as they are, but introduce a --quiet/-q flag that hides info messages from the output? In either case seems acceptable to keep the format as it is, unless we could form a consensus for making swift-log output as human readable as print output was? (I'm myself open to either possibility at this point)

xtremekforever · 2025-01-17T18:37:51Z

Looks like there is already a --verbose flag:

The default `--target` triple is Linux with the same CPU architecture with host triple

USAGE: swift-sdk-generator make-linux-sdk <options>

OPTIONS:
  --bundle-version <bundle-version>
                          An arbitrary version number for informational purposes. (default: 0.0.1)
  --sdk-name <sdk-name>   Name of the SDK bundle. Defaults to a string composed of Swift version, Linux distribution, Linux
                          release and target CPU architecture.
  --incremental           Experimental: avoid cleaning up toolchain and SDK directories and regenerate the SDK bundle
                          incrementally.
  -v, --verbose           Provide verbose logging output.

So I think setting all of these to .debug level would make sense, with the default log level as .info. --verbose would then change the log level to .debug to show all these messages.

xtremekforever · 2025-01-17T23:48:03Z

So what I ended up doing was setting some of the key actions to .info still, since I think it is useful to know if the generator is performing the essential steps:

$ swift run swift-sdk-generator make-linux-sdk --swift-version 6.0.3-RELEASE --host x86_64-unknown-linux-gnu --t
arget aarch64-unknown-linux-gnu
Building for debugging...
[12/12] Linking swift-sdk-generator
Build of product 'swift-sdk-generator' complete! (2.39s)
2025-01-17T18:33:56-0500 info org.swift.swift-sdk-generator : [SwiftSDKGenerator] Downloading required toolchain packages...
2025-01-17T18:33:57-0500 info org.swift.swift-sdk-generator : [SwiftSDKGenerator] Using downloaded artifacts from cache
2025-01-17T18:33:59-0500 info org.swift.swift-sdk-generator : packageCount=11 [SwiftSDKGenerator] Downloading Ubuntu packages...
2025-01-17T18:34:01-0500 info org.swift.swift-sdk-generator : [SwiftSDKGenerator] Unpacking and copying Swift binaries for the host triple...
2025-01-17T18:34:13-0500 info org.swift.swift-sdk-generator : [SwiftSDKGenerator] Unpacking Swift distribution for the target triple...
2025-01-17T18:34:25-0500 info org.swift.swift-sdk-generator : [SwiftSDKGenerator] Copying Swift core libraries for the target triple into Swift SDK bundle...
2025-01-17T18:34:26-0500 info org.swift.swift-sdk-generator : [SwiftSDKGenerator] Fixing up absolute symlinks...
2025-01-17T18:34:27-0500 info org.swift.swift-sdk-generator : [SwiftSDKGenerator] Generating toolset JSON file...
2025-01-17T18:34:27-0500 info org.swift.swift-sdk-generator : [SwiftSDKGenerator] Generating destination JSON file...
2025-01-17T18:34:27-0500 info org.swift.swift-sdk-generator : [SwiftSDKGenerator] Generating .artifactbundle info JSON file...
2025-01-17T18:34:27-0500 info org.swift.swift-sdk-generator : [SwiftSDKGenerator] 

All done! Install the newly generated SDK with this command:
swift experimental-sdk install ~/swift-sdk-generator/Bundles/6.0.3-RELEASE_ubuntu_jammy_aarch64.artifactbundle

After that, use the newly installed SDK when building with this command:
swift build --experimental-swift-sdk 6.0.3-RELEASE_ubuntu_jammy_aarch64

2025-01-17T18:34:27-0500 info org.swift.swift-sdk-generator : elapsedTime=31 seconds [GeneratorCLI] Generator run finished successfully.

However, when you pass --verbose to the generator, then .debug mode is enabled and everything gets printed! Let me know what you think @MaxDesiatov and @euanh

Sources/SwiftSDKGenerator/Generator/SwiftSDKGenerator+Download.swift

Sources/SwiftSDKGenerator/Generator/SwiftSDKGenerator.swift

Sources/SwiftSDKGenerator/Generator/SwiftSDKGenerator+Download.swift

- This was missed in the last set of changes, but important to note that the generator can now generate Swift SDKs to cross compile between different Linux distributions.

MaxDesiatov · 2025-01-22T10:09:09Z

@swift-ci test

xtremekforever · 2025-02-02T20:28:21Z

@MaxDesiatov I fixed the merge conflict. Sorry I didn't notice before- I came by a few times and just saw green checkmarks and not the conflict.

Please let me know if there's anything else here you'd like to me look at or refine!

MaxDesiatov · 2025-02-05T12:37:58Z

LGTM, but I'd like to get some feedback from @euanh and @kateinoigakukun

MaxDesiatov · 2025-02-05T14:49:52Z

@swift-ci test

README.md

MaxDesiatov · 2025-02-05T14:52:25Z

@swift-ci test

xtremekforever added 4 commits January 16, 2025 11:45

Update SwiftSDKGenerator and GeneratorCLI to use logger instead of print

5d93b4e

Merge branch 'main' into improvement/structured-logging

0ae3938

Use metadata instead of string interpolation for logger.info calls

4fe2774

- Easier for structured logging.

Make logGenerationStep an extension of Logger

8246d9e

- And pass a logger to the SwiftSDKRecipe and implementation to be used as well.

xtremekforever added 2 commits January 17, 2025 18:35

Remove logGenerationStep, log using .debug for detailed logs

7d0501b

Set logLevel from --verbose flag

8310aa2

Add logger to DownloadArtifactQuery

47f4c42

MaxDesiatov reviewed Jan 20, 2025

View reviewed changes

Sources/SwiftSDKGenerator/Generator/SwiftSDKGenerator+Download.swift Outdated Show resolved Hide resolved

MaxDesiatov reviewed Jan 20, 2025

View reviewed changes

Sources/SwiftSDKGenerator/Generator/SwiftSDKGenerator.swift Outdated Show resolved Hide resolved

Implement comments for logging output

92dba49

xtremekforever marked this pull request as ready for review January 21, 2025 01:29

MaxDesiatov reviewed Jan 21, 2025

View reviewed changes

Sources/SwiftSDKGenerator/Generator/SwiftSDKGenerator.swift Outdated Show resolved Hide resolved

Add log text to lsOutput print

0bbfce5

MaxDesiatov requested review from euanh and kateinoigakukun January 21, 2025 15:16

MaxDesiatov reviewed Jan 21, 2025

View reviewed changes

Sources/SwiftSDKGenerator/Generator/SwiftSDKGenerator+Download.swift Outdated Show resolved Hide resolved

xtremekforever added 3 commits January 21, 2025 21:09

Collect all downloaded artifact paths in single log message metadata

b632cf2

Merge branch 'main' into improvement/structured-logging

a836107

Update README supported platforms section to note Linux is now supported

7142b01

- This was missed in the last set of changes, but important to note that the generator can now generate Swift SDKs to cross compile between different Linux distributions.

xtremekforever requested a review from MaxDesiatov January 22, 2025 02:16

MaxDesiatov approved these changes Jan 22, 2025

View reviewed changes

MaxDesiatov added the enhancement New feature or request label Jan 22, 2025

Merge branch 'main' into improvement/structured-logging

0787866

kateinoigakukun approved these changes Feb 5, 2025

View reviewed changes

euanh approved these changes Feb 5, 2025

View reviewed changes

Merge branch 'main' into improvement/structured-logging

a3a1d4e

MaxDesiatov reviewed Feb 5, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

Apply suggestions from code review

f0ab4ed

MaxDesiatov enabled auto-merge (squash) February 5, 2025 14:52

MaxDesiatov merged commit c6ebe92 into swiftlang:main Feb 5, 2025
2 of 3 checks passed

xtremekforever deleted the improvement/structured-logging branch February 5, 2025 14:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Improvements to logging in the app using swift-log and structured logging #168

Improvements to logging in the app using swift-log and structured logging #168

Uh oh!

xtremekforever commented Jan 17, 2025

Uh oh!

MaxDesiatov commented Jan 17, 2025 •

edited

Loading

Uh oh!

xtremekforever commented Jan 17, 2025

Uh oh!

xtremekforever commented Jan 17, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MaxDesiatov commented Jan 22, 2025

Uh oh!

xtremekforever commented Feb 2, 2025 •

edited

Loading

Uh oh!

MaxDesiatov commented Feb 5, 2025 •

edited

Loading

Uh oh!

MaxDesiatov commented Feb 5, 2025

Uh oh!

Uh oh!

MaxDesiatov commented Feb 5, 2025

Uh oh!

Uh oh!

Uh oh!

Improvements to logging in the app using swift-log and structured logging #168

Improvements to logging in the app using swift-log and structured logging #168

Uh oh!

Conversation

xtremekforever commented Jan 17, 2025

Uh oh!

MaxDesiatov commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xtremekforever commented Jan 17, 2025

Uh oh!

xtremekforever commented Jan 17, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MaxDesiatov commented Jan 22, 2025

Uh oh!

xtremekforever commented Feb 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MaxDesiatov commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MaxDesiatov commented Feb 5, 2025

Uh oh!

Uh oh!

MaxDesiatov commented Feb 5, 2025

Uh oh!

Uh oh!

Uh oh!

MaxDesiatov commented Jan 17, 2025 •

edited

Loading

xtremekforever commented Feb 2, 2025 •

edited

Loading

MaxDesiatov commented Feb 5, 2025 •

edited

Loading