# Swift Server Guides

(https://hackmd.io/@kTmq8wwxT-S7kpJjg_Bevg/ryYcanUAxl)

Migration and updating for existing guides reflected in https://github.com/heckj/swift-org-website/blob/migration-plan-specifics/_info-architecture/0003-swift-docs-proposal.md#migration-plan. This document provides an overview and summary for
intended changes to update, broaden, and deepen the content for Swift Server.

⚙️ Update Process (planned - hoping to have these as issues on a swift docs repository)

* create issue for each document (GH issue, w/ matching radar)
* updated outline → HackMD (vs. updated content → HackMD)
* include in GH Issue - discuss outline in issue, or perhaps on forums if broader audience is warranted or needed
* vet outline w/ server group (time-boxed, consensus based)
* initial draft → PR on repo, referencing the issue
* review and iterate with PR
* close issue on merge

Migration Issues Created:
- https://github.com/swiftlang/swift-org-website/issues/1201
- https://github.com/swiftlang/swift-org-website/issues/1202

The existing guides follow a pattern that I interpret as related to a development cycle (the change code, build, test, evaluate cycle):

* building ([current](https://www.swift.org/documentation/server/guides/building.html))
* currently specifies “what” to use - swiftpm, eschews anything else
* touches on using docker for building
* mixes a bit at the tail end talking about other areas:
* basic performance during release builds
* enabling swift-backtrace
* 🧭 North Star: show examples of how to build executables (& libraries)
* (core path: developing on Mac, running on Linux)
* how to cross-compile from macOS → Linux
* how to compile in Linux while on macOS (using container)
* (reference separate article - Packaging - for building a container)
* differentiate between building for local dev & production
* build options that are common/useful for building
* -c release —product ...
* --static-swift-stdlib for linux (or not?)
* other options that do something specific?
* -Xcc -fno-omit-frame-pointer ?? better backtraces?
* --explicit-target-dependency-import-check ?
* --scratch-path to change from .build
* ✍️ updates
* add in a section on “build on macOS, deploy on linux”
* add in how to use “container” to build in linux
* add in a section on using dev container w/ VSCode (or a reference to how to use it - another article?)
* add in static linux SDK reference for static binary/cross compilation
* add reference to SwiftPM documentation for debug/release options
* add reference to “how to measure and optimize performance” overview (point to another article)
* remove ref to including swift-backtrace
* update/verify all code examples with “container” and “swift 6” as a baseline
* 🗺️ (Outline)
* configuration (`-c`) debug & release
* `swift build` - just checkin' - debug output, runable
* `swift build -c release` - optimized, runnable
* ? other CLI options while building? - other optimizations/controls that are common or useful?
* `-Xcc -fno-omit-frame-pointer` for potentially improved better backtraces?
* makes it easier to unwind stack traces with `perf` or `gdb`
* - generally only with `-c debug`
* -fomit-frame-pointer shouldn't be on by default, really, so that shouldn't be necessary, but if it is turned on for some reason then yes, adding -fno-omit-frame-pointer to turn it off again would be a good idea.
* `-Xswiftc -cross-module-optimization`
* increases code size, longer build times, but potentially better performance optimizations
* ref: https://forums.swift.org/t/stance-on-cross-module-optimization/73288
* - generally only with a `-c release`
* (checking with Alastair for backtracing and stack unwinding)
* --product for a focused build (aka -c release --product ...) to avoid building extra stuff
* choosing traits (Swift 6.3+)
* some packages started to expose controls activated by traits
* `--traits`
* `swift package show-traits`
* locating binaries (bin path)
* `swift build --show-bin-path`
* `swift build -c release --show-bin-path`
* building to run on other platforms
* using a container to check Linux build
* how to build using Swift toolchain in Linux using a container - `container` CLI (aka `docker` CLI)
* `container run --rm -it -c 4 -m 8g -v "$(pwd):/src" -w /src swift:6.2`
* (reference "how to build a container" to another article)
* compiling with the Static Linux SDK
* `--swift-sdk ...`
* `x86_64-swift-linux-musl`
* `aarch64-swift-linux-musl`
* When on linux
* compile dynamic
* `--static-swift-stdlib` - do we ever want to suggest this anymore?
* impact on backtraces - without it, backtraces aren't great?
* 6.0 -> 6.2: with - gives readable backtraces and long link times.
* Omitting it gives garbage backtraces and quick links
* The --static-swift-stdlib option causes us to statically link just the Swift standard library, so you don't need to ship libswiftCore.so with your program, but your program will still dynamically link the usual libswiftCore dependencies.
* Example use was to deliver a binary asset that can be run on a platform that might not have the Swift runtime installed. The main time was when I was producing JARs that were wrapping a Swift dynamic lib, but that dylib had statically linked the Swift stdlib.

* testing ([current](https://www.swift.org/documentation/server/guides/testing.html))
* 🧭 North Star: show examples of how to run tests for executables (& libraries, but focus on executables)
* (core path: developing on Mac, running on Linux)
* (presumption that tests are unit tests)
* show how to run test within a linux environment
* using container on the CLI
* show how to use sanitizers (thread, asan, while testing)
* show how to capture test coverage & get a simple report
* +200 for adding some reference to code coverage
* XCTest & Swift Testing (prefer Swift testing)
* reference “testing in a CI article” (new/separate article)
* XCTest still used - keep reference
* docker example of testing in linux using a container
* legacy references to test search, all focused on XCTest
* testing for production
* suggests using sanitizers - TSan, ASan
* touches into suggestions for what to do/run in CI
* ✍️ update
* update to Swift testing formats, include XCTest, but note legacy
* focus on unit testing, leave room for “functional” or “integration” tests
* break out content about about sanitizers into a separate CI article
* add examples for swift testing, reference swift-testing documentation
* drop content relevant to Swift 5.4 and earlier (test discovery, etc)
* 🗺️ (Outline)
* use `swift test`
* `-v`
* ``-vv`
* `list`
* `last`
* `--filter ...`
* `-s`
* run swift test in a container:
* `docker run -v "$PWD:/code" -w /code swift:latest swift test`
* Using sanitizers
* thread:
* `swift test --sanitize=thread`
* address:
* `swift test --sanitize=address`
* `swift test -c release --sanitize=address`
* (something about capturing test coverage)
* `swift test --enable-code-coverage/--disable-code-coverage`
* `--show-codecov-path`
* `--xunit-output <xunit-output>`
* ? how is this intended to be used
* `swift test --xunit-output fredfile` ==> `fredfile-swift-testing`
* `swift test --xunit-output .` ==> nothing
* https://forums.swift.org/t/swift-testing-and-vscode/70441
* https://github.com/swiftlang/swift-package-manager/issues/4752
* XCTest and swift-testing
* `--enable-xctest/--disable-xctest`
* `--enable-swift-testing/--disable-swift-testing`
*
* packaging ([current](https://www.swift.org/documentation/server/guides/packaging.html))
* 🧭 North Star: provide guidance on how to choose the packaging to use for your app
* mention explicitly swift-container-plugin
* docker
* single-stage build example
* doesn’t show/highlight multi-stage builds
* distroless - abandon
* archive (tarball)
* references that cross compiling isn’t an option (OUTDATED)
* source distribution
* static linking and curl/XML
* notes about additional dependencies needed
* ✍️ update
* add overview of options/guidepoint
* static binary
* container (builder & runtime)
* VM or bare metal - (tarball or local-to-host build)
* swift runtime dependencies - matching builder
* primarily revise into guidepost article that points to how-tos in other articles
* add references to using a lambda or compiling to a build-pack oriented setup (Heroku, Fly.io)
* check on static linking and dependencies - deprecate in favor of using “published docker images”
* reference static SDK option, content from build article?
* update Docker build to use a two-stage build, provide example
* break out distroless into it’s own article
* packaging-container (new)
* 🧭 North Star: guidance on how to build your app into a container
* use a pre-built container(s) (swift)
* notion of separate builder and runtime container images
* use your org’s distribution binaries (security cleaned/vetted)
* reference build article for how to compile, but provide Dockerfile example(s)
* use Container as a baseline OSS tool instead of Docker
* Dockerfile example - simple, local build
* Dockerfile example - two-stage build
* illustrate using caching to improve build times (package resolve sep. from build)
* (??) package resolve from Package.resolved file, if there?
* Enabling backtracing (simple example)
* explain single & multi-architecture builds
* mention swift-container-plugin (https://github.com/apple/swift-container-plugin) as an option
* example of how to inspect a built image
* verify architecture
* verify architecture of binaries in the container
* example of how to run locally
* ports
* ENV
* (?) variables/matrix builds
* tags/metadata
* volume mounts
* example of how to run locally (--rm -it yourContainer /bin/bash) to debug
* CI for apps (new)
* 🧭 North Star: show examples of continuous integration setups for Swift server packages - particularly executables)
* lean into expectation of using GitHub Actions, but keep technology neutral - that is, don’t use GH Action specifics features, but use in examples
* building
* build for testing (compilation check)
* testing
* test w/ ASAN
* test w/ TSAN
* capture code coverage, generate report
* linting
* lint/format check
* (?) caching
* (?) leveraging images, pre-compilation caches of any dependencies or binary dependencies
* CI for libraries (new)
* 🧭 North Star: show examples of continuous integration setups for Swift LIBRARY packages - very different expectations than executables
* lean into expectation of using GitHub Actions, but keep technology neutral - that is, don’t use GH Action specifics features, but use in examples
* building
* compilation on matrix of swift versions
* testing
* testing on matrix (swift versions)
* capturing & reporting coverage
* documentation
* vetting documentation/documentation coverage
* (?) .spi.yml (aside/tip for setting up an SPI.yml file for hosting docs)
* checking API breakage
* (?) are any of the Swiftlang GH action workflows something that are re-usable and can/should be leveraged?
* deploying ([current](https://www.swift.org/documentation/server/guides/deployment.html))
* 🧭 North Star: provide guidance on what options are available to deploy your app, and a guide post to find deeper information per deployment target
* VM - various cloud platforms
* container (for docker, kubernetes)
* build pack oriented (for lambda, heroku, fly.io)
* directional article that provides links to specific deployment targets
* lambda serverless
* AWS Fargate
* AWS EC2
* DigitalOcean
* Heroku
* Kubernetes & Docker
* GCP
* includes a note about making a debuggable configuration
* ✍️ update
* remove LLDB command as suggestion, lean into static backtracking
* migrate LLDB command into an article on “debugging” - “running/debugging” (Mac, linux)
* llvm-sanitizers ([current](https://www.swift.org/documentation/server/guides/llvm-sanitizers.html))
* swift build & test commands - ASAN, TSAN
* ✍️ deprecate and merge into build and test articles

* (new) docker-compose or something to run locally w/ a deploy example
* ? localstack for integration testing

* (new) how to wrangle a sidecar for metrics/telemetry/tracing

Performance focused content

* performance ([current](https://www.swift.org/documentation/server/guides/performance.html))
* 🧭 North Star: An overview and guide on how to approach performance measurement and optimization, focused on services/apps first - in particular server use cases, running on linux.
* measure, profile, benchmark
* is it slow, where is it slow,
* improving it, keeping it from regressing
* broadest measurement first - breakdown techniques
* broad algorithmic mistakes first - accidental quadratics, etc
* allocations and context switching optimization later
* building an expectation for what should be taking the most time
* how do you vet your hypothesis?
* flamegraphs (what they are, how to read) → ✍️ break out into it’s own article?
* malloc libraries (TCMalloc, Jemalloc)
* ✍️ reference to Xcode + instruments
* allocations instrument
* sampling/running sample on macOS
* linux-perf ([current](https://www.swift.org/documentation/server/guides/linux-perf.html))
* what is the “perf” command - how to use the linux command
* install
* how to use
* baremetal
* docker
* docker for mac
* vm
* memory leaks and usage ([current](https://www.swift.org/documentation/server/guides/memory-leaks-and-usage.html))
* techniques
* profiling tools (ref to allocations), valgrid, heaptrack
* review code
* on macOS using MallocStackLogging
* debugging with valgrind
* how to w/ examples
* talking to demangle command
* debugging using ASAN
* debugging using HEAPTRACK
* allocations ([current](https://www.swift.org/documentation/server/guides/allocations.html))
* explain heap and stack
* profiling
* instruments or perf
* allocations instrument on macOS
* how to create a flamegraph
* linux. perf tool intro
* perf probes for allocations
* allocation analysis
* counting allocations
* collecting raw data
* creating flame graphs from the data
* how to read the resulting flame graph
* dtrace allocation data on macOS
* some detail about Swift’s allocation patterns
* dwarf call stack unwinding and “missing data” explainer

✍️ possible new content

* ✍️ build beyond Swift Package (doesn’t yet exist)
* CMake - how to set up, create deliverable, establish tests, handle dependencies
* ✍️ creating artifactBundle
* pre-compiled C libraries for Swift on Server (ref/build on temporal.io work, prev. automerge experience)
* (example uses Rust library from Swift through C interface)
* leverage CMake article to provide binaries
* ✍️ debugging (doesn’t yet exist)
* general debugging techniques
* how to run locally - macOS/Xcode
* how to run locally - linux (container)
* how to run locally (docker-compose/multiple services)
* how to run locally - devcontainer (VSCode reference)?
* how to enable remote debugging
* socket-based debugging - container w/ permissions?
* optional “lldb” based command where permissions allow

lldb —batch -o "break set -n main —auto-continue 1 -C \"process handle SIGPIPE -s 0\"" -o run -k "image list" -k "register read" -k "bt all" -k "exit 134" ./my-program`

* printf debugging
* telemetry - logging
* telemetry - metrics
* telemetry - tracing
* single service vs. multi-service setups
* docker-compose
* referencing external services
* (swift-configuration reference?)
* running locally - in container
* running locally - in sep process
* ✍️ swift-backtracing & dealing with crashes
* dealing with crashes
* symbolicating, capturing crash reports
* LLDB usage to inspect
* ✍️ microbenchmarks & performance
* package-benchmark
* swift-collections-benchmark
* ✍️ performance characterization
* memory usage (RSS/resident) for operations (tuning for Kube)
* performance per request
* `wrk` and other driver commands
* synthetic workloads
* metrics and tracing to characterize
* workload characterization
* hot path, cold path flows
* long-tail/percentile distributions of timing (stochastic methods & histograms)
* ✍️ macro performance tuning
* measure-adjust-evaluate comparison loop
* tracing
* swift-performance-profiler
* (?parca)
* measure/characterizing load and bottleneck
* sampling/perf w/in a single server
* flamegraph generation from in-process profiler
* ✍️ functional & integration testing
* ?? maybe before performance characterization and optimization
* 🧭 North Star: talk about how to set up and run tests that require outside services and/or resources
* using containers for resources (locally, or in CI)
* leverage swift-testing exit tests as a test driver framework
* state setup and validation
* localstack - simulation of AWS services
* ✍️ GitHib Actions continuous integration guides/docs (doesn’t yet exist)
* lean in Github Actions?
* soundness
* docs check
* (Apple code leverages license header check)
* unit tests
* multiple platforms/matrix of swift versions
* NIO examples have some specific overrides
* cxx interop (https://github.com/apple/swift-nio/blob/main/.github/workflows/cxx_interop.yml)
* matrix build of swift versions
* macOS tests (https://github.com/apple/swift-nio/blob/main/.github/workflows/macos_tests.yml)
* matrix build of various Xcodes and platform combinations w/ some overrides enabled
* release builds
* for a library
* leverage guidance from Ecosystems group for N-2 platform support suggestion & testing
* are any of the workflows and actions re-usable (NIO, Apple, Swiftlang?)
* semantic version label check
* https://github.com/apple/swift-metrics/blob/main/.github/workflows/pull_request_label.yml
* https://github.com/apple/swift-nio/blob/main/.github/workflows/pull_request_label.yml
* https://github.com/apple/swift-nio/blob/main/.github/actions/pull_request_semver_label_checker/action.yml