KSP and Me

I’ve been using Kotlin Symbol Processing (KSP) for a few years so I thought I’d reflect on how I like to work with it to stay productive.

First things First

Let’s start by recognising if you are new to KSP it is hard to get up to speed, it’s not impossible but it will require some graft to really get stuck in. Many of the blog posts I read when I was starting were very good at helping you get something compiling but then pretty much finished there. Without someone holding my hand or giving me cues of where to look I was kind of stuck not really knowing the potential of the tool I was learning.

Don’t treat what you read on the internet as gospel

Many of the blog posts I read when starting out had a similar pattern of suggesting you should use the visitor pattern and KotlinPoet, without really saying why you’d want to use them. I’ve read the Gang of Four book many moons ago but had all but forgotten the visitor pattern and I’d never heard of KotlinPoet so that’s two things I was expected to learn just to follow an introductory tutorial.

Thankfully I’m a few years in and I’ve mostly managed to avoid using the visitor pattern for my use cases. My coding style these days leans more towards a functional style so less common OO patterns just feel alien and slow me down.

For example to get all of a class’ nested children I could use the visitor pattern something like this:

class MyVisitor: KSDefaultVisitor<Unit, Klass?>() {
    override fun visitClassDeclaration(classDeclaration: KSClassDeclaration, data: Unit) = Klass(
        classDeclaration.simpleName.asString(),
        classDeclaration.declarations.mapNotNull { it.accept(this, Unit) }.toList()
    )

    override fun defaultHandler(node: KSNode, data: Unit): Klass? = null
}

Tangent: None of the examples I read at the time actually accumulated results in this functional style using the second type parameter but instead opted but having an instance variable that you accessed after parsing was completed.

Or I could use a more functional style like this:

fun KSClassDeclaration.nestedClasses(): Klass = Klass(
    simpleName.asString(),
    declarations.filterIsInstance<KSClassDeclaration>().map(KSClassDeclaration::nestedClasses).toList()
)

The functional style I personally find more direct and I can see the recursion happening I’m not relying on learning what the various conformances to visitor are and which is right for my use case and the methods I need to use/implement (accept, defaultHandler) and why.

Anyway I’m not trying to sell one approach over the other because that’s for you and your team to thrash out. I’m mostly just saying if it works then use it, you don’t have to feel like I did that I was somehow holding it wrong because my code didn’t look like all the blog posts I was reading.

The other good thing to report is that I haven’t needed to learn KotlinPoet, again for the things I’ve worked on multiline string literals have been more than adequate. I mean I know what the Kotlin code I want to generate should look like so having an extra layer in the middle doesn’t add much for me personally.

Separate parsing and generating

When I started I kept trying to build up the final String of what the Kotlin source code should be whilst parsing the code. This is not a great idea as you soon tie yourself in knots. What compilers tend to do, which is the pattern I follow now is

• Parse the input into some intermediate representation
• Process the intermediate representation
• Render the output

For step 1 I like to take the types provided by KSP such as KSClassDeclaration and extract out the information I need into simple data class types. That way the processing logic I write next doesn’t need to know about KSP and the task is more focussed on gathering all the data that my processor thinks is interesting.

Once I have the data I’ll then do any validation, filtering or mapping to new representations. At this point I’m working with simple immutable data classs with well named properties, which is much preferred to having all my business logic calling all combinations of deeply nested resolve(), declaration, asString(), etc.

The final step is rendering, which is very often now just a collection of string templates that interpolate in the nicely structured data from the previous step.

I think there are a few great advantages to separating things out:

• You can generate different code for different targets (e.g. Kotlin/JS vs Kotlin/JVM) in a much simpler way
• Future readers don’t have to follow a potential mess of building a string whilst parsing
• More easily add unit tests around the business rules in the processor

Validate before you generate

Linked to the mistake mentioned in the section above about trying to do things in one go I would fully recommend writing out the code you want to generate manually and checking it works. I’ve found that I was constantly starting off with a simple picture of what I needed to generate and it seemed so obvious what was needed that I started writing the generation code. The issue is I’m not a very good developer and the simple code I imagine never really works and often requires changes. It’s much simpler to edit, compile and run code directly rather than trying to change the code to generate new code so that you can run and validate it.

Example use cases

The biggest pain point for me was not having that spark of inspiration for what I could be doing with KSP. Here’s a few things that me and my team have used KSP for:

• Validation tasks
  • Ensuring that a module correctly uses functions or computed properties, this is a bit niche but this module houses UI strings and if we use properties then every single string would end up in the JS artifact even if it wasn’t referenced. To avoid every JS artifact having every possible string we rely on the dead code elimination you get when the compiler notices you don’t invoke a function.
  • Ensuring that certain types conform to Serializable to support Android. If we forget the conformance then we could crash at runtime if an activity tries to serialize state.
• Generate type aliases
  • KMP doesn’t export typealises to iOS and the naming rules for types can be a little funky. Some times subtypes have dot separators (Parent.Child) and other times the symbol names are just smashed together (ParentChild). This is super confusing and we want to alias the most recent versions of some generated types so iOS developers never know about the actual versioning. The processor for this outputs Swift code, which is then packaged via SKIE.
• Generate type safe routing helpers
  • A colleague wrote a processor that will read various spring boot annotations to calculate the path for an endpoint and what arguments are required. This is then all used to generate typesafe helper functions that allow people to do routing in a much safer manor.
• Generate DSL versions
  • Me and a colleague wrote a pretty comprehensive processor that generates type safe versioned DSLs allowing us to migrate away from a system that meant adding a new version of our DSL required fairly specialised knowledge of our versioning system + many hours-days of work and resulted in inconsistent results to now mostly just bumping a number.
• Generating observability wrappers
  • Me and various colleagues wrote a processor that takes a class with some annotations sprinkled on it and generates a wrapper class that knows when properties are being written and will require us to recompute a new state. This processor also generates type safe bindings that allow us to bind our UI to these properties.
• Generate per request caches in webflux
  • I’m still learning webflux but a requirement came up to have a per request cache, this would normally be done with an @Cacheable annotation on a method in a normal thread based spring boot application. What I ended up spiking was having KSP look for an annotation of @Memoize which then generated a CoWebFilter to create a typed caffeine cache and slap it in the coroutine context. Then the KSP would generate a wrapper class that delegates to the original after trying the cache. This generated delegate wrapper would have a @Primary annotation so spring would wire it in rather than original.

There’s plenty more example uses out there these days if you look around but all of the above are either in live active projects or hopefully will be soon.

Conclusion

I think it’s great to have good documentation on how to use a library but sometimes the thing that is missing is the little bit of inspiration that get you thinking about how you could apply a technology to your project. I’m glad we embraced KSP and we have done away with so much boilerplate code and all the opportunities for mistakes and inconsistencies to sneak in that makes maintenance harder.

CustomTestStringConvertible

SwiftTesting’s parameterised tests are really great. I’ve been finding that I often want to give each example a nice name that shows in the test navigator as the default String(describing:) approach doesn’t always hit the mark.

Let’s take the contrived example of testing an enum initialiser that allows the enum to be constructed with a case insensitive string.

Attempt 1

As a first stab we might go for

struct SoftwareTests {
    @Test("Lowercase names can be used")
    func lowercaseNamesCanBeUsed() {
        #expect(Software("macos") == .macOS)
    }

    @Test("Uppercase names can be used")
    func uppercaseNamesCanBeUsed() {
        #expect(Software("MACOS") == .macOS)
    }

    @Test("Unknown names are mapped to unknown")
    func unknownNamesAreMappedToUnknown() {
        #expect(Software("??") == .unknown)
    }
}

This is fine and results in the test navigator showing our tests like this:

└── SoftwareTests
    ├── Lowercase names can be used
    ├── Uppercase names can be used
    └── Unknown names are mapped to unknown

This all looks fairly reasonable but even in this simple example we can see duplication. Each test repeats the exact same pattern in the expectation. Full disclosure I probably wouldn’t bother changing this code as it’s already fairly concise but let’s imagine that the test bodies were a little longer and there was duplicated set up happening in the bodies.

Attempt 2

In this case you’d want to jump to some parameterised tests which might look like this:

struct SoftwareTests {
    @Test(
        "Init is case insensitive and handles unknown cases",
        arguments: [
            (input: "macos", expected: Software.macOS),
            (input: "MACOS", expected: Software.macOS),
            (input: "??", expected: Software.unknown),
        ]
    )
    func initIsCaseInsensitiveAndHandlesUnknownCases(input: String, expected: Software) {
        #expect(Software(input) == expected)
    }
}

The duplication is gone and the different permutations are run in parallel which is great for test performance. The issue is that we’ve made the test navigator view a little less useful as it now looks like this:

└── SoftwareTests
    └── Init is case insensitive and handles unknown cases
        ├── "macos", .macOS
        ├── "MACOS", .macOS
        └── "??", .unknown

Those labels don’t really mean much unless you read the test implementation. Something to note is that the actual arguments declaration in the @Test annotation is using labels to make it easier to read the test set up to know which field is the input vs the expected. Although the code source is enhanced with these labels the test navigator is not so clear.

Attempt 3

Let’s fix the previous issue using the CustomTestStringConvertible protocol

struct SoftwareTests {
    @Test(
        "Init is case insensitive and handles unknown cases",
        arguments: CaseInsensitiveInit.allCases
    )
    func initIsCaseInsensitiveAndHandlesUnknownCases(fixture: CaseInsensitiveInit) {
        #expect(Software(fixture.input) == fixture.expected)
    }

    struct CaseInsensitiveInit: CustomTestStringConvertible, CaseIterable {
        let input: String
        let expected: Software
        let testDescription: String

        static let allCases: [CaseInsensitiveInit] = [
            .init(input: "macos", expected: .macOS, testDescription: "Lowercase names can be used"),
            .init(input: "MACOS", expected: .macOS, testDescription: "Uppercase names can be used"),
            .init(input: "??", expected: .unknown, testDescription: "Unknown names are mapped to unknown"),
        ]
    }
}

With this set up the test navigator is looking much nicer:

└── SoftwareTests
    └── Init is case insensitive and handles unknown cases
        ├── Lowercase names can be used
        ├── Uppercase names can be used
        └── Unknown names are mapped to unknown

We’ve restored the handy naming whilst keeping the ability for the framework to optimise and call all the cases in parallel.

Going further

With the above we have to add boiler plate but the benefits are quite useful. There are common types where we can provide helper functions to make this process a little smoother like booleans. If we create a little helper like this:

struct DescribedBool: CustomTestStringConvertible {
    let testDescription: String
    let value: Bool
}

func boolTestStringConvertible(label: (Bool) -> String) -> [DescribedBool] {
   [
    .init(testDescription: label(false), value: false),
    .init(testDescription: label(true), value: true),
   ]
}

Then we can write tests that can be described a lot easier

@Test("Display an appropriate install state", arguments: boolTestStringConvertible { "installed = \($0)"})
func displayAnAppropriateInstallState(fixture: DescribedBool) {
    // ...
}

NB: The above will hopefully work in future but due to way the macro currently works it doesn’t like there being a closure in the arguments position. We can work around this by adding a little shim

@Test("Display an appropriate install state", arguments: installedCases())
func displayAnAppropriateInstallState(fixture: DescribedBool) {
    // ...
}

private static func installedCases() -> [DescribedBool] {
    boolTestStringConvertible { "installed = \($0)" }
}

With this in place we get nice descriptions like this:

└── SoftwareTests
    └── Display an appropriate install state
        ├── installed = false
        └── installed = true

Conclusion

SwiftTesting parameterised tests are great. It’s also very easy just to slap a load of test cases in simple tuples and exercise a lot of things but maybe lose some clarity in the test navigator around what the tests are doing. Using CustomTestStringConvertible is a nice way to bring some order back and help yourself and other travellers of your code base to navigate some hopefully extensive tests suites.

Building Complex Things

I’m always fascinated when people build complex things, not so much by the final artifact but by the journey they travelled to get to the end result. Projects are very rarely plotted with a straight line from problem statement to final solution but when all you see is the final product it’s easy to discount the work that went into its creation with all the interesting choices and solutions to problems you’ll never know existed.

Here’s a retelling of a journey I went on to build a macOS virtualised CI solution. This is not a how to guide but in theory if you follow along you could build out your own working solution.

It Began

Our CI set up at work is a few Mac Studios that each have two CI runners installed per physical machine. The machines themselves had all been configured with a well crafted script that my colleague Sam made that installed all required tooling and got the environment into a good shape. The problem that we kept facing is that on first set up the machines were in a known good state but after that it was the wild west. Anyone could remote onto the machine and run anything they liked, whilst no one would ever do anything malicious, over time the machines drift or just generally become less healthy and builds become less repeatable.

I’d been thinking for a while about virtualisation and repeatable builds. All of our backend is set up around docker which gives you this but wouldn’t it be nice to have it for our builds that required macOS.

When Apple brought out the M1 chips they made virtualisation easier and they even had sample code for creating virtual machines. Although this was cool I didn’t get much further than having a play around as it felt like a lot more work would be required for me to use the technology.

I knew that people had gotten virtualisation to work to the point where they were offering cloud based CI services and after a bit of research I stumbled upon tart, which I bookmarked and then didn’t look at for months. It wasn’t until I saw some Tweets/Toots/Skeets (whatever the bloody platform was at the time) from Simon B. Støvring talking about his CI work that I started investigating properly.

The first virtual machine

Despite my hesitation on getting started it actually was quite simple to create a virtual machine and begin to mess around (I kicked myself for putting off trying sooner).

After installing tart from homebrew

brew install cirruslabs/cli/tart

It’s then a case of providing the url for an IPSW to a tart create invocation, that will then download the IPSW and create a virtual machine.

tart create example \
  --disk-size=120 \
  --from-ipsw=https://updates.cdn-apple.com/2024FallFCS/fullrestores/072-30094/44BD016F-6EE3-4EE5-8890-6F9AA008C537/UniversalMac_15.1.1_24B91_Restore.ipsw

The IPSW link above will no doubt go out of date pretty quickly but you can find the download urls at https://ipsw.me. I know the website looks a bit scammy and has adverts everywhere but it’s providing links to Apple domains so ¯_(ツ)_/¯.

After running the above (which will take a small eternity as the IPSW is a big download, don’t worry the download is cached for future invocations) you end up with… your terminal prompt back. To actually run the machine you just created you need to run

tart run example

example is the name I passed to the create command - if you create a machine with a different name make sure to run that instead.

This will boot the machine and allow you to start configuring the OS the same as if you fired up new macOS hardware for the first time. This was pretty exciting and brought back memories of the Dave Verwer course where I first learnt iOS and specifically the cool feeling from the first time deploying code I’d written onto a physical device.

At this point I was still a long way from the end goal but I’d made a start so the momentum kept me moving. The first thing I wanted to prove was could I actually get projects to build inside the virtual machine and was the performance alright. I think I discovered that I could get our project to build but for some reason the tests just would not run. When I hit this road block I moved things to the back burner again.

Asking for help

Weeks had passed and I was still keen to make this work and from following Simon B. Støvring on Mastodon I’d seen more posts he had made and discovered all the documentation that had been written for some tooling he’d made called Tartelet (see tartelet docs). Sure that I could make this work I tried again but kept getting the same result, I eventually just reached out on slack and got helpful responses from Simon and someone with the handle biscuit, which suggested two things to try

• prewarming simulators using something like https://github.com/biscuitehh/yeetd/blob/main/Resources/prewarm_simulators.sh
• bumping up the available memory on the virtual machines

I can’t specifically remember which one of the above it was but I now had proved to myself that I could indeed run our project inside virtual machines including the tests and the performance wasn’t noticeably worse than our existing set up.

Repeatable Configuration

The next thing I wanted to achieve was creating machines repeatably - for this I turned to the Tartelet docs mentioned before. The Tartelet docs talk you through sensible configuration to use for a CI runner but the configuration is done using the GUI. The problem I had is that I’m the type of person who locks my front door, walks to my car then turns around to check the front door is locked. This means that I just didn’t feel comfortable having to manually configure machines incase I mess a step up or mistype something.

After a bit of research I found the company that created tart (remember they provide a CI service) also has a repo that contains their configuration for building machines here. The first take away is that they are using Packer to provision machines. In this case Packer is using a tart plugin, which is using the tart tool under the hood. So after installing packer with mise and then the tart plugin I was set to explore building machines from scratch using just code.

mise install packer
packer plugins install github.com/cirruslabs/tart

Using the cirrus labs templates as a starting point I ended up with something like this to build a Sequoia machine

packer {
  required_plugins {
    tart = {
      version = ">= 1.12.0"
      source = "github.com/cirruslabs/tart"
    }
  }
}

source "tart-cli" "tart" {
  from_ipsw = "https://updates.cdn-apple.com/2024FallFCS/fullrestores/072-30094/44BD016F-6EE3-4EE5-8890-6F9AA008C537/UniversalMac_15.1.1_24B91_Restore.ipsw"
  vm_name = "base-sequoia"
  cpu_count = 8
  memory_gb = 8
  disk_size_gb = 100
  headless = false
  ssh_password = "runner"
  ssh_username = "runner"
  ssh_timeout = "120s"
  boot_command = [
    // hello, hola, bonjour, etc.
    // > Tap get started
    "<wait60s><spacebar>",

    // Language
    // > Typing english gets us to English UK
    "<wait10s>english<enter>",

    // Select Your Country or Region
    "<wait20s>united kingdom<leftShiftOn><tab><leftShiftOff><spacebar>",

    // Written and Spoken Languages
    // > Tap continue
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",

    // Accessibility
    // > Tap Not now
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",

    // Data & Privacy
    // > Tap continue
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",

    // Migration Assistant
    // > Tap Not now
    "<wait10s><tab><tab><tab><spacebar>",

    // Sign In with Your Apple ID
    // > Tap Set up later
    "<wait10s><leftShiftOn><tab><leftShiftOff><leftShiftOn><tab><leftShiftOff><spacebar>",

    // Are you sure you want to skip signing in with an Apple ID?
    // > Tap Skip
    "<wait10s><tab><spacebar>",

    // Terms and Conditions
    // > Tap Agree
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",

    // I have read and agree to the macOS Software License Agreement
    // > Tap Agree
    "<wait10s><tab><spacebar>",

    // Create a Computer Account
    // > Set username, account name, password all to runner
    "<wait10s>runner<tab><tab>runner<tab>runner<tab><tab><tab><spacebar>",

    // Enable Location Services
    // > Deselect and tap continue
    "<wait30s><leftShiftOn><tab><leftShiftOff><spacebar>",

    // Are you sure you don't want to use Location Services?
    // > Tap continue
    "<wait10s><tab><spacebar>",

    // Select Your Time Zone
    // > Type UTC and tap continue
    "<wait10s><tab><tab>UTC<enter><leftShiftOn><tab><tab><leftShiftOff><spacebar>",

    // Analytics
    // > Tap continue
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",

    // Screen Time
    // > Tap Set up later
    "<wait10s><tab><spacebar>",

    // Siri
    // > Deselect and tap continue
    "<wait10s><tab><spacebar><leftShiftOn><tab><leftShiftOff><spacebar>",

    // Choose your look
    // > Select light mode
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",

    // Welcome to Mac
    "<spacebar>",

    // Open terminal
    "<wait10s><leftAltOn>n<leftAltOff><wait3s><leftAltOn><leftShiftOn>g<leftShiftOff><leftAltOff>/Applications/Utilities/Terminal.app<enter><wait3s><leftAltOn>o<leftAltOff><wait3s>defaults write NSGlobalDomain AppleKeyboardUIMode -int 3<enter><wait5s><leftAltOn>q<leftAltOff>",

    // Open system settings
    "<wait10s><leftAltOn>n<leftAltOff><wait3s><leftAltOn><leftShiftOn>g<leftShiftOff><leftAltOff>/Applications/System Settings.app<enter><wait3s><leftAltOn>o<leftAltOff>",

    // Search for 'sharing'
    "<wait10s><leftAltOn>f<leftAltOff>sharing<enter>",

    // Tab to 'Screen Sharing' and enable it
    "<wait10s><tab><tab><tab><tab><tab><spacebar>",

    // Navigate to 'Remote Login' and enable it
    "<wait10s><tab><tab><tab><tab><tab><tab><tab><tab><tab><tab><tab><tab><spacebar>",

    // Close settings
    "<wait5s><leftAltOn>q<leftAltOff>"
  ]

  // A workaround for Virtualization.Framework's installation process not fully finishing in a timely manner
  create_grace_time = "30s"
  run_extra_args = []
}

build {
  sources = ["source.tart-cli.tart"]

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mEnable passwordless sudo\\033[0m'",
      "echo '└── echo runner | sudo -S sh -c \"mkdir -p /etc/sudoers.d/; echo \\047runner ALL=(ALL) NOPASSWD: ALL\\047 | EDITOR=tee visudo /etc/sudoers.d/runner-nopasswd\"'",
      "echo runner | sudo -S sh -c \"mkdir -p /etc/sudoers.d/; echo 'runner ALL=(ALL) NOPASSWD: ALL' | EDITOR=tee visudo /etc/sudoers.d/runner-nopasswd\""
    ]
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mEnable autologin\\033[0m'",
      "echo '├── printf \\047\\x0f\\xfc\\x3c\\x4d\\xb7\\xce\\xdd\\x8d\\x65\\xd0\\x6c\\x2c\\047 > /tmp/kcpassword'",
      "printf '\\x0f\\xfc\\x3c\\x4d\\xb7\\xce\\xdd\\x8d\\x65\\xd0\\x6c\\x2c' > /tmp/kcpassword",
      "echo '├── sudo mv /tmp/kcpassword /etc/kcpassword'",
      "sudo mv /tmp/kcpassword /etc/kcpassword",
      "echo '├── sudo chmod 600 /etc/kcpassword'",
      "sudo chmod 600 /etc/kcpassword",
      "echo '├── sudo chown root:wheel /etc/kcpassword'",
      "sudo chown root:wheel /etc/kcpassword",
      "echo '└── sudo defaults write /Library/Preferences/com.apple.loginwindow autoLoginUser runner'",
      "sudo defaults write /Library/Preferences/com.apple.loginwindow autoLoginUser runner"
    ]
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mDisable screensaver\\033[0m'",
      "echo '├── sudo defaults write /Library/Preferences/com.apple.screensaver loginWindowIdleTime 0'",
      "sudo defaults write /Library/Preferences/com.apple.screensaver loginWindowIdleTime 0",
      "echo '└── defaults -currentHost write com.apple.screensaver idleTime 0'",
      "defaults -currentHost write com.apple.screensaver idleTime 0"
    ]
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mDisable sleeping\\033[0m'",
      "echo '├── sudo systemsetup -setdisplaysleep Off 2> /dev/null'",
      "sudo systemsetup -setdisplaysleep Off 2> /dev/null",
      "echo '├── sudo systemsetup -setsleep Off 2> /dev/null'",
      "sudo systemsetup -setsleep Off 2> /dev/null",
      "echo '└── sudo systemsetup -setcomputersleep Off 2> /dev/null'",
      "sudo systemsetup -setcomputersleep Off 2> /dev/null"
    ]
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mDisable screen lock\\033[0m'",
      "echo '└── sysadminctl -screenLock off -password runner'",
      "sysadminctl -screenLock off -password runner"
    ]
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mDisable spotlight indexing\\033[0m'",
      "echo '└── sudo mdutil -a -i off'",
      "sudo mdutil -a -i off"
    ]
  }
}

If this is stored in a file called sequoia.pkr.hcl and you run packer build sequoia.pkr.hcl and then wait you’ll end up with a new machine in tart called base-sequoia that has various basic things configured like passwordless login, turning off screensaver etc. Arriving at the above configuration took a short eternity with a lot of trial and error tweaking things and rerunning.

Some notes on the hcl file above:

hcl is a configuration language that has all kinds of features, which I did start using but then in the spirit of not wanting future maintainers of this tooling having to learn yet another thing I opted to entirely wrap it. The above is generated by calling a thin kotlin dsl I wrote, which has a few advantages in my mind

• Other team mates can more easily help maintain things by calling the Kotlin dsl and not having to worry about learning hcl
• With a dsl I can make things like <leftShiftOn><tab><leftShiftOff> safer by managing the on/off state
• IDEs are going to be miles better at supporting Kotlin as opposed to a custom markup language
• I’m in control of the dsl output, which allowed me to do pretty printing of the commands

That last bullet deserves some expansion. By Default invoking shell commands in packer won’t actually tell you what is being invoked. In the output all you see is lines like Provisioning with shell script: /var/folders/wl/92q0hw051vnbwncp7lxsp3m80000gq/T/packer-shell2461613400. When you are debugging configuration or even just want to see the progress it’s super helpful to be able to see where you are up to especially in case of failure. With this in mind I made it so in the dsl you’d call a function like this

script("Disable sleeping") {
    """
    sudo systemsetup -setdisplaysleep Off 2> /dev/null
    sudo systemsetup -setsleep Off 2> /dev/null
    sudo systemsetup -setcomputersleep Off 2> /dev/null
    """.trimIndent()
}

This would generate this hcl configuration

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mDisable sleeping\\033[0m'",
      "echo '├── sudo systemsetup -setdisplaysleep Off 2> /dev/null'",
      "sudo systemsetup -setdisplaysleep Off 2> /dev/null",
      "echo '├── sudo systemsetup -setsleep Off 2> /dev/null'",
      "sudo systemsetup -setsleep Off 2> /dev/null",
      "echo '└── sudo systemsetup -setcomputersleep Off 2> /dev/null'",
      "sudo systemsetup -setcomputersleep Off 2> /dev/null"
    ]
  }

At run time you’d actually see this

==> tart-cli.tart: Provisioning with shell script: /var/folders/wl/92q0hw051vnbwncp7lxsp3m80000gq/T/packer-shell2461613400
    tart-cli.tart: 🟢 Disable sleeping
    tart-cli.tart: ├── sudo systemsetup -setdisplaysleep Off 2> /dev/null
    tart-cli.tart: warning: this combination of display sleep and system sleep may prevent system sleep.
    tart-cli.tart: setdisplaysleep: Never
    tart-cli.tart: ├── sudo systemsetup -setsleep Off 2> /dev/null
    tart-cli.tart: setsleep: Never (computer, display, hard disk)
    tart-cli.tart: └── sudo systemsetup -setcomputersleep Off 2> /dev/null
    tart-cli.tart: setcomputersleep: Never

It might not be pretty but I was borrowing the lines (├─, └─) from the tree command to give a bit of structure to show that the high level step is 🟢 Disable sleeping and after that you have the commands that make this process up starting with ├─, └─ and then the std{out,err} of those commands below that.

Marvel at what the above achieved

At this point I kept running tart list and tart run base-sequoia just to marvel at the fact I had indeed got a machine working and can yield the power of starting and stopping it on command.

More boring configuration

Having a machine with no software installed is pretty boring so the next step was spending ages figuring out the incantations to get things like Xcode and Ruby (for fastlane) installed. I was doing this like a cave man by starting the machine, opening the terminal and manually typing commands (as copy/paste doesn’t work across machines) until a colleague gave a disapproving look and said “why don’t you just ssh into the machine?”. Things went faster after that.

The basic config for getting Xcode installed looked something like

packer {
  required_plugins {
    tart = {
      version = ">= 1.12.0"
      source = "github.com/cirruslabs/tart"
    }
  }
}

source "tart-cli" "tart" {
  vm_base_name = "base-sequoia"
  vm_name = "xcode"
  cpu_count = 8
  memory_gb = 8
  disk_size_gb = 100
  headless = true
  ssh_password = "runner"
  ssh_username = "runner"
  ssh_timeout = "120s"
  run_extra_args = []
}

build {
  sources = ["source.tart-cli.tart"]

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mCreate directory for artifacts we want to install\\033[0m'",
      "echo '└── mkdir -p /Users/runner/Downloads/packer'",
      "mkdir -p /Users/runner/Downloads/packer"
    ]
  }

  provisioner "file" {
    sources = [
      pathexpand("~/XcodesCache/Command_Line_Tools_for_Xcode_16.1.dmg"),
      pathexpand("~/XcodesCache/Xcode_16.1.xip"),
      pathexpand("~/XcodesCache/iOS_18.1_Simulator_Runtime.dmg")
    ]
    destination = "/Users/runner/Downloads/packer/"
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mCreate ~/.zprofile\\033[0m'",
      "echo '└── echo \"export LANG=en_US.UTF-8\" >> ~/.zprofile'",
      "echo \"export LANG=en_US.UTF-8\" >> ~/.zprofile"
    ]
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mInstall command line tools\\033[0m'",
      "echo '├── hdiutil attach \"/Users/runner/Downloads/packer/Command_Line_Tools_for_Xcode_16.1.dmg\"'",
      "hdiutil attach \"/Users/runner/Downloads/packer/Command_Line_Tools_for_Xcode_16.1.dmg\"",
      "echo '├── sudo installer -pkg \"/Volumes/Command Line Developer Tools/Command Line Tools.pkg\" -target \"/Volumes/Macintosh HD\"'",
      "sudo installer -pkg \"/Volumes/Command Line Developer Tools/Command Line Tools.pkg\" -target \"/Volumes/Macintosh HD\"",
      "echo '└── hdiutil detach \"/Volumes/Command Line Developer Tools\"'",
      "hdiutil detach \"/Volumes/Command Line Developer Tools\""
    ]
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mInstall homebrew\\033[0m'",
      "echo '├── /bin/bash -c \"$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\"'",
      "/bin/bash -c \"$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\"",
      "echo '├── eval \"$(/opt/homebrew/bin/brew shellenv)\"'",
      "eval \"$(/opt/homebrew/bin/brew shellenv)\"",
      "echo '├── echo \\047eval \"$(/opt/homebrew/bin/brew shellenv)\"\\047 >> ~/.zprofile'",
      "echo 'eval \"$(/opt/homebrew/bin/brew shellenv)\"' >> ~/.zprofile",
      "echo '├── echo \\047export HOMEBREW_NO_AUTO_UPDATE=1\\047 >> ~/.zprofile'",
      "echo 'export HOMEBREW_NO_AUTO_UPDATE=1' >> ~/.zprofile",
      "echo '└── echo \\047export HOMEBREW_NO_INSTALL_CLEANUP=1\\047 >> ~/.zprofile'",
      "echo 'export HOMEBREW_NO_INSTALL_CLEANUP=1' >> ~/.zprofile"
    ]
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mInstall mise\\033[0m'",
      "echo '├── curl https://mise.run | sh'",
      "curl https://mise.run | sh",
      "echo '├── export PATH=\"$HOME/.local/bin:$PATH\"'",
      "export PATH=\"$HOME/.local/bin:$PATH\"",
      "echo '├── echo \\047eval \"$(~/.local/bin/mise activate zsh)\"\\047 >> ~/.zprofile'",
      "echo 'eval \"$(~/.local/bin/mise activate zsh)\"' >> ~/.zprofile",
      "echo '├── mkdir -p ~/.config || true'",
      "mkdir -p ~/.config || true",
      "echo '├── echo \\047[alias]\\047 >> ~/.config/mise.toml'",
      "echo '[alias]' >> ~/.config/mise.toml",
      "echo '└── echo \"xcodes = \\047asdf:younke/asdf-xcodes\\047\" >> ~/.config/mise.toml'",
      "echo \"xcodes = 'asdf:younke/asdf-xcodes'\" >> ~/.config/mise.toml"
    ]
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mInstall Xcode\\033[0m'",
      "echo '├── \"$HOME/.local/bin/mise\" trust --yes'",
      "\"$HOME/.local/bin/mise\" trust --yes",
      "echo '├── \"$HOME/.local/bin/mise\" use xcodes@\"1.6.0\" --yes'",
      "\"$HOME/.local/bin/mise\" use xcodes@\"1.6.0\" --yes",
      "echo '├── eval \"$(\"$HOME/.local/bin/mise\" activate --shims)\"'",
      "eval \"$(\"$HOME/.local/bin/mise\" activate --shims)\"",
      "echo '├── xcodes install \"16.1\" --path \"/Users/runner/Downloads/packer/Xcode_16.1.xip\" --experimental-unxip --empty-trash'",
      "xcodes install \"16.1\" --path \"/Users/runner/Downloads/packer/Xcode_16.1.xip\" --experimental-unxip --empty-trash",
      "echo '├── sudo xcodes select \"16.1\"'",
      "sudo xcodes select \"16.1\"",
      "echo '├── xcodebuild -runFirstLaunch'",
      "xcodebuild -runFirstLaunch",
      "echo '└── xcrun simctl runtime add \"/Users/runner/Downloads/packer/iOS_18.1_Simulator_Runtime.dmg\"'",
      "xcrun simctl runtime add \"/Users/runner/Downloads/packer/iOS_18.1_Simulator_Runtime.dmg\""
    ]
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mInstall Ruby\\033[0m'",
      "echo '├── eval \"$(/opt/homebrew/bin/brew shellenv)\"'",
      "eval \"$(/opt/homebrew/bin/brew shellenv)\"",
      "echo '├── brew install readline libyaml gmp'",
      "brew install readline libyaml gmp",
      "echo '└── \"$HOME/.local/bin/mise\" use ruby@\"3.3.0\"'",
      "\"$HOME/.local/bin/mise\" use ruby@\"3.3.0\""
    ]
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mDelete install artifacts\\033[0m'",
      "echo '└── rm -rf /Users/runner/Downloads/packer'",
      "rm -rf /Users/runner/Downloads/packer"
    ]
  }
}

There’s a few things to note with the above:

• Duplicated hardcoded version numbers - in reality these are interpolated in and not manually maintained.
• Assets related to installing Xcode are copied from the host machine to avoid needing to auth with Apple to download things.
• Homebrew is installed after Xcode command line tools - again to avoid being asked to auth with Apple to download the installer.

Marvel some more

Again it was time to reflect on how far we’d come and how it wasn’t easy. Figuring out the exact shell commands to install all the things you need was a combination of Google, ChatGPT, talking to colleagues and just trying loads of things.

Between the two packer files above I learnt a load of new things

• Various admin commands to disable screensavers and screen locks
• What on earth kcpassword was
• Mounting/unmounting volumes
• More in depth knowledge of installing/working with mise
• Ruby is still an absolute pain to install right

To even get this far though there’s a lot of foundational knowledge that I used

• Being comfortable with the command line
• Having the ability to navigate directories/files from the command line
• Understanding environment variables in shell environments
• Understanding file permissions
• Knowing which commands look safe to run that were found on random internet searches
• Knowing when to use sudo and not just using it blindly causing issues later on

What about CI?

Now we had machines running the next step was actually hooking up to the CI machinery to run some workloads. There were many misfires at this stage with trying different approaches. The most promising was to add more provisioning steps to install the CI runner inside the virtual machine and then orchestrate having 2 virtual machines booted that self registered with the CI server to run workloads. Once the virtual machine had run a workload it would then need to destroy itself and another machine be spun up in its place.

Although the above worked and seemed reasonable it was tricky and means that you always have to have virtual machines fired up even when not in use. It also felt a bit wrong having the virtual machines needing to know about CI when that information could be hidden from them - for instance I often just spin up the virtual machines to try things out in a clean environment but I don’t want to worry about it adding my personal machine to the CI work pool. Discussing this with a colleague we came to the conclusion that actually we could have a couple of CI runners on the bare metal and when they receive a work load they would clone a virtual machine and boot it, then copy the source files into the virtual machine before running the job. If you squint hard enough this feels a bit like docker where you’d have various layers being created to configure the environment then you’d copy your source code in and operate on that.

We messed around for a while trying to do this with combinations of tart commands directly and then one of us (can’t remember who) had the spark that we should just use Packer again for this as it has already massively simplified the task of copying files to/from the virtual machine as well as running shell tasks.

With this in mind we need a new packer file:

packer {
  required_plugins {
    tart = {
      version = ">= 1.12.0"
      source = "github.com/cirruslabs/tart"
    }
  }
}

source "tart-cli" "tart" {
  vm_base_name = "xcode"
  vm_name = "runner"
  cpu_count = 8
  memory_gb = 8
  disk_size_gb = 100
  headless = true
  ssh_password = "runner"
  ssh_username = "runner"
  ssh_timeout = "120s"
}

build {
  sources = ["source.tart-cli.tart"]

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mCreate workspace directory\\033[0m'",
      "echo '└── mkdir -p $HOME/workspace'",
      "mkdir -p $HOME/workspace"
    ]
  }

  provisioner "file" {
    source = "./"
    destination = "$HOME/workspace"
    direction = "upload"
  }

  provisioner "shell" {
    inline = [
      "echo '🟢 \\033[1mRun the entry point\\033[0m'",
      "echo '├── cd $HOME/workspace'",
      "cd $HOME/workspace",
      "echo '└── echo Hello, World!'",
      "echo 'Hello, World!'"
    ]
  }
}

This packer file will upload the current directory into the vm, then run echo 'Hello, World!'. Although this is pretty pointless it does demonstrate that everything works and we have the following:

• A clean machine is run by cloning the xcode machine
• We can copy our source code into the virtual machine
• We can execute arbitrary code as an entry point

Most CI solutions will allow you to create a template or reuse configuration. We use GoCD at work, which is “interesting” but we was able to find a way to make it so we could have a template that will invoke something like the packer file above but allow each pipeline to provide the entry point to invoke inside the virtual machine.

Entry points and configuration

When we started I think I was tunnel visioned and made it so that in the CI template if you provided an entry point of publish it would follow a convention of looking for an executable file in your repo called .gocd/publish. Although this worked it wasn’t super discoverable and if you wanted to share configuration between pipelines you were pretty much forced to start sourcing bash files from bash files 🤮.

The final straw for this set up was when we needed to pass environment variables to the workloads. My colleague took the fun task of changing this set up to instead read a configuration file written in json that describes the entry point, environment variables and any other config you might want to pass to a pipeline.

Getting data out of the virtual machine

When CI fails it’s helpful to know why and sometimes those details don’t appear in the logs but in various artifacts scattered throughout build folders. Packer gives a mechanism for this by allowing you to register an action on failure, in these cases we opt to copy files from a known directory to the host so they can be uploaded to the CI server before the virtual machine is deleted. Having the CI always upload files it finds in a directory is handy because it means you can put reports or artifacts in this special folder on the VM and know it will be uploaded to the CI regardless of whether there was a failure or not.

Android

Although we had plans to have Android builds run in docker to save the macOS runners we still needed to support Android because we have some Kotlin Multiplatform projects that we just want to build in one place. I’m not going to pretend to know what wizardry went on here but a big problem with the Virtualization framework is that is doesn’t support nested emulation, which means no Android emulators. To get around this my colleague created an ssh tunnel between the host and the VM and runs the emulators on the bare metal but connects Adb in the virtual machine via this tunnel. This works well.

Taking stock of things

The above represents the foundational thinking blocks that underpin the solution we built. It took quite a while to build out as it was mostly stewing as an idea whilst the pieces starting drifting together slowly from different experiments. Not all effort is equal - there we some tasks where it felt like I’d made a massive leap towards the end goal and then others where the work was important but it felt like a hard slog.

Where we ended up after some more iterations was we have a Kotlin Multiplatform command line tool that controls all of the above. There’s a dsl that hides away the hcl files and instead you have a nice Kotlin interface. The tooling has the ability to build machines in “layers” where each subsequent machine depends on the machine above existing. This means that you only need to rebuild machines (which can be slow) when required. We also have a json file that each repo uses to configure various things like caching, environment variables and the entry points themselves.

Other use cases

At some point we used Github’s virtual runners and when something went wrong it was an absolute nightmare to debug without access to the machine after the run. With this set up I’m never more than a few commands away from being in the same environment as the one that produced the failure so I can explore and try things out to get to the root issue.

Being able to build virtual machines easily in a repeatable way is empowering as it means if I want to jump on the beta release trains for macOS or Xcode versions I can do it without risking my primary machine.

I’ve also found it useful having the ability to have a clean machine with no software installed to help quickly iterate install scripts to provide to colleagues. It’s amazing how many assumptions you can make about other people’s system having similar set ups to your own which all fly out the window when you use a clean machine.

Conclusion

Working on this kicked my ass and I’d picked it up and put it down a few times but as it was a nagging idea I really wanted to see it through. Fortunately once it had some legs and I was pretty confident we could get it done I spent dedicated time on this with my colleague Jack. The end solution builds on the above adding several key things that are needed to productionise it. We’ve been using it for various macOS and Kotlin Multiplatform projects for ~6 months and all the iOS pipelines for ~2 months and despite some teething issues we’ve not really seen many issues.

In the above I’ve mentioned colleagues a few times and it’s worth stating that this project was probably only doable because of being fortunate to work with people who can solve problems, brainstorm and muck in across a wide range of technologies and tolerate listening to me talk nonsense and rewrite my stuff multiple times as I realised that the me who wrote code yesterday was an idiot.