Dev Conversations #2: Dave Verwer & Sven A. Schmidt
Dev Conversations is a monthly series where Swift Toolkit features members of Swift work groups, content creators, and people who contribute to the Swift community, focused on tooling, developer experience and Server Side Swift.
In these talks, we discuss open source projects, tips for professional growth, and many other valuable insights and perspectives from other developers.
Dave Verwer is well known in the iOS developer community, mostly for his iOSDevWeekly.com newsletter, which he’s been publishing for more than a decade. In the recent years, Dave and Sven A. Schmidt have been working on the Swift Package Index - a website that does much more than just listing Swift packages. It shows, for each listed package, a matrix of supported platforms, Swift versions, and also allows launching a playground with a package to try it out!
After listening to some of their podcast episodes, and peeking at the organization repositories on GitHub, I was sure it could yield a great talk about the different pieces they have built when creating the website. The Vapor server, the tools to digest packages, documentation hosting for package, the infrastructure that supports it all…
Sven and Dave both kindly accepted my invite to join another Dev Conversations episode. We chatted about its history, all the pieces behind the scenes, the state of Swift 6’s strict concurrency. Wrapping up, we also discussed how demanding this project became, and how essential is the community support, from GitHub Sponsors to the companies that contribute to it (MacStadium, Microsoft, and since 2023, Apple).
If you prefer reading or listening instead, you can find the written interview and the podcast links after the video, along with relevant links discussed in the chat.
Also available in your favorite podcast player 
Swift Toolkit
Hello there! Hello Sven. Hello Dave. Thanks for joining another Dev Conversations, where we host developers from the Swift community who contributes to the domains around tooling, developer experience and server-side Swift.
Dave
Thanks for having us, great to be here.
Sven
Hello! Thanks for having us!
Swift Toolkit
All right, so for maybe those who don't know yet the Swift Package Index, if you could describe it shortly.
Dave
How could anyone not know Swift Package Index? No, I'm kidding. Swift Package Index is a site that will help you find the best dependencies for your application. So any dependency that is compatible with the Swift Package Manager is welcome on Swift Package Index. And we index and discover a whole load of metadata about every package and display that to people who are browsing the site.
And we also test compatibility for different versions of Swift and different platforms. So we have a build system, which I'm sure we'll talk about later, that is constantly doing that compatibility testing. And then finally, the big thing that we also do is host documentation for Swift packages. So any open source Swift package that has DocC documentation within it, we will build and host versioned copies of that documentation for free for any.
Swift Toolkit
Right. And when you started it, from a product, both as a product and from a technical perspective, where do you start from? Maybe, Sven you want to talk about it?
Sven
So there was actually a predecessor project, the SwiftPM library by Dave that sort of kickstarted it all. And I was aware of the project that Dave had launched, I think in 2019, wasn't it, Dave? Yeah. And in early 2020, I had built something that allowed you to launch a playground with a package as a dependency so you can try a package in a playground.
Arena being featured in iOSDevWeeklyAnd I had the idea to connect that to Dave's project, SwiftPM Library, and we started talking. In March, I sent an email to Dave if he'd be interested, and then we had a call. Almost the next day or a couple days later, and it just so happened that Dave was discussing relaunching the project or rewriting the project in Vapor because the original project was written in Ruby on Rails.
I had experience with Vapor, I had time at the time and we sort of started talking: let's maybe join forces on this. I offered to to try my hand at porting SwiftPM library and see where we end up and that's where we ended up in a nutshell.
Swift Toolkit
Now, already connecting to the next question: I took a look at the organization on GitHub that you have and like there are many repositories, many different pieces. There's repo for the list of repositories itself. There's the server and the other executables. There's so much to talk about. And I think that that's where we could start.
So my original question was if you had any, if you could consider other frameworks, but that's, that's already answered, right? It was in Ruby on Rails and like, I think Vapor was a choice that you had to go with it, right?
Dave
Well, part of the thinking behind it was if it was truly to become a kind of a community hub for packages and it was going to benefit from being open source and if it was going to be open source, it had to really be written in Swift if it was for Swift developers.
So Vapor was the obvious choice when we actually came to, because I think the Swift PM library was more of a prototype. I mean, it worked, there was a fully functional package search engine, but it was really more of a prototype. It was kind of early days. And so when we decided to do it properly, Vapor was an obvious choice for the framework to do it in.
Swift Toolkit
I assume it was before before async-await, right? When it used SwiftNIO and had flatMap everywhere...
Dave
It was! It was actually right at the end of Vapor 3. So Vapor 3 was the stable version at the time and Vapor 4 was in kind of a late beta at the time. And I'd actually made a couple of commits to a Vapor 3 version.
And then when Sven and I chatted, Sven said, now I think we should do it in Vapor 4. So we kind of started with the beta of Vapor 4. And I believe that Vapor 5 was just announced. I think last week I read a blog post about it, we've had the entire lifespan of Vapor 4 in the Swift Package Index!
Vapor 5 announcement postSwift Toolkit
Awesome. And in what ways you think that it helped and in what ways it brought limitations and difficulties?
Sven
So I'm a huge fan of Swift on the server. I've had a couple of projects previously of similar scale that were written in Python, in Python Flask and Python Django, the two popular frameworks. And it really, really helps writing a server-side Swift project when you have strong typing.
The deployment story is so much smoother. I remember our services often struggling with memory in Python and the runtime exceptions that would kill you, because you would just lack the type safety, and lots of things happening at runtime that you have to tackle, while you tackle them at compile time in Swift.
I do see there's some complexity around the type system that's undeniable, and I think in particular the EventLoop Future type in Vapor, Vapor 4 before async-await is something that is a bit of a beast. It is a bit of a learning curve to start working with that...
Swift Toolkit
Right. All the flatMap, the operations around it.
Sven
There's no way to work with that unless you understand map, flatMap, and what that does and how you nest these things. And concurrency in that context is tricky when you have an array of EventLoop Futures. How do you transform those and how to ensure that they run in parallel or they run sequentially? And when do they do one or the other?
And more importantly, what's really tricky or was at the time was error handling in that context. Because how do errors that happen in the EventLoop Future context, how do they get propagated? So that was that was very tricky...
Swift Toolkit
But that's a thing of the past now!
Sven
That is something that's entire. Yeah, just a couple of weeks ago, we removed the last EventLoop Future from the Swift Package Index code base. Now it's all async-await and it has nothing but helped the project in terms of code readability, and it's so much nicer to work with. I mean, you do get the hang of it after while with EventLoop Futures, but there's just this nesting, and the flat mapping you can't get around, right?
It envelopes, you know, goes through your whole code base. Once you have these things, it's like async-await, right? Once you have something async-await, it bubbles up through your whole, yeah. But async await is much nicer to bubble up than EventLoop Futures. So we're in a better place now.
Dave
I'm not sure I ever got the hang of EventLoop Futures. It was something I really genuinely struggled with actually. Right from the very beginning I found those very challenging and if I still had to use them today I think I would still find them quite challenging.
Swift Toolkit
I think it reminds me to some extent working with Rx, RxSwift: everything is a stream.
But from a deployment perspective, like nowadays, I think we are in another stage where we have all the Linux distributions and so on. I remember deploying Vapor, having to manually install Swift in a Docker image. I assume you were also there.
Was deploying ever a problem for you?
Sven
No, the deployment story has actually always been easier than Python projects with Flask or Django, because there you also need to install everything, and you need to make sure that you have the C dependencies... Because you know in Python when you use SQLAlchemy and stuff, you always need to compile some little C project, and you need a C compiler then to build.
The huge benefits we gain just in terms of memory, I think it's like 50 MB or something around there. It's just ridiculously small. You couldn't spawn a Python process, let alone do anything with it in the memory footprint that we have in our current deployment.
And just how rock solid it is! We screwed up a couple of times, but that wasn't the Swift language's problem. That's just, you know, stuff that sometimes slips through, but just the sheer number of things that can't even slip through because you can't get it to compile in the first place! Just the refactoring we can do with confidence, they are really hard in Python. You need to have a really, really solid test suite in order to refactor with confidence in Python.
Swift Toolkit
The compiler is a safety net, right? That's exactly what I love about Swift on the server is, that you bring all that confidence that the compiler gives you.
Dave
And of course we have an extensive test suite as well. That's the other half of the confidence. And I don't think you can neglect that side of it either, certainly.
Swift Toolkit
That takes us also to the next question!
Dave
I'm here to help 😁
Swift Toolkit
I was looking into the repo of the server, and I saw there the dependency snapshot testing.
I said "wow, it actually makes sense!", because it's an app that you can test. But how are the ins and outs of it? How do you spin? Do you spin a browser instance, how it works, because I'm so used to testing on a simulator, how it goes for the web.
Dave
Yeah. So that's been through a few iterations actually over the four years that we've been running. In fact, it's more than four years. So we started with snapshot tests that actually used the image-based snapshotting, we use Point-Free's Swift snapshot testing library for our snapshot testing. And just to be clear, we do a couple of different types of snapshot tests. So we do snapshot tests that would be more like a unit test.
So if we just need to grab some text and make sure that text doesn't change between test runs, then for example, we snapshot test our SQL queries. So we output the SQL query and then have a snapshot test to make sure that's not different, without us knowing about it. And also little bits of JSON and little fragments of text that we put all over the place. We're gonna snapshot those and store those on disk. But then we also have full webpage snapshots, which are snapshots taken of using a browser to render the HTML and CSS and images into a browser, and then an image snapshot of those.
And we did use the snapshot testing library for that originally, but we did actually find some little bits of incompatibility there. During the transition from Intel to Apple Silicon, we found that sometimes there was like tiniest little pixel differences with the rendering of the browser that we weren't really in control of.
Swift Toolkit
But the tooling around it, is it pure Swift? Or does it have any other frameworks and tools around it?
Dave
These days we actually use a third party tool. So we now take our snapshot testing for the HTML: we output the HTML of a full page and store that in the repository. And then we run a tool called Percy, from BrowserStack. Percy is an online snapshot testing tool where it takes HTML, and renders that HTML using a variety of browsers. So it tests it with Chrome, Safari, Firefox, and gives you a web UI that says this pixel changed, this pixel changed, or this entire snapshot changed.
We find that very useful. Well, it's convenient and it's also a bigger test than a unit test. That's a full page test. Like it's a kind of integration of "is everything still producing the same webpage output?"
Sven
I just wanted to emphasize that it's nice to test both things, like the pure HTML and the visual representation of the HTML, because they can change independently, right? There could be invisible changes in the HTML that the webpage snapshot can't see, and stuff might move around due to browser changes or something that isn't actually visible in the HTML and have differences there.
Dave
We should give Percy a shout out for their generous support of open source because they provide a free Percy account for open source projects. It's great to see companies do that.
Swift Toolkit
Sure! We'll leave a link at the end of the page.
And besides the server, there are many other tools and packages that you built, I saw there. For example, I saw that there's a package to read the SPI.yml file and so many other tools.
SPI repos: the server and other toolsDave
So that's a file that package authors can put in their package repositories, that when we gather the metadata and when we query each package repository, we pick up that file and it's a way for package authors to give us preferences.
In terms of who wrote the package, maybe if they want to override the author information or if they want to provide some external documentation, or if they want to configure the build system in some way, we give them the ability to do that.
Swift Toolkit
But how do the pieces connect? I saw that to add the package, I should go on GitHub and create an issue. And then that issue that probably becomes a PR somewhere, I assume, or trigger some sort of flow. That's where that package goes in and fetches data?
For example, another question I wondered is, do you receive webhooks for each release once there's a new one, or once there's a tag, for example.
Dave
So the whole thing is kind of disconnected actually. We have three processes internally that do all of the updating of the package metadata. We have one called reconciliation, which you're right about the package adding process, all that does is it adds a Git repository URL into a JSON file in a separate repository under our organization on GitHub.
It constantly checks, every couple of minutes, the contents of that file. And if it sees any new packages or any packages that have been removed from that file, it will add a package record into our database and start the whole process. So that kind of gets it into the database.
We have then the next process that again, just constantly runs, is called analysis. And what that one does is it checks out the source code for the package. It runs some git commands to find out like how many versions you've got, and what the version dates were, and what the commit hashes.
Swift Toolkit
So no webhooks there.
Dave
No webhooks, we're just constantly polling. So we just, when we get to, it's like the Forth Bridge (in Scotland), isn't it? That they say that when they get to the end of painting it one way, they just turn around and start painting it the other way. Because by the time they finished, it already needs painting from the other end. So it's a bit like that: we're just constantly checking for package updates on all these repositories.
We have more than 7,000 packages at the moment. And so by the time we finish, we definitely need to start checking again. So yeah, that checks out the package source code, runs Swift dump package on it, which turns the manifest into a JSON file that we can read. And then also does a whole load of git commands to get history and things.
And then the final step is called ingestion and that checks online sources. So it goes to GitHub's API and it says, many stars does it have? What's the license for this package? And a whole load of other data that GitHub kindly make available to us. And again, that process is just constantly running. So all three of those together make up the metadata side of the process.
Swift Toolkit
And what about compatibility information and documentation for packages?
Dave
Once we find new versions that we don't have build information for or compatibility information for or documentation building performed for, we then trigger builds and documentation builds based on the discovery of those new versions. So the whole thing is based on that one bit of information, the package URL going into our database. And from there, everything else is kind of automatic and everything is discovered from there.
Swift Toolkit
Very interesting!
All right, and on the next topic, which is important and very timely and sometimes even controversial in the community: Swift 6 and strict concurrency. I see it in the same way that Swift was very strict about the type system when it launched, and we were used to it... In Objective-C you could have ID equals something, right? We still have Any in Swift, but still, it's not that used, or not recommended. And we got used to it. But it was a change that was hard.
And now we have this move that is supposed to bring the same strong or strictness about typing to the topic of concurrency. In my opinion, this is a change that it's even harder than the move from Swift 2 to 3 because it's not only syntax. You actually need to think differently in the way that you program. And you have this "Ready for Swift 6" report. You had two runs of it or is it constantly running? So what are your thoughts on that? And what made you have this idea and make it a report and have it as a source for the whole community?
The Swift 6 reportSven
Well, it was clear that, as you say, that's going to be a big change. And it became clear that it would be really interesting to get a view, like an ecosystem-wide view, what is actually the state of affairs? What happens if projects are compiled with Swift 6 mode? And I think it's really important to tell that story. Also correctly talking about the language modes, right?
You have to actually actively enable Swift 6 mode in order to get that view and to get that strictness, be exposed to that strictness. And we wanted to give a sort of an early warning system or an early state of affairs what happens if this starts running.
Not just for package authors, this is also interesting for the Swift compiler team because they are doing the beta process. And I think even now they are interested in in tuning what gets reported, and there are some features there that actually suppress reports because the compiler can be smarter about discovering when does something warrants a warning. Or when is it perhaps even provable to the compiler without you annotating something as Sendable, and it still works and take away that burden of you annotating your code, or as you say of you thinking differently.
Because sometimes you don't need to think differently, sometimes the compiler can do that inference for you and can see through your code and see that you're doing the same thing even though you're not explicit about it.
Swift Toolkit
To be honest, I feel I'm still behind. I know the basics, async-await, but when it gets to all the non-isolated, sendable, and many, many other keywords...
Dave
Well I think it's important to emphasize that if you're doing anything towards this now, you're ahead of the game, not behind!
Sven
Exactly!
Dave
Swift 6 is just out of beta. Only now is first time you can really be in this state of strict concurrency kind of readiness. But also you don't need to opt in to Swift language mode until you're ready to. You can use the Swift 6 compiler and Xcode 16 and all the modern tools without opting in to the Swift 6 language mode.
So it's absolutely in your control as a software developer when you make that choice.
Swift Toolkit
You decide when you flip the switch...
Sven
And I'd say, I'd even argue in many cases, the wise choice is just not to do that immediately, and give yourself time. Do it on a branch, you know, opt in, see what happens and pick things that you can address. You know, some things are easy to fix. You sometimes you have maybe used a var when a let would have been all right. And there are lots of easy, just low hanging fruit that you can pick off and then you can move that into your normal code base.
And over time, you can ensure that your Swift 6 language mode tracking branch has fewer fewer issues if you want to do it that way, or just every once in a while let it run and see how many warnings do you get. Effectively, that's what we're doing ecosystem-wide in our tests, we expose that number so you can see how your project is doing without you doing anything. It's just going to your package page.
But you can do that on your own and just see what warnings am I getting, which ones can I easily fix. And then also just give the compiler time because API annotation and future Swift versions will probably also reduce your burden, because that's just the way it's moving. That's another piece that is also moving forward, that you can just wait for.
Swift Toolkit
Yeah, these are definitely great tips!
Sven
And maybe one thing to stress is, because I see this often where people complain, this is throwing all these warnings at me, why is it doing that? It is doing that because there are actually issues in how you're dealing with concurrency. And we can actually tell that story in our project. So we are Swift 6 language mode compatible now. And we had to fix warnings as well.
And interestingly, we actually had rare crashes in production that were actually warned about. So we saw those warnings and it turned out these are actually, they're rare, but when you looked at the code, and after thinking about it, was not obvious in the code, but the compiler said: "look, look, this doesn't look right".
And the crashes went away. So it is actually telling you something. Yes, there are also false positives where it says something, and there isn't anything there, but the vast majority of cases it is actually highlighting things that you're probably lucky that they aren't biting you, but they will at some point potentially. So it's an improvement.
Swift Toolkit
Yeah, other day I saw someone complaining that in a project, which is only UIKit and AppKit, without zero concurrency, it was complaining about something. But the people from the Swift compiler team, are, I'm sure they are around the clock making sure that everything is working as it should be, even if we are not there yet.
Sven
Yeah, and I think it's just wise to treat this like any other API, sort of, you know, be cautious. It's version one of that thing and time will be on your side here. You don't feel pressured into moving too early because it'll, yeah, stuff will settle down. Like how did Ben Cohen, I think in his talk said, don't panic, know, just chill and give it time.
Swift Toolkit
All right. And to finalize, I had one question, which is more, not sure if personal, but more about feelings than code. So when you began the project of SPI, did you have in mind that one day it would get Apple to be an official sponsor? And in what other ways this is beneficial for the whole project and the whole community.
The sponsorship announcement on Swift.orgDave
No, not initially. When we started the open source project, as when you start any project, you have really no idea what it's gonna turn into. I think I certainly always knew that the idea was good, that we would need to have a package index, as a community we needed one. And I hoped it would become a significant and important project. And I'm sure Sven felt the same way, but we had absolutely no expectations that it would become what it has become.
But one thing that was very clear to us, as it did become something significant, is that a lot of open source projects are libraries rather than services, they don't come with quite the same level of maintenance and commitment that running a website, especially a website that does all of the build infrastructure that we have behind it. It's a very significant website to run.
And quite early on, we did realize that it was not something we were gonna just be able to put a few hours a week into, and have it be like a side project. It needed to be sustainable for us. It needed to sustain the amount of effort and responsibility that it was to keep that site available. And so we had no expectation of Apple support or anything like that, but what we did realize quite quickly is that we'd need to find some way to keep it sustainable.
And then the other thing to mention here is that the Apple support is much more than about just financial support. It is having an open line of communication with Apple and the Swift team.
Swift Toolkit
It's having their blessing in some way, right? Their seal of approval.
Sven
Yeah, absolutely!
Dave
Yes, definitely. And that relationship that we've built with Apple over the last few years is incredibly important to us, and absolutely makes the product that we have on the website better for everybody. So they've been wonderful to work with, but we also should also mention all of the other support that we get. So we have fantastic support on GitHub sponsors. The community itself can sponsor and support the site through our GitHub page. And there's a link to that on the website, of course, which takes you through to GitHub sponsors.
But also we have a couple of corporate sponsors at a time. those are companies that are supporting the project and they have a link on the homepage and a link in the supporters page as well.
And then we also get help from infrastructure sponsors as well. Mac Stadium is one of them. We just wrote a blog post about our new build system.
Swift Toolkit
I'll put a link to that post at the end, and also the link to the podcast episode where you talk about it. It is fascinating, I really enjoyed that episode!
Dave
And Microsoft as well. So Microsoft does all of our Linux hosting and our Linux build infrastructure. So we get lots of support from everywhere. Like it's not just that it's entirely supported by Apple. So it's very much, it is a community site and it's great to see Apple being part of community efforts like this. I think it's really admirable to see them do what they've done with the project.
Swift Toolkit
What about you, Sven? How did you feel when that became a reality?
Sven
That changed everything because when I got in touch with Dave in 2020 about this, I was looking for a new project and it soon became clear that this can't be a side project, as Dave said. We're nearing 3,500 issues on the server project, closed issues that we worked on and pull requests, more than 2,000 as well.
There's a lot of work going through that project. Also in terms of developer support. People have packages and they have issues, they reopen an issue. Like "my package says it's not compatible on Mac OS, but it builds", or "my documentation doesn't look right". And that sort of stuff we deal with.
And then, you know, the hosting and all that. So it was clear that this needs more work. You can't do this in an evening. But at the same time, I didn't, I mean, Dave is running iOS Dev Weekly and so he was in a better position there, but for me it was really critical to get something more substantial. We're super grateful. Early on it the only thing we had was GitHub sponsors and the corporate sponsors, but that was not enough to sustain two people's work on this, and it was just hugely helpful to get Apple and that made the thing like we're on super solid footing now to just look forward and without any concern how we operate.
Swift Toolkit
Yeah, it makes sense!
Congrats for this achievement. I don't think anyone would take that for granted. All right. I think we can wrap up here. In case people want to read more about SPI, I'm going to leave all relevant links below.
Thank you both so much. I truly feel honored and happy and grateful that you joined. Alright, sure. And see you next time.
Dave
It was our pleasure.
Sven
Thanks for having us!
Dave
All right, take care
Sven
Bye bye.