Releasing code is my thing…I used1 to ship Mac OS X updates at Apple, Firefox updates at Mozilla, theÂ web frontend at Facebook, and manyÂ mobile apps at Facebook. Because of those experiences I’ve seen a lot of what works and what doesn’t, which platforms do things better, and tweaksÂ that could improve the lives of developers shipping software.
Lately I’ve been giving talksÂ about howÂ to shipÂ mobile apps at scale forÂ conferences and startups2. While many of the experiences and challenges for popular apps like Facebook are unique, it is becoming clear that a lot of the experience shipping apps is universal. I decided to write down some of the current frustrations and suggested changes, starting with how I would change the App Store. It got so long that I broke it into a series ofÂ blog posts.
Note: I am sure there are other issues that can be fixed in the App Store, particularly related to discovery and monetization. I know nothing about those, as apps I worked onÂ had brand recognition with millions of users and were free.
Many of these suggestions I have already given directly toÂ the TestFlight / iTunes ConnectÂ team3, though some are new. I’ve ordered my suggestions from what I think would be the highest impact to lowest impact. Previous suggestions were:
🐥Â Add slow rollout / canary for production
As app developers weÂ hope every bugÂ is caught by automated testing, dogfooding, or beta testing before shipping our apps to the general public…but thatÂ isn’t always the case. Bugs are a fact of life when dealing with software, no matter how good your testing is or how many testers you have.
Long ago the industry learned how to cope with this inevitabilityÂ by slowly rolling out changes to production rather than all at once. Slowly rolling out changes to productionÂ makes it soÂ fewerÂ people are affected by major issues, increases overall quality, and reduces pressureÂ onÂ the development team when something does go wrong. The App StoreÂ does not support this…your app is either out to 0%Â or 100% of users.4
Slow rollout has been aÂ standard practice on web and desktop deployments for the last decade. In the last couple of yearsÂ support forÂ doing this on mobile was added to Google PlayÂ and something similar should be added to theÂ App Store as well.
On Android developers can ship app updates to a (self-defined) percentage of production users. If issues are found youÂ can halt the rollout to more users, push a fix, and resume the deployment. Because of this there areÂ way fewer “the sky is falling” moments when shipping on Android than iOS. Bugs that slipped through to production are found early and contained before they affect a significant portion of users.
An unintended consequence of not having slow rollout is that it makes Apple’s app review even more of a stressful bottleneck. Having a bad app out to 5% of your users is a lot different than having a bad app out to 100%. In the former case the bleeding has stopped and in the latter every second a fix isn’t shipped is critical.
Adding support for this standard industry practice will take pressure off app reviewers, increase the quality of apps shipping to production, and seriously reduce the churn iOS engineering teams deal with every time they ship. The good news is it can be entirely implemented on the server side…no OS changes are required!
Next up, I’ll talk about bug and crash reporting.
- I am now doing my own stealth startup in this space. ↩
- If you want me to talk at yours, email me at my first name at my last name.com or contact me on Twitter @LegNeato. ↩
- If anyone at Apple needs more context or wants confidential information, people in the Program Office know how to get in touch with me. ↩
- For 1.0 apps you can approximate slow rollout by shipping only to some country stores, but once you ship to a store you must always ship there for future updates. ↩