Releasing code is my thing…I used1 to ship Mac OS X updates at Apple, Firefox updates at Mozilla, the web frontend at Facebook, and many mobile apps at Facebook. Because of those experiences I’ve seen a lot of what works and what doesn’t, which platforms do things better, and tweaks that could improve the lives of developers shipping software.

Lately I’ve been giving talks about how to ship mobile apps at scale for conferences and startups2. While many of the experiences and challenges for popular apps like Facebook are unique, it is becoming clear that a lot of the experience shipping apps is universal. I decided to write down some of the current frustrations and suggested changes, starting with how I would change the App Store. It got so long that I broke it into a series of blog posts.

Note: I am sure there are other issues that can be fixed in the App Store, particularly related to discovery and monetization. I know nothing about those, as apps I worked on had brand recognition with millions of users and were free.

Many of these suggestions I have already given directly to the TestFlight / iTunes Connect team3, though some are new. I’ve ordered my suggestions from what I think would be the highest impact to lowest impact. Previous suggestions were:

  1. Get rid of app review in some situations.
  2. Raise the beta tester limit.

🐥 Add slow rollout / canary for production

As app developers we hope every bug is caught by automated testing, dogfooding, or beta testing before shipping our apps to the general public…but that isn’t always the case. Bugs are a fact of life when dealing with software, no matter how good your testing is or how many testers you have.

Long ago the industry learned how to cope with this inevitability by slowly rolling out changes to production rather than all at once. Slowly rolling out changes to production makes it so fewer people are affected by major issues, increases overall quality, and reduces pressure on the development team when something does go wrong. The App Store does not support this…your app is either out to 0% or 100% of users.4

Slow rollout has been a standard practice on web and desktop deployments for the last decade. In the last couple of years support for doing this on mobile was added to Google Play and something similar should be added to the App Store as well.

On Android developers can ship app updates to a (self-defined) percentage of production users. If issues are found you can halt the rollout to more users, push a fix, and resume the deployment. Because of this there are way fewer “the sky is falling” moments when shipping on Android than iOS. Bugs that slipped through to production are found early and contained before they affect a significant portion of users.

An unintended consequence of not having slow rollout is that it makes Apple’s app review even more of a stressful bottleneck. Having a bad app out to 5% of your users is a lot different than having a bad app out to 100%. In the former case the bleeding has stopped and in the latter every second a fix isn’t shipped is critical.

Adding support for this standard industry practice will take pressure off app reviewers, increase the quality of apps shipping to production, and seriously reduce the churn iOS engineering teams deal with every time they ship. The good news is it can be entirely implemented on the server side…no OS changes are required!

Next up, I’ll talk about bug and crash reporting.

  1. I am now doing my own stealth startup in this space.
  2. If you want me to talk at yours, email me at my first name at my last or contact me on Twitter @LegNeato.
  3. If anyone at Apple needs more context or wants confidential information, people in the Program Office know how to get in touch with me.
  4. For 1.0 apps you can approximate slow rollout by shipping only to some country stores, but once you ship to a store you must always ship there for future updates.
Tagged with: