Releasing code is my thing…I used1 to ship Mac OS X updates at Apple, Firefox updates at Mozilla, theÂ web frontend at Facebook, and manyÂ mobile apps at Facebook. Because of those experiences I’ve seen a lot of what works and what doesn’t, which platforms do things better, and tweaksÂ that could improve the lives of developers shipping software.
Lately I’ve been giving talksÂ about howÂ to shipÂ mobile apps at scale forÂ conferences and startups2. While many of the experiences and challenges for popular apps like Facebook are unique, it is becoming clear that a lot of the experience shipping apps is universal. I decided to write down some of the current frustrations and suggested changes, starting with how I would change the App Store. It got so long that I broke it into a series ofÂ blog posts.
Note: I am sure there are other issues that can be fixed in the app store, particularly related to discovery and monetization. I know nothing about those, as apps I worked onÂ had brand recognition with millions of users and were free.
Many of these suggestions I have already given directly toÂ the TestFlight / iTunes ConnectÂ team3, though some are new. I’ve ordered my suggestions from what I think would be the highest impact to lowest impact.Â This is the secondÂ suggestion, the first was to get rid of app review in some situations.
👥Â Raise the beta tester limit
While Apple really shouldÂ get rid of beta app review, there is another large beta pain point affecting virtually everyÂ successful app developer–limited testers.
Apple currently allows iOSÂ developers to ship pre-release apps to 2,000 beta testers via TestFlight. This isÂ not enough testersÂ to validateÂ apps broadly at scale.
It is impossible for 2,000 testers to cover all the combinations of OS versions, networks, devices, and usage patterns. Even if those factors do not affect your particular app, the inherent statistical properties of finding issues in a population point to the need for a larger testing audience.4
Facebook itself has over 11,000 employees so adding 2,000 external testers doesn’t provide value for an app that has millions of users. Even for smaller apps and companies, with the prevalence of A/B testing and feature flags you often need to split your beta audience into 2n groups to test the combinations together. This is virtually impossible with 2,000 testers.
Google Play has no such limit and leaves it up to the app developer to determine the optimal number of testers necessary to ensure production quality. Facebook hasÂ over 2,000,000 daily beta testers on Android makingÂ sure all the various devices, networks, languages, and features are tested and the app is ready for release. This large beta population allows Facebook to use beta as a predictor of production–the whole reason for having a beta andÂ something that is impossibleÂ with 2,000 testers.
Every major app developerÂ I have talked to thinksÂ shipping on iOS is scary while shipping on Android is a breeze–largely because of beta audience size. There are more frequent “oh shit” moments and mad scrambles on iOS for bugs that slipped to production due to lack of a large beta audience. OfÂ course, fixing those bugs quickly and pushing out a fix (after a perhaps lengthyÂ review) is scary too and can cause even more issues. No one feels good or safe pushing to the App Store because they are shipping an app tested by at max 2,000 testers to (hopefully)Â millions of users–it’s always push the button and pray.Â The 2,000 beta tester limit is hurting users by allowing more bugs to slip throughÂ and causing unnecessary churn for app developers.
Someone atÂ Apple told me that there is no technical reason why the limit is at 2,000. They saidÂ it is a policy that came down from on high. This policy appears to be created by someone who has never shipped an app and it needs to change.
Apple’s own iOS and Mac OS XÂ beta testing programs haveÂ millions of testers, so they clearly recognizeÂ the benefits of having a larger-than-2,000 testing pool. Not allowingÂ developers the same benefits is downright insulting.
Apple should change the TestFlight user limitÂ to 2,000 testers or 5% of active installs, whichever is greater. This will allow app developers to test at the scale of their app withoutÂ routingÂ around app review.
Next I have some suggestions forÂ how apps are deployed in the App Store.
- I am now doing my own stealth startup in this space. ↩
- If you want me to talk at yours, email me at my first name at my last name.com or contact me on Twitter @LegNeato. ↩
- If anyone at Apple needs more context or wants confidential information, people in the Program Office know how to get in touch with me. ↩
- As Philipp von Weitershausen pointed out on Facebook: “if you have an error that occurs for users, say, 5% of the time, and you want to make sure you catch it with 99% probability, you need log(1-.99)/log(1-.05) = 90 testers. Now imagine the error rate is much lower, and you have a bunch of those low-firing errors. And perhaps you want to catch more than 99% of them… You get into the 1,000s and 10,000s quickly.” ↩