Releasing code is my thing…I used1 to ship Mac OS X updates at Apple, Firefox updates at Mozilla, the web frontend at Facebook, and many mobile apps at Facebook. Because of those experiences I’ve seen a lot of what works and what doesn’t, which platforms do things better, and tweaks that could improve the lives of developers shipping software.

Lately I’ve been giving talks about how to ship mobile apps at scale for conferences and startups2. While many of the experiences and challenges for popular apps like Facebook are unique, it is becoming clear that a lot of the experience shipping apps is universal. I decided to write down some of the current frustrations and suggested changes, starting with how I would change the App Store. It got so long that I broke it into a series of blog posts.

Note: I am sure there are other issues that can be fixed in the app store, particularly related to discovery and monetization. I know nothing about those, as apps I worked on had brand recognition with millions of users and were free.

Many of these suggestions I have already given directly to the TestFlight / iTunes Connect team3, though some are new. I’ve ordered my suggestions from what I think would be the highest impact to lowest impact. This is the first suggestion with highest impact.

🚫 Get rid of app review in some situations

App review is the #1 pain I hear from app developers I talk with. Not knowing when your app will ship can be devastating for your business.

During app review seemingly arbitrary decisions are frequently made, and just because an app passed review last time doesn’t mean it will pass review again–even with no changes.4 Different reviewers have different standards, your app can get caught up in higher-level posturing related to Apple’s long-term business, and worst of all the process is a black box no matter how much Apple documents it. While your mobile infrastructure may be perfect and automated, you still throw your app over the fence to a human on Apple’s side. Your reviewer went home because they were feeling sick? Your app ain’t getting reviewed today.5

There is, of course, expedited review for when the sky is falling. The problem is determining when the sky is really falling and if you’ll regret using one on the problem at hand. One unicorn I gave a talk at said they often have a room full of people debating if something is “bad enough” to use one of those precious lifelines. This is madness and must stop.

At Facebook we were not exempt from app review, contrary to popular belief. In fact, I would say widely-installed apps like Facebook are held to even higher standards.6

There should be a larger discussion about what human app review accomplishes in a world where app developers use server-side switches to turn on and off code for A/B testing and feature gating.7 Companies like Facebook run thousands of tests every release–no two Facebook apps are alike.8 There is no way a human could test all of them or even a large subset.9 While app reviewers do get a diff of methods and such between app versions, developers can easily obfuscate if they desire.

There is no technical or legal way to stop developers from tweaking code already in the App Store as long as native apps are able to use the internet and webviews are supported. Enabling new features and substantially altering existing ones is only going to get more prevalent as React Native use increases and services like AppHub and CodePush sprout up. The usefulness of Apple’s up-front review by humans on code packaged in a zipfile is quickly being eroded.

That being said, Apple will probably never get rid of app review…nor do I think they should. I think app review should be overhauled though:

  1. Have a verified/top developer program. Make post-1.0 apps10 shipped by these developers auto-approved if Apple’s automated review systems pass (that is, remove humans from the process). This switches app review from an adversarial skeptical-by-default model to a trust but verify model.
    • App review is mainly there to protect people from getting taken advantage of11, something that doesn’t generally happen with popular/brand name apps and developers12. It is crazy that apps like Facebook and Google Maps go through the same process as Joe Blow’s Fart App–the risk to users is different.
    • Most popular apps now ship at least every two weeks and I only see the trend accelerating13. When there is a human in the process it is virtually impossible for Apple to catch up and puts a strain on Apple employees for little end-user benefit. If this change was made app reviewers could focus on more in-depth reviews for 1.0 apps, expedited reviews, and exploratory and/or verification testing from user reports.
  2. Allow the public to install apps while they are being reviewed.
    • This would dampen the severity of a major bug, as the app developer could instruct more adventurous people to install an in-review version if they are having issues or curious users could try it if they are experiencing problems.
    • This would also help with major app or feature launches. Currently it is a pain coordinating press and ad spend because you don’t know when your app will be approved and available in advance.  You can of course wait until the app is approved and hold for developer release, but because the App Store’s search index is only updated once a day14 it may not be available for users to find without a direct link. Allowing people to install apps still in review would make the coordination a lot easier and timing less critical.
    • This could also help keep app quality high by approximating a “slow rollout” strategy.
  3. No app review for beta apps. App review for beta apps is stupid, regardless of how quick an app is approved.
    • Beta users know they are getting something that is potentially harmful to their device and data and can easily fallback to the production version whenever they want. Apple should be focusing their resources / control on production, not beta.
    • Because Apple only reviews the initial version of an update,15 once you’ve gotten a beta build approved you can resubmit without review as much as you like.16 The beta review doesn’t actually prevent anything from shipping to beta testers and instead merely adds bureaucratic overhead.
    • Without this the value of beta as a rapid feedback channel is extremely limited.17
    • If this change was made app reviewers could focus on more in-depth reviews for 1.0 apps, expedited reviews, and exploratory and/or verification testing from user reports.

Based on my experience and discussions these changes would help virtually every developer out there–big and small–without giving up control or allowing substandard apps into the App Store.

Up next are some suggested changes to TestFlight beta testing.

  1. I am now doing my own stealth startup in this space.
  2. If you want me to talk at yours, email me at my first name at my last name.com or contact me on Twitter @LegNeato.
  3. If anyone at Apple needs more context or wants confidential information, people in the Program Office know how to get in touch with me.
  4. I have so many war stories I could tell…
  5. Some apps have dedicated reviewers, others are handled by a team. Every developer is affected by this during the annual holiday shutdown however.
  6. Any badness affects a large number of users. Apple also knows smaller developers will copy the bigger ones and they don’t want to deal with all the annoying “Facebook did it too” app submissions. I am not sure if it is official policy but Apple often appears to try to treat all developers equally. This means even though apps like Facebook are exceptional (integrated in the OS, installed on most phones, drives many App Store installs for other apps, makes a lot of money) they do not really get official exceptions in the review process.
  7. Hat tip to Ryan Nielsen for pointing out I should mention this.
  8. This causes a lot of angst for product managers who join Facebook and are used to a central list of everything going on.
  9. This is partly why Facebook invests in tooling, beta channels, and slow rollout so heavily.
  10. New/1.0 apps would still be reviewed.
  11. Of course there are other strategic reasons for Apple to have an upfront say (note that app quality is NOT one of them), but they’d still have their after-the-fact veto if anything threatened their business.
  12. Here is where cynical people will scoff, but it is true. Taking advantage of people’s trust would be bad for business at the very least. Most companies with popular apps like Facebook, Snapchat, Google, Twitter are also audited by government privacy watchdogs and organizations like the FTC.
  13. There are too many benefits to the business and we’ve been spoiled by the web’s deployment model.
  14. It is insane that Facebook and Google can index the internet in basically realtime but Apple doesn’t appear to be able to index their structured data on demand.
  15. Pointed out by my ex-colleague Colm Doyle on Facebook.
  16. As long you don’t bump the public build number.
  17. Facebook uses alpha and beta in Google Play religiously. They have over 100,000 alpha testers that get the code off master every night and over 2,000,000 beta testers that get the code off a release branch every night. It’s all automated and the world has not exploded.
Tagged with: