How to tell analysis from numerology

People see patterns, but sometimes they are coincidences. To analyze numbers in a meaningful way, you must be disciplined both in how you collect them and how you connect them. Otherwise, you’re just doing numerology.

Take Charles Arthur’s article in The Guardian called “90:9:1 – the odd ratio that technology keeps creating.” The intelligent reader asks (1) does technology keep creating it? and (2) if so, why?

This article points out how market share (and other things) seem to settle into a familiar pattern in different markets. Excerpts:

What do operating systems, browsers and search engines all have in common? It seems to be a ratio of 90:9:1 between the key players. One player dominates; then others get a minimal share.

There are now three main [mobile] platforms – Google’s Android, Apple’s iOS and Microsoft’s Windows Phone – for which worldwide shipments are currently running in a ratio of about 85:14:1 respectively.

Now look at desktop OS sales: the ratio stands in the most recent quarter at about 91:8:1 between Microsoft’s Windows, Apple’s Mac OSX, and “self-build” machines which probably get Linux.

It’s oddly reminiscent of the “1% rule” – a rule of thumb observed as far back as 2006, which states that if you have a group of 100 people interacting online, then one will generate some content, nine will provide feedback, and 90 will simply consume it. (Studies have broadly confirmed that principle.)

OK, let’s see why this is just numerology. First off, Arthur offers no plausible connection (because there is none) between the 90:9:1 rule of social networks and technology market share — a ratio that’s not really valid anymore in any case. So that’s a red herring.

As for technology markets, Arthur himself points out that right now, browser share and game consoles are more evenly divided with no 90% player. Markets evolve. If you’re trying to establish a trend or pattern, you can’t randomly cherry-pick data that fits your conclusion. (Even the 84:14:1 for mobile platforms isn’t that close.)

Arthur’s own faith in his conclusion is pretty shakable:

So is 90:9:1 an incontrovertible law for products? That remains to be seen; but it may be a guide to the future on mutually exclusive activities. Will “taxi” companies (Uber, Lyft, Hailo) follow the same path? Or machine-learning algorithm deployment? It might be worth checking back in a few years.

My definition of bullshit is “writing that wastes the reader’s time.” This article, like all numerology, certainly qualifies.

Even if The Guardian is full of crap, is there anything to this? Let’s think like analysts. While Arthur suggests we look at commoditization of components and “networked” markets, I’d like to get more rigorous.

In technology markets, various forces enable companies to gain dominant share. For example:

• Network effects — more users attract more users. That’s why Facebook is dominant.
• Platform effects — more users attract developers building on platforms, which generates more applications, which attract more users.
• Economies of scale — more buyers create efficiencies, which keep costs down, which give dominant players an advantage.

Other factors keep those share numbers from getting too dominant. For example:

• Niche players hang on to customers. Some users will be reluctant to give up on products they use like Blackberry, even if those products lose share. Other buyers may buy a niche product to maintain competition, or because they have special needs — they need a more advanced photo editor than Photoshop, for example.
• Disruptive newcomers attract fans. Think about what Google Docs is doing to Microsoft Office, for example.

Now we have identified some plausible mechanisms for when market shares could get into 90:10 or 90:9:1 alignments — when a single player generates growth that snowballs based on these effects. And we can see when those alignments will cease — when disruptive players remake markets.

Based on this analysis, you could identify when the game console market might generate a more lopsided alignment, or when the dominance of Microsoft in office suites might end. It accounts for the arc of technology markets, a topic that Sucharita Mulpuru and I explored at Forrester.

Rather than just play with numbers, we attempted to explain how shares shift and why. An analytical conclusion about patterns in numbers needs to account for all of the available data, have a plausible mechanism, and allow for future testing, so you can see if it’s actually predictive. That’s analysis. And it’s a lot more useful and interesting than numerology.