full-stack-deep-learning/fsdl-course

Great post from a16z on long-tailed AI problems: https://a16z.com/2020/08/12/taming-the-tail-adventures-in-improving-ai-economics/

My favorite quotes:
1 - Long-tailed distributions are also extremely common in machine learning, reflecting the state of the real world and typical data collection practices.

2 - The long tail can contribute to high variable costs beyond infrastructure. Even worse, AI businesses working on long-tailed problems can actually show diseconomies of scale – meaning the economics get worse over time relative to competitors.

3 - ML is not a religion, but science, engineering, and a little art.

4 - If you are working on a long-tail problem, it’s critical to determine the degree of consistency across customers, regions, segments, and other user cohorts. If the overlap is high, it’s likely you can serve most of your users with a global model (or ensemble).

5 - In "componentizing", each model addresses a global slice of data – rather than a particular customer, for instance – and that the sub-problems are relatively bounded and easy to reason about.

6 - Over time, models that share similar functionality can be joined together with a common “trunk” to reduce complexity. The goal is to make the trunk models as “thick” as possible (i.e. doing most of the work) while making the task-specific “branch” models as “thin” as possible – without sacrificing accuracy