The Science of Encouraging Honesty
Imagine you are about to invest in a company and you naturally wish to evaluate how much your investments are likely to be worth at the time of an exit. You ask the founders to give you their estimates and they come up with a probabilistic statement along the lines of:
- 10% chance our company becomes a billion dollar business without further investment
- 30% chance we succeed but we need further investment and your shares will be diluted by 40%
- 20% chance we fail at product/market fit but the team will still get acqui-hired for $50m
- 40% chance we go under and your investments are worth nothing, etc.
How do you ensure that the founders’ forecast is honest, that is it truthfully represent their current beliefs about what might happen? Well, statistics has an answer, known as strictly proper scoring rules.
Scoring rules of probabilistic forecasts
Scoring rules are used to evaluate probabilistic forecasts of some random quantity $X$. In our example $X$ would be the company’s exit valuation or the investors’ return on their investment. A forecast can be modelled as a probability distribution, $P$, over possible outcomes.
A scoring rule S assigns a payoff to every combination of a forecast $P$, and actual outcome $x$. So if someone gives a forecast $P$, and the actual value turns out to be $x$,they get a payoff of $S(P,x)$.
Let’s assume the founders did their homework and came up with a forecast, $P$, that represents their true knowledge and expectations. But they may choose to lie to you, and give you a different forecast, $Q$ instead. At the time of the exit, the company’s actual value $x$ is revealed, and the founders receive a payoff $S(Q,x)$. The payoff function S is called a strictly proper scoring rule if it ensures that it is the founders’ rational interest to reveal their true forecast $Q=P$.
If founder’s payoff is not a strictly proper scoring rule, they will be incentivised to lie to you.
An example of a strictly proper scoring rule is the logarithmic score
$$S(Q,x) = A + B·logQ(x).$$
$Q(x)$ is the probability assigned to the actual outcome, A and B are some constants. Because logarithm is a monotonically increasing function, the more probability one assigned to the true outcome, the more their payoff will be.
A logarithmic payoff function guarantees that the forecaster is rationally incentivised to reveal their true belief, $P$, about the quantity in question. It is not the only payoff mechanism that guarantees it. The slightly less straightforward Brier score may be closer to a practical suggestion.
Mechanisms that encourage truthfulness are important in business. A good example is the incentive compatible second-price auction schemes implemented in many advertisement marketplaces. When investing in a company, the stakes and risks involved are probably too complex to be modelled by simple payoff functions. But whenever you pay for predictions or forecasts, think about whether the forecaster is properly incentivised to give you their most honest estimates.