Fundraising Operations
Forecast Probabilities: Calibrate the Confidence You Report
Evaluate the confidence behind your forecasts. Use a resolved-event log, worked Brier score and a practical review of overconfidence.
What you will learn
- Write a resolvable probabilistic forecast.
- Compare stated probabilities with observed frequencies.
- Use a Brier score without confusing it with calibration alone.
A probability forecast becomes useful when the event, deadline and resolution rule are clear. Calibration asks whether events assigned a probability happen at roughly that frequency over many comparable forecasts. One confident prediction that comes true does not establish a calibrated forecasting process.
Write forecasts that can be resolved
“The pipeline looks strong” cannot be scored. “There is a 60% chance that account A signs the specified annual agreement by June 30” can be resolved if you define what signed means, which agreement qualifies and the relevant timezone. Keep the forecast as it existed before the result was known.
Do not turn an investor’s friendly response into a precise close probability without a defensible basis. A forecast is a judgment to evaluate, not a commitment from the other party.
Build the original forecast log
Five fictional, already-resolved binary forecasts
| Forecast | Stated probability | Outcome, 1=yes and 0=no | Squared error |
|---|---|---|---|
| A | 0.80 | 1 | 0.04 |
| B | 0.80 | 0 | 0.64 |
| C | 0.60 | 1 | 0.16 |
| D | 0.40 | 0 | 0.16 |
| E | 0.20 | 0 | 0.04 |
For this binary convention, the Brier score is the mean of (probability − outcome)². The example totals 1.04 across five forecasts, so its score is 0.208. Lower is better for this score on comparable events, but a single small log is not evidence of forecasting skill.
Read calibration separately from the score
The two forecasts assigned 80% resolved yes once out of two, or 50%. That is an observation to retain, not enough evidence to conclude that future 80% forecasts should all become 50%. Build a larger history of comparable events and report the number in each probability band.
scikit-learn’s calibration documentation explains reliability diagrams and notes that the Brier score reflects discrimination and uncertainty as well as reliability. It should not be treated as a pure calibration score.
Compare like with like
- Use forecasts with the same event definition and similar lead time.
- Keep unresolved events out of the scored result until the resolution rule is met.
- Retain canceled or invalid cases with a reason; do not quietly discard misses.
- Preserve earlier forecasts when making an updated estimate.
- Avoid mixing many predictions about the same event as if they were independent opportunities.
- Show base-rate comparisons using the same events and scoring convention.
Turn a review into better judgment
Look for recurring evidence behind overconfidence: treating verbal interest as a decision, ignoring approval dependencies or forgetting how much time remains. Change the questions you ask and the evidence you require before changing the number.
The sales-forecasting workflow remains the execution guide for a pipeline review. This lesson teaches how to evaluate probabilities after outcomes are known. It does not assign a default fundraising or sales probability to a stage.
Practice: compare two misses
Consider two separate hypothetical forecasts that both resolve no. An 80% forecast contributes 0.64 to the squared error; a 60% forecast contributes 0.36. Both classify the event as yes at a 50% threshold and miss, but the more confident miss receives the larger error. A simple correct/incorrect tally would miss that difference.
Keep the dated judgment, supporting evidence and next review with the relevant relationship in Finta. Use a separate reviewed calculation for the score; do not imply that a CRM stage is already a calibrated probability. The aim is better decisions, not a more scientific-looking percentage.