Probability Foundations

A softmax layer in your fraud model returns [0.70, 0.25, 0.05] for one row and [0.90, 0.80, 0.10] for the next. The first row is a legal probability distribution; the second is not, even though every number sits between 0 and 1. Interviewers probe exactly this boundary — "between 0 and 1" is not the only rule probabilities follow — and the rules you cannot skip are the three axioms of Kolmogorov. You spent Optimization Landscape Challenges reasoning about how models find minima under noisy gradients; probability is the language that describes that noise, and it starts with three rules.

Let P be a function that maps events to real numbers. The axioms:

Everything else in this topic — complement rule, inclusion-exclusion, conditional probability, Bayes' theorem — is derived from these three. The broken row [0.90, 0.80, 0.10] violates normalization: 0.9 + 0.8 + 0.1 = 1.8, and a model that emits it is outputting scores, not probabilities, no matter what its loss function is called.

Two derivations you should be able to reproduce on a whiteboard in under a minute. Complement rule: A and its complement A′ partition Ω, so by additivity P(A) + P(A′) = P(Ω) = 1, hence P(A′) = 1 − P(A). If a model says P(spam) = 0.03, you get P(not spam) = 0.97 for free. Inclusion-exclusion: for any two events (disjoint or not), P(A ∪ B) = P(A) + P(B) − P(A ∩ B). The subtraction exists because additivity only covers disjoint events; when A and B overlap, summing them counts the intersection twice.

Worked example — one roll of a fair six-sided die. Ω = {1, 2, 3, 4, 5, 6} with P(each outcome) = 1/6. Let A = {2, 4, 6} (even) and B = {4, 5, 6} (at least 4). Then P(A) = 3/6 = 0.5, P(B) = 3/6 = 0.5, and P(A ∩ B) = P({4, 6}) = 2/6 ≈ 0.333. Inclusion-exclusion gives P(A ∪ B) = 0.5 + 0.5 − 0.333 = 0.667, which matches counting {2, 4, 5, 6} directly: 4/6.

omega = set(range(1, 7))
P = lambda s: len(s) / len(omega)

A, B = {2, 4, 6}, {4, 5, 6}
union = A | B
print(P(union), P(A) + P(B) - P(A & B))  # 0.6666... 0.6666...
print(1 - P(A))                           # 0.5 — complement rule

Where ML breaks the rules on purpose: a softmax head is forced through exp-and-normalize precisely so the axioms hold for mutually exclusive classes. A multi-label sigmoid head deliberately does not normalize, because each label is its own event and two labels may both fire — the per-label probabilities need not sum to 1, and forcing them to would be a modeling bug. Knowing when the sum-to-one check is legitimate separates these two head designs.

Interview trap to pre-empt: "can a valid probability be negative?" No — and neither can a probability density. Raw logits and energy-based scores live outside the axioms; they are unnormalized scores until something like softmax makes them legal. Say that distinction out loud when the question comes up.

A feature team ships a fraud model with two indicators: device_changed (fires on 10% of sessions) and new_ip (fires on 20% of sessions). The product manager asks: "what fraction of sessions show both?" Your answer depends on one question nobody asked out loud: are the two events independent?

Independence means the occurrence of one event does not change the probability of the other: P(A ∩ B) = P(A) · P(B), equivalently P(A | B) = P(A). Under that assumption, P(both) = 0.10 × 0.20 = 0.02 — 2% of sessions.

Mutual exclusivity means the events cannot co-occur at all: P(A ∩ B) = 0. These are opposite ends of a spectrum, not synonyms. Independent events usually can co-occur; mutually exclusive events are maximally dependent — learning that A happened tells you B is impossible, so P(B | A) = 0 ≠ P(B). A stock exam trap follows immediately: "if A and B are mutually exclusive, are they independent?" No, unless one of them has probability zero.

Two rules govern how probabilities combine, and both reappear constantly in ML interviews:

Now the real numbers. In 100,000 logged sessions, device_changed fires 10,000 times and new_ip fires 20,000 times. If they were independent you would expect both on 0.02 × 100,000 = 2,000 sessions. Your log shows both on 6,000 — three times the independent prediction. The events are strongly dependent: people who change devices also tend to arrive from new IPs. Treating them as independent would understate the overlap by 3×, which compounds when a model multiplies many such terms.

That compounding is exactly what Naive Bayes does: it assumes features are conditionally independent given the class and multiplies their likelihoods together. The assumption is usually false in the raw data, yet Naive Bayes still classifies well because what matters for the decision is the ranking of class posteriors, not perfectly calibrated numbers. You will build that classifier in Bayesian Thinking for Models; the point here is to name the assumption before you lean on it.

Test independence empirically in Python instead of asserting it:

import numpy as np

rng = np.random.default_rng(0)
n = 100_000
a = rng.random(n) < 0.10   # device_changed
b = rng.random(n) < 0.20   # new_ip — drawn independently

print(np.mean(a & b), np.mean(a) * np.mean(b))
# ~0.020 vs ~0.020 — joint equals product: independence holds

# now couple them: P(new_ip | changed) = 0.5, P(new_ip | not) ≈ 0.17
b_dep = np.where(a, rng.random(n) < 0.50, rng.random(n) < 0.17)
print(np.mean(a & b_dep), np.mean(a) * np.mean(b_dep))
# ~0.050 vs ~0.020 — the joint exceeds the product: dependence

The empirical test is simple and worth memorizing: estimate P(A ∩ B) and P(A)P(B) from the data; independence is exactly the claim that they are equal. When they differ, the ratio P(A ∩ B) / (P(A)P(B)) is called lift, and anything above 1 means positive dependence.

Interview framing: when handed "two coins are flipped, what is P(both heads)?", state your assumption first (fair and independent gives 0.25), then say how the answer changes if the flips are dependent. The interviewer is grading whether you name the assumption, not whether you can multiply 0.5 by 0.5.

A kiosk advertises a medical test that is "99% accurate." Prevalence of the disease is 1 in 1,000. You test positive. What is the probability you are actually infected? Most candidates say 99%. The correct answer is about 9%. This computation — the one interviewers use to check whether you understand conditional probability rather than just its notation — is Bayes' rule, and it is the single highest-leverage formula in the probability half of ML interviews.

Conditional probability is defined by P(A | B) = P(A ∩ B) / P(B), whenever P(B) > 0. The intuition: conditioning on B shrinks the sample space down to B, and you ask what fraction of B's mass sits inside A. Example with a die: P(at least 4 | even) = P({4, 6}) / P({2, 4, 6}) = (2/6) / (3/6) = 2/3.

Bayes' rule falls out of the multiplication rule. Because P(A ∩ B) = P(A | B)P(B) = P(B | A)P(A), you can solve for the conditional you want but do not have:

P(A | B) = P(B | A) · P(A) / P(B)

Read it as: posterior = likelihood × prior / evidence. The evidence term P(B) is usually computed with the law of total probability, summing over every way B can happen.

Worked example — the positive test. Prior P(infected) = 0.001. Sensitivity P(+ | infected) = 0.99. False positive rate P(+ | not infected) = 0.01. Evidence:

P(+) = 0.99 × 0.001 + 0.01 × 0.999 = 0.00099 + 0.00999 = 0.01098

Posterior: P(infected | +) = 0.00099 / 0.01098 ≈ 0.090, about 9%.

The arithmetic tells the story: in a population of 100,000, roughly 100 infected people generate ~99 true positives, while ~99,900 healthy people generate ~999 false positives. True positives outnumbered almost 10 to 1 by false ones. The positive predictive value is driven by prevalence, not by the headline accuracy — one sentence that answers half of all probability follow-up questions.

Verify with Python, both analytically and by simulation:

import numpy as np

prior, tpr, fpr = 0.001, 0.99, 0.01
evidence = tpr * prior + fpr * (1 - prior)
posterior = tpr * prior / evidence
print(posterior)                       # 0.0901...

rng = np.random.default_rng(1)
infected = rng.random(100_000) < prior
test_pos = np.where(infected,
                    rng.random(100_000) < tpr,
                    rng.random(100_000) < fpr)
print(np.mean(infected[test_pos]))     # ~0.09, matching the formula

Where ML reuses this exact arithmetic: Naive Bayes classifiers compute a posterior per class by multiplying per-feature likelihoods times a class prior and picking the argmax; A/B analyses weigh an observed effect against how often effects of that size occur by chance. The confusion between P(+ | infected) and P(infected | +) — sometimes called the inverse fallacy — is the most-tested idea in probabilistic interviews, and now you can derive the correct direction instead of recalling it.

You have the probability machinery. The next step points it at real data: Descriptive Statistics and EDA covers measuring center, spread, and shape before assuming any distribution at all.

For a concrete calculation, suppose prevalence is 1%, sensitivity is 90%, and the false-positive rate is 5%. Among 10,000 people, about 100 have the condition and 90 test positive; among the 9,900 without it, about 495 also test positive. The positive predictive value is therefore 90 / (90 + 495), roughly 15%, despite a test that catches most true cases. The base rate matters because false positives are drawn from a much larger group. State whether a quoted rate is sensitivity, specificity, or predictive value, and identify the population to which the prevalence applies. A test's predictive value can change when used in a population with a different base rate.