Understanding the Kolmogorov–Smirnov (KS) Test: A Practical Guide for Model Validation

In a world increasingly shaped by predictive models—from credit scoring to fraud detection—it's critical to ensure our models are not only statistically sound but also interpretably effective. One of the most widely used tools for model validation in binary classification problems is the Kolmogorov–Smirnov (KS) statistic.

Despite its widespread use, particularly in credit risk modeling, KS is often misunderstood or applied mechanically. I'm writing about KS because recently, an Auditor went into a discussion over a low KS value of a scorecard, without even knowing or understanding KS and didn't even know what's the interpretation of this score.

This article aims to demystify the KS test: what it measures, how to compute it, and how to interpret the results in context.

What Is the KS Test?

The Kolmogorov–Smirnov test is a non-parametric test that measures the maximum distance between the empirical cumulative distribution functions (CDFs) of two samples. In the context of predictive modeling, it typically compares the CDFs of "goods" and "bads"—i.e., non-defaulting vs. defaulting accounts.

The idea is simple:

A good model should assign significantly different score distributions to different outcome classes.

KS measures the degree of separation between these two distributions.

Why We Use KS in Risk and Credit Scoring

In credit scoring:

"Goods" are customers who don't default.
"Bads" are those who do.

A well-performing model should rank "bads" with higher risk scores than "goods." The KS statistic quantifies this ability to discriminate.

How to Compute the KS Statistic

Here's a simplified step-by-step method to compute the KS statistic:

Step 1: Score Your Data

Apply your predictive model to generate probability scores or risk rankings for each observation.

Step 2: Bin the Data

Sort the data in descending order of score and divide it into bins (e.g., deciles).

Step 3: Compute Cumulative Distributions

For each bin, compute:

The cumulative % of goods
The cumulative % of bads

Step 4: Calculate KS

At each point, compute:

KS = | Cumulative % of goods – Cumulative % of bads |

The maximum difference across all bins is your KS statistic.

Example Interpretation

KS = 20%, observed in Bin 4.

How to Interpret KS Values

Important: High KS is not always "better" if the model is overfitted or not generalizable across segments or time periods.

Limitations of KS

Sensitive to Binning: Results may vary with the number of bins or bin boundaries.
Ignores Business Cost: KS treats all errors equally; it doesn't account for the cost of false positives/negatives.
Snapshot Metric: It doesn't reflect performance over time unless tracked as part of a stability report.
Not Suitable for Imbalanced Data Alone: KS may be misleading when there's a large class imbalance without further context.

KS vs. AUC (ROC)

AUC measures the probability that a randomly chosen "bad" has a higher score than a randomly chosen "good."
KS focuses on the maximum separation point.

They are related, but not interchangeable. KS is often favored in regulated credit risk environments due to its intuitive segmentation relevance.

When to Use KS

During scorecard development
For model monitoring and drift detection
In validation reports (e.g., for regulatory or audit purposes)
To set cutoff thresholds or analyze segmentation performance

Final Thought

"A model that predicts well must also separate well. KS helps quantify that separation—but it's not the whole story."

Use KS as part of a toolkit, not as a standalone verdict. Complement it with business KPIs, confusion matrix analysis, population stability index (PSI), and domain expertise.

Let's Discuss

How do you integrate KS in your model validation pipeline?
Have you found cases where a high KS misled the business?
What are your preferred metrics beyond KS for model interpretability?

I'd love to hear your experiences—especially from those working at the intersection of data science, credit risk, and real-world deployment.

Originally published on LinkedIn.