The prediction question and the intervention question
A repayment model asks how an outcome is associated with information available at a defined time. A causal study asks how the outcome would differ if a specific decision changed. The two questions overlap, but a strong predictor need not identify a useful intervention. Income can predict repayment without a credit policy that raises an applicant’s reported income improving repayment.
A concrete credit-builder-loan evaluation illustrates the distinction. In the study summarized by J-PAL, people choosing the product showed improving scores, yet the randomized offer comparison did not show an average score improvement. Selection into take-up carried information that was different from the product’s average causal effect. [1]
The missing counterfactual
For a particular household, researchers cannot observe both its income after receiving a loan and its income at the same moment without that loan. One of these potential outcomes is missing. A comparison group is an attempt to estimate the missing counterfactual, not merely a convenient set of records with a different loan status.
Suppose approved applicants earn $4,000 more a year than rejected applicants. That gap could reflect the loan’s effect, differences in initial earning capacity or both. If approval favored people with stable employment, the observed gap may largely reflect the underwriting rule. Matching on recorded characteristics can help only to the extent that the relevant differences are measured and modeled adequately.
The same issue arises within one group over time. A customer may take credit just before an expected increase in income. Comparing income before and after borrowing then combines the loan, the anticipated job change and wider economic conditions. More data can estimate that association precisely while leaving its causal interpretation unresolved.
Randomization identifies the assigned intervention
Random assignment can make treatment and comparison groups comparable on average before the intervention. The precise treatment still matters: an offer, easier application process, lower rate, larger limit and actual disbursement are different interventions. A randomized encouragement changes the probability of borrowing without necessarily forcing borrowing.
In the J-PAL credit-builder study, 1,531 interested credit-union members were assigned either immediate access or access conditional on completing an online education step. Take-up was about 30% versus 12%. This was not a trial in which every assigned person received a loan and every comparison person received none. The research setting and the mechanism affecting participation are part of the interpretation. [1]
A separate South African study gave some previously rejected, potentially creditworthy applicants a randomized second look. Loan officers were encouraged, not required, to approve them. That design again distinguishes assignment to reconsideration from actual lending. It does not justify describing the entire applicant population as randomly assigned to funded loans. [2]
Offer effects and borrower effects
Consider a hypothetical experiment with 1,000 people assigned an offer and 1,000 assigned no offer. If 300 in the offer group borrow and none in the comparison group do, take-up differs by 30 percentage points. Suppose average annual earnings across everyone assigned the offer are $600 higher. The intention-to-treat estimate is $600: the effect of assignment to the offer for the experimental population.
Dividing $600 by 0.30 gives $2,000. Under additional instrumental-variable assumptions, that ratio can identify an average effect for people whose borrowing was changed by the offer. Those assumptions include that assignment affects the outcome through borrowing rather than another channel, and a monotonic response to the encouragement. The calculation does not establish a $2,000 effect for every borrower. This distinction is the local-average-treatment-effect logic developed by Imbens and Angrist. [5]
The exclusion assumption is especially substantive. If the offer also includes counseling that directly changes employment search, assignment can affect earnings without borrowing. If comparison participants obtain loans elsewhere, the first-stage difference concerns the measured borrowing treatment, not simply use of the study lender. Arithmetic cannot repair a poorly defined intervention.
A debt-relief experiment answers a different question
Kluender, Mahoney, Wong and Yin studied medical debt relief through two randomized experiments. The published 2025 article concerns downstream debt that had been, or was about to be, sent to collections. Many average outcomes showed no detectable improvement, although relief reduced payments on existing medical bills. In the subsample whose debts would otherwise have been reported to credit bureaus, relief improved scores and credit limits without detected changes in borrowing or financial distress. [3]
This is evidence about the studied relief interventions and setting, rather than a universal verdict on credit access, healthcare affordability or all forms of debt forgiveness. Eliminating an old collection account is not equivalent to preventing a medical expense, supplying emergency cash or changing an interest rate. The stage at which an intervention occurs can alter its mechanism.
A hypothetical household with an old unpaid $5,000 bill may currently make no payments on it. Forgiving that balance reduces a liability, but does not necessarily free $5,000 of current cash. A household paying $200 each month on another debt could experience a different cash-flow change from relief. Face value alone is not a common treatment dose.
Quasi-experiments use institutional variation
Randomized trials are not the only source of causal evidence. Herkenhoff, Phillips and Cohen-Cole’s November 2016 working paper uses changes around bankruptcy-flag removal and linked employment, business and credit records. The identification strategy exploits an institutional change in credit access; it is not a randomized loan experiment. [4]
The underlying reasoning differs from comparing arbitrary high-score and low-score consumers. A discrete reporting change may provide a more informative comparison, but interpretation still depends on the design and possible channels. If a record change also affects how employers view an applicant, an employment response cannot automatically be labeled the effect of additional borrowed dollars alone.
Related designs have their own limits. A lending-score cutoff can identify effects near the threshold when its assumptions hold, rather than for all applicants. A policy comparison over time relies on a credible account of what would have happened without the change. Design labels summarize methods; they do not substitute for the identifying assumptions.
Outcomes, uncertainty and reach
Repayment, household consumption, business survival and lender profitability are different outcomes. A policy can improve one while weakening another. A zero estimated average can also coexist with positive and negative subgroup effects, although searching many subgroups after seeing results can create misleading apparent discoveries.
Follow-up length changes what is measured. A six-month study can miss later debt stress or business growth. Missing survey responses can matter if participation changes with treatment or outcomes. Administrative records reduce some reporting problems while introducing their own boundaries, such as activities occurring outside the observed lender or credit bureau.
The final distinction is between internal and external validity. A credible estimate for interested members of one credit union, marginal rejected applicants or holders of particular collection accounts does not automatically transfer to another product or economy. Causal research makes the intervention question answerable within a defined setting. Its value depends on preserving that definition rather than turning an identified local result into a universal claim about lending.
Sources
- J-PAL, Credit-building loan evaluation in the United States, study timeline 2014–2015; checked October 4, 2026SourceBack to text: ↑1↑2
- J-PAL, Small Individual Loans and Mental Health in South Africa, study-design description; checked October 4, 2026SourceBack to text: ↑
- Kluender, Mahoney, Wong and Yin, Effects of Medical Debt Relief, Quarterly Journal of Economics 140(2), May 2025SourceBack to text: ↑
- Herkenhoff, Phillips and Cohen-Cole, NBER Working Paper 22846, November 2016Technical reportBack to text: ↑
- Imbens and Angrist, Identification and Estimation of Local Average Treatment Effects, Econometrica, March 1994; author institution summarySourceBack to text: ↑