My AcademicTutor
Request Support
← Back to Machine Learning in Biostatistics

Module 3 · Machine Learning in Biostatistics

Model evaluation, validation and performance.

This module teaches how to judge whether a prediction model works beyond the data used to fit it. Students move from confusion matrices and ROC curves to calibration, resampling, bootstrap validation, leakage checks and responsible reporting.

View Module 3 lessons →Open Lesson 1.1Start learning

5

Lessons

0

Open now

R

Labs planned

July 2026

Full release

What this module builds

Performance judgement beyond accuracy.

Performance is not one number

Accuracy, sensitivity, specificity, ROC, AUC and calibration answer different questions. This module teaches students to read them together rather than choosing one headline metric.

Validation protects against overconfidence

A model can look excellent on the data that trained it and fail on new patients. Test sets, cross-validation and bootstrap validation help estimate future-patient performance more honestly.

Thresholds change decisions

A predicted risk is not a clinical decision until a threshold is chosen. Threshold choice changes false positives, false negatives and clinical workload.

Calibration matters in medicine

Clinical prediction models often communicate risk. If a model says 30% risk, the observed risk should be close to 30% among similar patients.

Leakage can destroy trust

If information from the future, the outcome process or the test set enters model fitting, the reported performance can become dangerously optimistic.

By the end

Students should evaluate health prediction models carefully.

Explain why training performance is usually too optimistic.
Interpret confusion matrices in clinical prediction settings.
Calculate and explain accuracy, sensitivity and specificity.
Understand ROC curves and AUC as discrimination summaries.
Explain why calibration is different from discrimination.
Use cross-validation and bootstrap validation conceptually.
Recognise data leakage and poor predictor timing.
Write cautious performance reports for medical ML models.

Module pathway

From test sets to responsible reporting.

The module shows how model evaluation moves from splitting data and counting errors to discrimination, calibration, validation, leakage prevention and transparent reporting.

Step 1

Split

Separate model fitting from model evaluation so performance is not judged only on data already seen by the model.

Step 2

Classify

Convert predicted risks into classifications using thresholds, then inspect true positives, false positives, false negatives and true negatives.

Step 3

Discriminate

Use ROC curves and AUC to understand how well the model ranks higher-risk and lower-risk observations.

Step 4

Calibrate

Check whether predicted probabilities match observed risk, especially when predictions are used for risk communication.

Step 5

Validate

Use cross-validation, bootstrap validation and leakage checks to estimate performance honestly.

Step 6

Report

Write a transparent performance summary that includes uncertainty, limitations and clinical caution.

Lesson design

The evaluation lessons will be output-led.

Clinical prediction examples based on health-data decisions
Detailed notes linking model evaluation to biostatistical reasoning
Browser R coding labs planned for every full lesson
Downloadable R scripts for local practice
Confusion matrix, ROC, calibration and validation outputs
Report sections translating output into interpretation
Caution boxes for leakage, optimism and threshold misuse

Current release state

Module 3 is open for preview.

Students can see the Module 3 pathway now. The full lessons are locked while they are redesigned with R scripts, confusion-matrix outputs, ROC and calibration visuals, validation reports and clinical interpretation.

Start learning →

Module lessons

Study the lessons in order.

All Module 3 lessons currently route to the waitlist. The full release will move from train/test splitting to classification metrics, calibration, validation and leakage prevention.

3.1

Lesson

Locked95–110 minHonest validation

Train/test split and resampling

Understand why training performance is optimistic, how test sets imitate unseen patients, and why resampling gives more stable performance estimates.

Train/test splitResamplingGeneralisation
Locked until July 2026

Start learning →

3.2

Lesson

Locked100–120 minDiscrimination

Classification metrics, sensitivity, specificity, ROC and AUC

Interpret confusion matrices, accuracy, sensitivity, specificity, ROC curves and AUC in clinical prediction models.

Confusion matrixROC/AUCThresholds
Locked until July 2026

Start learning →

3.3

Lesson

Locked100–120 minRisk reliability

Calibration, clinical usefulness and decision curves

Assess whether predicted probabilities agree with observed risk, and connect model predictions to decision thresholds and clinical usefulness.

CalibrationRisk predictionDecision curves
Locked until July 2026

Start learning →

3.4

Lesson

Locked100–120 minInternal validation

Cross-validation and bootstrap validation

Use cross-validation and bootstrap validation to estimate model performance more honestly than a single training performance summary.

Cross-validationBootstrapOptimism
Locked until July 2026

Start learning →

3.5

Lesson

Locked100–120 minResponsible evaluation

Bias, leakage and reproducibility in health ML

Identify common sources of biased model evaluation, including leakage, poor predictor timing, unrepresentative data and irreproducible workflows.

LeakageBiasReproducibility
Locked until July 2026

Start learning →

Join the waitlist

Get access updates when Module 3 opens in July 2026.

Module 3 lessons are currently locked while they are redesigned with validation R labs, ROC and calibration visuals, bootstrap workflows, leakage checks and report-style interpretation.

Module 3 overview stays open.
All Module 3 lessons remain opening in July 2026.
Lesson 1.1 remains available as the course preview.
Waitlist visitors can request early access or release updates.

Waitlist form

Request access.

Recommended start

Begin with the open foundation lesson.

Lesson 1.1 introduces the course structure: prediction question, R script, model output, interpretation, report writing and responsible modelling caution.

Open Lesson 1.1 →