How would you evaluate an AI study planner built from exam-performance history?

We’re building Zaiko inside MED.Z for Brazilian medical residency candidates. It reads a student’s study schedule and recorded accuracy by specialty/topic, then recommends priorities and helps assemble a daily plan.

I’m designing an evaluation framework around four questions: Are the recommendations grounded in the student’s own data? Are priorities pedagogically sensible? Do students follow the plan? Does performance improve over time?

What offline and online evals would you add before scaling this feature? I’m especially interested in failure cases, calibration, and ways to keep the learner in control.

Context: medz.app.br/