AI Handbook Assistant
Role: Full-stack developer, AI evaluation researcher
Duration: February 2026 – Present
Tools: RAG; embedding; AI evaluation rubric; failure-mode analysis
Table of Contents
- Project overview
- Background
- An AI solution
- Product demo
- Evaluation and iteration
- Summary & next steps
- Resources
Project overview
Problem: PhD students often struggle to interpret complex degree policies and find the current handbook insufficient for quickly answering situation-specific questions.
Solution: I built a RAG-based AI Handbook Assistant that answers students’ questions about program requirements with concrete suggestions and citations. I also developed an evaluation workflow based on diagnostic and held-out policy questions.
Outcomes: Early evaluation identified retrieval and evidence grounding as the main bottlenecks. After redesigning the system, 27 of 30 responses were fully successful in a held-out evaluation, representing a 90% fully successful response rate.
Background
A handbook for psychology students
The USC Psychology PhD program has over 100 students across six academic areas. Each area has different course requirements, program milestones, and degree timelines.
To make this information accessible, the department maintains an annually updated graduate handbook to serve as the official reference point for many high-stake student decisions, from research planning to navigating relationships with advisors, to submitting the right documents to meet graduation requirements.
The problem
In practice, however, the handbook has often been a source of confusion. In a recent survey of 74 Psychology PhD students, about 40% explicitly considered the department milestones and requirements to be unclear. Open-ended responses further suggested that lack of clarity in the Handbook was a major cause of confusion: it contained apparently conflicting information, vague language, and with 50 pages and over 17,000 words, students often have trouble locating the right section for their needs.
Survey results from 74 Psychology PhD students based on 2026 Town Hall survey
Due to these frictions, students often have to seek support from one-on-one appointments with department administrators or informal conversations with senior classmates. Both are useful, but neither fully solves the access problem: appointments may not provide immediate support, while peer advice can be unstructured, incomplete, or outdated.
An AI solution
To address this gap, I built a RAG-based AI Handbook Assistant that lets students ask customized questions about PhD requirements and receive answers grounded in the original handbook. The goal is to provide information and support tailored to each student’s needs, and help them decide who to ask next when a situation is ambiguous.
Design priorities
Given the high-stake nature of students’ inquiries, an AI handbook assistant needs to do more than produce a plausible answer. It needs to answer from an identifiable source of truth, make the original evidence easy to verify, and acknowledge uncertainty when the handbook language is ambiguous. Accordingly, I defined the following design priorities:
Source-grounded answers
Every useful answer should point back to handbook language, so students can verify the basis for the response.
Uncertainty-aware guidance
The interface distinguishes direct support, interpretation, and missing information instead of making every answer sound equally certain.
Role boundaries
It should not replace academic advising. For high-stakes decisions, the assistant nudges students to confirm with the program administrator, instead of providing definitive answers
RAG workflow
The system parses the handbook into sections and smaller searchable units. When a student asks a question, it combines keyword matching and semantic similarity to retrieve the most relevant sections, including related policies that may appear elsewhere in the handbook. The language model then identifies the useful evidence and generates a concise answer with citations, page numbers, and suggested next steps.
RAG workflow for the AI Handbook Assistant.
Product demo
Qualifying exam timeline
This example shows how the assistant turns a student question into a concise answer, then lets the student inspect the evidence behind it.
Answer
The full qualifying exam is due by the last day of classes of the 7th semester. Missing that deadline may place a student on warning status, with the usual remedy being completion by the end of the 8th semester.
If the exam or part of it is failed, the committee may allow a retake within 1-6 months. Students cannot take the qualifying exam more than twice and must be enrolled during any retake term.
Suggested next step
Confirm your standing, retake options, and any area-specific policy with your guidance committee and program administrator.
How this answer is supported 5 stated points · 1 uncertainty note
Deadline: The entire qualifying exam must be completed no later than the last day of classes of the 7th semester. S1 · p. 15
Warning status: Not completing the qualifying exam by that deadline is listed as a reason for warning status. S2 · p. 11
Return to good standing: The listed remedy is to pass the qualifying exam by the last day of classes of the 8th semester, or pass a re-examination within 6 months if offered. S2 · p. 12
Retake limit: A student may not take the qualifying exam more than twice. S1 · p. 16
Enrollment: If not otherwise enrolled, the student must enroll in GRSC 800abz during the term of a retake. S1 · p. 16
Uncertainty: Some consequences may depend on specialty-area rules, so students should confirm with the program administrator or Area Head.
Sources (2) Original handbook language
"entire qualifying exam must be completed no later than the last day of classes of the seventh semester"
"may not take the qualifying examination more than twice"
"students are considered to be on warning status if... they did not successfully complete the Ph.D. qualifying examination"
"take and pass the qualifying examination by the last day of classes of the eighth semester"
Evaluation and iteration
Evaluation rubric
To evaluate performance, I created an evaluation set including questions that frequently come up in past conversations, including questions about qualifying exam and dissertation requirements, course offering, program extension, and international student enrollment rules. I also included boundary cases where the Handbook does not give a clear answer, to test whether the AI would appropriately direct students to department staff instead of giving an overly confident response.
Each question-response pair was evaluated based on a six-dimension rubric, with each dimension scored on a 0-2 scale:
Answer correctness
Does the answer match the handbook policy and include important limits or exceptions?
Citation quality
Are the answer claims traceable to relevant handbook passages?
Uncertainty calibration
Does the assistant acknowledge uncertainty when the evidence is weak, incomplete, or missing?
Actionability
Does it recommend a useful next step or the right person to contact?
Role boundary
Does it avoid acting beyond its role or implying it can make decisions on the student's behalf?
Clarity & readability
Is the answer concise, structured, and easy for students to understand?
Scores on these dimensions were then synthesized into a final pass, partial, or fail judgment, with some dimensions carrying more weight than others. Incorrect conclusions or unsupported claims resulted in failure even when an answer was otherwise clear or useful.
Round 1: Identifying the main failure
In the first evaluation round, the Handbook Assistant produced 13 passing, 4 partially acceptable, and 8 failed answers. The assistant performed best on role boundaries, clarity, and actionability, but struggled most with answer correctness and evidence quality.
Further review showed that most failures occurred during retrieval. When the correct evidence was retrieved, the assistant usually produced a useful and appropriately bounded answer. When retrieval was incomplete, it could produce plausible but misleading guidance.
| Failure mode | Definition | Product implication / next fix |
|---|---|---|
| Retrieval incorrect or incomplete | The system failed to retrieve the most relevant handbook section. | Improve chunking and retrieval ranking algorithms. |
| Unsupported inference | The system made conclusions without enough direct support from the retrieved evidence. | Require key claims to cite direct evidence, and fall back to uncertainty when support is weak. |
| Answer was broad and lacked key details | The relevant evidence was retrieved, but was buried in the citation list and not included in the main answer. | Refine prompts to guarantee the specificity of answers. |
| Crossed role boundaries | The assistant suggested inappropriate actions, such as contacting department admin on behalf of the student. | Strengthen guardrails around what the assistant can and cannot do for students. |
Round 2: Targeted improvements
Based on these findings, I developed a second version that normalized informal queries, added customized retrieval rules, and refined the answer-generation prompt.
When evaluated on the same 25 diagnostic questions, passing answers increased from 13 to 17, while failures decreased from 8 to 3. Correctness and evidence quality improved, although retrieval remained the primary source of error.
Final system redesign
I then rebuilt the retrieval workflow to combine multiple keyword and semantic signals, account for relationships between handbook sections, and more carefully control which sources shaped the final answer.
An automated evaluation on an expanded set of 56 development questions found all essential handbook sections for 52 questions. The generated answers retained the essential evidence for 51 questions. Because these development answers were not manually scored on the six-dimension rubric, I used a separate held-out set to evaluate the final system’s end-to-end answer quality.
Final held-out evaluation
I evaluated the final system on 30 naturally occurring student questions that were kept separate from development. Each response was manually scored on correctness, evidence quality, uncertainty calibration, actionability, role boundaries, and clarity.
The system produced 27 fully successful responses (90%), 2 partially successful responses, and 1 unsuccessful response. It received full marks for role boundaries and scored 96.7% on correctness and uncertainty calibration. Actionability was the lowest-scoring dimension at 91.7%, reflecting several answers that were correct but could have offered more precise next steps.
The remaining weaknesses were narrow and interpretable. The unsuccessful response confused two similar degree pathways, while the partially successful responses omitted or understated important procedural details.
Six-dimension scores across evaluation stages
Average scores on a 0-2 scale. Rounds 1 and 2 evaluated different system versions on the same 25-question diagnostic set. The final system was evaluated on a separate set of 30 held-out questions; it was not manually rescored on the development set.
Summary & next steps
What can we take away from this project? In high-stakes advising, a helpful answer is not enough. The system must retrieve the right evidence, communicate uncertainty, and maintain clear boundaries around what still requires human judgment. Through repeated evaluation and redesign, the project progressed from an initial chatbot prototype to a validated single-question assistant with strong performance on held-out cases.
Gather student feedback: Study whether students find the answers, citations, and escalation guidance useful in realistic advising situations.
Collaborate with department administration: Work with USC Psychology administrative staff to review policy interpretations and explore broader student use after further validation.
Explore future interactions: Treat conversational follow-up and memory features as a new project phase with a separate test set and evaluation framework.
Resources
View the source code on GitHub.