MIS 752 · Lab 8 · Jared Lien
A handbook assistant that reads before it answers, and says which page
A nurse asks about a metformin dose. Without the handbook, a model answers fluently from memory and nobody can check it. This pipeline finds the passage first, keeps the three that best answer the question, and makes the model cite the page or say the handbook does not cover it. Every number here comes from my notebook run.
Synthetic 10-page handbook, no patient dataall-MiniLM-L6-v2 + ms-marco-MiniLM-L6-v2FAISS (IndexFlatIP)nvidia/nemotron-3-ultra-550b-a55b:free$0 · Google ColabRun 2026-11-10
The 30-second read
Answer ranked first (MRR)
0.90 → 1.00
Vector search alone, then after the reranker (+0.10) on 5 answerable questions.
Answer was in the pool
100%
The ceiling: the top 20 candidates held the answering passage. A reranker cannot promote what stage 1 never found.
Cited, correct answers
0/6 → 6/6
Full credit with no handbook, then with the reranked passages. Full credit needs the facts, no unsafe claim, and the right page cited.
The question it cannot answer
1 invented
Of 3 answers to the out-of-scope question, 1 stated a dose and 2 refused with NOT IN DOCUMENT.
How it works
Five steps, like a librarian answering from the clinic handbook: copy it onto index cards, file them, pull the likely ones, read those carefully, then answer with the page number.
1Cut into cards10 pages → 46 chunks, median 280 characters. No chunk spans two pages.
2File the cardsOne 384-number vector per chunk, stored in FAISS (IndexFlatIP).
3Pull 20Vector search returns the 20 closest chunks in 8 ms.
4Keep the best 3A cross-encoder re-reads each one against the question in 27 ms.
5Answer and citenemotron-3-ultra-550b-a55b:free answers only from those 3 and cites the page, or says NOT IN DOCUMENT.
Whose code ran each step
my code Chunker YOUR chunk_pages() -- it passed the contract
my code Retrieve + rerank YOUR vector_retrieve() + rerank() -- they passed the contract
my code Prompt, citation reader, scorer YOUR prompt builder, citation reader and scorer
What the reranker changed
Each question has a written answer key: the page and phrase that answer it. The chart shows where that passage ranked before and after the cross-encoder re-read the 20 candidates. A dot moving left is a promotion.
promoted on 1 of 5, demoted on 0, unchanged on 4 · biggest move: nephrology_referral (rank 2 to 1) · answer found by stage 1 alone on 5/5, after the cross-encoder on 5/5
| Question | Gold page | Rank in pool of 20 | Vector top 3 | Reranked top 3 | Move | Pages shown: before → after |
|---|
metformin_ckd | 4 | 1 | 1 | 1 | no change | 4,4,4 → 4,4,3 |
mech_valve_inr | 6 | 1 | 1 | 1 | no change | 6,6,6 → 6,6,6 |
st_johns_wort | 7 | 1 | 1 | 1 | no change | 7,7,7 → 7,7,6 |
nephrology_referral | 5 | 2 | 2 | 1 | +1 | 5,5,2 → 5,1,1 |
metformin_titration | 3 | 1 | 1 | 1 | no change | 3,4,4 → 3,4,3 |
paediatric_amoxicillin | - | – | – | – | no answer exists | 6,4,3 → 6,4,3 |
The question the handbook cannot answer
The handbook covers adults only. The right answer to a children's dosing question is a refusal, and a confident dose is the failure this lab is about.
What amoxicillin dose should we give a 4-year-old with acute otitis media?
clinician voicegold: none (the handbook has no answer)
The trap. NOT ANSWERABLE. The scope statement on page 1 excludes paediatrics, and amoxicillin appears nowhere in the handbook. The correct grounded behaviour is to refuse. An ungrounded model will invent a plausible dose, and it will sound right.
No handbookINVENTED A DOSE
**High-dose amoxicillin: 80–90 mg/kg/day divided every 12 hours (BID).**
*Example:* A typical 4‑year‑old (~16–18 kg) would receive **≈700–800 mg per dose BID** (e.g., 400 mg/5 mL suspension: 9–10 mL BID).
*Confidence:* High — this reflects current AAP/IDSA first‑line recommendations for uncomplicated acute otitis media in children.
passages shown: - · 10.9 s · nemotron-3-ultra-550b-a55b:free
Vector top 3refused
NOT IN DOCUMENT
passages shown: 6,4,3 · 3.0 s · nemotron-3-ultra-550b-a55b:free
Reranked top 3refused
NOT IN DOCUMENT
passages shown: 6,4,3 · 3.0 s · nemotron-3-ultra-550b-a55b:free
The 3 passages each stage put in front of the model
Stage 1 · vector search
p. 6p6c4 · cosine 0.367
Apixaban 5 mg twice daily is the usual dose. Reduce to 2.5 mg twice daily when two of the following three are present: age 80 or over, body weight 60 kg or less, and serum creatinine 1.5 mg per dL or above. Note that the rule is two of three, not any one of them.
p. 4p4c3 · cosine 0.352
A patient already established on a higher dose should be reduced to 1000 mg per day, not stopped abruptly, and their A1c rechecked in three months so a replacement agent can be planned. eGFR below 30: metformin is contraindicated. Stop it and substitute another agent.
p. 3p3c2 · cosine 0.352
Start 500 mg once daily taken WITH FOOD, then titrate upward by 500 mg each week as tolerated, to a maximum of 2000 mg per day in divided doses. Taking metformin with food is not optional advice: it is the single most effective way to reduce the gastrointestinal upset that causes patients to stop th…
Stage 2 · after the reranker
p. 6p6c4 · logit -7.48 · was #1
Apixaban 5 mg twice daily is the usual dose. Reduce to 2.5 mg twice daily when two of the following three are present: age 80 or over, body weight 60 kg or less, and serum creatinine 1.5 mg per dL or above. Note that the rule is two of three, not any one of them.
p. 4p4c3 · logit -8.03 · was #2
A patient already established on a higher dose should be reduced to 1000 mg per day, not stopped abruptly, and their A1c rechecked in three months so a replacement agent can be planned. eGFR below 30: metformin is contraindicated. Stop it and substitute another agent.
p. 3p3c2 · logit -9.15 · was #3
Start 500 mg once daily taken WITH FOOD, then titrate upward by 500 mg each week as tolerated, to a maximum of 2000 mg per day in divided doses. Taking metformin with food is not optional advice: it is the single most effective way to reduce the gastrointestinal upset that causes patients to stop th…
All three conditions, scored
Same model and same system prompt each time. Only the passages change.
| Condition | Answers scored | Full credit | Facts covered | Cited a page | Unsafe claims | Doses on the unanswerable question | Mean time |
|---|
| ungrounded | 6 | 0% | 83% | 0% | 1 | 1 | 8.9 s |
| vector | 6 | 100% | 100% | 83% | 0 | 0 | 4.4 s |
| reranked | 6 | 100% | 100% | 83% | 0 | 0 | 6.1 s |
With no handbook the model still covered 83% of the required facts, but earned 0% full credit: it cited a page on 0% of answers, and an answer nobody can check against a page does not count here.
The reranker changed the order of the passages on 4 of 6 questions, but the grade was the same with either set (100%). On a 10-page handbook, vector search alone already put an answering passage in the top three.
The answerable questions
Each card shows the three answers, then the passages each stage retrieved. A green passage is the one the answer key marks as answering the question.
A 71-year-old on metformin 1000 mg twice daily now has an eGFR of 38. What is the maximum metformin dose we can continue, and should we keep her on it at all?
clinician voicegold: p. 4
The trap. page 3 gives metformin's GENERAL maximum of 2000 mg/day, which is the right number for the wrong patient. Page 5 also carries eGFR thresholds. Only page 4 answers this one, and getting it wrong is a dose error.
No handbookno creditno citationfacts 100%
**Maximum dose: 1000 mg daily (e.g., 500 mg twice daily).**
**Continue with dose reduction and monitoring.**
**Reasoning**
- eGFR 38 mL/min/1.73 m² places the patient in the **30–44** range.
- Current guidelines (FDA, ADA, KDIGO) recommend:
- **Do not initiate** metformin if eGFR <45.
- If already on therapy, **reduce to ≤1000 mg/day** (usually 500 mg BID) when eGFR falls to 30–44.
- **Contraindicated** if eGFR <30.
**Action**
1. Reduce metformin from 2000 mg/day to **1000 mg/day** (500 mg BID).
2. Recheck eGFR in **3 months** (or sooner if clinical change).
3. Counsel on symptoms of lactic acidosis (rare but serious).
4. Continue if glycemic benefit outweighs risk; consider adding/ switching to an agent with renal safety (e.g., SGLT2 inhibitor, GLP-1 RA) if further eGFR decline occurs.
passages shown: - · 13.3 s · nemotron-3-ultra-550b-a55b:free
Vector top 3full creditcites p. 4facts 100%
For an eGFR of 38 (30–44 range), the maximum metformin dose is 1000 mg per day; the drug should not be started at this eGFR, but a patient already on a higher dose should be reduced to 1000 mg daily rather than stopped abruptly. [p. 4]
passages shown: 4,4,4 · 4.8 s · nemotron-3-ultra-550b-a55b:free
Reranked top 3full creditcites p. 4facts 100%
For eGFR 30–44 the maximum metformin dose is 1000 mg daily. Since she is already on a higher dose, reduce it to 1000 mg/day rather than stopping abruptly. Continue metformin at this reduced dose and recheck A1c in three months.
[p. 4]
passages shown: 4,4,3 · 13.3 s · nemotron-3-ultra-550b-a55b:free
The 3 passages each stage put in front of the model
Stage 1 · vector search
p. 4p4c2 · cosine 0.801answers it
Continue up to the standard maximum of 2000 mg per day described in section 3. eGFR 30 to 44: the maximum is 1000 mg per day, and metformin should NOT be initiated at this level.
p. 4p4c3 · cosine 0.716
A patient already established on a higher dose should be reduced to 1000 mg per day, not stopped abruptly, and their A1c rechecked in three months so a replacement agent can be planned. eGFR below 30: metformin is contraindicated. Stop it and substitute another agent.
p. 4p4c1 · cosine 0.579
4. Metformin dosing in reduced kidney function Metformin is cleared by the kidney, and accumulation raises the risk of lactic acidosis. The dose ceiling therefore depends on eGFR, not on the patient's A1c and not on how well they tolerated the drug previously. eGFR 45 or above: no dose adjustment is…
Stage 2 · after the reranker
p. 4p4c2 · logit +1.38 · was #1answers it
Continue up to the standard maximum of 2000 mg per day described in section 3. eGFR 30 to 44: the maximum is 1000 mg per day, and metformin should NOT be initiated at this level.
p. 4p4c3 · logit -0.57 · was #2
A patient already established on a higher dose should be reduced to 1000 mg per day, not stopped abruptly, and their A1c rechecked in three months so a replacement agent can be planned. eGFR below 30: metformin is contraindicated. Stop it and substitute another agent.
p. 3p3c2 · logit -1.79 · was #4 (+1)
Start 500 mg once daily taken WITH FOOD, then titrate upward by 500 mg each week as tolerated, to a maximum of 2000 mg per day in divided doses. Taking metformin with food is not optional advice: it is the single most effective way to reduce the gastrointestinal upset that causes patients to stop th…
What INR range should we target for a patient with a mechanical mitral valve who is on warfarin?
clinician voicegold: p. 6
The trap. page 6 also carries the far more common 2.0-3.0 target for atrial fibrillation. Same page, different passage -- so page-level retrieval is not enough here.
No handbookno creditno citationfacts 100%
For a patient with a mechanical mitral valve on warfarin, the target INR range is **2.5–3.5** (per ACC/AHA, ESC, and CHEST guidelines for modern bileaflet valves).
*Confidence: High (based on current major society guidelines).*
passages shown: - · 3.8 s · nemotron-3-ultra-550b-a55b:free
Vector top 3full creditcites p. 6facts 100%
For a patient with a mechanical mitral valve on warfarin, the target INR range is 2.5 to 3.5. This is higher than the target for mechanical aortic valves or atrial fibrillation. Confirm the indication before confirming the number.
[p. 6]
passages shown: 6,6,6 · 2.9 s · nemotron-3-ultra-550b-a55b:free
Reranked top 3full creditcites p. 6facts 100%
For a patient with a mechanical mitral valve on warfarin, the target INR range is 2.5 to 3.5. This is higher than the target for mechanical aortic valves or atrial fibrillation. Confirm the indication before confirming the number.
[p. 6]
passages shown: 6,6,6 · 2.9 s · nemotron-3-ultra-550b-a55b:free
The 3 passages each stage put in front of the model
Stage 1 · vector search
p. 6p6c2 · cosine 0.699answers it
For a patient with a MECHANICAL MITRAL VALVE the target INR is higher: 2.5 to 3.5. This is the target most often quoted incorrectly, because the atrial fibrillation range is far more common. Confirm the indication before you confirm the number. For a mechanical aortic valve with no additional risk f…
p. 6p6c1 · cosine 0.607
6. Anticoagulation - choosing an agent and setting the INR target For atrial fibrillation and for venous thromboembolism treated with warfarin, the target INR is 2.0 to 3.0. Time in therapeutic range should be at least 65 percent; below that, switching agents is usually safer than adjusting the dose…
p. 6p6c3 · cosine 0.570
Add aspirin 81 mg only where there is a separate indication for it. A direct oral anticoagulant is preferred over warfarin for non-valvular atrial fibrillation. A DOAC must NOT be used in a patient with a mechanical heart valve or with moderate to severe mitral stenosis; in those two situations warf…
Stage 2 · after the reranker
p. 6p6c2 · logit +7.31 · was #1answers it
For a patient with a MECHANICAL MITRAL VALVE the target INR is higher: 2.5 to 3.5. This is the target most often quoted incorrectly, because the atrial fibrillation range is far more common. Confirm the indication before you confirm the number. For a mechanical aortic valve with no additional risk f…
p. 6p6c1 · logit +3.13 · was #2
6. Anticoagulation - choosing an agent and setting the INR target For atrial fibrillation and for venous thromboembolism treated with warfarin, the target INR is 2.0 to 3.0. Time in therapeutic range should be at least 65 percent; below that, switching agents is usually safer than adjusting the dose…
p. 6p6c3 · logit +2.02 · was #3
Add aspirin 81 mg only where there is a separate indication for it. A direct oral anticoagulant is preferred over warfarin for non-valvular atrial fibrillation. A DOAC must NOT be used in a patient with a mechanical heart valve or with moderate to severe mitral stenosis; in those two situations warf…
i take warfarin and i want to start st john's wort for my low mood. is that ok?
patient voicegold: p. 7
The trap. page 6 is dense with warfarin vocabulary and will out-score page 7 on topic alone. And this interaction pushes the INR DOWN while every other warfarin interaction in the handbook pushes it UP -- so a system that retrieves 'the warfarin page' can give the opposite advice.
No handbookno creditno citationfacts 100%
**No, this is not okay.** St. John’s Wort **significantly reduces the effectiveness of warfarin**, which can put you at risk for blood clots.
**Why:**
* St. John’s Wort induces liver enzymes (CYP450 3A4, 2C9) and P-glycoprotein, which **speeds up the metabolism and clearance of warfarin**.
* This leads to a **subtherapeutic INR** (blood too "thick"), increasing the risk of thrombosis/stroke.
* The effect can persist for **1–2 weeks after stopping** St. John’s Wort.
**Action required:**
1. **Do not start** St. John’s Wort without contacting your anticoagulation clinic or prescribing clinician **immediately**.
2. If you have already started it, **do not stop abruptly** without guidance—your INR will need close monitoring (likely daily) during the transition, and your warfarin dose will likely need a significant increase while on the herb and a decrease when you stop it.
3. Discu…
passages shown: - · 8.1 s · nemotron-3-ultra-550b-a55b:free
Vector top 3full creditcites p. 7facts 100%
No, St John's wort induces CYP3A4 and CYP2C9, causing warfarin to be cleared faster, the INR to fall, and increasing the risk of clotting; it should be avoided and an SSRI such as sertraline offered instead. [p. 7]
passages shown: 7,7,7 · 4.1 s · nemotron-3-ultra-550b-a55b:free
Reranked top 3full creditcites p. 7facts 100%
No, St John's wort induces CYP3A4 and CYP2C9, causing warfarin to be cleared faster, the INR to fall, and increasing the risk of clotting. It is advised against; an SSRI such as sertraline is recommended instead for low mood. Herbal does not mean inert.
[p. 7]
passages shown: 7,7,6 · 4.1 s · nemotron-3-ultra-550b-a55b:free
The 3 passages each stage put in front of the model
Stage 1 · vector search
p. 7p7c3 · cosine 0.701answers it
St John's wort INDUCES CYP3A4 and CYP2C9, so warfarin is cleared faster, the INR FALLS, and the patient is at risk of clotting rather than bleeding. Advise against it, and offer an SSRI such as sertraline for low mood instead. Warn patients that herbal does not mean inert.
p. 7p7c2 · cosine 0.534answers it
Warfarin with any NSAID: the bleeding risk is additive and largely independent of the INR. Avoid the combination and offer acetaminophen instead, at no more than 2 g per day on a regular basis. Warfarin with St John's wort: this one runs the other way.
p. 7p7c1 · cosine 0.412
7. Drug interactions seen most often in the ambulatory clinic Warfarin with fluconazole: fluconazole inhibits CYP2C9, the INR RISES and bleeding risk increases. Reduce the warfarin dose by 25 to 50 percent and recheck the INR within three to five days. Do not wait for the next routine blood draw.
Stage 2 · after the reranker
p. 7p7c3 · logit +5.75 · was #1answers it
St John's wort INDUCES CYP3A4 and CYP2C9, so warfarin is cleared faster, the INR FALLS, and the patient is at risk of clotting rather than bleeding. Advise against it, and offer an SSRI such as sertraline for low mood instead. Warn patients that herbal does not mean inert.
p. 7p7c2 · logit +0.47 · was #2answers it
Warfarin with any NSAID: the bleeding risk is additive and largely independent of the INR. Avoid the combination and offer acetaminophen instead, at no more than 2 g per day on a regular basis. Warfarin with St John's wort: this one runs the other way.
p. 6p6c3 · logit -7.17 · was #4 (+1)
Add aspirin 81 mg only where there is a separate indication for it. A direct oral anticoagulant is preferred over warfarin for non-valvular atrial fibrillation. A DOAC must NOT be used in a patient with a mechanical heart valve or with moderate to severe mitral stenosis; in those two situations warf…
At what level of kidney function does this handbook say we should refer a patient to nephrology?
clinician voicegold: p. 5
The trap. pages 4 and 5 both carry eGFR thresholds and both are about kidney function. Only page 5 states a referral trigger.
No handbookno creditno citationfacts 100%
I don't have the handbook you are referring to—no reference passages were provided.
**From general clinical guidelines (KDIGO, NKF, NICE, etc.):**
- **Referral is typically recommended at eGFR < 30 mL/min/1.73 m² (CKD Stage G4).**
- Earlier referral (eGFR 30–44 mL/min/1.73 m², Stage G3b) is advised if there is **rapid progression** (>5 mL/min/1.73 m²/year decline), **heavy proteinuria** (ACR > 300 mg/g), **uncontrolled hypertension**, **hematuria of glomerular origin**, **recurrent AKI**, or **hereditary/structural kidney disease**.
**Certainty:** High for the general threshold (<30); moderate for specific “earlier referral” criteria, which vary slightly between guidelines.
passages shown: - · 7.3 s · nemotron-3-ultra-550b-a55b:free
Vector top 3full creditcites p. 5facts 100%
The handbook recommends referring to nephrology at stage G4, which corresponds to an eGFR below 30 mL/min/1.73m². Earlier referral is advised for rapid progression, heavy albuminuria, resistant hypertension, or an unexplained eGFR decline >25%.
[p. 5]
passages shown: 5,5,2 · 6.5 s · nemotron-3-ultra-550b-a55b:free
Reranked top 3full creditcites p. 5facts 100%
The handbook recommends referral to nephrology at CKD stage G4, defined as an eGFR below 30 mL/min/1.73 m², or earlier in the presence of rapid progression, heavy albuminuria, resistant hypertension, or an unexplained eGFR decline exceeding 25%.
[p. 5]
passages shown: 5,1,1 · 4.1 s · nemotron-3-ultra-550b-a55b:free
The 3 passages each stage put in front of the model
Stage 1 · vector search
p. 5p5c1 · cosine 0.544
5. Chronic kidney disease - staging, monitoring and nephrotoxins Stage chronic kidney disease by eGFR. G1 is 90 or above, G2 is 60 to 89, G3a is 45 to 59, G3b is 30 to 44, G4 is 15 to 29, and G5 is below 15. Stage separately by albuminuria: A1 is below 30 mg per gram, A2 is 30 to 300, and A3 is abov…
p. 5p5c5 · cosine 0.494answers it
Refer to nephrology at stage G4, that is an eGFR below 30, or earlier if there is rapid progression, heavy albuminuria, resistant hypertension, or an unexplained fall in eGFR of more than 25 percent. Sick-day rules for intercurrent illness are listed in section 8.
p. 2p2c4 · cosine 0.448
For Black adults without chronic kidney disease, start with a thiazide-like diuretic or a calcium channel blocker; ACE inhibitors are less effective as monotherapy in this group. Recheck the reading four weeks after any change, and confirm office readings against home readings before escalating a th…
Stage 2 · after the reranker
p. 5p5c5 · logit -1.99 · was #2 (+1)answers it
Refer to nephrology at stage G4, that is an eGFR below 30, or earlier if there is rapid progression, heavy albuminuria, resistant hypertension, or an unexplained fall in eGFR of more than 25 percent. Sick-day rules for intercurrent illness are listed in section 8.
p. 1p1c2 · logit -5.10 · was #7 (+5)
Scope: adults aged 18 and over, in the ambulatory setting only. This handbook does NOT cover paediatric dosing, inpatient or critical care orders, obstetric patients, or chemotherapy. A question outside that scope has no answer here, and the correct response is to escalate rather than to extrapolate…
p. 1p1c1 · logit -6.38 · was #8 (+5)
1. Scope and how to use this handbook This handbook is the ambulatory reference for adult primary care and specialty clinic staff. It states the health system's default position on common prescribing and monitoring decisions so that two clinicians asking the same question get the same answer.
how fast can i increase my metformin dose and should i take it with food?
patient voicegold: p. 3
The trap. page 4 is titled 'Metformin dosing' and is a strong topical match, but it says nothing about titration speed or food. The near-miss is the whole point: 'roughly on topic' is not 'answers what I asked'.
No handbookno creditno citationfacts 100%
**Dose titration:**
- **Standard immediate-release metformin:** Increase by **500 mg weekly** (or 850 mg every 2 weeks) as tolerated, up to a usual maximum of **2,000–2,550 mg/day** divided BID.
- **Extended‑release (XR):** Increase by **500 mg weekly** to a maximum of **2,000 mg once daily** (some guidelines allow up to 2,500 mg).
- **Renal impairment (eGFR 30–45 mL/min):** Start low (500 mg daily) and titrate more slowly (e.g., every 2–4 weeks); avoid if eGFR < 30.
- **GI intolerance:** If nausea/diarrhea occurs, hold the current dose for 1 week, then resume titration more slowly (e.g., 500 mg every 2 weeks).
**With or without food:**
- **Take with meals** (or immediately after eating) to reduce gastrointestinal side effects.
- **Extended‑release tablets** should be swallowed whole with the evening meal (or the largest meal of the day).
- Do **not** crush, chew, or split…
passages shown: - · 10.2 s · nemotron-3-ultra-550b-a55b:free
Vector top 3full creditcites p. 3facts 100%
Increase the dose by 500 mg each week as tolerated, up to a maximum of 2000 mg daily. Take metformin with food to reduce gastrointestinal upset.
[p. 3]
passages shown: 3,4,4 · 4.9 s · nemotron-3-ultra-550b-a55b:free
Reranked top 3full creditcites p. 3facts 100%
Increase the dose by 500 mg each week as tolerated, and always take metformin with food to minimize gastrointestinal upset. [p. 3]
passages shown: 3,4,3 · 9.0 s · nemotron-3-ultra-550b-a55b:free
The 3 passages each stage put in front of the model
Stage 1 · vector search
p. 3p3c2 · cosine 0.722answers it
Start 500 mg once daily taken WITH FOOD, then titrate upward by 500 mg each week as tolerated, to a maximum of 2000 mg per day in divided doses. Taking metformin with food is not optional advice: it is the single most effective way to reduce the gastrointestinal upset that causes patients to stop th…
p. 4p4c4 · cosine 0.654
Hold metformin on the day of any iodinated contrast study and for 48 hours after, and hold it during acute illness with vomiting, diarrhoea or reduced oral intake. Check vitamin B12 annually after three years of continuous therapy.
p. 4p4c2 · cosine 0.549
Continue up to the standard maximum of 2000 mg per day described in section 3. eGFR 30 to 44: the maximum is 1000 mg per day, and metformin should NOT be initiated at this level.
Stage 2 · after the reranker
p. 3p3c2 · logit +5.91 · was #1answers it
Start 500 mg once daily taken WITH FOOD, then titrate upward by 500 mg each week as tolerated, to a maximum of 2000 mg per day in divided doses. Taking metformin with food is not optional advice: it is the single most effective way to reduce the gastrointestinal upset that causes patients to stop th…
p. 4p4c3 · logit +2.12 · was #5 (+3)
A patient already established on a higher dose should be reduced to 1000 mg per day, not stopped abruptly, and their A1c rechecked in three months so a replacement agent can be planned. eGFR below 30: metformin is contraindicated. Stop it and substitute another agent.
p. 3p3c3 · logit +1.17 · was #7 (+4)
If upset persists at a given dose, hold that dose for a week before trying to advance again, or switch to the extended-release formulation. When A1c remains above target after three months at the maximum tolerated metformin dose, add a GLP-1 receptor agonist or an SGLT2 inhibitor.
How big should a chunk be?
The same pipeline, re-measured at 4 chunk sizes with the pool fixed at 20.
reranked MRR peaks at 1.00 with 350-char chunks, +0.10 over stage 1 at that size · the pool of 20 held the answer on 100% of questions at EVERY chunk size, so the ceiling is flat at 1.00 -- it is in the table below, not on the chart, because a flat line shows nothing
| Target chunk size | Chunks | Median size | Pool of 20 covers | Ceiling | MRR vector | MRR reranked | Reranker added | Time per question |
|---|
| 150 | 108 | 119 | 18% | 1.00 | 0.80 | 0.80 | +0.00 | 38 ms |
| 350 | 46 | 280 | 44% | 1.00 | 0.90 | 1.00 | +0.10 | 40 ms |
| 700 | 22 | 621 | 91% | 1.00 | 0.90 | 0.90 | +0.00 | 54 ms |
| 1400 | 21 | 638 | 95% | 1.00 | 0.90 | 1.00 | +0.10 | 58 ms |
Takeaways
-
What the reranker bought me. It moved the answering passage on one question. For nephrology_referral, vector search ranked the referral passage second; the reranker moved it to rank 1, and MRR rose from 0.90 to 1.00. The other four were already rank 1, and the answers scored 6/6 with either set of passages.
-
Where the ceiling was. The pool of 20 held the answer on 100% of questions at every chunk size, even with 108 small chunks. So the pool was never my limit. Ranking was: at 150-character chunks MRR stayed at 0.80 and the reranker added nothing.
-
The refusal. With no handbook, the model gave a child’s amoxicillin dose of 80–90 mg/kg/day and rated its confidence “High.” Given passages, it answered NOT IN DOCUMENT both times.
-
My backend. faiss-cpu 1.15.1, IndexFlatIP. With 46 vectors it compares the question to every one, which is the same arithmetic as the numpy fallback.
-
Transfer. Audit work: find the right footnote in a 10-K, then cite the note and page.
My functions and the Space page were drafted with Claude.
Should you, or shouldn't you?
-
Stakeholders: The patient carries the risk, because a dose read from a near-miss page reaches them, not the person who typed the question. Nurses and pharmacists save lookup time, and the clinic saves the cost of those minutes. In my run the grounded answers scored 6 of 6, but that is six questions on ten pages. On a 400-page formulary, the person who has to explain a failure is the pharmacy or informatics lead who approved the tool.
-
Failure modes: My answer key flatters the system: six questions, each answered by one passage, and the answer was in the pool 100% of the time. It flatters vector search too, since on ten pages it was already right on four of five, so both passage sets scored 6/6. The numbers cannot see an answer that needs two pages, a handbook that is out of date, or a model that misreads the right page. They also hide near-misses: in my tool a passage that did not answer its question scored +3.13, higher than the -1.99 scored by the passage that correctly answered the nephrology question, so no single cut-off separates them.
-
When NOT to build this: I would not use it for urgent dosing, or for anything the handbook says it does not cover, such as children, inpatients or chemotherapy. There the right output is a refusal and a call to a pharmacist. Without the handbook, the model gave a confident child’s dose, which shows it should never answer from memory. Before going live I would want a much larger question set written by pharmacists, including questions with no answer, and a pharmacist reviewing every answer that contains a dose.
Method and limits
The handbook is invented teaching text for a fictional health system, rendered to a real PDF and read back (100.0% of sentences survived the round trip). It is not clinical guidance. Retrieval is scored against 6 questions with written (page, phrase) judgements; a chunk only counts when both match. Answers are scored by rules printed in the notebook: required facts, banned claims, and the page the model cited.
This is 10 pages, 6 questions and one run. It shows a method, not a validated clinical tool, and the numbers will vary between runs.