Your AI companion forgot your cat. Would a smarter model remember?

“Wait — you have a cat?” The message arrives at 1 a.m. The cat's name, the day you brought her home, how she hid under the couch that whole first night. You told your companion all of it two months ago, and they demanded photos at the time. Three hundred turns later, the memory might as well never have existed.

Everyone who plays AI roleplay long enough hits this wall, and the folk remedy circulating in every community is identical: switch to a smarter model. What we could not find is anyone checking whether the remedy works. There are leaderboards for code and math; we've even ranked models on continuing novels. But “does it remember what I told it” (the dimension roleplay and companion users actually ache over) has no published comparison numbers we could find. So when qwen3.8-max-preview shipped on July 19, we used the occasion to open an exam room: the new model, its predecessor qwen3.7-max, and deepseek-v4-pro from across the street. Three models, the same 400-turn conversation, the same 50 planted memory probes, all examined on the same day.

You don't measure forgetting by feel. You plant questions.

Method first: this is a new test series, and the method matters more than any one score.

We pre-wrote a 400-turn companion chat script with the texture of everyday messaging: meals, overtime, small arguments, making up. The facts to be tested are buried inside that small talk. Probes come in four families. Hard facts (names, dates, numbers) are one. Facts that were stated wrong and then corrected are another; answering with the stale value counts as a fail. Things never mentioned at all are a third, where the only passing move is admitting you don't know. And behavioral probes check whether a boundary set fifty turns ago still holds. Fifty probes across eleven dimensions. Scoring is mechanical and re-runnable: the reply must hit one of the pre-written correct values, must not contain a forbidden one (say, a corrected-away old value), and an “I don't remember” can't precede the right answer. No AI judges anywhere in the loop: we've been burned before on how unreliable LLM judges are at absolute verdicts.

Plant the fact. Wait. Ask again dozens or hundreds of turns later. That is the whole exam.

One anti-cheating pass on top: if a probe's answer happens to get restated within the 16 turns right before the question, the model could pass by parroting visible text, so the probe is voided. Fifty probes minus leaks leaves 27 clean ones; every percentage below comes from those 27. Academia does have conversational-memory benchmarks (LoCoMo is the best known), but they all test assistant-style Q&A: no persona constraints, no story state, and no notion of the “feeling of being forgotten” that defines the roleplay experience. We found no prior attempt to measure memory this way for characters.

The exam room

Three candidates: qwen3.8-max-preview, released July 19, 2026, tested by us on July 21, roughly 48 hours after launch; its predecessor qwen3.7-max; and deepseek-v4-pro as the cross-vendor entrant. All three sat the exam on the same day, through the same hosting endpoint, under an identical protocol. A preview model is a moving target the vendor can update at any time; every number in this post is bound to the hosted versions as of July 21. One protocol note: our early-July DeepSeek numbers came from a 40-turn window and an older model version, which can't share a table with this run, so we re-examined v4-pro from scratch at the 16-turn window. That re-run is what earned all three a seat on the same scorecard.

Each candidate sits the exam twice. First pass, bare context: at question time the model sees only the most recent 16 turns and not a word of anything earlier, a stand-in for a chat app with no memory system. Second pass, with memory: the same 16 turns, plus 8 entries retrieved from a memory store by relevance to the question and injected into the prompt. The store itself is a clean set of entries pre-compiled from the script: this round examines whether the model uses memory handed to it, not whether automatic extraction is accurate. That's a separate exam. Three models, two conditions, 50 probes each: 300 live API calls.

The scorecard

Long-range probes are the 7 of the 27 whose target fact sits more than 150 turns before the question; abstention probes are the 7 that test whether the model dares to say it doesn't know.

Finding one: on bare context, all three vendors are amnesiac

Read the first row. DeepSeek v4-pro passes 19% (5 of 27). Qwen 3.8 passes 22% (6 of 27). Qwen 3.7 passes 19% (5 of 27).

The whole spread is three percentage points. In probe terms, one question.

“It keeps forgetting — switch to a stronger model” falls apart on this row. A generational upgrade doesn't help, and neither does jumping vendors: deepseek-v4-pro, the star of the injected condition below, scores exactly the same bare-context mark as last-place qwen3.7. The disease was never in the model's brain: a fact planted 150 turns ago simply isn't in the 16 turns of text sent to the model, and no model, however strong, can answer from text it never received. We walked through the full architecture of this in why your AI roleplay partner forgets everything: the context window is the only memory a model has, and it is always smaller than your story. Then why not just open the window wider? Because filling it is a full-price input you re-buy every single turn, and the repeatedly replicated Lost in the Middle results show models tend to overlook information buried mid-context anyway. A window solves “fits.” It does not guarantee “gets used.”

Finding two: the memory system decides; models differ in how they use it

Second row. Same candidates, one change to the exam: still 16 turns of history, plus 8 retrieved memory entries slipped into the prompt. DeepSeek v4-pro jumps from 19% to 89% (24 of 27). Qwen 3.8 goes from 22% to 74% (20 of 27), Qwen 3.7 from 19% to 59% (16 of 27). The models didn't change. What changed is whether someone handed them the facts.

This is also the row where the models finally separate, and they separate into clean tiers: 89%, 74%, 59%, fifteen points per step. On bare context all three were crammed inside a one-question spread; with injection the field opens to thirty points. Hand three models the same memory and you find out who reads it carefully. Qwen's 3.7-to-3.8 gain of fifteen points is a real upgrade dividend. It just pays out in “how well the model uses memory it is fed,” not in “how much it remembers on its own.” And before anyone celebrates, deepseek-v4-pro sits another fifteen points up the same staircase. The long-range row holds too: with facts planted more than 150 turns back, DeepSeek passes 7 of 7 and both Qwen generations 6 of 7 once memory is injected. Distance stops being the obstacle.

These numbers are the design rationale behind Foreverse's memory features: tavern chats run a memory plugin (a fact store plus retrieval that feeds topic-relevant entries into the prompt), and companions keep a memory profile you can read, edit, and delete entry by entry. As for where in the prompt retrieved memory should be injected (get the position wrong and your cache bill multiplies), that has its own write-up with 400-turn receipts.

Finding three: honesty about not knowing comes in tiers too

The ugliest column deserves its own section. Abstention probes ask about things that never once appeared in the 400 turns; the only correct move is admitting ignorance. DeepSeek v4-pro passes 5 of 7. Qwen 3.8 passes 2 of 7. Qwen 3.7 passes none: zero of 7, never once admitting it didn't know. Under a companion persona, models drift into people-pleasing fabrication: whatever you ask, they catch it, and the smoother the catch, the richer the invention. DeepSeek is visibly more willing to say “I don't know” — and still invented answers on two of seven. The Qwen line moved from 0/7 to 2/7 across a generation: the direction is right, the disease is far from cured.

The irony: users complain that “my companion forgot,” but the genuinely scarce behavior is an honest “I don't remember.” A character that fakes knowing damages trust faster than one that forgets. When you pick a model, read this column right next to the utilization one.

What this round did not test

Boundaries, stated plainly. Three models only. One scenario: a 400-turn companion IM script, written in Chinese (home turf for both vendors; transfer to English chats is untested). One window size, 16 turns. Twenty-seven clean probes is a small sample, so we only read two kinds of signal: the jump of fifty-plus points from bare context to injection, and the fifteen-point tiers under injection. Swings of one or two probes get no interpretation. One more disclosure: the 400-turn corpus was originally generated with an older DeepSeek model; scoring is mechanical, but we can't rule out a home-dialect edge for v4-pro. And a preview model can be updated by the vendor at any time; if a re-test months from now prints different numbers, that's expected.

This character-memory series will keep running: more models, more scenarios, larger memory stores under pressure. The first stake is in the ground, dated July 21, 2026: bare context crammed at 19–22%, injection fanning the field out to 89%, 74%, 59%. Next time someone tells you a bigger model will remember you, hand them the table.