64k context is nowhere near enough. I try to remember to compact Claude when I hit 200k but sometimes when I’m not paying attention I can hit 600k. Yeah I’ve written little Python scripts on my machine using qwen, but some of my work projects have 80k tokens of just documentation.
some of my work projects have 80k tokens of just documentation
I feel like there’s got to be better approaches than just shoving the entire documentation into the context. Humans don’t try to memorize the entire documentation. We just search for relevant pieces and remember those, or have them open in a webpage alongside the code. We probably only have like 10k context maximum.
Do the modern agent harnesses not have automatic compaction?
That sounds like a lot of work to just read some docs. RAG doesn’t require any training at all. I assume there are other approaches as well that would be faster than LoRA
64k context is nowhere near enough. I try to remember to compact Claude when I hit 200k but sometimes when I’m not paying attention I can hit 600k. Yeah I’ve written little Python scripts on my machine using qwen, but some of my work projects have 80k tokens of just documentation.
I feel like there’s got to be better approaches than just shoving the entire documentation into the context. Humans don’t try to memorize the entire documentation. We just search for relevant pieces and remember those, or have them open in a webpage alongside the code. We probably only have like 10k context maximum.
Do the modern agent harnesses not have automatic compaction?
The solution is to train a LoRA. It takes the base model, and overlays all your data on top.
That sounds like a lot of work to just read some docs. RAG doesn’t require any training at all. I assume there are other approaches as well that would be faster than LoRA