Bigger isn't always better when it comes to running AI models on your own computer. One developer discovered that switching from a 20-billion-parameter model to a 9-billion-parameter one actually improved their results, not because the smaller model was inherently superior, but because it was architected differently to handle longer conversations and complex tasks more efficiently .

What Is Context Window and Why Does It Matter More Than You Think?

When most people pick a local large language model (LLM), a tool that generates human-like text, they focus on parameter count,the number of individual settings the model adjusts during processing. The assumption is straightforward: more parameters equal better performance. But this overlooks a critical factor called context window, which is essentially your model's working memory .

Context window determines how much information the model can hold in mind at once. Everything in your conversation, your initial prompt, and the model's response must fit within this limit. A model with 20 billion parameters but a tiny context window will struggle with anything longer than a few paragraphs, while a smaller model with a massive context window can handle lengthy documents, research papers, or extended conversations without losing the thread .

The developer's experience illustrates this perfectly. Running OpenAI's gpt-oss 20B model through LM Studio, a free software tool that lets you run AI models locally on your computer, worked fine for quick questions and brainstorming. But when tasked with creating a longer UX design curriculum, the model kept hitting its context limits and couldn't generate complete responses .

How Architecture Changes Everything: The Qwen Switch

The breakthrough came when a colleague recommended the Qwen family of models, specifically because of their different internal architecture. Instead of using a standard transformer design, Qwen uses something called Gated DeltaNet (GDN), a hybrid approach that handles context fundamentally differently .

Standard transformer models, the dominant architecture in AI, create what's called a key-value cache for every token (roughly one word) in your context. The longer your conversation, the more memory this requires. GDN replaces most of these layers with a fixed-size memory state, meaning memory usage stays relatively constant even as context length grows dramatically .

The practical result: the developer switched to Qwen 3.5 9B, which is roughly half the size of the previous 20B model but supports up to 262,000 tokens of context, compared to the 128,000-token limit of the larger model. More importantly, Qwen could handle 60,000 tokens of context on the developer's modest setup,an 8GB graphics processing unit (GPU),without running out of memory, whereas the larger model struggled to manage even 30,000 tokens .

Settings Matter More Than You'd Expect

The initial switch to Qwen didn't go smoothly. The developer's first attempt to generate the UX curriculum still failed, and the instinct was to blame the model itself. But investigating LM Studio's settings revealed the real culprit: a "Limit Response Length" setting was capped at 1,643 tokens, cutting off responses mid-generation regardless of the model's actual capabilities .

This discovery highlighted a broader lesson for anyone running local AI models. Much of what feels like a hardware limitation or model weakness is actually a configuration issue. The developer had adjusted these settings days earlier while working on a different task and simply forgotten about it .

Beyond that initial fix, fine-tuning other parameters made a significant difference in Qwen's output quality:

• Thinking Mode: Qwen defaults to reasoning through problems before answering, which burns through tokens. Disabling this freed up token budget for actual responses.
• Presence and Repetition Penalties: Increasing these nudged the model toward more concise, less repetitive answers.
• Min-P Setting: Keeping this low helped the model avoid overthinking and over-explaining, a tendency Qwen has even with Thinking disabled.
• System Prompts: Instructing the model to be concise and skip unnecessary preamble significantly improved practical usefulness.
• Temperature: Lower values for precision-focused tasks, higher for general creative work.

How to Optimize Your Local LLM Setup for Longer Tasks

If you're running AI models locally and hitting limitations, here's a practical approach to troubleshooting and improvement:

• Audit Your Settings First: Before assuming your hardware or model is the problem, check LM Studio's configuration. Look for response length limits, context window settings, and parameter adjustments that might be artificially constraining performance.
• Match Architecture to Your Workflow: If your typical use involves long documents, research, or extended conversations, prioritize models with larger context windows and efficient architectures like GDN over raw parameter count.
• Test with Realistic Tasks: Run your model through the kinds of prompts you actually use. A model that scores well on benchmarks might not suit your specific workflow, so real-world testing matters more than published rankings.
• Use the Needle-in-a-Haystack Test: To verify your context window actually works as advertised, hide specific information in a large block of text and ask the model to retrieve it. This confirms the model is genuinely attending to content across the full context length.
• Adjust Parameters Incrementally: Small tweaks to presence penalties, repetition penalties, and temperature can dramatically improve output quality without requiring a model swap.

Real-World Performance: What Changed?

With context length set to 30,000 tokens, Thinking disabled, and parameters tuned, Qwen generated a comprehensive and actionable UX study guide that was significantly more practical than anything the larger model produced. When the developer pushed context to 60,000 tokens and ran a needle-in-a-haystack test, hiding key phrases in roughly 50,000 tokens of text, Qwen successfully retrieved the information, confirming the context window was genuinely functional across the entire range .

The real-world impact became apparent in extended workflows. The developer now comfortably runs 60,000-token sessions for UX research, design queries, and study sessions where context accumulates naturally through back-and-forth conversation. This represents a dramatic improvement over the previous setup, which would lose coherence or hit memory limits in similar scenarios .

At 60,000 tokens, the setup uses 7.6 out of 8GB of dedicated GPU memory, leaving minimal headroom but proving that a 9-billion-parameter model with efficient architecture can outperform a 20-billion-parameter model on modest hardware when the task demands extended context .

Why Reputation Can Lead You Astray

The developer initially avoided Qwen models because they appeared primarily in discussions about coding benchmarks. Since coding wasn't part of their workflow, they dismissed the entire family as a specialized developer tool. This assumption cost them months of suboptimal performance .

The lesson applies broadly: a model's reputation in one domain doesn't determine its usefulness in another. Qwen's strong performance on coding benchmarks doesn't diminish its capabilities for research, writing, design work, or any task requiring extended context and nuanced reasoning. Evaluating models based on your actual use case, not their marketing or benchmark reputation, often yields better results than chasing the highest parameter count .