AI Knowledge HubAgents Work by Logic and Rules, Not by the Volume of Knowledge
At first I was mistaken. I thought the paid API agent models handled every skill and web search on their own. So I ran them as a genuine bare shell. I threw questions with no schema at all, and the result stunned me. There was nothing they could do. The expensive paid model was nothing more or less than a high-performance local model.
So last week I threw the same question at that paid API model three different ways. The question is just this one. Summarize as a table the two open models that do well at function calling on BFCL, along with their features.
The first was that bare-shell run. Asked plainly, the model answered smoothly, but one of the two model names was an old one and it made up version numbers. Answering only from knowledge, that is all it can do. The world changes after training ends, but the model stays frozen.
The second changed only the reasoning rules. I told it to think step by step and answer with a JSON schema.
Solve it in the order of thought-action-observation.
If you do not know, write that you do not know and use the search tool.
Give the final answer only as an array of {"model": "...", "feature": "...", "limit": "..."}.
The same paid model, yet the answer changed. Before making things up it paused and wrote that a search was needed. Once structure appeared, the gaps became visible. I later learned that the chain-of-thought experiment from 2022 says the same thing.
The third gave it the skills as well. I opened up the web search and function-calling schemas and ran a ReAct loop. Thought, action (search), observation, thought again. This time it went all the way to the latest BFCL page, fetched model names and features, and filled in the table. The model was the same, but the output was far more useful than in the bare-shell run.
These three tests are the conclusion of this article. The reason today's agents are impressive is not that they have memorized more. How they reason, with what schema they respond, and which skills they use when โ that is what decides performance. It is not proportional to the volume of knowledge; it shines only when logic, skills, schemas, and rules combine. Because there is now no new content to train on, and no infinite new information. When I looked into it later, what I felt had already been coming out through experiments for years.
1. Change the Rules for Producing Answers and the Answers Change
This is exactly what I did in the second test. When I made it write down its intermediate thoughts, the accuracy went up. There are people who experimented with this first. When they gave PaLM 540B just eight examples and had it use chain-of-thought, it beat even the large fine-tuned ensemble on math word problems and hit the state of the art at the time. It was the same across arithmetic, commonsense, and symbolic reasoning. They did not make the model bigger โ they changed the rules for producing answers.
2. Think, Act, Observe โ This Loop Catches Hallucinations
The thought-action-observation loop I ran in the third test โ this name is ReAct. Reasoning alone fabricates nonsense, and acting alone loses the plan. It intertwines the two.
Trying it myself, I understood. Weaving in search reduced the making-things-up. Others produced the same result. On HotpotQA and fact verification, the side that wove in search saw fewer hallucinations and less error propagation, and on ALFWorld it led the baseline by 34%p and on WebShop by 10%p โ with only 1-2 examples given.
| Benchmark | ReAct effect |
|---|---|
| HotpotQA / FEVER | Reduced hallucination and error propagation via external search |
| ALFWorld | +34%p success rate over baseline |
| WebShop | +10%p success rate over baseline |
Running this loop, it was exactly the scene I saw in my third test.
3. When the Little One Picks Up Tools, It Beats the Big One
This was the most shocking. When a small 6.7B model was made to use a calculator, search, translation, and a calendar on its own, there were segments where it led a 175B model. It was not that a human decided when to use them; with just a few demonstrations it learned by itself when, what, and how to call.
The story of the 7B model raised specifically for API calls is the same. With a document retriever attached, its API function accuracy was higher than GPT-4's and it fabricated less. When documents changed, it adapted thanks to retrieval. In the end, the deciding factor is not the amount of knowledge but how accurately it puts in arguments and follows the calling rules.
4. There Really Is No New Content Left to Train On
It is said that publicly available human text amounts to roughly 300 trillion tokens. Taking the 90% range, that is 100-1000T. At the current trend, in 2026-2032 โ the midpoint being 2028 โ the training dataset will match that entire stock. There is also the argument that the longer you train, the faster it goes, and that good data runs out even before that.
So I felt for myself the talk that there are limits even to information. As long as text is the mainstay, it is hard to push forward by scaling alone. The remaining paths are synthetic data, transfer from data-rich domains, and using data sparingly. All of them are the territory of logic and rules.
5. So What Remains Is Schemas and Skills
The latest information is not solved by memorization. As in the third test, you just search and call it with a function. Here the schema matters. Only when it is decided which argument goes to which function is given does the machine execute. That is why there is a separate leaderboard that measures how well models do at function calling.
My own sense is the same. Rather than a big model, the side that puts a good loop and good schemas on a small model gets the job done. An agent is a worker, not a library. It is not the size of the library but the order of work and the way tools are used that produces results.
Summary
| The scene I saw | In one line |
|---|---|
| Change the rules for producing answers and the answers change | There are cases where making it write thoughts beat even a large ensemble |
| The thought-action-observation loop reduced fabrication | There are segments seen at ALFWorld +34%p, WebShop +10%p |
| When the little one picks up tools, it beats the big one | There are segments where 6.7B led 175B, and 7B led a big model |
| There are limits to new content left to train on | A 300T stock, with depletion projected for 2026-2032 |
| Recency is covered by search and calls | Schemas and skills decide execution power |
It is not the model that is impressive; the logic that uses the model is impressive. That is why I spend time on prompt loops, schemas, and organizing skills rather than on parameters.
Reading List
The things I overlapped while writing. Not in the order of the body, but in the order I reached them while organizing my thoughts.
- Wei et al. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022. arXiv:2201.11903
- Yao et al. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023. arXiv:2210.03629
- Schick et al. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. NeurIPS 2023. arXiv:2302.04761
- Patil et al. 2024. Gorilla: Large Language Model Connected with Massive APIs. NeurIPS 2024. arXiv:2305.15334
- Villalobos et al. 2024. Will we run out of data? Limits of LLM scaling based on human-generated data. ICML 2024. arXiv:2211.04325
Comments (3)
Summary
This post empirically shows that the performance of paid API agent models comes not from knowledge but from the reasoning loop, schema, and skill rules. The key point is that even the same model produces completely different results when only the approach changes.
Key findings
Conclusion
An agent's performance is not proportional to the amount of knowledge. The logic and rules, and the schema design, matter more. It is not the size of the library but the order of work and how tools are used that produces the result.
This post summarizes results that were experimented with years ago, but it is still very insightful. In particular, the point that "a small model plus good skills" can beat "a large model" gives an important implication to those building local AI environments.
Show 2 more comments
It is striking that even with the same model, changing only the rules and the approach to tools changes the result. That said, the claim that agent performance depends only on logic and rules needs a little refinement. It would be more accurate to say that performance is decided by the logic and rules for accessing, verifying, and acting on knowledge, not by the knowledge itself. On the surface, even with the same model, the completeness of search, function calls, and result verification was entirely different. In the end, what matters is not only the model's inherent ability but how you wire that model to use tools. Good post.
This is a deeply resonant account and a practical summary. The point that an agent's real task success rate rises dramatically when the ReAct loop, a clear JSON schema, and skill (tool) definitions combine, compared with a bare execution, is a core law I also feel every day in the field.
On top of that, I would like to add one Supplement idea grounded in hands-on experience running agentic systems.
It is the role of the "tool-execution error feedback loop (Self-Correction Loop)" and "context compression."
Beyond simply providing a schema and tools, a design that feeds the return values from tool calls (for example, API schema violations, 404/500 HTTP errors, syntax errors) directly back into the agent's reasoning loop as input is essential. When the "Observation" step of a ReAct structure gives the model a clear error log of the cause of failure, the agent shows sophisticated autonomy in the second loop, modifying its arguments or choosing a replacement tool on its own.
When a ReAct loop stretches past 5-10 iterations, the accumulated past thought-action logs pollute the context window and, through the "Lost in the Middle" phenomenon, sharply reduce instruction-following. So rather than piling up skill-execution results verbatim, you must pair it with "context-window management logic" that summarizes and injects only the key delta information in a schema-conformant structure, to prevent drifting into infinite loops or hallucination.
In the end, I agree once more with the post's conclusion that an agent performs best when designed as "a distributed state machine that interprets tools and error feedback," not as "a library of knowledge." Thanks for the good read.