--- title: "Does Graphify Actually Help an AI Coding Agent?" description: "A code graph earns trust only when it beats disciplined source search on a correct, source-backed task." canonical_url: "https://artificialcuriositylabs.ai/posts/does-graphify-actually-help-an-ai-coding-agent/" md_url: "https://artificialcuriositylabs.ai/posts/does-graphify-actually-help-an-ai-coding-agent.md" published_at: "2026-07-24T00:00:00.000Z" tags: - "agents" - "context-engineering" - "patterns" - "build-log" --- The thesis: a codebase map earns its place only when it helps an agent reach a correct answer with lower measured cost. Graphify promises a local knowledge graph for code, configuration, and documentation. It turns files into nodes and relationships, then lets an agent query the graph instead of rediscovering the repository through search. That is a plausible mechanism. It is not evidence of a productivity gain. The first result is a measurement rule: do not accept the token-reduction claim, the graph visualization, or the feeling that exploration was easier as proof. Test the tool on the work it is meant to improve. ## The Result So Far - Against opening four relevant files in full, Graphify reduced retrieval context from 15,186 to 1,618 tokens: an 83% reduction. - Against disciplined search plus narrow source snippets, it used 1,618 versus 1,488 tokens: 9% more. - In a three-question freshness-aware run, the wrapper arm had a 20-second median versus 17.4 seconds for disciplined search. It fully covered two of three answer keys; the baseline covered all three. The wrapper removes command-syntax errors and can narrow exploration. It has not yet shown a speed, token, or correctness advantage over careful search and source verification. ## Why the Test Has to Be Local The published evidence on AI coding productivity points in opposite directions. In a randomized trial, [METR](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) found that experienced open-source contributors using early-2025 AI tools took 19% longer on their own tasks, even though they believed the tools made them faster. METR now labels that result historical and says it no longer reflects current tools. The durable lesson is not that AI slows people down. The durable lesson is that reported confidence and measured completion are different things. [Databricks reached the same practical conclusion from the other direction](https://www.databricks.com/blog/benchmarking-coding-agents-databricks-multi-million-line-codebase): evaluate against real tasks from the codebase, review the task and its answer by hand, and use held-out tests rather than an LLM judge that rewards an answer for sounding correct. Graphify's own [documentation](https://github.com/Graphify-Labs/graphify/blob/v8/README.md) makes a narrower claim. Its local parser extracts structure and relationships; its graph can answer scoped queries without reopening every source file. The right experiment tests that exact claim: repository understanding, impact analysis, and architectural navigation. ## The Pilot Use twelve read-only tasks drawn from one unfamiliar but representative repository. Six run without the graph. Six run with a fresh code-only graph. A second set of matched tasks reverses the order. This counterbalanced design prevents the first condition from winning because the operator learned the repository during the trial. The task set needs four categories: | Category | Example question | Required evidence | | ------------- | ---------------------------------------------------------- | ----------------------------------------------------------- | | Ownership | Which service enforces a budget or access rule? | Handler, shared policy code, and test | | Change impact | What changes when a new endpoint or tool backend is added? | Application code, configuration, infrastructure, and docs | | Architecture | Why are two components separate instead of merged? | Current code plus the decision record | | Operations | What is the approved deployment or recovery path? | CLI contract, infrastructure definition, and operator guide | Before the first run, a reviewer writes an answer key for every task. The key names the facts that must appear and the files that establish them. It also lists dependencies that are easy to miss. No tool sees the answer key. Each task has a fixed stopping rule: the participant submits a short answer with citations to local files, then states the change or decision they would make. The reviewer scores the answer before seeing whether Graphify was available. ## What to Measure Time matters, but it is not enough. A fast answer that misses the infrastructure change is worse than a slow answer that finds it. Record five fields for every task: 1. **Correctness:** Did the answer match the answer key? 2. **Evidence coverage:** Did it cite the source files and decision records that establish the answer? 3. **Dependency coverage:** Did it identify every required cross-layer effect? 4. **Elapsed time:** From task start to submitted answer. 5. **Exploration cost:** Raw files opened, search calls, and model usage when the harness exposes it. The reviewer also records unsupported assertions. This is the failure mode a map can create: a clean-looking path through the graph can make an inferred connection feel established. Graphify distinguishes extracted and inferred edges. The evaluation must preserve that distinction. ## The Decision Rule This is a pilot, not a claim about the category. Twelve tasks establish whether the next test is worth running. Keep Graphify optional for unfamiliar, cross-layer questions. Admit it to the default workflow only when it is at least as correct and has either a 20% lower median completion time or a 20% lower measured retrieval-token cost. If it saves tokens but produces more unsupported claims, it failed. Keep it out of the default workflow when the task already identifies the files to change. A graph has its highest chance of helping during onboarding, architecture review, incident investigation, and change-impact analysis. It has little reason to sit between a builder and a two-file fix. ## The Safety Boundary The first run is code-only and local. Exclude secrets, state, local environment files, generated bundles, dependency directories, and deployment output. Do not enable document or media enrichment, strict hooks, shared graph hosting, or a project-scoped agent instruction during the pilot. That leaves one variable under test: whether a local structural map improves the agent's understanding of the repository. ## Results The first run is complete. It is an exploratory four-task slice, not the twelve-task counterbalanced study described above. Two fresh agents each completed one baseline task and one Graphify-assisted task. The Graphify condition required graph-first discovery, then limited raw source verification. The baseline condition used normal repository search and raw source reads. No task changed code, contacted AWS, or enabled a persistent agent hook. | Condition | Tasks | Median time | Raw source files opened | Answer quality | Unsupported assertions | | --------- | --------------------------------------: | ----------: | ----------------------: | ------------------------------------- | ---------------------: | | Baseline | Budget enforcement; conditional routing | 17.5s | 6.5 average | Correct, source-cited, and bounded | 0 | | Graphify | Cost attribution; load-balanced routing | 16.5s | 4 average | Correct after raw-source verification | 0 | The graph itself took 7.1 seconds to build from 299 code files. It produced 4,005 nodes and 10,229 edges in a 12 MB local artifact. Focused `explain` lookups took 0.32 to 0.36 seconds and returned source locations plus extracted-versus-inferred relationship labels. The read: the graph condition opened fewer raw source files. It did not clear the adoption threshold. The observed one-second median difference is too small, and the tasks were not identical in difficulty. This unpaired slice cannot establish a speed or token benefit. The tool is an optional discovery layer for unfamiliar code paths, not a replacement for source verification. Two implementation limits surfaced: - Broad natural-language graph queries returned hundreds of nodes and were truncated. The useful pattern was a narrow query followed by `explain` on a discovered symbol. - The code-only safety mode omitted Markdown decision records and documentation. That makes it useful for code and infrastructure tracing, but insufficient for an architecture question whose authoritative answer lives in a decision record. ## Token Result The graph also needs a token test. Its value proposition is not that source code disappears. The value proposition is that an agent reads a structural map first, then opens only the evidence required to verify the answer. The same load-balancing question was measured three ways. The count uses the [cl100k_base tokenizer](https://github.com/openai/tiktoken) over the captured tool output and source text. It excludes the agent's system prompt, tool-call envelope, and generated answer, so it measures retrieval context rather than total model cost. | Retrieval pattern | Input tokens | Result | | -------------------------------------------------------------- | -----------: | -------------------------------- | | Open four relevant files in full | 15,186 | High context cost | | Search, then read narrow source snippets | 1,488 | Lowest cost | | Broad graph query, graph explanation, then source verification | 2,515 | 69% more than disciplined search | | Graph explanation, then minimal source verification | 1,618 | 9% more than disciplined search | The result is clear. Graphify reduced context by 83% against opening all four files in full. It did not save tokens against a careful search-and-snippet workflow on this focused question. That is not a failure of the graph. It defines the right use case. Use it when the agent does not know where a cross-layer path begins, when the repository is unfamiliar, or when the alternative is broad file reading. Do not force it into a two-file question where search already identifies the exact symbols. The expanded study needs to record retrieval tokens for every task, with the same boundary: graph output, raw verification text, and the final generated answer counted separately. A graph-first workflow earns a default only if it reduces total context without reducing evidence quality. ## Release Context Graphify is getting attention because it is new, visible, and moving quickly. The [public repository](https://api.github.com/repos/Graphify-Labs/graphify) was created on April 3, 2026, and the [PyPI package](https://pypi.org/pypi/graphifyy/json) first shipped the next day. By July 24, [GitHub release history](https://github.com/Graphify-Labs/graphify/releases) recorded 172 public releases, including 27 releases in the 27 days from June 28 through July 24. The adoption signals are real but need interpretation. [PyPI Stats](https://pypistats.org/api/packages/graphifyy/overall?mirrors=false) recorded 3.60 million package-download events through July 24, while the [project site](https://graphify.com/) reported 93,125 GitHub stars on July 20. Package downloads are not active users, and star counts do not establish production use. They establish attention. Independent technical guides appeared across April through July, including [Analytics Vidhya](https://www.analyticsvidhya.com/blog/2026/04/graphify-guide/), [Foss Engineer](https://fossengineer.com/graphify-code-knowledge-graph/), and [Better Stack](https://betterstack.com/community/guides/ai/ai-development/graphify-codebase/). The Hacker News response was smaller: the July Show HN post had two points and no comments, according to the [public HN search record](https://hn.algolia.com/api/v1/search?query=Graphify-Labs%2Fgraphify&tags=story). The read: there is genuine launch momentum around a project that is less than four months old. The release rate is also a maturity signal. Treat the current CLI as fast-moving developer tooling, not stable shared infrastructure. ## Freshness And Safety Test An isolated temporary corpus tested local extraction, ignored-file handling, and incremental rebuilds without a model backend. | Check | Result | | ----------------------------------- | ----------------------------------------------------------------------------------- | | Python and Terraform/HCL extraction | Passed: expected symbols and resources appeared | | Ignored secret-like file | Passed: it did not appear in the graph | | Unchanged re-extraction | Passed: zero files changed and the graph hash stayed identical | | Add and delete source files | Passed: the new symbol appeared and deleted-file nodes disappeared | | Code-only follow-up update | Mixed: `update --force` added Markdown nodes after the initial code-only extraction | The last row matters. A code-only initial extraction is the correct safety boundary. Do not assume every follow-up command preserves that boundary without an explicit test on the exact version in use. Keep graph output outside the repository and retain `GRAPHIFY_QUERY_LOG_DISABLE=1` until query logging is independently verified. ## Expanded Agent Run A second agent run was intended to broaden the task sample across identity and routing questions. It did not produce a publishable performance score. The experimental instruction described the graph command too loosely. Multiple agents first invoked an invalid command before reaching the correct `graphify query "" --graph ` form. That invalidates elapsed time as a comparison. The answers still required raw source verification, and the graph output remained broad enough to need narrowing. This is useful operational evidence. Graphify needs a precise local skill or wrapper that supplies valid query syntax and a narrow-query pattern. Without that layer, agent setup errors erase any small retrieval-time benefit. ## Freshness-Aware Wrapper Run The wrapper was then tested on the same repository revision with a fresh code-only graph, three matched questions, and an explicit first command: `graph-discover query "" --repo --budget 800`. It passed the integration test: all three graph-first runs used valid syntax, checked the freshness record, and verified their conclusions in raw source. The wrapper also kept graph artifacts outside the repository. The result does not show a speed win. The baseline used disciplined search and narrow source reads; the wrapper run used graph discovery followed by source verification. The timing below is agent-reported and one run per condition. Five runs only had whole-second resolution because the local `date` command did not support millisecond output. | Question | Baseline time | Wrapper time | Baseline files read | Wrapper files read | Answer coverage | | ---------------------------------- | ------------: | -----------: | ------------------: | -----------------: | ---------------------------------------------------------- | | Team-budget enforcement | 17.4s | 34s | 5 | 3 | Full in both arms | | OAuth login and credential storage | 18s | 18s | 8 | 3 | Full in both arms | | Dynamic-routing rendering | 14s | 20s | 4 | 4 | Full baseline; wrapper missed one deployment-boundary fact | | Median | 17.4s | 20s | 5 | 3 | Baseline 3/3 full; wrapper 2/3 full | The read: the wrapper fixes the invalid-command problem and can reduce source exploration. It did not beat disciplined search on elapsed time in this sample, and the routing answer shows why correctness remains the first gate. The wrapper found the routing renderer but missed that the active data-plane configuration is separately rendered and a per-team dynamic reload is still a follow-up. No retrieval-token trace was captured for this run. Fewer files opened is not a token-savings result. The earlier focused token test still stands: Graphify beat opening four full files, but it did not beat careful search plus narrow snippets. The next study needs captured graph output and source excerpts under the same tokenizer, then a blinded answer-key score before comparing medians. ## Adoption Decision Keep Graphify available only behind an explicit, freshness-aware wrapper for unfamiliar, cross-layer questions. Do not commit graph artifacts, enable a repository hook, or use semantic document extraction yet. The next run needs the full twelve-task counterbalanced design, matched task difficulty, and an independent reviewer who does not know the condition. Until then, the result is directional evidence about navigation cost, not a published productivity claim. ## The Open Question A codebase graph can reduce rediscovery. It can also become stale, omit an indirect dependency, or encourage an agent to trust a summary over the code. The pilot measures the first benefit. It does not settle the freshness problem. The question is not whether an agent can draw a map. It is whether the map improves a correct decision before the code changes underneath it.