Open Source Knowledge Graph Tools for Web Data

Knowledge graphs have stopped being a research curiosity and turned into a practical infrastructure choice for teams building AI pipelines on web data. The field spent a stretch hyped on retrieval-augmented generation, then went through a round of disillusionment with it, and 2026 is shaping up as the year things settle into something more architectural: structured knowledge paired with agentic loops that actually reason over it.

The problem with flat vector retrieval was never really about model quality. A model fed similar-sounding paragraphs still has to guess at how the entities in them relate to each other, and that guesswork is the actual structural cause of multi-hop hallucinations. Ask a flat RAG system how a corporate acquisition affected a regulatory decision three steps downstream, and it reconstructs a chain of custody from word similarity alone. A knowledge graph skips the guessing. The relationships are already there as edges, so the system traverses a path instead of inferring one.

That accuracy gain is conditional, though, not automatic. A system that retrieves well still needs to know when to change tactics.

Not everyone buys the graph-first case, and the strongest pushback deserves airtime. A May 2026 SSRN survey of Agentic GraphRAG raises the possibility that sufficiently capable reasoning agents can resolve multi-hop relationships on the fly from flat retrieval alone, which would make building and maintaining a graph an expensive bit of overhead. That debate hasn't been settled by any empirical knockout punch. But the counterargument holds up under scrutiny: an agent reasoning over flat retrieval is only as good as what it retrieves, and graphs shrink the area where retrieval can go wrong. That matters most with web content, where entity relationships don't sit still. Companies merge, people change jobs, prices move, and a graph that encodes those relationships explicitly ages more gracefully than a pile of embeddings hoping the next query lands near the right chunk.

Web data is, in fact, close to the best possible argument for graphs. It's dense with entities, thick with relationships between them, and constantly shifting, which is exactly the combination that flat retrieval handles worst. HydraDB's 2026 guide to knowledge graph tools observes that GraphRAG capability is no longer a feature that sets one platform apart from another, since more and more platforms are bundling graph traversal, vector search, keyword matching, and reranking into one package. The capability is becoming table stakes.

Each layer has its own mature open source options in 2026, and the rest of this piece walks through all three.

What building from web data requires before any graph tool enters the picture

Before any graph database or extraction framework enters the picture, the pipeline needs clean input, and this is where most projects lose time they didn't budget for. A knowledge graph is only as good as what feeds it, and web data doesn't arrive in a shape anything downstream wants to touch.

Raw HTML is close to the worst possible input format for an AI pipeline. It's full of markup the model has to burn tokens parsing just to throw away, and that noise drags extraction quality down with it. The formats that actually work: Markdown for reading and structure-preserving tasks, because headings and lists help LLMs parse hierarchy while staying token-efficient, JSON for structured fields, and plain text as a fallback. None of this is exotic. It's closer to basic hygiene, the kind of thing a kitchen does before cooking rather than a technique that makes the meal taste better.

Two open source tools have confirmed their place in the 2026 crawling layer. Scrapy is the Python framework most teams reach for to build scalable spiders that pull public web data, and it now integrates with LLMs well enough to run as something closer to an AI scraping agent than a static spider. ScrapeGraphAI takes a different angle: it supports prompt-to-schema extraction, and developers can plug in their own LLM through OpenAI, Groq, Azure, or Gemini, or run something local through Ollama.

Adoption of AI in this layer lags the hype around it. The 2026 State of Web Scraping report found that only about half of professionals currently use AI anywhere in their scraping workflows, and most of the rest are still running traditional approaches. Most of those who do use AI apply it to generating code rather than to extraction itself. That's a telling detail, because it points to where AI's actual value sits in this stack.

The economics explain the pattern. AI-driven extraction is cheap on any single page but the cost adds up fast at volume, so the pattern that actually works is to use an LLM once to generate deterministic scraper code, then run that code cheaply and quickly at scale, with an AI agent watching for site changes and regenerating the code when something breaks. That separates where AI earns its keep (writing code, noticing when a site layout changed) from where it shouldn't be doing expensive reasoning on every single page (the extraction loop itself, which stays deterministic, cheap, and fast). The AI is the mechanic who fixes the assembly line when it jams, while a worker standing at every station does the job by hand.

All of this matters more because machines are already the dominant audience for a lot of the web. Cloudflare Radar data shows that training-related crawling made up the overwhelming majority of AI web traffic in 2025, outpacing search-related crawling at its peak. The web being built for machines to read is already underway, and pipelines that still treat scraping as an afterthought are behind the traffic patterns, not ahead of them.

The graph storage layer: eight open source databases, and the three that can serve GraphRAG on their own

Once clean data is flowing, the next decision is where it lives, and most teams need a graph database that can handle semantic retrieval alongside graph traversal, but only three of the eight serious open source options do this without a second database bolted on.

The most recently confirmed comparison of this kind comes from ArcadeDB's July 2026 write-up, which checked all eight projects against their own repositories as of 30 July 2026. The full field: ArcadeDB, Neo4j Community Edition, Memgraph, FalkorDB, JanusGraph, TerminusDB, Cayley, and Kuzu. ArcadeDB is the one publishing this comparison and clearly has a stake in how it reads, and the write-up says so directly rather than pretending otherwise.

The dividing line that matters most for GraphRAG is whether a database stores vector embeddings in the same place as the graph itself. Without that, an application has to run a graph query, run a vector query somewhere else, and merge the two results by hand, which is extra plumbing nobody wants to maintain. ArcadeDB, Memgraph, and FalkorDB clear that bar on their own. Memgraph runs under BSL 1.1, and licenses like this matter a great deal to a company planning to embed the database inside a product it ships to customers, since source-available licenses carry restrictions that pure open source licenses don't.

The other five aren't bad databases, they just aren't one-stop GraphRAG shops. Neo4j Community Edition runs under GPLv3 and brings the largest ecosystem of the bunch, the deepest well of Stack Overflow answers, and the best odds that a new hire already knows how to use it. That's the actual argument for picking Neo4j: hiring and community, not some edge in raw GraphRAG throughput.

JanusGraph deserves a specific callout for a specific problem: graphs that have genuinely outgrown a single machine. Running JanusGraph over Cassandra is the honest answer there, since a database that lives on one node simply can't shard, no matter how good its query engine is. If the data truly needs horizontal scale, that's the lane.

For anyone working in RDF natively, Turtle files, N-Triples, SPARQL queries, the right tool is a dedicated triplestore, and ArcadeDB's own comparison says it isn't built for that job. RDF-native graphs can get genuinely large, and triplestores are built for exactly that shape of problem.

Two of the eight databases in the comparison are no longer actively maintained, and the write-up says so without hedging. Picking an unmaintained engine means inheriting every bug it currently has, forever, since nobody is going to fix them. Whatever short-term convenience an old, familiar tool offers has to be weighed against that risk. A database that's stopped shipping updates is a little like a car that's stopped getting parts made for it: it'll run fine right up until something breaks, and then it just won't.

The entity extraction layer: frameworks for turning crawled web pages into graph structure

With storage settled, the harder question is how raw, crawled pages actually turn into nodes and edges, and the framework chosen here decides how much of that work gets automated versus how much needs a human drawing the schema by hand. This is the layer where LLMs do their heaviest lifting in the whole pipeline.

Four open source frameworks have matured enough by 2026 to handle the document-to-graph step reliably. Neo4j's LLM Graph Builder takes in PDFs, web pages, YouTube videos, and other unstructured sources and turns them into graph structure directly. LangChain's LLMGraphTransformer does the same document-to-graph conversion with schemas flexible enough to adapt per project. LlamaIndex's PropertyGraphIndex handles schema-guided construction across multiple data modalities. Microsoft's GraphRAG specializes in community detection at scale, useful once a graph has grown large enough that finding clusters of related entities becomes its own problem.

The framework only does half the job. The model doing the actual extraction matters just as much, and open source options recommended for this task as of 2026 include DeepSeek-R1-0528, a large mixture-of-experts model with extended context and reinforcement-learning-trained reasoning; Qwen3-235B-A22B, another MoE model with extended context, dual thinking and non-thinking modes, strong tool integration for agents, and support for over 100 languages; and GLM-4.5, a large MoE model built specifically with agent applications in mind. Context length affects whether a model can reason across an entire document at once rather than losing relationship context between chunks, which matters for large crawled documents and multi-page collections. And when the extraction pipeline is itself agentic, tool integration and agent-native design in the model start to matter as much as its raw reasoning scores, which is exactly where Qwen3's tool handling and GLM-4.5's agent focus earn their keep.

Extraction itself usually isn't the hard part. Entity resolution is, and practitioners describe it consistently as the most under-resourced piece of the whole job. Two different web pages might refer to the same company by its full legal name, an abbreviation, a stock ticker, or a nickname, and someone has to teach the system that all of those point to one node in the graph. Industry veterans have a blunt word for this work: soul-crushing. It's unglamorous, it's manual more often than anyone would like, and skipping it produces a graph full of duplicate entities that quietly poison every query downstream.

LLM output also needs a gate before it's trusted in production. SHACL, the Shapes Constraint Language, is the standard tool for catching bad relationships before they get written into a live graph. Model improvements won't make this step optional, since hallucination and non-determinism are baked into how these models work, not a bug release away from disappearing. The architecture that actually holds up pairs an adaptable AI extraction layer with a rigid, deterministic validation layer sitting right behind it. Teams consistently underestimate how long this takes to get right: building out one domain reliably takes weeks, and an enterprise-wide ontology covering many domains takes months.

The frontier beyond all this is streaming extraction, still a direction rather than something to build on tomorrow. LiveGraph, presented at CAIS 2026, is a compound AI system for continuously updating a knowledge graph as new data streams in, built on a formal algebra of five atomic operators that each have an inverse. Every update can be rolled back and every error traced to the specific component that caused it. It was evaluated against meeting transcripts from the AMI corpus, geopolitical event data from ICEWS18, and Wikipedia editor interactions, and it hit perfect temporal coherence in testing (S_C = 1.0). That's a meaningful marker, because it means live-web graph construction, graphs updating in real time off a continuous crawl, finally has a reliably debuggable foundation under it.

The retrieval layer: how agents query the graph

A graph that no agent can query well is a very expensive filing cabinet, so the retrieval layer is where all the work upstream actually pays off. The orchestration frameworks dominant in 2026, LangChain, LangGraph, CrewAI, AutoGen, and LlamaIndex, all support graph traversal, state management, planning logic, and tool registries that let an LLM-driven plan call out to fetch code mid-reasoning. These aren't five competing ways to do the same narrow thing.

Model Context Protocol is turning into the wiring that connects graphs to agents across all of these frameworks. Open source projects like mcp-memory-service and context-portal already expose knowledge graphs directly through MCP servers. The Open Knowledge Graphs catalog, maintained by Steve Hedden, has its own community-built MCP server, and that catalog indexes 1,803 ontologies and vocabularies as of March 2026. An agent with access to that server can query the catalog of ontologies itself, not just a single graph someone built, which turns a reference resource into something closer to a queryable layer of the pipeline in its own right.

Not every retrieval task needs heavy infrastructure. For most read-only agent work, a full browser is overkill, and a scraping API with JS rendering and anti-bot bypass is sufficient. Save the heavier, action-taking tooling for tasks that actually need to click, fill forms, or navigate multi-step flows, rather than defaulting every agent to carry that weight around for jobs that only ever read.

Each layer now has open source options mature enough to run in production, which is the real shift 2026 represents: not a new idea, but the plumbing finally catching up to it.