Named Entity Recognition with spaCy for Web Data

Scraped web text is a mess. Nav bars, cookie banners, half-broken sentences from a JavaScript widget that didn't render right, and buried in all of it: names, companies, dates, dollar amounts that actually matter. Named Entity Recognition (NER) is the tool that pulls those out and labels them, and spaCy is the library most teams reach for when they need to do it at scale, not just on one clean article at a time.

This piece walks through how that actually works: what spaCy's NER component is doing under the hood, which model to pick, how to clean web content before it even sees the model, and what to do when the model runs into entities it was never trained to recognize. There's a joke in here somewhere about teaching a robot to read the internet's handwriting, but the honest version is just: it takes a pipeline, not a single function call.

How spaCy's NER component works under the hood

spaCy first showed up in February 2015, and it's sat at version 3.8.4 since January 14, 2025. It's built in Python and Cython, and unlike a lot of research-first NLP tools, it was built from day one to run in production. That means it's expected to chew through entire web dumps, not just handle a demo notebook.

The piece doing the actual entity-spotting is called the EntityRecognizer. It uses a transition-based algorithm, which is a fancy way of saying it reads through a sentence token by token and decides, step by step, where an entity span starts and stops, without letting spans overlap. Once it's done, the results land in doc.ents, and every individual token gets tagged with token.ent_type_ and token.ent_iob_ so you know exactly which label it belongs to and whether it's inside, at the start of, or outside an entity span.

Two things about how this works matter a lot once you start feeding it scraped web pages instead of tidy news copy.

First, the loss function it trains against cares about getting the whole entity span right, start to finish. If human annotators disagreed about where an entity's boundary actually falls (does "New York Times Co." end after "Co." or before it?), the model tends to perform worse on exactly those boundary cases. Second, the algorithm leans hard on the tokens near the start of an entity to make its decision. Entities where the identifying information sits in the middle, or gets split across a line break because some HTML div cut a sentence in half, are the ones it struggles with most.

For the standard pipeline, en_core_web_sm runs text through three stages in sequence: a tagger, a dependency parser, then the entity recognizer, with each stage handing its processed Doc object to the next. Out of the box, the entity types worth knowing are PERSON, ORG, GPE (geopolitical entities like countries and cities), DATE, MONEY, PRODUCT, EVENT, FAC (facilities), LAW, NORP (nationalities and political/religious groups), and WORK_OF_ART. Keep those two architectural quirks in mind, because they're exactly why the cleaning step later in this piece isn't optional.

Choosing the right English model for a web data pipeline

spaCy ships four trained English pipelines, and picking the wrong one is an easy way to burn either accuracy or your compute budget for no reason.

en_core_web_sm is the lightweight option. No word vectors, trained on the OntoNotes 5.0 dataset, and it scores an NER F1 of 0.845. That's a solid starting point if you're processing a high volume of pages and speed matters more than squeezing out the last few points of accuracy. On the other end sits en_core_web_trf, which fine-tunes a RoBERTa-base transformer model and is spaCy's highest-accuracy English pipeline. Think of sm as the reliable sedan and trf as the sports car: faster lap times, but it drinks more gas.

Speed is where the tradeoff actually bites. On CPU, spaCy's lower overhead gives it an edge over heavier transformer-based alternatives. And compared to calling an LLM for the same task, local models (spaCy included) run substantially faster, benchmarks have put the gap at 7 to 30 times, according to comparisons of spaCy against LLM-based NER approaches. That gap is the difference between processing a scraped dataset overnight and watching a progress bar for a week.

A rough decision guide:

• High-volume scraping, general entity types: sm or md.
• Precision-critical work, financial or legal documents: trf.
• Entities that don't exist in OntoNotes at all (drug names, SKUs, internal product codes): custom training, covered further down.

None of these models hit 100%. In production, NER is considered reliable but not perfect, around 80% reliability is cited as a practical ceiling, and that number matters most in exactly the domains where being wrong is expensive. For financial or legal pipelines, model output needs a human or a rules layer checking it before anything downstream trusts it as fact.

Fetching and cleaning web content before NER can run

spaCy is upfront that scraping falls outside its job. That's a feature, not a gap: keeping the scraping step separate from the NER step means you can swap out how you fetch pages without touching the model logic at all.

For small jobs, BeautifulSoup paired with the Requests library covers most single-page or small-batch extraction, and it's the combination most tutorials reach for when pairing scraping with NER on something like news articles. For anything at real scale, a dedicated crawling tool is the approach the community leans on. The usual workflow collects everything into a dataset first, then runs spaCy's NER over that dataset as a separate step.

"Cleaning," in this context, means a specific handful of things:

• Strip out HTML tags, nav menus, headers, footers, and cookie banners.
• Normalize whitespace and line breaks. This one matters more than it sounds like it should, because remember, spaCy's entity recognizer leans on tokens near the start of a span to make its call, and fragmented, line-broken text breaks that assumption before the model even gets a chance.
• Handle encoding issues (stray characters, mismatched charsets) before tokenizing anything.

At real scale, a web data API that hands back clean Markdown or structured text instead of raw HTML removes this whole cleaning burden before it starts. That lets the pipeline begin at the NER step instead of the HTML-parsing step, which is a meaningful head start when you're processing thousands of pages a day.

Whichever path gets you there, the output of this stage should just be a plain text string, or a list of them for batch jobs, ready to hand to nlp(text). At scale, fetching and cleaning web content is a systems problem before it's an NLP problem. Anti-bot measures, rate limits, dynamically loaded pages that show up blank to a basic scraper. A reliable scraping layer is what makes everything downstream of it actually deterministic.

Building the core spaCy NER pipeline step by step

Once the text is clean, the actual NER call is short.

Load the model:

import spacy
nlp = spacy.load("en_core_web_sm")  # or en_core_web_trf for higher accuracy

Run the cleaned text through it:

doc = nlp(cleaned_text)

That single line runs the full pipeline, tagger, parser, and entity recognizer, in sequence.

Then loop over the results:

for ent in doc.ents:
    print(ent.text, ent.label_, ent.start_char, ent.end_char)

Each entity carries its text, its label, and its character offsets in the original string. Calling spacy.explain(ent.label_) turns a cryptic label like "GPE" into a plain-English description, which is handy when you're building a label key for whoever's consuming the output downstream.

For anything feeding another system, serialize the results to JSON:

[{"text": ent.text, "label": ent.label_, "start": ent.start_char,
  "end": ent.end_char, "description": spacy.explain(ent.label_)}
 for ent in doc.ents]

Running this pattern with en_core_web_trf gets the highest entity accuracy of spaCy's available English pipelines.

During development, displaCy's ENT visualizer renders entities as color-coded HTML in your browser or notebook so you can eyeball whether the pipeline is actually catching what it should. displaCy is a development tool, full stop. Serving its raw HTML output to end users in production opens up a cross-site scripting risk, so keep it on your local machine or in a notebook, not in a web response.

Two concrete examples show what this looks like in practice. One: tracking company acquisitions out of news headlines. NER pulls the ORG entities, and pairing that with the dependency parser tells you which company is doing the acquiring and which one's getting bought, since word order alone doesn't guarantee direction. Point it at a live news feed and it runs continuously. Two: recipe ingredient extraction, where a custom NER model gets paired with regex and Pandas in a scrape-clean-extract pipeline (Python, BeautifulSoup, Requests) to pull structured ingredient lists off recipe websites that were never built with structured data in mind.

Enforcing deterministic extraction with EntityRuler and Matchers

Statistical models are probabilistic by nature, which is exactly the problem when you need the same input to produce the same output every single time. That's where spaCy's rule-based tools come in: EntityRuler, Matcher, and PhraseMatcher.

EntityRuler gets added to the pipeline with nlp.add_pipe("entity_ruler"), and you feed it patterns tied to labels. (A "factory" in spaCy terms is just the registered function spaCy uses internally to build and wire up a pipeline component. Not something you need to think about day to day, but it's the mechanism under add_pipe.)

Rule-based matching earns its keep on exactly the stuff a model trained on OntoNotes news text will never reliably catch: brand names, product SKUs, regulatory codes, proprietary identifiers, specific URL patterns. A statistical model might guess. A rule either matches or it doesn't, and that determinism is exactly what you want when entities are about to get written into a database or trigger some downstream action, like flagging a compliance review.

EntityRuler can run before the statistical NER component or after it. Running it first lets rule-based matches feed into what the statistical model sees. Running it after lets the rules override or add to whatever the model already predicted. Neither is universally right, it depends on whether you trust your rules more than the model for the entities in question.

Matcher and PhraseMatcher work at the token and phrase level, and they're the right tool when you've got a known, fixed list to enforce, say, a list of competitor names pulled from a market intelligence feed that absolutely must get flagged every time they show up, model confidence be damned.

The pattern that tends to work best in practice: use statistical NER for the general stuff (PERSON, ORG, DATE), then layer EntityRuler on top for the domain-specific tokens. That combination gets you flexibility and precision without the cost of training a custom model from the ground up.

When and how to train a custom NER model on domain-specific web data

Sometimes the general-purpose model just doesn't have the vocabulary. Point en_core_web_trf, spaCy's most accurate transformer pipeline, at a clinical journal article, and it comes back with essentially nothing useful. Medical entity types were never part of its training data, so there's nothing for it to recognize.

The same pattern shows up repeatedly: even when the source text is well-written prose, a model with no exposure to a domain's specific vocabulary is going to miss things a domain-trained model would catch without breaking a sweat.

Training a custom model means annotating examples and serializing them using spaCy's DocBin class, which beats a plain pickle file on two counts: it's spaCy's recommended format for sharing and loading annotated training data efficiently.

Building a model completely from scratch with a fresh neural network takes a huge amount of annotated data, more than most teams have for a narrow domain. The more practical route is fine-tuning a pretrained model for a handful of epochs. For a genuinely narrow, well-defined entity type, a relatively small set of annotated examples can get you useful results, but the quality of the annotation boundaries matters more than the raw count, since remember, the loss function is scoring whole-span accuracy.

Watch for catastrophic forgetting. Fine-tune a model on new domain data and it can start losing accuracy on the general tasks it already knew how to do. That's a real risk anytime you fine-tune on narrow, web-scraped domain data, and the usual mitigation is being deliberate about how much domain-specific data you introduce and how aggressively you train.

This training workflow has held steady against spaCy 3.8 as of August 2026, and there hasn't been a new major version since v3, so what's documented now should keep working for a while.

For a one-off scrape or a short research task, though, building and training a custom model is often more work than the job needs. That's where the hybrid approach in the next section tends to win on speed.

Extending the pipeline with LLMs for entities spaCy cannot label alone

The spacy-llm package wires large language models directly into a spaCy pipeline through a serializable llm component. It's modular, handles prompting and response parsing for you, and the big draw is that it needs no training data to recognize a brand-new entity type. Ask it to find "clinical trial phase" mentions and it'll try, no annotated dataset required.

The mechanical pattern looks like this: run the spaCy pipeline as usual to get a Doc object, pull candidate spans out using doc.ents, doc.noun_chunks, or a custom matcher, then use those spans to build a structured prompt for an LLM API call. spaCy does the cheap, fast work of finding candidates. The LLM handles the harder judgment call of classifying something ambiguous.

LLMs earn their keep on out-of-domain text where no training data exists at all, on entity types you're inventing on the fly (zero-shot), and on coreference resolution across long documents, figuring out that "the company," "the firm," and "Acme Corp." three paragraphs later all refer to the same entity. Fine-tuned local models still win on in-domain, structured web content and on anything speed-sensitive, given that LLMs run 7 to 30 times slower than local models for this kind of task.

A meaningful share of LLM API responses, somewhere in the range of 5 to 15% by some estimates, fail to parse as valid JSON when you're using a naive prompt that just asks nicely for JSON back. That failure is silent unless you're checking for it. Log the raw response text before calling json.loads() on it, every time, and never assume the response came back clean.

And the two approaches aren't rivals, they're complementary. A lightweight spaCy rule or statistical check can sanity-check whatever the LLM hands back, catching cases where the model hallucinated an entity that isn't actually in the source text.

A medication extraction pipeline built on this pattern shows the ceiling of what's possible: run against domain-specific web content, it can extract medication-related entities that a general-purpose model would miss, and whether the person was still taking it, plus the primary psychiatric condition (with severity and diagnostic status), comorbidities, and side effects, all in a single pass. That's a level of structured detail no off-the-shelf OntoNotes model was ever built to produce.

Structuring entity output as AI-ready data for downstream consumption

Once entities are extracted, the format they land in decides whether anything downstream can actually use them.

displaCy's ENT renderer is genuinely useful during development. It fetches the JSON-formatted entity annotations and turns them into readable, color-coded HTML so you can sanity-check the pipeline by eye. It should stay a development tool, though: raw HTML generated from model output should never get served directly to users, since that's an open door for cross-site scripting.

For production, the target format is JSON: a list of dicts, each one carrying the entity text, its label, its start and end character positions, and a plain-English description from spacy.explain(). That structure is what makes the output "AI-ready" in any real sense, not a buzzword, just a format that a database, a search index, or another model can consume without a human translating it first.

From there, the JSON feeds whatever comes next, whether that is a database table for competitive intelligence tracking, a search index for content classification, a trigger for customer support automation, or a labeled dataset for training the next model down the line. The format is simple on purpose. The work that made it trustworthy, the cleaning, the model choice, the rules layer, the LLM fallback, happened upstream. By the time the JSON lands, it should just work.