Zonal relevance analysis for SEO

July to November 2025 · Python, Flask, spaCy, pymorphy3, SBERT, scikit-learn

My role. Subcontractor to an agency building an SEO platform, owning one vertical slice: content relevance scoring, competitor discovery from search results, and the zonal semantic analyser. Authentication, proxying, the keyword analyser and infrastructure belonged to other developers on the team. The final zonal scoring code has a second author.

The problem: telling an SEO team not just which words they are missing, but where on the page those words are missing.

Zonal analysis

Why zones

Keyword tools treat a page as a bag of words. Strip the HTML, count terms, compare against competitors. The advice that falls out is “use this word three more times”, which is close to useless, because where a term appears matters far more than how often.

A word in <title> is a claim about what the page is. The same word in a footer link is noise. Averaging them destroys the only signal worth having.

So text is labelled by structural role, and each role carries a weight:

L1_WEIGHTS = {
    "T": 3.0, "H1": 2.5, "H": 1.6, "TXT": 1.0,
    "A": 1.2, "MEDIA": 0.9, "UI": 0.8,
}

A second level refines this by content type, and those weights change with the page category, because a specifications table is the substance of a product page and clutter on an article:

Subzone Article Product Listing Service
TXT.SPECS 0.8 1.3 0.7 1.0
TXT.FAQ 1.1 1.0 0.9 1.2
A.CONTENT in-content links 1.3 0.9 0.9 1.1
A.PRODUCT_CARD_TITLE 0.0 0.0 1.4 0.0
UI.CALCULATOR 0.0 0.8 0.0 1.1

The zeros are as deliberate as the peaks. On an article, product card titles are not down-weighted but switched off: their presence means the extractor misclassified something, and counting them would poison the comparison.

The payoff is the recommendation column. Flat counting says “you use this term 18 times, the market uses 27”. Zoning says H1:+1, TXT:+8.

Extraction: three guards before the blacklist

Boilerplate removal by class name is the obvious approach and it quietly eats real content. class="main-footer" should go; class="downloads" should not match a rule for ads; a <footer> inside an <article> is part of the article.

So three protective checks run before any blacklist:

# Main and article are never removed
if tag in ('main', 'article'):
    return False
# A header or footer inside an article is content, not chrome
if tag in ('header', 'footer'):
    parent = element.getparent()
    while parent is not None:
        if _get_tag_safe(parent) in ('article', 'section', 'main'):
            return False
        parent = parent.getparent()
# If it contains an H1, it is probably not junk
if element.find('.//h1') is not None:
    return False

Class matching splits into tokens rather than testing substrings, with the reason recorded next to it: so that ads does not fire on downloads, while footer still fires on main-footer. Product card detection uses the same discipline, matching product- and product_ rather than product, so production is not a product.

Russian morphology makes lemmatisation mandatory rather than optional: without it, six inflected forms of one noun are six terms, each carrying a sixth of the real frequency. pymorphy3 handles that, behind an LRU cache of 50 000 entries, because the same tokens recur constantly across a competitor set.

Scoring

Importance is computed from the competitor corpus, never from the user’s own page:

importance[lemma] = 100.0 * (
    0.40 * df_component +        # share of competitor pages containing it
    0.25 * weighted_presence +   # zone-weighted presence across those pages
    0.15 * avg_tf_norm +         # average frequency, normalised
    0.10 * cluster_boost +       # appears in the user's topic cluster
    0.10 * cos_sim
)

Document frequency carries the largest share on purpose. A term on nine of ten ranking pages says something about the topic; a term appearing two hundred times on one page says something about that page’s author.

Frequency normalises against the 95th percentile rather than the maximum, so one keyword-stuffed competitor cannot compress everyone else toward zero:

avg_tf_norm = min(1.0, avg_total / max(1.0, p95_total))

A per-zone score answers “how much of what matters do we cover here”, and the empty case matters more than it looks:

if total_important == 0:
    score = 100.0          # nothing important lives here; not a gap
else:
    score = 100.0 * our_important_in_zone / total_important

Scoring an irrelevant zone at zero would send users off to pad a section that needs nothing.

Deviation is measured against competitor quantiles rather than the mean: below the median is underuse, above the 75th percentile is overuse. That gives a band of acceptable values instead of a single number to chase. Recommendations are capped at three zones and ordered by gap size, preferring high-weight zones first.

Where the language model is allowed to act

Rules own everything structural: boilerplate removal, all L1 assignment, product card context, every A.*, UI.* and MEDIA.* label, and the initial TXT.* guess. The model is given exactly one job, refining the TXT.* subzone label, for at most twelve blocks per page. Titles, headings, anchors and UI never reach it.

That boundary was the client’s requirement and it is the right one: heuristics must be able to produce a complete report on their own, so the model is an improvement rather than a dependency.

Four controls keep the cost bounded: the twelve-block cap, truncation to 1200 characters, exclusion of blocks whose DOM already determines the answer, and a priority ordering that sends the most ambiguous blocks first.

Rules then override the model on the way out. A specialised prediction is accepted only if a regex also agrees, unless the model reports high confidence in itself:

if conf < 0.9:
    if guess == "TXT.FAQ"   and not _looks_like_faq(text):           guess = "TXT.PARA"
    elif guess == "TXT.SPECS" and not _looks_like_specs(text, hint): guess = "TXT.PARA"
    elif guess == "TXT.TABLE" and not _looks_like_table(text, hint): guess = "TXT.PARA"

What that actually did, measured over 918 logged decisions

   
Model changed the heuristic’s label 135 of 644, 21%
Regex guard overrode the model 19 of 918, 2.1%
Guard reverted a correct model correction 0
Resolved to the null label TXT.OTHER 547 of 918, 59.6%
Confidence reported as exactly 1.0 76.5%

Every one of the nineteen overrides was against a stated confidence of 1.0, which is the argument for the veto in one line: the model is confidently wrong often enough to matter, and it never once reported low confidence when it was.

The typical override is a page footer reading “editorial telephone: …” confidently labelled as a specifications block. The typical win is a holiday-hours notice that no regex would ever classify correctly.

Since the confidence escape hatch opens at 0.9 and three quarters of responses claim 1.0, the veto is far weaker in practice than it reads. That is a design flaw, not a feature.

Competitor discovery

Competitor discovery from search results

Users pick competitors badly. They name the brands they think about rather than the pages actually ranking, and the whole analysis is then anchored to the wrong corpus. So the competitor set comes from the search results: a query, a depth, a country, a city, a language and a stated intent.

Results are over-fetched at two to three times the requested depth, because filtering removes so much that fetching exactly the requested count returns a short list.

Intent is scored 1 to 10 by a model, in batches of forty URLs, against a prompt naming both the topic and the user’s stated intent. One detail is worth recording because it caused a real complaint: any URL the model omits from its reply silently defaults to 3. Users reported “everything scores three” and the cause was not the scoring, it was the missing rows.

Competitor strength combines coverage across the submitted queries with average position, log-normalised, rescaled so the strongest competitor is exactly 100.

Performance: where the time actually goes

Measured across thirteen end-to-end runs, 92 to 96 percent of wall clock is serial language model calls.

Step 1  fetch HTML                          0.001 s   (cached)
Step 2  extract and classify blocks         0.240 s
Step 2.1 refine zones with the model      125.603 s
Step 3  lemmatise and count                 0.311 s
Step 4  aggregate competitor statistics     4.984 s
Step 5  compute importance                  0.657 s
Step 6  build results                       0.067 s

918 measured calls: median 1.79 s, mean 1.88 s, p95 2.45 s. Sixty calls at roughly two seconds reproduces that step exactly. There is no batching and no concurrency in the model client; the twelve-block cap is a limit, not a batch.

Everything else in the pipeline is essentially free. All of the optimisation value in this system sits in one place, and it is the place with no concurrency.

Cold start is 32 seconds, of which the Russian spaCy model is 20 and the sentence encoder is 6. In production across 425 analyses: median 22 seconds, p95 151 seconds, worst case 28 minutes.

What is wrong with it

Writing these down is the point of the exercise.

The sentence encoder sees roughly the first paragraph. paraphrase-multilingual-mpnet-base-v2 truncates at 128 word pieces, and whole page texts are passed to it in one call with no chunking. Pages in the corpus reach 900 000 characters. The page-level similarity score is therefore computed against an opening fragment, and nothing in the code says so.

Encoding is forced, not detected. Responses are decoded as UTF-8 unconditionally. A cp1251 page becomes mojibake that flows into lemmatisation and embeddings. Because forcing an encoding substitutes replacement characters rather than raising, there is not one decode error in the logs. The failure is completely invisible, which makes it worse than a crash.

Raising spaCy’s length limit converted an error into a crash. A clean ValueError: text of length 1470851 exceeds maximum of 1000000 was resolved by raising the maximum to ten million. The limit exists because the parser needs roughly a gigabyte per hundred thousand characters. Logs afterwards show parses attempted at 3.6 million characters alongside nine out-of-memory kills. I diagnosed this as a server-side restriction at the time. It was not; it was a default that existed for a reason.

Three subzones can never be populated. Breadcrumbs, banners and widgets are all in the boilerplate list and their subtrees are deleted before classification runs, yet UI.BREADCRUMBS and UI.BANNERS still carry non-zero weights. Dead configuration that looks live.

Two parallel count systems drift apart. Raw occurrence counts are displayed to the user while zone-weighted counts feed the recommendation, so a user reading a delta of minus three can receive advice that sums to something else. The split was a deliberate client request; wiring the weighted value into the recommendation was my mistake.

No accuracy number exists, and I would not trust one that did. Ground truth was established by having three separate model passes hand-count one lemma by zone on one saved page. They disagreed by a factor of two on the largest zone, mostly over whether an adjectival form counts as the same lemma. If annotators cannot reproduce each other, a reported accuracy figure would be decoration.

The deployment never matched the workload. A single gunicorn worker with a single thread and the default 30-second timeout, serving a synchronous request that routinely runs one to five minutes, against a memory limit well under what two loaded model instances need. I diagnosed it and supplied a working configuration. It was never applied, which is its own lesson about where a subcontractor’s influence ends.

What I would do differently

Batch and parallelise the model calls first. It is 95 percent of the runtime and it is sequential. Nothing else is worth optimising until that is.

Chunk long texts before encoding, or state plainly in the interface that the page-level score is computed on an excerpt.

Detect the encoding rather than asserting it, and let a decode failure be loud.

Fit the weights instead of choosing them. Title at 3.0 and headings at 1.6 are reasonable and untested. With ranking outcomes as labels they could be estimated, and I would expect them to differ by niche.

Detect the page category from the markup. The user selects it today, and that selection determines the entire second weight table, which makes it the largest single source of wrong output.