A small test of query wording and search results.

Why we ran it

We needed to know whether the collector could normalise and compare Google Indonesia result sets. Ten queries were enough to test that machinery, but not enough to study rankings.

A 50-query pilot was next on the list. We held it back after this run exposed three gaps: no repeat captures, no record of SERP module boundaries, and no retry when a result block ended early.

The raw table is public because those gaps are useful debugging information.

The query pairs

Indonesian question words often signal the kind of answer someone wants:

• Apa itu NPWP? requests a definition.
• Bagaimana cara punya NPWP? requests a procedure.
• Apa saja persyaratan membuat NPWP? requests a checklist.
• Siapa yang wajib memiliki NPWP? requests an eligibility rule.
• Kenapa BPJS tidak aktif? requests a causal explanation.

We wanted to see whether the collector noticed when a wording change came with a different result set. The awkward part is that wording and intent often change together. A pair that moves from “how” to “why” does more than swap grammar; it asks for another kind of answer.

Method

Ten Indonesian queries across three topics: removing ads from a phone, BPJS membership status, and NPWP. Within each topic the wording was varied to produce grammatical or intent contrasts.

• Google Indonesia, hl=id, gl=id, captured 19 July 2026.
• History-based personalisation reduced via pws=0. This does not disable personalisation; location, session, device, and datacenter effects remain.
• Up to ten results read per query, in displayed order, with destination URLs normalised.
• AI Overview presence recorded.
• One session. One date. No query was captured twice.

We compared each query pair in three ways: shared URLs, meaning results found in both searches; Jaccard similarity, the share of unique results common to both searches; and mean rank shift, the average position change for results that appeared in both. A higher Jaccard number means the two result sets looked more alike.

What went wrong

1. No test-retest baseline. No query was captured twice, so no overlap value on this page can be separated from ordinary session-to-session SERP churn. We do not know what Jaccard the same query scores against itself. Without that number, "high overlap" and "low overlap" have no reference point.
2. No unrelated-query control. We never measured the overlap between two arbitrary, unrelated queries. A Jaccard of 0.000 shows disjoint sets but gives no measure of significance.
3. Nine of twelve pairs are incomplete. Five of the ten queries returned fewer than ten results because the parser reached the end of the visible block. Those rows are flagged * throughout and should be read as overlap of the observed sets, not a top-10 comparison.
4. Unequal set sizes distort Jaccard. A pair where one query returned 7 results has a smaller union, which inflates the ratio relative to a 10-versus-10 pair. Raw Jaccard values are not comparable across rows. See the worked example below.
5. SERP modules were not separated. Video, social, organic, and other result types were pooled.
6. Several treatments changed grammar and intent together. Those pairs cannot isolate a grammar effect even in principle.
7. No page content was analysed. The pipeline collected URLs. It never fetched a ranking page, so this run says nothing about what those pages contained.
8. No controls for anything else. No matched pages were evaluated for backlinks, authority, freshness, speed, or any other ranking variable.

We ended up with one capture, missing results, and no baseline for ordinary SERP churn.

The ten queries

* Fewer than ten results: the parser reached the end of the visible result block. AI Overview appeared for all ten queries in this single capture. That is a snapshot, not a prevalence estimate.

All twelve pairwise comparisons

The table includes every pair produced by the run.

* At least one member returned fewer than ten observed results. Read as overlap of the observed sets, not a complete top-10 comparison. Mean rank shift is undefined where no URLs are shared.

Why the highest overlap score is misleading

B1–B2 (cek versus mengecek) has the highest observed-set Jaccard: 0.636. B1 also stopped after eight results. Here is what that does to the calculation.

B1 returned 8 URLs, B2 returned 10, and 7 were shared.
Jaccard = 7 / (8 + 10 − 7) = 7 / 11 = 0.636

Had B1 returned a full 10, with its 2 extra URLs both non-shared
(so 3 non-shared in total):
Jaccard = 7 / (10 + 10 − 7) = 7 / 13 = 0.538

0.538 is exactly the A1–A2 value.

B1–B2's apparent lead over A1–A2 rests on B1 returning two fewer results. The top-five figure, 4 of 5 shared, does not have that problem. The observed-set value does.

Nine rows have unequal or short sets. Sorting this table by Jaccard would partly sort it by parser performance.

What was in the capture

• The two BPJS procedure queries and the BPJS causal query (cara cek versus kenapa) shared no observed results. Two of those three queries, B1 and B3, returned only 8 results, so shared URLs could exist in the unobserved portion. Without an unrelated-query control, the meaning of zero is unclear.
• The four NPWP queries overlapped very little with one another (0.056–0.125 on observed sets) despite all four naming the same entity.
• The ads query with bagaimana prefixed shared 7 observed URLs with the bare version, while the top-five membership and order differed.
• The ads query carrying the constraints android, tiba-tiba muncul, and tanpa aplikasi shared 3 URLs with the short baseline. That contrast differs in semantic specificity as well as grammar, so it isolates nothing.
• Mean rank shift among shared URLs ran from 1.00 to 7.50 positions. With one capture and no retest, that range is uninterpretable.

The comparison code handled both cases. Why Google returned those sets is outside this test.

Wording changed, but so did search intent

cek → mengecek changes the word form but leaves the task alone. cara cek → kenapa changes the task from verification to explanation.

Calling both of these “grammar changes” would muddle two different treatments. The larger study would need separate groups for intent-preserving and intent-changing pairs.

Before the next run

We would capture each query on at least three dates and in two independent sessions per date. The collector would record SERP modules and retry short blocks. Query pairs would change one linguistic feature at a time, with entity, proposition, specificity, and intent held fixed where possible.

Turning a query into a content outline

For writing purposes, we use this rough sequence:

query grammar → inferred intent → expected answer type → passage structure

In plain English: read the query, identify what the searcher expects to receive, then choose a page structure that delivers it. This is a writing aid, not a ranking model.

These mappings come from content practice. This run did not test any of them.

Example: a BPJS procedure

For a procedural query, we would put the usable steps on the page:

<h1>Cara Mengecek Status BPJS Kesehatan</h1>
<p>Status kepesertaan BPJS Kesehatan dapat diperiksa melalui
aplikasi Mobile JKN.</p>

<h2>Cara mengecek melalui Mobile JKN</h2>
<ol>
  <li>Buka aplikasi Mobile JKN.</li>
  <li>Masuk menggunakan NIK atau nomor kartu dan kata sandi.</li>
  <li>Pilih menu <strong>Info Peserta</strong> di halaman utama.</li>
  <li>Periksa Status Kepesertaan: Aktif atau Tidak Aktif.</li>
</ol>

Steps verified 19 July 2026 against the BPJS Kesehatan Mobile JKN user manual. The menu is labelled Info Peserta, not "Informasi Peserta." An earlier draft used the wrong label.

The imperative sentences are there because these are instructions. We would not count imperatives or spread them through unrelated copy to satisfy a score.

A well-structured procedure can still be wrong. Device-specific instructions go stale silently, and fluent grammar can hide the error. The stamp therefore carries a date and source. A real study also needs a validity check outside the structure score.

Write passages that make sense on their own

Compare a paragraph that names its subject:

NPWP is the taxpayer identification number used in Indonesian tax administration. For Indonesian residents the 16-digit NIK now serves as the NPWP, and individuals register or activate it through the Directorate General of Taxes' Coretax system.

with one that drops the context:

It is important for many people. You can register there if required.

The first still makes sense when quoted on its own. The second does not.

NPWP/NIK and Coretax registration verified 19 July 2026 against the DJP Coretaxpedia individual registration guide and coretaxdjp.pajak.go.id.

HTML can expose that structure, but it cannot supply missing substance. We use headings for answer boundaries, ordered lists for procedures, unordered lists for requirements, tables for comparisons, and figures when a claim needs visual evidence.

What this does not prove

This run tells us nothing about ranking or citation. It did not inspect page content, record cited URLs, match citations to supporting passages, or repeat the capture to test stability.

A later citation study would record the ranking URL, the domain and URL cited in the AI answer, the supporting passage, its fit with the requested answer type, and whether the citation reappears across sessions.

How an SEO writer can use this

Before drafting, label the query as a definition, procedure, explanation, requirements list, eligibility question, or verification task. Put that answer near the top, use the matching format, keep important passages understandable on their own, cite the evidence, and date instructions that can go stale.

Those are editorial choices, not findings from this test. The report does not show that changing headings, adding steps, or rewriting sentences will improve rank.

The comparison code worked. The collector needs another pass.

Related

This is report 001 in the WebmasterNinja R&D report. The GEO scoring methodology explains how the product score is built and what it refuses to score.

Add to Chrome · free Back to overview