How many of your pages are saying the same thing? Semantic Duplicates

Most websites that have been around for a few years have the same problem. Nobody planned it. It just happened.

One team writes a page about volunteering. A year later, another team writes their own version because they didn't know the first one existed. An old page gets copied into a new section, and both stay live. A policy gets explained on four different pages, and three of them are now slightly out of date.

Visitors land on the wrong page. Editors update one version and forget the others. And Google has to guess which page is the "real" one, so often neither of them ranks well. SEO people call this keyword cannibalisation. I call it a mess that nobody has time to find.

So I built a Drupal module to find it.

Semantic Duplicates

Semantic Duplicates compares every page on your site with every other page and tells you which ones overlap. It is now available on drupal.org for Drupal 11.

It doesn't give you a confusing score. Each pair of pages gets a plain label:

• Duplicate content: the same text is on both pages.
• Competing pages: different words, but the same topic. This is the cannibalisation problem.
• Shared furniture: the pages only share things like contact blocks or disclaimers, so there's nothing to fix.
• Too short to be sure: the module tells you honestly instead of guessing.

Two ways of looking

The module checks your content in two ways.

The first compares the actual wording. It finds copied and partly copied text, and it doesn't need any AI at all.

The second compares meaning. It uses an AI model to spot two pages that cover the same subject in completely different words. This part is optional. If you don't set it up, the module still finds duplicated text.

Your content stays with you

This was important to me. Many of the organisations I work with, such as charities, universities and councils, can't send their content to an outside service. Sometimes it's privacy, sometimes procurement rules, sometimes it's just that nobody wants another monthly subscription.

With a local AI model running in Ollama, everything happens on your own server. No content leaves your site, no API key is needed, and there is nothing to pay for. It also works on pages that aren't public, which hosted SEO tools can't see.

It reports, you decide

The module never deletes, unpublishes or merges anything. Two similar pages are not always a mistake. A short summary and a detailed guide should overlap. A page for applicants and a page for staff might say similar things for good reasons.

What the module does is take thousands of possible page pairs and cut them down to a short list worth looking at, with a reason for each one. The decision is yours. When you decide to retire a page, I recommend the Redirect module so visitors and search engines still land in the right place.

Editors also get a gentle warning when they edit a page that looks like other content on the site. It never blocks them from saving.

Where this really helps: before a migration

This is the part closest to my own work. If you are moving from Drupal 7, 8 or 9 to Drupal 11, a migration is the best moment to clean up. Every duplicate page you carry across is a page you'll maintain forever. Every one you leave behind is time and money saved.

Running Semantic Duplicates before a migration gives you a clear list of what doesn't need to come with you.

Try it, and tell me what it gets wrong

This is an early alpha release, so please try it on a test site first. There's a demo submodule that adds twelve example pages, so you can see the report before using real content.

The most useful thing you can do is run it on a real site and tell me where it's wrong. False matches and missed duplicates are both worth reporting in the issue queue. Please mention roughly how big your site is and what kind of content it has.

And if you'd like help with the audit itself, or with the migration that comes after it, get in touch.