Exactly How Do You Define Duplicate Content And Should We Worry About It?

By Author: Donald Saunders
Total Articles: 84
Comment this article

The discussion over exactly how duplicate content is defined and whether duplicate content is a problem continues to rage and there is no sign of it going away. So exactly how is duplicate content defined and should we be worried about it?

The generally accepted view is that duplicate content does matter and, although one well respected SEO expert recently expressed the opposite view, even a cursory inspection of the huge mountain of material that has been published on the subject recently will clearly show that this is very much a minority view.

If we agree with the view that duplicate content does really matter, then just how do we define duplicate content? For instance, if I produce an original article for submission to an article directory and then alter that same article for submission to a second directory how will the search engines analyze these two articles and decide whether they contain duplicate content? The truth is that we don't know, however, here is one webmaster's view.

When checking for duplicate content was first done by the major search engines it was a simple case of looking at one web ...
... page as a whole against another and there was no attempt to start to cut apart the two pages and compare individual page elements. At that time you could use identical content and just add an introductory and concluding paragraph to one of the two pages to fly under the duplicate content radar. Sadly for many publishers those days have long since disappeared.

Nowadays, the major search engines dissect the two pages to allow them to compare individual elements and it is here that we find the core of today's disagreement. Most experts agree that attention is now focused upon the main content of a page rather than the structure of the page. The majority of webmasters make use of templates to build their pages which set the structure of each page including such things as navigation menus, headers and footers. This is generally believed to be quite acceptable and the major search engines do not class this as being duplicate content. What the major search engines are checking is the actual content which is contained within the body of the page. But precisely how do they go about checking this page content?

Some people argue that this checking is done at 'block' level (checking individual sentences or paragraphs), although other people contend that filters search for phrases or possibly even for individual words. None of us really knows of course although it might seem reasonable to assume that the most likely basis for examination would be to make use of either phrase or sentence matching.

Sentence matching is quite clear-cut and simply means breaking both pages down into chunks determined by the page's punctuation. Look, for instance, this sentence:

It is relatively simple to find a good deal on a sofa, as long as you know where to look.

This would be viewed as either a single sentence or two sentences, depending on whether you use the strict definition of a full-stop as being the end of a sentence or adopt an elastic approach and make use of other punctuation marks, such as commas.

Matching based on phrases is a little bit more difficult. What constitutes a phrase? Should a phrase have 2 words or 3 words or 4 words or�?

Let us say for a moment that a phrase is 3 words. In this case the following phrases would all be classed as duplicate content if they appeared on two pages which were being examined:

Did you know
In those days
One way to
Day to day
You can get
In the end
Take a look
At that time

All of these phrases are standard day to day phrases which could be used on pages about building a greenhouse, cycling, football or anything else you care to mention. Now some people would say that the major search engines do examine pages down to this level. As an example, when I asked the support staff of one particular content checker (Dupecop) about how their software checked duplicate content they said:

"DupeCop compares both individual words and 3-word phrases. It also ignores all punctuation and scans across sentences"

I was not surprised therefore that when I ran several articles through this software (comparing articles on the subject of Christmas decorations against articles about dogs) I found they had an average of 25% duplicate content!

Taking this into account, I believe that it would be ridiculous that the major search engines would filter down to this level. But just how low would the filters be set? At 4 words or 5 words or�? To be frank, your guess would be as good as mine.

Over the years I have written and published hundreds of articles and have monitored the results for signs of duplicate content penalties, as far as it is possible for anyone to do so. On the basis of my own experience I would say that filtering is not carried out down to the level of short phrases but is far more likely to stop at the sentence level. As a result, providing you re-write your articles down to sentence level, you ought not to have a problem in avoiding the filters. In actual fact, even if a couple of your sentences are duplicated you ought to be okay.