Web sites with dynamic content are usually, though not exclusively, business web sites with an on-line shopping facility.

They often rely on interaction with visitors who may typically enter parameters to define the content that is ultimately displayed, e.g. entering a product code or description on a shopping site, entering a country or region on a support site or entering some personal data to select a set of tailored content.

Effective web site marketing depends on top search engine positioning

These are sites that most need top search engine positioning and marketing these business web sites presents particular difficulties as the search engine spiders are incapable of entering the variable information.  In any case, the exercise may have to be performed many times over to generate a complete set of the variable content (for example, every product in your catalogue) before adequate indexing could be accomplished by the search engine.

How can a search engine spider index a page that doesn't exist?

The problem here is that the page content is generated “on the fly” from information entered or selections made by the visitor; it doesn’t exist until someone enters the information, or perhaps makes a menu selection, to generate it.

You can distinguish a dynamically generated page form the URL; it generally contains ? = % or perhaps all three.

It’s true that the search engines used to virtually give up and move on every time they ran into a query string in a URL (containing one or more of the above “special characters”).  These days they’ve become a little more enlightened and moved with the times as so many sites are now using dynamic content; the proliferation of sites with shopping carts for example.

Even Google now indexes some dynamic sites but there are still some “rules” to be aware of if you want to get your site indexed and ranked well.

1. Keep the number of parameters to 2 or less, e.g. www.mysite.com?product=widgetd&colour=blue

2. Do not use session ID’s in your URL.  To understand why, you have to understand a little about how ‘bots’ and search engines index the content on your site.  The spider (bot) goes and finds the site/page, brings back the info about who and where it is, and stick it in a database.  The search engine subsequently indexes it. When the search engine visits, the session ID will be different to that reported by the spider; what’s a poor search engine to do?  What’s the page in question, a duplicate page with the same content but a different URL? Spam?  In reality, the spider will probably recognise it as a session ID (they normally look something like sid=fdf4151fa3affca47f636508f0885d7e), and ignore either the page or the site completely.

3. Make sure that you have links to your dynamic pages from static (html) pages with entries like:
   
   < a href="http://www.yoursite.com/products?prod=blue_widget" >Product 1
                               and
   < a href="http://www.yoursite.com/products?prod=red_widget" >Product 2
   
   etc.  If not then all you stand to get indexed is
   
   www.yoursite.com/products.

As an alternative to this approach, create a site map and link individually to all dynamic pages; that way at least the search engine stands a chance of finding them – remember, no find, no index, no top search engine positioning.

One of the reasons that search engines were/are so reluctant to crawl dynamic content was/is the risk of deep crawling by following links on dynamic pages.  This could lead to a spider getting stuck in the bowels of site and unable to get out again; the so-called spider trap.

Be aware also that spiders will not follow links from dynamically generated pages.

Alternatively, use one of the 'work arounds'.

There are a number of well documented workarounds to the problem of search engines not following URL’s with query strings.

You can use the Apache PATH_INFO method to create dynamic pages with search engine-friendly URL’s like

www.yoursite.com/productswidgetblue.htm rather than

www.yoursite.com/products?prod=widget&colour=blue

You could achieve something similar with .htaccess, the “ForceType” directive and the Apache mod_rewrite module.

Much of what is available is only for the Apache web server so, chances are, if you’re using .asp or some similar technology on Microsoft’s Internet Information Server (IIS), you’ll have to look elsewhere. 

IISRewrite is a version of mod_rewrite for IIS and ISAPI_Rewrite Lite from isapirewrite.com is a free ISAPI filter for IIS.

Copyright © April 2004