Robots and Spiders

[Frank A. Reynolds]

Am I correct that search robots cannot index an online FileMaker Pro (5.5 Unlimited) database?

What is the most efficient way to help say Google's robot know the type of information that is in your FileMaker database?

Frank A. Reynolds

Hunt Institute

Carnegie Mellon

[Reed]

I guess you could run a script that periodically generates a static HTML page with the content you want indexed, then make that page redirect to your dynamic homepage.

[Steve T.]

Hmm... I used to want to stop bots and spiders from visiting my FM site since I suspected they'd cause it to crash by following long CDML URLs which they'd truncate to the first x many charcters (strong of 128? 256?). I suppose if you WANTED bots and spiders to crawl your site, you could make some CDML hard-coded links with relatively short string URLs and it'd work. Maybe you could even automate or have your site create those CDML strings just for bots by creating hidden or hard-to-find-by-a-human link to these URLs that a bot/spider would follow with glee.

I was going to read up on robots.txt files, too. Maybe there's something you can put in robots.txt to help them.

--ST

[DykstrL]

The best way to ensure that search page bots pcik up your site is to add meta-data to your pages. Here's the basic meta-data format that goes in the header:

<META http-equiv=Content-Type content="text/html; charset=windows-1252">

<META NAME="title" CONTENT="">

<META NAME="description" CONTENT="">

<META NAME="originatorJurisdiction" CONTENT="">

<META NAME="originatorDepartment" CONTENT="">

<META NAME="originatorDivision" CONTENT="">

<META NAME="originatorSection" CONTENT="">

<META NAME="originatorOffice" CONTENT="">

<META NAME="createDate" CONTENT="">

<META NAME="dateofLastModification" CONTENT="">

<META NAME="keywords" CONTENT="">

<META NAME="subjects" CONTENT="">

<META NAME="contactName" CONTENT="">

<META NAME="contactOrganization" CONTENT="">

<META NAME="contactStreetAddress1" CONTENT="">

<META NAME="contactStreetAddress2" CONTENT="">

<META NAME="contactCity" CONTENT="M">

<META NAME="contactState" CONTENT="">

<META NAME="contactZipcode" CONTENT="">

<META NAME="contactPhoneNumber" CONTENT="">

<META NAME="contactFaxNumber" CONTENT="">

<META NAME="contactNetworkAddress" CONTENT="">

You can leave blank or delete any meta-data elements you do not want to include.

[kfutter]

Unfortunately the meta tag method still doesn't allow your content to be searched or indexed, which is Google's prefered approach, and what you ultimately really want to search on any given site. Meta tags are notoriously unreliable and increasingly given less value by search engines (though there is some conflicting evidence that Dublin Core is more effective). I don't know whether search engines or bots can execute db queries and index the results, but at a guess I wouldn't think so.

Kevin