Posted by
Martin Pedersen  -  April 2010
Hi again,

We are also experiencing problems with the crawler (presumably the cache crawler) visiting parts of the site that we have explicitly excluded in the settings for the persistent cache (the files are excluded also in the search crawler settings by the way).

The debuglog gives the following clue that the crawler is desperately trying to index a page that does not respond well to crawling, hence the exclusion.

11:08:19  : Running crawler synchronously for VCFile(N/E:0:[Intranet]::/lobby/ajax/calendars.xml) (1353 msec); successful result: "no"

For your reference the regexp we use in the settings for the cache crawler looks like this: /lobby/.*

Any thoughts?

Search this thread: