Just when you think it could not happen, it does anyway…
I have just discovered that Internet Explorer 6.0 has the habit of omitting the trailing slash of a domain name, whenever it is not explicitly appended in a request. This only works for requests of the webroot (like www.example.com), because in all other cases Apache will automatically launch a 301 redirect to the url version with a trailing slash. This is irritating to me because all other browsers will automatically add the trailing slash if it is missing.

Here are some log entries to illustrate what it looks like when you omit the slash of the domain name:

192.168.0.16 - - [22/Jan/2008:09:42:11 +0100] "GET / HTTP/1.1"
200 41369 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
192.168.0.16 - - [22/Jan/2008:09:42:13 +0100]
"GET /wp-content/themes/nodepet/style.css HTTP/1.1"
304 - "http://www.nodepet.com" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"

I marked the referrer string as bold. By sending this broken referrer string, IE is likely to break scripts which rely on the usual behaviour (i.e. referrer checker against hotlinking or script automation) and deny access to legit visitors. Curiously, if you do add the slash to your request from the start, IE 6.0 will behave like any other browser:

192.168.0.1 - - [22/Jan/2008:09:48:16 +0100] "GET / HTTP/1.1"
200 41369 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
192.168.0.1 - - [22/Jan/2008:09:48:17 +0100]
"GET /wp-content/themes/nodepet/style.css HTTP/1.1"
304 - "http://www.nodepet.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"

Up to now I thought a good indicator for a bot generated fake referrer is the missing slash and would make a good SetEnvIf rule to deny access on, however now it seems like this approach is generating false positives. On the other hand, by loosening the check, I open the up the flood gates for spambots, which is not really something I am keen on.