Mozilla/4.0 (compatible;)

This user agent was in the middle of many page requests in my Apache logs, requesting content referenced by link tags in the head section.

After a bit of research on one of the link tag URLs, I ran this script:

Code:

IPS=`grep Author access_log | cut -f 1 -d ' '  | sort | uniq`
for IP in $IPS
do
        echo Testing "$IP"
        host "$IP"  
done

In almost every case, the requests came from large organizations - corporations, government agencies, and the military.

These institutions often use proxy servers, and Mozilla/4.0 (compatible must be a common user agent setting for the proxy server requests.

In the one case where it wasn’t a large organization, it was a blacklisted IP, and the user agent was Java.

The sample set was limited, but the pattern was clear.

	IPS=`grep Author access_log \| cut -f 1 -d ' ' \| sort \| uniq`
	`for IP in $IPS`
	`do`
	`echo Testing "$IP"`
	`host "$IP"`
	`done`

web notes - a blog