spanishcivilwarstamps Posted November 3, 2014 Share Posted November 3, 2014 Hello everybody, I'm getting thousands of visits from the referrer st3.cwl.yahoo.com, each of them from a different IP address. While these visits do not reflect on my Google Analytics dashboard, they get registered in my Prestashop stats when I chech Online Visitors, and they consume CPU resources in my webserver, from 70 to 100% usage. I've cheched the address http://st3.cwl.yahoo.com and it does not exist. At the moment of writting this message, I've already got 1548 visits from st3.cwl.yahoo.com as showing in the "Visitors Origin" section of Prestashop stats, and it is still going on. This is not the first time I detect this; in past monthes I've seen the same behaviour, at least one time per month. This not appears to be a Yahoo crawler. Here there is an example of a registry of one of these visits from my hosting latest visitors records: IP: 5.135.250.45 URL: en/ugt-political-labels/2032--ugt-casa-del-pueblo-5c-red-gg-1989-used.html Time: 11/3/14 7:48AM Size (bytes): 115035 Referring URL: http://st3.cwl.yahoo.com/ User Agent: Mozilla/4.0 (compatible; MSIE 5.01;Windows NT 5.0) Does anyone know what is it and how to stop it? I am using Prestashop 1.5.6.2 Link to comment Share on other sites More sharing options...
musicmaster Posted November 3, 2014 Share Posted November 3, 2014 There are more sites with this problem. See http://climprofesional.com/robots.txt You could try their solution... Link to comment Share on other sites More sharing options...
spanishcivilwarstamps Posted November 3, 2014 Author Share Posted November 3, 2014 Thanks musicmaster, but st3.cwl.yahoo.com is not a user agent, it is a referrer. I tried this solution in the past and it does not work. Link to comment Share on other sites More sharing options...
musicmaster Posted November 3, 2014 Share Posted November 3, 2014 You can try changing .htaccess. See for example: http://linux.icydog.net/apache/refspam.php Problem is that Prestashop will overwrite it the next time you insert a new product when you have pretty url's enabled. But it might be better than nothing. Link to comment Share on other sites More sharing options...
tuk66 Posted November 3, 2014 Share Posted November 3, 2014 It is Yahoo crawler. Try to add: User-agent: Slurp Disallow: / User-agent: Yahoo! Slurp Disallow: / to robots.txt. Link to comment Share on other sites More sharing options...
spanishcivilwarstamps Posted November 3, 2014 Author Share Posted November 3, 2014 Thanks tuk66, but I don't think it is Yahoo crawler. The full user value for Slurp agent in access logs is this: Mozilla/5.0 (compatible; Yahoo Slurp; http://help.yahoo.com/help/us/ysearch/slurp)While my logs reflect this for the user agent with referrer st3.cwl.yahoo.com: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0) And all visits coming from IPs belonging to OVH ISP. Moreover, I got hits up to 10 times a minute. I don't know a single one honest crawler doing this. In total I've had 1825 hits from the referrer st3.cwl.yahoo.com in 3.5 hours. It only has stopped when I've noticed the company that monitors my web security, but they did not tell me yet what's this about. Link to comment Share on other sites More sharing options...
tuk66 Posted November 4, 2014 Share Posted November 4, 2014 You are right. The referring URL is just a cloak. Link to comment Share on other sites More sharing options...
Recommended Posts