zaglebie Posted August 24, 2013 Share Posted August 24, 2013 I have installed module which list all comments from store on one page, they are listed on url /module/blockcomments/comments.php BUT in robots.txt we all have disallow: /modules/ i've tried Allow: /modules/blockcomments/comments.php Disallow: /modules/ which should work - but it doesnt, what to do ? Link to comment Share on other sites More sharing options...
vekia Posted August 24, 2013 Share Posted August 24, 2013 everything looks fine above, so it should work as you said. the question is: why you think that this doesn't work? Link to comment Share on other sites More sharing options...
zaglebie Posted August 24, 2013 Author Share Posted August 24, 2013 because google webmaster tool or other tools like http://phpweby.com/services/robots are telling me this modules/blockcomments/comments.php Sieć Odrzucone przez plik robots.txt Link to comment Share on other sites More sharing options...
zaglebie Posted August 24, 2013 Author Share Posted August 24, 2013 (edited) when i remove disallow: /modules/ then its fine: Access allowed by line 18 Allow: /modules/blockcomments/comments.php but with disallow: /modules line i got: Access blocked by line 19 Disallow: /modules/ which means that robot is ignoring line 18 where i'm allowing to access one file, very strange Edited August 24, 2013 by zaglebie (see edit history) Link to comment Share on other sites More sharing options...
zaglebie Posted August 25, 2013 Author Share Posted August 25, 2013 bump Link to comment Share on other sites More sharing options...
PrestaHeroes USA Posted August 25, 2013 Share Posted August 25, 2013 This is supposed to work User-agent: Googlebot Disallow: /directory/ Allow: /directory/subdirectory/ Link to comment Share on other sites More sharing options...
PrestaHeroes USA Posted August 25, 2013 Share Posted August 25, 2013 actually forget my previous post. tip: robots.txt is not really a ps function and you are much better served looking outside of ps for this sort of answer. This is directly from Google https://groups.google.com/d/msg/Google_Webmaster_Help-Indexing/36MQedZxMVo/NGLdQOfFaDoJ Link to comment Share on other sites More sharing options...
Dh42 Posted August 26, 2013 Share Posted August 26, 2013 The problem with your robots.txt file is the order. You are allowing a file, then disallowing it. They are read from the top down so you can set something and then unset it. What you basically need to do is reverse the order, have the disallow rule first then have the allow rule. 1 Link to comment Share on other sites More sharing options...
zaglebie Posted August 27, 2013 Author Share Posted August 27, 2013 (edited) [deleted] Edited August 27, 2013 by zaglebie (see edit history) Link to comment Share on other sites More sharing options...
zaglebie Posted August 27, 2013 Author Share Posted August 27, 2013 (edited) i've put allow below disallow and will wait for googlebot crawling, will let you all know.. Edited August 27, 2013 by zaglebie (see edit history) Link to comment Share on other sites More sharing options...
zaglebie Posted August 30, 2013 Author Share Posted August 30, 2013 it worked - thanks Link to comment Share on other sites More sharing options...
vekia Posted August 31, 2013 Share Posted August 31, 2013 Well, I learned something new! thread marked as [solved] 1 Link to comment Share on other sites More sharing options...
Recommended Posts