Default Robots.txt in Joomla 2.5 Affects Google Rank

If you still use Joomla 2.5 (like me :P), Google will now make you think about mobile usability of your site. Recently I got a warning on Google Webmaster Tools that bunch of pages on www.utilizewindows.com are not mobile friendly, despite the fact that I actually use responsive template on it.
To check things out, first I’ve used the “Fetch as Google” feature in Webmaster Tools to see how Google sees my site. This was the result:

This was just plain wrong. Below this picture there was also a list of resources (CSS and JavaScript files) that were not available to Google:

Note the folders (marked yellow) in which the CSS and JS resources are located, but not available to Google. Well, all those folders were actually blocked by the default robots.txt file which is shipped with Joomla.
This problem was particularly evident when I checked my pages to see if they were acknowledged by Google as mobile friendly. To do that I’ve used Mobile-Friendly Test.

This was a great impact on the Google rank for my whole site. To try and resolve this problem I have simply commented out all of the entries in robots.txt file that disallowed Google to crawl resources (files) in those directories. For example, Google noted that Googlebot couldn’t get resource http://www.utilizewindows.com/libraries/expose/interface/css/joomla.css. This resources is in libraries folder, and that folder is disallowed in default robots.txt file provided by Joomla!. I have simply commented out that row in robots.txt file by putting “#” in front of the entry. So instead of

Disallow: /libraries/

I now have

# Disallow: /libraries/

in my robots.txt file (I could also simply delete those entries). I’ve made this change for all the folders that were disallowed in robots.txt and from which Google tried to, but couldn’t get resources because of the disallow entry.
After submitting my new robots.txt file to Webmaster Tools, I’ve checked my pages again. The results were positive and visible immediately! For example, this is how Googlebot sees pages on www.utilizewindows.com now:

Also, Google now sees the site as mobile friendly:

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: