4 Useful Hints About robots.txt File For CS-Cart Stores

The robots.txt file is a file telling the crawling bots how to scan your website: what to crawl and what to ignore. Crawlers influence on how you may be ranked by the search engines. So, a proper completing of that file is all about SEO good practice to rank higher in search. How to enhance this tiny file to make it work for you? Here is a cheat sheet list from our tech guys.

#1 – Sitemap

Indicate the sitemap in your robots.txt file to make the site structure visible for bots to crawl and index your site better. Using the sitemap directive tells search engines where to find your XML sitemap.

#2 – Crawl Delay

Indicate the crawl delay intervals to define how frequently bots can scan your website. If the crawling period is too short, then it may overload your website making troubles for real customers to interact with your content.

#3 – Service Pages

Disallow indexing non-relevant service pages like printout information, for example. It makes sense to allow content and product pages. 

#4 – Dispatch=debugger 

Draw attention to *dispatch=debugger* agent as the default Cs-Cart’s robots.txt file does not configure that properly making visible some sensitive data (including developers’ credentials). Preventing bots from checking out your private folders will make them much harder to find and index. Read more

Extra hints:

  • Ban bad bots! Learn more about bad bots from here.
  • Check the bot-to-visitor ratio to know how attractive your online store is in the eyes of the real people visiting your website. I the number of bots prevails, it should give you a pause of thought.

Closing

Adopt the best practices and rank higher with CS⁠-⁠Cart! Below is an example of a proper-configured robots.txt file drawn specifically for the CS⁠-⁠Cart platform. Note that this file still needs to be fine-tuned to fit your website.

And if you don’t want to tackle with tech issues, don’t hesitate to contact our developers!

GET EXPERT ADVICE!

Share:
Share on twitter
Share on linkedin
Share on pinterest
Share on pocket
Table of Contents
Proven eCommerce Tips Straight to Your Inbox

Get access to exclusive tips, strategies, and case studies.

Какой-то текст ошибки

By clicking Subscribe or emailing us, you agree with Privacy Policy

More Posts
How to choose developers and not waste time and money?
Download the checklist “10 hallmarks of an unreliable developer” and find out:
 
  • How to evaluate a portfolio and make sure that the company can be trusted?
  • How do you know if a contractor has the experience you need?
  • What questions should you ask a sales manager to protect yourself?
Какой-то текст ошибки
By clicking Receive by email or emailing us, you agree with Privacy Policy
Got Question?

Want to work with us?

Just send us your ideas or concerns about starting your online store or marketplace and we will give you the help you need.

By clicking Submit or emailing us, you agree with Privacy Policy

Free
Checklist