- How do I request VidaMonti not to crawl parts or all of my site?
There is a standard for robot exclusion at https://www.robotstxt.org/orig.html. You can put a file on your server called robots.txt that can exclude VidaMontibot or other “web crawlers.” VidaMontibot has a user-agent of “VidaMontibot”. There is another standard for telling robots not to index a web page or follow links on it, which may be more helpful in some cases, since it can be used more conveniently on a page-by-page basis. It involves placing a “META” element into a page of HTML, and is described here. Remember, changing your server’s robots.txt file or changing the “META” elements on its pages will not cause an immediate change in what results VidaMonti returns. It is likely that it will take a while for any changes you make to propagate to VidaMonti’s next index of the web.
- Why is VidaMontibot asking for a file called robots.txt which is not on my server?
Robots.txt is a standard document that can tell VidaMontibot not to download some or all information from your web server. For information on how to create a robots.txt file, see in here.
- Why do you have to add a piece code into robots.txt file of my ecommerce site?
To make sure 100% that your products along with your ecommerce site will be found on VidaMonti, it’s advisable to add the user-agent of “VidaMontibot” into the robots.txt file of your ecommerce site. Except your products page, any other page can be disallowed to not be indexed by VidaMonti crawlers.
- Why is VidaMontibot trying to download incorrect links from my server?
It is a property of the web that many links will be broken or outdated at any given time. Whenever anyone types a link incorrectly that points to your site or fails to update their product pages to reflect changes in your server, VidaMontibot will try to download an incorrect link from your site. Also, this is why you may get hits on a machine that is not even a web server.
- Why is VidaMontibot downloading information from our “secret” web server?
It is almost impossible to keep a web server secret by not publishing any links to it. As soon as someone follows a link from your “secret” server to another web server, it is likely that your “secret” URL is in the referrer tag, and it can be stored and possibly published by the other web server in its referrer log. So, if there is a link to your “secret” web server or page on the web anywhere, it is likely that VidaMontibot and other “web crawlers” will find it.
- Why isn’t VidaMontibot obeying my robots.txt file?
In order to save bandwidth VidaMontibot only downloads the robots.txt file every week or so. So, it may take a while for VidaMontibot to learn of any changes that might have been made to your robots.txt file. Also, VidaMontibot is distributed on several machines. Each of these keeps its own record of your robots.txt file. Also, check that your syntax is correct against the standard at: https://www.robotstxt.org/orig.html. If there still seems to be a problem, please let us know, and we will correct it.
- How do I register my ecommerce site with VidaMontibot so it will be indexed?
See the Add Your Website to VidaMonti form.
- Why are there hits from multiple machines at VidaMonti.com all with user-agent VidaMontibot?
VidaMontibot was designed to be distributed on several machines to improve performance and scale as the web grows. Also, to cut down on bandwidth usage we would like to run many crawlers which run on machines close to the sites they are indexing in the network.