About VapeBot

VapeBot is Vape Compare’s web crawling bot (or sometimes referred as spider), and it’s very similar to Google’s web crawler. In order for us to display your prices correctly to your potential buyers and keep them up to date, we need to regularly visit your website and check the prices of your products.  Therefore, our crawler will visit your website and read the prices of your products in a time interval that’s been specifically programmed in a way that it won’t overload your servers and won’t affect at all your website’s performance. Ideally, we’d like to visit your website on a daily basis to make sure your potential buyers will be aware of any price reductions that will happen on your website, so they can take advantage of the discounted prices.

Visiting frequency

On larger websites with hundreds of products, we’ll be using “machine learning” techniques, in order to determine the time interval VapeBot will visit your website. The crawler will learn about your categories and products, and it will identify the categories that are most likely to change prices, and it will visit those pages more frequently, ignoring the other products for longer periods of time. For example, if the product prices in a specific category were changed 10 times in a month, it will crawl the products from within that category every 3 days – if the prices were changed 4 times in a month, it will visit your website once a week.

We’d like to make sure we won’t interfere with your web shop and your users experience on your website, and we are committed to provide unique value to your business and truly help you growing your online sales. Therefore, we’d like to work together with you and listen to you, and be totally flexible in how we approach the web crawling. Please get in touch with us if you think we crawl your website too often, and let us know any of your concerns, we’ll be more than happy to cooperate.

Crawl rate at each visiting session

While most of the web crawlers extract data from a website at the rate of hundreds of pages/second, the web crawlers that respect website owners and care about more than just getting the data (ex. GoogleBot, BingBot, BaiduSpider, etc.) , with every visit they crawl the websites at the rate of 1 page every 10-30 seconds, or even more. That will ensure that the website’s server won’t be over loaded, and the crawling process won’t affect at all the performance of the website and user experience. We are also committed to use this approach when reading your prices. With every visiting session, we’ll only load one page at a time, at the (current) rate of 15 seconds. The only exception will be when we add your products for the first time in our database – but that’s a one off process,  and still, it will be at the rate of 1 crawl/second, which shouldn’t cause any problems even to cheap, shared hosting based websites.
If you have any questions or suggestions regarding our technology and services, please feel free to send us an email to info {at} vape-compare.co.uk

If you wish to change the crawl rate for your website (for example if you want us to update your prices daily, weekly, etc.), please send us an email to vapebot {at} vape-compare.co.uk

VapeBot changelog

Version 1.2.0 (19.07.2015) Major improvement in crawling technology

  • 3 new powerful indexing spiders added
  • Fully automated and autonomous new products finding and indexing
  • Automatic product category recognition
  • Http header error  detection and url filtering accordingly
  • Total web spiders: 5

Version 1.1.0 (17.06.2015)

  • Improved crawling rules
  • 100% price accuracy
  • Crawler politeness more precise and more “respectful” with vendor websites
  • Discounts page introduced
  • Advanced semi-manual new product adding to database feature
  • Improved back-end management and crawl monitoring
  • Advanced front end stats page about crawl sessions
  • 404 and non existent page automatic recognition by VapeBot
  • Total web spiders: 2

Version 1.0.0 launched on 27.05.2015

Total web spiders: 1

Warning: Invalid argument supplied for foreach() in /home/iphonere/public_html/vape-compare.co.uk/wp-content/themes/twentyfifteen/content-page.php on line 403