neeva wordmark
icon close
Neeva wordmark

About Neevabot

Updated: May 6th, 2021

  • Neevabot is the web crawler for the search engine neeva.com.

  • Requests from Neevabot set the User-Agent to:

  • Mozilla/5.0 (compatible; Neevabot/1.0; +https://neeva.com/neevabot)
    and originate from IPs where reverse DNS shows it in the *.neevabot.com domain. For example:
  • $ host 100.26.127.17
    17.127.26.100.in-addr.arpa domain name pointer 100-26-127-17.neevabot.com
  • Standard directives in robots.txt that target Neevabot are respected. For example, the following will allow Neevabot to crawl all pages, except those under /private/:
  • User-Agent: Neevabot
    Allow: /
    Disallow: /private/
  • Neevabot also supports the crawl-delay directive in robots.txt files. It interprets the value as the minimum duration, in seconds, between the start of consecutive requests. For example, assume you have specified the following in your robots.txt file:
  • User-Agent: Neevabot
    Crawl-delay: 5
    Neevabot will split each day into 5 second intervals and issue at most one request to your domain inside each interval.
  • If there is no rule targeting Neevabot, but there is a rule targeting Googlebot, then Neevabot will follow the Googlebot directives. For example, Neevabot will fetch all pages, except those under /private/ with the following robots.txt:
  • User-Agent: *
    Disallow: /
    User-Agent: Googlebot
    Allow: /
    Disallow: /private/
  • If you have any questions, or if you think Neevabot is misbehaving on your site, please do not hesitate to contact us at neevabot@neeva.co.