Parent

Methods

WebRobots

Public Class Methods

new(user_agent, options = nil) click to toggle source

Creates a WebRobots object for a robot named user_agent, with optional options.

  • :http_get => a custom method, proc, or anything that responds to .call(uri), to be used for fetching robots.txt. It must return the response body if successful, return an empty string if the resource is not found, and return nil or raise any error on failure. Redirects should be handled within this proc.

# File lib/webrobots.rb, line 20
def initialize(user_agent, options = nil)
  @user_agent = user_agent
  @parser = RobotsTxt::Parser.new(user_agent)
  @parser_mutex = Mutex.new

  options ||= {}
  @http_get = options[:http_get] || method(:http_get)

  @robotstxt = create_cache()
end

[Validate]

Generated with the Darkfish Rdoc Generator 2.