Parent

Included Modules

Files

HTTPClient

The HTTPClient class provides several methods for accessing Web resources via HTTP.

HTTPClient instance is designed to be MT-safe. You can call a HTTPClient instance from several threads without synchronization after setting up an instance.

clnt = HTTPClient.new
clnt.set_cookie_store('/home/nahi/cookie.dat')
urls.each do |url|
  Thread.new(url) do |u|
    p clnt.head(u).status
  end
end

How to use

At first, how to create your client. See initialize for more detail.

  1. Create simple client.

    clnt = HTTPClient.new
  2. Accessing resources through HTTP proxy. You can use environment variable 'http_proxy' or 'HTTP_PROXY' instead.

    clnt = HTTPClient.new('http://myproxy:8080')

How to retrieve web resources

See get_content.

  1. Get content of specified URL. It returns a String of whole result.

    puts clnt.get_content('http://dev.ctor.org/')
  2. Get content as chunks of String. It yields chunks of String.

    clnt.get_content('http://dev.ctor.org/') do |chunk|
      puts chunk
    end

Invoking other HTTP methods

See head, get, post, put, delete, options, propfind, proppatch and trace. It returns a HTTP::Message instance as a response.

  1. Do HEAD request.

    res = clnt.head(uri)
    p res.header['Last-Modified'][0]
  2. Do GET request with query.

    query = { 'keyword' => 'ruby', 'lang' => 'en' }
    res = clnt.get(uri, query)
    p res.status
    p res.contenttype
    p res.header['X-Custom']
    puts res.content

How to POST

See post.

  1. Do POST a form data.

    body = { 'keyword' => 'ruby', 'lang' => 'en' }
    res = clnt.post(uri, body)
  2. Do multipart file upload with POST. No need to set extra header by yourself from httpclient/2.1.4.

    File.open('/tmp/post_data') do |file|
      body = { 'upload' => file, 'user' => 'nahi' }
      res = clnt.post(uri, body)
    end

Accessing via SSL

Ruby needs to be compiled with OpenSSL.

  1. Get content of specified URL via SSL. Just pass an URL which starts with 'https://'.

    https_url = 'https://www.rsa.com'
    clnt.get_content(https_url)
  2. Getting peer certificate from response.

    res = clnt.get(https_url)
    p res.peer_cert #=> returns OpenSSL::X509::Certificate
  3. Configuring OpenSSL options. See HTTPClient::SSLConfig for more details.

    user_cert_file = 'cert.pem'
    user_key_file = 'privkey.pem'
    clnt.ssl_config.set_client_cert_file(user_cert_file, user_key_file)
    clnt.get_content(https_url)

Handling Cookies

  1. Using volatile Cookies. Nothing to do. HTTPClient handles Cookies.

    clnt = HTTPClient.new
    clnt.get_content(url1) # receives Cookies.
    clnt.get_content(url2) # sends Cookies if needed.
  2. Saving non volatile Cookies to a specified file. Need to set a file at first and invoke save method at last.

    clnt = HTTPClient.new
    clnt.set_cookie_store('/home/nahi/cookie.dat')
    clnt.get_content(url)
    ...
    clnt.save_cookie_store
  3. Disabling Cookies.

    clnt = HTTPClient.new
    clnt.cookie_manager = nil

Configuring authentication credentials

  1. Authentication with Web server. Supports BasicAuth, DigestAuth, and Negotiate/NTLM (requires ruby/ntlm module).

    clnt = HTTPClient.new
    domain = 'http://dev.ctor.org/http-access2/'
    user = 'user'
    password = 'user'
    clnt.set_auth(domain, user, password)
    p clnt.get_content('http://dev.ctor.org/http-access2/login').status
  2. Authentication with Proxy server. Supports BasicAuth and NTLM (requires win32/sspi)

    clnt = HTTPClient.new(proxy)
    user = 'proxy'
    password = 'proxy'
    clnt.set_proxy_auth(user, password)
    p clnt.get_content(url)

Invoking HTTP methods with custom header

Pass a Hash or an Array for extheader argument.

extheader = { 'Accept' => '*/*' }
clnt.get_content(uri, query, extheader)

extheader = [['Accept', 'image/jpeg'], ['Accept', 'image/png']]
clnt.get_content(uri, query, extheader)

Invoking HTTP methods asynchronously

See head_async, get_async, post_async, put_async, delete_async, options_async, propfind_async, proppatch_async, and trace_async. It immediately returns a HTTPClient::Connection instance as a returning value.

connection = clnt.post_async(url, body)
print 'posting.'
while true
  break if connection.finished?
  print '.'
  sleep 1
end
puts '.'
res = connection.pop
p res.status
p res.content.read # res.content is an IO for the res of async method.

Shortcut methods

You can invoke get_content, get, etc. without creating HTTPClient instance.

ruby -rhttpclient -e 'puts HTTPClient.get_content(ARGV.shift)' http://dev.ctor.org/
ruby -rhttpclient -e 'p HTTPClient.head(ARGV.shift).header["last-modified"]' http://dev.ctor.org/

HTTPClient - HTTP client library. Copyright (C) 2000-2009 NAKAMURA, Hiroshi <nahi@ruby-lang.org>.

This program is copyrighted free software by NAKAMURA, Hiroshi. You can redistribute it and/or modify it under the same terms of Ruby's license; either the dual license version in 2003, or any later version.


HTTPClient - HTTP client library. Copyright (C) 2000-2009 NAKAMURA, Hiroshi <nahi@ruby-lang.org>.

This program is copyrighted free software by NAKAMURA, Hiroshi. You can redistribute it and/or modify it under the same terms of Ruby's license; either the dual license version in 2003, or any later version.

Attributes

follow_redirect_count[RW]

How many times get_content and post_content follows HTTP redirect. 10 by default.

proxy_auth[R]

HTTPClient::ProxyAuth

Proxy authentication handler.

request_filter[R]

An array of request filter which can trap HTTP request/response. See HTTPClient::WWWAuth to see how to use it.

ssl_config[R]

HTTPClient::SSLConfig

SSL configurator.

test_loopback_response[R]

An array of response HTTP message body String which is used for loop-back test. See test/* to see how to use it. If you want to do loop-back test of HTTP header, use test_loopback_http_response instead.

www_auth[R]

HTTPClient::WWWAuth

WWW authentication handler.

Public Class Methods

new(*args) click to toggle source

Creates a HTTPClient instance which manages sessions, cookies, etc.

HTTPClient.new takes 3 optional arguments for proxy url string, User-Agent String and From header String. User-Agent and From are embedded in HTTP request Header if given. No User-Agent and From header added without setting it explicitly.

proxy = 'http://myproxy:8080'
agent_name = 'MyAgent/0.1'
from = 'from@example.com'
HTTPClient.new(proxy, agent_name, from)

You can use a keyword argument style Hash. Keys are :proxy, :agent_name and :from.

HTTPClient.new(:agent_name = 'MyAgent/0.1')
# File lib/httpclient.rb, line 340
def initialize(*args)
  proxy, agent_name, from = keyword_argument(args, :proxy, :agent_name, :from)
  @proxy = nil        # assigned later.
  @no_proxy = nil
  @www_auth = WWWAuth.new
  @proxy_auth = ProxyAuth.new
  @request_filter = [@proxy_auth, @www_auth]
  @debug_dev = nil
  @redirect_uri_callback = method(:default_redirect_uri_callback)
  @test_loopback_response = []
  @session_manager = SessionManager.new(self)
  @session_manager.agent_name = agent_name
  @session_manager.from = from
  @session_manager.ssl_config = @ssl_config = SSLConfig.new(self)
  @cookie_manager = WebAgent::CookieManager.new
  @follow_redirect_count = 10
  load_environment
  self.proxy = proxy if proxy
end
timeout_scheduler() click to toggle source

CAUTION: caller must aware of race condition.

# File lib/httpclient/timeout.rb, line 115
def timeout_scheduler
  @timeout_scheduler ||= TimeoutScheduler.new
end

Public Instance Methods

debug_dev() click to toggle source

Returns debug device if exists. See debug_dev=.

# File lib/httpclient.rb, line 361
def debug_dev
  @debug_dev
end
debug_dev=(dev) click to toggle source

Sets debug device. Once debug device is set, all HTTP requests and responses are dumped to given device. dev must respond to << for dump.

Calling this method resets all existing sessions.

# File lib/httpclient.rb, line 369
def debug_dev=(dev)
  @debug_dev = dev
  reset_all
  @session_manager.debug_dev = dev
end
default_redirect_uri_callback(uri, res) click to toggle source

A default method for redirect uri callback. This method is used by HTTPClient instance by default. This callback allows relative redirect such as

Location: ../foo/

in HTTP header.

# File lib/httpclient.rb, line 570
def default_redirect_uri_callback(uri, res)
  newuri = URI.parse(res.header['location'][0])
  if https?(uri) && !https?(newuri)
    raise BadResponseError.new("redirecting to non-https resource")
  end
  unless newuri.is_a?(URI::HTTP)
    newuri = uri + newuri
    STDERR.puts("could be a relative URI in location header which is not recommended")
    STDERR.puts("'The field value consists of a single absolute URI' in HTTP spec")
  end
  puts "redirect to: #{newuri}" if $DEBUG
  newuri
end
delete(uri, extheader = {}, &block) click to toggle source

Sends DELETE request to the specified URL. See request for arguments.

# File lib/httpclient.rb, line 605
def delete(uri, extheader = {}, &block)
  request(:delete, uri, nil, nil, extheader, &block)
end
delete_async(uri, extheader = {}) click to toggle source

Sends DELETE request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

# File lib/httpclient.rb, line 695
def delete_async(uri, extheader = {})
  request_async(:delete, uri, nil, nil, extheader)
end
get(uri, query = nil, extheader = {}, &block) click to toggle source

Sends GET request to the specified URL. See request for arguments.

# File lib/httpclient.rb, line 590
def get(uri, query = nil, extheader = {}, &block)
  request(:get, uri, query, nil, extheader, &block)
end
get_async(uri, query = nil, extheader = {}) click to toggle source

Sends GET request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

# File lib/httpclient.rb, line 677
def get_async(uri, query = nil, extheader = {})
  request_async(:get, uri, query, nil, extheader)
end
get_content(uri, query = nil, extheader = {}, &block) click to toggle source

Retrieves a web resource.

uri

a String or an URI object which represents an URL of web resource.

query

a Hash or an Array of query part of URL. e.g. { "a" => "b" } => 'host/part?a=b'. Give an array to pass multiple value like

["a", "b"], ["a", "c"]

> 'host/part?a=b&a=c'.

extheader

a Hash or an Array of extra headers. e.g. { 'Accept' => '/' } or [['Accept', 'image/jpeg'], ['Accept', 'image/png']].

&block

Give a block to get chunked message-body of response like get_content(uri) { |chunked_body| ... }. Size of each chunk may not be the same.

get_content follows HTTP redirect status (see HTTP::Status.redirect?) internally and try to retrieve content from redirected URL. See redirect_uri_callback= how HTTP redirection is handled.

If you need to get full HTTP response including HTTP status and headers, use get method. get returns HTTP::Message as a response and you need to follow HTTP redirect by yourself if you need.

# File lib/httpclient.rb, line 518
def get_content(uri, query = nil, extheader = {}, &block)
  follow_redirect(:get, uri, query, nil, extheader, &block).content
end
head(uri, query = nil, extheader = {}) click to toggle source

Sends HEAD request to the specified URL. See request for arguments.

# File lib/httpclient.rb, line 585
def head(uri, query = nil, extheader = {})
  request(:head, uri, query, nil, extheader)
end
head_async(uri, query = nil, extheader = {}) click to toggle source

Sends HEAD request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

# File lib/httpclient.rb, line 671
def head_async(uri, query = nil, extheader = {})
  request_async(:head, uri, query, nil, extheader)
end
no_proxy() click to toggle source

Returns NO_PROXY setting String if given.

# File lib/httpclient.rb, line 411
def no_proxy
  @no_proxy
end
no_proxy=(no_proxy) click to toggle source

Sets NO_PROXY setting String. no_proxy must be a comma separated String. Each entry must be 'host' or 'host:port' such as; HTTPClient#no_proxy = 'example.com,example.co.jp:443'

'localhost' is treated as a no_proxy site regardless of explicitly listed. HTTPClient checks given URI objects before accessing it. 'host' is tail string match. No IP-addr conversion.

You can use environment variable 'no_proxy' or 'NO_PROXY' for it.

Calling this method resets all existing sessions.

# File lib/httpclient.rb, line 426
def no_proxy=(no_proxy)
  @no_proxy = no_proxy
  reset_all
end
options(uri, extheader = {}, &block) click to toggle source

Sends OPTIONS request to the specified URL. See request for arguments.

# File lib/httpclient.rb, line 610
def options(uri, extheader = {}, &block)
  request(:options, uri, nil, nil, extheader, &block)
end
options_async(uri, extheader = {}) click to toggle source

Sends OPTIONS request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

# File lib/httpclient.rb, line 701
def options_async(uri, extheader = {})
  request_async(:options, uri, nil, nil, extheader)
end
post(uri, body = '', extheader = {}, &block) click to toggle source

Sends POST request to the specified URL. See request for arguments.

# File lib/httpclient.rb, line 595
def post(uri, body = '', extheader = {}, &block)
  request(:post, uri, nil, body, extheader, &block)
end
post_async(uri, body = nil, extheader = {}) click to toggle source

Sends POST request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

# File lib/httpclient.rb, line 683
def post_async(uri, body = nil, extheader = {})
  request_async(:post, uri, nil, body, extheader)
end
post_content(uri, body = nil, extheader = {}, &block) click to toggle source

Posts a content.

uri

a String or an URI object which represents an URL of web resource.

body

a Hash or an Array of body part. e.g. { "a" => "b" } => 'a=b'. Give an array to pass multiple value like

["a", "b"], ["a", "c"]

> 'a=b&a=c'.

When you pass a File as a value, it will be posted as a multipart/form-data. e.g. { 'upload' => file }

extheader

a Hash or an Array of extra headers. e.g. { 'Accept' => '/' } or [['Accept', 'image/jpeg'], ['Accept', 'image/png']].

&block

Give a block to get chunked message-body of response like post_content(uri) { |chunked_body| ... }. Size of each chunk may not be the same.

post_content follows HTTP redirect status (see HTTP::Status.redirect?) internally and try to post the content to redirected URL. See redirect_uri_callback= how HTTP redirection is handled.

If you need to get full HTTP response including HTTP status and headers, use post method.

# File lib/httpclient.rb, line 544
def post_content(uri, body = nil, extheader = {}, &block)
  follow_redirect(:post, uri, nil, body, extheader, &block).content
end
propfind(uri, extheader = PROPFIND_DEFAULT_EXTHEADER, &block) click to toggle source

Sends PROPFIND request to the specified URL. See request for arguments.

# File lib/httpclient.rb, line 615
def propfind(uri, extheader = PROPFIND_DEFAULT_EXTHEADER, &block)
  request(:propfind, uri, nil, nil, extheader, &block)
end
propfind_async(uri, extheader = PROPFIND_DEFAULT_EXTHEADER) click to toggle source

Sends PROPFIND request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

# File lib/httpclient.rb, line 707
def propfind_async(uri, extheader = PROPFIND_DEFAULT_EXTHEADER)
  request_async(:propfind, uri, nil, nil, extheader)
end
proppatch(uri, body = nil, extheader = {}, &block) click to toggle source

Sends PROPPATCH request to the specified URL. See request for arguments.

# File lib/httpclient.rb, line 620
def proppatch(uri, body = nil, extheader = {}, &block)
  request(:proppatch, uri, nil, body, extheader, &block)
end
proppatch_async(uri, body = nil, extheader = {}) click to toggle source

Sends PROPPATCH request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

# File lib/httpclient.rb, line 713
def proppatch_async(uri, body = nil, extheader = {})
  request_async(:proppatch, uri, nil, body, extheader)
end
proxy() click to toggle source

Returns URI object of HTTP proxy if exists.

# File lib/httpclient.rb, line 376
def proxy
  @proxy
end
proxy=(proxy) click to toggle source

Sets HTTP proxy used for HTTP connection. Given proxy can be an URI, a String or nil. You can set user/password for proxy authentication like HTTPClient#proxy = 'user:passwd@myproxy:8080'

You can use environment variable 'http_proxy' or 'HTTP_PROXY' for it. You need to use 'cgi_http_proxy' or 'CGI_HTTP_PROXY' instead if you run HTTPClient from CGI environment from security reason. (HTTPClient checks 'REQUEST_METHOD' environment variable whether it's CGI or not)

Calling this method resets all existing sessions.

# File lib/httpclient.rb, line 390
def proxy=(proxy)
  if proxy.nil?
    @proxy = nil
    @proxy_auth.reset_challenge
  else
    @proxy = urify(proxy)
    if @proxy.scheme == nil or @proxy.scheme.downcase != 'http' or
        @proxy.host == nil or @proxy.port == nil
      raise ArgumentError.new("unsupported proxy #{proxy}")
    end
    @proxy_auth.reset_challenge
    if @proxy.user || @proxy.password
      @proxy_auth.set_auth(@proxy.user, @proxy.password)
    end
  end
  reset_all
  @session_manager.proxy = @proxy
  @proxy
end
put(uri, body = '', extheader = {}, &block) click to toggle source

Sends PUT request to the specified URL. See request for arguments.

# File lib/httpclient.rb, line 600
def put(uri, body = '', extheader = {}, &block)
  request(:put, uri, nil, body, extheader, &block)
end
put_async(uri, body = nil, extheader = {}) click to toggle source

Sends PUT request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

# File lib/httpclient.rb, line 689
def put_async(uri, body = nil, extheader = {})
  request_async(:put, uri, nil, body, extheader)
end
redirect_uri_callback=(redirect_uri_callback) click to toggle source

Sets callback proc when HTTP redirect status is returned for get_content and post_content. default_redirect_uri_callback is used by default.

If you need strict implementation which does not allow relative URI redirection, set strict_redirect_uri_callback instead.

clnt.redirect_uri_callback = clnt.method(:strict_redirect_uri_callback)
# File lib/httpclient.rb, line 493
def redirect_uri_callback=(redirect_uri_callback)
  @redirect_uri_callback = redirect_uri_callback
end
request(method, uri, query = nil, body = nil, extheader = {}, &block) click to toggle source

Sends a request to the specified URL.

method

HTTP method to be sent. method.to_s.upcase is used.

uri

a String or an URI object which represents an URL of web resource.

query

a Hash or an Array of query part of URL. e.g. { "a" => "b" } => 'host/part?a=b' Give an array to pass multiple value like

["a", "b"], ["a", "c"]

> 'host/part?a=b&a=c'

body

a Hash or an Array of body part. e.g. { "a" => "b" } => 'a=b'. Give an array to pass multiple value like

["a", "b"], ["a", "c"]

> 'a=b&a=c'.

When the given method is 'POST' and the given body contains a file as a value, it will be posted as a multipart/form-data. e.g. { 'upload' => file } See HTTP::Message.file? for actual condition of 'a file'.

extheader

a Hash or an Array of extra headers. e.g. { 'Accept' => '/' } or [['Accept', 'image/jpeg'], ['Accept', 'image/png']].

&block

Give a block to get chunked message-body of response like get(uri) { |chunked_body| ... }. Size of each chunk may not be the same.

You can also pass a String as a body. HTTPClient just sends a String as a HTTP request message body.

When you pass an IO as a body, HTTPClient sends it as a HTTP request with chunked encoding (Transfer-Encoding: chunked in HTTP header). Bear in mind that some server application does not support chunked request. At least cgi.rb does not support it.

# File lib/httpclient.rb, line 659
def request(method, uri, query = nil, body = nil, extheader = {}, &block)
  uri = urify(uri)
  if block
    filtered_block = proc { |res, str|
      block.call(str)
    }
  end
  do_request(method, uri, query, body, extheader, &filtered_block)
end
request_async(method, uri, query = nil, body = nil, extheader = {}) click to toggle source

Sends a request in async style. request method creates new Thread for HTTP connection and returns a HTTPClient::Connection instance immediately.

Arguments definition is the same as request.

# File lib/httpclient.rb, line 727
def request_async(method, uri, query = nil, body = nil, extheader = {})
  uri = urify(uri)
  do_request_async(method, uri, query, body, extheader)
end
reset(uri) click to toggle source

Resets internal session for the given URL. Keep-alive connection for the site (host-port pair) is disconnected if exists.

# File lib/httpclient.rb, line 734
def reset(uri)
  uri = urify(uri)
  @session_manager.reset(uri)
end
reset_all() click to toggle source

Resets all of internal sessions. Keep-alive connections are disconnected.

# File lib/httpclient.rb, line 740
def reset_all
  @session_manager.reset_all
end
set_auth(domain, user, passwd) click to toggle source

Sets credential for Web server authentication.

domain

a String or an URI to specify where HTTPClient should use this

credential.  If you set uri to nil, HTTPClient uses this credential
wherever a server requires it.

user

username String.

passwd

password String.

You can set multiple credentials for each uri.

clnt.set_auth('http://www.example.com/foo/', 'foo_user', 'passwd')
clnt.set_auth('http://www.example.com/bar/', 'bar_user', 'passwd')

Calling this method resets all existing sessions.

# File lib/httpclient.rb, line 444
def set_auth(domain, user, passwd)
  uri = urify(domain)
  @www_auth.set_auth(uri, user, passwd)
  reset_all
end
set_basic_auth(domain, user, passwd) click to toggle source

Deprecated. Use set_auth instead.

# File lib/httpclient.rb, line 451
def set_basic_auth(domain, user, passwd)
  uri = urify(domain)
  @www_auth.basic_auth.set(uri, user, passwd)
  reset_all
end
set_proxy_auth(user, passwd) click to toggle source

Sets credential for Proxy authentication.

user

username String.

passwd

password String.

Calling this method resets all existing sessions.

# File lib/httpclient.rb, line 462
def set_proxy_auth(user, passwd)
  @proxy_auth.set_auth(user, passwd)
  reset_all
end
strict_redirect_uri_callback(uri, res) click to toggle source

A method for redirect uri callback. How to use:

clnt.redirect_uri_callback = clnt.method(:strict_redirect_uri_callback)

This callback does not allow relative redirect such as

Location: ../foo/

in HTTP header. (raises BadResponseError instead)

# File lib/httpclient.rb, line 553
def strict_redirect_uri_callback(uri, res)
  newuri = URI.parse(res.header['location'][0])
  if https?(uri) && !https?(newuri)
    raise BadResponseError.new("redirecting to non-https resource")
  end
  unless newuri.is_a?(URI::HTTP)
    raise BadResponseError.new("unexpected location: #{newuri}", res)
  end
  puts "redirect to: #{newuri}" if $DEBUG
  newuri
end
trace(uri, query = nil, body = nil, extheader = {}, &block) click to toggle source

Sends TRACE request to the specified URL. See request for arguments.

# File lib/httpclient.rb, line 625
def trace(uri, query = nil, body = nil, extheader = {}, &block)
  request('TRACE', uri, query, body, extheader, &block)
end
trace_async(uri, query = nil, body = nil, extheader = {}) click to toggle source

Sends TRACE request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

# File lib/httpclient.rb, line 719
def trace_async(uri, query = nil, body = nil, extheader = {})
  request_async(:trace, uri, query, body, extheader)
end

[Validate]

Generated with the Darkfish Rdoc Generator 2.