API Reference¶
Classes¶
-
class
HTTPConnection
(hostname, port=None, ssl_context=None, proxy=None)¶ This class encapsulates a HTTP connection.
Methods whose name begin with
co_
return coroutines. Instead of blocking, a coroutines will yield aPollNeeded
instance that encapsulates information about the IO operation that would block. The coroutine should be resumed once the operation can be performed without blocking.HTTPConnection
instances can be used as context managers. Thedisconnect
method will be called on exit from the managed block.-
co_discard
()¶ Read and discard current response body
This method returns a coroutine.
discard
is a regular method implementing the same functionality.
-
co_read
(len_=None)¶ Read up to len_ bytes of response body data
This method may return less than len_ bytes, but will return
b''
only if the response body has been read completely.If len_ is
None
, this method returns the entire response body.This method returns a coroutine.
read
is a regular method implementing the same functionality.
-
co_read_response
()¶ Read response status line and headers
Return a
HTTPResponse
instance containing information about response status, reason, and headers. The response body data must be retrieved separately (e.g. usingread
orreadall
).This method returns a coroutine.
read_response
is a regular method implementing the same functionality.
-
co_readall
()¶ Read and return complete response body
This method returns a coroutine.
readall
is a regular method implementing the same functionality.
-
co_readinto
(buf)¶ Read response body data into buf
Return the number of bytes written or zero if the response body has been read completely.
buf must implement the memoryview protocol.
This method returns a coroutine.
readinto
is a regular method implementing the same functionality.
-
co_send_request
(method, path, headers=None, body=None, expect100=False)¶ Send a new HTTP request to the server
The message body may be passed in the body argument or be sent separately. In the former case, body must be a bytes-like object. In the latter case, body must be an a
BodyFollowing
instance specifying the length of the data that will be sent. If no length is specified, the data will be send using chunked encoding.headers should be a mapping containing the HTTP headers to be send with the request. Multiple header lines with the same key are not supported. It is recommended to pass a
CaseInsensitiveDict
instance, other mappings will be converted toCaseInsensitiveDict
automatically.If body is a provided as a bytes-like object, a
Content-MD5
header is generated automatically unless it has been provided in headers already.This method returns a coroutine.
send_request
is a regular method implementing the same functionality.
-
co_write
(buf)¶ Write request body data
ExcessBodyData
will be raised when attempting to send more data than required to complete the request body of the active request.This method returns a coroutine.
write
is a regular method implementing the same functionality.
-
connect
()¶ Connect to the remote server
This method generally does not need to be called manually.
-
discard
()¶ Read and discard current response body
This method may block.
co_discard
provides a coroutine implementing the same functionality without blocking.
-
disconnect
()¶ Close HTTP connection
-
get_ssl_cipher
()¶ Get active SSL cipher
If plain HTTP is used, return
None
. Otherwise, the call is delegated to the underlying SSL socketscipher
method.
-
get_ssl_peercert
(binary_form=False)¶ Get peer SSL certificate
If plain HTTP is used, return
None
. Otherwise, the call is delegated to the underlying SSL socketsgetpeercert
method.
-
read
(len_=None)¶ Read up to len_ bytes of response body data
This method may return less than len_ bytes, but will return
b''
only if the response body has been read completely.If len_ is
None
, this method returns the entire response body.This method may block.
co_read
provides a coroutine implementing the same functionality without blocking.
-
read_raw
(size)¶ Read size bytes of uninterpreted data
This method may be used even after
UnsupportedResponse
orInvalidResponse
has been raised. It reads raw data from the socket without attempting to interpret it. This is probably only useful for debugging purposes to take a look at the raw data received from the server. This method blocks if no data is available, and returnsb''
if the connection has been closed.Calling this method will break the internal state and switch the socket to blocking operation. The connection has to be closed and reestablished afterwards.
Don’t use this method unless you know exactly what you are doing.
-
read_response
()¶ Read response status line and headers
Return a
HTTPResponse
instance containing information about response status, reason, and headers. The response body data must be retrieved separately (e.g. usingread
orreadall
).This method may block.
co_read_response
provides a coroutine implementing the same functionality without blocking.
-
readall
()¶ Read and return complete response body
This method may block.
co_readall
provides a coroutine implementing the same functionality without blocking.
-
readinto
(buf)¶ Read response body data into buf
Return the number of bytes written or zero if the response body has been read completely.
buf must implement the memoryview protocol.
This method may block.
co_readinto
provides a coroutine implementing the same functionality without blocking.
-
reset
()¶ Reset HTTP connection
This method resets the status of the HTTP connection after an exception has occured. Any cached data and pending responses are discarded.
-
response_pending
()¶ Return
True
if there are still outstanding responsesThis includes responses that have been partially read.
-
send_request
(method, path, headers=None, body=None, expect100=False)¶ Send a new HTTP request to the server
The message body may be passed in the body argument or be sent separately. In the former case, body must be a bytes-like object. In the latter case, body must be an a
BodyFollowing
instance specifying the length of the data that will be sent. If no length is specified, the data will be send using chunked encoding.headers should be a mapping containing the HTTP headers to be send with the request. Multiple header lines with the same key are not supported. It is recommended to pass a
CaseInsensitiveDict
instance, other mappings will be converted toCaseInsensitiveDict
automatically.If body is a provided as a bytes-like object, a
Content-MD5
header is generated automatically unless it has been provided in headers already.This method may block.
co_send_request
provides a coroutine implementing the same functionality without blocking.
-
write
(buf)¶ Write request body data
ExcessBodyData
will be raised when attempting to send more data than required to complete the request body of the active request.This method may block.
co_write
provides a coroutine implementing the same functionality without blocking.
-
timeout
= None¶ If a regular
HTTPConnection
method is unable to send or receive data for more than this period (in seconds), it will raiseConnectionTimedOut
. Coroutines are not affected by this attribute.
-
trace_fh
= None¶ Filehandler for tracing
-
-
class
HTTPResponse
(method, path, status, reason, headers, length=None)¶ This class encapsulates information about HTTP response. Instances of this class are returned by the
HTTPConnection.read_response
method and have access to response status, reason, and headers. Response body data has to be read directly from theHTTPConnection
instance.-
headers
= None¶ HTTP Response headers, a
email.message.Message
instance
-
length
= None¶ Length of the response body or
None
if not known. This attribute contains the actual length of the transmitted response. That means that for responses where RFC 2616 mandates that no request body be sent (e.g. in response to HEAD requests or for 1xx response codes) this value is zero. In these cases, the length of the body that would have been send can be extracted from theContent-Length
response header.
-
method
= None¶ HTTP Method of the request this was response is associated with
-
path
= None¶ Path of the request this was response is associated with
-
reason
= None¶ HTTP reason phrase returned by the server
-
status
= None¶ HTTP status code returned by the server
-
-
class
BodyFollowing
(length=None)¶ Sentinel class for the body parameter of the
send_request
method. Passing an instance of this class declares that body data is going to be provided in separate method calls.If no length is specified in the constructor, the body data will be send using chunked encoding.
-
class
CaseInsensitiveDict
(data=None, **kwargs)¶ A case-insensitive
dict
-like object.Implements all methods and operations of
collections.abc.MutableMapping
as well ascopy
.All keys are expected to be strings. The structure remembers the case of the last key to be set, and
iter()
,keys()
anditems()
will contain case-sensitive keys. However, querying and contains testing is case insensitive:cid = CaseInsensitiveDict() cid['Accept'] = 'application/json' cid['aCCEPT'] == 'application/json' # True list(cid) == ['Accept'] # True
For example,
headers['content-encoding']
will return the value of a'Content-Encoding'
response header, regardless of how the header name was originally stored.If the constructor,
update()
, or equality comparison operations are given multiple keys that have equal lower-case representions, the behavior is undefined.-
lower_items
()¶ Like
items()
, but with all lowercase keys.
-
-
class
PollNeeded
¶ This class encapsulates the requirements for a IO operation to continue.
PollNeeded
instances are typically yielded by coroutines.-
poll
(timeout=None)¶ Wait until fd is ready for requested IO
This is a convenince function that uses
poll
to wait untilfd
is ready for the requested type of IO. If poll is unavailable (e.g. on Windows) it usesselect
instead.If timeout is specified, return
False
if the timeout is exceeded without the file descriptor becoming ready.
-
fd
¶ File descriptor that the IO operation depends on
-
mask
¶ Event mask specifiying the type of required IO
This attribute defines what type of IO the provider of the
PollNeeded
instance needs to perform on fd. It is expected that, when fd is ready for IO of the specified type, operation will continue without blocking.The type of IO is specified as a poll compatible event mask, i.e. a bitwise combination of
select.POLLIN
andselect.POLLOUT
.
-
-
class
AioFuture
(crt, loop=None)¶ This class wraps a coroutine that yields
PollNeeded
instances into anasyncio
compatibleFuture
.This is done by registering a callback with the event loop that resumes the coroutine when the requested IO is available.
Functions¶
-
is_temp_network_error
(exc)¶ Return true if exc represents a potentially temporary network problem
DNS resolution errors (
socket.gaierror
orsocket.herror
) are considered permanent, becauseHTTPConnection
employs a heuristic to convert these exceptions toHostnameNotResolvable
orDNSUnavailable
instead.
Exceptions¶
Dugong functions may pass through any exceptions raised by the
socket and ssl.SSLSocket
methods. In
addition to that, the following dugong-specific exceptions may be
raised as well:
-
exception
ConnectionClosed
(msg=None)¶ Raised if the connection was unexpectedly closed.
This exception is raised also if the server declared that it will close the connection (by sending a
Connection: close
header). Such responses can still be read completely, but the next attempt to send a request or read a response will raise the exception. To re-use the connection after the server has closed the connection, callHTTPConnection.reset
before further requests are send.This behavior is intentional, because the caller may have already issued other requests (i.e., used pipelining). By raising an exception, the caller is notified that any pending requests have been lost and need to be resend.
-
exception
InvalidResponse
(msg=None)¶ Raised if the server produced an invalid response (i.e, something that is not proper HTTP 1.0 or 1.1).
-
exception
UnsupportedResponse
(msg=None)¶ This exception is raised if the server produced a response that is not supported. This should not happen for servers that are HTTP 1.1 compatible.
If an
UnsupportedResponse
exception has been raised, this typically means that synchronization with the server will be lost (i.e., dugong cannot determine where the current response ends and the next response starts), so the connection needs to be reset by calling thedisconnect()
method.
-
exception
ExcessBodyData
(msg=None)¶ Raised when trying to send more data to the server than announced.
-
exception
StateError
(msg=None)¶ Raised when attempting an operation that doesn’t make sense in the current connection state.
-
exception
ConnectionTimedOut
(msg=None)¶ Raised if a regular
HTTPConnection
method (i.e., no coroutine) was unable to send or receive data for the timeout specified in theHTTPConnection.timeout
attribute.
-
exception
HostnameNotResolvable
(hostname)¶ Raised if a host name does not resolve to an ip address.
Dugong raises this exception if a resolution attempt results in a
socket.gaierror
orsocket.herror
exception with errnosocket.EAI_AGAIN
orsocket.EAI_NONAME
, but at least one of the hostnames inDNS_TEST_HOSTNAMES
can be resolved.
Raised if the DNS server cannot be reached.
Dugong raises this exception if a resolution attempt results in a
socket.gaierror
orsocket.herror
exception with errnosocket.EAI_AGAIN
orsocket.EAI_NONAME
, and none of the hostnames inDNS_TEST_HOSTNAMES
can be resolved either.
Constants¶
-
MAX_LINE_SIZE
= 65535¶ Maximal length of HTTP status line. If the server sends a line longer than this value,
InvalidResponse
will be raised.
-
MAX_HEADER_SIZE
= 65535¶ Maximal length of a response header (i.e., for all header lines together). If the server sends a header segment longer than this value,
InvalidResponse
will be raised.
-
DNS_TEST_HOSTNAMES
= (('www.google.com', 80), ('www.iana.org', 80), ('C.root-servers.org', 53))¶ Sequence of
(hostname, port)
tuples that are used by dugong to distinguish between permanent and temporary name resolution problems.
Thread Safety¶
Dugong is not generally threadsafe. However, simultaneous use of the
same HTTPConnection
instance by two threads is supported if once
thread is restricted to sending requests, and the other thread
restricted to reading responses.
Avoiding Deadlocks¶
The HTTPConnection
class allows you to send an unlimited number of
requests to the server before reading any of the responses. However, at some
point the transmit and receive buffers on both the ends of the connection
will fill up, and no more requests can be send before at least some of the
responses are read, and attempts to send more data to the server will
block. If the thread that attempts to send data is is also responsible for
reading the responses, this will result in a deadlock.
There are several ways to avoid this:
- Do not send a new request before the last response has been read. This is the easiest solution, but it means that no HTTP pipelining can be used.
- Use different threads for sending requests and receiving responses.
- Use the coroutine based API (see Pipelining with Coroutines in the tutorial).