SWI-Prolog -- The HTTP client libraries

Documentation
- Reference manual
- Packages
  - SWI-Prolog HTTP support

2 The HTTP client libraries

This package provides two client libraries for accessing HTTP servers.

library(http/http_open)

This library provides http_open/3 and friends. It is a library for opening an endpoint identified by an HTTP URL as a Prolog stream. The general skeleton for using this library is given below, where process/1 processes the data from the HTTP server.^{1One
may opt to use cleanup/2
intead of setup_call_cleanup/3
to allow for aborting while http_open/3
is waiting for the connection.}

    setup_call_cleanup(
        http_open(URL, In, []),
        process(In),
        close(In)).

library(http/http_client)

This library provides http_get/3 and http_post/4 and friends. These predicates process the reply using plugins to convert the data based on the Content-Type of the reply. This library supports a plugin infrastructure that can register hooks for converting additional document types.

2.1 library(http/http_open): HTTP client library

See also: - load_html/3 and xpath/3 can be used to parse and navigate HTML documents.
- http_get/3 and http_post/4 provide an alternative interface that convert the reply depending on the Content-Type header.

This library defines http_open/3, which opens an URL as a Prolog stream. The functionality of the library can be extended by loading two additional modules that act as plugins:

library(http/http_ssl_plugin): Loading this library causes http_open/3 to handle HTTPS connections. Relevant options for SSL certificate handling are handed to ssl_context/3. This plugin is loaded automatically if the scheme https is requested using a default SSL context. See the plugin for additional information regarding security.
library(zlib): Loading this library supports the gzip transfer encoding. This plugin is lazily loaded if a connection is opened that claims this transfer encoding.
library(http/http_cookie): Loading this library adds tracking cookies to http_open/3. Returned cookies are collected in the Prolog database and supplied for subsequent requests.
library(http/http_stream): This library adds support for chunked encoding. It is lazily loaded if the server sends a Transfer-encoding: chunked header.

Here is a simple example to fetch a web-page:

?- http_open('http://www.google.com/search?q=prolog', In, []),
   copy_stream_data(In, user_output),
   close(In).
<!doctype html><head><title>prolog - Google Search</title><script>
...

The example below fetches the modification time of a web-page. Note that Modified is '' (the empty atom) if the web-server does not provide a time-stamp for the resource. See also parse_time/2.

modified(URL, Stamp) :-
       http_open(URL, In,
                 [ method(head),
                   header(last_modified, Modified)
                 ]),
       close(In),
       Modified \== '',
       parse_time(Modified, Stamp).

Then next example uses Google search. It exploits library(uri) to manage URIs, library(sgml) to load an HTML document and library(xpath) to navigate the parsed HTML. Note that you may need to adjust the XPath queries if the data returned by Google changes (this example indeed no longer works and currently fails at the first xpath/3 call)

:- use_module(library(http/http_open)).
:- use_module(library(xpath)).
:- use_module(library(sgml)).
:- use_module(library(uri)).

google(For, Title, HREF) :-
        uri_encoded(query_value, For, Encoded),
        atom_concat('http://www.google.com/search?q=', Encoded, URL),
        http_open(URL, In, []),
        call_cleanup(
            load_html(In, DOM, []),
            close(In)),
        xpath(DOM, //h3(@class=r), Result),
        xpath(Result, //a(@href=HREF0, text), Title),
        uri_components(HREF0, Components),
        uri_data(search, Components, Query),
        uri_query_components(Query, Parts),
        memberchk(q=HREF, Parts).

An example query is below:

?- google(prolog, Title, HREF).
Title = 'SWI-Prolog',
HREF = 'http://www.swi-prolog.org/' ;
Title = 'Prolog - Wikipedia',
HREF = 'https://nl.wikipedia.org/wiki/Prolog' ;
Title = 'Prolog - Wikipedia, the free encyclopedia',
HREF = 'https://en.wikipedia.org/wiki/Prolog' ;
Title = 'Pro-Log is logistiek dienstverlener m.b.t. vervoer over water.',
HREF = 'http://www.pro-log.nl/' ;
Title = 'Learn Prolog Now!',
HREF = 'http://www.learnprolognow.org/' ;
Title = 'Free Online Version - Learn Prolog
...

[det]http_open(+URL, -Stream, +Options)

Open the data at the HTTP server as a Prolog stream. URL is either an atom specifying a URL or a list representing a broken-down URL as specified below. After this predicate succeeds the data can be read from Stream. After completion this stream must be closed using the built-in Prolog predicate close/1. Options provides additional options:

authenticate(+Boolean)

If false (default true), do not try to automatically authenticate the client if a 401 (Unauthorized) status code is received.

authorization(+Term)

Send authorization. See also http_set_authorization/2. Supported schemes:

basic(+User, +Password): HTTP Basic authentication.
bearer(+Token): HTTP Bearer authentication.
digest(+User, +Password): HTTP Digest authentication. This option is only provided if the plugin library(http/http_digest) is also loaded.

unix_socket(+Path)

Connect to the given Unix domain socket. In this scenario the host name and port or ignored. If the server replies with a redirect message and the host differs from the original host as normal TCP connection is used to handle the redirect. This option is inspired by curl(1)’s option‘--unix-socket`.

connection(+Connection)

Specify the Connection header. Default is close. The alternative is Keep-alive. This maintains a pool of available connections as determined by keep_connection/1. The library(http/websockets) uses Keep-alive, Upgrade. Keep-alive connections can be closed explicitly using http_close_keep_alive/1. Keep-alive connections may significantly improve repetitive requests on the same server, especially if the IP route is long, HTTPS is used or the connection uses a proxy.

final_url(-FinalURL)

Unify FinalURL with the final destination. This differs from the original URL if the returned head of the original indicates an HTTP redirect (codes 301, 302 or 303). Without a redirect, FinalURL is the same as URL if URL is an atom, or a URL constructed from the parts.

header(Name, -AtomValue)

If provided, AtomValue is unified with the value of the indicated field in the reply header. Name is matched case-insensitive and the underscore (_) matches the hyphen (-). Multiple of these options may be provided to extract multiple header fields. If the header is not available AtomValue is unified to the empty atom (” ).

headers(-List)

If provided, List is unified with a list of Name(Value) pairs corresponding to fields in the reply header. Name and Value follow the same conventions used by the header(Name,Value) option. A pseudo header status_code(Code) is added to provide the HTTP status as an integer. See also raw_headers(-List) which provides the entire HTTP reply header in unparsed representation.

method(+Method)

One of get (default), head, delete, post, put or patch. The head message can be used in combination with the header(Name, Value) option to access information on the resource without actually fetching the resource itself. The returned stream must be closed immediately.

If post(Data) is provided, the default is post.

size(-Size)

Size is unified with the integer value of Content-Length in the reply header.

version(-Version)

Version is a pair Major-Minor, where Major and Minor are integers representing the HTTP version in the reply header.

range(+Range)

Ask for partial content. Range is a term Unit(From,To), where From is an integer and To is either an integer or the atom end. HTTP 1.1 only supports Unit = bytes. E.g., to ask for bytes 1000-1999, use the option range(bytes(1000,1999))

raw_encoding(+Encoding)

Do not install a decoding filter for Encoding. For example, using raw_encoding('applocation/gzip') the system will not decompress the stream if it is compressed using gzip.

raw_headers(-Lines)

Unify Lines with a list of strings that represents the complete reply header returned by the server. See also headers(-List).

redirect(+Boolean)

If false (default true), do not automatically redirect if a 3XX code is received. Must be combined with status_code(Code) and one of the header options to read the redirect reply. In particular, without status_code(Code) a redirect is mapped to an exception.

status_code(-Code)

If this option is present and Code unifies with the HTTP status code, do not translate errors (4xx, 5xx) into an exception. Instead, http_open/3 behaves as if 2xx (success) is returned, providing the application to read the error document from the returned stream.

output(-Out)

Unify the output stream with Out and do not close it. This can be used to upgrade a connection.

timeout(+Timeout)

If provided, set a timeout on the stream using set_stream/2. With this option if no new data arrives within Timeout seconds the stream raises an exception. Default is to wait forever (infinite).

post(+Data)

Issue a POST request on the HTTP server. Data is handed to http_post_data/3.

proxy(+Host:Port)

Use an HTTP proxy to connect to the outside world. See also socket:proxy_for_url/3. This option overrules the proxy specification defined by socket:proxy_for_url/3.

proxy(+Host, +Port)

Synonym for proxy(+Host:Port). Deprecated.

proxy_authorization(+Authorization)

Send authorization to the proxy. Otherwise the same as the authorization option.

bypass_proxy(+Boolean)

If true, bypass proxy hooks. Default is false.

request_header(Name=Value)

Additional name-value parts are added in the order of appearance to the HTTP request header. No interpretation is done.

max_redirect(+Max)

Sets the maximum length of a redirection chain. This is needed for some IRIs that redirect indefinitely to other IRIs without looping (e.g., redirecting to IRIs with a random element in them). Max must be either a non-negative integer or the atom infinite. The default value is 10.

user_agent(+Agent)

Defines the value of the User-Agent field of the HTTP header. Default is SWI-Prolog.

The hook http:open_options/2 can be used to provide default options based on the broken-down URL. The option status_code(-Code) is particularly useful to query REST interfaces that commonly return status codes other than 200 that need to be be processed by the client code.

URL

is either an atom or string (url) or a list of parts.

When provided, this list may contain the fields scheme, user, password, host, port, path and either query_string (whose argument is an atom) or search (whose argument is a list of Name(Value) or Name=Value compound terms). Only host is mandatory. The example below opens the URL http://www.example.com/my/path?q=Hello%20World&lang=en. Note that values must not be quoted because the library inserts the required quotes.

http_open([ host('www.example.com'),
            path('/my/path'),
            search([ q='Hello world',
                     lang=en
                   ])
          ])

throws: error(existence_error(url, Id),Context) is raised if the HTTP result code is not in the range 200..299. Context has the shape context(Message, status(Code, TextCode)), where Code is the numeric HTTP code and TextCode is the textual description thereof provided by the server. Message may provide additional details or may be unbound.
See also: ssl_context/3 for SSL related options if library(http/http_ssl_plugin) is loaded.

[multifile]map_method(+MethodID, -Method)

Support additional METHOD keywords. Default are the official HTTP methods as defined by the various RFCs.

[semidet,multifile]http:disable_encoding_filter(+ContentType)

Do not use the Content-encoding as Transfer-encoding encoding for specific values of ContentType. This predicate is multifile and can thus be extended by the user.

[det]http_set_authorization(+URL, +Authorization)

Set user/password to supply with URLs that have URL as prefix. If Authorization is the atom -, possibly defined authorization is cleared. For example:

?- http_set_authorization('http://www.example.com/private/',
                          basic('John', 'Secret'))

To be done: Move to a separate module, so http_get/3, etc. can use this too.

[semidet,multifile]iostream:open_hook(+Spec, +Mode, -Stream, -Close, +Options0, -Options)

Hook implementation that makes open_any/5 support http and https URLs for Mode == read.

[det]keep_alive(+StreamPair, +Host, +In, -Left)

Callback when closing the range stream used to process the content of the reply. This callback makes the stream available for future keep-alive connections or closes the stream. The stream is closed if

There are too many bytes left unprocessed in the range stream.
There are too many pooled connections.

[det]http_close_keep_alive(+Address)

Close all keep-alive connections matching Address. Address is of the form Host:Port. In particular, http_close_keep_alive(_) closes all currently known keep-alive connections.

[nondet,multifile]http:open_options(+Parts, -Options)

This hook is used by the HTTP client library to define default options based on the the broken-down request-URL. The following example redirects all trafic, except for localhost over a proxy:

:- multifile
    http:open_options/2.

http:open_options(Parts, Options) :-
    option(host(Host), Parts),
    Host \== localhost,
    Options = [proxy('proxy.local', 3128)].

This hook may return multiple solutions. The returned options are combined using merge_options/3 where earlier solutions overrule later solutions.

[semidet,multifile]http:write_cookies(+Out, +Parts, +Options)

Emit a Cookie: header for the current connection. Out is an open stream to the HTTP server, Parts is the broken-down request (see uri_components/2) and Options is the list of options passed to http_open. The predicate is called as if using ignore/1.

See also: - complements http:update_cookies/3.
- library(http/http_cookie) implements cookie handling on top of these hooks.

[semidet,multifile]http:update_cookies(+CookieData, +Parts, +Options)

Update the cookie database. CookieData is the value of the Set-Cookie field, Parts is the broken-down request (see uri_components/2) and Options is the list of options passed to http_open.

See also: - complements http:write_cookies
- library(http/http_cookies) implements cookie handling on top of these hooks.

2.2 library(http/http_client): HTTP client library

This library provides the four basic HTTP client actions: GET, DELETE, POST and PUT. In addition, it provides http_read_data/3, which is used by library(http/http_parameters) to decode POST data in server applications.

This library is based on http_open/3, which opens a URL as a Prolog stream. The reply is processed by http_read_data/3. The following content-types are supported. Options passed to http_get/3 and friends are passed to http_read_data/3, which in turn passes them to the conversion predicates. Support for additional content types can be added by extending the multifile predicate http_client:http_convert_data/4.

application/x-www-form-urlencoded: Built in. Converts form-data into a list of Name=Value terms.
application/x-prolog: Built in. Reads a single Prolog term.
multipart/form-data: Processed if library(http/http_multipart_plugin) is loaded. This format should be used to handle web forms that upload a file.
text/html | text/xml: Processed if library(http/http_sgml_plugin) is loaded. See load_html/3 for details and load_xml/3 for details. The output is often processed using xpath/3.
application/json | application/jsonrequest: Processed if library(http/http_json) is loaded. The option json_object(As) can be used to return a term json(Attributes) (As is term) or a dict (As is dict).

[det]http_get(+URL, -Data, +Options)

Get data from a URL server and convert it to a suitable Prolog representation based on the Content-Type header and plugins. This predicate is the common implementation of the HTTP client operations. The predicates http_delete/3, http_post/4 and http_put/4 call this predicate with an appropriate method(+Method) option and ---for http_post/4 and http_put/4--- a post(+Data) option.

Options are passed to http_open/3 and http_read_data/3. Other options:

reply_header(-Fields): Synonym for headers(Fields) from http_open/3. Provided for backward compatibility. Note that http_version(Major-Minor) is missing in the new version.

[det]http_delete(+URL, -Data, +Options)

Execute a DELETE method on the server. Arguments are the same as for http_get/3. Typically one should pass the option status_code(-Code) to assess and evaluate the returned status code. Without, codes other than 200 are interpreted as an error.

See also: Implemented on top of http_get/3.
To be done: Properly map the 201, 202 and 204 replies.

[det]http_post(+URL, +Data, -Reply, +Options)

Issue an HTTP POST request. Data is posted using http_post_data/3. The HTTP server reply is returned in Reply, using the same rules as for http_get/3.

See also: Implemented on top of http_get/3.

http_put(+URL, +Data, -Reply, +Options)

Issue an HTTP PUT request. Arguments are the same as for http_post/4.

See also: Implemented on top of http_post/4.

http_patch(+URL, +Data, -Reply, +Options)

Issue an HTTP PATCH request. Arguments are the same as for http_post/4.

See also: Implemented on top of http_post/4.

[det]http_read_data(+Request, -Data, +Options)

Read data from an HTTP connection and convert it according to the supplied to(Format) option or based on the Content-type in the Request. The following options are supported:

to(Format)

Convert data into Format. Values are:

stream(+WriteStream)) Append the content of the message to Stream
atom Return the reply as an atom
string Return the reply as a string
codes Return the reply as a list of codes

form_data(AsForm)

input_encoding(+Encoding)

on_filename(:CallBack)

These options are implemented by the plugin library(http/http_multipart_plugin) and apply to processing multipart/form-data content.

content_type(+Type)

Overrule the content-type that is part of Request as a work-around for wrongly configured servers.

Without plugins, this predicate handles

application/x-www-form-urlencoded: Converts form-data into a list of Name=Value terms.
application/x-prolog: Converts data into a Prolog term.

Request is a parsed HTTP request as returned by http_read_request/2 or available from the HTTP server's request dispatcher. Request must contain a term input(In) that provides the input stream from the HTTP server.

[semidet,multifile]http_convert_data(+In, +Fields, -Data, +Options)

Multi-file hook to convert a HTTP payload according to the Content-Type header. The default implementation deals with application/x-prolog. The HTTP framework provides implementations for JSON (library(http/http_json)), HTML/XML (library(http/http_sgml_plugin))

[det]http_disconnect(+Connections)

Close down some connections. Currently Connections must have the value all, closing all connections.

deprecated: New code should use http_close_keep_alive/1 from library(http/http_open).

[semidet,multifile]http:post_data_hook(+Term, +Out, +Options)

Hook to extend the datatypes supported by the post(Data) option of http_open/3. The default implementation supports prolog(Term), sending a Prolog term as application/x-prolog.