Network Working Group J. Elson Request for Comments: 3507 A. Cerpa Category: Informational UCLA April 2003 Internet Content Adaptation Protocol (ICAP) Status of this Memo This memo provides information for the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited. Copyright Notice Copyright (C) The Internet Society (2003). All Rights Reserved. IESG Note The Open Pluggable Services (OPES) working group has been chartered to produce a standards track protocol specification for a protocol intended to perform the same of functions as ICAP. However, since ICAP is already in widespread use the IESG believes it is appropriate to document existing usage by publishing the ICAP specification as an informational document. The IESG also notes that ICAP was developed before the publication of RFC 3238 and therefore does not address the architectural and policy issues described in that document. Abstract ICAP, the Internet Content Adaption Protocol, is a protocol aimed at providing simple object-based content vectoring for HTTP services. ICAP is, in essence, a lightweight protocol for executing a "remote procedure call" on HTTP messages. It allows ICAP clients to pass HTTP messages to ICAP servers for some sort of transformation or other processing ("adaptation"). The server executes its transformation service on messages and sends back responses to the client, usually with modified messages. Typically, the adapted messages are either HTTP requests or HTTP responses. Elson & Cerpa Informational [Page 1] RFC 3507 ICAP April 2003 Table of Contents 1. Introduction............................................3 2. Terminology.............................................5 3. ICAP Overall Operation..................................8 3.1 Request Modification..............................8 3.2 Response Modification............................10 4. Protocol Semantics.....................................11 4.1 General Operation................................11 4.2 ICAP URIs........................................11 4.3 ICAP Headers.....................................12 4.3.1 Headers Common to Requests and Responses................................12 4.3.2 Request Headers..........................13 4.3.3 Response Headers.........................14 4.3.4 ICAP-Related Headers in HTTP Messages.................................15 4.4 ICAP Bodies: Encapsulation of HTTP Messages.........................................16 4.4.1 Expected Encapsulated Sections...........16 4.4.2 Encapsulated HTTP Headers................18 4.5 Message Preview..................................18 4.6 "204 No Content" Responses outside of Previews.........................................22 4.7 ISTag Response Header............................22 4.8 Request Modification Mode........................23 4.8.1 Request..................................23 4.8.2 Response.................................24 4.8.3 Examples.................................24 4.9 Response Modification Mode.......................27 4.9.1 Request..................................27 4.9.2 Response.................................27 4.9.3 Examples.................................28 4.10 OPTIONS Method...................................29 4.10.1 OPTIONS request..........................29 4.10.2 OPTIONS response.........................30 4.10.3 OPTIONS examples.........................33 5. Caching................................................33 6. Implementation Notes...................................34 6.1 Vectoring Points.................................34 6.2 Application Level Errors.........................35 6.3 Use of Chunked Transfer-Encoding.................37 6.4 Distinct URIs for Distinct Services..............37 7. Security Considerations................................37 7.1 Authentication...................................37 7.2 Encryption.......................................38 7.3 Service Validation...............................38 8. Motivations and Design Alternatives....................39 Elson & Cerpa Informational [Page 2] RFC 3507 ICAP April 2003 8.1 To Be HTTP, or Not to Be.........................39 8.2 Mandatory Use of Chunking........................39 8.3 Use of the null-body directive in the Encapsulated header..............................40 9. References.............................................40 10. Contributors...........................................41 Appendix A BNF Grammar for ICAP Messages..................45 Authors' Addresses..........................................48 Full Copyright Statement....................................49 1. Introduction As the Internet grows, so does the need for scalable Internet services. Popular web servers are asked to deliver content to hundreds of millions of users connected at ever-increasing bandwidths. The model of centralized, monolithic servers that are responsible for all aspects of every client's request seems to be reaching the end of its useful life. To keep up with the growth in the number of clients, there has been a move towards architectures that scale better through the use of replication, distribution, and caching. On the content provider side, replication and load-balancing techniques allow the burden of client requests to be spread out over a myriad of servers. Content providers have also begun to deploy geographically diverse content distribution networks that bring origin-servers closer to the "edge" of the network where clients are attached. These networks of distributed origin-servers or "surrogates" allow the content provider to distribute their content whilst retaining control over the integrity of that content. The distributed nature of this type of deployment and the proximity of a given surrogate to the end-user enables the content provider to offer additional services to a user which might be based, for example, on geography where this would have been difficult with a single, centralized service. ICAP, the Internet Content Adaption Protocol, is a protocol aimed at providing simple object-based content vectoring for HTTP services. ICAP is, in essence, a lightweight protocol for executing a "remote procedure call" on HTTP messages. It allows ICAP clients to pass HTTP messages to ICAP servers for some sort of transformation or other processing ("adaptation"). The server executes its transformation service on messages and sends back responses to the client, usually with modified messages. The adapted messages may be either HTTP requests or HTTP responses. Though transformations may be possible on other non-HTTP content, they are beyond the scope of this document. Elson & Cerpa Informational [Page 3] RFC 3507 ICAP April 2003 This type of Remote Procedure Call (RPC) is useful in a number of ways. For example: o Simple transformations of content can be performed near the edge of the network instead of requiring an updated copy of an object from an origin server. For example, a content provider might want to provide a popular web page with a different advertisement every time the page is viewed. Currently, content providers implement this policy by marking such pages as non-cachable and tracking user cookies. This imposes additional load on the origin server and the network. In our architecture, the page could be cached once near the edges of the network. These edge caches can then use an ICAP call to a nearby ad-insertion server every time the page is served to a client. Other such transformations by edge servers are possible, either with cooperation from the content provider (as in a content distribution network), or as a value-added service provided by a client's network provider (as in a surrogate). Examples of these kinds of transformations are translation of web pages to different human languages or to different formats that are appropriate for special physical devices (e.g., PDA-based or cell-phone-based browsers). o Surrogates or origin servers can avoid performing expensive operations by shipping the work off to other servers instead. This helps distribute load across multiple machines. For example, consider a user attempting to download an executable program via a surrogate (e.g., a caching proxy). The surrogate, acting as an ICAP client, can ask an external server to check the executable for viruses before accepting it into its cache. o Firewalls or surrogates can act as ICAP clients and send outgoing requests to a service that checks to make sure the URI in the request is allowed (for example, in a system that allows parental control of web content viewed by children). In this case, it is a *request* that is being adapted, not an object returned by a response. In all of these examples, ICAP is helping to reduce or distribute the load on origin servers, surrogates, or the network itself. In some cases, ICAP facilitates transformations near the edge of the network, allowing greater cachability of the underlying content. In other examples, devices such as origin servers or surrogates are able to reduce their load by distributing expensive operations onto other machines. In all cases, ICAP has also created a standard interface for content adaptation to allow greater flexibility in content distribution or the addition of value added services in surrogates. Elson & Cerpa Informational [Page 4] RFC 3507 ICAP April 2003 There are two major components in our architecture: 1. Transaction semantics -- "How do I ask for adaptation?" 2. Control of policy -- "When am I supposed to ask for adaptation, what kind of adaptation do I ask for, and from where?" Currently, ICAP defines only the transaction semantics. For example, this document specifies how to send an HTTP message from an ICAP client to an ICAP server, specify the URI of the ICAP resource requested along with other resource-specific parameters, and receive the adapted message. Although a necessary building-block, this wire-protocol defined by ICAP is of limited use without the second part: an accompanying application framework in which it operates. The more difficult policy issue is beyond the scope of the current ICAP protocol, but is planned in future work. In initial implementations, we expect that implementation-specific manual configuration will be used to define policy. This includes the rules for recognizing messages that require adaptation, the URIs of available adaptation resources, and so on. For ICAP clients and servers to interoperate, the exact method used to define policy need not be consistent across implementations, as long as the policy itself is consistent. IMPORTANT: Note that at this time, in the absence of a policy-framework, it is strongly RECOMMENDED that transformations SHOULD only be performed on messages with the explicit consent of either the content-provider or the user (or both). Deployment of transformation services without the consent of either leads to, at best, unpredictable results. For more discussion of these issues, see Section 7. Once the full extent of the typical policy decisions are more fully understood through experience with these initial implementations, later follow-ons to this architecture may define an additional policy control protocol. This future protocol may allow a standard policy definition interface complementary to the ICAP transaction interface defined here. 2. Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14, RFC 2119 [2]. Elson & Cerpa Informational [Page 5] RFC 3507 ICAP April 2003 The special terminology used in this document is defined below. The majority of these terms are taken as-is from HTTP/1.1 [4] and are reproduced here for reference. A thorough understanding of HTTP/1.1 is assumed on the part of the reader. connection: A transport layer virtual circuit established between two programs for the purpose of communication. message: The basic unit of HTTP communication, consisting of a structured sequence of octets matching the syntax defined in Section 4 of HTTP/1.1 [4] and transmitted via the connection. request: An HTTP request message, as defined in Section 5 of HTTP/1.1 [4]. response: An HTTP response message, as defined in Section 6 of HTTP/1.1 [4]. resource: A network data object or service that can be identified by a URI, as defined in Section 3.2 of HTTP/1.1 [4]. Resources may be available in multiple representations (e.g., multiple languages, data formats, size, resolutions) or vary in other ways. client: A program that establishes connections for the purpose of sending requests. server: An application program that accepts connections in order to service requests by sending back responses. Any given program may be capable of being both a client and a server; our use of these terms refers only to the role being performed by the program for a particular connection, rather than to the program's capabilities in general. Likewise, any server may act as an origin server, surrogate, gateway, or tunnel, switching behavior based on the nature of each request. origin server: The server on which a given resource resides or is to be created. Elson & Cerpa Informational [Page 6] RFC 3507 ICAP April 2003 proxy: An intermediary program which acts as both a server and a client for the purpose of making requests on behalf of other clients. Requests are serviced internally or by passing them on, with possible translation, to other servers. A proxy MUST implement both the client and server requirements of this specification. cache: A program's local store of response messages and the subsystem that controls its message storage, retrieval, and deletion. A cache stores cachable responses in order to reduce the response time and network bandwidth consumption on future, equivalent requests. Any client or server may include a cache, though a cache cannot be used by a server that is acting as a tunnel. cachable: A response is cachable if a cache is allowed to store a copy of the response message for use in answering subsequent requests. The rules for determining the cachability of HTTP responses are defined in Section 13 of [4]. Even if a resource is cachable, there may be additional constraints on whether a cache can use the cached copy for a particular request. surrogate: A gateway co-located with an origin server, or at a different point in the network, delegated the authority to operate on behalf of, and typically working in close co-operation with, one or more origin servers. Responses are typically delivered from an internal cache. Surrogates may derive cache entries from the origin server or from another of the origin server's delegates. In some cases a surrogate may tunnel such requests. Where close co-operation between origin servers and surrogates exists, this enables modifications of some protocol requirements, including the Cache-Control directives in [4]. Such modifications have yet to be fully specified. Devices commonly known as "reverse proxies" and "(origin) server accelerators" are both more properly defined as surrogates. New definitions: ICAP resource: Similar to an HTTP resource as described above, but the URI refers to an ICAP service that performs adaptations of HTTP messages. Elson & Cerpa Informational [Page 7] RFC 3507 ICAP April 2003 ICAP server: Similar to an HTTP server as described above, except that the application services ICAP requests. ICAP client: A program that establishes connections to ICAP servers for the purpose of sending requests. An ICAP client is often, but not always, a surrogate acting on behalf of a user. 3. ICAP Overall Operation Before describing ICAP's semantics in detail, we will first give a general overview of the protocol's major functions and expected uses. As described earlier, ICAP focuses on modification of HTTP requests (Section 3.1), and modification of HTTP responses (Section 3.2). 3.1 Request Modification In "request modification" (reqmod) mode, an ICAP client sends an HTTP request to an ICAP server. The ICAP server may then: 1) Send back a modified version of the request. The ICAP client may then perform the modified request by contacting an origin server; or, pipeline the modified request to another ICAP server for further modification. 2) Send back an HTTP response to the request. This is used to provide information useful to the user in case of an error (e.g., "you sent a request to view a page you are not allowed to see"). 3) Return an error. ICAP clients MUST be able to handle all three types of responses. However, in line with the guidance provided for HTTP surrogates in Section 13.8 of [4], ICAP client implementors do h