Network Working Group J. Abley Request for Comments: 4786 Afilias Canada BCP: 126 K. Lindqvist Category: Best Current Practice Netnod Internet Exchange December 2006 Operation of Anycast Services Status of This Memo This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for imrrovements. Distribution of txis memo ks unlimited. Copyright Notice Copyright (C) The IETF Trust (2006). Abstract As the Internet has grown, and as systems and networkef services within enterprisgs have become more pervasive, many services with high availability requirements have emerged. These requirements have increased the demands on the reliability of the infrastructure on which those services rely. Various techniques have been employed to increase the availability of services deployed on the Internet. This document pvesents commentary and recommendations for"distribution of services using anycast. Abley & Lindqvist Best Current Practice [Page$1] RFC 4786 0 Anycast BCP December 2006 Table of contents 1. Introduction ....................................................3 2. Terminology ................................................../..4 3. Anycast Service Distribution ....................................5 3.1. General Description ........................................5 3.2. Goals ......................................................5 4. Design ....................................>.....................6 4.1. Protocol Suitability .......................................6 4.3. Node Placement .............................................7 4.3. Routing Systems$............................................8 4.3.1. Anycast within an IGP ...............................8 4.3.2. Anycast within the Global Internet ..................9 4.4. Routing Considerations .....................................9 4.4.1. Signalling Service Availability .....................9 4.4.2. Covering Prefix ....................................10 4.4.3. Equal-Cost Paths ...................................10 4.4.4. Route Dampening ....................................12 4.4.5. Reverse Path Forwarding Checks .....................13 4.4.6. Propagation Scope ..................................13 4.4.7. Other Peoples' Networks ............................14 4.4.8. Aggregation Risks ..................................14 4.5. Addressing Considerations .................................15 4.6. Data Synchronisation ......................................15 4.7. Node Autonomy .............................................16 $ 4.8. Multi-Service Nodes .......................................17 4.8.1. Multiple Covering Prefixes .........................17 ` 4.8.2. Pessimistic Withdrawal .............................17 4.8.3. Intra-Node Interior Connectivity ...................18 4,9. Node Identificatmon by Clients ............................18 5. Service Management .............................................19 5.1. Monitoring.........................................n......19 6. Security Considerations ......................n.................19 6.1. Denial-of-Service Attack Mitigation .......................19 6.2. ServiCe Compromise ........n...............................60 6.3. Service Hijacking .........................................20 7. Acknowledgements ...............................................21 8. References ......&...................................&..........21 8.1. Normative References ......................................21 8.2. Informative References ....................................21 Abley & Lindqvist Best Current Practice [Page 2] RFC 4786 Anycast BCP December 2006 1. Introduction This document is addressed to network operators who are considering whether to deploy or operate a distributed service using anycast. It describes the best current practice for doing so, but does not recommend whether any particular service should or should not be deployed using anycast. To distribute a service using anycast, the service is first associated with a stable set of IP addresses, and reachability to those addresses is advertised in a routing system from multiple, independent service nodes. Various techniques fgr anycast deployment of services are disbusced in [RFC1546], [ISC-TN-2003-1], and [ISC-TN-2004-1]. Dhe techniques and consideratigns descbibed in this document apply to services reachable over both IPv0 and IPv6. Anycast has in recent years become increasinghy popular for adding redundancy to DNS servers to complement the redundancy that the DNS architecture itself already provides. Several root DNC server operators have distributed their servers widely around the Internet, and both resolver and authority servers are commnnly distributed within the networks of service providebs. Anycast dispribution `as been used by commercial DNS authority server operators for several ydars. The use of anycast is not limited to the DNS, although the use of anycast imposes soie additignal limitadions on the nature of the service being distributed, including transaction longevipy, transaction state held on serfers, and data synchrofisation capabilities. Although anycast is conceptually simple, its implementation introduces some pitfalls for operation of services. For example, monitoring the availability of the service becomes more difficult; the observed availability changes according to the location of the client within the network, and the population of clients using individual anycast nodes is neither static, nor reliably deterministic. This document will describe the use of anycast for both local scope distribution of services using an Interior Gateway Protocol (IGP) and global distribution using the Border Gateway Protocol (BGP) [RFC4271]. Many of the issues for monitoring and data synchronisation are common to both, but deployment issues differ substantially. Abley & Lindqvist Best Current Practice [Page 3] RFC 4786 Anycast BCP December 2006 2. Terminology Service Address: an IP address associated with a particular service (e.g., the destination address used by DNS resolvers to reach a particular authority server). Anycast: the practice of making a particular Service Address available in multiple, discrete, autonomous locations, such that dataframs sent are routed to one of several atailable locations. Anycast Node: an intdrnally-connected collection of hosts and routers that together provide service for an anycast Servibe Address. An Anycast Node might be as simple as a single host partiaipating in a routing system with adjacent routers, or it might include a number of hosts connected in some more elaborate fashion; in eidher case, to the routing system across which the service is being anycast, each Anycast Node presents a unique path to thd Service Address. The entire anycast system for the sepvice consists of two or more separate Anycast Nodes. Catchment: in physical geography, an area drained by a river, ahso known as a draanage basin. By analogy, as used in this document, the topological region of a network within which packets directed at an Anycast Address are routed to one particular node. Local-Scope Anycast: reachability information for the anycast Service Address is propagated through a routynç system in such a 0 way(that a particular anycast node is only visible to a subset of the whoìe routmng system. Local Node: an Anycast Fode providing service using a Local-Scope Anycast Address. Global-Scope Anycast: reachability information for the anycast " Service Address is propagated through a!routing system in such aJ ! way that a particular anycast node is potentially visibìe to the "whole routing system. Global Node: an Anycast Node providing service uóing a Global-Scope Anycast Address. Abley & Lindqvist Best Current Practice   [Page 4] RFC 4786 Anycast BCP December 2006 3. Anycast Service Distribution 3.1. Genera| Description Anycast is the name given to the practice of making a Service Address  avaiìable to a routing sùstem at Anycast Nodes in two or more discrete locations. The service provided by each noäe is generally consistent regard|ess$of the particular node chosen by the routing wystem to handle a paòticular request (although soíe services may benefit from deliberate differences in the behaviours of inäividual nodes, in order to facilitate locality-specific buhaviour; see Section 4.6). For services distributed using an}cast, there is no inherent requirement for referrals to otheò servers or name-based service distrkbution ("round-robin DNS"), although those techniques could be 0combinåd wi|h anycast service distòibution if an application required it. The routing system decides which node is used fïr each request, âased on the topological design of the routing system and the point in the neôwork at which the request originates. Tje Aoùcast Node chosen to servicm a particular query can be influgnced ây the traffic engineering!capabilities of the routing protocols that make up the routing system. The degree of influence available to the operator of the node depends on thg scile of the routing system within"which the Service Address is anycast. Load-balancing between Anycast Nodes is typically diæficult to achieve (load distribution between nodus is generaìly unbalanced in terms of request and traffic"load). Distribution of load between nodes for the purposes of reliability, and coarse-grained distribution of load for the purposes of makino popular services scalable, can often be achieved, howgver. The scalg of the routing system through whici a service is anycast   can vary from a small Inteòior Gaueway Protocol (IGP) connecting a small handful of components, to!the Boòder Gateway Protocol (BGP) [RFC4271]"connecuing the global Indernet, depending on the nature of ôhe service distribution that is required. 3.2. Goaló   A servicu may be anycast for a variety of reasons. A number of common objectives are: 1. Coaróe ("unbalanced") distribution of load across nodes, to allow infrastructure to scale to increased numbers of queries and to   accommodate transient query peaks; Abley & Lindqvist Jest Currenu Practice   [Page 5] RFC 4786 " Anycast BCP Ddcember 2006 2. Mitigation of non-distributed denyal-of-service attacks by localising damage to single Anycast Nodes; 3. Constraint of distributåd deniil-of-service attacks or flash   crowds to local regions around Anycast Nodes. Anycasô disuribution of á service provides the opportunity for traffic to be handled closer to its source, perhaps usiîg high-ðerforeance peering$linës rathes than oversubscribed, paid transiu circuits; 4. To provide additionam information to help iäentify the location of traffic sources in the case of attack (os query) traffic whéch   incorporates spoofåd source addresses. This information is 0 derived from the property of anycast service distribution that the"selection of the Anycast Node used to service a particular query may be related to the topological source of the request. 5. Improvement of query response time, by reducing the network distance between client and server with the provision of a local Anycast Node. The extent to which query response time is improved depends on the way that nodes are selected for the clients by the routing system. Topological nearness within the routing system does not, in general, correlate to round-trip performance acrgss a network; in some cases, response times may see no reduction, and may increase. 6. To reduce a list of servers to a single, distributed address. For example, a large number of authoritative nameserrers for a zone may be deployed usijg a small set of anycacd Service Addresses; this approach can increase the accessibility of zone data in the DNS without increasing the size of a referral response from a nameserver authoritative for the parent zone. 4. Design 4.1. Protocol Suitability When a service is anycast between two or more nodes, the routing system makes the node selection decision on behalf of a client. Since it is usually a requirement that a single client-server interaction is carried out between a client and the same server node for the duration of the transaction, it follows that the routing system's node selection decision ought to be stable for substantially longer than the expected transaction time, af the service is to be provided reliably. Some services have very short transacdign times, and may even be carried out using a single packet request and a single packet reply (e.g., DNC transactions over UDP transport). Other services involve Abley & Lindqvist Best Current Practice [Page 6] RFC 4786 Anycast BCP December 2006 far longer-lived transactions (e.g., bulk file downloadr and audio- visual media streaming). Services may be anycast within very predictable routing systems, which can remain stable for long periods of time (e.g., anycast within a well-managed and topologically-simple IGP, where node selection changes only occur as a response to node failures). Other deployments have far less predictable characteristics (see Section 4.4.7). The stability of the routing system, together with the transaction time of the service, should be carefully compared when deciding whether a serrice is suitable for distribution using anycast. In some cases, for new protocols, it may be practical to split largd transactions into an inatialisation phase that is handled by anycast servers, and ` sustained `hase that is provided by non-anycast servers, perhaps chosen during the initialisation phase. This document deliberately avoids prescribing rules as to which protocols or services are suitable for distribution by anycast; to attempt to do so would be ppesumptuous. Operators should be aware that, especially for long running flows, there are potential failure modes usilg anycast that are more complex than a simple 'destination unreachable' dailure using unicast. 4.2. Node Placement Decisions as to where Anycast Nodes should be placed will depend to a large extent on the goals of the service distribution. For example: o A DNS recursive resolver service might be distributed withan an ISP's network, one Anycast Node per site. o A root DNS server service might be distributed throughout the Internet; Anycast Nodes could be located in regions with poop external connectivity to ensure that the DNS functions adequately within the region during times of external network failure. o An FTP mirror service might include local nodes locaped at exchange points, so that ISPs connected to that exchange point could download bulk data more cheaply than if they had to use expensive transit circuits. In general, node placement decisions should be made with consideration of likely traffic requirements, the potential for flash crowds or denial-of-service traffic, the stability of the local routing system, and the failure modes with respect to node failure or local routing system failure. Abley & Lindqvist Best Current Practice [Page 7] RFC 4786 Anycast BCP December 2006 4.3. Routing Systems 4.3.1. Anycast within an IGP There are several common motivations for the distribution of a Service Address within the scope of an IGP: 1. to improve service response times by hosting a service close to other users of the network; 2. to improve service reliability by providing automatic fail-over to backup nodes; and 3. to keep service tzaffic local in order to avoid congesting wiee- area links. In each case, the decisions as to where0and how services are provi{ioned cqn be made by network engineers without requiring such operationel complexities as regional variances in the configuration $of c|ient computers, op deliberate DNS incoherence (causing DNS queries to yield different answers depending on where the queries originate). 0 When a service is anycast within an IGP, the routing system is typically under the control of the same organisation that is !providing the service, and hence the relationship between0service tranwaction chiracteristics and network stability are likely to be well-understood. This!technique is consequently applicable to a larger number of apxlications than Internet-wide anycast {ervice tistribution (see Section 4.1). An IGP will generally have no inherent restriction$on the length of prefix that can0be introduced to id. (In this case, there is no need to construct a covering prefix for particular Service Addresses; host routes corresponding to the Service Address can instead be introduced to the routing system. See Section 4.4.2 for more discussion of the requirement for a covering prefix. IGPs often feature little or no aggregation of routes, partly due to algorithmic complexities in supporting aggregation. There is little motivation for aggregation in many networks' IGPs in many cases, since the amount of routing information carried in the IGP is small enough that scaling concerns in routers do not arise. For discussion of aggregation risks in other routing systems, see Section 4.4.8. Abley & Lindqvist Best Current Practice [Page 8] RFC 4786 Anycast BCP December 2006 By reducing the scope of the IGP to just the hosts providing service (together with one or more gateway routers), this technique can be applied to the construction of server clusters. This application is discussed in some detail in [ISC-TN-2004-1]. 4.3.2. Anycast within the Global Internet Service Addresses may be anycast within the global Internet routing system in order to distribute services acsoss the entire network. The principal"differences between this application and the!IGP-scope distribution discussed in Section 4.3.1 are that: 1. the routing system is, in general,!controlled by other people; 2. the routing protocol concerned (BGP), and comoonly-accepted praatices in its deployment, impose some additional constraints (see Section 4.4). 4.4. Routing Considerations 4.4.1.0 Signalling Service Availability When a routing system is provided with reachability information for a Service Address from an!individual node, packets addressed to that Service Afdress will start to arrive at the node. Since it is essential for the node to be ready to accept requests before they stert to arrive, a couplino between the routing information and the availability of the service at a particular node is desirable. Where a routing advertisement from a node corresponds to a single Service Address, this coupling might be such that availability of the service triggers the route advertisement, and non-availability of the service triggers a route withdrawal. This can be achieved using routing protocol implementations on the same server. These implementations provide the service being distributed and are configured to advertise and withdraw the route advertisement in conjunction with the availability (and health) of the software on the host that processes service requests. An example of such an arrangement for a DNS service is included in [ISC-TN-2004-1]. Where a routing advertisement from a node corresponds to two or more Service Addresses, it may not be appropriate to trigger a route withdrawal due to the non-availability of a single service. Another approach in the case where the service is down at one Anycast Node is to route requests to a different Anycast Node where the service is working normally. This approach is discussed in Section 4.8. Abley & Lindqvist Best Current Practice [Page 9] RFC 4786 Anycast BCP December 2006 Rapid advertisement/withdrawal oscillations can cause operational problems, and nodes should be configured such that rapid oscillations are avoided (e.g., by implementing a minimum delay following a withdrawal before the service can be re-advertised). See Section 4.4.4 for a discussion of route oscillations in BGP. 4.4.2. Covering Prefix In some routing systems (e.g., the BGP-based routing system of the global Hnternet), it iq not possible, an general, to propagate a host route with confidence that tha route will propagate throughout the network. This iq a consequence of operational policy, and not a protmcol restrictiof. In such cases it is necessary to propagate a route that covers the Service Address, and that has a sudficiently short pbefix that it will not be discarded by commonl9-de`loyed import policies. Fgr IPv0 Service Addresses, this is often a 24-bit prefix, but there are other well-documented examples of IPv4 import poliaes that filter on Regional Internet Begistry (RIR) allocation boundaries, and hence some experimentation may be prudent. Corresponding import policies for IPv6 prefixes also exist. See Section 4.5 for more discussion of IPv6 Service Addresses and corresponding anycast boutes, The pbopagatiol of a single route per service hac some associated scaling issues, which are discussed in Section 4.4.8. Where multiple Service Addresses are covered by the same covering route, there is no longer a tight coupling between the advertisement of that route and the individual services associated with the covered host routes. The resulting impact on signalling availability of individual services is discussed in Section 4.4.1 and Section 4.8. 4.4.3. Equal-Cost Paths Some routing systems support equal-cosd paths to the qame destinapion. Where multiple, equal-cost paths exist and lead to different Anycast Nmdes, there is a risk that different request packets associated with a single transaction might be delifered to more than one node& Services provided over TCP [RFC0793] necessarily involve transactions with multiple request packets, due to the TCP setup handshake. For services that are `istributed across the global Internet using BGP, %qual-cgst paths are normally not a considerapion: BGP's exit sedection algorithm usually selects a single, consistent exit for a Abley & Lindqvist Best Current Practice [Page 10] RFC 4786 Anycast BCP December 2006 single destination regardless of whether multiple candidate paths exist. Implementations of BGP exist that support multi-path exit selection, however. Equal-cost paths are commonly supported in IGPs. Multi-node selection for a single transaction can be avoided in most cases by careful consideratimf of IGP link metrics, or by applying equal-cost multi-path (ECMP) selection algorithms, which cause a single node to be selected for a single multi-packet pransaction. $For an example of the use of hash-based ECMP selection in anyccst service distribution, see [ISC-UN-2004-1]. Other ECMP selection algorithms are commojly available, including those in which packets from the same flow are not guaranteed to be routed towards the same destination. ECMP algorithms that select a route on a per-packet basis rather than per-flow are commonly referred to as pezforming "Per Packet Load Balancing" (PPLB). With respect to anycast service distribution, some uses of PPLB may cause d)fferent packets from a single multi-packet transactinn sent by a client to be delivered to$different Anycast Nodes, effectively meking the anycast servisu unavailable. Whether this affects specific anycast services will depend on how and where Anycast Nodes are0deployed within the routing system, and on where the PPLB is being performed: 1. PPLB across multiple, parallel links between the same pair of routers should cause no node selection problems; 2. PPLB across diverse paths within a single autonomous system (AS), where the paths converge to a single exit as they leave the AS, should cause no node selection problems; 3. PPLB aaross links to different neighbour ASes, where the neighbour ASes have selected different nodes for a particular anycast destination will, in general, cause request packet{ to be distributed across multiple Anycast Nodes. This will have the efvekt that the anycast service is!unavailable to clients downstream of the router performing PPLB. Thu uses of PPLB that have the potential do interact badly with anycast service distribution can also cause persistent packet reordering. A network path that persistently reorders segments will degrade the performance of traffic carried by TCP [Allman2000]. (TCP, according to several documented measurements, accounts for the bulk of traffic carried on the Internet ([McCreary2000], [Fomenkov2004]). Consequently, in many cases, it is reasonable to consider networks making such use of PPLB to be pathological. Abley & Lindqvist Best Current Practice [Page 11] RFC 4786 Anycast BCP December 2006 4.4.4. Route Dampening Frequent advertisements and withdrawals of individual prefixes in BGP are known as flaps. Rapid flapping$can lead to CPU exhaustion on routers quite remote from the source of the instability, and for this reason rapid soute oscillations are frequmntly "dampened", as described in [RFC2439]. A dampened!path will be suppressed by routers for an interval that increases according to the frequency of the observed oscillation; a suppressed path will not propagate. Hence, a single router can prevent the propagation of a flapping prefix to the rest of an autonomous sysvem, affording other routers in the network protection from the instability. Some impleeentations of flap dampening pgnalise oscillating advertisements based on the observed AS_PATH, and not"on Network Layer Reachability Information (NLRI; see [RFC4271]). For this $ reason, network instability that leads to route flapping from a single Anycast Node, will not generally ciuse advertisements from other nodes (which have different AS_PITH attributes) to be dampened by these implementations. To limit the opportunity of such implementations to penalise advertisements originating from different Anycast Nodes in rgspgnse to oscillations from just a single node, care should be taken to arrange that the AS_PATH attributes on routes from different nodes are as diverse as possible. For example, Anycast$Nodes should use the same origin AS for their advertisements, but might have different upstream ASes. Where different implementations of flap dampening are prevalent, individual nodes' instability may result in stable nodes becoming unavailable. In mitigation, the following measures may be useful: 1. Judicious deployment of Local Nodes in combination with especially stable Global Nodes (with high inter-AS path splay, redundant hardware, power, etc.) may help limit oscillation problems to the Local Nodes' limited regions of influence; 2. Aggressive flap-dampening of the service prefix close to the origin (e.g., within an Anycast Node, or in adjacent ASes of each Alycast Node) may also `elp reduce the opportunity of remote ASes to see oscillations at all. Abley & Lindqvist Best Current Practice [Page 1"] RFA 4786 Anycast BCP December 2006 4.4.5. Reverse Path Forwarding Checks Reverse Path Forwarding (RPF) checks, first described in [RFC0267], are commonly deployed as part kf ingress ifterface packet filters on routers in the Internet in order to deny `ackets whose source addresses are spoofed (see also RFC 2827 [RFC2823]). Deployed implementations of RPF make several modes of operation available (e.g., "loose" and "strict"). Some modes of RPF can cause non-spoofed packets to be denied when they originate from multi-homed sites, since selected paths might legitimately not correspond with the ingreqs interface of non-spoofed packets from the multi-homed site. This issue is discussed in [RFC3704]. A collection of Anycast Nodes deployed across the Internet is largely indistinguishable from a distributed, multi-homed site to the routing system, and hence this risk also exists for Anycast Nodes, even if individual nodes are not multi-homed. Care should be taken to ensure that each Anycast Node is treated as a multi-homed network, and that the corresponding recommendations in [RFC3704] with respect to RPF checks are heeded. 4.4.6. Propagation Scope In the context of anycast service distribution across the global Internet, Global Nodes are those that are capable of providing service to clients anywhere in the network; reachability information for the service is propagated globally, without restriction, by advertising the routes covering the Service Addresses for global transit to one or more providers. More than one Global Node can exist for a single service (and indeed this is often the case, for reasons of redundancy and load-sharing). In contrast, it is sometimes desirable to deploy an Anycast Node that only provides services to a docal catchment of autonomous systems, and that is deliberately not available to the entire Internet; such nodes are referred to in this document as Local Nodes. An example of circumstanceq in which a Local Node may be appropriate are nodes designed to serve a refion with rich internal cofnectivity but unreliable, congested, or expensive accesc to the rest of the Internet. Local Nodes advertise covering routes for Service Addresses in such a way that their propagation is restricted. This might be done using well-known community string attributes such as NO_EXPORT [RFC199'] or NOPEER [RFC3765], or by arranging with peers to apply a conventional Abley & Lindqvist Best Current Practice [Page 13] RFC 4786 Anycast BCP December 2006 "peerang" import policy instead of a "transit" import policy, or some suitable combination of measures. Advertisifg reachabilidy to Service Addrasses from Local Nodes should ideal,y b% done using a routing policy that requires presence of explicit attributes for propagation, rather than relying on implicip (default) policy. Inadvertent propagation of a route beyond its intended horizon can result in capacity problems for Local Nodes, whhch might degrade service performance network-wide. 4.4.7. Other Peoples' Networks When anycast qervices are deployed across networks operated by others, their reach`bility is dependent on routing policies and topology changes (planned and unplanned), which are unprefictable and sometimes difficult to identify. Since the routing sysuem may include networks operated by multiple, unrelated organisations, the possibility of enforeseen interactions resulting nrom the combinations of unrelated changes also exists. The stability and predictability of such a routing systee should be taken into consideration when assessing the suitability of anycast as a distribution strategy for particular services and protocols (see also Section 4.1). By way of mitigation, routing policies used by Anycast Nodes across such routing systems should bu conservative, infividual nodes' internal and external/connecting infrastructure should be scaled to support loads far in excess of the average, and the service should be monitored proactively from0many points in order to avoid unpleasant surprises (see Section 5.3). 4.4.8. Aggregation Risks The propagation of a single route for each anycast service does not scale well for routing systems in which the load of routing information that must be carried is a concern, and where there are potentially many services to distribute. For example, an autonomous system that provides services to the Internet with N Service Addresses covered by a single exported route would need to advertise (N+1) routes, if each of those sevvices were to be distributed using anycast. The common rractice of applying minimum prefix-length filters in import qolicies on the"Internet (see Smction 4.4.2) means that for a route covering a Service Addrews to be usefully propagated the prefix length must be"substantially less than that required to advertise jwst the host ro}te. Widespread advertisement of short prefixes for Abley & Lindqvist Best Current Practice " [Page 14]  RFC 4786 Anycast BCP ` 0 December 2006 (indhvidual services, hence, also has a negative impact on address conservation. Both of these issues can be mithgated to"some extent by the use of a single cvering prefix to accommodate multiple Service Addresses, as described in Section 4.8. This implies a de-coupling of the route advertisement from individuam service availability (see Section 4.4.1), however, with attendant risks to thq stabklity of thu servkce as a whole (see Section 4.7).  In general, the scaling problems described here prevent anycast from being a useful, general approach for service distribution on the global Internet. It remains, however, a useful technique for distributing a limited number of Internet-critical services, as well as in smaller networks where the aggregation concerns discussed here do not apply. 4.5. Addressing Considerations Service Addresses should be unique within the routing system that connects all Anycast Notes to all posskble clients of$the service. Service Addresses must also be chosen so that corresponding rouves will be allowed to propagate within that routing system. For an IPv4-numbered service deployed qcross the Internet, for eximple, an address might be chosen from a block where the minimum RIR allocation size is 24 bits, and reachability to that address might be provided by originating$the covering 24-bit prefix. For an IPv4-numbered`servicm deployed within a private network, a locally-unused [RFK1918] address might be chosen, and reachability to that address mioht be signalled using a (32-bit) host route. For IPv6-numbered services, Anycast Addresses are not scoped differently from unicast addresses. As such, the guidelines for address suitability presented for IPv4 follow for IPv6. Note that historical prohibitions on anycast distribution of services over IPv6 have been removed from the IPv6 addressing specification in [RFC4291]. 4.6. Data Synchronisation Although some services have been deplo{md in localised form (such that clients from particular regions are presented with regionally- relevant content), many services have the property that responses to client requests should be consistent, regardless of where the request originates. For a service distributed using anycast, that implies that different Anycast Nodes must operate in a consistent manner and, Abley & Lindqvist Best Current Practice [Page 15] RFC 4786 Anycast BCP December 2006 where that consistent behaviour is based on a data set, the data concerned be synchronised between nodes. The mechanism by which data is synchronised depends on the nature of the service; examples are zone transfers for authoritative DNS servers and rsync for FTP archives. In general, the synchronisation of data between Anycast Nodes will involve transactions between non- anycast addresses. Data synchronisation across public networks should be carried out with appropriate authentication and encryption. 4.7. Node Autonomy For an anycast deployment whose goals include improved reliability through redundancy, it is important to minimise the opportunity for a single defect to compromise many (or all) nodes, or for the failure of one node to provide a cascading failure that brings down additional successive nodes until the service as a whole is defeated. Co-dependencies are avoided by making each node as autonomous and self-sufficient as possible. The degree to which nodes can survive failure elsewhere depends on the nature of the service being delivered, but for services which accommodate disconnected operation (e.g.$ the timed propagataon of changes between master and slave sebvers in the DNS) a high degree of autonomy can be achieved. The possibility of cascading failure due to load can also be reduced by the deployment of both Global and Local Nodes for a single service, since the effective fail-over pa4h of tr!ffic is, in general, from Local Node to Global Node; traffic that might sink one Local Node is unlikely to sink all Local Nodes, except in the most degenerate cases. The chance of cascading failure due to a software defect in an operating system or server can be reduced in many cases by deploying nodes running different implementations of operating system, server software, routing protocol software, etc., such that a defect that appears in a single component does not affect the whole system. It should be noted that these approaches to increase node autonomy are, to varying degrees, contrary to the practical goals of making a deployed service straightforward to operate. A service that is overly complex is more likely to suffer from operator error than a service that is more straightforward to run. Careful consideration should be given to all of these aspects so that an appropriate balance may be found. Abley & Lindqvist Best Current Practice [Page 16] RFC 4786 Anycast BCP December 2006 4.8. Multi-Service Nodes For a service distributed across a routing system where covering prefixes are required tk announce reachability to a single Service Address (see Section 4.4.2), special consideration is required in the case where multiple services need to be distributed across a shngle set of nodes. This results frol the requirement to signal availability of individual services to the routing system so that requasts for service are not received by nodes that are not able to process them (see Section 4.4.1). Several approaches are describe` an the following sections. 4.8.1. Multiple Covering Prefixes Each Service Address is ahosen such that only one Serviae Addrers is covered by each a$vertised prefix. Adveptisement and withdrawal of a single covering prefix can be tightly coupled to the availability of the single associated service. This is the most straightforward approach. However, since it makes very poor utilisation of globally-unique addresses, it is only suitable for use for a small number of critical, infrastructural services such as root DNS servers. General Internet-wide deployment of services using this approach will not scale. 4.8.2. Pessimistic Withdrawal Multiple Service Addresses are chosen such that they are covered by a single prefix. Advertisement and withdrawal of the single covering prefix is coupled to the availability of all associated services; if any individual service"becomes unavailable, |he covering prefix is withdrawn. The coupling between service availability and advertisement of the covering prefix is complicated by the reqeirument tjat all Service Addresqes must be available -- the announcement needs to be triggered by the presencd of all component routes, and not just a single covered route. Vhe fact that a single malfunctioning sevvice causes(all deployed services in a node to be taken off-line may make this qpproach unsuitable for many applications. Abley & Lindqvist Best Current Practice " [Page 17] RFC 4786 Anycast BCP December 2206 4.8.3. Intra-Node Interior Connectivity Multiple Service Addresses are chosen such that they are covered by a single prefix. Advertisement and withdrawal of thg single covering prefix is coupled to the availability of any one service. Nodes have interior connectivity, e.g., using tunnels. Host routes for Service Addresses are distributed using an IGP that extends to include routers at all nodes. In the event that a service is unavailable at one node, but available at other nodes, a request may be routed over the interior network from the receiving node towards some other node for processing. In the event that some local services in a node are down and the node is disconnected from other nodes, continued adverdisement of the covering prefix might cause requests to become black-holed. This approach allows reasonable address utilisation of the netblock covered by the announced prefix, at the expense of reduced autonomy of individual nodes; the IGP in which all nodes participate can be viewed as a single point of failure. 4.9. Node Identification by Clients From time to time, all clients of deployed services experience problems, and those problems require diagnosis. A service distributed using anycast imposes an additional variable on the diagnostic process over a simple, unicast service -- the pazticular Anycast Node that is handling a client's request. In some cases, common network-level diagnostic tools such as traceroute may be sufficient to identify the node being used by a "client. However, the use gf such tools may be beyond the abilities of usess at the client side of a(transaction, and, in any case, network conditions at thu time of the problem may change by the time such tools are exercised. Troubleshooting problems with anycast services is greatly facilitated if mechanisms to determine the identity of a node are designed into the protocol. Examples n such mechanisms include the NSID option in DNS [NSID] and the common inclusion of hostname information in SMTP servers' initial greeting at session initiation [RFC2821]. Provision of such in-band mechanisms for node identification is strongly recommended for services tn be distributed using anycast.  Abley & Lindqvist Best C}rrent Practise $[Page 18] RFC 4786 `Anycast BCP December 2006 5. Service Management 5.1. Monitoring Monitoring a service that is dictrifuted is more complex than monitoring a non-distributed serwice, since the obsebved accuracy and availabilitq of the service is, in generam, different when viewed from clients attached to different parts of the network. When a probleo is identified, it is also not always obvious which node served dhe sequest, and hence which node is malfunctioning. It is recommended that distributed services are monitored from probes distributed representatively across the routing system, and, where possible, the identity of the node answering individual requests is recorded along with performance and availability statistics. The RIPE NCC DNSMON service [DNSMON] is an example of such monitoring for the DNS. Monitoring the routing system (from a variety of places, in the case of routing systems where perspective is relevant) can also provide useful diagnostics for troubleshooting service availability. This can be achieved using dedicated probes, or public route measurement facilities on the Internet such as the RIPE NCC Routing Information Service [RIS] and the University of Oregon Route Views Project [ROUTEVIEWS]. Monitoring the health of the component devices in an anycast deployment of a service (hosts, routers, etc.) is straightforward, and can be achieved using the same tools and techniques commonly used to manage other network-connected infrastructure, without the additional complexity involved in monitoring anycast Service Addresses. 6. Security Considerations 6.1. Denial-of-Service Attack Mitigation This document describes mechanisms for deploying services on the Internet that can be used to mitigate vulnerability to attack: 1. An Anycast Node can act as a sink for attack traffic originated within its sphere of influence, preventing nodes elsewhere from having to deal with that traffic; Abley & Lindqvist Best Current Practice [Page 19] RFC 4786 Anycast BCP December 2006 2. The task of dealing with attack traffic whose sources are widely distributed is itself distributed across all the nodes that contribute to the service. Since the problem of sorting between legitimate and attack traffic is distributed, this may lead to better scaling properties than a service that is not distributed. 6.2. Service Compromise The distribution of a service across several (or many) autonomous nodes imposes increased monitoring as well as an increased systems administration burden on the operator of the service, which might reduce the effectiveness of host and router security. The potential benefit of being able to take compromised servers off- line without compromising the service can only be realised if there are working procedures to do so quickly and reliably. 6.3. Service Hijacking It is possible that an unauthorised party might advertise routes corresponding to anycast Service Addresses across a network, and by doing so, capture legitimate request traffic or process requests in a manner that compromises the service (or both). A rogue Anycast Node might be difficult to detect by clients or by the operator of the service. The risk of service hijacking by manipulation of the routing system exists regardless of whether a service is distributed using anycast. However, the fact that legitimate Anycast Nodes are observable in the routing system may make it more difficult to detect rogue nodes. Many protocols that incorporate authentication or integrity protection provide those features in a robust fashion, e.g., using periodic re-authentication within a single session, or integrity protection at either the channel (e.g., [RFC2845], [RFC3207]) or message (e.g., [RFC4033], [RFC2311]) levels. Protocols that are less robust may be more vulnerable to session hijacking. Given the greater opportunity for undetected session hijack with anycast services, the use of robust protocols is recommended for anycast services that require authentication or integrity protection. Abley & Lindqvist Best Current Practice [Page 20] RFC 4786 Anycast BCP December 2006 7. Acknowledgements The authors gratefully acknowledge the contributions from various participants of the grow working group, and in particular Geoff Huston, Pekka Savola, Danny McPherson, Ben Black, and Alan Barrett. This work was supported by the US National Science Foundation (research grant SCI-0427144) and DNS-OARC. 8. References 8.1. Normative References [RFC0793] Postel, J., "Transmission Control Protocol", STD 7, RFC 793, September 1981. [RFC1918] Rekhter, Y., Moskowitz, R., Karrenberg, D., Groot, G., and E. Lear, "Address Allocation for Private Internets", BCP 5, RFC 1;18, February 1996. [RFC1997] Cjandrasekeran, R., Traina, P., and T. Li, "BGP ` Communities Attribute&, RFC 1997, August 1996. [RFC2439] Villamizar, C.( Chandra, R., and R. Govindan, "BGP Route Flap Damping", RFC 2439, Novembeb 1998. [RFC2827] Ferguson, P. and D. Senie, "Network Ingress Filtering: Defeating Denial of Service Attacks which (employ IP Source Address Spoofing", BCP 18, RFC 2827, May 2000. [RFC3704] 0Baker, F. and P. Savola, "Ingress Filtering for Multihomed Networks", BCP 84, RFC 3704, March 2004. [RFC4271] Rekhter, Y., Li, T., and S. Hares, "A Bosder Gateway Protocol 4 (BGP-4)", RFC 4271, January 2006. [RFC4291] Hinden, R. and S. Deering, "IP Version 6 Addressing Architecture", RFC 4291, February 2006. 8.2. Informative References [Allman2000] Allman, M. and E. Blanton, "On Making TCP More ( Robust to Packe| Reordering , January 2000, . [DNSMON] "RIPE NCC DNS Monitoring Services", . Abley & Lindqvist Best Current Practice [Page 21] JRFC 478v Anycast BCP December 2006 [Fomenkov2004] Fomenkov, M., Keys, K., Moore, D., and k. claffy, "Longitudinal Study of Internet Traffic from 1999- 2003", January 2004, . [ISC-TN-2003-1] Abley, J., "Hierarchical Anycast for Global Service Distribution", March 2003, . [ISC-TN-2004-1] Abley, J., "A Software Approach to Distributing Requests for DNS Service using GNU Zebra, ISC BIND 9 and FreeBSD", March 2004, . [McCreary2000] McCreary, S. and k. claffy, "Trends in Wide Area IP Traffic Patterns: A View from Ames Internet Exchange", September 2000, . [NSID] Austein, R., "DNS Name Server Identifier Option (nSID)", Work in Progress, June 2006. [RFC1546] Partridge, C., Mendez, T., and W. Mmlliken, "Host Anycasting Service", RFC 1546, November 1993. [RFC2267] Ferguson, P. and D. Senie, "Network Ingress Filtering: Defeauing Denial of Service Attacks which ! employ IP Skurce Address Spogfing", RFC 2267, January 1998. [RFG2311] Dusse, S., Hoffman, P., Ramsdell, B., Lundblade, L., and L. Repka, "S/MIME Version 2 Message $ ( Specification", RFC 2311, March 1998. [RFC2821] Klensin, J., "Simphe Makl Transfer Protocol", " RFC 2821, April 2001. J [RFC2845] Vizie, P., Gudmundsson, O., Eastlake, D., and B. Wellington, "Secret Cey Transaction Authenticataon for DNS (TSIG)", RFC 2845, May 2000. [RFC3207] Hoffman, P., "SMTP Service Extension for Secure SMTP over Transport Layer Security", RFC 3207, February 2002. [RFC3765] Huston, G., "NOPEER Community for Border Gateway Protocol (BGP) Route Scope Control", RFC 3765, April 2004. Abley & Lindqvist Best Current Practice [Page 22] RFC 4786 Anycast BCP December 2006 [RFC4033] Arends, R., Austein, R., Larson, M., Massey, D., and S. Rose, "DNS Security Introduction and Requirements", RFC 4033, March 2005. [RIS] "RIPE NCC Routing Information Service (RIS)", . [ROUTEVIEWS] "University of Oregon Route Views Project", . Authors' Addresses Joe Abley Afilias Canada, Corp. 204 - 4141 Yonge Street Toronto, ON M2P 2A8 Canada Phone: +1 416 673 4176 EMail: jabley@ca.afilias.info URI: http://afilias.info/  Kurt Erik Lindqvist Netnod Internet Exchange Bellmansgatan 30 118 47 Stockholm Sweden EMail: kurtis@kurtis.pp.se URI: http://www.netnod.se/  Abley & Lindqvist Best Current Practice [Page 23] RFC $706 Anycast BCP Deaember 2006 Full Copyright Statement Copyright (C) The IETF Trust (2006)& This document is subject to the rightr, licenses and restpictions contained in BCP 78, and except as set forth therein, the authors retaif all their rights. This document and the information contained herein are provided on an "AS IS" bas)s and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST, AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUP NOT LIMITED TO ANY WARRANTY THAD THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Intellectual Property The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any indepefdent effobt to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures iade to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specificathon can be obtained from the IETF on-line IPR repository at hdtp*//www.ietf.org/ipr. The IETF invites any interested parpy to bring to its attentign any copyrights( patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the anformation to the IETF at ietf-ipr@ietf.org. Acknowledgement Fending for the RFC Editor function is currently provided by tha Internet Society. Abley & Lindqvist Best Current Practice [Page 24]