Traffic Management

Istio’s traffic routing rules let you easily control the flowof traffic and API calls between services. Istio simplifies configuration ofservice-level properties like circuit breakers, timeouts, and retries, and makesit easy to set up important tasks like A/B testing, canary rollouts, and stagedrollouts with percentage-based traffic splits. It also provides out-of-boxfailure recovery features that help make your applicationmore robust against failures of dependent services or the network.

Istio’s traffic management model relies on the Envoyproxies that are deployed along with your services. All traffic that your meshservices send and receive (data plane traffic) is proxied through Envoy, makingit easy to direct and control traffic around your mesh without making anychanges to your services.

If you’re interested in the details of how the features described in this guidework, you can find out more about Istio’s traffic management implementation in thearchitecture overview. The rest ofthis guide introduces Istio’s traffic management features.

Introducing Istio traffic management

In order to direct traffic within your mesh, Istio needs to know where all yourendpoints are, and which services they belong to. To populate its ownservice registry, Istio connects to a servicediscovery system. For example, if you’ve installed Istio on a Kubernetes cluster,then Istio automatically detects the services and endpoints in that cluster.

Using this service registry, the Envoy proxies can then direct traffic to therelevant services. Most microservice-based applications have multiple instancesof each service workload to handle service traffic, sometimes referred to as aload balancing pool. By default, the Envoy proxies distribute traffic acrosseach service’s load balancing pool using a round-robin model, where requests aresent to each pool member in turn, returning to the top of the pool once eachservice instance has received a request.

While Istio’s basic service discovery and load balancing gives you a workingservice mesh, it’s far from all that Istio can do. In many cases you might wantmore fine-grained control over what happens to your mesh traffic.You might want to direct a particular percentage of traffic to a new version ofa service as part of A/B testing, or apply a different load balancing policy totraffic for a particular subset of service instances. You might also want toapply special rules to traffic coming into or out of your mesh, or add anexternal dependency of your mesh to the service registry. You can do all thisand more by adding your own traffic configuration to Istio using Istio’s trafficmanagement API.

Like other Istio configuration, the API is specified using Kubernetes customresource definitions (CRDs), which you can configureusing YAML, as you’ll see in the examples.

The rest of this guide examines each of the traffic management API resourcesand what you can do with them. These resources are:

• Virtual services
• Destination rules
• Gateways
• Service entries
• Sidecars

This guide also gives an overview of some of thenetwork resilience and testing features thatare built in to the API resources.

Virtual services

Virtual services,along with destination rules, are the key building blocks of Istio’s trafficrouting functionality. A virtual service lets you configure how requests arerouted to a service within an Istio service mesh, building on the basicconnectivity and discovery provided by Istio and your platform. Each virtualservice consists of a set of routing rules that are evaluated in order, lettingIstio match each given request to the virtual service to a specific realdestination within the mesh. Your mesh can require multiple virtual services ornone depending on your use case.

Why use virtual services?

Virtual services play a key role in making Istio’s traffic management flexibleand powerful. They do this by strongly decoupling where clients send theirrequests from the destination workloads that actually implement them. Virtualservices also provide a rich way of specifying different traffic routing rulesfor sending traffic to those workloads.

Why is this so useful? Without virtual services, Envoy distributestraffic using round-robin load balancing between all service instances, asdescribed in the introduction. You can improve this behavior with what you knowabout the workloads. For example, some might represent a different version. Thiscan be useful in A/B testing, where you might want to configure traffic routesbased on percentages across different service versions, or to directtraffic from your internal users to a particular set of instances.

With a virtual service, you can specify traffic behavior for one or more hostnames.You use routing rules in the virtual service that tell Envoy how to send thevirtual service’s traffic to appropriate destinations. Route destinations canbe versions of the same service or entirely different services.

A typical use case is to send traffic to different versions of a service,specified as service subsets. Clients send requests to the virtual service host as ifit was a single entity, and Envoy then routes the traffic to the differentversions depending on the virtual service rules: for example, “20% of calls go tothe new version” or “calls from these users go to version 2”. This allows you to,for instance, create a canary rollout where you gradually increase thepercentage of traffic that’s sent to a new service version. The traffic routingis completely separate from the instance deployment, meaning that the number ofinstances implementing the new service version can scale up and down based ontraffic load without referring to traffic routing at all. By contrast, containerorchestration platforms like Kubernetes only support traffic distribution basedon instance scaling, which quickly becomes complex. You can read more about howvirtual services help with canary deployments in Canary Deployments using Istio.

Virtual services also let you:

• Address multiple application services through a single virtual service. Ifyour mesh uses Kubernetes, for example, you can configure a virtual serviceto handle all services in a specific namespace. Mapping a singlevirtual service to multiple “real” services is particularly useful infacilitating turning a monolithic application into a composite service builtout of distinct microservices without requiring the consumers of the serviceto adapt to the transition. Your routing rules can specify “calls to these URIs ofmonolith.com go to microservice A”, and so on. You can see how this worksin one of our examples below.
• Configure traffic rules in combination withgateways to control ingressand egress traffic.

In some cases you also need to configure destination rules to use thesefeatures, as these are where you specify your service subsets. Specifyingservice subsets and other destination-specific policies in a separate objectlets you reuse these cleanly between virtual services. You can find out moreabout destination rules in the next section.

Virtual service example

The following virtual service routesrequests to different versions of a service depending on whether the requestcomes from a particular user.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
  - reviews
  http:
  - match:
    - headers:
        end-user:
          exact: jason
    route:
    - destination:
        host: reviews
        subset: v2
  - route:
    - destination:
        host: reviews
        subset: v3

The hosts field

The hosts field lists the virtual service’s hosts - in other words, the user-addressabledestination or destinations that these routing rules apply to. This is theaddress or addresses the client uses when sending requests to the service.

hosts:
- reviews

The virtual service hostname can be an IP address, a DNS name, or, depending onthe platform, a short name (such as a Kubernetes service short name) that resolves,implicitly or explicitly, to a fully qualified domain name (FQDN). You can alsouse wildcard (”*”) prefixes, letting you create a single set of routing rules forall matching services. Virtual service hosts don’t actually have to be part of theIstio service registry, they are simply virtual destinations. This lets you modeltraffic for virtual hosts that don’t have routable entries inside the mesh.

Routing rules

The http section contains the virtual service’s routing rules, describingmatch conditions and actions for routing HTTP/1.1, HTTP2, and gRPC traffic sentto the destination(s) specified in the hosts field (you can also use tcp andtls sections to configure routing rules forTCP andunterminatedTLStraffic). A routing rule consists of the destination where you want the trafficto go and zero or more match conditions, depending on your use case.

Match condition

The first routing rule in the example has a condition and so begins with thematch field. In this case you want this routing to apply to all requests fromthe user “jason”, so you use the headers, end-user, and exact fields to selectthe appropriate requests.

- match:
   - headers:
       end-user:
         exact: jason

Destination

The route section’s destination field specifies the actual destination fortraffic that matches this condition. Unlike the virtual service’s host(s), thedestination’s host must be a real destination that exists in Istio’s serviceregistry or Envoy won’t know where to send traffic to it. This can be a meshservice with proxies or a non-mesh service added using a service entry. In thiscase we’re running on Kubernetes and the host name is a Kubernetes service name:

route:
- destination:
    host: reviews
    subset: v2

Note in this and the other examples on this page, we use a Kubernetes short name for thedestination hosts for simplicity. When this rule is evaluated, Istio adds a domain suffix basedon the namespace of the virtual service that contains the routing rule to getthe fully qualified name for the host. Using short names in our examplesalso means that you can copy and try them in any namespace you like.

Using short names like this only works if thedestination hosts and the virtual service are actually in the same Kubernetesnamespace. Because using the Kubernetes short name can result inmisconfigurations, we recommend that you specify fully qualified host names inproduction environments.

The destination section also specifies which subset of this Kubernetes serviceyou want requests that match this rule’s conditions to go to, in this case thesubset named v2. You’ll see how you define a service subset in the section ondestination rules below.

Routing rule precedence

Routing rules are evaluated in sequential order from top to bottom, with thefirst rule in the virtual service definition being given highest priority. Inthis case you want anything that doesn’t match the first routing rule to go to adefault destination, specified in the second rule. Because of this, the secondrule has no match conditions and just directs traffic to the v3 subset.

- route:
  - destination:
      host: reviews
      subset: v3

We recommend providing a default “no condition” or weight-based rule (describedbelow) like this as the last rule in each virtual service to ensure that trafficto the virtual service always has at least one matching route.

More about routing rules

As you saw above, routing rules are a powerful tool for routing particularsubsets of traffic to particular destinations. You can set match conditions ontraffic ports, header fields, URIs, and more. For example, this virtual servicelets users send traffic to two separate services, ratings and reviews, as ifthey were part of a bigger virtual service at http://bookinfo.com/. Thevirtual service rules match traffic based on request URIs and direct requests tothe appropriate service.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: bookinfo
spec:
  hosts:
    - bookinfo.com
  http:
  - match:
    - uri:
        prefix: /reviews
    route:
    - destination:
        host: reviews
  - match:
    - uri:
        prefix: /ratings
    route:
    - destination:
        host: ratings
...
  http:
  - match:
      sourceLabels:
        app: reviews
    route:
...

For some match conditions, you can also choose to select them using the exactvalue, a prefix, or a regex.

You can add multiple match conditions to the same match block to AND yourconditions, or add multiple match blocks to the same rule to OR your conditions.You can also have multiple routing rules for any given virtual service. Thislets you make your routing conditions as complex or simple as you like within asingle virtual service. A full list of match condition fields and their possiblevalues can be found in theHTTPMatchRequest reference.

In addition to using match conditions, you can distribute trafficby percentage “weight”. This is useful for A/B testing and canary rollouts:

spec:
  hosts:
  - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 75
    - destination:
        host: reviews
        subset: v2
      weight: 25

You can also use routing rules to perform some actions on the traffic, forexample:

• Append or remove headers.
• Rewrite the URL.
• Set a retry policy for calls to this destination.

To learn more about the actions available, see theHTTPRoute reference.

Destination rules

Along with virtual services,destination rulesare a key part of Istio’s traffic routing functionality. You can think ofvirtual services as how you route your traffic to a given destination, andthen you use destination rules to configure what happens to traffic for thatdestination. Destination rules are applied after virtual service routing rulesare evaluated, so they apply to the traffic’s “real” destination.

In particular, you use destination rules to specify named service subsets, suchas grouping all a given service’s instances by version. You can then use theseservice subsets in the routing rules of virtual services to control thetraffic to different instances of your services.

Destination rules also let you customize Envoy’s traffic policies when callingthe entire destination service or a particular service subset, such as yourpreferred load balancing model, TLS security mode, or circuit breaker settings.You can see a complete list of destination rule options in theDestination Rule reference.

Load balancing options

By default, Istio uses a round-robin load balancing policy, where each serviceinstance in the instance pool gets a request in turn. Istio also supports thefollowing models, which you can specify in destination rules for requests to aparticular service or service subset.

• Random: Requests are forwarded at random to instances in the pool.
• Weighted: Requests are forwarded to instances in the pool according to aspecific percentage.
• Least requests: Requests are forwarded to instances with the least number ofrequests.

See theEnvoy load balancing documentationfor more information about each option.

Destination rule example

The following example destination rule configures three different subsets forthe my-svc destination service, with different load balancing policies:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: my-destination-rule
spec:
  host: my-svc
  trafficPolicy:
    loadBalancer:
      simple: RANDOM
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
    trafficPolicy:
      loadBalancer:
        simple: ROUND_ROBIN
  - name: v3
    labels:
      version: v3

Each subset is defined based on one or more labels, which in Kubernetes arekey/value pairs that are attached to objects such as Pods. These labels areapplied in the Kubernetes service’s deployment as metadata to identifydifferent versions.

As well as defining subsets, this destination rule has both a default trafficpolicy for all subsets in this destination and a subset-specific policy thatoverrides it for just that subset. The default policy, defined above the subsetsfield, sets a simple random load balancer for the v1 and v3 subsets. In thev2 policy, a round-robin load balancer is specified in the correspondingsubset’s field.

Gateways

You use a gateway tomanage inbound and outbound traffic for your mesh, letting you specify whichtraffic you want to enter or leave the mesh. Gateway configurations are appliedto standalone Envoy proxies that are running at the edge of the mesh, ratherthan sidecar Envoy proxies running alongside your service workloads.

Unlike other mechanisms for controlling traffic entering your systems, such asthe Kubernetes Ingress APIs, Istio gateways let you use the full power andflexibility of Istio’s traffic routing. You can do this because Istio’s Gatewayresource just lets you configure layer 4-6 load balancing properties such asports to expose, TLS settings, and so on. Then instead of addingapplication-layer traffic routing (L7) to the same API resource, you bind aregular Istio virtual service to the gateway. This lets youbasically manage gateway traffic like any other data plane traffic in an Istiomesh.

Gateways are primarily used to manage ingress traffic, but you can alsoconfigure egress gateways. An egress gateway lets you configure a dedicated exitnode for the traffic leaving the mesh, letting you limit which services can orshould access external networks, or to enablesecure control of egress trafficto add security to your mesh, for example. You can also use a gateway toconfigure a purely internal proxy.

Istio provides some preconfigured gateway proxy deployments(istio-ingressgateway and istio-egressgateway) that you can use - both aredeployed if you use our demo installation,while just the ingress gateway is deployed with ourdefault or sds profiles. Youcan apply your own gateway configurations to these deployments or deploy andconfigure your own gateway proxies.

Gateway example

The following example shows a possible gateway configuration for external HTTPSingress traffic:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: ext-host-gwy
spec:
  selector:
    app: my-gateway-controller
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - ext-host.example.com
    tls:
      mode: SIMPLE
      serverCertificate: /tmp/tls.crt
      privateKey: /tmp/tls.key

This gateway configuration lets HTTPS traffic from ext-host.example.com into the mesh onport 443, but doesn’t specify any routing for the traffic.

To specify routing and for the gateway to work as intended, you must also bindthe gateway to a virtual service. You do this using the virtual service’sgateways field, as shown in the following example:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: virtual-svc
spec:
  hosts:
  - ext-host.example.com
  gateways:
    - ext-host-gwy

You can then configure the virtual service with routing rules for the externaltraffic.

Service entries

You use aservice entry to addan entry to the service registry that Istio maintains internally. After you addthe service entry, the Envoy proxies can send traffic to the service as if itwas a service in your mesh. Configuring service entries allows you to managetraffic for services running outside of the mesh, including the following tasks:

• Redirect and forward traffic for external destinations, such as APIsconsumed from the web, or traffic to services in legacy infrastructure.
• Define retry, timeout, andfault injection policies for external destinations.
• Run a mesh service in a Virtual Machine (VM) byadding VMs to your mesh.
• Logically add services from a different cluster to the mesh to configure amulticluster Istio meshon Kubernetes.

You don’t need to add a service entry for every external service that you wantyour mesh services to use. By default, Istio configures the Envoy proxies topassthrough requests to unknown services. However, you can’t use Istio featuresto control the traffic to destinations that aren’t registered in the mesh.

Service entry example

The following example mesh-external service entry adds the ext-resourceexternal dependency to Istio’s service registry:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: svc-entry
spec:
  hosts:
  - ext-svc.example.com
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  location: MESH_EXTERNAL
  resolution: DNS

You specify the external resource using the hosts field. You can qualify itfully or use a wildcard prefixed domain name.

You can configure virtual services and destination rules to control traffic to aservice entry in a more granular way, in the same way you configure traffic forany other service in the mesh. For example, the following destination ruleconfigures the traffic route to use mutual TLS to secure the connection to theext-svc.example.com external service that we configured using the service entry:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: ext-res-dr
spec:
  host: ext-svc.example.com
  trafficPolicy:
    tls:
      mode: MUTUAL
      clientCertificate: /etc/certs/myclientcert.pem
      privateKey: /etc/certs/client_private_key.pem
      caCertificates: /etc/certs/rootcacerts.pem

See theService Entry referencefor more possible configuration options.

Sidecars

By default, Istio configures every Envoy proxy to accept traffic on all theports of its associated workload, and to reach every workload in the mesh whenforwarding traffic. You can use a sidecar configuration to do the following:

• Fine-tune the set of ports and protocols that an Envoy proxy accepts.
• Limit the set of services that the Envoy proxy can reach.

You might want to limit sidecar reachability like this in larger applications,where having every proxy configured to reach every other service in the mesh canpotentially affect mesh performance due to high memory usage.

You can specify that you want a sidecar configuration to apply to all workloadsin a particular namespace, or choose specific workloads using aworkloadSelector. For example, the following sidecar configuration configuresall services in the bookinfo namespace to only reach services running in thesame namespace and the Istio control plane (currently needed to use Istio’spolicy and telemetry features):

apiVersion: networking.istio.io/v1alpha3
kind: Sidecar
metadata:
  name: default
  namespace: bookinfo
spec:
  egress:
  - hosts:
    - "./*"
    - "istio-system/*"

See the Sidecar referencefor more details.

Network resilience and testing

As well as helping you direct traffic around your mesh, Istio provides opt-infailure recovery and fault injection features that you can configure dynamicallyat runtime. Using these features helps your applications operate reliably,ensuring that the service mesh can tolerate failing nodes and preventinglocalized failures from cascading to other nodes.

Timeouts

A timeout is the amount of time that an Envoy proxy should wait for replies froma given service, ensuring that services don’t hang around waiting for repliesindefinitely and that calls succeed or fail within a predictable timeframe. Thedefault timeout for HTTP requests is 15 seconds, which means that if the servicedoesn’t respond within 15 seconds, the call fails.

For some applications and services, Istio’s default timeout might not beappropriate. For example, a timeout that is too long could result in excessivelatency from waiting for replies from failing services, while a timeout that istoo short could result in calls failing unnecessarily while waiting for anoperation involving multiple services to return. To find and use your optimal timeoutsettings, Istio lets you easily adjust timeouts dynamically on a per-servicebasis using virtual services without having to edit yourservice code. Here’s a virtual service that specifies a 10 second timeout forcalls to the v1 subset of the ratings service:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: ratings
spec:
  hosts:
  - ratings
  http:
  - route:
    - destination:
        host: ratings
        subset: v1
    timeout: 10s

Retries

A retry setting specifies the maximum number of times an Envoy proxy attempts toconnect to a service if the initial call fails. Retries can enhance serviceavailability and application performance by making sure that calls don’t failpermanently because of transient problems such as a temporarily overloadedservice or network. The interval between retries (25ms+) is variable anddetermined automatically by Istio, preventing the called service from beingoverwhelmed with requests. By default, the Envoy proxy doesn’t attempt toreconnect to services after a first failure.

Like timeouts, Istio’s default retry behavior might not suit your applicationneeds in terms of latency (too many retries to a failed service can slow thingsdown) or availability. Also like timeouts, you can adjust your retry settings ona per-service basis in virtual services without having totouch your service code. You can also further refine your retry behavior byadding per-retry timeouts, specifying the amount of time you want to wait foreach retry attempt to successfully connect to the service. The following exampleconfigures a maximum of 3 retries to connect to this service subset after aninitial call failure, each with a 2 second timeout.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: ratings
spec:
  hosts:
  - ratings
  http:
  - route:
    - destination:
        host: ratings
        subset: v1
    retries:
      attempts: 3
      perTryTimeout: 2s

Circuit breakers

Circuit breakers are another useful mechanism Istio provides for creatingresilient microservice-based applications. In a circuit breaker, you set limitsfor calls to individual hosts within a service, such as the number of concurrentconnections or how many times calls to this host have failed. Once that limithas been reached the circuit breaker “trips” and stops further connections tothat host. Using a circuit breaker pattern enables fast failure rather thanclients trying to connect to an overloaded or failing host.

As circuit breaking applies to “real” mesh destinations in a load balancingpool, you configure circuit breaker thresholds indestination rules, with the settings applying to eachindividual host in the service. The following example limits the number ofconcurrent connections for the reviews service workloads of the v1 subset to100:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: reviews
spec:
  host: reviews
  subsets:
  - name: v1
    labels:
      version: v1
    trafficPolicy:
      connectionPool:
        tcp:
          maxConnections: 100

You can find out more about creating circuit breakers inCircuit Breaking.

Fault injection

After you’ve configured your network, including failure recovery policies, youcan use Istio’s fault injection mechanisms to test the failure recovery capacityof your application as a whole. Fault injection is a testing method thatintroduces errors into a system to ensure that it can withstand and recover fromerror conditions. Using fault injection can be particularly useful to ensurethat your failure recovery policies aren’t incompatible or too restrictive,potentially resulting in critical services being unavailable.

Unlike other mechanisms for introducing errors such as delaying packets orkilling pods at the network layer, Istio’ lets you inject faults at theapplication layer. This lets you inject more relevant failures, such as HTTPerror codes, to get more relevant results.

You can inject two types of faults, both configured using avirtual service:

• Delays: Delays are timing failures. They mimic increased network latency oran overloaded upstream service.
• Aborts: Aborts are crash failures. They mimic failures in upstream services.Aborts usually manifest in the form of HTTP error codes or TCP connectionfailures.

For example, this virtual service introduces a 5 second delay for 1 out of every 1000requests to the ratings service.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: ratings
spec:
  hosts:
  - ratings
  http:
  - fault:
      delay:
        percentage:
          value: 0.1
        fixedDelay: 5s
    route:
    - destination:
        host: ratings
        subset: v1

For detailed instructions on how to configure delays and aborts, seeFault Injection.

Working with your applications

Istio failure recovery features are completely transparent to theapplication. Applications don’t know if an Envoy sidecar proxy is handlingfailures for a called service before returning a response. This means thatif you are also setting failure recovery policies in your application codeyou need to keep in mind that both work independently, and therefore mightconflict. For example, suppose you can have two timeouts, one configured ina virtual service and another in the application. The application sets a 2second timeout for an API call to a service. However, you configured a 3second timeout with 1 retry in your virtual service. In this case, theapplication’s timeout kicks in first, so your Envoy timeout and retryattempt has no effect.

While Istio failure recovery features improve the reliability andavailability of services in the mesh, applications must handle the failureor errors and take appropriate fallback actions. For example, when allinstances in a load balancing pool have failed, Envoy returns an HTTP 503code. The application must implement any fallback logic needed to handle theHTTP 503 error code..

See also

Istio as a Proxy for External Services

Configure Istio ingress gateway to act as a proxy for external services.

Multi-Mesh Deployments for Isolation and Boundary Protection

Deploy environments that require isolation into separate meshes and enable inter-mesh communication by mesh federation.

Secure Control of Egress Traffic in Istio, part 3

Comparison of alternative solutions to control egress traffic including performance considerations.

Secure Control of Egress Traffic in Istio, part 2

Use Istio Egress Traffic Control to prevent attacks involving egress traffic.

Secure Control of Egress Traffic in Istio, part 1

Attacks involving egress traffic and requirements for egress traffic control.

Version Routing in a Multicluster Service Mesh

Configuring Istio route rules in a multicluster service mesh.