• Secure Gateways (File Mount)
    • Before you begin
    • Generate server certificate and private key
    • Configure a TLS ingress gateway with a file mount-based approach
    • Configure a mutual TLS ingress gateway
    • Configure a TLS ingress gateway for multiple hosts
      • Create a server certificate and private key for bookinfo.com
      • Redeploy istio-ingressgateway with the new certificate
      • Configure traffic for the bookinfo.com host
    • Troubleshooting
      • Troubleshooting for mutual TLS
    • Cleanup
    • See also

    Secure Gateways (File Mount)

    The Control Ingress Traffic taskdescribes how to configure an ingress gateway to expose an HTTPservice to external traffic. This task shows how to expose a secure HTTPSservice using either simple or mutual TLS.

    The TLS required private key, server certificate, and root certificate, are configuredusing a file mount based approach.

    Before you begin

    • Perform the steps in the Before you beginand Determining the ingress IP and portssections of the Control Ingress Traffic task. After performingthose steps you should have Istio and the httpbin service deployed,and the environment variables INGRESS_HOST and SECURE_INGRESS_PORT set.

    • For macOS users, verify that you use curl compiled with the LibreSSL library:

    1. $ curl --version | grep LibreSSL
    2. curl 7.54.0 (x86_64-apple-darwin17.0) libcurl/7.54.0 LibreSSL/2.0.20 zlib/1.2.11 nghttp2/1.24.0

    If a version of LibreSSL is printed as in the output above, your curl should work correctly with theinstructions in this task. Otherwise, try another installation of curl, for example on a Linux machine.

    Generate server certificate and private key

    For this task you can use your favorite tool to generate certificates and keys. The commands below useopenssl

    • Create a root certificate and private key to sign the certificate for your services:
    1. $ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=example.com' -keyout example.com.key -out example.com.crt
    • Create a certificate and a private key for httpbin.example.com:
    1. $ openssl req -out httpbin.example.com.csr -newkey rsa:2048 -nodes -keyout httpbin.example.com.key -subj "/CN=httpbin.example.com/O=httpbin organization"
    2. $ openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 0 -in httpbin.example.com.csr -out httpbin.example.com.crt

    Configure a TLS ingress gateway with a file mount-based approach

    In this section you configure an ingress gateway with port 443 to handle HTTPStraffic. You first create a secret with a certificate and a private key. Thesecret is mounted to a file on the /etc/istio/ingressgateway-certs path. You can thencreate a gateway definition that configures a server on port 443.

    • Create a Kubernetes secret to hold the server’s certificate and private key.Use kubectl to create the secret istio-ingressgateway-certs in namespaceistio-system . The Istio gateway will load the secret automatically.

    The secret must be named istio-ingressgateway-certs in the istio-system namespace to align with theconfiguration of the Istio default ingress gateway used in this task.

    1. $ kubectl create -n istio-system secret tls istio-ingressgateway-certs --key httpbin.example.com.key --cert httpbin.example.com.crt
    2. secret "istio-ingressgateway-certs" created

    Note that by default all the pods in the istio-system namespace can mount this secret and access theprivate key. You may want to deploy the ingress gateway in a separate namespace and create the secret there, so thatonly the ingress gateway pod will be able to mount it.

    Verify that tls.crt and tls.key have been mounted in the ingress gateway pod:

    1. $ kubectl exec -it -n istio-system $(kubectl -n istio-system get pods -l istio=ingressgateway -o jsonpath='{.items[0].metadata.name}') -- ls -al /etc/istio/ingressgateway-certs
    • Define a Gateway with a server section for port 443.

    The location of the certificate and the private key must be /etc/istio/ingressgateway-certs, or the gateway will fail to load them.

    1. $ kubectl apply -f - <<EOF
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: Gateway
    4. metadata:
    5.   name: httpbin-gateway
    6. spec:
    7.   selector:
    8.     istio: ingressgateway # use istio default ingress gateway
    9.   servers:
    10.   - port:
    11.       number: 443
    12.       name: https
    13.       protocol: HTTPS
    14.     tls:
    15.       mode: SIMPLE
    16.       serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
    17.       privateKey: /etc/istio/ingressgateway-certs/tls.key
    18.     hosts:
    19.     - "httpbin.example.com"
    20. EOF
    • Configure routes for traffic entering via the Gateway. Define the same VirtualService as in the Control Ingress Traffic task:
    1. $ kubectl apply -f - <<EOF
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: VirtualService
    4. metadata:
    5.   name: httpbin
    6. spec:
    7.   hosts:
    8.   - "httpbin.example.com"
    9.   gateways:
    10.   - httpbin-gateway
    11.   http:
    12.   - match:
    13.     - uri:
    14.         prefix: /status
    15.     - uri:
    16.         prefix: /delay
    17.     route:
    18.     - destination:
    19.         port:
    20.           number: 8000
    21.         host: httpbin
    22. EOF
    • Access the httpbin service with HTTPS by sending an https request using curl to SECURE_INGRESS_PORT.

    The —resolve flag instructs curl to supply theSNI value httpbin.example.com when accessing the gateway IPover TLS. The —cacert option instructs curl to use your generated certificate to verify the server.

    The -HHost:httpbin.example.com flag is included but only really needed if SECURE_INGRESS_PORT is differentfrom the actual gateway port (443), for example, if you are accessing the server via a mapped NodePort.

    By sending the request to the /status/418 URL path, you get a nice visual clue that your httpbin service wasindeed accessed. The httpbin service will return the418 I’m a Teapot code.

    1. $ curl -v -HHost:httpbin.example.com --resolve httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST --cacert example.com.crt https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418
    2. ...
    3. Server certificate:
    4.   subject: CN=httpbin.example.com; O=httpbin organization
    5.   start date: Oct 27 19:32:48 2019 GMT
    6.   expire date: Oct 26 19:32:48 2020 GMT
    7.   common name: httpbin.example.com (matched)
    8.   issuer: O=example Inc.; CN=example.com
    9.   SSL certificate verify ok.
    10. SSL certificate verify ok.
    11. ...
    12. HTTP/2 418
    13. ...
    14. -=[ teapot ]=-
    16.    _...._
    17.  .'  _ _ `.
    18. | ."` ^ `". _,
    19. \_;`"---"`|//
    20.   |       ;/
    21.   \_     _/
    22.     `"""`

    It might take time for the gateway definition to propagate so you might get the following error:Failed to connect to httpbin.example.com port <your secure port>: Connection refused. Wait for a minute andthen retry the curl call.

    Look for the Server certificate section in the curl output and specifically a line with the matched common name:common name: httpbin.example.com (matched). The line SSL certificate verify ok in the output indicatesthat the server’s certificate was verified successfully. If all went well, you should also see a returnedstatus of 418 along with a nice drawing of a teapot.

    Configure a mutual TLS ingress gateway

    In this section you extend your gateway’s definition from the previous section to supportmutual TLS between external clients and the gateway.

    • Create a Kubernetes Secret to hold the CA certificate thatthe server will use to verify its clients. Create the secret istio-ingressgateway-ca-certs in namespace istio-systemusing kubectl. The Istio gateway will automatically load the secret.

    The secret must be named istio-ingressgateway-ca-certs in the istio-system namespace to align with theconfiguration of the Istio default ingress gateway used in this task.

    1. $ kubectl create -n istio-system secret generic istio-ingressgateway-ca-certs --from-file=example.com.crt
    2. secret "istio-ingressgateway-ca-certs" created
    • Redefine your previous Gateway to change the tls mode to MUTUAL and to specify caCertificates:

    The location of the certificate must be /etc/istio/ingressgateway-ca-certs, or the gatewaywill fail to load them. The file (short) name of the certificate must be identical to the one you created the secretfrom, in this case example.com.crt.

    1. $ kubectl apply -f - <<EOF
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: Gateway
    4. metadata:
    5.   name: httpbin-gateway
    6. spec:
    7.   selector:
    8.     istio: ingressgateway # use istio default ingress gateway
    9.   servers:
    10.   - port:
    11.       number: 443
    12.       name: https
    13.       protocol: HTTPS
    14.     tls:
    15.       mode: MUTUAL
    16.       serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
    17.       privateKey: /etc/istio/ingressgateway-certs/tls.key
    18.       caCertificates: /etc/istio/ingressgateway-ca-certs/example.com.crt
    19.     hosts:
    20.     - "httpbin.example.com"
    21. EOF
    • Access the httpbin service by HTTPS as in the previous section:
    1. $ curl -HHost:httpbin.example.com --resolve httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST --cacert example.com.crt https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418
    2. curl: (35) error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

    It might take time for the gateway definition to propagate so you might still get 418. Wait for a minute andthen retry the curl call.

    This time you will get an error since the server refuses to accept unauthenticated requests. You need to pass _curl_a client certificate and your private key for signing the request.

    • Create a client certificate for the httpbin.example.com service. You can designate the client by thehttpbin-client.example.com URI, or use any other URI.
    1. $ openssl req -out httpbin-client.example.com.csr -newkey rsa:2048 -nodes -keyout httpbin-client.example.com.key -subj "/CN=httpbin-client.example.com/O=httpbin's client organization"
    2. $ openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 0 -in httpbin-client.example.com.csr -out httpbin-client.example.com.crt
    • Resend the previous request by curl, this time passing as parameters your client certificate (additional —cert option)and your private key (the —key option):
    1. $ curl -HHost:httpbin.example.com --resolve httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST --cacert example.com.crt --cert httpbin-client.example.com.crt --key httpbin-client.example.com.key https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418
    3. -=[ teapot ]=-
    5.    _...._
    6.  .'  _ _ `.
    7. | ."` ^ `". _,
    8. \_;`"---"`|//
    9.   |       ;/
    10.   \_     _/
    11.     `"""`

    This time the server performed client authentication successfully and you received the pretty teapot drawing again.

    Configure a TLS ingress gateway for multiple hosts

    In this section you will configure an ingress gateway for multiple hosts, httpbin.example.com and bookinfo.com.The ingress gateway will present to clients a unique certificate corresponding to each requested server.

    Unlike the previous sections, the Istio default ingress gateway will not work out of the box because it is onlypreconfigured to support one secure host. You’ll need to first configure and redeploy the ingress gatewayserver with another secret, before you can use it to handle a second host.

    Create a server certificate and private key for bookinfo.com

    1. $ openssl req -out bookinfo.com.csr -newkey rsa:2048 -nodes -keyout bookinfo.com.key -subj "/CN=bookinfo.com/O=bookinfo organization"
    2. $ openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 0 -in bookinfo.com.csr -out bookinfo.com.crt

    Redeploy istio-ingressgateway with the new certificate

    • Create a new secret to hold the certificate for bookinfo.com:
    1. $ kubectl create -n istio-system secret tls istio-ingressgateway-bookinfo-certs --key bookinfo.com.key --cert bookinfo.com.crt
    2. secret "istio-ingressgateway-bookinfo-certs" created
    • To include a volume mounted from the new created secret, update the istio-ingressgateway deployment.To patch the istio-ingressgateway deployment, create the following gateway-patch.json file:
    1. $ cat > gateway-patch.json <<EOF
    2. [{
    3.   "op": "add",
    4.   "path": "/spec/template/spec/containers/0/volumeMounts/0",
    5.   "value": {
    6.     "mountPath": "/etc/istio/ingressgateway-bookinfo-certs",
    7.     "name": "ingressgateway-bookinfo-certs",
    8.     "readOnly": true
    9.   }
    10. },
    11. {
    12.   "op": "add",
    13.   "path": "/spec/template/spec/volumes/0",
    14.   "value": {
    15.   "name": "ingressgateway-bookinfo-certs",
    16.     "secret": {
    17.       "secretName": "istio-ingressgateway-bookinfo-certs",
    18.       "optional": true
    19.     }
    20.   }
    21. }]
    22. EOF
    • Apply istio-ingressgateway deployment patch with the following command:
    1. $ kubectl -n istio-system patch --type=json deploy istio-ingressgateway -p "$(cat gateway-patch.json)"
    • Verify that the key and certificate have been successfully loaded in the istio-ingressgateway pod:
    1. $ kubectl exec -it -n istio-system $(kubectl -n istio-system get pods -l istio=ingressgateway -o jsonpath='{.items[0].metadata.name}') -- ls -al /etc/istio/ingressgateway-bookinfo-certs

    tls.crt and tls.key should appear in the directory contents.

    Configure traffic for the bookinfo.com host

    • Deploy the Bookinfo sample application, without a gateway:

    Zip

    1. $ kubectl apply -f @samples/bookinfo/platform/kube/bookinfo.yaml@
    • Define a gateway for bookinfo.com:
    1. $ kubectl apply -f - <<EOF
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: Gateway
    4. metadata:
    5.   name: bookinfo-gateway
    6. spec:
    7.   selector:
    8.     istio: ingressgateway # use istio default ingress gateway
    9.   servers:
    10.   - port:
    11.       number: 443
    12.       name: https-bookinfo
    13.       protocol: HTTPS
    14.     tls:
    15.       mode: SIMPLE
    16.       serverCertificate: /etc/istio/ingressgateway-bookinfo-certs/tls.crt
    17.       privateKey: /etc/istio/ingressgateway-bookinfo-certs/tls.key
    18.     hosts:
    19.     - "bookinfo.com"
    20. EOF
    • Configure the routes for bookinfo.com. Define a VirtualService like the one insamples/bookinfo/networking/bookinfo-gateway.yaml:
    1. $ kubectl apply -f - <<EOF
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: VirtualService
    4. metadata:
    5.   name: bookinfo
    6. spec:
    7.   hosts:
    8.   - "bookinfo.com"
    9.   gateways:
    10.   - bookinfo-gateway
    11.   http:
    12.   - match:
    13.     - uri:
    14.         exact: /productpage
    15.     - uri:
    16.         exact: /login
    17.     - uri:
    18.         exact: /logout
    19.     - uri:
    20.         prefix: /api/v1/products
    21.     route:
    22.     - destination:
    23.         host: productpage
    24.         port:
    25.           number: 9080
    26. EOF
    • Send a request to the Bookinfo productpage:
    1. $ curl -o /dev/null -s -v -w "%{http_code}\n" -HHost:bookinfo.com --resolve bookinfo.com:$SECURE_INGRESS_PORT:$INGRESS_HOST --cacert example.com.crt -HHost:bookinfo.com https://bookinfo.com:$SECURE_INGRESS_PORT/productpage
    2. ...
    3. Server certificate:
    4.   subject: CN=bookinfo.com; O=bookinfo organization
    5.   start date: Oct 27 20:08:32 2019 GMT
    6.   expire date: Oct 26 20:08:32 2020 GMT
    7.   common name: bookinfo.com (matched)
    8.   issuer: O=example Inc.; CN=example.com
    9. SSL certificate verify ok.
    10. ...
    11. 200
    • Verify that httbin.example.com is accessible as previously. Send a request to it and see again the teapot youshould already love:
    1. $ curl -HHost:httpbin.example.com --resolve httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST --cacert example.com.crt --cert httpbin-client.example.com.crt --key httpbin-client.example.com.key https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418
    2. ...
    3. -=[ teapot ]=-
    5.    _...._
    6.  .'  _ _ `.
    7. | ."` ^ `". _,
    8. \_;`"---"`|//
    9.   |       ;/
    10.   \_     _/
    11.     `"""`

    Troubleshooting

    • Inspect the values of the INGRESS_HOST and SECURE_INGRESS_PORT environmentvariables. Make sure they have valid values, according to the output of thefollowing commands:
    1. $ kubectl get svc -n istio-system
    2. $ echo INGRESS_HOST=$INGRESS_HOST, SECURE_INGRESS_PORT=$SECURE_INGRESS_PORT
    • Verify that the key and the certificate are successfully loaded in theistio-ingressgateway pod:
    1. $ kubectl exec -it -n istio-system $(kubectl -n istio-system get pods -l istio=ingressgateway -o jsonpath='{.items[0].metadata.name}') -- ls -al /etc/istio/ingressgateway-certs

    tls.crt and tls.key should exist in the directory contents.

    • If you created the istio-ingressgateway-certs secret, but the key and thecertificate are not loaded, delete the ingress gateway pod and force theingress gateway pod to restart and reload key and certificate.
    1. $ kubectl delete pod -n istio-system -l istio=ingressgateway
    • Verify that the Subject is correct in the certificate of the ingress gateway:
    1. $ kubectl exec -i -n istio-system $(kubectl get pod -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}')  -- cat /etc/istio/ingressgateway-certs/tls.crt | openssl x509 -text -noout | grep 'Subject:'
    2.     Subject: CN=httpbin.example.com, O=httpbin organization
    • Verify that the proxy of the ingress gateway is aware of the certificates:
    1. $ kubectl exec -ti $(kubectl get po -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}') -n istio-system -- pilot-agent request GET certs
    2. {
    3.   "ca_cert": "",
    4.   "cert_chain": "Certificate Path: /etc/istio/ingressgateway-certs/tls.crt, Serial Number: 100212, Days until Expiration: 370"
    5. }
    • Check the log of istio-ingressgateway for error messages:
    1. $ kubectl logs -n istio-system -l istio=ingressgateway
    • For macOS users, verify that you use curl compiled with the LibreSSLlibrary, as described in the Before you begin section.

    Troubleshooting for mutual TLS

    In addition to the steps in the previous section, perform the following:

    • Verify that the CA certificate is loaded in the istio-ingressgateway pod:
    1. $ kubectl exec -it -n istio-system $(kubectl -n istio-system get pods -l istio=ingressgateway -o jsonpath='{.items[0].metadata.name}') -- ls -al /etc/istio/ingressgateway-ca-certs

    example.com.crt should exist in the directory contents.

    • If you created the istio-ingressgateway-ca-certs secret, but the CAcertificate is not loaded, delete the ingress gateway pod and force it toreload the certificate:
    1. $ kubectl delete pod -n istio-system -l istio=ingressgateway
    • Verify that the Subject is correct in the CA certificate of the ingress gateway:
    1. $ kubectl exec -i -n istio-system $(kubectl get pod -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}')  -- cat /etc/istio/ingressgateway-ca-certs/example.com.crt | openssl x509 -text -noout | grep 'Subject:'
    2. Subject: O=example Inc., CN=example.com

    Cleanup

    • Delete the Gateway configuration, the VirtualService, and the secrets:
    1. $ kubectl delete gateway --ignore-not-found=true httpbin-gateway bookinfo-gateway
    2. $ kubectl delete virtualservice httpbin
    3. $ kubectl delete --ignore-not-found=true -n istio-system secret istio-ingressgateway-certs istio-ingressgateway-ca-certs
    4. $ kubectl delete --ignore-not-found=true virtualservice bookinfo
    • Delete the directories of the certificates and the repository used to generate them:
    1. $ rm -rf example.com.crt example.com.key httpbin.example.com.crt httpbin.example.com.key httpbin.example.com.csr httpbin-client.example.com.crt httpbin-client.example.com.key httpbin-client.example.com.csr bookinfo.com.crt bookinfo.com.key bookinfo.com.csr
    • Remove the patch file you used for redeployment of istio-ingressgateway:
    1. $ rm -f gateway-patch.json
    • Shutdown the httpbin service:

    Zip

    1. $ kubectl delete --ignore-not-found=true -f @samples/httpbin/httpbin.yaml@

    See also

    Istio as a Proxy for External Services

    Configure Istio ingress gateway to act as a proxy for external services.

    Deploy a Custom Ingress Gateway Using Cert-Manager

    Describes how to deploy a custom ingress gateway using cert-manager manually.

    Configuring Istio Ingress with AWS NLB

    Describes how to configure Istio ingress with a network load balancer on AWS.

    Ingress Gateway without TLS Termination

    Describes how to configure SNI passthrough for an ingress gateway.

    Ingress Gateways

    Describes how to configure an Istio gateway to expose a service outside of the service mesh.

    Kubernetes Ingress with Cert-Manager

    Demonstrates how to obtain Let's Encrypt TLS certificates for Kubernetes Ingress automatically using Cert-Manager.