Fixing the Certificate Issue on Chrome for Cloud Pak Common Service

It seems like the recent Chrome more strictly validates the HTTPS certificate and blocks the connection if it deems that the cert is not valid. When using Cloud Pak common services running on-prem OpenShift, one of the issues caused by this is that Chrome blocks access to the console totally as shown in below,

The Reason

OpenShift maps the Kubernetes ingress object to the OpenShift Route object automatically. We can then access the services inside the cluster through the router directly without using the Load Balancer type of service exposure commonly seen on the public cloud.

OpenShift achieve this by registering a wildcard domain name in the DNS server, such as *.apps.<your domain name>. This wildcard domain name will be resolved to the infra node’s IP/Load Balancer IP. The Router pod binds the HTTP port (80) and HTTPS port (443) to the infra nodes’ host network. When the traffic comes to the router, it will forward the traffic to the Kubernetes’ service accordingly.

For Cloud Pak common services, there are two ingresses to be exposed one is the management and the other one is for the applications. For example,

• management ingress: https://icp-console.apps.ocp.fyre.io.cpak/
• application ingress: https://icp-proxy.apps.ocp.fyre.io.cpak/

Again it is implemented with the OpenShift route, which is briefly shown in the chart below.

For the HTTPS traffic, the OpenShift Router has three types of termination strategy.

• Edge: Terminate at the router. Use HTTP traffic within the cluster. The certificate is the OpenShift Router’s certificate
• Passthrough: No termination. Use the certification of the Pod service.
• Reencrypt: Terminate at the router. The certificate is the OpenShift Router’s certificate. Then re-encrypt the traffic to talk to the Pod service. A CA cert can be defined for the traffic.

By default, the Cloud Pak is using Passthrough to implement the route object for the management ingress. This causes the problem of the Chrome browser. Before we access the Cloud Pak management console, we may already test the OpenShift console with the wildcard URL, such as *.apps.ocp.fyre.io.cpak. The certificate is the OpenShift Router’s certificate.

When we access the management console through the URL of https://icp-console.apps.ocp.fyre.io.cpak/ , the certificate is the Cloud Pak management ingress Pod’s certificate. However, the wildcard cert has already been used before. Chrome deems that this cert is fake and therefore it blocks the access totally.

We can’t use the certificate from the management pod directly. We need to use the re-encrypt termination strategy instead.

The Fix

First, we need to switch to re-encrypt termination. Secondly, as the management ingress is using client cert authentication, we have to find the CA certificate for the management ingress, and use that CA cert to create cert for the router to communicate with the management ingress.

oc -n kube-system get secret icp-management-ingress-tls-secret -o jsonpath="{ .data.ca\.crt }" | base64 -d -i
-----BEGIN CERTIFICATE-----
MIIFpzCCA4+gAwIBAgIUIfNdkqW2mvrhMxAfXVIftLf66AQwDQYJKoZIhvcNAQEL
...Skipped...
5Dn9NQu7aBYvUqMC55RPGYPTbhVjUgWKRLHqVDr8fgT0HFE+iEfF+wL4G75GYyl1
ywPIDc/dlx71Xk0=
-----END CERTIFICATE-----

Now let's re-apply the Route object with this CA cert,

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: icp-console
  namespace: kube-system
spec:
  host: icp-console.apps.ocp.fyre.io.cpak
  to:
    kind: Service
    name: management-ingress
    weight: 100
  tls:
    insecureEdgeTerminationPolicy: Redirect  
    termination: reencrypt
    destinationCACertificate: |-
      -----BEGIN CERTIFICATE-----
     MIIFpzCCA4+gAwIBAgIUIfNdkqW2mvrhMxAfXVIftLf66AQwDQYJKoZIhvcNAQEL
...Skipped...
5Dn9NQu7aBYvUqMC55RPGYPTbhVjUgWKRLHqVDr8fgT0HFE+iEfF+wL4G75GYyl1
ywPIDc/dlx71Xk0=
      -----END CERTIFICATE-----
status: 
  ingress: []

Apply it, the console can now be accessed using the Chrome browser.