Airgap installation is always a challenging thing for OpenShift. By setting up a mirror registry and applying ImageContentSourcePolicy CRD to the cluster, we can instruct the OCI container engine to retrieve the source image from its mirrored image hosted in the mirror registry. This solves the airgap images for the cluster and the apps.
There is still a 3rd type of image for an airgap environment to tackle, that is the Operator related images. This paper documents the Operator based installation in an air-gapped environment, the steps, and the hiccups, and how it is being resolved.
OpenShift manages operators through Operator Catalog.
An Operator catalog is a repository of metadata that Operator Lifecycle Manager (OLM) can query to discover and install Operators and their dependencies on a cluster.
To install an operator in an air-gapped environment, we first need to sync the operator catalog. Starting from OCP4.6, the operator catalog is released as a container image, called an index image.
Create a custom pruned index image
Let's see we are interested in the RedHat-operators of serverless (Knative), pipeline (Tekton), and service mesh (Istio) only. We can keep the operators that we are interested in, prune the rest.
First, we must know what are the available operators from the RedHat operator provider.
In an internet-connected box, run the following index image,
podman run -p50051:50051 --name operator-index -d registry.redhat.io/redhat/redhat-operator-index:v4.7
Then download the grpcurl tool, run
grpcurl -plaintext localhost:50051 api.Registry/ListPackages > packages.out
From the output, we know the operators we are interested in are named as
We then use the opm tool to create a new pruned index image. The opm tool can be downloaded from this URL with OCP 4.7.2 release.
opm index prune -f registry.redhat.io/redhat/redhat-operator-index:v4.7 -p servicemeshoperator,serverless-operator,openshift-pipelines-operator-rh -t my-redhat-operator-index:v4.7
The -f tells where is the source image, -p is the list of operators to keep. Opm will prune the operators, and tag the image created. Notice the image will be saved in the local container storage.
As we need to get this image from a registry in the next step, we need to set up a local registry and push the image into it. Assuming this registry is
ocp47-mirror-registry:5000 The step of setting it up can be referred to in my past paper. To push it into this registry,
podman tag my-redhat-operator-index:v4.7 ocp47-mirror-registry:5000/olm-mirror/my-redhat-operator-index:v4.7podman login ocp47-mirror-registry:5000podman push ocp47-mirror-registry:5000/olm-mirror/my-redhat-operator-index:v4.7
Mirror the catalog into the disk
Download oc command. Assuming we are using the 4.7.2 version.
mkdir -p olmmirror
cd olmmirroroc adm catalog mirror ocp47-mirror-registry:5000/olm-mirror/my-redhat-operator-index:v4.7 file://mirror -a /run/user/1000/containers/auth.json
Use our pruned image from a registry, mirror the catalog (meta data and images) into the file system. Notice the file://mirror, will be mapped to “v2/mirror” in the current directory.
In my setup, the mirrored data is about 30GB and took 30 minutes.
Now tar v2 directory, and copy it to the target offline mirror registry server. As some of the files/directories are symbolic links, we have to use tar to keep the structure correct. It was found the loading will validate the type of the file.
It is noticed that the custom pruned index image is included in the mirror directory, located at
olm-mirror is the namespace we used for the image.
Therefore we don’t need to copy and load this image separately.
Load the catalog into the mirrored registry
In the mirrored registry server, untar the file, cd the directory where you can see the “v2” directory. Run the following command to load the images into the registry.
cd olmmirroroc adm catalog mirror file://mirror/olm-mirror/my-redhat-operator-index:v4.7 dev-airgap47-lb.dev-airgap47.ibmcloud.io.cpak:5000/olm-mirror -a /run/user/1000/containers/auth.json
However, the following error message pop up,
error: unable to retrieve source image file://mirror/olm-mirror/my-redhat-operator-index/openshift-serverless-1-tech-preview/eventing-channel-broker-rhel8 manifest sha256:72d7c6d34f6b83dd65f583b50620936358150f503658480344363f3acefa95fc: open v2/mirror/olm-mirror/my-redhat-operator-index/openshift-serverless-1-tech-preview/eventing-channel-broker-rhel8/manifests/sha256:72d7c6d34f6b83dd65f583b50620936358150f503658480344363f3acefa95fc: no such file or directory
Not exactly sure if this is a bug or expected behavior. To resolve it move all namespace level directory such as
openshift-serverless-1-tech-preview into the
olm-mirror/my-redhat-operator-index subdirectory to satisfy it.
Rerun it. Now we have all the images loaded into the mirror registry.
oc adm catalog mirror command creates an ImageContentSourcePolicy YAML file in a manifest subdirectory after running.
However check it out,
The source is not correct. Change all these prefix
mirror/olm-mirror/my-redhat-operator-index to the actual source of the image, which is
Apply this YAML to the cluster. Monitor the nodes are updated for the mirror registry changes.
Apply the catalog source
A catalog source YAML file is also created in the manifest directory. Update it with some name change,
displayName: Mirrored Operator Catalog
Apply this YAML.
Watch the OperatorHub is populated with the 3 operators we selected. We can then install the operators.