Airgap/Disconnected Installation of OpenShift 3.11

4 min readNov 21, 2019

Some environment is disconnected from the Internet. This adds a little bit of challenge to the OpenShift Container Platform(OCP) installation. This paper summarized the steps for OCP 3.11 installation in an airgap/disconnected environment.

1. RPM packages

During the installation, OCP ansible script validates the required RPMs and install it accordingly from the yum servers. On an Internet-connected Redhat server, we can use the reposync command to download the RPMs to a local directory. Assuming the RPMs have been downloaded and tar-zipped to a file.

In the connected environment, create a new file server VM. I am using Ubuntu as a sample. Install the apache2 package,

sudo apt install -y apache2

Untar the yum RPMs file into /var/www/html/repos.

2. Docker images

Docker images for OCP also need to be downloaded. On the same file server, install docker and run the docker registry container,

sudo docker run -d -p 5000:5000 --name registry registry:2

As we are still internet-connected, login to the Redhat registry with your registered ID and password, download the docker images.

docker login -u {{ .u }} -p {{ .p }} registry.redhat.io

I compiled the list of the images required in a YAML file attached below.

images:
  - registry.redhat.io/openshift3/apb-base:v3.11.135
  - registry.redhat.io/openshift3/apb-tools:v3.11.135
  - registry.redhat.io/openshift3/automation-broker-apb:v3.11.135
  - registry.redhat.io/openshift3/csi-attacher:v3.11.135
  - registry.redhat.io/openshift3/csi-driver-registrar:v3.11.135
  - registry.redhat.io/openshift3/csi-livenessprobe:v3.11.135
  - registry.redhat.io/openshift3/csi-provisioner:v3.11.135
  - registry.redhat.io/openshift3/grafana:v3.11.135
  - registry.redhat.io/openshift3/local-storage-provisioner:v3.11.135
  - registry.redhat.io/openshift3/manila-provisioner:v3.11.135
  - registry.redhat.io/openshift3/mariadb-apb:v3.11.135
  - registry.redhat.io/openshift3/mediawiki:v3.11.135
  - registry.redhat.io/openshift3/mediawiki-apb:v3.11.135
  - registry.redhat.io/openshift3/mysql-apb:v3.11.135
  - registry.redhat.io/openshift3/ose-ansible-service-broker:v3.11.135
  - registry.redhat.io/openshift3/ose-cli:v3.11.135
  - registry.redhat.io/openshift3/ose-cluster-autoscaler:v3.11.135
  - registry.redhat.io/openshift3/ose-cluster-capacity:v3.11.135
  - registry.redhat.io/openshift3/ose-cluster-monitoring-operator:v3.11.135
  - registry.redhat.io/openshift3/ose-console:v3.11.135
  - registry.redhat.io/openshift3/ose-configmap-reloader:v3.11.135
  - registry.redhat.io/openshift3/ose-control-plane:v3.11.135
  - registry.redhat.io/openshift3/ose-deployer:v3.11.135
  - registry.redhat.io/openshift3/ose-descheduler:v3.11.135
  - registry.redhat.io/openshift3/ose-docker-builder:v3.11.135
  - registry.redhat.io/openshift3/ose-docker-registry:v3.11.135
  - registry.redhat.io/openshift3/ose-efs-provisioner:v3.11.135
  - registry.redhat.io/openshift3/ose-egress-dns-proxy:v3.11.135
  - registry.redhat.io/openshift3/ose-egress-http-proxy:v3.11.135
  - registry.redhat.io/openshift3/ose-egress-router:v3.11.135
  - registry.redhat.io/openshift3/ose-haproxy-router:v3.11.135
  - registry.redhat.io/openshift3/ose-hyperkube:v3.11.135
  - registry.redhat.io/openshift3/ose-hypershift:v3.11.135
  - registry.redhat.io/openshift3/ose-keepalived-ipfailover:v3.11.135
  - registry.redhat.io/openshift3/ose-kube-rbac-proxy:v3.11.135
  - registry.redhat.io/openshift3/ose-kube-state-metrics:v3.11.135
  - registry.redhat.io/openshift3/ose-metrics-server:v3.11.135
  - registry.redhat.io/openshift3/ose-node:v3.11.135
  - registry.redhat.io/openshift3/ose-node-problem-detector:v3.11.135
  - registry.redhat.io/openshift3/ose-operator-lifecycle-manager:v3.11.135
  - registry.redhat.io/openshift3/ose-ovn-kubernetes:v3.11.135
  - registry.redhat.io/openshift3/ose-pod:v3.11.135
  - registry.redhat.io/openshift3/ose-prometheus-config-reloader:v3.11.135
  - registry.redhat.io/openshift3/ose-prometheus-operator:v3.11.135
  - registry.redhat.io/openshift3/ose-recycler:v3.11.135
  - registry.redhat.io/openshift3/ose-service-catalog:v3.11.135
  - registry.redhat.io/openshift3/ose-template-service-broker:v3.11.135
  - registry.redhat.io/openshift3/ose-tests:v3.11.135
  - registry.redhat.io/openshift3/ose-web-console:v3.11.135
  - registry.redhat.io/openshift3/postgresql-apb:v3.11.135
  - registry.redhat.io/openshift3/registry-console:v3.11.135
  - registry.redhat.io/openshift3/snapshot-controller:v3.11.135
  - registry.redhat.io/openshift3/snapshot-provisioner:v3.11.135
  - registry.redhat.io/openshift3/ose-ansible:v3.11.135
  - registry.redhat.io/openshift3/metrics-cassandra:v3.11.135
  - registry.redhat.io/openshift3/metrics-hawkular-metrics:v3.11.135
  - registry.redhat.io/openshift3/metrics-hawkular-openshift-agent:v3.11.135
  - registry.redhat.io/openshift3/metrics-heapster:v3.11.135
  - registry.redhat.io/openshift3/metrics-schema-installer:v3.11.135
  - registry.redhat.io/openshift3/oauth-proxy:v3.11.135
  - registry.redhat.io/openshift3/ose-logging-curator5:v3.11.135
  - registry.redhat.io/openshift3/ose-logging-elasticsearch5:v3.11.135
  - registry.redhat.io/openshift3/ose-logging-eventrouter:v3.11.135
  - registry.redhat.io/openshift3/ose-logging-fluentd:v3.11.135
  - registry.redhat.io/openshift3/ose-logging-kibana5:v3.11.135
  - registry.redhat.io/openshift3/prometheus:v3.11.135
  - registry.redhat.io/openshift3/prometheus-alertmanager:v3.11.135
  - registry.redhat.io/openshift3/prometheus-node-exporter:v3.11.135#Other images
  - registry.redhat.io/rhel7/etcd:3.2.22
  - registry.redhat.io/cloudforms46/cfme-openshift-postgresql
  - registry.redhat.io/cloudforms46/cfme-openshift-memcached
  - registry.redhat.io/cloudforms46/cfme-openshift-app-ui
  - registry.redhat.io/cloudforms46/cfme-openshift-app
  - registry.redhat.io/cloudforms46/cfme-openshift-embedded-ansible
  - registry.redhat.io/cloudforms46/cfme-openshift-httpd
  - registry.redhat.io/cloudforms46/cfme-httpd-configmap-generator
  - registry.redhat.io/rhgs3/rhgs-server-rhel7
  - registry.redhat.io/rhgs3/rhgs-volmanager-rhel7
  - registry.redhat.io/rhgs3/rhgs-gluster-block-prov-rhel7
  - registry.redhat.io/rhgs3/rhgs-s3-server-rhel7#S2i images
  - registry.redhat.io/jboss-amq-6/amq63-openshift
  - registry.redhat.io/jboss-datagrid-7/datagrid71-openshift
  - registry.redhat.io/jboss-datagrid-7/datagrid71-client-openshift
  - registry.redhat.io/jboss-datavirt-6/datavirt63-openshift
  - registry.redhat.io/jboss-datavirt-6/datavirt63-driver-openshift
  - registry.redhat.io/jboss-decisionserver-6/decisionserver64-openshift
  - registry.redhat.io/jboss-processserver-6/processserver64-openshift
  - registry.redhat.io/jboss-eap-6/eap64-openshift
  - registry.redhat.io/jboss-eap-7/eap71-openshift
  - registry.redhat.io/jboss-webserver-3/webserver31-tomcat7-openshift
  - registry.redhat.io/jboss-webserver-3/webserver31-tomcat8-openshift
  - registry.redhat.io/openshift3/jenkins-2-rhel7
  - registry.redhat.io/openshift3/jenkins-agent-maven-35-rhel7
  - registry.redhat.io/openshift3/jenkins-agent-nodejs-8-rhel7
  - registry.redhat.io/openshift3/jenkins-slave-base-rhel7
  - registry.redhat.io/openshift3/jenkins-slave-maven-rhel7
  - registry.redhat.io/openshift3/jenkins-slave-nodejs-rhel7
  - registry.redhat.io/rhscl/mongodb-32-rhel7
  - registry.redhat.io/rhscl/mysql-57-rhel7
  - registry.redhat.io/rhscl/perl-524-rhel7
  - registry.redhat.io/rhscl/php-56-rhel7
  - registry.redhat.io/rhscl/postgresql-95-rhel7
  - registry.redhat.io/rhscl/python-35-rhel7
  - registry.redhat.io/redhat-sso-7/sso70-openshift
  - registry.redhat.io/rhscl/ruby-24-rhel7
  - registry.redhat.io/redhat-openjdk-18/openjdk18-openshift
  - registry.redhat.io/redhat-sso-7/sso71-openshift
  - registry.redhat.io/rhscl/nodejs-6-rhel7
  - registry.redhat.io/rhscl/mariadb-101-rhel7

Notice the images with the specific tag version. You can browse the Redhat registry to get the latest tag version. Not the “latest” tag, but the specific tag version such as v3.11.135 . In addition, the etcd image has its own version formating like etcd:3.2.22

As we are just running an HTTP version of the Docker registry, we need to inform the docker engine that this registry is not using HTTPS. Assuming the hostname of the ubuntu server is ubuntu-fileserver, update /etc/docker/daemon.json with the following content,

{
  "insecure-registries" : ["ubuntu-fileserver:5000"]
}

Restart the Docker service. We are ready to pull and push each of the images into the registry. For example,

sudo docker pull registry.redhat.io/openshift3/apb-base:v3.11.135sudo docker tag registry.redhat.io/openshift3/apb-base:v3.11.135 ubuntu-fileserver:5000/openshift3/apb-base:v3.11.135sudo docker push ubuntu-fileserver:5000/openshift3/apb-base:v3.11.135

Once the RPMs and Docker images are ready, we can clone the VM and move it to the disconnected zone for OCP installation.

3. Configure local yum server for the RedHat Enterprise Linux server

Upload the following file to the location of /etc/yum.repos.d/local.repo

[rhel-7-server-rpms]
name=rhel-7-server-rpms
baseurl=http://ubuntu-fileserver/repos/rhel-7-server-rpms
enabled=1
gpgcheck=0[rhel-7-server-extras-rpms]
name=rhel-7-server-extras-rpms
baseurl=http://ubuntu-fileserver/repos/rhel-7-server-extras-rpms
enabled=1
gpgcheck=0[rhel-7-server-ansible-2.6-rpms]
name=rhel-7-server-ansible-2.6-rpms
baseurl=http://ubuntu-fileserver/repos/rhel-7-server-ansible-2.6-rpms
enabled=1
gpgcheck=0[rhel-7-server-ose-3.11-rpms]
name=rhel-7-server-ose-3.11-rpms
baseurl=http://ubuntu-fileserver/repos/rhel-7-server-ose-3.11-rpms
enabled=1
gpgcheck=0

Validate the yum server is working by yum repolist You should see the count of the packages at the last column of the output.

4. Ansible inventory file

A sample inventory file for the installation is attached as below,

[OSEv3:children]
masters
nodes
etcd[OSEv3:vars]
ansible_ssh_user=root
openshift_deployment_type=openshift-enterpriseopenshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]
openshift_master_default_subdomain=apps.ocp.mydomain.iooreg_url=ubuntu-fileserver:5000/openshift3/ose-${component}:${version}openshift_image_tag=v3.11.135
osm_etcd_image=ubuntu-fileserver:5000/rhel7/etcd:3.2.22openshift_docker_insecure_registries=ubuntu-fileserver:5000os_firewall_use_firewalld=True[masters]
master.ocp.mydomain.io[etcd]
master.ocp.mydomain.io[nodes]
master.ocp.mydomain.io openshift_node_group_name='node-config-master'
infra.ocp.mydomain.io openshift_node_group_name='node-config-infra'
worker1.ocp.mydomain.io openshift_node_group_name='node-config-compute'
worker2.ocp.mydomain.io openshift_node_group_name='node-config-compute'
worker3.ocp.mydomain.io openshift_node_group_name='node-config-compute'

Notice the oreg_url variable points to the local file server with the place holder ${component} and ${version} which will be substituted dynamically during installation.

Using the var named as openshift_image_tag to specify the image tag explicitly as what has been downloaded. Etcd image is an exception out of oreg_url , use the variable of osm_etcd_image and give the full URL path of the image.

As the docker registry use HTTP protocol, set the variable of openshift_docker_insecure_registries .

With these settings, the air-gapped/disconnected installation of OCP can proceed successfully.

Airgap/Disconnected Installation of OpenShift 3.11

1. RPM packages

2. Docker images

3. Configure local yum server for the RedHat Enterprise Linux server

4. Ansible inventory file

Written by Zhimin Wen