OpenShift: DNS Server Misbehaving

Zhimin Wen
4 min readJan 16, 2023
Image by Arek Socha from Pixabay

I am deploying an Operator based product onto the OpenShift cluster, however the reconciling error message of the operator gives me,

DNS Error: Server Misbehaving. Please check DNS Settings.

Sure enough, the DNS server is having problem. But it worth a study on how the DNS system works in the OpenShift cluster.

Resolving HTTP Handler

Lets have a toy HTTP handler program to resolve a hostname.


func dnscheck(w http.ResponseWriter, r *http.Request) {
host := r.FormValue("host")
resolver := net.Resolver{}
ips, err := resolver.LookupIPAddr(r.Context(), host)
if err != nil {
fmt.Fprintf(w, "Failed to resolve %s: %v", host, err)
return
} else {
fmt.Fprintf(w, "Resolved %s to %v", host, ips)
}
}

Build the container image, push into the OpenShift registry, deploy as a Deployment, expose the service. Then we can test it as below,

We can resolve a service name within in the same namespace. (Command breaks into two lines just for viewing purpose)

curl "http://app-service-dns-misbehaving.apps.dev-ocp410.ibmcloud.io.cpak/\
dnscheck?host=app-service"
Resolved app-service to [{172.30.73.189 }]

We can also resolve a external name such as,

curl "http://app-service-dns-misbehaving.apps.dev-ocp410.ibmcloud.io.cpak/\
dnscheck?host=google.com"
Resolved google.com to [{142.250.113.138 } {142.250.113.139 } {142.250.113.102 } {142.250.113.113 } {142.250.113.100 } {142.250.113.101 } {2607:f8b0:4023:1000::71 } {2607:f8b0:4023:1000::8b } {2607:f8b0:4023:1000::65 } {2607:f8b0:4023:1000::64 }]

Name Resolving in a Pod

When a pod is being deployed, based on the .spec.dnsPolicy settings the kubelet will create the /etc/resolv.conf for the pods, where the DNS server for the pods is specified,

The default value for dnsPolicy is “ClusterFirst”, then what we will have for /etc/resolv.conf is listed as below,

oc exec -it app-6ccb45c4f7-whsxh -- bash -c "cat /etc/resolv.conf"

search dns-misbehaving.svc.cluster.local svc.cluster.local cluster.local dev-ocp410.ibmcloud.io.cpak
nameserver 172.30.0.10
options ndots:5

--

--