OpenShift: DNS Server Misbehaving
--
I am deploying an Operator based product onto the OpenShift cluster, however the reconciling error message of the operator gives me,
DNS Error: Server Misbehaving. Please check DNS Settings.Sure enough, the DNS server is having problem. But it worth a study on how the DNS system works in the OpenShift cluster.
Resolving HTTP Handler
Lets have a toy HTTP handler program to resolve a hostname.
func dnscheck(w http.ResponseWriter, r *http.Request) {
host := r.FormValue("host")
resolver := net.Resolver{}
ips, err := resolver.LookupIPAddr(r.Context(), host)
if err != nil {
fmt.Fprintf(w, "Failed to resolve %s: %v", host, err)
return
} else {
fmt.Fprintf(w, "Resolved %s to %v", host, ips)
}
}Build the container image, push into the OpenShift registry, deploy as a Deployment, expose the service. Then we can test it as below,
We can resolve a service name within in the same namespace. (Command breaks into two lines just for viewing purpose)
curl "http://app-service-dns-misbehaving.apps.dev-ocp410.ibmcloud.io.cpak/\
dnscheck?host=app-service"
Resolved app-service to [{172.30.73.189 }]
We can also resolve a external name such as,
curl "http://app-service-dns-misbehaving.apps.dev-ocp410.ibmcloud.io.cpak/\
dnscheck?host=google.com"
Resolved google.com to [{142.250.113.138 } {142.250.113.139 } {142.250.113.102 } {142.250.113.113 } {142.250.113.100 } {142.250.113.101 } {2607:f8b0:4023:1000::71 } {2607:f8b0:4023:1000::8b } {2607:f8b0:4023:1000::65 } {2607:f8b0:4023:1000::64 }]Name Resolving in a Pod
When a pod is being deployed, based on the .spec.dnsPolicy settings the kubelet will create the /etc/resolv.conf for the pods, where the DNS server for the pods is specified,
The default value for dnsPolicy is “ClusterFirst”, then what we will have for /etc/resolv.conf is listed as below,
oc exec -it app-6ccb45c4f7-whsxh -- bash -c "cat /etc/resolv.conf"
search dns-misbehaving.svc.cluster.local svc.cluster.local cluster.local dev-ocp410.ibmcloud.io.cpak
nameserver 172.30.0.10
options ndots:5
