Golang Application Tracing in Instana
The approach for the cloud age
The traditional monitoring solution that preciously treats each component individually is no longer being able to cope with the cloud-native world anymore, simply because the number of moving parts is too many in nowadays infrastructure and applications. Auto-discovery, auto-monitoring is kind of a must for cloud age monitoring.
Instana is such a tool that “Free your Dev+Ops teams from the burden of manual observability and application monitoring” by automating the discovery and monitoring for the infrastructure, services, and application.
For the infrastructure, such as the OS and Kubernetes platform, it’s intuitive to understand that once an agent is deployed on these standardized platforms, the auto-discovery and auto-monitoring can be performed. The same applies to those middlewares also.
But how about the applications? The answer will be something like the protocol, as long as the application conforms to some protocols that both sides mutually agree on, then the Instana is able to auto-discover and auto-monitor the application. Notice this is possible nowadays as the line between Dev/Ops is blurred.
Using the Golang application as an example, this paper examines these protocols for Instana to monitor a Golang HTTP handler application. In the last part, we will also examine the deployment of such Instana enabled applications on both VM and Kubernetes.
The init() function
In my sample Golang app, import Instana's go-sensor. Define the init() function as below.
import (
... instana "github.com/instana/go-sensor"
"github.com/opentracing/opentracing-go" ...
)var iSensor *instana.Sensorfunc init() {
instana.InitSensor(&instana.Options{
Service: "myapp",
LogLevel: instana.Debug,
EnableAutoProfile: true}) iSensor = instana.NewSensor("myapp-tracing")
}
By doing so, the basic metrics, auto profiling are enabled, and code instrumentation can be performed.
- Enable Basic Metric Collecting
The InitSensor() call enables the basic golang metric collection, such as the Heap usage, GC status, and so on. A sample is shown in the following graph. (Goto Infrastructure, Filter by “myapp”)
2. Enable AutoProfile
Meantime, I have turned on AutoProfile by setting the EnableAutoProfile fields as true.
Unlike the native Golang profiling, in which you need to instruct your app to start the profiling manually, Instana’s Golang profiling is auto-scheduled and continuously performed. The following graph shows the profiling result. (Goto Analyze, Profiles, filter by “myapp”)
The flame chart to identify the hotspot is available on the dashboard. A sample for CPU flame chart is shown as below,
The profiling is continuous, each small icon of a clock on the graph indicates a profiling result. By default, the profiling occurs every 2 minutes.
3. Enable Code Instrumentation
The above two are not enough if you want to track the application usage. In order for Instana to discover the type of the application and monitor it automatically, code instrumentation is required. The instana.NewSensor() in the init() function, enables the code instrumentation.
In the sample app, I am creating an HTTP server using the common golang HTTP handler. For an example
func greet(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hello")
}To enable the auto-discover and auto monitor, the handler register function need to be modified from the default, http.HandleFunc("/", greet), to the below
http.HandleFunc("/", instana.TracingHandlerFunc(iSensor, "/", greet)
)Basically, we wrap the handler function with Instana’s handler function so that code instrumentation can be run.
Now if you Goto Application, Services, Filter myapp, the following is shown,
Instana auto-discover the HTTP service of myapp and start to monitor the HTTP type of application. We can drill down further to find more details of the HTTP app.
Click on the Analyze Calls, we can see the HTTP get calls.
For each specific call, we can find its details as shown below.
Notice there is no further tracing for the GET/ call. This brings us to the tracing topic.
Enable Function Level Tracing
Inside the handler, Let’s add the tracing capability.
func myhandler(w http.ResponseWriter, r *http.Request) {
var span opentracing.Span
if parent, ok := instana.SpanFromContext(r.Context()); ok {
span = parent.Tracer().StartSpan("writeHandler", opentracing.ChildOf(parent.Context()))
defer span.Finish()
}...
span.SetTag("count in million", num)...
Instana’s tracing is an implementation of the Opentracing. The concept of tracer, span applies without a difference.
As we wrap our handler with Instana’s TracingHandlerFunc function, a parent span is already available. Get it from the context, then we start a new child span and finish it using the defer function.
We can set some taggings to identify the call when analyzing the call later.
Inside the handler, we can further trace a sub-call say it goes call another service or simply calling another function. The same technique to get the parent tracer can be applied by getting the tracer from the context.
if parent, ok := instana.SpanFromContext(r.Context()); ok {
span = parent.Tracer().StartSpan("writeHandler", opentracing.ChildOf(parent.Context()))
defer span.Finish()
}Following is another screenshot of such case,
Deployment
After enabling the basic metric collecting, profiling, and tracing at the code level, how do we deploy the app to let it communicate to the backend and report the monitoring and tracing?
Instana is taking the agent and server approach. When we init the go-sensor, the sensor will try to communicate with the agent on the host VM and in turn communicate with the Instana agent.
When the sensor is inited with the instana.Options, it can take the AgentHost (default to localhost) and AgentPort (default to 42699) fields, or read from the environment variable INSTANA_AGENT_HOST and INSTANA_AGENT_PORT respectively.
When we deploy the trace-enabled application on a VM where an Instana agent is installed, we just use the default agent host and port value for it to register to the backend.
When deploy onto Kubernetes, the configuration change sightly. Instana agents run on each node of the cluster with a DaemonSet. It uses the hostNetwork to run the agent listening on the default 42699 port. Based on this, without changing the code configuration, we can set the environment variables for the sensor to register to the agent in the YAML configuration. A sample deployment YAML of the Golang app is shown below,
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
labels:
app: myapp
spec:
replicas: 1
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: image-registry.openshift-image-registry.svc:5000/go-trace/myapp:v1.0
ports:
- containerPort: 8080
env:
- name: INSTANA_AGENT_HOST
valueFrom:
fieldRef:
fieldPath: status.hostIP
The INSTANA_AGENT_HOST is taken the referenced field value of status.hostIP, which is the node’s IP. We didn’t define the port, the sensor program take the default value.
The Golang app running in Kubernetes, therefore, can be auto-discovered and monitored in Instana with the metrics, auto profiling, and tracing capabilities.
