5 min readOct 24, 2018

Implement Capistrano/SSHKit in Golang

I enjoyed a lot using Capistrano/SSHkit deployment with Ruby. With the right toolings, I can manage multiple servers concurrently, finish a task in a snap, perform consistently across different server at a different time. More importantly, the Rake tasks code is a document that describes what you need to do precisely.

However, years after, it did have some unsatisfied area. Other than the sometimes slowness issue of Rake, one of the major issues I encountered was how to deploy it to a green field where there is no Ruby presence and all the dependent library is not there yet…

Thanks to Golang, I am now able to achieve the target deployment goal and yet resolve the problem of deployment of the tool itself.

Puzzle Piece #1: Mage as a Task Describer (Rake as in Golang)

Mage is the right tool to define the tasks in plain golang.

Mage is a make/rake-like build tool using Go. You write plain-old go functions, and Mage automatically uses them as Makefile-like runnable targets

Take an example in the case that I need to create a Docker Image. Using the voodoos of Mage, the task skeleton can be defined as below,

// +build magepackage mainfunc init() {
}// Cross Compile for Linux
func T01_CrossCompile() {
}//Upload cross compiled exe
func T02_Upload() {
}//Build docker image
func T03_Dockerfile() {
}// Push to dockerhub
func T04_Push() {
}

Run mage -l I have my task ready defined,

λ mage -l
Targets:
  t01_CrossCompile    Cross Compile for Linux in
  t02_Upload          Upload cross compiled exe
  t03_Dockerfile      Build docker image
  t04_Push            Push to dockerhub

I can trigger it by using mage -v t01_CrossCompile and I can compile it into a single executable file with

mage -compile MyDeployer.exe

I have the single executable file ready to be deployed to anywhere without any dependency, and run my tasks just as

λ MyDeployer.exe
Targets:
  t01_CrossCompile    Cross Compile for Linux
  t02_Upload          Upload cross compiled exe
  t03_Dockerfile      Build docker image
  t04_Push            Push to dockerhub

There are other cool features in Mage, such as namespace, importing other magefile and so on.

With mage and golang, the issue of the deployment of the deployer is easily resolved.

Puzzle Piece 2: Task Execution

There are really two portions of the problem to tackle with. One is to run a task locally, the other one is to run the task remotely. I have developed a set of utilities to help the task executions.

A. Run Task Locally

Mage already provide a sh utilities to run a command locally. To make the output more easy to read, I implement the execute() function to display stdout and stderr messages with some color.

Sample mage tasks

// +build magepackage mainimport (
 "os" "github.com/zhiminwen/magetool/shellkit"
)func init() {
 os.Setenv("MAGEFILE_VERBOSE", "true")
}//Run Cmd
func RunCmd() {
 shellkit.Execute("cmd", "/c", "type test.log & sleep 10")
}//dir
func Dir() {
 shellkit.Execute("cmd", "/c", "time /t 1>&2 & exit 1")
}

The task displays the stdout in green color, the stderr in red color. It marks the final result with a green tick √ if the exit code is zero, and a red cross x if the exit code is non zero. The overall elapsed time is also reported.

The execute() function use the os/exec library, after defining the command, it gets holds of the stdout and stderr pipes, reads and displays the output accordingly in a separate goroutine. Once the command execution returns, the main goroutine reports the overall time taken and final status.

The output is achieved with the following interface in the “fmtkit” package. You can define and implement your own formatter.

type Formatter interface {
 Header(cmd string, args ...string)
 NormalLine(prefix, line string)
 ErrorLine(prefix, line string)
 Footer(duration time.Duration, err error)
}

B. Program the task remotely

Majority of the time we need to run the task remotely and in a programmed dynamic way. To achieve that the following features are a must to have for any tools or library:

Init the client dynamically
Run command dynamically
Capture command output
Upload a file or upload a string
Download a file
Run command with interactive input

I have these features implemented in my “github.com/zhiminwen/magetool/sshkit” package. They are illustrated as below,

To init the client

//Use password
client1 := sshkit.NewSSHClient("192.168.5.10", "22", "ubuntu", "password", "")//Use private key
s2 := sshkit.NewSSHClient("192.168.5.10", "22", "ubuntu", "", "mykeyfile"// Use a slice of the client
var servers []*sshkit.SSHClientservers = append(servers, sshkit.NewSSHClient("192.168.5.10", "22", "ubuntu", "password", ""))servers = append(servers, sshkit.NewSSHClient("192.168.5.10", "22", "ubuntu2", "password", ""))

The actual connection only happens when there is a need.

2. To execute a command

// Run against a single client
client1.Execute("id; hostname")//Run against slice of servers
sshkit.Execute(servers, "id; hostname")

The sample output is the same as the local command where color code is applied.

The implementation is similar where we get stdio and stderr pipe from the ssh session, display the result using the fmtkit, wait until the command execution finishes, finally report the status.

3. Capture a result

result, err := client.Capture("hostname")

Once we get the result, we can apply some logic to run the next action command dynamically. This is particularly useful when running against a series of clients. For that, we have an ExecFunc() function, see the following example.

sshkit.ExecuteFunc(servers, func(t *sshkit.SSHClient) {
  id, err := t.Capture("id -u")
  if err != nil {
   log.Printf("Failed to get id:%v", err)
   return
  }  if id == "1000" {
   return
  }  cmd := fmt.Sprintf("echo %s; hostname", id)
  t.Execute(cmd)
 })

Here among a list of remote SSH servers, we only run the command when the user id is not 1000. (The example might be too trivial, but the idea is there, hopefully)

4. Upload and Download

client1.Upload("example.txt", "/tmp/example.txt")client1.Download("/etc/hosts", "hosts.txt")

With the Golang io.Reader and io.Writer interface, we can upload a string content to a remote file, or download a remote file as a string content.

client1.Put(`This is a test content`, "mytest.txt")list, err := servers[0].Get("/etc/passwd")

5. Interact with command

Sometimes, its needed to interact through the pseudo TTY. See the following example,

client1.Execute(`rm -rf .ssh/known_hosts`)
client1.ExecuteInteractively("scp ubuntu@ubuntu:/tmp/test.txt /tmp/test.txt.dupped", map[string]string{
  `\(yes/no\)`: "yes",
  "password":   "password",
 })

The ExecuteInteractively() takes two arguments. The first one is the command to execute, the second is a type of map[string][string]. The key of the map is the regex string for matching the output, the value is the content to be entered through the TTY.

Sample program to build a Docker image

As the end, let me put the sample code to achieve the task at the beginning of the story. I will develop the go code on my Windows laptop, cross build as a Linux executable file, build the docker image on my Linux VM, finally push it to docker hub.

The full code sample on Github: https://github.com/zhiminwen/magetool/blob/master/sample/dockerBuild/dockerBuilder.go