GO’CIRCUIT
A project by Petar Maymounkov.

Build, cross-build and deploy an application

 

Each circuit application consists of one or more command-line executables, which we call commands for short, and a single worker executable. Commands are administration tools that start, stop or alter the execution of a circuit application. Both target exeutable types can be built as easily as running the standard Go build tool go build. This method is particularly suited for commands, which are usually meant to be executed from the developer's shell.

Worker executables, however, typically run on a remote cluster with hardware differing from that of the developer's machine. To build a worker executable for the “cluster hardware”, one would have to log into a cluster machine and perform the build there. Doing so manually prohibits fast build-deploy iterations, especially during development.

To remedy this, the circuit includes a complementary build system that makes cross-building a breeze. Furthermore, included is an deploy tool which integrates with the cross-build system and is responsible for uploading newly built worker executables to the cluster machines. This toolchain is often more convenient to use even when building circuit applications that are intended to run entirely on the developer's machine.

Table of contents

Synopsis
Overview
Build
Cross-build
Prepare the build server
Configure the cross-build
Run the cross-build
Deploy
Background
Run the deploy tool
Results of deploying

Synopsis

% CIR=app.config 4crossbuild                   # cross-build worker
% CIR=app.config 4deploy < list_of_hosts       # deploy worker
% go install app_start_cmd_pkg                 # build launch command
% app_start_cmd                                # deploy

Overview

Each circuit application consists of one or more command-line programs—the commands—and a worker program—the worker. Commands are what you invoke on the command-line to launch your circuit application, or to alter its behavior mid-way through its execution in some application specific way. The worker is an executable that is to be pre-installed on all hosts in your cluster. When a spawn of a new worker is performed within a circuit application, the underlying spawning mechanism locates the worker executable on the target hosts and executes it.

Building a circuit application entails building the command and worker binaries for the respective target systems where they will be executed. Command executables are almost always executed from the developer's terminal or a similar administrator's terminal (e.g. engineers often use Apple MacBook personal terminals to manage a Linux PC-powered datacenter). They can therefore be built natively on the developer's terminal using a straight up invocation of go build.

Worker executables, on the other hand, are destined for execution in the datacenter, which will often run on different hardware. Currently, worker executables cannot be cross-compiled on the developer's machine due to dependence on C drivers for Apache Zookeeper. Cross-building can be accomplished manually by performing a native build on a target host. Doing so manually poses a practical impedence to rapid build-deploy cycles. To remedy this, we offer an entirely optional cross-build system that fully automates the common case. In our experience, the cross-build system has been so handy that we use it for native builds as well.

Illustration of the build–deploy workflow.

In the remainder of this chapter, we describe a common workflow (illustrated above) whereby command executables are built on the (local) developer console, worker executables are cross-built on a remote host, then deployed to the cluster, and eventually the commands are invoked locally to deploy the application.

Build

Building a command executable manually is done in a straightforward manner using the go build tool:

  1. Add the GOPATH directory of your application sources to the GOPATH environment variable. For example,
    % export GOPATH=/projects/myapp:$GOPATH
    
    would instruct the Go compiler to look for your packages in /projects/myapp/src.
  2. Make sure you have the CGO_CFLAGS and CGO_LDFLAGS environment variables pointing to the Zookeeper include and library files (see the circuit build manual for details) on this machine.
  3. For each executable that you would like to build, change to the package directory of that executable and simply run go build.

The manual build method is useful for building the commands of your circuit application as they are usually invoked from the developer's shell. For an example, see the Hello, world! tutorial.

Cross-build

In the realm of cross-building we call the developer's environment—where the cross-build is initiated—the build client, and the remote host environment—where the actual build is taking place—the build server. At high level, the developer invokes the cross-build tool at the command line and supplies to it the application configuration file. The cross-build tool logs into the build server using the ssh credentials of the developer, executes the build tool against the application configuration and downloads the resulting binaries back to the build client.

Prepare the build server

A few simple provisions need to be made for a host machine to be a build server.

  1. The following software must be installed on the build server:
  2. The Apache Zookeeper C driver's include and library files must be present on the build server local file system. (Their location is specified to the build system in the configuration file, discussed in the next section.) You can refer to the instructions for installing Zookeeper during the initial circuit build.
  3. The developer's environment at the build client must be able to connect via ssh into the build server without using a password.

    To do so, generate ssh credentials on the build client using ssh-keygen. Then concatenate the file ~/.ssh/id_rsa.pub—at the build client—to the end of ~/.ssh/authorized_keys—at the build server. And make sure sshd is accepting connections to the build server.

  4. When the cross-build tool executes shell scripts on the build server through ssh, it obtains a much more limited system-mandated shell environment than what one gets with an interactive login. Often this limited environment contains a basic PATH variable that does not capture binary install directories used by higher-level packaging tools. It is hence not uncommon that the tools installed in Step 1 might not be “visible” by default within the restricted shell.

    To address this issue in a non-invasive fashion, the Build section of the application configuration contains a field, called PrefixPath, that allows one to specify a PATH environment variable value that will be prepended to the default PATH value of the restricted build server shell. So, for example,

    {
    	…
    	"Build": {
    		…
    		"PrefixPath": "/usr/local/bin:/usr/bin"
    	}
    }
    

Configure the cross-build

The key step in the process is writing the application configuration file which describes how to compile the final product. That said, the process is pretty straightforward.

The cross-build process utilizes the Deploy and Build sections of the configuration. These are well documented in the configuration reference. Regardless, we review them briefly here for continuity. Consider a running example:

{
	…
	"Deploy": {
		…
		"Worker":           "worker_binary_name"
	},
	"Build": {
		"Host":             "build.datacenter.net",
		"Jail":             "/home/petar/build/myapp",
		"ZookeeperInclude": "/home/petar/gocircuit/misc/starter-kit-linux/zookeeper/include",
		"ZookeeperLib":     "/home/petar/gocircuit/misc/starter-kit-linux/zookeeper/lib",
		"Tool":             "/home/petar/gocircuit/bin/4build",
		"PrefixPath":       "/usr/local/bin",

		"AppRepo":          "{git}git@github.com:petar/circuit_app_project.git",
		"AppSrc":           "/GOPATH_relative_to_app_repo",

		"GoRepo":           "{hg}{branch:release}code.google.com/p/go",
		"RebuildGo":        false,

		"CircuitRepo":      "{hg}code.google.com/p/gocircuit",
		"CircuitSrc":       "/",

		"WorkerPkg":        "worker_pkg",
		"CmdPkgs":          ["cmd/myapp-spawn"],
		"ShipDir":          "/home/petar/ship/myapp"
	}
}

Build server environment:

Application sources:

Go Compiler sources:

Circuit sources:

Output:

Run the cross-build

Assuming the application configuration file is called, say, app.config, launching the cross-build is fairly simple:

% CIR=app.config 4crossbuild
If cross-building terminates successfully, you will find (i) a worker executable, (ii) an internal “helper” executable, as well as (iii) zero or more command executables—all delivered to the local shipping directory. In our example above, this would be /home/petar/ship/myapp.

Deploy

Background

The deploy tool, 4deploy, integrates with the cross-build tool by sharing the application configuration file. What this means in practice is that 4deploy can be invoked directly after 4crossbuild without any additional configuration steps. The deploy tool will automatically know where to find the built binaries resulting from the cross-build and will proceed to upload and install them to the cluster hosts that you specify.

The Deploy section of your application configuration file serves a double purpose. It tells the deploy tool where to install the circuit binaries—within the local file system of each cluster host—and it tells the underlying spawning mechanism where to look for these binaries at runtime. Within the configuration file, the Deploy section might look something like this

{
	…
	"Deploy": {
		"Dir":    "/deploy/firehose_dev_v2",
		"Worker": "circuit-firehose_dev_v2"
	},
}

The deploy tool also peeks at the Build section of the configuration file, in order to determine the location of the built binaries on the local machine. In particular, this is the directory specified as Build.ShipDir, as in the following example,

{
	…
	"Build": {
		…
		"ShipDir": "/Users/petar/ship/firehose_dev_v2",
	}
}

Running the deploy tool

The deploy tool can be invoked after a successful cross-build has completed. Similar to other circuit tools,

  1. 4deploy expects that the environment variable CIR points to a local file holding the application configuration,
  2. It takes no command-line arguments and it expects the list of target hosts to be supplied on its standard input, one host name (no port) per line. And, finally,
  3. The user environment where 4deploy is being invoked should be able to log into each target host with a password-less ssh session.

Without further ado, a typical invocation of 4deploy looks like so

% 4deploy < cluster_hosts
Here cluster_hosts is a text file containing host names, one per line. The tool will install to all hosts in parallel, using throttling techniques to prevent contention when the target host cluster is big (to the tune of 100 or more machines). The tool will try to install the software on all hosts. If any of the hosts fails to install, the 4deploy invocation will return a non-zero execution error code as well as informative print outs.

Results of deploying

The result of running 4deploy is quite simple. If we let /deploy be the deploy directory and worker be the name of the worker binary—both specified in the configuration—then the invocation of 4deploy results in creating the following directory structure on each target cluster host:

/deploy/bin                      # directory for circuit binaries
/deploy/bin/worker               # worker executable
/deploy/bin/4clear-helper        # internal tool executable
/deploy/jail                     # empty directory for worker runtime jails