GO’CIRCUIT
A project by Petar Maymounkov.

Hello, world: Implement, build, cross-build, deploy

 
This is an end-to-end self-contained tutorial, assuming only that the Go Circuit is installed. The tutorial introduces the concept of spawning remote goroutines programmatically, while maintaining transparent cross-process type safety. The tutorial then goes on to describe the build, deploy and sustenance workflows.

Overview

In this tutorial, we implement, build and deploy a “Hello, world!” circuit app from scratch. The final product will be a binary, called hello-spawn, which when executed from a terminal will behave as follows:

Here's a schematic view of this:

Illustration of the Hello, world! application logic.

Running hello-spawn on the developer console, will produce a similar output instantaneously:

% ./hello
Staring...
Spawned R72fd87ca40beca64@127.0.0.1:55118
Time at remote on spawn: 2:59PM
(Application exits.)

On the remote machine, where the spawned worker is executing, one would see (there is a way to observe the output of a remote worker):

(Delay of 30 seconds.)
Hello world!
(Worker exits.)

In both figures above, the text in brackets is not part of the output; it represents annotations that clarify what is taking place. The source code of the project described here can be found in the tutorials/hello directory within the circuit repository.

Prerequisites

For this tutorial you need to have built the Go Circuit project and setup your environment, as described in download, build and install manual. In particular, make sure that you have the CGO_* and GOPATH environment variables set. E.g. if you are using sh, you could check like so:

% env | grep CGO
CGO_LDFLAGS=/Users/petar/gocircuit/misc/starter-kit-osx/zookeeper/lib/libzookeeper_mt.a
CGO_CFLAGS=-I/Users/petar/gocircuit/misc/starter-kit-osx/zookeeper/include
% env | grep GOPATH
GOPATH=/Users/petar/gocircuit

Your actual directories may vary.

Implementation

Moving right along, begin by creating a directory for this project's source root. From within your home directory, execute

% mkdir -p hello/src/hello

The main directory hello will enclose your project (sources and other things), whereas hello/src will be the GOPATH of your project. Finally, no source files will reside outside of , so as to make sure that all project-related package paths begin with hello/…. As in every Go project, you need to make the Go compiler aware of your project, by including its root directory in the GOPATH environment variable. Do this as follows:

% export GOPATH=$GOPATH:/Users/petar/hello

Notice, we are retaining any prior GOPATH and adding the new project path, separated by a colon.

Main logic

Take a look at this listing. This is the body of our hello-spawn program, and you should place it in src/hello/cmd/hello-spawn/main.go within the root project directory. (The column numbers in the left gutter are not part of the source listing.)

 	package main

	import (
1:		_ "circuit/load/cmd"	// Link the circuit into this executable
2:		"circuit/use/circuit"	// Import the circuit language API
 		"time"
3:		"hello/x"		// Import the package implementing the worker function
	)

 	func main() {
		println("Starting...")

4:		retrn, addr, err := circuit.Spawn(
			"localhost",                // Host where to spawn the new worker
			[]string{"/hello"},         // List of anchors for the new worker
			x.App{},                    // User type that encloses the worker function
			"world!",                   // Argument to pass to worker function
		)
		if err != nil {
			println("Oh well", err.Error())
			return
		}

5:		println("Spawned", addr.String())

6:		remoteTime := retrn[0].(time.Time)
		println("Time at remote on spawn:", remoteTime.Format(time.Kitchen))
	}

We now explain the program line-by-line, following the progression in which one designs the program:

Worker logic

We are going to place the worker function in package hello/x, located in src/hello/x/hello.go.

	package x

	import (
		"fmt"
		"time"
 1:		"circuit/use/circuit"
	)

 2:	type App struct{}

 3:	func (App) Main(suffix string) time.Time {
 4:		circuit.RunInBack(func() {
			time.Sleep(30*time.Second)
			fmt.Printf("Hello %s\n", suffix)
		})
 5:		return time.Now()
	}

 6:	func init() { circuit.RegisterFunc(App{}) }

Worker executable

When spawning a new worker, the spawning mechanism expects to find a worker executable on the target host. Once started, the worker effectively waits for incoming network requests from other workers, asking it to spawn one function or another.

For a worker to be able to execute any user-defined worker function, it needs to have any such function linked into it. For this reason, implementing the worker program is the responsibility of the application programmer. This task however is quite straightforward.

Below is the source listing for the worker executable for this tutorial. Save this code as src/hello/worker/worker.go:

package main

import (
        // Package worker ensures that this executable will act as a circuit worker
        _ "circuit/load/worker"

        // Importing hello/x ensures that the hello tutorial's logic is linked into the worker executable
        _ "hello/x"
)

// Main will never be executed.
func main() {}

The import of circuit/load/worker has the side-effect of making this program behave as a circuit worker. In particular, the init() function of this package hijacks program execution on startup and main() is never started. This would be a boilerplate import in every worker program one writes.

The import of package x has the side-effect of executing its init() function which, if you recall, registers the user-defined type x.App with the circuit runtime. As a consequence, the resulting worker executable will be able to accept spawn requests for x.App.

Principles of build and deploy

In this section, we explain the principles of operation of the build and deploy systems. Step-by-step hands-on instructions for performing the build will come in the following sections.

To build this application, we have to understand how the circuit works and what exactly is being built. Two executable files are to be built:

The command executable is often invoked from the same or similar system to the developer console, which is why one can build it directly using the go build tool. And we cover this in detail later.

The worker executable, on the other hand, is usually hosted on a “datacenter” machine that typically differs in hardware from the developer console. The worker cannot be cross-built on the developer console, because the circuit links against C libraries (for Apache Zookeeper) and in such cases, the Go toolchain requires a native build on the target hardware.

To facilitate the process of logging into a remote host and performing a build there, the circuit provides a cross-build tool that does this automatically. Furthermore, the output of this tool (which is delivered locally to the developer console) is integrated with a subsequent deploy tool that simplifies the deployment of the freshly built binaries to the remote host(s).

How cross-building works

The cross-build tool is embodied in the command-line program 4crossbuild. To build the worker binary, we first need to create a configuration file (and we are going to do this in the next section) that specifies the build parameters for the application at hand and then invoke the cross-build tool with this configuration. The invocation itself is quite simple:

% CIR=app.config 4crossbuild
We point the CIR environment variable to the configuration file and run 4crossbuild in this environment.

Let us understand what 4crossbuild does at high level. First, it connects to a remote build host via ssh. There it executes the circuit command 4build, which does the actual building. When the build completes, the resulting binaries are downloaded back to the developer console in a designated directory.

4build creates a build jail directory which encloses all build-related files. In this jail, 4build fetches the Go compiler repository, the circuit repository, as well as the repository of the user application. It then goes on to build the Go compiler (if this is the first invocation of 4build on this project) as well as the user application's worker executable and any number of command executables. At the end, all resulting binaries are downloaded back to the developer console where 4crossbuild was invoked.

Note that 4build requires that prerequsite technologies for building the Go Compiler be installed on the build host a priori and be accessible through the PATH environment there.

How deploying works

After a successful cross-build, the next step is to upload the compiled binaries to all hosts that might be used as targets for spawning new workers. This is achieved by the deploy tool, embodied in 4deploy.

The deploy tool integrates nicely with the cross-build tool. The product of cross-building is delivered to a local directory, which we call the shipping directory. It will contain a worker executable and possibly other files.

The deploy tool is aware of the shipping directory (as it shares the configuration file with the cross-build tool), so it can be invoked immediately after the cross-build completes. It expects a list of target hosts on its standard input, one per line. A typical invocation looks like:

% CIR=app.config 4deploy < list_of_hosts

The deploy tool connects to each of the supplied hosts in parallel, using ssh, and takes the necessary steps to install the worker executables there.

You must configure your developer environment to be able to connect to each of these hosts via a password-less ssh login.

Configuration

Create the following configuration file inside the main project directory hello and name it app.config:

{
    "Build": {
            "Host":             "localhost",
            "Jail":             "/Users/petar/.hello/build",
            "ZookeeperInclude": "/Users/petar/gocircuit/misc/starter-kit-osx/zookeeper/include",
            "ZookeeperLib":     "/Users/petar/gocircuit/misc/starter-kit-osx/zookeeper/lib",
            "Tool":             "/Users/petar/gocircuit/bin/4build",
            "PrefixPath":       "",

            "AppRepo":          "{rsync}/Users/petar/gocircuit",
            "AppSrc":           "/tutorials/hello",

            "GoRepo":           "{rsync}/Users/petar/go",
            "RebuildGo":        false,

            "CircuitRepo":      "{rsync}/Users/petar/gocircuit",
            "CircuitSrc":       "/",

            "WorkerPkg":        "hello/worker",
            "CmdPkgs":          ["hello/cmd/hello-spawn"],
            "ShipDir":          "/Users/petar/.hello/ship"
    },
    "Deploy": {
            "Dir":              "/Users/petar/.hello/deploy",
            "Worker":           "4hello"
    }
}

This listing is merely an example. You would have to modify some of the parameters to fit your environment, as we describe next.

Even though cross-building is usually invoked to perform a remote build on different hardware, nothing prevents the remote build host from being localhost. This is what we intend to do in this tutorial to make it accessible to most people. You can make your own changes accordingly, if you prefer to use an actual remote build host.

 

Build

The "Build" configuration section instructs the build tool how to prepare the binary distribution of your circuit application.


Build host environment:


App sources:


Go compiler:


Circuit sources:


Output:

Deploy

The "Deploy" configuration specifies how your circuit app distribution is installed on your cluster hosts. The build system also needs this configuration to read the name of the worker binary that you have chosen.

Cross-build

Having saved the configuration file, the cross-build can be invoked from any location. Go ahead and build the tutorial app:

% CIR=app.config 4crossbuild
Building circuit on localhost
/Users/petar/gocircuit/bin/4build \
	'-binary=4hello' '-jail=/Users/petar/.hello/build' \
	'-app={rsync}/Users/petar/gocircuit' '-appsrc=/tutorials/hello' \
	'-workerpkg=worker' '-show=true' '-go={rsync}/Users/pet
localhost:4build/err|  No previous build flags found in jail.
localhost:4build/err|  Building Go compiler
localhost:4build/err|  % cd
localhost:4build/err|  % rsync -acrv --delete --exclude .git --exclude .hg --exclude *.a /Users/petar/go/* /Users/petar/.hello/build/go/
localhost:4build/err|  rsync -acrv --delete --exclude .git --exclude .hg --exclude *.a /Users/petar/go/* /Users/petar/.hello/build/go/
localhost:4build/err|  building file list ... done
…
localhost:4build/err|  Shipping install package
localhost:4build/err|  --Packaging worker 4hello
localhost:4build/err|  --Packaging helper 4clear-helper
localhost:4build/err|  --Packaging command cmd/hello-spawn
localhost:4build/err|  Build successful!
Downloading from /var/folders/l9/fhmztmg504bdt4pjc6jtm6_00000gq/T/4d65822107fcfd52
Download successful.
Done.
%

Deploy

After a successful build, you can go ahead and deploy the compiled binaries to your hosts. In our case, the deploy will be to the local host:
% echo localhost | CIR=app.config 4deploy
Installing circuit.
Installing on localhost
Done.
%

Build the app spawning command

We implemented the program hello-spawn to start the tutorial circuit application. Since we intend to invoke this command from the developer console, we need to build it on the developer console (on native hardware).

Since our cross-build actually happened on the local host, hello-spawn was already built and can be found in the shipping directory. In general, however, the cross-build will be performed on a remote host with different hardware. In this case, we would still have to build hello-spawn on the developer hardware. We are going to demonstrate how to do this here.

As mentioned before, one builds a program binary the same way one builds any Go binary—using the go build tool. First, remember to make sure that the app sources are within the GOPATH environment:

% export GOPATH=/Users/petar/gocircuit/tutorials/hello:$GOPATH
And make sure that all other environment that was prepared while building and installing the circuit is in effect. Next, change to the source directory of hello-spawn. In my case (modify accordingly):
% cd /Users/petar/gocircuit/tutorials/hello/src/cmd/hello-spawn
% go build
This step will leave you with a program binary, hello-spawn, inside the same source directory. At this point we are ready for the final step: running the application.

Launch Zookeeper

The current implementation of the circuit requires a running Zookeeper instance for a variety of its internal functions. (This is a limitation that will be lifted eventually.) Before we can run our application, we need to launch a Zookeeper instance.

One way to do so is to use the pre-packaged Zookeeper binaries that come with the circuit. If you are running on OS X, for example, the following command will do the trick:

% /Users/petar/gocircuit/misc/start-kit-osx/zookeeper/usr/start.sh
Of course, modify the directory path to your environment.

Run the application

All circuit command-line tools, application commands (like hello-spawn) as well as worker executables, henceforth circuit executables, need to be aware of “where the circuit is” before they can begin execution. An individual circuit is embodied in a Zookeeper cluster and the root directory inside it that houses all circuit-related data. Hence, before a circuit executable can begin operation, it needs to be made aware of these parameters.

Additionally, a circuit executable needs to be aware of the deploy configuration used so as to be able to find the circuit worker binary when spawning new workers on remote hosts.

These configuration parameters are passed to circuit executables in the same manner we passed the configuration to the cross-build and deploy tools. We let the CIR environment variable points to the same configuration file, app.config, that we prepared for the build and deploy, but we need to add one more section to it.

We are going to add a Zookeeper section, pointing to the local instance of Zookeeper that we launched in the previous section. Here is the new addition to the configuration:

{
	…
	"Zookeeper": {
		"Workers": ["localhost:2181"],
		"Dir":     "/circuit/hello"
	}
}

We are now ready to run. In this example, from the source directory of hello-spawn:

% CIR=app.config ./hello-spawn
Starting ...
Spawned R0de1473b3c192b4c@127.0.0.1:65530
Time at remote on spawn: 12:42AM
%

After hello returns, the spawned process will linger for another 30 seconds, by design. During this time, we can find this process by listing the contents of the anchor file system:

% CIR=app.config 4ls /...
/hello
/hello/R0de1473b3c192b4c
/host
/host/localhost
/host/localhost/R0de1473b3c192b4c
%

The paths ending with something looking like R6aca01d945c3b861 are files, corresponding to workers. The others are directories. Each worker process, uniquely defined by its worker ID—the R6aca01d945c3b861 part—will be seen within multiple directories: Recall that we can specify multiple anchor directories when we spawn a new worker. Furthermore, the circuit registers all new workers under the anchor /host/{target_host_name}/{worker_id} automatically.

To showcase some of the other tools in the circuit toolbox, try obtaining the stack trace of the running worker in real-time. (You can only do this withing the 30 seconds while that worker is still alive.) Copy any one of the anchors corresponding to the worker from the output of 4ls, and feed it into the stack trace tool 4stk:

% CIR=app.config 4stk /hello/R6aca01d945c3b861
goroutine profile: total 12
1 @ 0x10fcee 0x10faf7 0x10cb12 0xbd137 0x87b8f 0x86beb 0x60452 0x67755 0x6b05d 0x187a0
#       0x10fcee        runtime/pprof.writeRuntimeProfile+0x9e          /Users/petar/.hello/build/go/src/pkg/runtime/pprof/pprof.go:540
#       0x10faf7        runtime/pprof.writeGoroutine+0x87               /Users/petar/.hello/build/go/src/pkg/runtime/pprof/pprof.go:502
#       0x10cb12        runtime/pprof.(*Profile).WriteTo+0xb2           /Users/petar/.hello/build/go/src/pkg/runtime/pprof/pprof.go:229
#       0xbd137         circuit/sys/acid.(*Acid).RuntimeProfile+0x107   /Users/petar/.hello/build/circuit/src/circuit/sys/acid/acid.go:53
#       0x87b8f         reflect.Value.call+0xe9f                        /Users/petar/.hello/build/go/src/pkg/reflect/value.go:474
#       0x86beb         reflect.Value.Call+0x9b                         /Users/petar/.hello/build/go/src/pkg/reflect/value.go:345
#       0x60452         circuit/sys/lang.call+0x432                     /Users/petar/.hello/build/circuit/src/circuit/sys/lang/call.go:50
#       0x67755         circuit/sys/lang.(*Runtime).serveCall+0x495     /Users/petar/.hello/build/circuit/src/circuit/sys/lang/value.go:113
#       0x6b05d         circuit/sys/lang.func·008+0x36d                 /Users/petar/.hello/build/circuit/src/circuit/sys/lang/runtime.go:97

1 @ 0x185f4 0x534b 0x5948 0x2515d 0x251b5 0x2050 0x16838 0x187a0
#       0x2515d circuit/load/worker.init·1+0x2d /Users/petar/.hello/build/circuit/src/circuit/load/worker/worker.go:26
#       0x251b5 circuit/load/worker.init+0x45   /Users/petar/.hello/build/circuit/src/circuit/load/worker/worker.go:0
#       0x2050  main.init+0x40                  /Users/petar/.hello/build/app/tutorials/hello/src/worker/worker.go:0
#       0x16838 runtime.main+0x88               /Users/petar/.hello/build/go/src/pkg/runtime/proc.c:179

…
%

This concludes our first tutorial!