GO’CIRCUIT
A project by Petar Maymounkov. Partially supported by DARPA XData.
Blog Twitter Source Issues Code review gocircuit-user gocircuit-dev

Paradigm for scale-free engineering

The circuit reduces the human development and sustenance costs of complex massively-scaled systems nearly to the level of their single-process counterparts. It is a combination of proven ideas from the Erlang ecosystem of distributed embedded devices and Go's ecosystem of Internet application development.

The circuit extends the reach of Go's linguistic environment to multi-host/multi-process applications. In simple terms, the Go Circuit was born from the desire to be able to write:

feedback := make(chan int)
circuit.Spawn("host25.datacenter.net", func() {
	feedback <- 1
})
<-feedback
println("Roundtrip complete.")

The Spawn operator will start its argument function on a desired remote host in a new goroutine, while making it possible to communicate between the parent and child goroutines using the same Go code that you would use to communicate between traditional goroutines. Above, the channel feedback is transparently “stretched” between the parent goroutine, executing locally, and the child goroutine, executing remotely and hosting the anonymous function execution.

Using the circuit one is able to write complex distributed applications — involving multiple types of collaborating processes — within a single circuit program. The circuit language used therein is syntactically identical to Go while also:

  • Providing facilities for spawning goroutines on remote hardware, and
  • Treating local and remote goroutines in the same manner, both syntactically and semantically.

As a result, distributed application code becomes orders of magnitude shorter, as compared to using traditional alternatives. For isntance, we have been able to write large real-world cloud applications — e.g. streaming multi-stage MapReduce pipelines — in as many as 200 lines of code from the ground up.

For lifecycle maintenance, the circuit provides a powerful toolkit that can introspect into, control and modify various dynamic aspects of a live circuit application. Robust networking protocols allow for complex runtime maneuvers like, for instance, surgically replacing components of running cloud applications with binaries from different versions of the source tree, without causing service interruption.

The transparent source of the circuit runtime makes it easy to instrument circuit deployments with custom logic that has full visibility of cross-runtime information flow dynamics. Out of the box the circuit comes with a set of tools for debugging and profiling in-production applications with minimal impact on uptime.