Karl Matthias

Member of Technical Staff at Mozi. Co-Author of “Docker: Up and Running” from O’Reilly Media

Cgo: When and (Usually) When Not to Use it

Cgo is really good. If you’ve worked with C libraries in other languages, I expect you already know it’s not always much fun. But Cgo is pretty easy to use, makes a lot of the hard work easy, and can appear to make working with a C library at the code level just like working with Go code. But that appearance of ease sometimes leads people down the wrong path.

Go Gopher by Renee French.

Cgo is really good, but it’s also full of pitfalls, and when using Cgo in your project, as Rob Pike said, and Dave Cheney later wrote, you are not writing Go, even if it looks like it. Dave’s point is that you probably shouldn’t use Cgo in most cases, and that’s echoed by other smart people. I agree with that caution, if not to the same extent.

There are a number of cases where you can and maybe need to use Cgo. So if you’re in that situation, what do you do? Cgo turned out to be a huge win for us and we couldn’t have done the project without it. This post is based on my own experience with writing a production Cgo app that sees a reasonable level of traffic. The intent is not to provide exhaustive truth about Cgo, but instead, to give some guidance about things that worked for us. I’m focusing on using C code from Go. While it’s also possible to build a Go library and use it from C, that’s out of scope for this post.

What I’ve written here is based on a talk, with the same title, that I gave at the Gathering of International Gophers #2 before the Paris dotGo conference in early November 2017. People asked for my slides and some asked for a blog post. So here it is!

The Case Against Cgo

Before we talk about how you might use Cgo effectively, let’s briefly remind ourselves why this is not the first path you should choose. I’ll not repeat a lot of what has been written elsewhere. But in general, here are some problems with using Cgo in your application:

It breaks a lot of Go’s awesome tooling
Everyone loves C builds and compiles, amirite?!
Puts Go’s concurrency promise at risk
Might break your static binary, or at least really make you work for it
Breaks cross-compiling almost always
You get to manage memory in C by hand
Calls into Cgo are much slower than native Go calls

I find that two of the things I love most about Go are the awesome—near zero effort—tooling and static binaries as the build artifact. Close behind those are the concurrency model and cross-compiling for macOS, Windows, and Linux. All of those either don’t work or are made much more difficult by using Cgo in your application.

The tradeoff could be worth it: it was for us in the project where we used it. My point is just that to think hard about these things before you write them off. These are what drew many of us to Go in the first place! They are part of what makes it a language in which you can easily “get shit done.” And while you’re mulling over the cost of giving up or limiting those, remember how much time Go’s GC saves you.

When to Use Cgo

I realize that sounds like maybe the worst sales pitch in Go history. Or at least a pretty bad sales pitch. So, let’s turn to the good parts. There are some really good reasons to use Cgo. For us, at Nitro, it was a C library we wanted to use that had no real equivalent in Go. This library would have taken many mulitples of the project cost to write in Go, not to mention the time it would have consumed. Using that library saved us time and money and the end result has been great for us. So here are some reasons that might be true for you, too:

There is no Go equivalent of a library you need and can’t write in Go.
You have some legacy code that needs a web frontend.
You have to consume a proprietary library or SDK.
You have legacy business logic in C that e.g. has no functional tests and is hard or risky to rewrite or replace.

That’s not an exhaustive list. You might have some other good reason. So let’s assume now that you have a good reason. How do you use Cgo effectively?

How to Do It

We learned some lessons along they way to shipping our application. I’m not going to claim to be a Cgo expert. But these are things that have worked for us and, given that we had to find some of these out on our own, it seems like giving back to the community to put some of these out there.

Know the C

The first thing that should be said before you embark on a Cgo project is that you need to be comfortable in C! You will be reading C code. Ideally you have all the source code: having all the code will make your life a lot easier here. If it’s a proprietary library you’re still going to be pouring over header files. You need to be comfortable with that.

You will also very likely have to write some C code. Do not embark on this project thinking you will only be writing Go code. In the real world there are lots of places you need to write little bridge functions, or struct definitions, or other small workarounds.

You should understand how the linker and C includes work. You wont’t get away from them.

Love the `Makefile`

It doesn’t have to be make. Use cmake or bazel or whatever you know. But, plan on using a build system from the beginning, because you’re going to need it. Go can leverage go generate and go build but this isn’t a Go project any more! In the very simplest case you might get away without a build system, but with a project of any complexity it’s not feasible. How about C build dependencies? You may have to even build the C library itself. You need to make sure the C code is either buildable by Go, or pre-built. This generally means several steps are required.

These are the things make was designed for, and it’s still reasonably good at.

Isolate the Code

Isolating the Cgo wrappers into their own package/project worked out really well for us. If you do that, you can in turn isolate the Cgo build and the rest of the project can use Go tooling as normal. This means that unless the Cgo code changes and needs a rebuild, you will not have to do much in your consuming applications.

But keep in mind that you still might need a Makefile in that project to make sure this one is built, or to make sure that the linking to various libraries happens in the way you intend.

All of the above scenarios are simplified by putting the Cgo wrapper code into its own Go package. It will also discourage you from proliferating Cgo calls all over your Go codebase and help contain it in a consumable form.

Use Statically Linked C Libraries

This is not always possible. But when it is possible you can preserve the Go output as a static binary by compiling only against static C libraries, much as you would if you were generating a static binary in C. I found that you have to work pretty hard with the build flags to get Linux to do this because it really tries hard to use shared libraries. Sometimes you need to modify the build flags of the upstream libraries several layers deep.

It’s a pain but it’s worth it. Getting this right means you don’t lose the ease of distribution that you normally get with Go binaries. At least with one platform… you still have major complications with cross-compiling. But in most of the scenarios we talked about, you’ve already agreed to forego cross-compilation.

Write C Bridge Functions

There were lots of places where we needed to write little functions to glue the libraries we were consuming to the Go application. This is necessary for at least a few reasons:

Go cannot call C macros! This is a big one. It’s surprising how many C libraries rely on macros for a large portion of their public API. You can’t call them from Go, so you write C functions to call the macros you need, and then you access those from Go. This can be tricky. If your library is heavily macro-laden, think hard before proceeding.
You don’t want to call in to C from Go many times in a row when possible. If it makes sense in your scenario, you can sometimes get a big performance win by calling several C functions in one Cgo function call. A good place to do this might be looping over a C function, for example. Implementing the loop in C might save a bundle of overhead.
C programmers often do typecasting that Go’s stronger type system just won’t let you do (thankfully!). But when you’re using someone else’s C library, you might be expected to do that. Writing a wrapper in C to do the typecasting can really get you out of a jam.

Write Test Wrappers

You can’t call Cgo directly from Go test code! That means if you have things you need to test that are in C, you need to write some wrapper functions in your Go package that allow you to call them. This is pretty trivial but took a little reworking to solve once we knew this was a problem. Knowing this ahead of time, plan at exposing whatever C functionality you need to have in tests via Go wrappers.

Let Go Help with Memory Management

One of the worst (and, yes, most powerful) “features” in C is that you must do all of your own memory management. Some libraries have their own memory management schemes and you should leverage those if you are calling that code. But in places where you are responsible for managing memory on your own, you can use a few strategies to help you:

Allocate memory from Go when possible and pass it into C. Note that this cannot contain references to other Go memory. Thus, this is best used for memory buffers or other places you’d use a byte slice. (see the Cgo Godoc
Leverage defer to make sure you clean up temporary C memory by the end of your Go function. Be aware of the lifecycle of the C memory, though.
Try very, very hard not to pass C memory outside of a single Go function. If you need to do that, a good solution for us was to assign these to fields in a struct on which your Go functions are then operating. That lets you allocate the C memory for the life of the struct without losing the reference. But you still have to clean it up on your own!

Think About Concurrency

Go makes concurrency comparatively easy vs C. Your C library may be managing concurrency on its own. You can’t pass references to shared/synchronized C structures into Goroutines without introducing big problems. There is not a simple answer here because so much depends on the library in question. Sometimes your library has thread-local storage in C, or the library tracks some resource by thread ID, and you don’t know which thread Go will schedule your goroutine on. It’s possible to pin a goroutine to a single kernel thread if necessary (runtime.LockOSThread()). That’s not ideal and may not be the right solution. The general suggestion here is to just plan how you will handle concurrency between Go and C before you dive in. Think through the issues.

Wrap Up

There is undoubtedly more. But this should give you a pretty good head start. Cgo is powerful and capable, and deceivingly easy to use from a code level. In the right circumstances it could be a lifesaver for your project. Ours worked out well and has been in production for some months now without memory leaks, thread leaks, deadlocks, or other major issues. Cgo was instrumental in making it a success and has been pretty easy to manage. But it took a lot of work to get us there. I’m hoping the lessons we learned along the way may help others make the most of Cgo’s rather awesome features.

But remember the caveats! Great power does not come wihout a cost.