Embedding Python & GoLang Codes Each Other

In old days, Prior to beginning any project, most development teams go through fevered discussions to decide the best language for their software. Nowadays, most of the time, this discussion boils down to Python and Golang. But software developers today know actually, there is no the best language, there is the best proper language for our project. On the other hand, projects today are so vast and advanced, so deciding on a language is not easy and right. This is why microservice architectures were developed. Separated small models can be developed easily with proper different languages and technologies. Approaching of Microservice has different challenges. But if you have a gigantic project, there will be different extra problems except for selecting language or technology too.

Anyway, I don’t mean to beat a dead horse, but I wanted to mention talking about different approaches and no superiority on their each other totally. If you adopted a monolith approach and decided to continue with Golang, you will surely need it one day. That is Python! Or vice versa, you should use the abilities of Golang.

Embedding Python in GoLang

Introducing CGO

Cgo enables the creation of Go packages that call C code. Cgo is not a language nor a compiler. It’s a Foreign Function Interface (FFI), a mechanism we can use in Go to invoke functions and services written in a different language, specifically C. Here is a simple example using:

Everything in the comments above the import “C” is C code and will be compiled with the GCC. Make sure you have a GCC installed. You need to include “limits.h” to print the value of the “INT_MAX” constant and we write a “sum” function in the comments.

Embedding CPython

We can access the CPython interpreter by CGO library in Golang. Because Python supports C API to can embed “C” codes like Golang. So a Go program that, technically speaking, embeds CPython is not as complicated as you might expect.

CPython; is simply a programming language that allows us to write C extensions for Python. It does this by enabling us to specify a type (static type) on C functions and variables with Python syntax, and by running the C code it generates as a Python extension, we get C-speed functions that we can use in Python.

There is massive document for Python C API below. You can change the version.

https://docs.python.org/3.7/c-api/index.html

Jython, IronPython and PyPy are the current “other” implementations of the Python programming language; these are implemented in Java, C# and RPython (a subset of Python), respectively. Jython compiles your Python code to Java bytecode, so your Python code can run on the JVM.

You can see we put a “#cgo” directive in the header comments. Those directives are passed to the toolchain to let you change the build workflow. In this case, we tell CGO to invoke “pkg-config” to gather the flags needed to build and link against a library called “python-3.7” and pass those flags to the C compiler.

You have to install Python-3.7 and pkg-config development libraries.

Dependencies

Installation on Debian for Python3

# apt-get install pkg-config python3-dev

Installation on Centos for Python3

# yum install pkgconfig python3-devel

If you want to use Python2, install python2.7 development libraries and change the pkg-config invocation like below.

// #cgo pkg-config: python-2.7

You can write codes with “cgo” command sets directly, but a library would be nice to hide the “cgo” details. Go friendly thin wrapper around the C API and get rid of CGO details.

If you are using Python2, you can use the GO library “github.com/sbinet/go-python”.

package main

// #cgo pkg-config: python-2.7
import (
    python "github.com/sbinet/go-python"
)

func main() {
    python.Initialize()
    python.PyRun_SimpleString("print 'hello, world!'")
    python.Finalize()
}

But If you want to use Python3, You should use that one: “github.com/DataDog/go-python3”

package main
import "github.com/DataDog/go-python3"

func main() {
   defer python3.Py_Finalize()
   python3.Py_Initialize()
   python3.PyRun_SimpleString("print('hello world')")
}

The library developed by DataDog only supports Python 3.7 for now, but there are many forks that support new Python versions

Embedding GoLang in Python

Build Go package as C shared library or object

Every Go the main package can be built as a C shared library. Any C shared library can be distributed, installed, and linked to every C application and applications in other programming languages that can reference C shared libraries.

go build -o main.so -buildmode=c-shared main.go

After executing the command above on the GolangCode below, you will get a package as C shared library. Then you can use this module (“main.so”) in your Python scripts.

Now we can use “main.so” shared object in our Python example code. The “ctypes” foreign function library is used to call the exported Go functions as shown in the following codes.

The lib variable represents the loaded symbols from the shared object file. Python class “GoString” map to their respective C struct types. Because actually “String” is a special type. It is a sequence of characters and an immutable object.

Output:

Add(15,17) = 32
Title = Our Title Is Nice

Some Notes About Shared Objects

“.so” file extension is a standard for shared library objects. It is not compulsory but it is conventional in Linux/Unix Systems. It is “.dll” is the same meaning in Windows Systems.
There is a global environment named “LD_LIBRARY_PATH” in Linux System for searching shared objects. Our code did not raise errors because of setting the exact location.

lib = cdll.LoadLibrary(“./main.so”)

If you had used “main.so” instead of “./main.so”, you would got an error about not finding the file, but you can set or add a new path to your LD_LIBRARY_PATH environment.

export LD_LIBRARY_PATH=.

or

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/batur/lib

What was the Goal?

The aim of this document is giving a simple approach about embedding GoLang and Python codes each other and show it is not so hard what you expected. I love writing codes with both :)