A Python-Unity interface with ZeroMQ

If you do some computer vision project or simply need to pass images or data from Python to Unity and vice versa, you have to build some rapid interface. You can use OS API, but if you need a cross-platform solution you should consider ZeroMQ sockets.

SETUP NETMQ

First, let us install a C# port of ZeroMQ — NetMQ. For that, we need NuGet for Unity, which we can get in the release section on the official GitHub repo.

The package file to download

Then we simply open the package through your file manager (Explorer, Nautilus, etc.) and import it.

The process of import

Now we have the NuGet button in Unity’s menu (sometimes we should restart Unity for it).

Type NetMQ in the search and click install.

Unfortunately, NuGetForUnity cannot install the “System.Private.ServiceModel” package, because now it cannot deal with the runtime directory of NuGet packages correctly. So we should install it manually.

Open https://www.nuget.org/packages/System.Private.ServiceModel/4.4.0 (specify the version you need). Click on “Download package”, change the file’s type from .nupkg to .zip, extract the “runtime” folder to Assets > Packages >System.Private.ServiceModel.*.*.*; remove the “unix” folder if you run Windows, and “win7” if wise versa, leave only the“netstandard2.0” directory inside.

What you will have in the case of Windows

SETUP THE UNITY SCENE

  1. Create the RawImage GameObject:

2. Set Anchor Preset to stretch both X and Y:

3. Set Left, Top, Right, etc. values that way:

Finally, you will see the Game window became white:

SIMPLE SERVER

Assume we have to pass a dictionary contains either bool, int, str, and an image as a list. We can use the send_json method to send the whole data object. To draw the image in Unity we will use RawImage’s texture method LoadImage, which loads PNG/JPG image array, but I would not advise to encode and pass .png as it serializes too slow.

SIMPLE CLIENT

The imports:

We need an additional thread to receive messages out of Unity’s event loop. Therefore we declare the Receiver class that contains a thread returning data through the callback:

Then the callback adds an Action to the runOnMainThread queue invoking this action in the Update method.

Add the component on for example MainCamera and set Image to RawImage.

Run the server Python script and the Unity scene.

Image is being sent as well as “Hello there!”

SOCKET TYPES

Note the type of socket we use in the server is zmq.REP—reply, and RequestSocket in the client. This socket type supposes two-way communication (send-receive-send-…). But it is quite ineffective and illogical. We had better use PUSH/PULL sockets.

ONE-WAY COMMUNICATION

UNCOMPRESSED IMAGES’ SENDING

Earlier, we compressed images because of slow JSON serialization but we can avoid it sending the image as an array of bytes:

— Here we change the color scheme to RGB and mirror image horizontally due to the lower-left origin of Unity.

The GitHub repo.

Senior computer science student.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store