A Python-Unity interface with ZeroMQ

If you do some computer vision project or simply need to pass images or data from Python to Unity and vice versa, you have to build some rapid interface. You can use OS API, but if you need a cross-platform solution you should consider ZeroMQ sockets.

SETUP NETMQ

The package file to download

Then we simply open the package through your file manager (Explorer, Nautilus, etc.) and import it.

The process of import

Now we have the NuGet button in Unity’s menu (sometimes we should restart Unity for it).

Type NetMQ in the search and click install.

Unfortunately, NuGetForUnity cannot install the “System.Private.ServiceModel” package, because now it cannot deal with the runtime directory of NuGet packages correctly. So we should install it manually.

Open https://www.nuget.org/packages/System.Private.ServiceModel/4.4.0 (specify the version you need). Click on “Download package”, change the file’s type from .nupkg to .zip, extract the “runtime” folder to Assets > Packages >System.Private.ServiceModel.*.*.*; remove the “unix” folder if you run Windows, and “win7” if wise versa, leave only the“netstandard2.0” directory inside.

What you will have in the case of Windows

SETUP THE UNITY SCENE

2. Set Anchor Preset to stretch both X and Y:

3. Set Left, Top, Right, etc. values that way:

Finally, you will see the Game window became white:

SIMPLE SERVER

Assume we have to pass a dictionary contains either bool, int, str, and an image as a list. We can use the send_json method to send the whole data object. To draw the image in Unity we will use RawImage’s texture method LoadImage, which loads PNG/JPG image array, but I would not advise to encode and pass .png as it serializes too slow.

SIMPLE CLIENT

We need an additional thread to receive messages out of Unity’s event loop. Therefore we declare the Receiver class that contains a thread returning data through the callback:

Then the callback adds an Action to the runOnMainThread queue invoking this action in the Update method.

Add the component on for example MainCamera and set Image to RawImage.

Run the server Python script and the Unity scene.

Image is being sent as well as “Hello there!”

SOCKET TYPES

ONE-WAY COMMUNICATION

UNCOMPRESSED IMAGES’ SENDING

— Here we change the color scheme to RGB and mirror image horizontally due to the lower-left origin of Unity.

The GitHub repo.

Senior computer science student.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store