A Python-Unity interface with ZeroMQ

If you do some computer vision project or simply need to pass images or data from Python to Unity and vice versa, you have to build some rapid interface. You can use OS API, but if you need a cross-platform solution, you should consider ZeroMQ sockets.

SETUP NETMQ

First, let us install a C# port of ZeroMQ — NetMQ. For that, we need NuGet for Unity, which we can get in the release section on the official GitHub repo.

The package file to download

Then we just open the package through your file manager (Explorer, Nautilus, etc.) and import it.

The process of import

Now we have the NuGet button in Unity’s menu (sometimes, we should restart Unity for it).

Type NetMQ in the search and click install.

SETUP THE UNITY SCENE

1. Create the RawImage GameObject:

2. Set Anchor Preset to stretch both X and Y:

3. Set Left, Top, Right, etc. values that way:

Finally, you will see the Game window became white:

SIMPLE SERVER

Assume we have to pass a dictionary containing either bool, int, str, and an image as a list. We can use the send_json method to send the whole data object. To draw the image in Unity, we will use RawImage’s texture method LoadImage, which loads PNG/JPG image array, but I would not advise encoding and passing .png as it serializes too slowly.

SIMPLE CLIENT

The imports:

We need an additional thread to receive messages out of Unity’s event loop. Therefore we declare the Receiver class that contains a thread returning data through the callback:

Then the callback adds an Action to the runOnMainThread queue invoking this action in the Update method.

Add the component on, for example, MainCamera, and set Image to RawImage.

Run the server Python script and the Unity scene.

Image is being sent as well as “Hello there!”

SOCKET TYPES

Note the type of socket we use in the server is zmq.REP—reply, and RequestSocket in the client. This socket type supposes two-way communication (send-receive-send-…). But it is quite ineffective and illogical. We had better use PUSH/PULL sockets.

ONE-WAY COMMUNICATION

SENDING UNCOMPRESSED IMAGES

Earlier, we compressed images because of slow JSON serialization, but we can avoid it sending the image as an array of bytes:

— Here, we change the color scheme to RGB and mirror the image horizontally due to the lower-left origin of Unity.

The GitHub repo.