If you do some computer vision project or simply need to pass images or data from Python to Unity and vice versa, you have to build some rapid interface. You can use OS API, but if you need a cross-platform solution you should consider ZeroMQ sockets.
First, let us install a C# port of ZeroMQ — NetMQ. For that, we need NuGet for Unity, which we can get in the release section on the official GitHub repo.
Then we simply open the package through your file manager (Explorer, Nautilus, etc.) and import it.
Now we have the NuGet button in Unity’s menu (sometimes we should restart Unity for it).
Type NetMQ in the search and click install.
Unfortunately, NuGetForUnity cannot install the “System.Private.ServiceModel” package, because now it cannot deal with the runtime directory of NuGet packages correctly. So we should install it manually.
Open https://www.nuget.org/packages/System.Private.ServiceModel/4.4.0 (specify the version you need). Click on “Download package”, change the file’s type from .nupkg to .zip, extract the “runtime” folder to Assets > Packages >System.Private.ServiceModel.*.*.*; remove the “unix” folder if you run Windows, and “win7” if wise versa, leave only the“netstandard2.0” directory inside.
SETUP THE UNITY SCENE
- Create the RawImage GameObject:
2. Set Anchor Preset to stretch both X and Y:
3. Set Left, Top, Right, etc. values that way:
Finally, you will see the Game window became white:
Assume we have to pass a dictionary contains either bool, int, str, and an image as a list. We can use the send_json method to send the whole data object. To draw the image in Unity we will use RawImage’s texture method LoadImage, which loads PNG/JPG image array, but I would not advise to encode and pass .png as it serializes too slow.
We need an additional thread to receive messages out of Unity’s event loop. Therefore we declare the Receiver class that contains a thread returning data through the callback:
Then the callback adds an Action to the runOnMainThread queue invoking this action in the Update method.
Add the component on for example MainCamera and set Image to RawImage.
Run the server Python script and the Unity scene.
Note the type of socket we use in the server is zmq.REP—reply, and RequestSocket in the client. This socket type supposes two-way communication (send-receive-send-…). But it is quite ineffective and illogical. We had better use PUSH/PULL sockets.
UNCOMPRESSED IMAGES’ SENDING
Earlier, we compressed images because of slow JSON serialization but we can avoid it sending the image as an array of bytes:
— Here we change the color scheme to RGB and mirror image horizontally due to the lower-left origin of Unity.
The GitHub repo.