Ufdjrw

In a current project I have to perform the following tasks (among others):

• capture video frames from five IP cameras and stitch a panorama


• run machine learning based object detection on the panorama


• stream the panorama so it can be displayed in a UI

Currently, the stitching and the streaming runs in one docker container, and the object detection runs in another, reading the panorama stream as input.

Since I need to increase the input resolution for the the object detector while maintaining the stream resolution for the UI, I have to look for alternative ways of getting the stitched (full resolution) panorama (~10 MB per frame) from the stitcher container to the detector container.

My thoughts regarding potential solutions:

• shared volume. Potential downside: One extra write and read per frame might be too slow?


• Using a message queue or e.g. redis. Potential downside: yet another component in the architecture.


• merging the two containers. Potential downside(s): Not only does it not feel right, but the two containers have completely different base images and dependencies. Plus I'd have to worry about parallelization.

Since I'm not the sharpest knife in the docker drawer, what I'm asking for are tips, experiences and best practices regarding fast data exchange between docker containers.

Usually most communication between Docker containers is over network sockets. This is fine when you're talking to something like a relational database or an HTTP server. It sounds like your application is a little more about sharing files, though, and that's something Docker is a little less good at.

If you only want one copy of each component, or are still actively developing the pipeline: I'd probably not use Docker for this. Since each container has an isolated filesystem and its own user ID space, sharing files can be unexpectedly tricky (every container must agree on numeric user IDs). But if you just run everything on the host, as the same user, pointing at the same directory, this isn't a problem.

If you're trying to scale this in production: I'd add some sort of shared filesystem and a message queueing system like RabbitMQ. For local work this could be a Docker named volume or bind-mounted host directory; cloud storage like Amazon S3 will work fine too. The setup is like this:

• Each component knows about the shared storage and connects to RabbitMQ, but is unaware of the other components.


• Each component reads a message from a RabbitMQ queue that names a file to process.


• The component reads the file and does its work.


• When it finishes, the component writes the result file back to the shared storage, and writes its location to a RabbitMQ exchange.

In this setup each component is totally stateless. If you discover that, for example, the machine-learning component of this is slowest, you can run duplicate copies of it. If something breaks, RabbitMQ will remember that a given message hasn't been fully processed (acknowledged); and again because of the isolation you can run that specific component locally to reproduce and fix the issue.

This model also translates well to larger-scale Docker-based cluster-computing systems like Kubernetes.

Running this locally, I would absolutely keep separate concerns in separate containers (especially if individual image-processing and ML tasks are expensive). The setup I propose needs both a message queue (to keep track of the work) and a shared filesystem (because message queues tend to not be optimized for 10+ MB individual messages). You get a choice between Docker named volumes and host bind-mounts as readily available shared storage. Bind mounts are easier to inspect and administer, but on some platforms are legendarily slow. Named volumes I think are reasonably fast, but you can only access them from Docker containers, which means needing to launch more containers to do basic things like backup and pruning.

Alright, Let's unpack this:

• IMHO Shared Volume works just fine, but gets way too messy over time. Especially if you're handling Stateful services.


• MQ: This seems like a best option in my opinion. Yes, it's another component in your architecture, but it makes sense to have it rather than maintaining messy shared Volumes or handling massive container images (if you manage to combine 2 container images)


• Yes, You could potentially do this, but not a good idea. Considering your use case, I'm going to go ahead and make an assumption that you have a massive list of dependencies which could potentially lead to a conflict. Also, lot of dependencies = larger image = Larger attack surface - which from a security perspective is not a good thing.

If you really want to run multiple processes in one container, it's possible. There are multiple ways to achieve that, however I prefer supervisord.

https://docs.docker.com/config/containers/multi-service_container/