Vehicle Incident RabbitMQ Consumer

The Client

SA Taxi is a South African-based transport firm, with a unique insight into the safety of their clients.

The Concept

Having thousands of messages in the air at one time can be a difficult thing to manage, even more so when those messages could be life or death. This RabbitMQ consumer is the bridge between the critical incidents happening for a transport firm and its resolution team. The idea behind the consumer was that when an incident occurred (accident, blown tire, everything being on fire ext.) then it would report the incident and as much information with it as possible. For example, if there was an accident then the IOT sensors in the vehicle would send out a message with coordinates of where the vehicle is, the time of the incident, and even how far along the route it was.

Now, this isn’t the only time that we need to get messages from vehicles, its whereabouts would also be needed at minimum semi-regular intervals. This is where the Cartrack and NetStar trackers came in. They would send up to date information of where the vehicle was, then we could see how far along its route it is, calculate the speed and change routes depending on traffic. One problem the Cartrack and NetStar messages have a completely different structure.

The Approach

So because of the number of messages that need to be dealt with concurrently and that these are critical events that cannot be missed, the choice of the queue was RabbitMQ. This is because of the need for a guarantee on a per-message basis. It also meant that if the need arose, the queue could be scaled to match the load.

The consumer was the next part of this process, we needed something that could handle high concurrency without breaking the bank. Because of this, Golang was used to create the consumer because of its remarkable routine structure. 

Architecturally the system needed to be able to do 2 things, send critical incidents into PubSub topic while non-critical tracking systems would need to be mapped into a pre-set structure and sent to PubSub topic B. Now with the critical incidents, we wanted the fastest throughput because time is valuable in that instance. So no mapping would need to be done at all, just catch and push. But there would need to be quite a bit needed to get the Cartrack and Netstar to have the same structure so that they could be sent to the same topic, they had very similar information but that information was manifest differently. 

So we needed something that could point to multiple RabbitMQ instances that could be different versions, so I made it so there was a “Consumer” which was basically the entire pipeline from connecting to the RabbitMQ instance to mapping. This meant that there could easily be multiple “Consumers” all pointing to different instances of RabbitMQ, it also means that each individual consumer can have its own mapping depending on the config that it’s given.

Finally, once the consuming and mapping is done then it’s time to send the message through to the correct topic. For this build, we were only using Google PubSub so the simplest thing to do is to make a “Publisher” interface that could be extended later if more publishers are required. It also means that the “Publisher” can hold 2 different configs depending on which consumer it’s connected to. 

This project was a fun one to do, from the requirements gathering to planning and implementing the sprints. Everything that happened on this project I did.