Urgency and Impact
For this particular queue, rules are such that we allow a certain amount of messages to be there at any given time. A size of the queue may significantly increase in a short period of time due to the various reasons, mostly related to Firebase servers being down due to maintenance or some updates. Short living spikes can be ignored with no consequences, but if the queue size remains high for a longer period of time, it may start to affect the delivery of the messages, their order, etc.
There are following reasons why the queue may build up within a longer period of time:
- servers experience lack of free resources
- server is overloaded with the number of requests. load balancing is required
- Redis is under a heavy load and it starts to slow down the process of fetching the messages from the queue
Alerts
We don’t have any alerts set up for this queue as for now. We will observe data and come up with the alerting strategy if needed later.
Data sources
GET https://statistics.production.almightypush.com/ui/#!/default/controllers_stock_statistics_senders_monitoring_list_handler
The output always represents the current state of messages in the queue for different firebase accounts.
Data is being collected by senders-monitoring running on all senders locations. Their output (senders-queue-monitoring) is being sent to the master location where it is collected by statistics-listener