Shopware 6: Monitoring Queue on Stage with a bitbucket pipeline, Part 2

The first part about how to find all the Shopware 6.5 instances can be found here.

This part is about how to monitor the queue

My goal is to know, when a queue on our staging server is not working properly. The idea is simple, I loop over all Shopware 6.5+ instances and collect the numbers. If any of the number is above a certain threshold I assume the queue is broken. Finding a proper threshold is a discussion for itself, I decided to use 10k. Because the last two times we had problems, we discovered it when the queue already collected 500k and 350k messages, therefore a limit of 10k seems like a low enough baseline to be alarmed.

How do we do this?

Since Shopware 6.5 there is a new CLI command: messenger:stats this returns the stats for a shopware instance and looks like this:

12:11:33 c-1234 ~/customer.test.mc.winkelwagen.de/current $ bin/console messenger:stats
 ----------- -------
  Transport   Count
 ----------- -------
  failed      3360
  async       4128
 ----------- -------

Unfortuantely is there no JSON flag or something to get an easily readable format, so we need to parse this.

    #[NoReturn] public function run(): void
    {
        $paths = $this->shopware65Finder->getShopware65Paths();
        foreach ($paths as $path) {
            $numbers = $this->collectNumbers($path);
            $this->output($path, $numbers);
            $this->checkNumbers($numbers);
            echo "\n\n";
        }
        exit($this->errorCode);
    }

So the idea:

• We get the paths as in the post yesterday
• We loop over them
  • We collect the numbers and make an array our of them
  • We output them, so we have a log of the monitoring
  • We check the numbers against our 10k threshold
• And return an error code if needed

We collect the numbers and make an array our of them

private function collectNumbers(mixed $path): array
    {
        $command = "php $path/bin/console messenger:stats -vv 2>&1";
        $stats = shell_exec($command);

        $statsAsArray = array_slice(explode("\n", $stats ?? ''), 3);
        $statsAsArray = array_slice($statsAsArray, 0, -3);

        $stats = [];

        foreach ($statsAsArray as $number) {
            $parts = array_values(array_filter(explode(' ', $number), static function ($content) {
                return $content !== '';
            }));
            $stats[$parts[0]] = $parts[1];
        }
        return $stats;
    }

I’m not sure what is interesting here for you, we run the CLI command and get the table. We remove the first three and the last three lines from it.

Then we iterate through the remaining lines which are only names of the queue with a ton of spaces around. We explode them by spaces and get an array which looks like this:

[
    0 => '',
    1 => '',
    2 => 'failed',
    3 => '',
    4 => '',
    5 => '',
    6 => '',
    7 => '',
    8 => '3360',
    9 => '',
    10 => '',
    11 => '',
];

So we filter all empty values.

But be careful, array_filter filters everything which falsifies, that means: empty string, 0, null, false, etc.

So we implement our own function to only filter empty string.

At the end we run the remaining array through array_values because we don’t know the keys of our message count, because array_filter preserves the keys and the key depends on the spaces between the name and the count and the spaces depend on the length of the queue name.

And at the end we create an array which has the name as key and the count as value.

We output them, so we have a log of the monitoring

    private function output(mixed $path, array $numbers): void
    {
        echo 'Path: ' . $path . "\n";
        foreach ($numbers as $key => $number) {
            echo $key . ': ' . $number . "\n";
        }
    }

Nothing fancy on the output, iterate over the numbers and echo them.

We check the numbers against our 10k threshold

    private function checkNumbers(array $numbers): void
    {
        foreach ($numbers as $key => $number) {
            if ($number > self::THRESHOLD) {
                $this->errorCode = 1;
                echo 'ERROR: ' . $key . ' is ' . $number . ' and should be less than ' . self::THRESHOLD . "\n";
            }
        }
    }

And again, I don’t think anything interesting here, but for the sake of completeness. We iterate again over the numbers and if the number is higher than the threshold we echo an error and set the errorCode returned to something else then the default 0

Running it regularly on our staging server

The last thing is to run the script on a regular basis and tell someone if something is odd. I didn’t want to fiddle with any deployment and stuff, therefore I just copied all the code to our staging server and created a scheduled bitbucket pipeline to run the existing php file.

image: atlassian/default-image:latest

pipelines:
  custom:
    queueMonitor:
      - step:
          name: Queue Monitor
          script:
            - |
              ssh web-user@mc.winkelwagen.de << EOF
                 php /var/www/share/bin/tools/bin/monitorQueues.php
              EOF

And if you are wondering, the monitorQueues.php looks like this:

<?php

namespace Winkelwagen\Tools\Exec;

require __DIR__ . '/../vendor/autoload.php';

use Winkelwagen\Tools\Factory;

(new Factory())->createMessengerQueueMonitor()->run();