Failed/aborted reloads

• 1: Alert emails
• 2: Alerts via Slack and Microsoft Teams
• 3: Storing script logs of failed reloads to disk
• 4: InfluxDB
• 5: New Relic
• 6: MQTT
• 7: Signl4
• 8: Webhooks

1 - Alert emails

Scheduled vs manual app reloads

It might not be obvious at first, but there are several kinds of reloads in Qlik Sense Enterprise on Windows:

1. Reloads started from QMC. These are usually created and managed in the QMC. Quite often they are also combined into reload chains. The common thing about these reloads is that they - under the hood - are managed by Sense’s scheduling service.
2. Manual reloads started from the script editor. When developing apps in the standard Sense client/script editor you usually reload the apps from there. This does trigger an app reload, but not via the Sense scheduling service. Instead the reload is done directly in the engine service.

The reload failure notifications described here work by looking at log entries written by the scheduling service. When that service writes information to the logs about a failed reload, your logging appender will detect it and send a UDP message to Butler - who will forward the message to all the notification destinations configured in the config file.

It’s also possible to have the log appender send emails without using Butler. It works perfectly fine, but the emails will be very basic when it comes to formatting and you will not get any of the features offered by Butler (last few lines of the reload script log included in the email, customizable email subjects etc).

Alert emails

Butler can send two kinds of alert emails:

• When a scheduled reload task fails.
• When a running reload task is stopped.

Alert emails can be formatted using HTML, use CSS styling, emojis etc.
There’s no reason an alert email can’t look good!

Alert emails viewed on a mobile phone give direct insight into what has happened:

In a regular email client a reload failed email could look like below.

Note the end of the script - the last few lines of the reload log are often very useful when it comes to understanding what caused the reload failure.

Basic alert emails also possible

Qlik Sense Enterprise on Windows uses the log4net logging framework to create log files. Log4net is quite flexible and can - among other things - send emails when events such as reload failures occur. There is however little flexibility when it comes to layout and contents of those emails. They are text only (no formatting, tables, different fonts, colors etc) and the email subjects cannot contain any dynamic fields (for example the name of the failed reload task).

The goal of Butler’s alert emails is to address these limitations and offer a flexible foundation not only for emails, but for all kinds of alerts.

If you want to explore what’s possible using just the features offered by log4net, Christof Schwarz has a good post on sending basic notification emails when scheduled reloads fail, with links to Levi Turner’s great examples.

Alert emails to app owners

Qlik Sense can be configured in many ways. In some companies all apps are owned by a central service account.
Other companies set the developer as app owner also for published apps.

In the latter case it might be relevant to send the app owner a notification email when a reload task fails or is aborted. That way the developer is immediately made aware of the issue and can act on it as needed.

This feature assumes the app owner’s user account (in the Sense user directory) has an email address associated with it. When syncing users from Active Directory the users’ emails are often brought along into Sense, but there is no guarantee for this.

If an email address is available for a Sense user, the QMC user section can look like this:

Alert emails only for some tasks

Sometimes there is a desire to only have email alerts for some tasks.
One example can be a Sense cluster that hosts both development and production apps, maybe separated on different servers.

As of Butler 7.4.0 it is possible to control per task if an alert email should be sent when the task fails or is aborted from the QMC.

Conceptually it works like this:

Instructions for how to configure this feature is available here.

Note: This feature is similar to - but independent from - the “task specific email recipients” feature below. Either feature can be enabled or disabled independently of the other in Butler’s config file.

Task specific email recipients

They may be cases where all alert emails should normally go to for example a Sense administrator, but some alerts should instead (or also) go to some other recipients.

An example could be a sales related Sense app. If it fails reloading the standard alert email should go to the Sense administrator, but there should also be an alert email sent to the sales operations team, to notify them that they won’t find updated numbers in the Sales app.

Butler handles this scenario by using a custome propperty (its name is configurable in the Butler config file) to set alert email recipients on a per-task basis.

Conceptually it works like this:

Instructions for how to configure this feature is available here.

Note: This feature is similar to - but independent from - the “alert emails only for some tasks” feature below. Either feature can be enabled or disabled independently of the other in Butler’s config file.

How it works

Butler uses a templating engine called Handlebars. It is used when sending all kinds of alert emails supported by Butler.

Slack, MS Teams and MQTT messages are currently not using the templating engine - this is however likely to change in coming Butler versions. Feel free to add (or +1) a request on GitHub if this is of interest to you!

Template fields

The Handlebars templating engine looks for template fields in the template files you create.

A complete list of template fields - including descriptions - is available in the Reference section.

Not all failed reloads will cause alert emails

While not obvious at first, there are different kinds of reloads taking place in a Qlik Sense Enterprise environment:

• Reloads started by the Sense Scheduler service. These reloads always have a task associated with them.

• Reloads started from Sense’s standard script editor. These reloads are not started by the Sense scheduler, but rather directly in the Sense engine. Progress for such reloads will therefore go to the engine logs.

The log appenders that drive Butler’s alerts rely on the Scheduler logs - not the engine logs.
This is an intentional design decision.

It is certainly possible to add log appenders also for engine logs and that way get notified when any reload fail. The question is whether that’s an interesting use case. In most cases sys admins aren’t very interested in reloads that fail during app development - they only care about failures caused by apps in production - i.e. app reload tasks managed by the Sense Scheduler. Thus, Butler currently doesn’t deal with reload failures reported from the Sense engine.

References

• Qlik’s documenation around log appenders and how to hook into the Sense logs is somewhat brief, but does provide a starting point if you want to dive deeper into this topic.

• The main log4net documentation (log4net is the logging framework used by Qlik Sense Enterprise) can also be useful.

These links describe how emails can be sent from the log4net logging framework itself, directly to the recipient. Butler includes sameple XML files for this use case too, but Butler takes things further by using the data in the Sense logs to pull in more data around the failed or stopped reload.

In other words - Butler’s alert emails are significantly more flexible and contain information (such as script logs) that are not availble using purely log4net.

Seeing is believing

The video below is available at Ptarmigan Labs’ YouTube channel and also in the Butler playlist.

2 - Alerts via Slack and Microsoft Teams

Teams, Slack and email notifications

Microsoft Teams, Slack and email are all notification destinations.

Alert messages/notifications come in two variants: “basic” and “formatted”.

Formatted messages

These messages take full advantage of the formatting available in each notification destination.

Slack has its own way for creating messages with rich layout and formatting - as does Teams and email.

Formatted messages are created using template files.
Each notification destination has its own set of template files. It’s therefore possible to take advantage of each destination’s specific features when it comes to formatting the messages sent by Butler.

Message templates can include “template fields”. These are placeholders that are replaced with actual event values before the message is sent.

The GitHub repository includes fully functional template files for all destinations.

Basic messages

Basic message formats are available for all notification destinations.

This message type is useful if you only want a short, basic notification that a task failed or was aborted. The basic formats are pre-defined and are thus easy to set up.

Microsoft Teams notifications

Basic and formatted reload task failure notifications can look like this in Teams:

The configuration needed for setting this up is described here.

Slack notifications

Basic and formatted reload task failure notifications can look like this in Teams:

The configuration needed for setting this up is described here.

Seeing is believing

The video below is available at Ptarmigan Labs’ YouTube channel and also in the Butler playlist.

3 - Storing script logs of failed reloads to disk

Reload script logs

When doing a scheduled reload or a reload started from the QMC, Sense will create a detailed log file that includes all the commands executed during the reload.

If a reload for reason fails it can be very useful to look at these reload logs.

The latest reload log file for each reload task is available via the QMC, but logs for previous reload attempts are not available via the QMC.

Using the same mechanism used by reload failure alerts in general, Butler can be configured to store the reload logs of all failed reloads to disk.

The reload logs are stored in the directory configured in the Butler config file, with separate directories for each date:

.
├── butler.exe
├── log
│   └── butler.2022-04-07.log
├── production.yaml
└── scriptlog
    ├── 2022-04-06
    │   ├── 2022-04-06_15-36-12_appId=deba4bcf-47e4-472e-97b2-4fe8d6498e11_taskId=0d815a99-1ca3-4131-a398-6878bd735fd8.log
    │   └── 2022-04-06_22-42-35_appId=66bc109d-286a-415b-8355-1422abb22133_taskId=e959f40a-67be-4a5b-ae83-a292f96ba078.log
    └── 2022-04-07
        └── 2022-04-07_05-49-16_appId=deba4bcf-47e4-472e-97b2-4fe8d6498e11_taskId=0d815a99-1ca3-4131-a398-6878bd735fd8.log

All in all this makes it a lot easier to find log files for failed reloads.

Configuration of this feature is described here.

4 - InfluxDB

Visualising failed reloads in Grafana

When a reload fails, Butler can send information about the failed reload to InfluxDB.
The data stored in InfluxDB is described here.

Once the data is in InfluxDB it can be visualized in Grafana or similar tools.
Grafana has a good log viewer that can be used to visualize the data.

Note how even the script log is stored in InfluxDB, so you can see the last few lines of the reload script log in Grafana.
This makes it easy to right away see what went wrong, especially when dealing with reloads that happened a while back.

Configuration

Configuration of this feature is described here.

5 - New Relic

View reload alerts in New Relic

When investigating reload failures it can often be useful to have access to the entire reload log, as this usually tells immediately what went wrong.
Butler forwards a very comprehensive set of data to New Relic when a reload fails, including the failing part of the app’s script.

This means that companies using New Relic as their enterprise monitoring solution can now also use it to monitor their Qlik Sense reloads, as well as all the Sense real-time metrics provided by Butler’s sibling tool Butler SOS.

As of this wrtiting, New Relic also offers a free tier that will be more than enough for most Butler users.
It is thus possible to get started with monitoring Sense reloads in New Relic without any additional cost.

More information about how Butler integrates with New Relic can be found here.

Configuration

Configuration of this feature is described here.

6 - MQTT

MQTT as unified message bus

When a reload fails, Butler can send information about the failed reload to an MQTT broker.

MQTT is a lightweight messaging protocol that is commonly used in IoT applications, but it is a mature and versatile protocol that can be used in many different scenarios.

In short, MQTT works by having a broker that clients can connect to. Clients can publish messages to the broker, and clients can subscribe to messages from the broker.
This makes MQTT a great way to integrate different systems in a publish/subscribe pattern.

By sending information about failed reloads to an MQTT broker, Butler can be integrated with any system that can consume MQTT messages - which is a lot of systems.

The information included in the MQTT message is described here.

Here is an example of how the information about a failed reload can be viewed in MQTT Explorer:

Configuration

Configuration of this feature is described here.

7 - Signl4

Mobile alerting

Signl4 has a great mobile app that can be used to receive reload-failed alerts from Butler.

It can look something like this:

More information about how Butler integrates with Signl4 can be found here.

Configuration

Configuration of this feature is described here.

8 - Webhooks

When nothing else works

Webhooks may be somewhat limited in terms of what they can do, but their simplicity is also their strength.

When you need to send information about failed/aborted reloads to a system that doesn’t have a dedicated integration with Butler, webhooks may be a good solution.

Butler offers a lot of flexibility in terms of how each webhook is configured.
You can…

• specify the URL to send the webhook to.
• specify the HTTP method to use (POST, PUT, GET).
• use htto or https.
• specify if a custom certificate should be used. In this case the root CA certificate is provided to Butler per webhook, which means Butler can be integrated to many systems, each using their own self-signed certificate.
• specify if an untrusted certificate should be accepted. This is useful when integrating with systems that use self-signed certificates and you don’t have access to the root CA certificate.

Sending reload-failed informnation to a GET http webhook provided by Node-RED can look like this in the Node-RED editor:

Configuration

Configuration of this feature is described here.