Concepts

• 1: Key-value store
• 2: Scheduler
• 3: Failed/aborted reloads
• 3.1: Alert emails
• 3.2: Alerts via Slack and Microsoft Teams
• 3.3: Storing script logs of failed reloads to disk
• 3.4: InfluxDB
• 3.5: New Relic
• 3.6: MQTT
• 3.7: Signl4
• 3.8: Webhooks
• 4: Successful reloads
• 5: Starting Sense tasks in style
• 6: Incident management tools
• 6.1: New Relic
• 6.2: Signl4
• 7: Qlik Sense licenses
• 8: File system access: copy/move/delete files
• 9: MQTT integration
• 10: UDP client
• 11: Monitor Windows services
• 12: Real-time metrics

1 - Key-value store

Storing key-value pairs in Butler

The key-value (=KV) feature in Butler is a basic variant of the more complex KV databases available out there, with etcd, Apache Ignite and memcached being popular open source options. All the major cloud providers also have their own KV database products.

Butler’s KV features are basic, yet very useful in the context of Qlik Sense. The only assumptions are:

• You have a value you want to store for some time outside of your Sense app.
• There is a unique key for each value you want to store.

Put differently: Think of Butler’s KV store as a way to stash away some values, then get them back later when they are needed again.

Each KV pair is associated with a namespace. Namespaces are simply a way to categorize KV pairs.

There is also an optional, per KV-pair Time To Live (TTL) feature. If used it will auto-delete the KV pair when a certain time has passed from the KV-pair’s last update.

The API docs shows what methods are available to work with KV pairs.

How can a key-value store be used in Sense apps?

As mentioned above - A KV store can be useful whenever you need to stash data away for a while and then get it back. I.e. keeping track of the state of something.

For example

• Easily pass parameters beteen apps in a reload chain Let’s assume data is created when appA reloads as part of an hourly reload schedule. That data is needed in appB, which is triggered to reload when appA finishes its reload. But how do you get the data from appA to appB?

  Historically you solve this by writing the data to a temporary QVD or CSV file. This still works of course, but if it’s only some dimensional value that needs to be passed, a KV store might be a cleaner option.

• Keep a time limited state The TTL feature is useful to keep things tidy. If you know for sure that your KV pair only needs to be stored for a limited time, it’s good practice to either delete it when its no longer needed, or set a TTL when the KV pair is first created.

  This way you keep the KV namespaces relevant and reasonable in size.

• Use app IDs as namespace names If you need to keep state between successive reloads of a particular app, you can use the app ID as namespace. That way it will be very clear which a specific KV pair belongs to.

• Keep track of what users should be notified after an app reload is complete Let’s say you have a button in an app that when clicked kicks of a reload of the app (or some other app). Let’s also assume several users might be interested in triggering a refresh of this dataset.

  By pushing each user’s username to a KV namespace when they request the data refresh (by clicking that button in the app), it’s possible to notify them using Teams, Slack, email etc as the last step of the app’s reload script (i.e. when the app ist just about done refreshing the data).

  The effect is a solution where users can request a data refresh and be notified when the new data is available.

• Keeping state in visualisation extensions Extensions are built using Javascript, and they can thus also make use of the KV store.

  There might be times when several extension instances in an app need to keep in sync or share some state - a KV store might be useful there.
  The KV store could even allow an extension to share state with its siblings in other Sense apps.

Persistence of key-value data

As of current Butler version (v4.0), KV pairs are not persisted to disk.
Is this good or bad? It depends:

• Good as it reduces complexity in Butler.
• Bad as all KV pairs are lost when Butler is restarted. Now, Butler tends to be very stable, so spontaneous restarts are usually not a problem. But the flushing of all KV data is still something to consider when for example upgrading Butler to new versions.

2 - Scheduler

What is a scheduler?

In the context of Qlik Sense, a scheduler is a tool that triggers Qlik Sense tasks at some specific time or interval.
Qlik Sense Enterprise has its own, built-in scheduler that can be accessed via the QMC.

The QMC interface to Sense’s standard scheduler lets you create schedules for two kinds of tasks:

• App reload tasks
• User directory sync tasks

What’s wrong with Sense’s own scheduler?

The built-in scheduler in Qlik Sense is ok in most aspects, but lack significantly in some.
Specifically, it doesn’t allow you to run a task certain hours of the day. At least not without resorting to creating lots of task triggers - which is not an attractive option from a maintenance perspective.

This is a quite common scenario and thus Butler gets its own scheduler to solve the issue.

The Butler scheduler

Butler’s scheduler is based on cron.
Cron has been the standard scheduler for decades in most Linux systems, it’s thus a proven concept.

Features of the Butler scheduler:

• 6 position cron pattern. The leftmost position represents seconds.
• 5 postition patterns are also supported, the leftmost position then represents minutes.
• Hundreds of schedules tested and confirmed working as expected.
• A Qlik Sense task ID is associated with each schedule. When the schedule fires, the associated task is started.
• Schedules can be added either manually in the YAML schedules file (as defined in the main Butler config file) or using Butler’s API. A sample schedule file is included in the GitHub repository. Schedules added using the API will be stored in the schedule YAML file referenced in the main Butler config file.

The two supported schedule formats look like this:

    ┌───────────── seconds (0 - 59)
    │ ┌───────────── minute (0 - 59)
    │ │ ┌───────────── hour (0 - 23)
    │ │ │ ┌───────────── day of the month (1 - 31)
    │ │ │ │ ┌───────────── month (0 - 11)
    │ │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday)
    │ │ │ │ │ │
    * * * * * *

Valid cron patterns are:

    Asterisk. E.g. *
    Ranges. E.g. 1-3,5
    Steps. E.g. */2

There are quite a few online tools for creating cron patterns, for example crontab.guru and crontab-generator.

Using a tool like these can save some time when setting things up, but keep in mind that most online tools use minutes as the smallest unit. Some manual tweaking might thus be needed before using the generated pattern in Butler’s scheduler.

3 - Failed/aborted reloads

Below follows a list of destinations to which Butler can send notifications when a reload task fails or is aborted.

A comparison of the different alert destinations can be found here.

3.1 - Alert emails

Scheduled vs manual app reloads

It might not be obvious at first, but there are several kinds of reloads in Qlik Sense Enterprise on Windows:

1. Reloads started from QMC. These are usually created and managed in the QMC. Quite often they are also combined into reload chains. The common thing about these reloads is that they - under the hood - are managed by Sense’s scheduling service.
2. Manual reloads started from the script editor. When developing apps in the standard Sense client/script editor you usually reload the apps from there. This does trigger an app reload, but not via the Sense scheduling service. Instead the reload is done directly in the engine service.

The reload failure notifications described here work by looking at log entries written by the scheduling service. When that service writes information to the logs about a failed reload, your logging appender will detect it and send a UDP message to Butler - who will forward the message to all the notification destinations configured in the config file.

It’s also possible to have the log appender send emails without using Butler. It works perfectly fine, but the emails will be very basic when it comes to formatting and you will not get any of the features offered by Butler (last few lines of the reload script log included in the email, customizable email subjects etc).

Alert emails

Butler can send two kinds of alert emails:

• When a scheduled reload task fails.
• When a running reload task is stopped.

Alert emails can be formatted using HTML, use CSS styling, emojis etc.
There’s no reason an alert email can’t look good!

Alert emails viewed on a mobile phone give direct insight into what has happened:

In a regular email client a reload failed email could look like below.

Note the end of the script - the last few lines of the reload log are often very useful when it comes to understanding what caused the reload failure.

Basic alert emails also possible

Qlik Sense Enterprise on Windows uses the log4net logging framework to create log files. Log4net is quite flexible and can - among other things - send emails when events such as reload failures occur. There is however little flexibility when it comes to layout and contents of those emails. They are text only (no formatting, tables, different fonts, colors etc) and the email subjects cannot contain any dynamic fields (for example the name of the failed reload task).

The goal of Butler’s alert emails is to address these limitations and offer a flexible foundation not only for emails, but for all kinds of alerts.

If you want to explore what’s possible using just the features offered by log4net, Christof Schwarz has a good post on sending basic notification emails when scheduled reloads fail, with links to Levi Turner’s great examples.

Alert emails to app owners

Qlik Sense can be configured in many ways. In some companies all apps are owned by a central service account.
Other companies set the developer as app owner also for published apps.

In the latter case it might be relevant to send the app owner a notification email when a reload task fails or is aborted. That way the developer is immediately made aware of the issue and can act on it as needed.

This feature assumes the app owner’s user account (in the Sense user directory) has an email address associated with it. When syncing users from Active Directory the users’ emails are often brought along into Sense, but there is no guarantee for this.

If an email address is available for a Sense user, the QMC user section can look like this:

Alert emails only for some tasks

Sometimes there is a desire to only have email alerts for some tasks.
One example can be a Sense cluster that hosts both development and production apps, maybe separated on different servers.

As of Butler 7.4.0 it is possible to control per task if an alert email should be sent when the task fails or is aborted from the QMC.

Conceptually it works like this:

Instructions for how to configure this feature is available here.

Note: This feature is similar to - but independent from - the “task specific email recipients” feature below. Either feature can be enabled or disabled independently of the other in Butler’s config file.

Task specific email recipients

They may be cases where all alert emails should normally go to for example a Sense administrator, but some alerts should instead (or also) go to some other recipients.

An example could be a sales related Sense app. If it fails reloading the standard alert email should go to the Sense administrator, but there should also be an alert email sent to the sales operations team, to notify them that they won’t find updated numbers in the Sales app.

Butler handles this scenario by using a custome propperty (its name is configurable in the Butler config file) to set alert email recipients on a per-task basis.

Conceptually it works like this:

Instructions for how to configure this feature is available here.

Note: This feature is similar to - but independent from - the “alert emails only for some tasks” feature below. Either feature can be enabled or disabled independently of the other in Butler’s config file.

How it works

Butler uses a templating engine called Handlebars. It is used when sending all kinds of alert emails supported by Butler.

Slack, MS Teams and MQTT messages are currently not using the templating engine - this is however likely to change in coming Butler versions. Feel free to add (or +1) a request on GitHub if this is of interest to you!

Template fields

The Handlebars templating engine looks for template fields in the template files you create.

A complete list of template fields - including descriptions - is available in the Reference section.

Not all failed reloads will cause alert emails

While not obvious at first, there are different kinds of reloads taking place in a Qlik Sense Enterprise environment:

• Reloads started by the Sense Scheduler service. These reloads always have a task associated with them.

• Reloads started from Sense’s standard script editor. These reloads are not started by the Sense scheduler, but rather directly in the Sense engine. Progress for such reloads will therefore go to the engine logs.

The log appenders that drive Butler’s alerts rely on the Scheduler logs - not the engine logs.
This is an intentional design decision.

It is certainly possible to add log appenders also for engine logs and that way get notified when any reload fail. The question is whether that’s an interesting use case. In most cases sys admins aren’t very interested in reloads that fail during app development - they only care about failures caused by apps in production - i.e. app reload tasks managed by the Sense Scheduler. Thus, Butler currently doesn’t deal with reload failures reported from the Sense engine.

References

• Qlik’s documenation around log appenders and how to hook into the Sense logs is somewhat brief, but does provide a starting point if you want to dive deeper into this topic.

• The main log4net documentation (log4net is the logging framework used by Qlik Sense Enterprise) can also be useful.

These links describe how emails can be sent from the log4net logging framework itself, directly to the recipient. Butler includes sameple XML files for this use case too, but Butler takes things further by using the data in the Sense logs to pull in more data around the failed or stopped reload.

In other words - Butler’s alert emails are significantly more flexible and contain information (such as script logs) that are not availble using purely log4net.

Seeing is believing

The video below is available at Ptarmigan Labs’ YouTube channel and also in the Butler playlist.

3.2 - Alerts via Slack and Microsoft Teams

Teams, Slack and email notifications

Microsoft Teams, Slack and email are all notification destinations.

Alert messages/notifications come in two variants: “basic” and “formatted”.

Formatted messages

These messages take full advantage of the formatting available in each notification destination.

Slack has its own way for creating messages with rich layout and formatting - as does Teams and email.

Formatted messages are created using template files.
Each notification destination has its own set of template files. It’s therefore possible to take advantage of each destination’s specific features when it comes to formatting the messages sent by Butler.

Message templates can include “template fields”. These are placeholders that are replaced with actual event values before the message is sent.

The GitHub repository includes fully functional template files for all destinations.

Basic messages

Basic message formats are available for all notification destinations.

This message type is useful if you only want a short, basic notification that a task failed or was aborted. The basic formats are pre-defined and are thus easy to set up.

Microsoft Teams notifications

Basic and formatted reload task failure notifications can look like this in Teams:

The configuration needed for setting this up is described here.

Slack notifications

Basic and formatted reload task failure notifications can look like this in Teams:

The configuration needed for setting this up is described here.

Seeing is believing

The video below is available at Ptarmigan Labs’ YouTube channel and also in the Butler playlist.

3.3 - Storing script logs of failed reloads to disk

Reload script logs

When doing a scheduled reload or a reload started from the QMC, Sense will create a detailed log file that includes all the commands executed during the reload.

If a reload for reason fails it can be very useful to look at these reload logs.

The latest reload log file for each reload task is available via the QMC, but logs for previous reload attempts are not available via the QMC.

Using the same mechanism used by reload failure alerts in general, Butler can be configured to store the reload logs of all failed reloads to disk.

The reload logs are stored in the directory configured in the Butler config file, with separate directories for each date:

.
├── butler.exe
├── log
│   └── butler.2022-04-07.log
├── production.yaml
└── scriptlog
    ├── 2022-04-06
    │   ├── 2022-04-06_15-36-12_appId=deba4bcf-47e4-472e-97b2-4fe8d6498e11_taskId=0d815a99-1ca3-4131-a398-6878bd735fd8.log
    │   └── 2022-04-06_22-42-35_appId=66bc109d-286a-415b-8355-1422abb22133_taskId=e959f40a-67be-4a5b-ae83-a292f96ba078.log
    └── 2022-04-07
        └── 2022-04-07_05-49-16_appId=deba4bcf-47e4-472e-97b2-4fe8d6498e11_taskId=0d815a99-1ca3-4131-a398-6878bd735fd8.log

All in all this makes it a lot easier to find log files for failed reloads.

Configuration of this feature is described here.

3.4 - InfluxDB

Visualising failed reloads in Grafana

When a reload fails, Butler can send information about the failed reload to InfluxDB.
The data stored in InfluxDB is described here.

Once the data is in InfluxDB it can be visualized in Grafana or similar tools.
Grafana has a good log viewer that can be used to visualize the data.

Note how even the script log is stored in InfluxDB, so you can see the last few lines of the reload script log in Grafana.
This makes it easy to right away see what went wrong, especially when dealing with reloads that happened a while back.

Configuration

Configuration of this feature is described here.

3.5 - New Relic

View reload alerts in New Relic

When investigating reload failures it can often be useful to have access to the entire reload log, as this usually tells immediately what went wrong.
Butler forwards a very comprehensive set of data to New Relic when a reload fails, including the failing part of the app’s script.

This means that companies using New Relic as their enterprise monitoring solution can now also use it to monitor their Qlik Sense reloads, as well as all the Sense real-time metrics provided by Butler’s sibling tool Butler SOS.

As of this wrtiting, New Relic also offers a free tier that will be more than enough for most Butler users.
It is thus possible to get started with monitoring Sense reloads in New Relic without any additional cost.

More information about how Butler integrates with New Relic can be found here.

Configuration

Configuration of this feature is described here.

3.6 - MQTT

MQTT as unified message bus

When a reload fails, Butler can send information about the failed reload to an MQTT broker.

MQTT is a lightweight messaging protocol that is commonly used in IoT applications, but it is a mature and versatile protocol that can be used in many different scenarios.

In short, MQTT works by having a broker that clients can connect to. Clients can publish messages to the broker, and clients can subscribe to messages from the broker.
This makes MQTT a great way to integrate different systems in a publish/subscribe pattern.

By sending information about failed reloads to an MQTT broker, Butler can be integrated with any system that can consume MQTT messages - which is a lot of systems.

The information included in the MQTT message is described here.

Here is an example of how the information about a failed reload can be viewed in MQTT Explorer:

Configuration

Configuration of this feature is described here.

3.7 - Signl4

Mobile alerting

Signl4 has a great mobile app that can be used to receive reload-failed alerts from Butler.

It can look something like this:

More information about how Butler integrates with Signl4 can be found here.

Configuration

Configuration of this feature is described here.

3.8 - Webhooks

When nothing else works

Webhooks may be somewhat limited in terms of what they can do, but their simplicity is also their strength.

When you need to send information about failed/aborted reloads to a system that doesn’t have a dedicated integration with Butler, webhooks may be a good solution.

Butler offers a lot of flexibility in terms of how each webhook is configured.
You can…

• specify the URL to send the webhook to.
• specify the HTTP method to use (POST, PUT, GET).
• use htto or https.
• specify if a custom certificate should be used. In this case the root CA certificate is provided to Butler per webhook, which means Butler can be integrated to many systems, each using their own self-signed certificate.
• specify if an untrusted certificate should be accepted. This is useful when integrating with systems that use self-signed certificates and you don’t have access to the root CA certificate.

Sending reload-failed informnation to a GET http webhook provided by Node-RED can look like this in the Node-RED editor:

Configuration

Configuration of this feature is described here.

4 - Successful reloads

What’s this?

Butler can optionally track successfully completed reload tasks.

Historically few Sense admins have probably paid much attention to successful reloads (but rather looked at failed ditto), but from a performance planning perspective it’s relevant to also monitor successful reloads.

Monitoring some successful reloads or all

Butler can be configured to track all successful reloads, or only those that have a specific custom property set.
This can be useful if you want to track successful reloads for some apps but not others.

• To enable this feature for all reload tasks, set Butler.influxDb.reloadTaskSuccess.allReloadTasks.enable to true.
• To enable this feature for only some reload tasks, set Butler.influxDb.reloadTaskSuccess.byCustomProperty.enable to true.
  This setting will have no effect if Butler.influxDb.reloadTaskSuccess.allReloadTasks.enable is set to true.
• The name of the custom property is configured in the Butler config file, Butler.influxDb.reloadTaskSuccess.byCustomProperty.customPropertyName.
• The value of the custom property that will enable per-task-tracking is found in Butler.influxDb.reloadTaskSuccess.byCustomProperty.enabledValue.

Using tags

Both Sense apps and reload tasks can have tags attached to them (set in the QMC).
Butler can be configured to get these tags and include them in the data sent to the enabled destinations.

In the case of InfluxDB the Sense tags will be sent as InfluxDB tags, which means they can be used to filter data in InfluxDB queries and Grafana dashboards.

The config file also allows for adding static tags to the data sent to InfluxDB, this is set in Butler.influxDb.reloadTaskSuccess.tag.static.

Grafana dashboards

Given the information stored in InfluxDB, several interesting Grafana dashboards can be created:

• Distribution of reload durations. If filtering on a specific reload task this will show how much the task duration vary. Can be very useful when investigating server performance problems.
• Number of successful reloads per (for example) 15-minute blocks. This shows how reloads are distributed over the course of a day, and can be useful when deciding when to schedule reloads.

Sample dashboard showing the above charts:

Here is another set of charts, also showing what metadata is available for each reload task:

How it works

Reload success UDP server

Butler includes a UDP server to which Sense’s logging framework Log4Net can send reload success messages.

The UDP server is configured in the Butler config file, and is disabled by default. When enabled, the UDP server listens for UDP messages on the configured port.

When a UDP message is received, the UDP server will parse the message to determine what it is about and to extract relevant data, then dispatch the message to the enabled destinations.
The XML appenders must be correctly deployed on the Sense servers for this to work.
See this page for more information on setting up the XML appenders.

In the case of successful reload tasks, the UDP server will determine which app was reloaded, the duration of the reload task, who started the task etc.
Butler will then send this information to the enabled destinations.

Supported destinations

The following destinations are supported:

• InfluxDB

Config file settings

Butler:
...
  ...          
  # InfluxDB settings
  influxDb:
    enable: false                   # Master switch for InfluxDB integration. If false, no data will be sent to InfluxDB.
    ...
    ...
    reloadTaskSuccess:
      enable: true
      allReloadTasks:
        enable: false
      byCustomProperty:
        enable: true
        customPropertyName: 'Butler_SuccessReloadTask_InfluxDB'
        enabledValue: 'Yes'
      tag: 
        static:                 # Static attributes/dimensions to attach to events sent to InfluxDb
          # - name: event-specific-tag 1
          #   value: abc 123
        dynamic:
          useAppTags: true      # Should app tags be sent to InfluxDb as tags?
          useTaskTags: true     # Should task tags be sent to InfluxDb as tags?
  ...
  ...          

5 - Starting Sense tasks in style

What’s wrong with starting tasks from the QMC?

Nothing really. At least not if starting tasks is considerd a manual activity.

There are also scenarios that benefit from automation.
Consider a Sense environment that gets its data from some source system, for example a financial ERP system.:

Sense could poll that source system for data ever so often - or the source system could start the needed tasks when new data is available.

Or simply: Notifications are usually (always?) a better solution than polling for new data.

The good news is that Butler includes solid support for starting Sense tasks from 3rd party systems.

Below the concept is described on a high level.
Actual examples are available here.

Start tasks by IDs, tags and custom properties

Butler’s /v4/reloadtask/<taskid>/start API endpoint is used to start tasks.
For historical reasons there are two variants: POST and PUT. Both do the exactly the same though.

The general idea of this API is

• One task should be started. The task is identified by its task ID.
  Done using just a properly formatted URL. This is usually the most common scenario.
• More than one task should be started. The tasks are identified by their IDs.
  The first task can be specified in the URL and the rest in an array in the call’s body.
  Or all task IDs can be specified in the call’s body.
• One or more tasks should be started. The tasks are identified by tags set in the QMC.
  The tags are specified in the call’s body.
  All tasks having the tags included in the call will be started.
• One or more tasks should be started. The tasks are identified by custom property values set in the QMC.
  Same principle as for tags, but using custom properties instead.
  All tasks having the custom property/value combinations in the call will be started.

Start tasks by ID

While this may be the most obvious way to control what task(s) should be started, it requires the caller to know the exact ID of the task of interest.
If the task for some reason is re-created its ID will change.

Still, there are certainly cases where task IDs are relevant and the easiest option to use and set up.

Start tasks by tags

Given the example below, a single call to Butlers API could start all four of the tasks tagged Butler 5.0 demo.
Or the three tasks tagged startTask1.
Or all of those tasks, if both tags were passed to the Butler API.

Start tasks by custom properties

Using custom properties to identify which tasks to start works similarly to how tags are used (see above).
The main difference is that the caller must know the name of the custom property and at least one of the possible values for that custom property, in order to start the associated task.

In the example below, calling the Butler API with parameters taskGroup=tasks2 would result in all tasks having the taskGroup custom property set to tasks2 to be started.

6 - Incident management tools

6.1 - New Relic

New Relic is not primarily an incident management tool, but rather a complete SaaS platform for handling metrics, logs, events and trace messages.

Their event handling however has some alert handling features built in, and these can be nicely integrated with Butler and Qlik Sense.
Furthermore, New Relic also integrates with several dedicated incident manamgenet tools (PagerDuty, VictorOps, OpsGenie and others) and also other notification channels such as Slack, Teams, email, and generic webhooks.
Toghether these capabilities makes New Relic a very good match to the features offered by Butler and Butler SOS.

The concept looks like this:

1. Alerts (for example a reload task fails in Qlik Sense) are sent to New Relic using their event and log APIs.
   Butler integrates tightly with those APIs, creating a seamless, almost instantaneous forwarding of incidents.
2. Metadata about the failed or aborted reload task and associate Sense app is sent to New Relic together with the last parts of the reload script log.
   The script log can then be viewed from within New Relic’s web interface and usually provides instant insight into what caused the reload to fail.
3. As part of the setup process, “alert conditions” are created in New Relic. These define when New Relic alerts should be created, given the event data sent from Butler.
4. New Relic “alert policies” are used to group together several conditions and also associate notification channels (Slack, Teams, PagerDuty, …) to each alert policy.
5. New Relic incidents stay in an open state until they are acknowledged, which can be done from the web interface or from within email or Teams/Slack messages.

The Getting started section has hands-on instructions for setting up Butler to work with New Relic.

New Relic screen shots

Sample New Relics dashboards are found below.

Please note that some of the charts (server RAM, CPU, user sessions etc) in these dashboards use data from Butler SOS.
Butler provides data for failed/aborted reloads tables and charts.

6.2 - Signl4

Signl4 offers both an online SaaS service and a mobile app.

The concept looks like this:

1. Alerts (for example an important Sense app fails reloading) are sent to Signl4 using their APIs.
   Using their API an incident is created.
   Butler integrates with these APIs, creating a seamless, almost instantaneous handling of incidents.
2. The incident is assigned to whoever is on duty at that time. This person (or group of people in larger organisations) will be notified via their Signl4 mobile apps.
3. The person on duty can acknowledge that an incident is being looked into and eventually also that the incident has been resolved.
4. If no-one acknowledge the incidient within a configurable time the incident can be escalated.
5. … and much more.

The Getting started section has hands-on instructions for setting up Butler to work with Signl4.

Example: Qlik Sense and Signl4 on Android

Below are screen shots from an Android phone in which the Signl4 mobile app is installed.

7 - Qlik Sense licenses

Monitor user licenses

In order to use Qlik Sense end users must be assigned some kind of access license.
There are different types of licenses, for example Professional, Analyzer and Analyzer Capacity licenses.

Butler can monitor the usage of these licenses and store the data in InfluxDB, from where the license data can be visualized in Grafana.

This makes it easy to track (and alert if needed) on the number of used licenses, how many are available and when it’s time to get more licenses.

Release inactive user licenses

Butler can automatically release Professional and Analyzer user licenses that have been inactive for a certain period of time.

This is useful in environments where some users use Sense sporadically, for example only during certain times of the year.
In such cases it’s a waste of resources to keep the license assigned to the user when it’s not being used.

Here is how it works:

• Butler will evaluate allocation of professional and analyzer access licenses on a configurable schedule, for example once per day.
• Licenses that have not been used for a certain period of time (e.g. 30 days) will be released.
  • Separate inactivity periods can be configured for professional and analyzer licenses.
• Licenses that are within quarantine period will never be released.
• It is possible to exclude certain users from having their licenses released, i.e. guarantee they will always have a license available. Can be useful for administrators or users that need guaranteed access to Sense.
  • The following criteria can be used to exclude users from having their licenses released:
    • Specific users (by userDirectory\userId)
    • Specific tag(s) assigned to a user.
    • Specific custom property value(s) assigned to a user.
    • Users belonging to a user directory.
    • Users being marked as (not) active in the QMC.
    • Users being marked as (not) blocked in the QMC.
    • Users being marked as (not) externally removed in the QMC.

Disclaimer

This feature has the potential to let more users use your Sense environment, compared to a scenario where no release of licenses is done.

In order to avoid users not being able to access Sense you should still ensure to have a good margin, i.e. get more licenses from Qlik once you are running low. This is the only (and correct and proper) way to ensure that users are not denied access to Sense due to missing licenses.

Also, you must ensure that managing licenses this way is not in conflict with your license agreement with Qlik.

Butler’s license release feature uses APIs that are publicly documented by Qlik (example here).

The same APIs are used by the QMC to release licenses.
Butler simply automates what is otherwise a manual task in the QMC.

8 - File system access: copy/move/delete files

Unrestricted file system access is a security risk

Qlik Sense locked down things quite a bit compared to its QlikView predecessor.

In QlikView your app scripts could do almost anything with any file on the server’s disks as long as the QlikView service account had access to the file.
This was not ideal from a security perspective and Qlik Sense therefore introduced the concept of folder data connetions and in general much stricter file system access restrictions.

With this change Qlik Sense had a much better position with respect to security, as access to files was now boxed by the folder data connection the access used (by means of lib:// statements).
It’s also possible to include .qvs script files via the same mechanism.

The problem now is that it’s no longer possible to do file level operations on individual or groups of files.
No more deleting, copying or moving of files from within the load script.

Now - there is a per-server setting that disables this new “standard mode” and reverts back to what’s known as “legacy mode”, which is essentially how QlikView worked (and still works). But then the Sense environment is once again vulerable to badly written or even malicious Sense apps.

Butler adds controlled file system access to Qlik Sense Enterprise

Butler’s solution is to add a set of REST API endpoints for file copy/move/delete operations, but only allow these to operate on pre-defined folders.

For example, you might have a QVD folder at e:\data\qvd\sales\temp.
You also need to remove old QVDs from that folder.

This could be done with scheduled BAT/CMD files or PowerShell scripts, but it might be better/more flexible/easier/preferred to do this cleanup from the load script of a Sense app.

The solution: Add e:\data\qvd\sales\temp to Butler’s list of folders in which files can be deleted, then call Butler’s /v4/filedelete API endpoint from within your app’s load script. Done!

Convenience subs

Butler includes a set of Subs that make it easy to use the file copy/move/delete APIs.
These subs are found in this .qvs file as well as embedded in the Butler demo app.

The examples section shows how to use these subs - or call the Butler APIs directly.

9 - MQTT integration

What is MQTT?

MQTT is a light weight publish-subscribe (“pub-sub”) protocol.

Used in both the telecomms industry and various Internet of Things applications, there are client libraries available for many different languages and platforms. This is important, as there is a good chance other systems can find a way of sending MQTT messages, which Butler can then listen for/subscribe to.

Outgoing MQTT from Butler itself

If MQTT is enabled, Butler will forward events (reload failed, user opened a session to Sense etc) to MQTT. These events are sent to the MQTT topics defined in the Butler.mqttConfig section of Butler’s config file.

Outgoing MQTT - publish

Butler can post messages to MQTT topics. The /v4/mqttpublishmessage API endpoint is used for this.
This way Butler can send status information and notifications to other systems, outside of Qlik Sense. Use cases include:

• Notify downstream systems that Sense has finished creating some data set that the downstream system depends on.

• Send debug or trace messages to MQTT instead of to the Sense log. Using a MQTT client (there are multiple ones on both Windows, OSX and Linux) you can then monitor the messages in real time. Very useful during development of tricky Sense load scripts!

• Start Sense tasks (typically reloads) from the Sense load script. Very useful when you need to trigger a second app reload once the first app’s load script reaches some specific point of execution.
  This way the scheduling and execution of Sense tasks can be made much more flexible than using the built in QMC scheduler.

  Note: While possible to start reload tasks using MQTT, it’s usually easier to do this using Butler’s REST API.

• Send messages to platforms such as Node-RED. Node-RED is an open source platform (with a graphical editor) intended for integrating different systems and data sources. As it is built on node.red there are many different modules available, offering integrations with all sorts of systems and protocols.
  Using Node.RED together with Qlik Sense and Butler, it is possible to interface with social media from the Sense load script (send a Tweet when some condition occur during app reload, for example).

Incoming MQTT - subscribe

Butler subscribes to all MQTT messages in the topic specied in the config setting Butler.mqttConfig.subscriptionRootTopic.
Which in MQTT lingo means “listen to all messages in the this topic, as well as in any subtopics”.

When Butler gets a message as a result of this subscription it is analysed and if the topic matches any of the predefined topics with special meaning, the associated tasks are carried out.

Topics with special meaning are:

• Start Sense task. The exact topic is defined in config property Butler.mqttConfig.taskStartTopic.
  Note that this topic must be a subtopic to the topic specified in Butler.mqttConfig.subscriptionRootTopic!
  Starts the Sense task identified by the ID sent in the message body. More info in the examples section.

As Butler listens to all messages in the topic tree specified by Butler.mqttConfig.subscriptionRootTopic it can easily be extended with handlers for additional topics.

10 - UDP client

Butler includes a very basic UDP client, which can be used to send test messages to Butler’s UDP servers.
This can be useful when debugging a Butler server, when adding new UDP handlers etc.
The client is built using node.js, and is found in the src/udp_client directory.

Run the app to show its help text (in this case the UDP client is executed on a Mac):

$ node udp_client.js
Usage: node udp_client.js [options]

This app sends messages to the UDP server(s) built into Butler (or any other UDP
server)

Options:
  --version   Show version number                                      [boolean]
  -i, --ip    IP address of UDP server message will be sent to        [required]
  -p, --port  Port on UDP server                                      [required]
  -m, --msg   Message to send                          [default: "Test message"]
  -h, --help  Show help                                                [boolean]

Missing required arguments: i, p
$

Testing the failed task UDP server

Sending a message to port 9998 will test the UDP server responsible for handling task failure messages:

Sending a message to Butler looks like this (with a fake IP address):

$ node udp_client.js --ip 1.2.3.4 -p 9998 -m "Abc;123;456;test"
UDP message sent to 1.2.3.4:9998, 16 bytes.
$

The resulting Slack message looks like this:

11 - Monitor Windows services

Be the first to know when a service stops

If a Sense service stops (for whatever reason), end users will most likely be impacted one way or another.

Maybe apps will not be reloaded with new data, or users will not be able to access the hub or QMC.

Some companies have dedicated teams that monitors the IT infrastructure, but in many cases the Sense platform is monitored by the same team that develops and maintains the Sense apps.

For these teams, it is important to be notified as soon as possible when a Sense service stops - Butler can help with this.

Grafana dashboards and alerts

Butler can send service related alerts to InfluxDB, and Grafana can then be used to create dashboards and alerts based on the data in InfluxDB.

A Grafana “state timeline” chart showing the status of Qlik Sense services across 4 servers can look like below.
Note the red bar indicating that all Qlik Sense services were restarted at one point, but also note that a couple of the servers had additional, shorter outages:

With the above in place, Grafana alerts can be created that will trigger when a service stops, sending messages to incident management systems such as PagerDuty, OpsGenie and VictorOps - or to Slack, Teams, email etc.

Many servers, many services - and permissions

Butler can monitor services on multiple servers, and it can monitor multiple services on each server.

It should be noted though that each check is executed sequentially, so if you have many servers and many services, it will take some time to complete all checks.
This is especially true if you are monitoring services on remote servers, as the latency will add up.

In order to monitor services on remote servers, Butler needs to be able to connect to the remote server using WMI (Windows Management Instrumentation).

Long story short, the easiest way to set things up is to make the account used to run Butler a member of the local Administrators group on the remote server.
This should be done on all remote servers that Butler will monitor services on.

How it works

Butler uses a state machine to keep track of the status of each service.
The available states and the possible transitions between them are:

When a service check reveals that a service has changed state compared to the state stored in the state machine, Butler will send an alert message to the configured destinations.

Note 1: Windows services technically have more states than the ones shown above, but for our use case the above states are sufficient.

Note 2: The state machine is not persisted to disk, so if Butler is restarted it will not remember the state of the services it was monitoring before the restart.

Note 3: The states shown above are the ones used by Butler internally. The actual alert messages sent to the configured destinations may use different wording.

Config file settings

The configuration for Windows service monitoring is done in several sections in the Butler config file.

First, the general services configuration section defines

• Which services should be monitored on what servers.
• How often the checks should be executed.
• What destinations should receive service related alerts.

Butler:
  ...
  ...
  # Monitor Windows services.
  # This feature only works when Butler is running on Windows Server or desktop.
  # On other OSs service monitoring will be automatically disabled.
  serviceMonitor:
    enable: false                         # Main on/off switch for service monitoring
    frequency: every 5 minutes            # https://bunkat.github.io/later/parsers.html
    monitor:
      - host: <hostname or IP>            # Host name of Windows computer where services are running
        services:                         # List of services to monitor
          - name: postgresql-x64-12       # Posgress/repository db
            friendlyName: Repository DB
          - name: QlikSenseEngineService
            friendlyName: Engine
          - name: QlikSensePrintingService
            friendlyName: Printing
          - name: QlikSenseProxyService
            friendlyName: Proxy
          - name: QlikSenseRepositoryService
            friendlyName: Repository
          - name: QlikSenseSchedulerService
            friendlyName: Scheduler
          - name: QlikSenseServiceDispatcher
            friendlyName: Service Dispatcher
    alertDestination:               # Control to thich destinations service related alerts are sent
      influxDb:                     # Send service alerts to InfluxDB
        enable: true
      newRelic:                     # Send service alerts to New Relic
        enable: true
      email:                        # Send service alerts as emails
        enable: true                
      mqtt:                         # Send service alerts as MQTT messages
        enable: true
      teams:                        # Send service alerts as MS Teams messages
        enable: true
      slack:                        # Send service alerts as Slack messages
        enable: true
      webhook:                      # Send service alerts as outbound webhooks/http calls
        enable: true
  ...
  ...

Secondly, the configuration for the various destinations may also contain settings specific to service monitoring.
Not all destinations support this, but for example the Teams destination does:

Butler:
   ...
   ...
  # Settings for notifications and messages sent to MS Teams
  teamsNotification:
    enable: true
    ...
    ...
    serviceStopped:
      webhookURL: <web hook URL from MS Teams>
      messageType: formatted          # formatted / basic. Formatted means that template file below will be used to create the message.
      basicMsgTemplate: 'Windows service stopped: "{{serviceName}}" on host "{{host}}"'       # Only needed if message type = basic
      rateLimit: 30                   # Min seconds between messages for a given Windows service. Defaults to 5 minutes.
      templateFile: /path/to/teams/template/directory/service-stopped.handlebars
    serviceStarted:
      webhookURL: <web hook URL from MS Teams>
      messageType: formatted          # formatted / basic. Formatted means that template file below will be used to create the message.
      basicMsgTemplate: 'Windows service started: "{{serviceName}}" on host "{{host}}"'       # Only needed if message type = basic
      rateLimit: 30                   # Min seconds between messages for a given Windows service. Defaults to 5 minutes.
      templateFile: /path/to/teams/template/directory/service-started.handlebars
   ...
   ...

Similarly, service state changes can be sent to outbound webhooks:

Butler:
  ...
  ...
  # Settings for notifications and messages sent using outgoing webhooks
  webhookNotification:
    enable: true
    ...
    ...
    serviceMonitor:
      rateLimit: 15               # Min seconds between outgoing webhook calls, per Windows service that is monitored. Defaults to 5 minutes.
      webhooks:
        - description: 'This outgoing webhook is used to...'
          webhookURL: http://host.my.domain:port/some/path    # outgoing webhook that Butler will call
          httpMethod: POST                                    # GET/POST/PUT. Note that the body and URL query parameters differs depending on which method is used
        - description: 'This outgoing webhook is used to...'
          webhookURL: http://host.my.domain:port/some/path    # outgoing webhook that Butler will call
          httpMethod: PUT                                     # GET/POST/PUT. Note that the body and URL query parameters differs depending on which method is used
        - description: 'This outgoing webhook is used to...'
          webhookURL: http://host.my.domain:port/some/path    # outgoing webhook that Butler will call
          httpMethod: GET                                     # GET/POST/PUT. Note that the body and URL query parameters differs depending on which method is used

To get MQTT messages when a service stops or starts, configure the MQTT section of the config file:

Butler:
  ...
  ...
  mqttConfig:
    enable: true                                     # Should Qlik Sense events be forwarded as MQTT messages?
    brokerHost: <FQDN or IP of MQTT server>
    brokerPort: 1883
    azureEventGrid:
      enable: false              # If set to true, Butler will connect to an Azure Event Grid MQTT Broker, using brokerHost and brokerPort above 
      clientId: <client ID>
      clientCertFile: <path to client certificate file>
      clientKeyFile: <path to client key file>
    ...
    ...
    serviceRunningTopic: qliksense/service_running
    serviceStoppedTopic: qliksense/service_stopped
    serviceStatusTopic: qliksense/service_status
  ...
  ...

12 - Real-time metrics