About

• 1: Butler
• 2: Use cases
• 2.1: Retired use cases
• 3: The Butler family
• 4: Versions
• 5: Contribution guidelines
• 6: Telemetry

1 - Butler

The Butler project is all about adding useful features to the client-managed version of Qlik Sense, also known as Qlik Sense Enterprise on Windows, QSEoW.
Some of the features can be used from Sense load scripts, other features provide integration with 3rd party systems.
Most of Butler’s features try to make daily life for a Qlik Sense administrator a bit easier.

The goal is to integrate battle-proven concepts and best-of-breed open-source tools into Butler and thus make them available to developers of Sense apps or those responsible for running Sense clusters.
In some cases it might be possible to use these tools from within Sense also without Butler - in those cases Butler simply tries to make things easier, lowering the barriers to get started and get things done.

There is also a clear goal that Butler should be very configurable. In practice this means that individual features can be turned on/off as needed, improving security and lowering memory usage.

Butler is written in Node.js and runs on most modern operating systems.
As of version 7.2 Butler offers pre-built binaries for Windows, macOS and Linux§, meaning that Node.js no longer has to be separately installed to use Butler.
This simplifies things - just download, configure and run.

You can run Butler on the same server as Qlik Sense, in a Docker container on a Linux server, in Kubernetes, on Mac OS, on Raspberry Pi (not a good idea.. but possible and proven to work).

Butler is a member of a group of tools collectively referred to as the “Butler family”, more info is available here.

This picture might be useful to understand what Butler does and how it fits into the larger system map around Qlik Sense:

2 - Use cases

• Instant notifications when reload tasks fail or are stopped
• Forward failed reload events to incident management systems (New Relic, Signl4)
• Use InfluxDB/Grafana or New Relic to track Butler memory usage
• Save a copy of the complete reload log for all failed reload tasks
• Start reload tasks from load script or from upstream systems
• Start any reload task from within any Qlik Sense or web app
• Start reload tasks via REST API based on task tags or custom properties
• Trigger full/partial app reloads from load script or upstream systems
• Flexible scheduling of app reloads in Qlik Sense Enterprise on Windows
• Passing parameters between reload tasks
• Storing state across several apps
  • Time-to-live (TTL) for key-value pairs
• Make new data reach end users as quickly as possible
• Using MQTT to notify downstream systems that Sense is done processing data
• Create directories, copy/move/delete files
• Extract metadata for apps
• Easily post messages to Slack
• Monitor Windows services
• Monitor and release Qlik Sense user licenses

Instant notifications when reload tasks fail or are stopped

Information about failing tasks can be sent as emails, to Microsoft Teams, Slack, as MQTT messages or outgoing webhooks.

Email, Slack and MS Teams notifications all use a templating concept where HTML/Markdown template files describe what the alert message should look like. Before the alert is sent the template is populated with actual data from the failed reload task.

Both subject and body of email can use the template fields.

For both Slack and Teams there are options to use more flexible/configurable alert formats and more basic pre-configured alerts.

The result is a very poweful tool for QSEoW sysadmins, who get real-time insight into what’s happening with respect to task execution.

More info here.

Forward failed reload events to incident management systems (New Relic, Signl4)

Butler offers advanced failed reload alerts via Slack, Teams, email and outgoing webhooks.
Configurable templates means you can customize emails/Teams/Slack messages.

Sometimes you want a bit more structure though.
This is especially true when Sense is used in the enterprise.

Butler integrates with both Signl4 and New Relic.
Both offer incident management features on both the web and via mobile clients.

Information about failed/aborted reloads can be sent to one or more New Relic accounts.
Tags for the reload task and associated app is sent to New Relic as metadata for the event/log entry that’s created there.

Use InfluxDB/Grafana or New Relic to track Butler memory usage

Butler can be configured to log its own memory usage to InfluxDB, from where it can be visualised using Grafana.

If you prefer using New Relic One that’s possible too - sending Butler memory metrics to New Relic is super simple: Just add your New Relic credentials in the YAML config file or as command line options when starting Butler and you’re set.

Save a copy of the complete reload log for all failed reload tasks

Let’s say a scheduled reload task fails.

This can happen due to lots of reasons, from uncontrollable events that are impossible to predict to bugs in the script of a Sense app.

No matter what the cause is, as a Sense administrator you probably want to investigate the script reload logs.

Butler can send notifications (Slack, Teams, email, webhooks, …) when reloads fail.
These notifications can include the last 20-30-40 lines of the script log and this usually gives a good idea of what caused the reload to fail.

But what if you want to look at the complete reload log of that failed app reload?

So far you would have to dig into the log directory on the Sense server, find that specific reload log among potentially thousands of other log files. Not very effective.

As of version 7.2 Butler can store a copy of the complete reload log in a directory that you specify.
The log files are stored in separate directories, one for each date.
This makes it easy to find the log file you are interested in.

Start reload tasks from load script or from upstream systems

Trigger Sense reload tasks from a reload script: This makes it possible to start different Sense tasks based on what data has been read from a database, what time of day it is etc.
Starting a task from the reload script is as easy as Call StartTask('fbf645f0-0c92-40a4-af9a-6e3eb1d3c35c').

Trigger Sense reloads from external systems: When new data is available in a source database, that database can trigger a reload in Sense, and the data is loaded from the database into Sense. This way delays caused by Sense polling for data are minimized and data arrives at end users as quickly as possible.

Starting reload tasks using REST API is described here.
Using MQTT messages to achieve this is described here.

Start any reload task from within any Qlik Sense or web app

Some HTML and Javascript magic is also needed, but given Butler’s start-task API it’s pretty easy to set up a button in a Sense app (or any web app!) to start any Sense reload task.

This can for example be used to allow end users to start an Extract-Transform when they (the user) need refreshed data.

More info here.

Start reload tasks via REST API based on task tags or custom properties

Using tags and/or custom properties to identify what tasks should be started can be easier than having to know the tasks IDs. This both makes it easier for 3rd party systems to start Qlik Sense tasks and easier for Sense admins to manage which tasks should be startable by 3rd party systems.

More info in Concepts and Examples sections.

Trigger full/partial app reloads from load script or upstream systems

Sometimes you just want to reload an app without also having to create a reload task.
When it comes to partial app reloads it’s not even possible to do these from a Sense reload task.

Butler’s API makes prvovides a solution: Just pass in an app ID to reload together with task IDs of the tasks that should be started when the app is done reloading (different tasks can be started depending on app reload success or failure).

The partial reload feature is of special interest as it can be used to trigger faster incremental execution of of Extract-Transform reload chains. Great for keeping data in Sense apps updated during the course of a day!

More info here.

Flexible scheduling of app reloads in Qlik Sense Enterprise on Windows

Using the scheduler built into Qlik Sense you can’t for example create schedules that are limited to a parts of a day.
This is a pretty common scenario though - you want to reload an app hourly from say 3 am to 3 pm.

You can set this up in Sense, but it involves creating a lot of triggers for the reload task, which becomes a nightmare to maintain.

Butler’s task scheduler is built on Cron, which has been used in Linux for decades. Battle proven and very flexible.

Passing parameters between reload tasks

This has always been hard both in QlikView and Sense.

Butler’s key-value store makes it much easier to pass values from one app to the next in a reload chain.

Storing state across several apps

The key-value store can also be used to keep state in general across several apps or parts of a Qlik Sense environment.

Maybe a Development cluster needs to share information in real time with the Test and Production clusters?
Easily solved using Butler’s key-value store.

A demo showing parameter passing between apps is found here.

Time-to-live (TTL) for key-value pairs

Key-value pairs can optionally be set up with a time-to-live (ttl) parameter. If ttl is set, the KV pair will auto-delete when the ttl expires.

The key-value store is described here.

Make new data reach end users as quickly as possible

See above. Have the upstream data source initiate Sense app reloads, either via Butler’s REST API or via MQTT messages sent to Butler.

Using MQTT to notify downstream systems that Sense is done processing data

Use Butler’s API endpoints for MQTT handling to send and receive MQTT publish-subscribe messages.
MQTT (and the pubsub concept in general) is a great way for systems to communicate reliably with each other.

A demo app is available, showing how MQTT messages can be sent from Sense load scripts. More info here.

Create directories, copy/move/delete files

In “standard mode” apps reloading in Qlik Sense Enterprise on Windows can’t access the file system of the Sense servers. This is a good thing because it adds a lot of security.

From time to time you need to delete temp QVDs though, or copy or move data files from one directory to another.

Butler has REST API endpoints for these use cases, but as those endpoints are locked down to only work on specific, configurable directiories they don’t result in the same security issues as seen in for example QlikView or Sense running in legacy mode.

More info here.

Extract metadata for apps

Exporting apps as JSON can be very useful for backup purposes. Doing regular snapshots of all apps in a Sense cluster is a fast and space-effective way of keeping point-in-time backups.

The REST API documentation has full docs for the /v4/app/{appId}/dump endpoint.

Easily post messages to Slack

Slack messages can include full formatting (web links, text formatting etc), as well as “poking” users.
I.e. notifying specific Slack users that they have a new message.

Can for example be used to notify user(s) that an app has reloaded with new data, or that some error condition has occured.

More info here.

Monitor Windows services

If Butler is running on Windows (server or even desktop) it can monitor one or more Windows services.
This feature is not available when running Butler on Linux, macOS or in Docker.

Monitoring here means tracking the services’ state and sending messages to email, InfluxDB, New Relic, MQTT, Webhooks, Slack or Teams when services stop or start.

It can for example be used to get alerts if a Qlik Sense service for some reason stops.
The concept is not limited to Qlik Sense services though - any Windows service can be monitored.

Monitor and release Qlik Sense user licenses

Butler can monitor the usage of Qlik Sense user licenses and store the data in InfluxDB, from where the license data can be visualized in Grafana. This makes it easy to track (and alert if needed) on the number of used licenses, how many are available and when it’s time to get more licenses.

Butler can also automatically release Professional and Analyzer user licenses that have been inactive for a certain period of time. This is useful in environments where some users use Sense sporadically, for example only during certain times of the year. In such cases it’s a waste of resources to keep the license assigned to the user when it’s not being used.

More info here.

2.1 - Retired use cases

Some features age with grace, others don’t.
Here’s a list of features that are candidates for removal from Butler, or that have already been removed.

Candidates for removal in coming versions

Real-time metrics around active users

While a good idea in theory, Butler just wasn’t the vehicle for this.

The way Butler approached this was to have Sense’s log4net logging framework send UDP messages to Butler when users logged in/out or sessions started/ended. This certainly works (quite well in fact!), but it also has inherent issues.
For example, when Butler was started it wouldn’t capture currently active users or sessions - it was only after some event captured in the logs that Butler would update it’s internal counters. This meant that it would take some time (sometimes quite long) until the metrics were even approaching the real number of users using Sense.

There was also the risk of Butler missing UDP messages and not registering the associated log event.

The affected API endpoints are:

/v4/activeusercount
/v4/activeusers

3 - The Butler family

Butler started out with a very specific need to start Sense reloads from outside systems.
Over the years a few projects (for example Butler SOS, which simplifies day 2 operations ([1], [2]) have spun off from the original Butler project, and still other projects have been created from scratch to solve specific challenges around developing Sense apps and running Qlik Sense server environments.

All members of the Butler family are available on Ptarmigan Labs’ GitHub page.

Projects with production grade release status are (as of this writing):

Butler

The original Butler. Offers various utilities that make it easier to develop Sense apps, as well as simplifying day 2 operations.

butler.ptarmiganlabs.com. (This site!)

Butler SOS

Real-time operational metrics for Qlik Sense. A must-have if you are responsible for a Sense environment with more than a dozen or so users.

Butler SOS makes it possible to detect and alert on issues as they happen, rather than in retrospect much later.

Several storage and visualisation options available, including InfluxDB + Grafana, and New Relic.

butler-sos.ptarmiganlabs.com

Butler Sheet Icons

Automates the creation of sheet icons for both Qlik Sense Cloud and client-managed Qlik Sense Enterprise on Windows (QSEoW) applications.

It’s a cross platform command line tool which given the correct Sense credentials will take screen shots of all sheets in a Sense app (or all apps on a Sense server!), then create thumbnail versions of those screenshots.
Finally those thumbnails will be set as sheet icons.

No more manual screenshot taking, resizing images, navigating hundreds of sheets in dozens of apps.
Start Butler Sheet Icons instead and go get a nice fika.

The tool can be used stand-along or as part of an automated release process.

https://github.com/ptarmiganlabs/butler-sheet-icons

Ctrl-Q

Given the name of this tool it doesn’t sound like a member of the Butler family.
Let’s say Ctrl-Q is a sibling of the Butler bunch.

While the Butler tools are (usually) intended to solve and simplify rather specific use cases, Ctrl-Q is aimed at being the lazy Qlik developer’s best friend.

Let’s say there is some manual, tedious, time consuming and error prone activity that a Qlik Sense developer is faced with.
For example importing dozens of apps from QVF files and creating a hundred associated reload tasks.
Ctrl-Q lets you do this with a single command, using definitions in an Excel file. Instead of spending a day on this the actual execution takes a minute or so.

In other words: Ctrl-Q focus on high-value use cases that are difficult or impossible to solve using other tools.

github.com/ptarmiganlabs/ctrl-q

Butler CW

Butler Cache Warmer. Cache warming is the process of proactively forcing Sense apps to be loaded into RAM, so they are readily available when users open them.
Using Butler CW is an easy way to make your end users’ experience of Sense a little better.

github.com/ptarmiganlabs/butler-cw

Butler App Duplicator

No matter if you are a single developer creating Sense apps, or have lots of developers doing this, having app templates is a good idea:

• Lowered barrier of entry for new Sense developers.
• Productivity boost when developing Sense apps.
• Encouraging a common coding standard across all apps.

github.com/ptarmiganlabs/butler-app-duplicator

Butler Spyglass

This tool is mainly of interest if you have lots of QVDs and apps, but when that’s the case it’s of paramount importance to understand what apps use which QVDs. In other words what data lineage looks like.

Butler Spyglass also extracts full load scripts for all Sense apps, creating a historical record of all load scripts for all Sense apps.

github.com/ptarmiganlabs/butler-spyglass

Butler Notifier

This tool makes it easy to tap into the Qlik Sense notification API. From there you can get all kinds of notifications, including task reload failures and changes in session state (user login/logout etc).

github.com/ptarmiganlabs/butler-notifier

Butler Icon Uploader

Visual looks is important when it comes to analytics, and this holds true also for Sense apps.

The Butler Icon Uploader makes it easy to upload icon libraries (for example Font Awesome) to Qlik Sense Enterprise. With such icons available it is then easy for app developers to use professional quality sheet and app icons in their Sense apps.

github.com/ptarmiganlabs/butler-icon-upload

4 - Versions

Documentation updates

Starting with Butler version 4.0, this documentation site will offer both latest and earlier site versions.

Select which doc site to view in drop-down list in the upper right corner of the page.

5 - Contribution guidelines

Butler is developed in Node.js, with support from various NPM modules.

We use Hugo to format and generate this documentation site, the Docsy theme for styling and site structure.
Hugo is an open-source static site generator that provides us with templates, content organisation in a standard directory structure, and a website generation engine. You write the pages in Markdown (or HTML if you want), and Hugo wraps them up into a website.

All submissions, including submissions by project members, require review. We use GitHub pull requests for this purpose. Consult GitHub Help for more information on using pull requests.

Creating an issue

If you’ve found a problem - or have a feature suggestion - with Butler itself or the documentation, but you’re not sure how to fix it yourself, please create an issue in the Butler repo. You can also create an issue about a specific doc page by clicking the Create Issue button in the top right hand corner of the page.

6 - Telemetry

What’s telemetry

From Wikipedia:

Telemetry is the in situ collection of measurements or other data at remote points and their automatic transmission to receiving equipment (telecommunication) for monitoring.

In the context of software tools (including Butler) telemetry is often used to describe the process of sending information about the tool itself to some monitoring system.

Why telemetry in Butler

This is a very, very good question.

For many years there was no telemetry at all in Butler.

Development of new features were driven mainly by what features were needed at the time.
Or the fact that Qlik released some new feature in Sense and Butler was a way to test that new feature from the perspective of the Sense APIs.

That’s all good but today Butler is a rather significant tool with features spanning quite different areas (alerting, task scheduling, key-value store and more).

This multitude of features is also one of the core reasons for adding telemetry to Butler:

Without this information the Butler developers will keep working in the dark, not really knowing where to focus their efforts.

On the other hand - with access to telemetry data a lot of possibilities open up for the Butler developers:

• If telemetry shows that no one uses a particular feature, maybe that feature should be scheduled for deprecation?
• The opposite of the previous: If lots of users use a specific Butler feature, then that feature is a candidate for future focus and development.
• Telemetry will show if lots of users run Butler on old Node.js versions. Knowing this its possible to set a migration schedule for what Node.js versions are supported - avoiding hard errors when some old Node.js version is no longer supported by Butler.
• Same thing for understanding what operating systems Butler runs (and should be supported) on.

Configuring Butler’s telemetry

Instructions here.

The details

Where is telemetry data sent

The telemetry data is sent to the PostHog service, using their database in the European Union.

Deleting telemetry data

Even though no-one (not even the Butler developers or Ptarmigan Labs who manage the telemetry database!) has any way of ever connecting the data sent by your Butler instance to you (it’s all anonymized, remember?), there can be cases where telemetry data must be deleted.

The legal page has more information about this.

Field level description of telemetry data

A telemetry message from Butler contains the information below.

{
  "ts": "2021-04-23T23:11:51.431Z",
  "data": {
    service: "butler",
    serviceVersion: "7.2.1",
    system: {
      id: "3de76798c85894844ac20100cf2142c9a58cc90d7e9dd31a22c94b68048c3ee5",
      arch: "x64",
      platform: "darwin",
      release: "12.3.1",
      distro: "macOS",
      codename: "macOS Monterey",
      virtual: false,
      hypervisor: undefined,
      nodeVersion: "v16.4.0",
    },
    enabledFeatures: {
      api: {
        apiListEnbledEndpoints: true,
        base62ToBase16: true,
        base16ToBase62: true,
        butlerping: true,
        createDir: true,
        createDirQVD: true,
        fileDelete: true,
        fileMove: true,
        fileCopy: true,
        keyValueStore: true,
        mqttPublishMessage: true,
        scheduler: {
          createNewSchedule: true,
          getSchedule: true,
          getScheduleStatusAll: true,
          updateSchedule: true,
          deleteSchedule: true,
          startSchedule: true,
          stopSchedule: true,
        },
        senseAppReload: true,
        senseAppDump: true,
        senseListApps: true,
        senseStartTask: true,
        slackPostMessage: true,
      },
      feature: {
        heartbeat: false,
        dockerHealthCheck: true,
        uptimeMonitor: true,
        uptimeMonitor_storeInInfluxdb: true,
        uptimeMonitor_storeInNewRelic: true,
        teamsNotification: true,
        teamsNotification_reloadTaskFailure: true,
        teamsNotification_reloadTaskAborted: true,
        slackNotification: true,
        slackNotification_reloadTaskFailure: true,
        slackNotification_reloadTaskAborted: true,
        emailNotification: true,
        emailNotification_reloadTaskFailure: true,
        emailNotification_reloadTaskAborted: true,
        webhookNotification: false,
        webhookNotification_reloadTaskFailure: false,
        webhookNotification_reloadTaskAborted: false,
        signl4Notification_reloadTaskFailure: true,
        signl4Notification_reloadTaskAborted: true,
        newRelicNotification_reloadTaskFailure: true,
        newRelicNotification_reloadTaskAborted: true,
        scheduler: true,
        mqtt: true,
        userActivityLogging: false,
      },
    },
  }
}

Creating an anonymous ID field

The id field deserves a bit more explanation.

It’s purpose is to uniquely identify the Butler instance - nothing else.
If Butler is stopped and started agagin the same ID should be generated.

Some sensitive information is used to create the ID, but as the ID is anonymized before sent as part of the telemetry data, no sensitive information leaves your servers.

The ID field is created as follows:

1. Combine the following information to a single string

   1. MAC address of default network interface
   2. IPv4 address of default network interface
   3. IP or FQDN of Sense server where repository service is running
   4. System unique ID as reported by the OS. Not all OSs support this though, which is why field 1-3 above are also needed to get a unique ID.

2. Run the created string through a one-way hashing/message digest function. Butler uses Node.js’ own Crypto library to create a SHA-256 hash, using the default network interface’s MAC address as salt.
   Security is increased due to the fact that the salt never leaves the server where Butler is running.

   The bottom line is that it’s impossible to reverse the process and get your the IP, host name etc used in step 1 above.
   Then again - this is cryptography and things change.
   But if you trust the certificates securing Sense itself, then the ID anonymization used by Butler should be ok too.

3. The result is a string that uniquely identifies the Butler instance at hand, without giving away any sensitive data about the system where Butler is running.

See below for an example of what the id field looks like.
The id field is shown during Butler startup as “Instance ID”.

2023-12-08T13:15:21.936Z info: --------------------------------------
2023-12-08T13:15:21.937Z info: Starting Butler
2023-12-08T13:15:21.937Z info: Log level      : info
2023-12-08T13:15:21.937Z info: App version    : 9.1.1
2023-12-08T13:15:21.937Z info: Instance ID    : 3de76798c85894844ac20100cf2142c9a58cc90d7e9dd31a22c94b68048c3ee5
2023-12-08T13:15:21.937Z info:
2023-12-08T13:15:21.937Z info: Node version   : v18.5.0
2023-12-08T13:15:21.937Z info: Architecture   : x64
2023-12-08T13:15:21.937Z info: Platform       : darwin
2023-12-08T13:15:21.937Z info: Release        : 14.1
2023-12-08T13:15:21.938Z info: Distro         : macOS
2023-12-08T13:15:21.938Z info: Codename       : macOS Sonoma
2023-12-08T13:15:21.938Z info: Virtual        : false
2023-12-08T13:15:21.938Z info: Processors     : 1
2023-12-08T13:15:21.938Z info: Physical cores : 8
2023-12-08T13:15:21.938Z info: Cores          : 16
2023-12-08T13:15:21.938Z info: Docker arch.   : undefined
2023-12-08T13:15:21.938Z info: Total memory   : 68719476736
2023-12-08T13:15:21.938Z info:
2023-12-08T13:15:21.938Z info: Config file    : /Users/goran/tools/butler/production.yaml
2023-12-08T13:15:21.938Z info: API rate limit : 100
2023-12-08T13:15:21.938Z info: --------------------------------------

Telemetry FAQ

1. What data is included in the telemetry messages?
   See above for details.
   In general the telemetry includes information about which Butler API endpoints and features are enabled vs disabled.
   A unique, anonymized ID is included too, it’s unique to each Butler instance and is used soley to distinguish between different Butler instances.
   Finally some information about Butler’s execution environment is included. Things like operating system, Node.js version used etc.

2. Can my Sense environment be identified via telemetry data?
   Short answer: No.
   Longer answer: No information about your Sense environment is sent as part of telemetry. No IP addresses or server names, no IDs of Sense apps/tasks/etc, no information about what actual data passed through Butler’s APIs, or any other data that can be linked to your Sense environment is included in the telemetry data.