Documentation

• 1: About
• 1.1: Butler
• 1.2: Use cases
• 1.2.1: Retired use cases
• 1.3: The Butler family
• 1.4: Versions
• 1.5: Contribution guidelines
• 1.6: Telemetry
• 2: Getting started
• 2.1: Overview
• 2.2: Installing Butler
• 2.2.1: Decide how to run Butler
• 2.2.2: Running Butler as a native, pre-built application
• 2.2.3: Running Butler in Docker
• 2.2.4: Running Butler as a Node.js application
• 2.3: Setup
• 2.3.1: Which config file to use
• 2.3.2: Connecting to a Qlik Sense server
• 2.3.3: Configuring Butler's REST API
• 2.3.4: Reload related alerts
• 2.3.4.1: Reload alerts sent as emails
• 2.3.4.2: Reload alerts in InfluxDB
• 2.3.4.3: Reload alerts via New Relic
• 2.3.4.4: Reload alerts via Slack
• 2.3.4.5: Reload alerts via Microsoft Teams
• 2.3.4.6: Reload alerts via MQTT
• 2.3.4.7: Reload alerts via outgoing webhooks
• 2.3.5: Reload script logs
• 2.3.6: Monitoring Windows services
• 2.3.6.1: Sending Windows service alerts as email
• 2.3.6.2: Sending Windows service alerts to New Relic
• 2.3.6.3: Storing Windows service alerts in InfluxDB
• 2.3.6.4: Sending Windows service alerts to Slack
• 2.3.6.5: Sending Windows service alerts to Microsoft Teams
• 2.3.6.6: Sending Windows service alerts as MQTT messages
• 2.3.6.7: Sending Windows service alerts as outgoing webhooks (=http messages)
• 2.3.7: Qlik Sense licenses
• 2.3.8: Configuring the Butler scheduler
• 2.3.9: Configuring the key-value store
• 2.3.10: Configuring file system access via REST API
• 2.3.11: Incident management tools
• 2.3.11.1: New Relic
• 2.3.11.2: Signl4
• 2.3.12: Setting up MQTT messaging
• 2.3.13: Configuring Butler heartbeats
• 2.3.14: Configuring Butler metrics & monitoring
• 2.3.15: Docker healthcheck
• 2.3.16: Creating Sense data connections
• 2.3.17: Forwarding user activity events to Butler
• 2.3.18: Control which tasks can be started via Butler's API
• 2.3.19: Configuring telemetry
• 2.4: Day 2 operations
• 2.4.1: Standalone app
• 2.4.2: Docker
• 2.4.3: Node.js app
• 2.5: Upgrade
• 3: Concepts
• 3.1: Key-value store
• 3.2: Scheduler
• 3.3: Failed/aborted reloads
• 3.3.1: Alert emails
• 3.3.2: Alerts via Slack and Microsoft Teams
• 3.3.3: Storing script logs of failed reloads to disk
• 3.3.4: InfluxDB
• 3.3.5: New Relic
• 3.3.6: MQTT
• 3.3.7: Signl4
• 3.3.8: Webhooks
• 3.4: Successful reloads
• 3.5: Starting Sense tasks in style
• 3.6: Incident management tools
• 3.6.1: New Relic
• 3.6.2: Signl4
• 3.7: Qlik Sense licenses
• 3.8: File system access: copy/move/delete files
• 3.9: MQTT integration
• 3.10: UDP client
• 3.11: Monitor Windows services
• 3.12: Real-time metrics
• 4: Examples
• 4.1: Swagger UI: Try out the Butler API using the API docs
• 4.2: Flexible scheduling of Sense reload tasks
• 4.3: Reload task chaining with parameters
• 4.4: Monitoring Butler's memory usage using Grafana
• 4.5: Start Sense tasks using Butler APIs
• 4.5.1: Start Sense tasks using REST API
• 4.5.2: Start Sense tasks from load script of Sense apps
• 4.5.3: Start Sense tasks using MQTT
• 4.6: Controlled and secure file system operations
• 4.7: Sense apps that explain and highlight various Butler features
• 4.7.1: Using Butler APIs from Sense load script
• 4.7.2: Partial loads in Qlik Sense
• 4.7.3: Post message to Slack
• 4.7.4: Publish message to MQTT
• 4.8: Run Butler as a Windows service
• 4.9: When things don't work as expected
• 5: Reference docs
• 5.1: Command line options
• 5.2: Config file syntax
• 5.3: REST API documentation
• 5.4: REST API documentation (alternative format)
• 5.5: Scheduler
• 5.6: Key-value store
• 5.7: Alert template fields
• 5.7.1: Alert template fields for failed/aborted reload tasks
• 5.7.2: Alert template fields for Windows services events
• 5.8: Information stored in InfluxDB
• 5.9: Helper subs that make it easy to use Butler APIs
• 5.10: Test cases
• 6: Security considerations
• 7: Legal stuff

1 - About

1.1 - Butler

The Butler project is all about adding useful features to the client-managed version of Qlik Sense, also known as Qlik Sense Enterprise on Windows, QSEoW.
Some of the features can be used from Sense load scripts, other features provide integration with 3rd party systems.
Most of Butler’s features try to make daily life for a Qlik Sense administrator a bit easier.

The goal is to integrate battle-proven concepts and best-of-breed open-source tools into Butler and thus make them available to developers of Sense apps or those responsible for running Sense clusters.
In some cases it might be possible to use these tools from within Sense also without Butler - in those cases Butler simply tries to make things easier, lowering the barriers to get started and get things done.

There is also a clear goal that Butler should be very configurable. In practice this means that individual features can be turned on/off as needed, improving security and lowering memory usage.

Butler is written in Node.js and runs on most modern operating systems.
As of version 7.2 Butler offers pre-built binaries for Windows, macOS and Linux§, meaning that Node.js no longer has to be separately installed to use Butler.
This simplifies things - just download, configure and run.

You can run Butler on the same server as Qlik Sense, in a Docker container on a Linux server, in Kubernetes, on Mac OS, on Raspberry Pi (not a good idea.. but possible and proven to work).

Butler is a member of a group of tools collectively referred to as the “Butler family”, more info is available here.

This picture might be useful to understand what Butler does and how it fits into the larger system map around Qlik Sense:

1.2 - Use cases

• Instant notifications when reload tasks fail or are stopped
• Forward failed reload events to incident management systems (New Relic, Signl4)
• Use InfluxDB/Grafana or New Relic to track Butler memory usage
• Save a copy of the complete reload log for all failed reload tasks
• Start reload tasks from load script or from upstream systems
• Start any reload task from within any Qlik Sense or web app
• Start reload tasks via REST API based on task tags or custom properties
• Trigger full/partial app reloads from load script or upstream systems
• Flexible scheduling of app reloads in Qlik Sense Enterprise on Windows
• Passing parameters between reload tasks
• Storing state across several apps
  • Time-to-live (TTL) for key-value pairs
• Make new data reach end users as quickly as possible
• Using MQTT to notify downstream systems that Sense is done processing data
• Create directories, copy/move/delete files
• Extract metadata for apps
• Easily post messages to Slack
• Monitor Windows services
• Monitor and release Qlik Sense user licenses

Instant notifications when reload tasks fail or are stopped

Information about failing tasks can be sent as emails, to Microsoft Teams, Slack, as MQTT messages or outgoing webhooks.

Email, Slack and MS Teams notifications all use a templating concept where HTML/Markdown template files describe what the alert message should look like. Before the alert is sent the template is populated with actual data from the failed reload task.

Both subject and body of email can use the template fields.

For both Slack and Teams there are options to use more flexible/configurable alert formats and more basic pre-configured alerts.

The result is a very poweful tool for QSEoW sysadmins, who get real-time insight into what’s happening with respect to task execution.

More info here.

Forward failed reload events to incident management systems (New Relic, Signl4)

Butler offers advanced failed reload alerts via Slack, Teams, email and outgoing webhooks.
Configurable templates means you can customize emails/Teams/Slack messages.

Sometimes you want a bit more structure though.
This is especially true when Sense is used in the enterprise.

Butler integrates with both Signl4 and New Relic.
Both offer incident management features on both the web and via mobile clients.

Information about failed/aborted reloads can be sent to one or more New Relic accounts.
Tags for the reload task and associated app is sent to New Relic as metadata for the event/log entry that’s created there.

Use InfluxDB/Grafana or New Relic to track Butler memory usage

Butler can be configured to log its own memory usage to InfluxDB, from where it can be visualised using Grafana.

If you prefer using New Relic One that’s possible too - sending Butler memory metrics to New Relic is super simple: Just add your New Relic credentials in the YAML config file or as command line options when starting Butler and you’re set.

Save a copy of the complete reload log for all failed reload tasks

Let’s say a scheduled reload task fails.

This can happen due to lots of reasons, from uncontrollable events that are impossible to predict to bugs in the script of a Sense app.

No matter what the cause is, as a Sense administrator you probably want to investigate the script reload logs.

Butler can send notifications (Slack, Teams, email, webhooks, …) when reloads fail.
These notifications can include the last 20-30-40 lines of the script log and this usually gives a good idea of what caused the reload to fail.

But what if you want to look at the complete reload log of that failed app reload?

So far you would have to dig into the log directory on the Sense server, find that specific reload log among potentially thousands of other log files. Not very effective.

As of version 7.2 Butler can store a copy of the complete reload log in a directory that you specify.
The log files are stored in separate directories, one for each date.
This makes it easy to find the log file you are interested in.

Start reload tasks from load script or from upstream systems

Trigger Sense reload tasks from a reload script: This makes it possible to start different Sense tasks based on what data has been read from a database, what time of day it is etc.
Starting a task from the reload script is as easy as Call StartTask('fbf645f0-0c92-40a4-af9a-6e3eb1d3c35c').

Trigger Sense reloads from external systems: When new data is available in a source database, that database can trigger a reload in Sense, and the data is loaded from the database into Sense. This way delays caused by Sense polling for data are minimized and data arrives at end users as quickly as possible.

Starting reload tasks using REST API is described here.
Using MQTT messages to achieve this is described here.

Start any reload task from within any Qlik Sense or web app

Some HTML and Javascript magic is also needed, but given Butler’s start-task API it’s pretty easy to set up a button in a Sense app (or any web app!) to start any Sense reload task.

This can for example be used to allow end users to start an Extract-Transform when they (the user) need refreshed data.

More info here.

Start reload tasks via REST API based on task tags or custom properties

Using tags and/or custom properties to identify what tasks should be started can be easier than having to know the tasks IDs. This both makes it easier for 3rd party systems to start Qlik Sense tasks and easier for Sense admins to manage which tasks should be startable by 3rd party systems.

More info in Concepts and Examples sections.

Trigger full/partial app reloads from load script or upstream systems

Sometimes you just want to reload an app without also having to create a reload task.
When it comes to partial app reloads it’s not even possible to do these from a Sense reload task.

Butler’s API makes prvovides a solution: Just pass in an app ID to reload together with task IDs of the tasks that should be started when the app is done reloading (different tasks can be started depending on app reload success or failure).

The partial reload feature is of special interest as it can be used to trigger faster incremental execution of of Extract-Transform reload chains. Great for keeping data in Sense apps updated during the course of a day!

More info here.

Flexible scheduling of app reloads in Qlik Sense Enterprise on Windows

Using the scheduler built into Qlik Sense you can’t for example create schedules that are limited to a parts of a day.
This is a pretty common scenario though - you want to reload an app hourly from say 3 am to 3 pm.

You can set this up in Sense, but it involves creating a lot of triggers for the reload task, which becomes a nightmare to maintain.

Butler’s task scheduler is built on Cron, which has been used in Linux for decades. Battle proven and very flexible.

Passing parameters between reload tasks

This has always been hard both in QlikView and Sense.

Butler’s key-value store makes it much easier to pass values from one app to the next in a reload chain.

Storing state across several apps

The key-value store can also be used to keep state in general across several apps or parts of a Qlik Sense environment.

Maybe a Development cluster needs to share information in real time with the Test and Production clusters?
Easily solved using Butler’s key-value store.

A demo showing parameter passing between apps is found here.

Time-to-live (TTL) for key-value pairs

Key-value pairs can optionally be set up with a time-to-live (ttl) parameter. If ttl is set, the KV pair will auto-delete when the ttl expires.

The key-value store is described here.

Make new data reach end users as quickly as possible

See above. Have the upstream data source initiate Sense app reloads, either via Butler’s REST API or via MQTT messages sent to Butler.

Using MQTT to notify downstream systems that Sense is done processing data

Use Butler’s API endpoints for MQTT handling to send and receive MQTT publish-subscribe messages.
MQTT (and the pubsub concept in general) is a great way for systems to communicate reliably with each other.

A demo app is available, showing how MQTT messages can be sent from Sense load scripts. More info here.

Create directories, copy/move/delete files

In “standard mode” apps reloading in Qlik Sense Enterprise on Windows can’t access the file system of the Sense servers. This is a good thing because it adds a lot of security.

From time to time you need to delete temp QVDs though, or copy or move data files from one directory to another.

Butler has REST API endpoints for these use cases, but as those endpoints are locked down to only work on specific, configurable directiories they don’t result in the same security issues as seen in for example QlikView or Sense running in legacy mode.

More info here.

Extract metadata for apps

Exporting apps as JSON can be very useful for backup purposes. Doing regular snapshots of all apps in a Sense cluster is a fast and space-effective way of keeping point-in-time backups.

The REST API documentation has full docs for the /v4/app/{appId}/dump endpoint.

Easily post messages to Slack

Slack messages can include full formatting (web links, text formatting etc), as well as “poking” users.
I.e. notifying specific Slack users that they have a new message.

Can for example be used to notify user(s) that an app has reloaded with new data, or that some error condition has occured.

More info here.

Monitor Windows services

If Butler is running on Windows (server or even desktop) it can monitor one or more Windows services.
This feature is not available when running Butler on Linux, macOS or in Docker.

Monitoring here means tracking the services’ state and sending messages to email, InfluxDB, New Relic, MQTT, Webhooks, Slack or Teams when services stop or start.

It can for example be used to get alerts if a Qlik Sense service for some reason stops.
The concept is not limited to Qlik Sense services though - any Windows service can be monitored.

Monitor and release Qlik Sense user licenses

Butler can monitor the usage of Qlik Sense user licenses and store the data in InfluxDB, from where the license data can be visualized in Grafana. This makes it easy to track (and alert if needed) on the number of used licenses, how many are available and when it’s time to get more licenses.

Butler can also automatically release Professional and Analyzer user licenses that have been inactive for a certain period of time. This is useful in environments where some users use Sense sporadically, for example only during certain times of the year. In such cases it’s a waste of resources to keep the license assigned to the user when it’s not being used.

More info here.

1.2.1 - Retired use cases

Some features age with grace, others don’t.
Here’s a list of features that are candidates for removal from Butler, or that have already been removed.

Candidates for removal in coming versions

Real-time metrics around active users

While a good idea in theory, Butler just wasn’t the vehicle for this.

The way Butler approached this was to have Sense’s log4net logging framework send UDP messages to Butler when users logged in/out or sessions started/ended. This certainly works (quite well in fact!), but it also has inherent issues.
For example, when Butler was started it wouldn’t capture currently active users or sessions - it was only after some event captured in the logs that Butler would update it’s internal counters. This meant that it would take some time (sometimes quite long) until the metrics were even approaching the real number of users using Sense.

There was also the risk of Butler missing UDP messages and not registering the associated log event.

The affected API endpoints are:

/v4/activeusercount
/v4/activeusers

1.3 - The Butler family

Butler started out with a very specific need to start Sense reloads from outside systems.
Over the years a few projects (for example Butler SOS, which simplifies day 2 operations ([1], [2]) have spun off from the original Butler project, and still other projects have been created from scratch to solve specific challenges around developing Sense apps and running Qlik Sense server environments.

All members of the Butler family are available on Ptarmigan Labs’ GitHub page.

Projects with production grade release status are (as of this writing):

Butler

The original Butler. Offers various utilities that make it easier to develop Sense apps, as well as simplifying day 2 operations.

butler.ptarmiganlabs.com. (This site!)

Butler SOS

Real-time operational metrics for Qlik Sense. A must-have if you are responsible for a Sense environment with more than a dozen or so users.

Butler SOS makes it possible to detect and alert on issues as they happen, rather than in retrospect much later.

Several storage and visualisation options available, including InfluxDB + Grafana, and New Relic.

butler-sos.ptarmiganlabs.com

Butler Sheet Icons

Automates the creation of sheet icons for both Qlik Sense Cloud and client-managed Qlik Sense Enterprise on Windows (QSEoW) applications.

It’s a cross platform command line tool which given the correct Sense credentials will take screen shots of all sheets in a Sense app (or all apps on a Sense server!), then create thumbnail versions of those screenshots.
Finally those thumbnails will be set as sheet icons.

No more manual screenshot taking, resizing images, navigating hundreds of sheets in dozens of apps.
Start Butler Sheet Icons instead and go get a nice fika.

The tool can be used stand-along or as part of an automated release process.

https://github.com/ptarmiganlabs/butler-sheet-icons

Ctrl-Q

Given the name of this tool it doesn’t sound like a member of the Butler family.
Let’s say Ctrl-Q is a sibling of the Butler bunch.

While the Butler tools are (usually) intended to solve and simplify rather specific use cases, Ctrl-Q is aimed at being the lazy Qlik developer’s best friend.

Let’s say there is some manual, tedious, time consuming and error prone activity that a Qlik Sense developer is faced with.
For example importing dozens of apps from QVF files and creating a hundred associated reload tasks.
Ctrl-Q lets you do this with a single command, using definitions in an Excel file. Instead of spending a day on this the actual execution takes a minute or so.

In other words: Ctrl-Q focus on high-value use cases that are difficult or impossible to solve using other tools.

github.com/ptarmiganlabs/ctrl-q

Butler CW

Butler Cache Warmer. Cache warming is the process of proactively forcing Sense apps to be loaded into RAM, so they are readily available when users open them.
Using Butler CW is an easy way to make your end users’ experience of Sense a little better.

github.com/ptarmiganlabs/butler-cw

Butler App Duplicator

No matter if you are a single developer creating Sense apps, or have lots of developers doing this, having app templates is a good idea:

• Lowered barrier of entry for new Sense developers.
• Productivity boost when developing Sense apps.
• Encouraging a common coding standard across all apps.

github.com/ptarmiganlabs/butler-app-duplicator

Butler Spyglass

This tool is mainly of interest if you have lots of QVDs and apps, but when that’s the case it’s of paramount importance to understand what apps use which QVDs. In other words what data lineage looks like.

Butler Spyglass also extracts full load scripts for all Sense apps, creating a historical record of all load scripts for all Sense apps.

github.com/ptarmiganlabs/butler-spyglass

Butler Notifier

This tool makes it easy to tap into the Qlik Sense notification API. From there you can get all kinds of notifications, including task reload failures and changes in session state (user login/logout etc).

github.com/ptarmiganlabs/butler-notifier

Butler Icon Uploader

Visual looks is important when it comes to analytics, and this holds true also for Sense apps.

The Butler Icon Uploader makes it easy to upload icon libraries (for example Font Awesome) to Qlik Sense Enterprise. With such icons available it is then easy for app developers to use professional quality sheet and app icons in their Sense apps.

github.com/ptarmiganlabs/butler-icon-upload

1.4 - Versions

Documentation updates

Starting with Butler version 4.0, this documentation site will offer both latest and earlier site versions.

Select which doc site to view in drop-down list in the upper right corner of the page.

1.5 - Contribution guidelines

Butler is developed in Node.js, with support from various NPM modules.

We use Hugo to format and generate this documentation site, the Docsy theme for styling and site structure.
Hugo is an open-source static site generator that provides us with templates, content organisation in a standard directory structure, and a website generation engine. You write the pages in Markdown (or HTML if you want), and Hugo wraps them up into a website.

All submissions, including submissions by project members, require review. We use GitHub pull requests for this purpose. Consult GitHub Help for more information on using pull requests.

Creating an issue

If you’ve found a problem - or have a feature suggestion - with Butler itself or the documentation, but you’re not sure how to fix it yourself, please create an issue in the Butler repo. You can also create an issue about a specific doc page by clicking the Create Issue button in the top right hand corner of the page.

1.6 - Telemetry

What’s telemetry

From Wikipedia:

Telemetry is the in situ collection of measurements or other data at remote points and their automatic transmission to receiving equipment (telecommunication) for monitoring.

In the context of software tools (including Butler) telemetry is often used to describe the process of sending information about the tool itself to some monitoring system.

Why telemetry in Butler

This is a very, very good question.

For many years there was no telemetry at all in Butler.

Development of new features were driven mainly by what features were needed at the time.
Or the fact that Qlik released some new feature in Sense and Butler was a way to test that new feature from the perspective of the Sense APIs.

That’s all good but today Butler is a rather significant tool with features spanning quite different areas (alerting, task scheduling, key-value store and more).

This multitude of features is also one of the core reasons for adding telemetry to Butler:

Without this information the Butler developers will keep working in the dark, not really knowing where to focus their efforts.

On the other hand - with access to telemetry data a lot of possibilities open up for the Butler developers:

• If telemetry shows that no one uses a particular feature, maybe that feature should be scheduled for deprecation?
• The opposite of the previous: If lots of users use a specific Butler feature, then that feature is a candidate for future focus and development.
• Telemetry will show if lots of users run Butler on old Node.js versions. Knowing this its possible to set a migration schedule for what Node.js versions are supported - avoiding hard errors when some old Node.js version is no longer supported by Butler.
• Same thing for understanding what operating systems Butler runs (and should be supported) on.

Configuring Butler’s telemetry

Instructions here.

The details

Where is telemetry data sent

The telemetry data is sent to the PostHog service, using their database in the European Union.

Deleting telemetry data

Even though no-one (not even the Butler developers or Ptarmigan Labs who manage the telemetry database!) has any way of ever connecting the data sent by your Butler instance to you (it’s all anonymized, remember?), there can be cases where telemetry data must be deleted.

The legal page has more information about this.

Field level description of telemetry data

A telemetry message from Butler contains the information below.

{
  "ts": "2021-04-23T23:11:51.431Z",
  "data": {
    service: "butler",
    serviceVersion: "7.2.1",
    system: {
      id: "3de76798c85894844ac20100cf2142c9a58cc90d7e9dd31a22c94b68048c3ee5",
      arch: "x64",
      platform: "darwin",
      release: "12.3.1",
      distro: "macOS",
      codename: "macOS Monterey",
      virtual: false,
      hypervisor: undefined,
      nodeVersion: "v16.4.0",
    },
    enabledFeatures: {
      api: {
        apiListEnbledEndpoints: true,
        base62ToBase16: true,
        base16ToBase62: true,
        butlerping: true,
        createDir: true,
        createDirQVD: true,
        fileDelete: true,
        fileMove: true,
        fileCopy: true,
        keyValueStore: true,
        mqttPublishMessage: true,
        scheduler: {
          createNewSchedule: true,
          getSchedule: true,
          getScheduleStatusAll: true,
          updateSchedule: true,
          deleteSchedule: true,
          startSchedule: true,
          stopSchedule: true,
        },
        senseAppReload: true,
        senseAppDump: true,
        senseListApps: true,
        senseStartTask: true,
        slackPostMessage: true,
      },
      feature: {
        heartbeat: false,
        dockerHealthCheck: true,
        uptimeMonitor: true,
        uptimeMonitor_storeInInfluxdb: true,
        uptimeMonitor_storeInNewRelic: true,
        teamsNotification: true,
        teamsNotification_reloadTaskFailure: true,
        teamsNotification_reloadTaskAborted: true,
        slackNotification: true,
        slackNotification_reloadTaskFailure: true,
        slackNotification_reloadTaskAborted: true,
        emailNotification: true,
        emailNotification_reloadTaskFailure: true,
        emailNotification_reloadTaskAborted: true,
        webhookNotification: false,
        webhookNotification_reloadTaskFailure: false,
        webhookNotification_reloadTaskAborted: false,
        signl4Notification_reloadTaskFailure: true,
        signl4Notification_reloadTaskAborted: true,
        newRelicNotification_reloadTaskFailure: true,
        newRelicNotification_reloadTaskAborted: true,
        scheduler: true,
        mqtt: true,
        userActivityLogging: false,
      },
    },
  }
}

Creating an anonymous ID field

The id field deserves a bit more explanation.

It’s purpose is to uniquely identify the Butler instance - nothing else.
If Butler is stopped and started agagin the same ID should be generated.

Some sensitive information is used to create the ID, but as the ID is anonymized before sent as part of the telemetry data, no sensitive information leaves your servers.

The ID field is created as follows:

1. Combine the following information to a single string

   1. MAC address of default network interface
   2. IPv4 address of default network interface
   3. IP or FQDN of Sense server where repository service is running
   4. System unique ID as reported by the OS. Not all OSs support this though, which is why field 1-3 above are also needed to get a unique ID.

2. Run the created string through a one-way hashing/message digest function. Butler uses Node.js’ own Crypto library to create a SHA-256 hash, using the default network interface’s MAC address as salt.
   Security is increased due to the fact that the salt never leaves the server where Butler is running.

   The bottom line is that it’s impossible to reverse the process and get your the IP, host name etc used in step 1 above.
   Then again - this is cryptography and things change.
   But if you trust the certificates securing Sense itself, then the ID anonymization used by Butler should be ok too.

3. The result is a string that uniquely identifies the Butler instance at hand, without giving away any sensitive data about the system where Butler is running.

See below for an example of what the id field looks like.
The id field is shown during Butler startup as “Instance ID”.

2023-12-08T13:15:21.936Z info: --------------------------------------
2023-12-08T13:15:21.937Z info: Starting Butler
2023-12-08T13:15:21.937Z info: Log level      : info
2023-12-08T13:15:21.937Z info: App version    : 9.1.1
2023-12-08T13:15:21.937Z info: Instance ID    : 3de76798c85894844ac20100cf2142c9a58cc90d7e9dd31a22c94b68048c3ee5
2023-12-08T13:15:21.937Z info:
2023-12-08T13:15:21.937Z info: Node version   : v18.5.0
2023-12-08T13:15:21.937Z info: Architecture   : x64
2023-12-08T13:15:21.937Z info: Platform       : darwin
2023-12-08T13:15:21.937Z info: Release        : 14.1
2023-12-08T13:15:21.938Z info: Distro         : macOS
2023-12-08T13:15:21.938Z info: Codename       : macOS Sonoma
2023-12-08T13:15:21.938Z info: Virtual        : false
2023-12-08T13:15:21.938Z info: Processors     : 1
2023-12-08T13:15:21.938Z info: Physical cores : 8
2023-12-08T13:15:21.938Z info: Cores          : 16
2023-12-08T13:15:21.938Z info: Docker arch.   : undefined
2023-12-08T13:15:21.938Z info: Total memory   : 68719476736
2023-12-08T13:15:21.938Z info:
2023-12-08T13:15:21.938Z info: Config file    : /Users/goran/tools/butler/production.yaml
2023-12-08T13:15:21.938Z info: API rate limit : 100
2023-12-08T13:15:21.938Z info: --------------------------------------

Telemetry FAQ

1. What data is included in the telemetry messages?
   See above for details.
   In general the telemetry includes information about which Butler API endpoints and features are enabled vs disabled.
   A unique, anonymized ID is included too, it’s unique to each Butler instance and is used soley to distinguish between different Butler instances.
   Finally some information about Butler’s execution environment is included. Things like operating system, Node.js version used etc.

2. Can my Sense environment be identified via telemetry data?
   Short answer: No.
   Longer answer: No information about your Sense environment is sent as part of telemetry. No IP addresses or server names, no IDs of Sense apps/tasks/etc, no information about what actual data passed through Butler’s APIs, or any other data that can be linked to your Sense environment is included in the telemetry data.

2 - Getting started

2.1 - Overview

Getting started: 1-2-3

The main components of Butler are outlined in the system diagram above.

1. Installation

Follow the installation instructions - they will guide through the setup process, including requirements and customisation.

2. Setup

Once everything is installed you need to edit the configuration file to suit your specific needs.

3. Try it out!

Feel free to browse through the concepts and examples to get an understanding of how to use Butler.

2.2 - Installing Butler

Getting started

Sorry - there is no installer for Butler.

The pre-built binaries for Windows, macOS, Linux and Docker simply work as-is when combined with a properly set up configuration file.

If you still want to run Butler as Node.js app you will first need to install Node.js.

The instructions on the pages below should provide good guidance, if you still run into troubles you can always reach out via the GitHub discussion forums.

What’s required to use Butler

• A Butler config file adapted to your specific Qlik Sense environment
• Certificates exported from Qlik Sense Enterprise.

2.2.1 - Decide how to run Butler

The short answer is: Almost anywhere.

The pre-built binaries for Windows, macOS, Linux and Docker should cater for most use cases.

If you have some other, more exotic platform or operating system you want to run Butler on that’s probably possible too.
Butler is built on Node.js and as long as Node.js is available on the platform/operating system of your choice there is a good chance Butler will run there.

Butler has been successfully used on Windows Server, Windows 10, various Linux distributions, in Docker, Kubernetes, on Mac OS and even on Raspberry Pis. And a Raspberry Pi based Kubernetes cluster.

Your platform options thus typically fall into three categories:

Butler as a stand-alone executable

Here you will be using the pre-built Butler binaries (Windows, Linux, Mac OS) that are available for Butler 7.2 and later.

When using third party tools these binaries can be started as services.
For example, on Windows the free NSSM tool is a great way to run Butler as a Windows service.
Another good tool is PM2 which works well on Linux-ish platforms.

The Butler stand-alone executables are available on the GitHub releases page.

Butler in a container: Docker and Kubernetes

If you have access to or can set up a container runtime environment, this is a great way to running Butler.

Installation is less error prone compared to installing Butler as a native Node.js app, you get all the benefits from the Docker ecosystem (monitoring of running containers etc), and upgrades to future Butler versions become trivial.

If you have access to a Kubernetes cluster, that is usually an even better option than Docker. Kubernetes can be daunting when first approached, but will give you superb reliability, failover and restarts if a server goes down or becomes unresponsive etc. All major cloud providers (Microsoft Azure, Google, Amazon etc) offer Kubernetes services.

Rancher’s K3s is a very good way to get started with self hosted Kubnernetes. Fully featured, well supported and a vibrant developer community.

Butler as a Node.js application

This option means you will first install Node.js on your server of choice, then Butler and it’s dependencies.

It works perfectly well but is the most demanding when it comes to amount of work needed to get started.

2.2.2 - Running Butler as a native, pre-built application

Downloading the app

Download Butler for your preferred operating systym.

Latest version is available on GitHub.

Installation steps

Installing Butler is quite simple.
The steps below outline the process.

Additional information is found on the Day 2 operations page.

• Decide where to install Butler
  It is usually a good starting point to run Butler on the Sense server. If there are more than one server in the Sense cluster, Butler can be placed on the reload server (as the /createDir endpoint then can be used to create folders in which QVD and other files can be stored).

  On the other hand, you might want to keep the Sense servers as clean as possible (with respect to software running on them). If that is a priority you should install Butler on some other server.

  The bottom line is that Butler can run on any server, as long as there is network connectivity to the Sense server(s).

  It’s usually a good idea to keep 3rd party tools installed in the same directory tree, to maintenance as easy as possible.
  A good place for Butler could be c:\tools\butler or d:\tools\butler on Windows, for example.

• Download Butler
  Download the latest version from the releases page.
  Make sure to get the binary file for your preferred operating system.

  Unzip the downloaded file, then copy or move the butler binary to the desired directory (e.g. c:\tools\butler) and that’s it.

2.2.3 - Running Butler in Docker

Installation steps

The following steps give some guidance on how to get Butler running on Docker.
Here Mac OS was used, things will look different on Linux and Windows.

Note: While the console logs below refer to an older version of Butler’s Docker image, the steps involved are the same also for current/most recent version of Butler.

proton:~ goran$ mkdir /Users/goran/butler
proton:~ goran$ cd /Users/goran/butler
proton:butler goran$ mkdir -p config/certificate
proton:butler goran$
proton:butler goran$ wget https://raw.githubusercontent.com/ptarmiganlabs/butler/master/src/docker-compose.yaml
--2021-10-25 17:07:23--  https://raw.githubusercontent.com/ptarmiganlabs/butler/master/src/docker-compose.yaml
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 660 [text/plain]
Saving to: ‘docker-compose.yaml’

docker-compose.yaml 100%[=====================================================================================================================================>]     660  --.-KB/s    in 0s

2021-10-25 17:07:23 (42.0 MB/s) - ‘docker-compose.yaml’ saved [660/660]


proton:butler goran$ cat docker-compose.yaml
# docker-compose.yml
version: '3.3'
services:
  butler:
    image: ptarmiganlabs/butler:6.1.0
    container_name: butler
    restart: always
    ports:
      - "8080:8080"       # REST API available on port 8180 to services outside the container
      - "9998:9998/udp"   # UDP port for task failure events
    volumes:
      # Make config file accessible outside of container
      - "./config:/nodeapp/config"
      - "./log:/nodeapp/log"
    environment:
      - "NODE_ENV=production"
    logging:
      driver: json-file
      options:
        max-file: "5"
        max-size: "5m"
proton:butler goran$

At this point you should

1. Export certificates from the Qlik Sense QMC. Export a full set of certificates in PEM format, no psasword on the certificates.
2. Copy the certificates to the ./config/certificate directory.
3. Copy the template config file from the GitHub repository to the ./config directory, modify it as needed based on your system(s) and which Butler features you want enabled, and rename it to for example production.yaml.
   You can name the config file anything, but its name has to match the NODE_ENV environment variable, as set it the docker-compose.yaml file.
4. Optional. Copy the template schedule file to the location specified in Butler’s config file. This is only needed if you manually want to add schedules. If using the API to create schedules, there is no need to first manually create a schedules file (the schedule file will be created by Butler in this case).

When done, you should see something like this:

proton:butler goran$ pwd
/Users/goran/butler
proton:butler goran$ ls -la
total 8
drwxr-xr-x   4 goran  staff   128 Sep 26 16:36 .
drwxr-xr-x+ 59 goran  staff  1888 Sep 26 16:24 ..
drwxr-xr-x   4 goran  staff   128 Sep 26 16:36 config
-rw-r--r--   1 goran  staff   565 Sep 26 16:25 docker-compose.yml
proton:butler goran$
proton:butler goran$ ls -la config/
total 8
drwxr-xr-x  4 goran  staff   128 Sep 26 16:36 .
drwxr-xr-x  4 goran  staff   128 Sep 26 16:36 ..
drwxr-xr-x  6 goran  staff   192 Sep 26 16:36 certificate
-rw-r--r--  1 goran  staff  1861 Sep 26 16:36 production.yaml
proton:butler goran$
proton:butler goran$ ls -la config/certificate/
total 32
drwxr-xr-x  6 goran  staff   192 Sep 26 16:36 .
drwxr-xr-x  4 goran  staff   128 Sep 26 16:36 ..
-rw-r--r--@ 1 goran  staff  1166 Sep 26 16:36 client.pem
-rw-r--r--@ 1 goran  staff  1702 Sep 26 16:36 client_key.pem
-rw-r--r--@ 1 goran  staff  1192 Sep 26 16:36 root.pem
proton:butler goran$

At this point everything is ready and you can start the Butler container using docker-compose:

proton:butler goran$ docker-compose up
Creating network "butler_default" with the default driver
Pulling butler (ptarmiganlabs/butler:6.1.0)...
6.1.0: Pulling from ptarmiganlabs/butler
7d63c13d9b9b: Already exists
bb262aff53d8: Already exists
24467fa1084c: Already exists
d318401bbcfd: Already exists
fef5c41ac380: Already exists
da4caec0e1fa: Pull complete
d69466c67eaa: Pull complete
ad6e84e85ade: Pull complete
56b17f947d30: Pull complete
9aa9ea345c5a: Pull complete
Digest: sha256:046989e7d440b1fde2db6abfb2cc5eab740b82559ef392c32287ba188bae6235
Status: Downloaded newer image for ptarmiganlabs/butler:6.1.0
Creating butler ... done
Attaching to butler
butler    | 2021-10-25T16:35:31.739Z info: Adding normalized fileCopy directories {
butler    |   "fromDir": "/Users/goran/butler-test-dir1",
butler    |   "toDir": "/Users/goran/butler-test-dir2"
butler    | }
butler    | 2021-10-25T16:35:31.739Z info: Adding normalized fileCopy directories {
butler    |   "fromDir": "/Users/goran/butler-test-dir2",
butler    |   "toDir": "/Users/goran/butler-test-dir1"
butler    | }
butler    | 2021-10-25T16:35:31.740Z info: Adding normalized fileCopy directories {
butler    |   "fromDir": "/Users/goran/butler-test-dir1/abc",
butler    |   "toDir": "/Users/goran/butler-test-dir1"
butler    | }
butler    | 2021-10-25T16:35:31.741Z info: Adding normalized fileCopy directories {
butler    |   "fromDir": "/from/some/directory2",
butler    |   "toDir": "/to/some/directory2"
butler    | }
butler    | 2021-10-25T16:35:31.742Z info: Adding normalized fileMove directories {
butler    |   "fromDir": "/Users/goran/butler-test-dir1",
butler    |   "toDir": "/Users/goran/butler-test-dir2"
butler    | }
butler    | 2021-10-25T16:35:31.743Z info: Adding normalized fileMove directories {
butler    |   "fromDir": "/Users/goran/butler-test-dir2",
butler    |   "toDir": "/Users/goran/butler-test-dir1"
butler    | }
butler    | 2021-10-25T16:35:31.744Z info: Adding normalized fileMove directories {
butler    |   "fromDir": "/Users/goran/butler-test-dir1/abc",
butler    |   "toDir": "/Users/goran/butler-test-dir1"
butler    | }
butler    | 2021-10-25T16:35:31.745Z info: Adding normalized fileMove directories {
butler    |   "fromDir": "/Users/goran/butler-test-dir2/abc-dest",
butler    |   "toDir": "/Users/goran/butler-test-dir1"
butler    | }
butler    | 2021-10-25T16:35:31.745Z info: Adding normalized fileDelete directory /Users/goran/butler-test-dir1
butler    | 2021-10-25T16:35:31.746Z info: Adding normalized fileDelete directory /Users/goran/butler-test-dir1
butler    | 2021-10-25T16:35:31.747Z info: Adding normalized fileDelete directory /Users/goran/butler-test-dir2/abc-dest
butler    | 2021-10-25T16:35:31.747Z info: Enabled API endpoints: [
butler    |   "activeUserCount",
butler    |   "activeUsers",
butler    |   "apiListEnbledEndpoints",
butler    |   "base62ToBase16",
butler    |   "base16ToBase62",
butler    |   "butlerping",
butler    |   "createDir",
butler    |   "createDirQVD",
butler    |   "fileDelete",
butler    |   "fileMove",
butler    |   "fileCopy",
butler    |   "keyValueStore",
butler    |   "mqttPublishMessage",
butler    |   "createNewSchedule",
butler    |   "getSchedule",
butler    |   "getScheduleStatusAll",
butler    |   "updateSchedule",
butler    |   "deleteSchedule",
butler    |   "startSchedule",
butler    |   "stopSchedule",
butler    |   "senseAppReload",
butler    |   "senseAppDump",
butler    |   "senseListApps",
butler    |   "senseStartTask",
butler    |   "slackPostMessage",
butler    |   "getBookmarkList",
butler    |   "applyBookmark",
butler    |   "getSessions",
butler    |   "deleteSession"
butler    | ]
butler    | 2021-10-25T19:06:41.265Z info: CONFIG: Influxdb enabled: false
butler    | 2021-10-25T19:06:41.265Z info: CONFIG: Influxdb host IP: 192.168.100.20
butler    | 2021-10-25T19:06:41.265Z info: CONFIG: Influxdb host port: 8086
butler    | 2021-10-25T19:06:41.265Z info: CONFIG: Influxdb db name: butler
butler    | 2021-10-25T19:06:41.567Z info: --------------------------------------
butler    | 2021-10-25T19:06:41.567Z info: Starting Butler
butler    | 2021-10-25T19:06:41.568Z info: Log level      : verbose
butler    | 2021-10-25T19:06:41.568Z info: App version    : 6.1.0
butler    | 2021-10-25T19:06:41.568Z info: Instance ID    : b6292735c80987393c5cf1a5c685e8548b46e6385b940789e2599936e20d5080
butler    | 2021-10-25T19:06:41.568Z info:
butler    | 2021-10-25T19:06:41.569Z info: Node version   : v16.11.1
butler    | 2021-10-25T19:06:41.569Z info: Architecture   : x64
butler    | 2021-10-25T19:06:41.569Z info: Platform       : linux
butler    | 2021-10-25T19:06:41.569Z info: Release        : 11
butler    | 2021-10-25T19:06:41.570Z info: Distro         : Debian GNU/Linux
butler    | 2021-10-25T19:06:41.570Z info: Codename       : bullseye
butler    | 2021-10-25T19:06:41.570Z info: Virtual        : false
butler    | 2021-10-25T19:06:41.570Z info: Processors     : 4
butler    | 2021-10-25T19:06:41.570Z info: Physical cores : 4
butler    | 2021-10-25T19:06:41.571Z info: Cores          : 4
butler    | 2021-10-25T19:06:41.571Z info: Docker arch.   : undefined
butler    | 2021-10-25T19:06:41.571Z info: Total memory   : 6233116672
butler    | 2021-10-25T19:06:41.571Z info: --------------------------------------
butler    | 2021-10-25T19:06:41.571Z info: Client cert: /nodeapp/config/certificate/client.pem
butler    | 2021-10-25T19:06:41.571Z info: Client cert key: /nodeapp/config/certificate/client_key.pem
butler    | 2021-10-25T19:06:41.572Z info: CA cert: /nodeapp/config/certificate/root.pem
butler    | 2021-10-25T19:06:41.584Z info: MAIN: Didn't load schedules from file
butler    | 2021-10-25T19:06:41.627Z info: MAIN: REST server listening on http://0.0.0.0:8080
butler    | 2021-10-25T19:06:41.633Z info: MAIN: Started Docker healthcheck server on port 12398.
butler    | 2021-10-25T19:06:46.029Z info: /v4/senselistapps called from 192.168.176.1

What you see on your screen will depend on which Butler version you are using and what features are enabled.

Let’s make sure things are working by opening a new terminal window and from there requesting a list of all apps on the server:

proton:~ goran$
proton:~ goran$ curl "http://localhost:8080/v4/senselistapps"
[{"id":"492a1bca-1c41-4a01-9104-543a2334c465","name":"2018 sales targets"},
{"id":"5b243cb2-8d00-44c9-b865-08b00a0af18b","name":"App 1"},
...
...
{"id":"181d101f-986c-49c5-a457-d351058c05b4","name":"Template app 1 DEV"}]
proton:~ goran$

Nice, looking good.

In the terminal where you ran docker-compose, you will see a new line saying that a app list was retrieved:

butler    | 2021-10-25T19:20:50.356Z info: /v4/senselistapps called from 192.168.176.1

2.2.4 - Running Butler as a Node.js application

Selecting an OS

While Qlik Sense Enterprise is a Windows only system, Butler should be able to run on any OS where Node.js is available.
Butler has been succesfully used - during development and production - on Windows, Linux (Debian and Ubuntu tested) and mac OS.

Installation steps

The steps below outline the steps needed to install Butler as a native Node.js application on for example Windows Server.

Additional information is found on the Day 2 operations page.

• Install node.js
  Butler has been developed and tested using the 64 bit version of Node.js. The most recent LTS (Long Term Support) version is usually a good choice.

• Decide where to install Butler
  It is usually a good starting point to run Butler on the Sense server. If there are more than one server in the Sense cluster, Butler can be placed on the reload server (as the /createDir endpoint then can be used to create folders in which QVD and other files can be stored).

  On the other hand, you might want to keep the Sense servers as clean as possible (with respect to software running on them). If that is a priority you should install Butler on some other server.

  The bottom line is that Butler can run on any server, as long as there is network connectivity to the Sense server(s).

• Download Butler
  Download the repository zip from the releases page.

  Do not just clone the Butler repository as that will give you the latest development version, which may not yet be fully tested and packaged.
  The exception is of course if you want to contribute to Butler development - then forking and cloning the repository is the right thing to do.

• Install node dependencies
  From a Windows command prompt (assuming the Butler ZIP file/repository was saved to d:\node\butler):

    d:
    cd \node\butler\src
    npm install  
  

  This will download and install all Node.js modules used by Butler.
  On some OSs you’ll get some warnings during the installation - they are usually harmless. Try to run Butler even if you got some warnings, chances are good that things will work just fine. This is common on especially Windows Server and is a result of some Butler dependencies being primarily developed on Linux rather than Windows.

2.3 - Setup

2.3.1 - Which config file to use

A description of the config file format is available here.

Select which config file to use

Butler uses configuration files in YAML format.

A default config file called production_template.yaml is included in the release Zip files on the download page (starting with version 9.3.0). It is also available in the GitHub repository.

Make a copy of it, then rename the copy default.yaml, production.yaml, staging.yaml or something else suitable to your specific use case.
Update it as needed (see the config file reference page for details).

Trying to run Butler with the default config file (the one on GitHub) will not work - you must adapt it to your server environment. For example, you need to enter the IP or host name of you Sense server(s), the IP or host name where Butler is running etc.

Finally, Butler must somehow be given instructions about where to look for the config file.
This can be done in several ways depending on how Butler is used, see below.

Config file for stand-alone Butler

Let’s run Butler on a Windows Server using PowerShell, without any options or parameters:

PS C:\tools\butler> .\butler.exe
Usage: butler [options]

Butler gives superpowers to client-managed Qlik Sense Enterprise on Windows!
Advanced reload failure alerts, task scheduler, key-value store, file system access and much more.

Options:
  -V, --version                        output the version number
  -c, --configfile <file>              path to config file
  -l, --loglevel <level>               log level (choices: "error", "warn", "info", "verbose", "debug", "silly")
  --new-relic-account-name  <name...>  New Relic account name. Used within Butler to differentiate between different target New Relic accounts
  --new-relic-api-key <key...>         insert API key to use with New Relic
  --new-relic-account-id <id...>       New Relic account ID
  --test-email-address <address>       send test email to this address. Used to verify email settings in the config file.
  --test-email-from-address <address>  send test email from this address. Only relevant when SMTP server allows from address to be set.
  --no-qs-connection                   don't connect to Qlik Sense server at all. Run in isolated mode
  --api-rate-limit                     set the API rate limit, per minute. Default is 100 calls/minute. Set to 0 to disable rate limiting.
  -h, --help                           display help for command
PS C:\tools\butler>

There is an option --configfile (or its short version -c) that let us control which config file to use.
In this example the config file .\config\butler-config.yaml is used.
Let’s try again with the -c option:

PS C:\tools\butler> dir


    Directory: C:\tools\butler


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----        20/06/2022     16:27       68426646 butler.exe
-a----        20/06/2022     17:17          34762 production.yaml


PS C:\tools\butler> .\butler.exe -c c:\tools\butler\production.yaml
PS C:\tools\butler> .\butler.exe -c .\config\butler-config.yaml
2023-12-10T13:46:32.939Z info: Enabled API endpoints: [
  "apiListEnbledEndpoints",
  "base62ToBase16",
  "base16ToBase62",
  "butlerping",
  "createDir",
  "createDirQVD",
  "fileDelete",
  "fileMove",
  "fileCopy",
  "keyValueStore",
  "mqttPublishMessage",
  "postNewRelicMetric",
  "postNewRelicEvent",
...
...

Butler now starts nicely using the specified config file.

Config file when running Butler as a Node.js app

When running Butler as a Node.js app, i.e. starting it with node butler.js, Butler will look for a config file in the ./config subdirectory.

The name of the config file matters.
Butler looks for an environment variable called “NODE_ENV” and then tries to load a config file named with the value found in NODE_ENV.

Example: NODE_ENV=production

Butler will look for a config file config/production.yaml.

Config file when running Butler in a Docker container

The template docker-compose.yaml file in the GitHub repository shows how to specify which config file that will be used:

# docker-compose.yml
version: '3.3'
services:
  butler:
    image: ptarmiganlabs/butler:latest
    container_name: butler
    restart: always
    ports:
      - "8080:8080"       # REST API available on port 8180 to services outside the container
      - "9998:9998/udp"   # UDP port for task failure events
    volumes:
      # Make config file accessible outside of container
      - "./config:/nodeapp/config"
      - "./log:/nodeapp/log"
    environment:
      - "NODE_ENV=production"
    logging:
      driver: json-file
      options:
        max-file: "5"
        max-size: "5m"

Here the environment variable NODE_ENV is set to “production”, and the host OS’ ./config directory is mapped to the container’s /nodeapp/config directory.

As there is no --configfile command line option present the default setting will be used, which is to look for the config file in the config directory right under the directory where the docker-compose.yaml file is located.
The file name is determined by Butler (running in the container) looking at the NODE_ENV env variable.

Bottom line is that the ./config/production.yaml (relative to the location of docker-compose.yaml) file will be used.

Running several Butler instances in parallel

If you have several Sense clusters (for example DEV, TEST and PROD environments) you may want to run several Butler instances.

The solution is to create several config files: butler_dev.yaml, butler_test.yaml and butler_prod.yaml.

In this scenario three instances of Butler should be started, each given a different config file via the --configfile command line option.

Note: If running several Butler instances in parallel, you must also ensure that each one uses unique port numbers for their respective REST APIs, UDP servers etc.

Setting environment variables

The method for setting environment variables varies between operating systems:

On Windows:

set NODE_ENV=production

Mac OS or Linux

export NODE_ENV=production

If using Docker, the NODE_ENV environment varible is set in the docker-compose.yml file (as already done in the template docker-compose file.)

2.3.2 - Connecting to a Qlik Sense server

What’s this?

In order to interact with a Qlik Sense Enterprise on Windows (QSEoW) environment, Butler needs to know a few things about that environment. This is true no matter if the Sense cluster consists of a single Sense server or many.

Settings in main config file

---
Butler:
  ...
  ...
  # Certificates to use when connecting to Sense. Get these from the Certificate Export in QMC.
  cert:
    clientCert: <path/to/cert/client.pem>
    clientCertKey: <path/to/cert/client_key.pem>
    clientCertCA: <path/to/cert/root.pem>
    # If running Butler in a Docker container, the cert paths MUST be the following
    # clientCert: /nodeapp/config/certificate/client.pem
    # clientCertKey: /nodeapp/config/certificate/client_key.pem
    # clientCertCA: /nodeapp/config/certificate/root.pem

  configEngine:
    # engineVersion: 12.170.2        # Qlik Associative Engine version to use with Enigma.js. Ver 12.170.2 works with Feb 2019
    engineVersion: 12.612.0         # Qlik Associative Engine version to use with Enigma.js. Works with Feb 2020 and others
    host: <FQDN or IP of Sense server where Sense Engine is running>
    port: <Port to connect to, usually 4747>
    useSSL: true
    headers:
      X-Qlik-User: UserDirectory=Internal;UserId=sa_repository
    rejectUnauthorized: false

  configQRS:
    authentication: certificates
    host: <FQDN or IP of Sense server where QRS is running>
    useSSL: true
    port: 4242
    headerKey: X-Qlik-User                                      # Header used to identify what user connection to QRS is made as
    headerValue: UserDirectory=Internal; UserId=sa_repository   # What user connection to QRS is made as
    rejectUnauthorized: false       # Set to false to ignore warnings/errors caused by Qlik Sense's self-signed certificates.
                                    # Set to true if the Qlik Sense root CA is available on the computer where Butler SOS is running.
  ...
  ...

2.3.3 - Configuring Butler's REST API

What’s this?

Butler offers a set of REST API endpoints. While these endpoints are tested for stability and correct functionality as part of each release, it’s always good practice to only enable the endpoints really needed.

Thus, individual endpoints of Butler’s API can be turned on or off in the main config file.

Configuring the REST API

Butler:
  ...
  ...
  restServerConfig:
    enable: false                                     # Should Butler's REST API be started? Must be true if *any* API endpoints are to be used.
    serverHost: <FQDN or IP (or localhost) of server where Butler is running>   # Use 0.0.0.0 to listen on all network interfaces (e.g. when running in Docker!).
    serverPort: 8080                                  # Port where Butler's REST is available. Any free port on the server where Butler is running can bse used.
    backgroundServerPort: 8081

Ports used by Butler

Butler exposes its REST API on a TCP port defined in the Butler.restServerConfig.serverPort setting in the config file.

Similarly, the host name Butler listens at is defined by the Butler.restServerConfig.serverHost setting. This would typically be the IP number, host name or fully qualified domain name of the computer where Butler is running.

Note that Butler uses two ports for its REST API: One external facing port and one used internally. Both must be dedicated to Butler on the computer where Butler is running.

Using two ports (one external facing and one internal) is not ideal, but it was an easy yet stable way of solving some technical challenges around Butler’s use of the X-HTTP-Method-Override HTTP header. Just make sure that the two settings Butler.restServerConfig.serverPort and Butler.restServerConfig.backgroundServerPort aren’t the same and aren’t already in use, and all should be fine.

Rate limiting the REST API

Butler’s REST API can be rate limited to prevent abuse.

Rate limiting is configured by the --api-rate-limit command line parameter when starting Butler.

The parameter takes a single integer value, which is the number of API calls allowed per minute.
Set to 0 to disable rate limiting.

Enabling individual API endpoints

Each enabled endpoint will result in Butler using more memory and CPU. Thus only enable the endpoints that are needed.

Endpoint specific settings

In some cases some extra configuration is needed to make an API endpoint function properly.
This information is configured in the Butler.restServerEndpointsConfig section in the config file.

Settings in main config file

---
Butler:
  ...
  ...
  # Enable/disable individual REST API endpoints. Set config item below to true to enable that endpoint.
  restServerEndpointsEnable:
    apiListEnbledEndpoints: false
    base62ToBase16: false
    base16ToBase62: false
    butlerping: false
    createDir: false
    createDirQVD: false
    fileDelete: false
    fileMove: false
    fileCopy: false
    keyValueStore: false
    mqttPublishMessage: false
    newRelic:
      postNewRelicMetric: false
      postNewRelicEvent: false
    scheduler:
      createNewSchedule: false
      getSchedule: false
      getScheduleStatusAll: false
      updateSchedule: false
      deleteSchedule: false
      startSchedule: false
      stopSchedule: false
    senseAppReload: false
    senseAppDump: false
    senseListApps: false
    senseStartTask: false
    slackPostMessage: false 

  restServerEndpointsConfig:
    newRelic:
      postNewRelicMetric:          # Setings used by post metric to New Relic API endpoint
        destinationAccount:
          - First NR account
          - Second NR account
        # As of this writing the valid options are
        # https://insights-collector.eu01.nr-data.net/metric/v1
        # https://insights-collector.newrelic.com/metric/v1
        url: https://insights-collector.eu01.nr-data.net/metric/v1
        header:                   # Custom http headers
          - name: X-My-Header
            value: Header value
        attribute: 
          static:                   # Static attributes/dimensions to attach to the metrics data sent to New Relic.
            - name: env
              value: prod
      postNewRelicEvent:            # Setings used by post event to New Relic API endpoint
        destinationAccount:
          - First NR account
          - Second NR account
        # Note that the URL path should *not* be included in the url setting below!
        # As of this writing the valid options are
        # https://insights-collector.eu01.nr-data.net
        # https://insights-collector.newrelic.com 
        url: https://insights-collector.eu01.nr-data.net/
        header:                   # Custom http headers
          - name: X-My-Header
            value: Header value
        attribute: 
          static:                   # Static attributes/dimensions to attach to the metrics data sent to New Relic.
            - name: env
              value: prod
  ...
  ...

2.3.4 - Reload related alerts

Alert types

These alert types are available:

• Reload task failure. Send alerts when reload tasks fail, no matter if they were started on schedule or manually from the QMC.

• Reload task aborted. Send alerts when reload tasks are manually aborted in the QMC.

Alert destinations and options

Alerts can be sent to these destinations, with different options available for each destination.
Each destination can be individually enabled/disabled in the config file.

How it works

In order for Butler initiated alerts to become a reality, Butler must somehow be notified that the event of interest (for example a failed reload task) has occurred.
This is achieved by adding a log appender to Qlik Sense Enterprise on Windows.

Log appenders offer a way to hook into Qlik Sense’s logging subsystem, which is called log4net.

By adding a carefully crafted .xml file in the right location on the Sense server(s), you can make Sense notify Butler by means of UDP messages when the events of interest occur. Conceptually it looks like this:

So what happens when a scheduled reload task fails?
Let’s look at the steps:

1. A reload task is started by the Sense scheduler, either on a time schedule, as a result of some other task(s) finishing or manually by a user in the QMC or from the Hub.

2. When the task’s state changes, entries are written to the Sense scheduler’s log files using log4net (which is built into Qlik Sense). If the filter defined in the log appender (= the .xml file on the Sense server) matches the log entry at hand, the associated action in the log appender will be carried out.

3. Log appenders can do all kinds of things, everything from writing custom log files, sending basic emails, writing to databases and much more.
   Here we’re interested in the log appender sending a UDP message from Qlik Sense to Butler.

4. The log appender provided as part of Butler will make log4net send a UDP message to Butler, including various info about the reload task that just failed or was stopped/aborted.

5. Butler will look at the incoming event and determine what it is about.
   For example: Is the event about a reload task failure, a reload that has been aborted/stopped, or something else?
   Butler thus first works as a dispatcher. In a second step, after the initial dispatch, the event is sent to the relevant handler function within Butler.

Response times are usually very good - Butler will typically get the UDP message within a few seconds after (for example) the reload failing, with alerts going out shortly thereafter.

Adding a log appender

This is possibly the trickiest part to get right when it comes to setting up log4net based alerts.
Still, if you start from the sample .xml file provided in the Butler repository on GitHub it’s not too hard.
Those sample .xml files are also included in the release Zip files available on the Butler releases page.

The steps are:

1. In this case you want to be notified when certain events occur in the scheduler log files.

   This is important: Qlik Sense Enterprise on Windows consists of many different subsystems (engine, proxy, scheduler, printing etc) - here we’re interested in log events from the scheduler subsystem.

   Add a file LocalLogConfig.xml in the C:\ProgramData\Qlik\Sense\Scheduler folder on the Sense server whose scheduler you want to get events from. If you have multiple Sense servers with schedulers running on them, the .xml file should be deployed on each server (assuming you want events from all the servers).

2. The contents of LocalLogConfig.xml will determine what events are forwarded to Butler, or what other actions will be taken by log4net. See below for examples.

3. Sense will eventually detect and load the new xml file, but it might take a while (minutes). Restarting the Qlik Sense Scheduler Windows service will make the changes take effect immediately.

Forwarding task reload events to Butler

Here’s the XML that should go into C:\ProgramData\Qlik\Sense\Scheduler\LocalLogConfig.xml to enable the various kinds of Butler task reload alerts.

• The remoteAddress property should be set to the host name or IP where Butler is running.

• The remotePort property should match the port number specified in Butler’s config file. Note that Butler uses different ports for task related and user activity related events.

• The first appender looks for the text “Max retries reached” in the System.Scheduler.Scheduler.Master.Task.TaskSession log stream. That log entry will be created when a reload task has failed and also carried out all its retries. Once the search string is found a UDP message will be sent to port 9998 on IP 10.11.12.13.

• The second appender looks for “Execution State Change to Aborting” in the System.Scheduler.Scheduler.Master.Task.TaskSession log stream. That log entry occurs when a user stops a running reload from the QMC’s task view, or using the Sense APIs. When the search string is found a UDP message is once again sent to 10.11.12.13:9998, but with a different messsage (as specified in the conversionpattern property of the appender).

• The third appender looks for “Reload complete” in the System.Scheduler.Scheduler.Slave.Tasks.ReloadTask log stream.
  That log entry occurs when a reload task has completed successfully.

Here is an XML file that would forward log events as UDP messages to Butler:

<?xml version="1.0" encoding="UTF-8"?>

<configuration>
    <!-- Appender for detecting reload task failures. Only the last of potentially several retries is reported -->
    <appender name="TaskFailureLogger" type="log4net.Appender.UdpAppender">
        <filter type="log4net.Filter.StringMatchFilter">
            <param name="stringToMatch" value="Max retries reached" />
        </filter>
        <filter type="log4net.Filter.DenyAllFilter" />
        <param name="remoteAddress" value="<IP of server where Butler is running>" />
        <param name="remotePort" value="9998" />
        <param name="encoding" value="utf-8" />
        <layout type="log4net.Layout.PatternLayout">
            <converter>
                <param name="name" value="hostname" />
                <param name="type" value="Qlik.Sense.Logging.log4net.Layout.Pattern.HostNamePatternConverter" />
            </converter>
            <param name="conversionpattern" value="/scheduler-reload-failed/;%hostname;%property{TaskName};%property{AppName};%property{User};%property{TaskId};%property{AppId};%date;%level;%property{ExecutionId};%message" />
        </layout>
    </appender>

    <!-- Appender for detecting aborted reloads -->
    <appender name="AbortedReloadTaskLogger" type="log4net.Appender.UdpAppender">
        <filter type="log4net.Filter.StringMatchFilter">
            <param name="stringToMatch" value="Execution State Change to Aborting" />
        </filter>
        <filter type="log4net.Filter.DenyAllFilter" />
        <param name="remoteAddress" value="<IP of server where Butler is running>" />
        <param name="remotePort" value="9998" />
        <param name="encoding" value="utf-8" />
        <layout type="log4net.Layout.PatternLayout">
            <converter>
                <param name="name" value="hostname" />
                <param name="type" value="Qlik.Sense.Logging.log4net.Layout.Pattern.HostNamePatternConverter" />
            </converter>
            <param name="conversionpattern" value="/scheduler-reload-aborted/;%hostname;%property{TaskName};%property{AppName};%property{User};%property{TaskId};%property{AppId};%date;%level;%property{ExecutionId};%message" />
        </layout>
    </appender>

    <!-- Appender for detecting successful reload tasks -->
    <appender name="ReloadTaskSuccessLogger" type="log4net.Appender.UdpAppender">
        <filter type="log4net.Filter.StringMatchFilter">
            <param name="stringToMatch" value="Reload complete" />
        </filter>
        <filter type="log4net.Filter.DenyAllFilter" />
        <param name="remoteAddress" value="<IP of server where Butler is running>" />
        <param name="remotePort" value="9998" />
        <param name="encoding" value="utf-8" />
        <layout type="log4net.Layout.PatternLayout">
            <converter>
                <param name="name" value="hostname" />
                <param name="type" value="Qlik.Sense.Logging.log4net.Layout.Pattern.HostNamePatternConverter" />
            </converter>
            <param name="conversionpattern" value="/scheduler-reloadtask-success/;%hostname;%property{TaskName};%property{AppName};%property{User};%property{TaskId};%property{AppId};%date;%level;%property{ExecutionId};%message" />
        </layout>
    </appender>

    <!-- Send message to Butler on task failure -->
    <!-- Send message to Butler on task abort -->
    <logger name="System.Scheduler.Scheduler.Master.Task.TaskSession">
        <appender-ref ref="TaskFailureLogger" />
        <appender-ref ref="AbortedReloadTaskLogger" />
    </logger>

    <!-- Send message to Butler on reload task success -->
    <logger name="System.Scheduler.Scheduler.Slave.Tasks.ReloadTask">
        <appender-ref ref="ReloadTaskSuccessLogger" />
    </logger>

</configuration>

The above configuration is enough to support all task reload alerts currently supported by Butler.

Sending basic alert emails from log4net

If you are happy with the more basic/limited reload-failed alert emails provided by log4net, you can add a SMTP appender like this (the example below is for sending emails using Google GMail, customise as needed):

<?xml version="1.0"?>
<configuration>
    <!-- Mail appender-->
    <appender name="MailAppender" type="log4net.Appender.SmtpAppender">
        <filter type="log4net.Filter.StringMatchFilter">
            <param name="stringToMatch" value="Message from ReloadProvider" />
        </filter>
        <filter type="log4net.Filter.DenyAllFilter" />
        <evaluator type="log4net.Core.LevelEvaluator">
            <param name="threshold" value="ERROR"/>
        </evaluator>
        <param name="to" value="<email address to send failed task notification emails to>" />
        <param name="from" value="<sender email address used in notification emails>" />
        <param name="subject" value="Qlik Sense failed task (server <servername>)" />
        <param name="smtpHost" value="smtp.gmail.com" />
        <param name="port" value="587" />
        <param name="EnableSsl" value="true" />
        <param name="Authentication" value="Basic" />
        <param name="username" value="<Gmail username>" />
        <param name="password" value="<Gmail password>" />
        <param name="bufferSize" value="0" /> <!-- Set this to 0 to make sure an email is sent on every error -->
        <param name="lossy" value="true" />
        <layout type="log4net.Layout.PatternLayout">
            <param name="conversionPattern" value="%newline%date %-5level %newline%property{TaskName}%newline%property{AppName}%newline%message%newline%newline%newline" />
        </layout>
    </appender>

    <!--Send mail on task failure-->
    <logger name="System.Scheduler.Scheduler.Slave.Tasks.ReloadTask">
        <appender-ref ref="MailAppender" />
    </logger>
</configuration>

References

• Qlik’s documenation around log appenders and how to hook into the Sense logs is somewhat brief, but does provide a starting point if you want to dive deeper into this topic.

• The main log4net documentation (log4net is the logging framework used by Qlik Sense Enterprise) can also be useful.

These links describe how emails can be sent from the log4net logging framework itself, directly to the recipient. Butler includes sameple XML files for this use case too, but Butler takes things further by using the data in the Sense logs to pull in more data around the failed or stopped reload.

In other words - Butler’s alert emails are significantly more flexible and contain information (such as script logs) that are not availble using purely log4net.

2.3.4.1 - Reload alerts sent as emails

What’s this?

Butler can send two kinds of alert emails:

• When a scheduled, running reload task fails.
• When a scheduled, running reload task is somehow stopped/aborted.

Butler has a de-duplication feature that ensure each email address that has qualified for an alert email only gets ONE email per alert.

See the Concepts section for additional details and sample alert emails.

Basic vs formatted email alerts

If you want Butler to send email alerts you must provide an email template file.

For some other alert destinations (Slack and Teams) Butler offers a “basic” option. A fixed format alert is then sent by Butler.
The closest thing available for emails is to use the mail log appender described here, but if you set up a log appender AND have Butler running, you might as well use the formatted email option as it provides much more flexibility than log4net’s email appender.

Rate limiting

Butler has rate limiting feature to ensure alert recipients are not spammed with too many alert emails.

The rate limit is configured (in seconds) in the main config file and can be set independently for reload-failed and reload-aborted emails.
The corresponding config settings are Butler.emailNotification.reloadTaskFailure.rateLimit and Butler.emailNotification.reloadTaskAborted.rateLimit.

Rate limiting is done based on task ID + email address.

Sending test emails to verify correct settings

It can be tricky to find the correct settings to use Butler with email servers.
Butler itself uses a very generic email components to send emails, but corporate email servers may impose restrictions on from where/what servers emails will be accepted, encryption may be used together with non-standard network ports etc.

Butler offers a command line option that when used will send a simple test email to the specified email address.
This makes is very easy to test if the email settings in Butler’s config file are working or not.
When this command line option is used Butler will start normally, but also send a test email during startup.

The command line option is --test-email-address <address>.
The sender of the test email can be specified with --test-email-from-address <address>.

PS C:\tools\butler> .\butler.exe
Usage: butler [options]

Butler gives superpowers to client-managed Qlik Sense Enterprise on Windows!
Advanced reload failure alerts, task scheduler, key-value store, file system access and much more.

Options:
  -V, --version                        output the version number
  -c, --configfile <file>              path to config file
  -l, --loglevel <level>               log level (choices: "error", "warn", "info", "verbose", "debug", "silly")
  --new-relic-account-name  <name...>  New Relic account name. Used within Butler to differentiate between different target New Relic accounts
  --new-relic-api-key <key...>         insert API key to use with New Relic
  --new-relic-account-id <id...>       New Relic account ID
  --test-email-address <address>       send test email to this address. Used to verify email settings in the config file.
  --test-email-from-address <address>  send test email from this address. Only relevant when SMTP server allows from address to be set.
  --no-qs-connection                   don't connect to Qlik Sense server at all. Run in isolated mode
  --api-rate-limit                     set the API rate limit, per minute. Default is 100 calls/minute. Set to 0 to disable rate limiting.
  -h, --help                           display help for command
PS C:\tools\butler>

If the settings in the config file’s Butler.emailNotification.smtp section are valid and correct a command like this can be used:
butler.exe -c ./config/production.yaml --test-email-address myname@somedomain.com. Adapt config file location and email address as needed.

The resulting email looks like this:

Sending alert emails to app owners

Butler can optionally send alert emails to the owner of apps that failed reloading/were aborted.

This feature is controlled by the config file properties Butler.emailNotification.reloadTaskAborted.appOwnerAlert.enable and Butler.emailNotification.reloadTaskFailure.appOwnerAlert.enable.

If set to true the app owner will be added to the send list of alert emails, in addition to the recipients specied in Butler.emailNotification.reloadTaskAborted.recipients and Butler.emailNotification.reloadTaskFailure.recipients.

The sections of the config file dealing with app owner notification emails looks like this:

appOwnerAlert:
  enable: true              # Should app owner get notification email (assuming email address is available in Sense user directory)
  includeOwner:
    includeAll: true                            # true = Send notification to all app owners except those in exclude list
                                                # false = Send notification to all app owners in the include list
    user:
      - directory: <Sense user directory>
        userId: <userId>
      - directory: <Sense user directory>
        userId: <userId>
  excludeOwner:
    user:
      - directory: <Sense user directory>
        userId: <userId>
      - directory: <Sense user directory>
        userId: <userId>

It works like this:

• If appOwnerAlert.enable is set to false no app owner emails will be sent. If it’s set to true the rules below apply.
• If appOwnerAlert.includeOwner.includeAll is set to true all app owners will get notification emails when apps the own fail/are aborted…
  • … except those app owners listed in the appOwnerAlert.excludeOwner.user array.
  • That array thus provides a way to exclude some app owners (e.g. system accounts) to receive notifcation emails.
• If appOwnerAlert.includeOwner.includeAll is set to false it’s still possible to add individual app owners to the appOwnerAlert.includeOwner.user array.
  Those users will then receive notification emails for apps they own.

Send alerts only for some tasks

Some reload tasks may be more important than others.
I.e. some tasks should generate alert emails when they fail, but others not.

Butler controls which tasks to send alerts for by looking at a specific Qlik Sense custom property.

• If the config file setting Butler.emailNotification.reloadTaskFailure.alertEnableByCustomProperty.enable is set to false, all failed reload tasks will cause alert emails.
• If that setting is true only some tasks will cause alert emails:
  • If a task has the value specified in Butler.emailNotification.reloadTaskFailure.alertEnableByCustomProperty.enabledValue set for the custom property named as specified in Butler.emailNotification.reloadTaskFailure.alertEnableByCustomProperty.customPropertyName, the alert will be sent.
  • If a task does not have that custom property set, no alert will be sent for that task.
    • A task can still cause an alert to be sent if a specific email address is specified for the task, see below for details.

Some configuration is needed to make this work:

1. Make changes to the config file. Specifically the three settings mentioned above needs to be reviewed and updated as needed.
2. Create a custom property in Sense.
   1. The name and value of the custom property must match the one in the config file, Butler.emailNotification.reloadTaskFailure.alertEnableByCustomProperty.customPropertyName and Butler.emailNotification.reloadTaskFailure.alertEnableByCustomProperty.enabledValue.
   2. The custom property should be available on reload tasks.
3. Set the custom property for reload tasks for which alert emails should be sent.

Aborted reload tasks (as compared to the failed reload tasks described above) are handled the same way, with their own settings in the config file.

In the QMC the custom property can look like this:

Send alerts to specific people, for some tasks

It’s possible to send alert emails to specific email addresses and control this on a per-task basis.

This is achieved by using a Sense custom property that contains the email addresses alerts should be sent to, for the task in question.

• These config setting control which custom properties are used to store email addresses:
  • Butler.emailNotification.reloadTaskFailure.alertEnableByEmailAddress.customPropertyName
  • Butler.emailNotification.reloadAborted.alertEnableByEmailAddress.customPropertyName

Email specific alert recpients is independent from the feature where alerts can be switched on/off for individual tasks (see above).

In other words: If an email address has been designated as recipient of alert emails, that address will always receive alert emails for all failed or aborted reload tasks.

Having set two different (blurred out) recipients of alert emails for a reload task:

Settings in config file

---
Butler:
  ...
  ...
  # Qlik Sense related links used in notification messages
  qlikSenseUrls:
    qmc: <Link to Qlik Sense QMC>
    hub: <Link to Qlik Sense Hub>
  ...
  ...
  # Settings needed to send email notifications when for example reload tasks fail.
  # Reload failure notifications assume a log appender is configured in Sense AND that the UDP server in Butler is running.
  emailNotification:
    enable: false
    reloadTaskAborted:
      enable: false
      appOwnerAlert:
        enable: true              # Should app owner get notification email (assuming email address is available in Sense user directory)
        includeOwner:
          includeAll: true                            # true = Send notification to all app owners except those in exclude list
                                                      # false = Send notification to app owners in the include list
          user:
            - directory: <Sense user directory>
              userId: <userId>
            - directory: <Sense user directory>
              userId: <userId>
        excludeOwner:
          user:
            - directory: <Sense user directory>
              userId: <userId>
            - directory: <Sense user directory>
              userId: <userId>
      # Custom property used to control which aborted tasks will cause alert emails to be sent
      # If this setting is true, alerts will not be sent for all tasks, but *only* for tasks with the CP set to the enabledValue.
      # If this setting is false, alerts will be sent for all aborted reload tasks.
      alertEnableByCustomProperty:
        enable: true
        customPropertyName: 'Butler_AbortedAlertEnableEmail'
        enabledValue: 'Yes'
      # Custom property used to say that alerts for a certain task should be sent to zero or more recipients
      # These alerts will be sent irrespective of the alertEnableByCustomProperty.enable setting.
      alertEnabledByEmailAddress:
        customPropertyName: 'Butler_AbortedAlertSendToEmail'
      rateLimit: 600                                  # Min seconds between emails for a given taskID. Defaults to 5 minutes.
      headScriptLogLines: 15                          # Number of lines from start of script to include in email
      tailScriptLogLines: 15                          # Number of lines from end of script to include in email
      priority: high                                  # high/normal/low
      subject: 'Qlik Sense reload aborted: "{{taskName}}"'  # Email subject. Can use template fields
      bodyFileDirectory: path/to/email_templates      # Directory where email body template files are stored
      htmlTemplateFile: aborted-reload                # Name of email body template file to use
      fromAdress: Qlik Sense (no-reply) <qliksense-noreply@mydomain.com>
      recipients:                                     # Array of email addresses to which the notification email will be sent
        - <Email address 1>
        - <Email address 2>
    reloadTaskFailure:
      enable: false
      appOwnerAlert:
        enable: true              # Should app owner get notification email (assuming email address is available in Sense user directory)
        includeOwner:
          includeAll: true                            # true = Send notification to all app owners except those in exclude list
                                                      # false = Send notification to app owners in the include list
          user:
            - directory: <Sense user directory>
              userId: <userId>
            - directory: <Sense user directory>
              userId: <userId>
        excludeOwner:
          user:
            - directory: <Sense user directory>
              userId: <userId>
            - directory: <Sense user directory>
              userId: <userId>
      # Custom property used to control which task failures will cause alert emails to be sent
      # If this setting is true, alerts will not be sent for all tasks, but *only* for tasks with the CP set to the enabledValue.
      # If this setting is false, alerts will be sent for all failed reload tasks.
      alertEnableByCustomProperty:
        enable: false
        customPropertyName: 'Butler_FailedAlertEnableEmail'
        enabledValue: 'Yes'
      # Custom property used to say that alerts for a certain task should be sent to zero or more recipients
      # These alerts will be sent irrespective of the alertEnableByCustomProperty.enable setting.
      alertEnabledByEmailAddress:
        customPropertyName: 'Butler_FailedAlertSendToEmail'
      rateLimit: 600                                  # Min seconds between emails for a given taskID. Defaults to 5 minutes.
      headScriptLogLines: 15                          # Number of lines from start of script to include in email
      tailScriptLogLines: 15                          # Number of lines from end of script to include in email
      priority: high                                  # high/normal/low
      subject: 'Qlik Sense reload failed: "{{taskName}}"'   # Email subject. Can use template fields
      bodyFileDirectory: path/to/email_templates      # Directory where email body template files are stored
      htmlTemplateFile: failed-reload                 # Name of email body template file to use
      fromAdress: Qlik Sense (no-reply) <qliksense-noreply@mydomain.com>
      recipients:                                       # Array of email addresses to which the notification email will be sent
        - <Email address 1>
        - <Email address 2>
    serviceStopped:
      rateLimit: 30                   # Min seconds between emails for a given service. Defaults to 5 minutes.
      priority: high                  # high/normal/low
      subject: '❌ Windows service stopped on host {{host}}: "{{serviceDisplayName}}"'
      bodyFileDirectory: path/to/email_templates/email_templates
      htmlTemplateFile: service-stopped
      fromAdress: Qlik Sense (no-reply) <qliksense-noreply@mydomain.com>
      recipients:
        - <Email address 1>
        - <Email address 2>
    serviceStarted:
      rateLimit: 30                   # Min seconds between emails for a given service. Defaults to 5 minutes.
      priority: high                  # high/normal/low
      subject: '✅ Windows service started on host {{host}}: "{{serviceDisplayName}}"'
      bodyFileDirectory: path/to/email_templates/email_templates
      htmlTemplateFile: service-started
      fromAdress: Qlik Sense (no-reply) <qliksense-noreply@mydomain.com>
      recipients:
        - <Email address 1>
        - <Email address 2>
    smtp:                                             # Email server settings. See https://nodemailer.com/smtp/ for details on the meaning of these fields.
      host: <FQDN or IP or email server, e.g. smtp.gmail.com>
      port: <port on which SMTP server is listening>
      secure: true                                    # true/false
      tls:
        serverName:                                   # If specified the serverName field will be used for TLS verification instead of the host field.
        ignoreTLS: false
        requireTLS: true
        rejectUnauthorized: false
      auth:
        enable: true
        user: <Username, email address etc>
        password: <your-secret-password>
  ...
  ...
  udpServerConfig:
    enable: false                                     # Should the UDP server responsible for receving task failure and session events be started? true/false
    serverHost: <FQDN or IP (or localhost) of server where Butler is running>
    portTaskFailure: 9998
  ...
  ...

Templates: Configuring email appearance

Alert emails use standard HTML formatting. Inline CSS can be used (if so desired) for fine tuning the visual look of the alert email.

Butler’s process for sending alert emails is

1. Figure out which email body template file should be used. This is determine by two set of fields in the main config file:
   1. For reload failure emails these config file properties are used: Butler.emailNotification.reladTaskFailure.bodyFileDirectory and Butler.emailNotification.reladTaskFailure.htmlTemplateFile
   2. For aborted reload emails these config file properties are used: Butler.emailNotification.reloadTaskAborted.bodyFileDirectory and Butler.emailNotification.reloadTaskAborted.htmlTemplateFile
2. For email subjects, these config properties are used: Butler.emailNotification.reladTaskFailure.subject and Butler.emailNotification.reloadTaskAborted.subject
3. Process the body template, replacing template fields with actual values.
4. Process the email subject template, replacing template fields with actual values.
5. Send the email.

A couple of sample template files are found in the src/config/email_templates directory of the GitHub repository.

Template fields reference

A complete list of template fields - including descriptions - is available in the Reference section.

2.3.4.2 - Reload alerts in InfluxDB

What’s this?

Butler can store information about both successful and failed reload tasks in InfluxDB.

• If enabled, Butler will store information about all failed reload tasks to InfluxDB.
• For successful reload tasks, there are two options:
  • Store information about all successful reload tasks to InfluxDB.
  • Store information about only some successful reload tasks to InfluxDB.
    Which tasks to store information about is controlled using a custom property on the reload task.

Once the information about the reload task is in InfluxDB, it can be used to create dashboards in Grafana.

This way it is possible to get a good, continuous overview of the reload activity in your Qlik Sense environment.
You can also use the information to create alerts in Grafana using it’s comprehensive alerting capabilities, including alerting to Slack, Teams, email, etc.

As with the other reload failures destinations, Butler detects failures of reload tasks that were started from the QMC.

Please note that InflixDB must be enabled and correctly configured in the Butler config file for the below features to work.

Monitor failed reload tasks

If enabled using the Butler.influxDb.reloadTaskFailure.enable setting, Butler will store information about all failed reload tasks in InfluxDB.

The information stored includes (among other things):

• The name and ID of the app that the failed reload task was reloading.
• The name and ID of the reload task.
• The name of the Qlik Sense node/server that the task was running on.
• User who started the reload task. This will be the service account when the task was started by a schedule or via a task chain/trigger.
• Execution ID of the reload. This is a unique ID that is generated by Qlik Sense for each reload task execution, it can be used to cross-reference the reload task with related entries in the Qlik Sense log files.
• Last Butler.influxDb.reloadTaskFailure.tailScriptLogLines lines of the Sense log file for the reload task.
• Static tags defined in the config file’s Butler.influxDb.reloadTaskFailure.tag.static section.
• Dynamic app tags, i.e. Sense tags for the app being reloaded, if enabled in the config file Butler.influxDb.reloadTaskFailure.tag.dynamic.useAppTags section.
• Dynamic reload task tags, i.e. Sense tags for the reload task being executed, if enabled in the config file Butler.influxDb.reloadTaskFailure.tag.dynamic.useTaskTags section.

A complete definition of all information sent to InfluxDB is available in the reference section.

Monitor successful reload tasks

Butler can monitor all reload tasks for successful completion, or only some of them.

Monitor all successful reload tasks

If enabled using the Butler.influxDb.reloadTaskSuccess.allReloadTasks.enable setting, Butler will store information about all successful reload tasks in InfluxDB.

The information stored is almost the same as for failed reload tasks, except that the Sense script log file is not included (as it can potentially be very large).

Monitor only some successful reload tasks

If enabled using the Butler.influxDb.reloadTaskSuccess.byCustomProperty.enable setting, Butler will store information about only some successful reload tasks in InfluxDB.

Which tasks to store information about is controlled using a custom property on the reload task.
The name of the custom property is defined in the Butler.influxDb.reloadTaskSuccess.byCustomProperty.customPropertyName setting.
The value of the custom property that will be used to indicate that the reload task should be monitored is defined in the Butler.influxDb.reloadTaskSuccess.byCustomProperty.enabledValue setting.

Static vs dynamic tags

Butler offers two kinds of tags: Static and dynamic.

Static tags are defined in the config file and are the same for all messages stored in InfluxDB.
An example of a static tag could be the name of the Qlik Sense server that Butler is running on, or whether the message related to a production or test Qlik Sense environment.

Dynamic attributes are determined at run-time when the message is stored in InfluxDB.

Settings in config file

---
Butler:
  ...
  ...
  influxDb:
    ...
    ...
    reloadTaskFailure:
      enable: true
      tailScriptLogLines: 20
      tag: 
        static:                 # Static tags to attach to data stored in InflixDB
          - name: butler_instance
            value: prod-1
        dynamic:
          useAppTags: true      # Should app tags be stored in InfluxDB as tags?
          useTaskTags: true     # Should task tags be stored in InfluxDB as tags?      
    reloadTaskSuccess:
      enable: true
      allReloadTasks:
        enable: false
      byCustomProperty:
        enable: true
        customPropertyName: 'Butler_SuccessReloadTask_InfluxDB'
        enabledValue: 'Yes'
      tag: 
        static:                 # Static attributes/dimensions to attach to events sent to InfluxDb
          # - name: event-specific-tag 1
          #   value: abc 123
        dynamic:
          useAppTags: true      # Should app tags be sent to InfluxDb as tags?
          useTaskTags: true     # Should task tags be sent to InfluxDb as tags?
  ...
  ...

2.3.4.3 - Reload alerts via New Relic

What’s this?

Butler can send two kinds of messages to New Relic:

• When a scheduled or started from the QMC reload task fails.
• When a scheduled or started from the QMC reload task is somehow stopped/aborted.

See the Concepts section for examples on what a New Relic alert can look like.

This page has additional info on how to set up Butler to work with New Relic.

A complete reference to the config file format is found here.

Different kinds of New Relic messages

Two kinds of messages can be sent to New Relic: Events and log messages.

The difference between them is that New Relic events are meant to be used for alerting, while New Relic log messages are meant to be used for troubleshooting.

Events are more flexible in terms of what data can be included in them, whereas log messages are just that - parts of Sense log files sent to New Relic.

Together they provide a powerful combination of alerting and troubleshooting capabilities, but they can also be enabled independently of each other.

Destination accounts

New Relic does not have very good access control capabilities for their dashboards, so if you want certain people to see only some reload alerts, and other people to see other alerts, you need to create multiple New Relic accounts.

Butler supports this scenario and can send messages to one or more New Relic accounts.
It is possible to specify per reload task which New Relic account(s) to send alerts to.

Three pieces of information is needed for each New Relic account that Butler should send messages to:

• The name of the New Relic account. This is just a name that you choose, it is not used for anything other than to identify the account in Butler’s config file and in the custom properties of Qlik Sense reload tasks.
• The New Relic account ID.
• The New Relic insert/API key. This is basically a secret key that is used to authenticate Butler with New Relic.

Account numbers and insert keys are available in the New Relic UI, under “Account settings” > “Data sharing”.

Authentication and credentials

Butler looks for New Relic account names, account ID and API keys in two places:

1. The command line, using the --new-relic-account-name, --new-relic-account-id and --new-relic-api-key options.
   1. If you have multiple New Relic accounts they should be listed in sequence, separated by space.
   2. Account names can include spaces, but should then be enclosed in double quotes.
   3. Example: --new-relic-account-name "First New Relic account" "Second New Relic account" --new-relic-api-key 1234567890abcdef 0987654321fedcba --new-relic-account-id 1234567 7654321
2. The config file, in the Butler.thirdPartyToolsCredentials.newRelic section.

Standard attributes

When sending messages to New Relic you can include “attributes”.

Attributes are key/value pairs that can be used to provide additional information about the message.
They can be added to both events and log messages.

Attributes can be used in New Relic dashboards to filter and group messages in various ways.

Static vs dynamic attributes

Butler offers two kinds of attributes: Static and dynamic.

Static attributes are defined in the config file and are the same for all messages sent to New Relic.
An example of a static attribute could be the name of the Qlik Sense server that Butler is running on, or whether the message related to a production or test Qlik Sense environment.

Dynamic attributes are determined at run-time when the message is sent to New Relic.

Examples include:

• Sense tags that are assigned to the reload task that failed. Their names are qs_appTag_<tag name>
• App tags of the app that failed to reload. Their names are qs_taskTag_<tag name>

Shared settings

Some settings are shared between events and log messages, these are found in the sharedSettings sections of the config file. Values there will be used for both events and log messages, unless they are overridden in the respective events or logMessages sections of the config file.

Settings in config file

---
Butler:
  ...
  ...
  thirdPartyToolsCredentials:
    newRelic:         # Array of New Relic accounts/insert keys. Any data sent to New Relic will be sent to both accounts. 
      - accountName: First NR account
        insertApiKey: <API key 1 (with insert permissions) from New Relic> 
        accountId: <New Relic account ID 1>
      - accountName: Second NR account
        insertApiKey: <API key 2 (with insert permissions) from New Relic> 
        accountId: <New Relic account ID 2>
  ...
  ...
  incidentTool:
    newRelic:
      enable: false
      destinationAccount:
        event:                    # Failed/aborted reload tasks are sent as events to these New Relic accounts
          - First NR account
          - Second NR account
        log:                      # Failed/aborted reload tasks are sent as log entries to these New Relic accounts
          - First NR account
          - Second NR account
      # New Relic uses different API URLs for different kinds of data (metrics, events, logs, ...)
      # There are different URLs depending on whther you have an EU or US region New Relic account.
      # The available URLs are listed here: https://docs.newrelic.com/docs/accounts/accounts-billing/account-setup/choose-your-data-center/
      url:
        # As of this writing the valid options are
        # https://insights-collector.eu01.nr-data.net
        # https://insights-collector.newrelic.com 
        event: https://insights-collector.eu01.nr-data.net

        # Valid options are (1) EU/rest of world and 2) US)
        # https://log-api.eu.newrelic.com/log/v1
        # https://log-api.newrelic.com/log/v1 
        log: https://log-api.eu.newrelic.com/log/v1
      reloadTaskFailure:
        destination:
          event: 
            enable: false
            sendToAccount:              # Which reload task failures are sent to New Relic as events
              byCustomProperty:
                enable: false            # Control using a task custom property which reload task failures are sent as events
                customPropertyName: 'Butler_FailedTask_Event_NewRelicAccount'
              always:
                enable: false            # Controls which New Relic accounts ALL failed reload tasks are sent to (as events)
                account: 
                  - First NR account
                  - Second NR account
            attribute: 
              static:                 # Static attributes/dimensions to attach to events sent to New Relic.
                - name: event-specific-attribute 1  # Example
                  value: abc 123                    # Example
              dynamic:
                useAppTags: true      # Should app tags be sent to New Relic as attributes?
                useTaskTags: true     # Should task tags be sent to New Relic as attributes?
          log:
            enable: false
            tailScriptLogLines: 20
            sendToAccount:              # Which reload task failures are sent to New Relic as log entries
              byCustomProperty:
                enable: false            # Control using a task custom property which reload task failures are sent as log entries
                customPropertyName: 'Butler_FailedTask_Log_NewRelicAccount'
              always:
                enable: false            # Controls which New Relic accounts ALL failed reload tasks are sent to (as logs)
                account: 
                  - First NR account
                  - Second NR account
            attribute: 
              static:                 # Static attributes/dimensions to attach to events sent to New Relic.
                - name: log-specific-attribute 1    # Example
                  value: def 123                    # Example
              dynamic:
                useAppTags: true      # Should app tags be sent to New Relic as attributes?
                useTaskTags: true     # Should task tags be sent to New Relic as attributes?
        sharedSettings:
          rateLimit: 15             # Min seconds between events sent to New Relic for a given taskID. Defaults to 5 minutes.
          header:                   # Custom http headers
            - name: X-My-Header     # Example
              value: Header value 1 # Example
          attribute: 
            static:                 # Static attributes/dimensions to attach to events sent to New Relic.
              - name: service       # Example
                value: butler       # Example
              - name: environment   # Example
                value: prod         # Example
      reloadTaskAborted:
        destination:
          event: 
            enable: false
            sendToAccount:              # Which reload task aborts are sent to New Relic as events
              byCustomProperty:
                enable: false            # Control using a task custom property which reload task aborts are sent as events
                customPropertyName: 'Butler_AbortedTask_Event_NewRelicAccount'
              always:
                enable: false            # Controls which New Relic accounts ALL aborted reload tasks are sent to (as events)
                account: 
                  - First NR account
                  - Second NR account
            attribute: 
              static:                 # Static attributes/dimensions to attach to events sent to New Relic.
                - name: event-specific-attribute 2  # Example
                  value: abc 123                    # Example
              dynamic:
                useAppTags: true      # Should app tags be sent to New Relic as attributes?
                useTaskTags: true     # Should task tags be sent to New Relic as attributes?
          log:
            enable: false
            tailScriptLogLines: 20
            sendToAccount:              # Which reload task aborts are sent to New Relic as log entries
              byCustomProperty:
                enable: true            # Control using a task custom property which reload task aborts are sent as log entries
                customPropertyName: 'Butler_AbortedTask_Log_NewRelicAccount'
              always:
                enable: false          # Controls which New Relic accounts ALL aborted reload tasks are sent to (as logs)
                account: 
                  - First NR account
                  - Second NR account
            attribute: 
              static:                 # Static attributes/dimensions to attach to events sent to New Relic.
                - name: log-specific-attribute 2    # Example
                  value: def 123                    # Example
              dynamic:
                useAppTags: true      # Should app tags be sent to New Relic as attributes?
                useTaskTags: true     # Should task tags be sent to New Relic as attributes?
        sharedSettings:
          rateLimit: 15             # Min seconds between events sent to New Relic for a given taskID. Defaults to 5 minutes.
          header:                   # Custom http headers
            - name: X-My-Header     # Example
              value: Header value 2 # Example
          attribute: 
            static:                 # Static attributes/dimensions to attach to events sent to New Relic.
              - name: service       # Example
                value: butler       # Example
              - name: environment   # Example
                value: prod         # Example
      serviceMonitor:
        destination:
          event: 
            enable: false
            sendToAccount:                # Windows service events are sent to these New Relic accounts
              - First NR account
              - Second NR account
            attribute: 
              static:                     # Static attributes/dimensions to attach to events sent to New Relic.
                - name: event-specific-attribute
                  value: abc 123
              dynamic:
                serviceHost: true         # Should host where service is running be sent to New Relic as attribute?
                serviceName: true         # Should service name be sent to New Relic as attribute?
                serviceDisplayName: true  # Should service display name be sent to New Relic as attribute?
                serviceState: true        # Should service state be sent to New Relic as attribute?
          log:
            enable: false
            sendToAccount:                # Windows service log entries are sent to these New Relic accounts
              - First NR account
              - Second NR account
            attribute: 
              static:                     # Static attributes/dimensions to attach to events sent to New Relic.
                - name: log-specific-attribute
                  value: def 456
              dynamic:
                serviceHost: true         # Should host where service is running be sent to New Relic as attribute?
                serviceName: true         # Should service name be sent to New Relic as attribute?
                serviceDisplayName: true  # Should service display name be sent to New Relic as attribute?
                serviceState: true        # Should service state be sent to New Relic as attribute?
        monitorServiceState:              # Control whih service states are sent to New Relic
          running:
            enable: true
          stopped:
            enable: true
        sharedSettings:
          rateLimit: 5                    # Min seconds between events/logs sent to New Relic for a given host+service. Defaults to 5 minutes.
          header:                         # Custom http headers
            - name: X-My-Header           # Example
              value: Header value 2       # Example
          attribute: 
            static:                       # Static attributes/dimensions to attach to events sent to New Relic.
              - name: service             # Example
                value: butler             # Example
              - name: environment         # Example
                value: prod               # Example

  ...
  ...