This is the multi-page printable view of this section. Click here to print.
Reference docs
Detailed, more technical descriptions of Butler’s REST API and other parts of Butler.
- 1: Command line options
- 2: Config file syntax
- 3: REST API documentation
- 4: REST API documentation (alternative format)
- 5: Scheduler
- 6: Key-value store
- 7: Alert template fields
- 7.1: Client managed Qlik Sense Enterprise on Windows
- 7.1.1: Alert template fields for failed/aborted reload tasks
- 7.1.2: Alert template fields for Windows services events
- 7.2: Qlik Sense Cloud
- 8: Information stored in InfluxDB
- 9: Helper subs that make it easy to use Butler APIs
- 10: Test cases
1 - Command line options
Description of Butler’s command line options.
Command line options
When starting Butler, you can pass command line options to customize its behavior.
Looks like this:
Usage: butler [options]
Butler gives superpowers to client-managed Qlik Sense Enterprise on Windows!
Advanced reload failure alerts, task scheduler, key-value store, file system access and much more.
Options:
-V, --version output the version number
-c, --configfile <file> path to config file
-l, --loglevel <level> log level (choices: "error", "warn", "info", "verbose", "debug", "silly")
--new-relic-account-name <name...> New Relic account name. Used within Butler to differentiate between different target New Relic accounts
--new-relic-api-key <key...> insert API key to use with New Relic
--new-relic-account-id <id...> New Relic account ID
--test-email-address <address> send test email to this address. Used to verify email settings in the config file.
--test-email-from-address <address> send test email from this address. Only relevant when SMTP server allows from address to be set.
--no-qs-connection don't connect to Qlik Sense server at all. Run in isolated mode
--api-rate-limit set the API rate limit, per minute. Default is 100 calls/minute. Set to 0 to disable rate limiting.
--skip-config-verification Disable config file verification (default: false)
-h, --help display help for command
-V, –version
Output the version number of Butler.
-c, –configfile
Specifies the configuration file to use.
Valid values: A path to a configuration file.
Default: Whatever is specified in the NODE_ENV environment variable, with a .yaml extension added. Butler will look for that file in the ./config directory.
Example:
-cor--configfileare not specified.NODE_ENVis set toproduction. Butler will try to read settings from./config/production.yaml.
-l, –loglevel
Specifies the log level to use.
When set, this overrides the log level specified in the configuration file.
Valid values: ’error’, ‘warn’, ‘info’, ‘verbose’, ‘debug’, ‘silly’
Default: ‘info’
New Relic related options
When using New Relic as backend for storing logs, information about failed reloads etc, you can specify New Relic credentials in the config file - but that is not ideal from a security perspective.
To avoid that, you can specify the New Relic credentials on the command line using the following options.
–new-relic-account-name
List of New Relic account names. Used within Butler to differentiate between different target New Relic accounts to which data can be sent. This name has nothing to do with the account name used in New Relic - it’s purely for Butler’s internal use.
Specifically, it’s at multiple places in the config file where you can specificy to which New Relic account to send data.
Enclose account names in quotes if they contain spaces.
Separate multiple account names with a space.
Example: --new-relic-account-name "Account 1" "Account 2"
–new-relic-api-key
List of New Relic API keys. Used to authenticate with New Relic.
Enclose API keys in quotes if they contain spaces.
Separate multiple API keys with a space. Note that the order of the API keys must match the order of the account names, i.e. the first API key corresponds to the first account name, the second API key corresponds to the second account name, and so on.
Example: --new-relic-api-key "API key 1" "API key 2"
–new-relic-account-id
List of New Relic account IDs. Used to identify the New Relic account to which data should be sent.
Enclose account IDs in quotes if they contain spaces.
Separate multiple account IDs with a space. Note that the order of the account IDs must match the order of the account names, i.e. the first account ID corresponds to the first account name, the second account ID corresponds to the second account name, and so on.
Send test email
Butler has several alerting features for sending emails when some event happens (typoically something failing…) in Sense.
To verify that the email settings in the config file are correct, you can send a test email using the following options.
–test-email-address
Send a test email to this address, using the SMTP configuration in the config file.
Used to verify email settings in the config file.
This is the destination address for the test email.
–test-email-from-address
Send a test email to this address, using the SMTP configuration in the config file.
Used to verify email settings in the config file.
This is the sender address for the test email. Only relevant when the SMTP server allows the from address to be set.
–no-qs-connection
Don’t connect to Qlik Sense server at all. Run in isolated mode.
This is probably only relevant when you’re developing Butler and don’t have a Qlik Sense server available, or when you want to automatically exporting the API docs for Butler’s REST API (this is done during the build process).
–api-rate-limit
Set the rate limit of Butler’s REST API, per minute. Default is 100 calls/minute. Set to 0 to disable rate limiting.
Useful when you want to ensure that Butler (or the server it runs on) doesn’t get overloaded with requests.
–skip-config-verification
Disable config file verification.
By default, Butler verifies the config file when it starts. If the config file is invalid, Butler will log an error and exit.
Use this option to disable config file verification.
-h, –help
Display help for command.
2 - Config file syntax
Description of Butler’s config file.
Main Butler config file
The production_template.yaml config file looks like this (sorry for the incorrect syntax coloring, the issue is noted and is being worked on):
Info
Starting with Butler version 9.0 there is a check that the config file has the correct format.
This means that if you forget to add or change some setting in the main YAML config file, Butler will tell you what’s missing and refuse to start.
A consequence of this is that all settings are now mandatory, even if you don’t use them.
Trying to start Butler without some mandatory settings in the config file will result in an error message like this:
Adding the missing settings and restarting Butler will result in a successful startup.
---
Butler:
# General notes:
# - File and directory paths in this sample config file use Linux/Mac syntax, i.e. using forward slashes.
# Windows paths work just as well, just make sure to quote them with single or double quotes.
# - All entries in the config file are mandatory in the sense that they must be present.
# However, if a feature is not used the corresponding config entries can contain
# any value (for example the provided default ones).
# - Butler will start using the settings in this file if the follwing settings are set first:
# - Butler.cert.clientCert: Set to the path of the client certificate file. If relative paths cause issues, use an absolute path.
# - Butler.cert.clientCertKey: Set to the path of the client key file. If relative paths cause issues, use an absolute path.
# - Butler.cert.clientCertCA: Set to the path of the CA certificate file. If relative paths cause issues, use an absolute path.
# - Butler.configEngine.host: Set to the IP or FQDN of the host where the Sense engine service is running.
# - Butler.configEngine.port: Set to the port where the Sense engine service is listening.
# - Butler.configQRS.host: Set to the IP or FQDN of the host where the Qlik Repository Service (QRS) is running.
# - Butler.configQRS.port: Set to the port where the Qlik Repository Service (QRS) is listening.
# - Having set the above settings, Butler will start and run, but it will not do anything useful until you configure
# the various monitoring and notification settings, as described at https://butler.ptarmiganlabs.com.
# Logging configuration
logLevel: info # Log level. Possible log levels are silly, debug, verbose, info, warn, error
fileLogging: false # true/false to enable/disable logging to disk file
logDirectory: log # Directory where log files are stored (no trailing / )
anonTelemetry: true # Can Butler send anonymous telemetry data?
# More info on whata data is collected: https://butler.ptarmiganlabs.com/docs/about/telemetry/
# Please consider leaving this at true - it really helps future development of Butler!
# Should Butler start a web server that serves an obfuscated view of the Butler config file?
configVisualisation:
enable: true
host: localhost # Hostname or IP address where the web server will listen. Should be localhost in most cases.
port: 3100 # Port where the web server will listen. Change if port 3100 is already in use.
obfuscate: true # Should the config file shown in the web UI be obfuscated?
# Heartbeats can be used to send "I'm alive" messages to any other tool, e.g. an infrastructure monitoring tool
heartbeat:
enable: false
remoteURL: http://my.monitoring.server/some/path/
frequency: every 30 seconds # https://bunkat.github.io/later/parsers.html#text
# Docker health checks are used when running Butler as a Docker container.
# The Docker engine will call the container's health check REST endpoint with a set interval to determine
# whether the container is alive/well or not.
# If you are not running Butler in Docker you can safely disable this feature.
dockerHealthCheck:
enable: false # Control whether a REST endpoint will be set up to serve Docker health check messages
port: 12398 # Port the Docker health check service runs on (if enabled)
# Uptime monitor
uptimeMonitor:
enable: false # Should uptime messages be written to the console and log files?
frequency: every 15 minutes # https://bunkat.github.io/later/parsers.html#text
logLevel: verbose # Starting at what log level should uptime messages be shown?
storeInInfluxdb:
enable: false # Should Butler memory usage be logged to InfluxDB?
storeNewRelic:
enable: false
destinationAccount:
- First NR account
- Second NR account
# There are different URLs depending on whther you have an EU or US region New Relic account.
# The available URLs are listed here: https://docs.newrelic.com/docs/accounts/accounts-billing/account-setup/choose-your-data-center/
# As of this writing the options for the New Relic metrics API are
# https://insights-collector.eu01.nr-data.net/metric/v1
# https://metric-api.newrelic.com/metric/v1
url: https://insights-collector.eu01.nr-data.net/metric/v1 # Where should uptime data be sent?
header: # Custom http headers
- name: X-My-Header
value: Header value
metric:
dynamic:
butlerMemoryUsage:
enable: true # Should Butler's memory/RAM usage be sent to New Relic?
butlerUptime:
enable: true # Should Butler's uptime (how long since it was started) be sent to New Relic?
attribute:
static: # Static attributes/dimensions to attach to the data sent to New Relic.
- name: metricType
value: butler-uptime
- name: service
value: butler
- name: environment
value: prod
dynamic:
butlerVersion:
enable: true # Should the Butler version be included in the data sent to New Relic?
# Credentials for third party systems that Butler integrate with.
# These can also be specified via command line parameters when starting Butler.
# Command line options takes precedence over settings in this config file.
thirdPartyToolsCredentials:
newRelic: # Array of New Relic accounts/insert keys. Any data sent to New Relic will be sent to both accounts.
# - accountName: First NR account
# insertApiKey: <API key 1 (with insert permissions) from New Relic>
# accountId: <New Relic account ID 1>
# - accountName: Second NR account
# insertApiKey: <API key 2 (with insert permissions) from New Relic>
# accountId: <New Relic account ID 2>
# InfluxDB settings
influxDb:
enable: false # Master switch for InfluxDB integration. If false, no data will be sent to InfluxDB.
hostIP: influxdb.mycompany.com # IP or FQDN of Influxdb server
hostPort: 8086 # Port where Influxdb is listening. Default=8086
auth:
enable: false # Does InfluxDB require login?
username: user_joe
password: joesecret
dbName: butler # Name of database in InfluxDB to which Butler's data is written
# Default retention policy that should be created in InfluxDB when Butler creates a new database there.
# Any data older than retention policy threshold will be purged from InfluxDB.
retentionPolicy:
name: 10d
duration: 10d
tag:
static: # Static tags to attach to all data stored in InflixDB
# - name: butler_instance
# value: dev
reloadTaskFailure:
enable: true
tailScriptLogLines: 20
tag:
static: # Static tags to attach to data stored in InflixDB
- name: butler_instance
value: prod-1
dynamic:
useAppTags: true # Should app tags be stored in InfluxDB as tags?
useTaskTags: true # Should task tags be stored in InfluxDB as tags?
reloadTaskSuccess:
enable: true
allReloadTasks:
enable: false
byCustomProperty:
enable: true
customPropertyName: 'Butler_SuccessReloadTask_InfluxDB'
enabledValue: 'Yes'
tag:
static: # Static attributes/dimensions to attach to events sent to InfluxDb
# - name: event-specific-tag 1
# value: abc 123
dynamic:
useAppTags: true # Should app tags be sent to InfluxDb as tags?
useTaskTags: true # Should task tags be sent to InfluxDb as tags?
# Store script logs of failed reloads on disk.
# The script logs will be stored in daily directories under the specified main directory below
# NOTE: Use an absolute path when running Butler as a standalone executable!
scriptLog:
storeOnDisk:
clientManaged:
reloadTaskFailure:
enable: false
logDirectory: /path/to/scriptlogs/qseow
qsCloud:
appReloadFailure:
enable: false
logDirectory: /path/to/scriptlogs/qscloud
# Qlik Sense (client-managed) related links used in notification messages
qlikSenseUrls:
qmc: <URL to Qlik Sense QMC>
hub: <URL to Qlik Sense Hub>
appBaseUrl: <URL to Qlik Sense hub>/<virtual proxy, if any>/sense/app # Base URL for Qlik Sense apps, for example http://sense.mycompany.net/sense/app. App ID will be appended to this URL.
# Links available as template variables in notification messages
genericUrls:
- id: ptarmiganlabs_com
linkText: Ptarmigan Labs home page
comment: The home page of the company behind Butler
url: https://ptarmiganlabs.com
- id: butler_docs
linkText: Butler documentation
comment: The documentation for Butler
url: https://butler.ptarmiganlabs.com
# Settings for monitoring Qlik Sense version info
# Version info is retrieved from the hostname:9032/v1/systeminfo endpoint in Qlik Sense
qlikSenseVersion:
versionMonitor:
enable: false # Should Qlik Sense version info be retrieved?
frequency: every 24 hours # https://bunkat.github.io/later/parsers.html#text
host: <FQDN or IP of Qlik Sense central node>
rejectUnauthorized: false # Set to false to ignore warnings/errors caused by Qlik Sense's self-signed certificates.
destination:
influxDb: # Store version data in InfluxDB.
# If enabled, version info will be stored as measurements in InfluxDB.
enable: false
tag:
static: # Static attributes/tags to attach to the data sent to InflixDB
- name: foo
value: bar
# Settings for monitoring Qlik Sense licenses
qlikSenseLicense:
serverLicenseMonitor:
enable: false
frequency: every 24 hours # https://bunkat.github.io/later/parsers.html#text
alert: # Alert if the number of days left on the license is below the threshold
# License expiry alerts on a global level are enabled here, then configured on
# a per-destination basis elsewhere in this config file.
thresholdDays: 60
destination:
influxDb: # Store license data in InfluxDB
enable: false
tag:
static: # Static attributes/tags to attach to the data sent to InflixDB
- name: foo
value: bar
mqtt:
enable: false
sendRecurring: # Send license data to the MQTT broker at the frequency specified above
enable: true
sendAlert: # Send an MQTT alert if the number of days left on the license is below the threshold
enable: true
webhook:
enable: false
sendRecurring: # Send license data to webhook(s) at the frequency specified above
enable: true
sendAlert: # Send alert to webhook(s) if the number of days left on the license is below
# the threshold or the license has already expired
enable: true
licenseMonitor: # Monitor Qlik Sense accesds license usage
enable: false
frequency: every 6 hours # https://bunkat.github.io/later/parsers.html#text
destination:
influxDb: # Store license data in InfluxDB
enable: false
tag:
static: # Static attributes/tags to attach to the data sent to InflixDB
- name: foo
value: bar
licenseRelease: # Release unused Qlik Sense access licenses
enable: false # true/false. If true, Butler will release unused licenses according to settings below
dryRun: true # true/false. If true, Butler will not actually release any licenses, just log what it would have done.
frequency: every 24 hours # https://bunkat.github.io/later/parsers.html#text
neverRelease: # Various ways of defining which users should never have their licenses released
user: # Users who should never have their licenses released
- userDir: 'INTERNAL'
userId: 'sa_repository'
- userDir: 'INTERNAL'
userId: 'sa_api'
- userDir: 'USERDIR'
userId: 'qs_admin_account'
tag: # Users with these tags will never have their licenses released
- License do not release
- some other tag
customProperty: # Users with these custom properties will never have their licenses released
- name: LicenseManage
value: do-not-release
userDirectory: # List of user directories whose users should never have their licenses released
- INTERNAL
- ADMIN
inactive: Ignore # Ignore/Yes/No. The value is case insensitive
# No = Don't release licenses for users marked as "Inactive=No" in the QMC
# Yes = Don't release licenses for users marked as "Inactive=Yes" in the QMC
# Ignore = Disregard this setting
blocked: Ignore # Ignore/Yes/No, No = Don't release licenses for users marked as "Blocked=No" in the QMC
removedExternally: ignore # Ignore/Yes/No, No = Don't release licenses for users marked as "Removed externally=No" in the QMC
licenseType: # License types to monitor and release
analyzer:
enable: true # Monitor and release Analyzer licenses
releaseThresholdDays: 30 # Number of days a license can be unused before it is released
professional:
enable: true # Monitor and release Professional licenses
releaseThresholdDays: 30 # Number of days a license can be unused before it is released
destination:
influxDb: # Store info about released licenses in InfluxDB
enable: false
tag:
static: # Static attributes/tags to attach to the data sent to InflixDB
- name: foo
value: bar
# Settings for notifications and messages sent to MS Teams
teamsNotification:
enable: false
reloadTaskFailure:
enable: false
webhookURL: <web hook URL from MS Teams>
messageType: formatted # formatted / basic. Formatted means that template file below will be used to create the message.
basicMsgTemplate: 'Qlik Sense reload failed: "{{taskName}}"' # Only needed if message type = basic
rateLimit: 300 # Min seconds between emails for a given taskID. Defaults to 5 minutes.
headScriptLogLines: 10
tailScriptLogLines: 10
templateFile: /path/to/teams/template/directory/failed-reload-qseow.handlebars
reloadTaskAborted:
enable: false
webhookURL: <web hook URL from MS Teams>
messageType: formatted # formatted / basic. Formatted means that template file below will be used to create the message.
basicMsgTemplate: 'Qlik Sense reload aborted: "{{taskName}}"' # Only needed if message type = basic
rateLimit: 300 # Min seconds between emails for a given taskID. Defaults to 5 minutes.
headScriptLogLines: 10
tailScriptLogLines: 10
templateFile: /path/to/teams/template/directory/aborted-reload-qseow.handlebars
serviceStopped:
webhookURL: <web hook URL from MS Teams>
messageType: formatted # formatted / basic. Formatted means that template file below will be used to create the message.
basicMsgTemplate: 'Windows service stopped: "{{serviceName}}" on host "{{host}}"' # Only needed if message type = basic
rateLimit: 30 # Min seconds between messages for a given Windows service. Defaults to 5 minutes.
templateFile: /path/to/teams/template/directory/service-stopped.handlebars
serviceStarted:
webhookURL: <web hook URL from MS Teams>
messageType: formatted # formatted / basic. Formatted means that template file below will be used to create the message.
basicMsgTemplate: 'Windows service started: "{{serviceName}}" on host "{{host}}"' # Only needed if message type = basic
rateLimit: 30 # Min seconds between messages for a given Windows service. Defaults to 5 minutes.
templateFile: /path/to/teams/template/directory/service-started.handlebars
# Settings for notifications and messages sent to Slack
slackNotification:
enable: false
restMessage:
webhookURL: <web hook URL from Slack> # Webhook to use when sending basic Slack messages via Butler's REST API
reloadTaskFailure: # Reload task failed in QSEoW
enable: false
webhookURL: <web hook URL from Slack>
channel: sense-task-failure # Slack channel to which task failure notifications are sent
messageType: formatted # formatted / basic. Formatted means that template file below will be used to create the message.
basicMsgTemplate: 'Qlik Sense reload failed: "{{taskName}}"' # Only needed if message type = basic
rateLimit: 300 # Min seconds between emails for a given taskID. Defaults to 5 minutes.
headScriptLogLines: 10
tailScriptLogLines: 10
templateFile: /path/to/slack/template/directory/failed-reload-qseow.handlebars
fromUser: Qlik Sense
iconEmoji: ':ghost:'
reloadTaskAborted: # Reload task aborted in QSEoW
enable: false
webhookURL: <web hook URL from Slack>
channel: sense-task-aborted # Slack channel to which task stopped notifications are sent
messageType: formatted # formatted / basic. Formatted means that template file below will be used to create the message.
basicMsgTemplate: 'Qlik Sense reload aborted: "{{taskName}}"' # Only needed if message type = basic
rateLimit: 300 # Min seconds between emails for a given taskID. Defaults to 5 minutes.
headScriptLogLines: 10
tailScriptLogLines: 10
templateFile: /path/to/slack/template/directory/aborted-reload-qseow.handlebars
fromUser: Qlik Sense
iconEmoji: ':ghost:'
serviceStopped:
webhookURL: <web hook URL from Slack>
channel: qliksense-service-alert # Slack channel to which Windows service stopped notifications are sent
messageType: formatted # formatted / basic. Formatted means that template file below will be used to create the message.
basicMsgTemplate: 'Windows service stopped: "{{serviceName}}" on host "{{host}}"' # Only needed if message type = basic
rateLimit: 30 # Min seconds between messages for a given Windows service. Defaults to 5 minutes.
templateFile: /path/to/slack/template/directory/service-stopped.handlebars
fromUser: Qlik Sense
iconEmoji: ':ghost:'
serviceStarted:
webhookURL: <web hook URL from Slack>
channel: qliksense-service-alert # Slack channel to which Windows service stopped notifications are sent
messageType: formatted # formatted / basic. Formatted means that template file below will be used to create the message.
basicMsgTemplate: 'Windows service started: "{{serviceName}}" on host "{{host}}"' # Only needed if message type = basic
rateLimit: 30 # Min seconds between messages for a given Windows service. Defaults to 5 minutes.
templateFile: /path/to/slack/template/directory/service-started.handlebars
fromUser: Qlik Sense
iconEmoji: ':ghost:'
# Settings needed to send email notifications when for example reload tasks fail.
# Reload failure notifications assume a log appender is configured in Sense AND that the UDP server in Butler is running.
emailNotification:
enable: false
reloadTaskSuccess:
enable: false
# Custom property used to control which task successes will cause alert emails to be sent
# If this setting is true, alerts will not be sent for all tasks, but *only* for tasks with the CP set to the enabledValue.
# If this setting is false, alerts will be sent for all failed reload tasks.
alertEnableByCustomProperty:
enable: false
customPropertyName: 'Butler_SuccessAlertEnableEmail'
enabledValue: 'Yes'
# Custom property used to say that alerts for a certain task should be sent to zero or more recipients
# These alerts will be sent irrespective of the alertEnableByCustomProperty.enable setting.
alertEnabledByEmailAddress:
customPropertyName: 'Butler_SuccessAlertSendToEmail'
rateLimit: 60 # Min seconds between emails for a given taskID/recipient combo. Defaults to 5 minutes.
headScriptLogLines: 15
tailScriptLogLines: 25
priority: high # high/normal/low
subject: '✅ Qlik Sense reload success: "{{taskName}}"'
bodyFileDirectory: path/to/email_templates
htmlTemplateFile: success-reload-qseow
fromAddress: Qlik Sense (no-reply) <qliksense-noreply@ptarmiganlabs.com>
recipients:
- <Email address 1>
- <Email address 2>
reloadTaskAborted:
enable: false
appOwnerAlert:
enable: true # Should app owner get notification email (assuming email address is available in Sense user directory)
includeOwner:
includeAll: true # true = Send notification to all app owners except those in exclude list
# false = Send notification to app owners in the include list
user:
- directory: <Sense user directory>
userId: <userId>
- directory: <Sense user directory>
userId: <userId>
excludeOwner:
user:
- directory: <Sense user directory>
userId: <userId>
- directory: <Sense user directory>
userId: <userId>
# Custom property used to control which aborted tasks will cause alert emails to be sent
# If this setting is true, alerts will not be sent for all tasks, but *only* for tasks with the CP set to the enabledValue.
# If this setting is false, alerts will be sent for all aborted reload tasks.
alertEnableByCustomProperty:
enable: true
customPropertyName: 'Butler_AbortedAlertEnableEmail'
enabledValue: 'Yes'
# Custom property used to say that alerts for a certain task should be sent to zero or more recipients
# These alerts will be sent irrespective of the alertEnableByCustomProperty.enable setting.
alertEnabledByEmailAddress:
customPropertyName: 'Butler_AbortedAlertSendToEmail'
rateLimit: 600 # Min seconds between emails for a given taskID. Defaults to 5 minutes.
headScriptLogLines: 15 # Number of lines from start of script to include in email
tailScriptLogLines: 15 # Number of lines from end of script to include in email
priority: high # high/normal/low
subject: 'Qlik Sense reload aborted: "{{taskName}}"' # Email subject. Can use template fields
bodyFileDirectory: path/to/email_templates # Directory where email body template files are stored
htmlTemplateFile: aborted-reload # Name of email body template file to use
fromAddress: Qlik Sense (no-reply) <qliksense-noreply@mydomain.com>
recipients: # Array of email addresses to which the notification email will be sent
- <Email address 1>
- <Email address 2>
reloadTaskFailure:
enable: false
appOwnerAlert:
enable: true # Should app owner get notification email (assuming email address is available in Sense user directory)
includeOwner:
includeAll: true # true = Send notification to all app owners except those in exclude list
# false = Send notification to app owners in the include list
user:
- directory: <Sense user directory>
userId: <userId>
- directory: <Sense user directory>
userId: <userId>
excludeOwner:
user:
- directory: <Sense user directory>
userId: <userId>
- directory: <Sense user directory>
userId: <userId>
# Custom property used to control which task failures will cause alert emails to be sent
# If this setting is true, alerts will not be sent for all tasks, but *only* for tasks with the CP set to the enabledValue.
# If this setting is false, alerts will be sent for all failed reload tasks.
alertEnableByCustomProperty:
enable: false
customPropertyName: 'Butler_FailedAlertEnableEmail'
enabledValue: 'Yes'
# Custom property used to say that alerts for a certain task should be sent to zero or more recipients
# These alerts will be sent irrespective of the alertEnableByCustomProperty.enable setting.
alertEnabledByEmailAddress:
customPropertyName: 'Butler_FailedAlertSendToEmail'
rateLimit: 600 # Min seconds between emails for a given taskID. Defaults to 5 minutes.
headScriptLogLines: 15 # Number of lines from start of script to include in email
tailScriptLogLines: 15 # Number of lines from end of script to include in email
priority: high # high/normal/low
subject: 'Qlik Sense reload failed: "{{taskName}}"' # Email subject. Can use template fields
bodyFileDirectory: path/to/email_templates # Directory where email body template files are stored
htmlTemplateFile: failed-reload # Name of email body template file to use
fromAddress: Qlik Sense (no-reply) <qliksense-noreply@mydomain.com>
recipients: # Array of email addresses to which the notification email will be sent
- <Email address 1>
- <Email address 2>
serviceStopped:
rateLimit: 30 # Min seconds between emails for a given service. Defaults to 5 minutes.
priority: high # high/normal/low
subject: '❌ Windows service stopped on host {{host}}: "{{serviceDisplayName}}"'
bodyFileDirectory: path/to/email_templates/email_templates
htmlTemplateFile: service-stopped
fromAddress: Qlik Sense (no-reply) <qliksense-noreply@mydomain.com>
recipients:
- <Email address 1>
- <Email address 2>
serviceStarted:
rateLimit: 30 # Min seconds between emails for a given service. Defaults to 5 minutes.
priority: high # high/normal/low
subject: '✅ Windows service started on host {{host}}: "{{serviceDisplayName}}"'
bodyFileDirectory: path/to/email_templates/email_templates
htmlTemplateFile: service-started
fromAddress: Qlik Sense (no-reply) <qliksense-noreply@mydomain.com>
recipients:
- <Email address 1>
- <Email address 2>
smtp: # Email server settings. See https://nodemailer.com/smtp/ for details on the meaning of these fields.
host: <FQDN or IP or email server, e.g. smtp.gmail.com>
port: <port on which SMTP server is listening>
secure: true # true/false
tls:
serverName: # If specified the serverName field will be used for TLS verification instead of the host field.
ignoreTLS: false
requireTLS: true
rejectUnauthorized: false
auth:
enable: true
user: <Username, email address etc>
password: <your-secret-password>
# Incident management tools integration
# Used to trigger incidents in these tools when task reloads fail or are aborted.
incidentTool:
signl4:
enable: false # Enable/disable Signl4 integration as a whole
url: https://connect.signl4.com/webhook/abcde12345
reloadTaskFailure:
enable: false # Enable/disable reload failed handling in Signl4
rateLimit: 15 # Min seconds between emails for a given taskID. Defaults to 5 minutes
serviceName: Qlik Sense # Signl4 "service name" to use
severity: 1 # Signl4 severity level for failed reloads
includeApp:
includeAll: false
appId:
- 47d38f73-628f-44e1-a62c-841604b123ff
reloadTaskAborted:
enable: false # Enable/disable reload aborted handling in Signl4
rateLimit: 15 # Min seconds between emails for a given taskID. Defaults to 5 minutes
serviceName: Qlik Sense # Signl4 "service name" to use
severity: 10 # Signl4 severity level for aborted reloads
includeApp:
includeAll: false
appId:
- 47d38f73-628f-44e1-a62c-841604b123ff
newRelic:
enable: false
destinationAccount:
event: # Failed/aborted reload tasks are sent as events to these New Relic accounts
# - First NR account
# - Second NR account
log: # Failed/aborted reload tasks are sent as log entries to these New Relic accounts
# - First NR account
# - Second NR account
# New Relic uses different API URLs for different kinds of data (metrics, events, logs, ...)
# There are different URLs depending on whther you have an EU or US region New Relic account.
# The available URLs are listed here: https://docs.newrelic.com/docs/accounts/accounts-billing/account-setup/choose-your-data-center/
url:
# As of this writing the valid options are
# https://insights-collector.eu01.nr-data.net
# https://insights-collector.newrelic.com
event: https://insights-collector.eu01.nr-data.net
# Valid options are (1) EU/rest of world and 2) US)
# https://log-api.eu.newrelic.com/log/v1
# https://log-api.newrelic.com/log/v1
log: https://log-api.eu.newrelic.com/log/v1
reloadTaskFailure:
destination:
event:
enable: false
sendToAccount: # Which reload task failures are sent to New Relic as events
byCustomProperty:
enable: false # Control using a task custom property which reload task failures are sent as events
customPropertyName: 'Butler_FailedTask_Event_NewRelicAccount'
always:
enable: false # Controls which New Relic accounts ALL failed reload tasks are sent to (as events)
account:
# - First NR account
# - Second NR account
attribute:
static: # Static attributes/dimensions to attach to events sent to New Relic.
- name: event-specific-attribute 1 # Example
value: abc 123 # Example
dynamic:
useAppTags: true # Should app tags be sent to New Relic as attributes?
useTaskTags: true # Should task tags be sent to New Relic as attributes?
log:
enable: false
tailScriptLogLines: 20
sendToAccount: # Which reload task failures are sent to New Relic as log entries
byCustomProperty:
enable: false # Control using a task custom property which reload task failures are sent as log entries
customPropertyName: 'Butler_FailedTask_Log_NewRelicAccount'
always:
enable: false # Controls which New Relic accounts ALL failed reload tasks are sent to (as logs)
account:
# - First NR account
# - Second NR account
attribute:
static: # Static attributes/dimensions to attach to events sent to New Relic.
- name: log-specific-attribute 1 # Example
value: def 123 # Example
dynamic:
useAppTags: true # Should app tags be sent to New Relic as attributes?
useTaskTags: true # Should task tags be sent to New Relic as attributes?
sharedSettings:
rateLimit: 15 # Min seconds between events sent to New Relic for a given taskID. Defaults to 5 minutes.
header: # Custom http headers
- name: X-My-Header # Example
value: Header value 1 # Example
attribute:
static: # Static attributes/dimensions to attach to events sent to New Relic.
- name: service # Example
value: butler # Example
- name: environment # Example
value: prod # Example
reloadTaskAborted:
destination:
event:
enable: false
sendToAccount: # Which reload task aborts are sent to New Relic as events
byCustomProperty:
enable: false # Control using a task custom property which reload task aborts are sent as events
customPropertyName: 'Butler_AbortedTask_Event_NewRelicAccount'
always:
enable: false # Controls which New Relic accounts ALL aborted reload tasks are sent to (as events)
account:
# - First NR account
# - Second NR account
attribute:
static: # Static attributes/dimensions to attach to events sent to New Relic.
- name: event-specific-attribute 2 # Example
value: abc 123 # Example
dynamic:
useAppTags: true # Should app tags be sent to New Relic as attributes?
useTaskTags: true # Should task tags be sent to New Relic as attributes?
log:
enable: false
tailScriptLogLines: 20
sendToAccount: # Which reload task aborts are sent to New Relic as log entries
byCustomProperty:
enable: false # Control using a task custom property which reload task aborts are sent as log entries
customPropertyName: 'Butler_AbortedTask_Log_NewRelicAccount'
always:
enable: false # Controls which New Relic accounts ALL aborted reload tasks are sent to (as logs)
account:
# - First NR account
# - Second NR account
attribute:
static: # Static attributes/dimensions to attach to events sent to New Relic.
- name: log-specific-attribute 2 # Example
value: def 123 # Example
dynamic:
useAppTags: true # Should app tags be sent to New Relic as attributes?
useTaskTags: true # Should task tags be sent to New Relic as attributes?
sharedSettings:
rateLimit: 15 # Min seconds between events sent to New Relic for a given taskID. Defaults to 5 minutes.
header: # Custom http headers
- name: X-My-Header # Example
value: Header value 2 # Example
attribute:
static: # Static attributes/dimensions to attach to events sent to New Relic.
- name: service # Example
value: butler # Example
- name: environment # Example
value: prod # Example
serviceMonitor:
destination:
event:
enable: false
sendToAccount: # Windows service events are sent to these New Relic accounts
# - First NR account
# - Second NR account
attribute:
static: # Static attributes/dimensions to attach to events sent to New Relic.
- name: event-specific-attribute
value: abc 123
dynamic:
serviceHost: true # Should host where service is running be sent to New Relic as attribute?
serviceName: true # Should service name be sent to New Relic as attribute?
serviceDisplayName: true # Should service display name be sent to New Relic as attribute?
serviceState: true # Should service state be sent to New Relic as attribute?
log:
enable: false
sendToAccount: # Windows service log entries are sent to these New Relic accounts
# - First NR account
# - Second NR account
attribute:
static: # Static attributes/dimensions to attach to events sent to New Relic.
- name: log-specific-attribute
value: def 456
dynamic:
serviceHost: true # Should host where service is running be sent to New Relic as attribute?
serviceName: true # Should service name be sent to New Relic as attribute?
serviceDisplayName: true # Should service display name be sent to New Relic as attribute?
serviceState: true # Should service state be sent to New Relic as attribute?
monitorServiceState: # Control whih service states are sent to New Relic
running:
enable: true
stopped:
enable: true
sharedSettings:
rateLimit: 5 # Min seconds between events/logs sent to New Relic for a given host+service. Defaults to 5 minutes.
header: # Custom http headers
- name: X-My-Header # Example
value: Header value 2 # Example
attribute:
static: # Static attributes/dimensions to attach to events sent to New Relic.
- name: service # Example
value: butler # Example
- name: environment # Example
value: prod # Example
# Settings for notifications and messages sent using outgoing webhooks
webhookNotification:
enable: false
reloadTaskFailure:
rateLimit: 300 # Min seconds between outgoing webhook calls for a given taskID. Defaults to 5 minutes.
webhooks:
# - description: 'This outgoing webhook makes a POST and is used to...' # Informational only
# webhookURL: http://host.my.domain:port/some/path # outgoing webhook that Butler will call
# httpMethod: POST # GET/POST/PUT
# cert:
# enable: false # Set to true to use a custom CA certificate when calling the webhookURL
# rejectUnauthorized: true # Set to false to ignore warnings/errors caused by self-signed certificates used on the webhooks server.
# certCA: /path/to/ca-certificate.pem # Path to the CA certificate file
# - description: 'This outgoing webhook makes a PUT and is used to...' # Informational only
# webhookURL: https://host.my.domain:port/some/path # Outgoing webhook that Butler will call
# httpMethod: PUT # GET/POST/PUT.
# cert:
# enable: true # Set to true to use a custom CA certificate when calling the webhookURL
# rejectUnauthorized: false # Set to false to ignore warnings/errors caused by self-signed certificates used on the webhooks server.
# certCA: /path/to/ca-certificate.pem # Path to the CA certificate file
# - description: 'This outgoing webhook makes a GET and is used to ..' # Informational only
# webhookURL: https://host.my.domain:port/some/path # Outgoing webhook that Butler will call
# httpMethod: GET # GET/POST/PUT
# cert:
# enable: true # Set to true to use a custom CA certificate when calling the webhookURL
# rejectUnauthorized: true # Set to false to ignore warnings/errors caused by self-signed certificates used on the webhooks server.
# certCA: /path/to/ca-certificate.pem # Path to the CA certificate file
reloadTaskAborted:
rateLimit: 300 # Min seconds between outgoing webhook calls for a given taskID. Defaults to 5 minutes.
webhooks:
# - description: 'This outgoing webhook makes a GET and is used to ..' # Informational only
# webhookURL: http://host.my.domain:port/some/path # Outgoing webhook that Butler will call
# httpMethod: GET # GET/POST/PUT
# cert:
# enable: true # Set to true to use a custom CA certificate when calling the webhookURL
# rejectUnauthorized: true # Set to false to ignore warnings/errors caused by self-signed certificates used on the webhooks server.
# certCA: /path/to/ca-certificate.pem # Path to the CA certificate file
serviceMonitor:
rateLimit: 300 # Min seconds between outgoing webhook calls, per Windows service that is monitored. Defaults to 5 minutes.
webhooks:
# - description: 'This outgoing webhook makes a POST and is used to...' # Informational only
# webhookURL: http://host.my.domain:port/some/path # Outgoing webhook that Butler will call
# httpMethod: POST # GET/POST/PUT. Note that the body and URL query parameters differs depending on which method is used
# cert:
# enable: true # Set to true to use a custom CA certificate when calling the webhookURL
# rejectUnauthorized: true # Set to false to ignore warnings/errors caused by self-signed certificates used on the webhooks server.
# certCA: /path/to/ca-certificate.pem # Path to the CA certificate file
qlikSenseServerLicenseMonitor: # Outgoing webhook that Butler will call with info on Qlik Sense server license status
rateLimit: 300 # Min seconds between outgoing webhook calls, per Windows service that is monitored. Defaults to 5 minutes.
webhooks:
# - description: 'This outgoing webhook makes a PUT and is used to ...'
# webhookURL: http://host.my.domain:port/some/path
# httpMethod: PUT # GET/POST/PUT. Note that the body and URL query parameters differs depending on which method is used
# cert:
# enable: false # Set to true to use a custom CA certificate when calling the webhookURL
# rejectUnauthorized: true # Set to false to ignore warnings/errors caused by self-signed certificates used on the webhooks server.
# certCA: foo
qlikSenseServerLicenseExpiryAlert: # Outgoing webhook that Butler will call when Qlik Sense server license is about to expire
rateLimit: 300 # Min seconds between outgoing webhook calls, per Windows service that is monitored. Defaults to 5 minutes.
webhooks:
# - description: 'This outgoing webhook makes a POST and is used to ...'
# webhookURL: https://host.my.domain:port/some/path
# httpMethod: POST # GET/POST/PUT. Note that the body and URL query parameters differs depending on which method is used
# cert:
# enable: true # Set to true to use a custom CA certificate when calling the webhookURL
# rejectUnauthorized: true # Set to false to ignore warnings/errors caused by self-signed certificates used on the webhooks server.
# certCA: /path/to/ca-certificate.pem # Path to the CA certificate file
# Scheduler for Qlik Sense tasks
scheduler:
enable: false # Should Butler's reload task scheduler be started?
configfile: config/schedule.yaml # Path to file containing task start schedules
# Key-value store
keyValueStore:
enable: false # Should Butler's key-value store be enabled?
maxKeysPerNamespace: 1000 # Max keys that can be stored per namespace. Defaults to 1000 if not specified in this file.
mqttConfig:
enable: false # Should Qlik Sense events be forwarded as MQTT messages?
brokerHost: <FQDN or IP of MQTT server>
brokerPort: 1883
azureEventGrid:
enable: false # If set to true, Butler will connect to an Azure Event Grid MQTT Broker, using brokerHost and brokerPort above
clientId: <client ID>
clientCertFile: <path to client certificate file>
clientKeyFile: <path to client key file>
taskFailureSendFull: true
taskAbortedSendFull: true
subscriptionRootTopic: qliksense/# # Topic that Butler will subscribe to
taskStartTopic: qliksense/start_task # Topic for incoming messages used to start Sense tasks. Should be subtopic to subscriptionRootTopic
taskFailureTopic: qliksense/task_failure
taskFailureFullTopic: qliksense/task_failure_full
taskFailureServerStatusTopic: qliksense/butler/task_failure_server
taskAbortedTopic: qliksense/task_aborted
taskAbortedFullTopic: qliksense/task_aborted_full
serviceRunningTopic: qliksense/service_running
serviceStoppedTopic: qliksense/service_stopped
serviceStatusTopic: qliksense/service_status
qlikSenseServerLicenseTopic: qliksense/qliksense_server_license # Topic to which Sense server license info is published
qlikSenseServerLicenseExpireTopic: qliksense/qliksense_server_license_expire # Topic to which Sense server license expiration alerts are published
qlikSenseCloud: # MQTT settings for Qlik Sense Cloud integration
event:
mqttForward: # QS Cloud events forwarded to MQTT topics, which Butler will subscribe to
enable: false
broker: # Settings for MQTT broker to which QS Cloud events are forwarded
host: mqttbroker.company.com
port: <port>
username: <username>
password: <password>
topic:
subscriptionRoot: qscloud/# # Topic that Butler will subscribe to
appReload: qscloud/app/reload
udpServerConfig:
enable: false # Should the UDP server responsible for receving task failure/aborted events be started?
serverHost: <FQDN or IP (or localhost) of server where Butler is running>
portTaskFailure: 9998
restServerConfig:
enable: false # Should Butler's REST API be started? Must be true if *any* API endpoints are to be used.
serverHost: <FQDN or IP (or localhost) of server where Butler is running> # Use 0.0.0.0 to listen on all network interfaces (e.g. when running in Docker!).
serverPort: 8080 # Port where Butler's REST is available. Any free port on the server where Butler is running can bse used.
backgroundServerPort: 8081 # Port used internally by Butler's REST API. Any free port on the server where Butler is running can bse used.
# List of directories between which file copying via the REST API can be done.
# Butler will try to clean up messy paths like this one, which resolves to /Users/goran/butler-test-dir1
# How? First you have /Users/goran/butler-test-dir1//abc which cleans up to /Users/goran/butler-test-dir1/abc,
# then up one level (..).
fileCopyApprovedDirectories:
# - fromDirectory: /Users/goran/butler-test-dir1//abc//..
# toDirectory: /Users/goran/butler-test-dir2
# - fromDirectory: /Users/goran/butler-test-dir2
# toDirectory: /Users/goran/butler-test-dir1
# - fromDirectory: /from/some/directory2
# toDirectory: /to/some/directory2
# List of directories between which file moves via the REST API can be done.
fileMoveApprovedDirectories:
# - fromDirectory: /Users/goran/butler-test-dir1//abc//..
# toDirectory: /Users/goran/butler-test-dir2
# - fromDirectory: /Users/goran/butler-test-dir2
# toDirectory: /Users/goran/butler-test-dir1
# - fromDirectory: /from/some/directory2
# toDirectory: /to/some/directory2
# List of directories in which file deletes via the REST API can be done.
fileDeleteApprovedDirectories:
# - /Users/goran/butler-test-dir1
# - /Users/goran/butler-test-dir1//abc//..
# - /Users/goran/butler-test-dir2
# If set to true, Butler will be started with a focus on creating an API documentation file
# All configuration relating to outbound connetions (to Sense, email servers, MQTT broker etc) will be disabled.
# NOTE: This setting should always be false (or just deleted), unless you want to regenerate the API doc files.
restServerApiDocGenerate: false
# Enable/disable individual REST API endpoints. Set config item below to true to enable that endpoint.
restServerEndpointsEnable:
apiListEnbledEndpoints: false
base62ToBase16: false
base16ToBase62: false
butlerping: false
createDir: false
createDirQVD: false
fileDelete: false
fileMove: false
fileCopy: false
keyValueStore: false
mqttPublishMessage: false
newRelic:
postNewRelicMetric: false
postNewRelicEvent: false
scheduler:
createNewSchedule: false
getSchedule: false
getScheduleStatusAll: false
updateSchedule: false
deleteSchedule: false
startSchedule: false
stopSchedule: false
senseAppReload: false
senseAppDump: false
senseListApps: false
senseStartTask: false
slackPostMessage: false
restServerEndpointsConfig:
newRelic:
postNewRelicMetric: # Setings used by post metric to New Relic API endpoint
destinationAccount:
# - First NR account
# - Second NR account
# As of this writing the valid options are
# https://insights-collector.eu01.nr-data.net/metric/v1
# https://insights-collector.newrelic.com/metric/v1
url: https://insights-collector.eu01.nr-data.net/metric/v1
header: # Custom http headers
- name: X-My-Header
value: Header value
attribute:
static: # Static attributes/dimensions to attach to the metrics data sent to New Relic.
- name: env
value: prod
postNewRelicEvent: # Setings used by post event to New Relic API endpoint
destinationAccount:
# - First NR account
# - Second NR account
# Note that the URL path should *not* be included in the url setting below!
# As of this writing the valid options are
# https://insights-collector.eu01.nr-data.net
# https://insights-collector.newrelic.com
url: https://insights-collector.eu01.nr-data.net/
header: # Custom http headers
- name: X-My-Header
value: Header value
attribute:
static: # Static attributes/dimensions to attach to the metrics data sent to New Relic.
- name: env
value: prod
# Controls which tasks can be started via Butler's REST API.
# Enabling this feature gives Qlik Sense sysadmins a way to control which tasks can be started by third party systems and applications.
# If this feature is disabled all tasks can be started via the API (assuming the start task API itself is enabled).
# Note that the taskId, tag and customProperty sections below are additive. I.e. a task only has to appear in one of those sections to be on the "allowed" list.
startTaskFilter:
enable: false
allowTask:
taskId:
# Zero or more approved/allowed task IDs
# If Butler.startTaskFilter.enable is true, only task IDs listed below will be started by Butler
# - e3b27f50-b1c0-4879-88fc-c7cdd9c1cf3e
# - 7552d9fc-d1bb-4975-9a38-18357de531ea
# - fb0f317d-da91-4b86-aafa-0174ae1e8c8f
tag:
# Zero or more tags used to indicate that a task is approved to be started by Butler.
# Use the Qlik Sense QMC to set tags on tasks.
# If Butler.startTaskFilter.enable is true, only tasks with the tags below will be started by Butler
# - startTask1
# - startTask2
customProperty:
# Zero or more custom properties name/value pairs used to indicate that a task is approved to be started by Butler.
# Use the Qlik Sense QMC to set custom properties on tasks.
# If Butler.startTaskFilter.enable is true, only tasks with the custom property values below will be started by Butler
# - name: taskGroup
# value: tasks1
# - name: taskGroup
# value: tasks2
# Monitor Windows services.
# This feature only works when Butler is running on Windows Server or desktop.
# On other OSs service monitoring will be automatically disabled.
serviceMonitor:
enable: false # Main on/off switch for service monitoring
frequency: every 30 seconds # https://bunkat.github.io/later/parsers.html#text
monitor:
# - host: <hostname or IP> # Host name of Windows computer where services are running
# services: # List of services to monitor
# - name: postgresql-x64-12 # Posgress/repository db
# friendlyName: Repository DB
# - name: QlikSenseEngineService
# friendlyName: Engine
# - name: QlikSensePrintingService
# friendlyName: Printing
# - name: QlikSenseProxyService
# friendlyName: Proxy
# - name: QlikSenseRepositoryService
# friendlyName: Repository
# - name: QlikSenseSchedulerService
# friendlyName: Scheduler
# - name: QlikSenseServiceDispatcher
# friendlyName: Service Dispatcher
alertDestination: # Control to thich destinations service related alerts are sent
influxDb: # Send service alerts to InfluxDB
enable: true
newRelic: # Send service alerts to New Relic
enable: true
email: # Send service alerts as emails
enable: true
mqtt: # Send service alerts as MQTT messages
enable: true
teams: # Send service alerts as MS Teams messages
enable: true
slack: # Send service alerts as Slack messages
enable: true
webhook: # Send service alerts as outbound webhooks/http calls
enable: true
qlikSenseCloud: # Settings for Qlik Sense Cloud integration
enable: false
event:
mqtt: # Which QS Cloud tenant should Butler receive events from, in the form of MQTT messages?
tenant:
id: tenant.region.qlikcloud.com
tenantUrl: https://tenant.region.qlikcloud.com
authType: jwt # Authentication type used to connect to the tenant. Valid options are "jwt"
auth:
jwt:
token: <JWT token> # JWT token used to authenticate Butler when connecting to the tenant
# Qlik Sense Cloud related links used in notification messages
qlikSenseUrls:
qmc: <URL to QMC in Qlik Sense Cloud>
hub: <URL to Hub in Qlik Sense Cloud>
comment: This is a comment describing the tenant and its settings # Informational only
alert:
# Settings for notifications and messages sent to MS Teams
teamsNotification:
reloadAppFailure:
enable: false
alertEnableByTag:
enable: false
tag: Butler - Send Teams alert if app reload fails
basicContentOnly: false
webhookURL: <URL to MS Teams webhook>
messageType: formatted # formatted / basic
basicMsgTemplate: 'Qlik Sense Cloud app reload failed: "{{appName}}"' # Only needed if message type = basic
rateLimit: 15 # Min seconds between emails for a given appId. Defaults to 5 minutes.
headScriptLogLines: 15
tailScriptLogLines: 15
templateFile: /path/to/teams_templates/failed-reload-qscloud-workflow.handlebars
# Settings for notifications and messages sent to Slack
slackNotification:
reloadAppFailure:
enable: false
alertEnableByTag:
enable: false
tag: Butler - Send Slack alert if app reload fails
basicContentOnly: false
webhookURL: <URL to Slack webhook>
channel: sense-task-failure # Slack channel to which task failure notifications are sent
messageType: formatted # formatted / basic. Formatted means that template file below will be used to create the message.
basicMsgTemplate: 'Qlik Sense Cloud app reload failed: "{{appName}}"' # Only needed if message type = basic
rateLimit: 60 # Min seconds between emails for a given appId. Defaults to 5 minutes.
headScriptLogLines: 10
tailScriptLogLines: 20
templateFile: /path/to/slack_templates/failed-reload-qscloud.handlebars
fromUser: Qlik Sense
iconEmoji: ':ghost:'
# Settings needed to send email notifications when for example reload tasks fail.
# Reload failure notifications assume a log appender is configured in Sense AND that the UDP server in Butler is running.
emailNotification:
reloadAppFailure:
enable: false # Enable/disable app reload failed notifications via email
alertEnableByTag:
enable: false
tag: Butler - Send email if app reload fails
appOwnerAlert:
enable: false # Should app owner get notification email (assuming email address is available in Sense)?
includeOwner:
includeAll: true # true = Send notification to all app owners except those in exclude list
# false = Send notification to app owners in the include list
user: # Array of app owner email addresses that should get notifications
# - email: anna@somecompany.com
# - email: joe@somecompany.com
excludeOwner:
user:
# - email: daniel@somecompany.com
rateLimit: 60 # Min seconds between emails for a given appId/recipient combo. Defaults to 5 minutes.
headScriptLogLines: 15
tailScriptLogLines: 25
priority: high # high/normal/low
subject: '❌ Qlik Sense reload failed: "{{taskName}}"'
bodyFileDirectory: /path/to//email_templates
htmlTemplateFile: failed-reload-qscloud
fromAddress: Qlik Sense (no-reply) <qliksense-noreply@ptarmiganlabs.com>
recipients:
# - emma@somecompany.com
# - patrick@somecompany.com
# Certificates to use when connecting to Sense. Get these from the Certificate Export in QMC.
cert:
clientCert: <path/to/cert/client.pem>
clientCertKey: <path/to/cert/client_key.pem>
clientCertCA: <path/to/cert/root.pem>
# If running Butler in a Docker container, the cert paths MUST be the following
# clientCert: /nodeapp/config/certificate/client.pem
# clientCertKey: /nodeapp/config/certificate/client_key.pem
# clientCertCA: /nodeapp/config/certificate/root.pem
configEngine:
engineVersion: 12.612.0 # Qlik Associative Engine version to use with Enigma.js. Works with Feb 2020 and others
host: <FQDN or IP of Sense server where Sense Engine is running>
port: <Port to connect to, usually 4747>
useSSL: true
headers:
static: # http headers that are sent with every request to QRS. The "X-Qlik-User" is mandatory.
- name: X-Qlik-User # Header used to identify what user connection to QRS is made as
value: UserDirectory=Internal;UserId=sa_repository # What user connection to QRS is made as
rejectUnauthorized: false # Set to false to ignore warnings/errors caused by Qlik Sense's self-signed certificates.
# Set to true if the Qlik Sense root CA is available on the computer where Butler is running.
configQRS:
authentication: certificates
host: <FQDN or IP of Sense server where QRS is running>
useSSL: true
port: <Port to connect to, usually 4242>
headers:
static: # http headers that are sent with every request to QRS. The "X-Qlik-User" is mandatory.
- name: X-Qlik-User # Header used to identify what user connection to QRS is made as
value: UserDirectory=Internal;UserId=sa_repository # What user connection to QRS is made as
rejectUnauthorized: false # Set to false to ignore warnings/errors caused by Qlik Sense's self-signed certificates.
# Set to true if the Qlik Sense root CA is available on the computer where Butler is running.
configDirectories:
qvdPath: <Path to folder under which QVDs are stored>
Comments
-
Note that you can enable/disable most features independently of each other. This means Butler can be configured to do exactly what you need, and nothing more.
-
The default location cert/key files are found in (assuming a standard install of Qlik Sense)
C:\ProgramData\Qlik\Sense\Repository\Exported Certificates\<name specified during certificate export>The files needed by Butler are
client.pem,client_key.pemandroot.pem.
3 - REST API documentation
OpenAPI/Swagger style documentation for the Butler API.
Remember
The “Try it out” feature of the API documentation below does not work when accessed from butler.ptarmiganlabs.com. This is only expected as this site does not know anything about where your Butler instance is running.
The same feature is however also available from Butler itself, see this page.
4 - REST API documentation (alternative format)
Alternative documentation for the Butler API.
Butler API documentation (9.4.0)
Download OpenAPI specification:Download
Butler is a microservice that provides add-on features to Qlik Sense Enterprise on Windows. Butler offers both a REST API and things like failed reload notifications etc.
This page contains the API documentation. Full documentation is available at https://butler.ptarmiganlabs.com
Get an array of all enabled API endpoints.
Get an array of all enabled API endpoints, using the key names from the Butler config file.
Note: Endpoints are enabled/disabled in the Butler main configuration file.
Responses
Response samples
- 200
Content type
application/json
[- "activeUserCount",
- "activeUsers",
- "apiListEnbledEndpoints"
]Converts strings from base62 to base16.
Converts strings from base62 to base16.
query Parameters
| base62 required | string Example: base62=6DMW88LpSok9Z7P7hUK0wv7bF The base62 encoded string that should be converted to base16 |
Responses
Response samples
- 200
- 400
- 500
Content type
application/json
{- "base62": "6DMW88LpSok9Z7P7hUK0wv7bF",
- "base16": "3199af08bfeeaf5d420f27ed9c01e74370077"
}Converts strings from base16 to base62.
Converts strings from base16 to base62.
query Parameters
| base16 required | string Example: base16=3199af08bfeeaf5d420f27ed9c01e74370077 The base16 encoded string that should be converted to base62 |
Responses
Response samples
- 200
- 400
- 500
Content type
application/json
{- "base16": "3199af08bfeeaf5d420f27ed9c01e74370077",
- "base62": "6DMW88LpSok9Z7P7hUK0wv7bF"
}Copy file(s) between well defined, approved locations.
Copying of files is only posttible between pre-approved directories. Defining approved source and destination directories is done in Butler's config file.
If the source directory contains subdirectories, these will be copied too.
Request Body schema: application/json
| fromFile | string Name of source file. |
| toFile | string Name of destination file. Can be different from source file name, if needed. |
| overwrite | boolean Controls whether destination file should be overwritten if it already exists. Note that the copy operation will silently fail if you set this to false and the destination exists. Defaults to false. |
| preserveTimestamp | boolean When true, the timestamp of the source file(s) will be preserved on the destination file(s). When false, timestamp behaviour is OS-dependent. Defaults to false. |
Responses
Request samples
- Payload
Content type
application/json
{- "fromFile": "subfolder/file1.qvd",
- "toFile": "archive/file1_20200925.qvd",
- "overwrite": false,
- "preserveTimestamp": false
}Response samples
- 201
- 400
- 403
- 500
Content type
application/json
{- "fromFile": "subfolder/file1.qvd",
- "toFile": "archive/file1_20200925.qvd",
- "overwrite": false,
- "preserveTimestamp": false
}Move file(s) between well defined, approved locations.
Moving of files is only posttible between pre-approved directories. Defining approved source and destination directories is done in Butler's config file.
If the source directory contains subdirectories, these will be moved too.
Request Body schema: application/json
| fromFile | string Name of source file. |
| toFile | string Name of destination file. Can be different from source file name, if needed. |
| overwrite | boolean Controls whether destination file should be overwritten if it already exists. Defaults to false. |
Responses
Request samples
- Payload
Content type
application/json
{- "fromFile": "subfolder/file1.qvd",
- "toFile": "archive/file1_20200925.qvd",
- "overwrite": false
}Response samples
- 201
- 400
- 403
- 500
Content type
application/json
{- "fromFile": "subfolder/file1.qvd",
- "toFile": "archive/file1_20200925.qvd",
- "overwrite": false
}Delete file(s) in well defined, approved locations.
It is only possible to delete files in pre-approved directories, or subdirectories thereof. Defining approved directories is done in Butler's config file.
Request Body schema: application/json
| deleteFile | string Name of file to be deleted. Use forward/backward slashes in paths as needed, depending on whether Butler runs on Windows/non-Windows platform. |
Responses
Request samples
- Payload
Content type
application/json
{- "deleteFile": "data/qvdstore/sales/file1.qvd"
}Response samples
- 204
- 400
- 403
- 500
Content type
application/json
{ }Creates a directory in designated QVD directory.
Creates a directory in QVD directory (which is defined in Butler's config file).
Request Body schema: application/json
| directory | string Directory that should be created. |
Responses
Request samples
- Payload
Content type
application/json
{- "directory": "subfolder/2020-10"
}Response samples
- 201
- 400
- 500
Content type
application/json
{- "directory": "subfolder/2020-10"
}Creates a directory anywhere in the file system.
If the directory already exists nothing will happen. If permissions don't allow a directory to be created, or if the path is invalid, an error will be returned.
Request Body schema: application/json
| directory | string Path to directory that should be created. Can be a relative or absolute path. |
Responses
Request samples
- Payload
Content type
application/json
{- "directory": "/Users/joe/data/qvds/2020"
}Response samples
- 201
- 400
- 500
Content type
application/json
{- "directory": "/Users/joe/data/qvds/2020"
}Get the value associated with a key, in a specific namespace.
path Parameters
| namespace required | string Example: Sales ETL step 2 |
query Parameters
| key required | string Example: key=Last extract timestamp |
Responses
Response samples
- 200
- 400
- 500
Content type
application/json
{- "namespace": "Sales ETL step 2",
- "key": "Last extract timestamp",
- "value": "2020-09-29 17:14:56"
}Create a new key-value pair in the specified namespace.
If the specified key already exists it will be overwritten.
If the posted data has a TTL, it will start counting when the post occur. I.e. if a previouly posted key also had a TTL, it will be replace with the most recent TTL.
path Parameters
| namespace required | string Example: Sales ETL step 2 Name of namespace. |
Request Body schema: application/json
| key | string Key to use |
| value | string Value to set |
| ttl | number Time to live = how long (milliseconds) the key-value pair should exist before being automatically deleted |
Responses
Request samples
- Payload
Content type
application/json
{- "key": "ce68c8ca-b3ff-4371-8285-7c9ce5040e42_parameter_1",
- "value": "12345.789",
- "ttl": 10000
}Response samples
- 201
- 400
- 500
Content type
application/json
{- "namespace": "Sales ETL step 2",
- "key": "Last extract timestamp",
- "value": "2020-09-29 17:14:56",
- "ttl": 60000
}Checks if a key exists in a namespace.
Returns true if the specified key exists, otherwise false.
path Parameters
| namespace required | string Example: Sales ETL step 2 |
query Parameters
| key required | string Example: key=Last extract timestamp |
Responses
Response samples
- 200
- 400
- 500
Content type
application/json
{- "keyExists": true,
- "keyValue": {
- "namespace": "Sales ETL step 2",
- "key": "Last extract timestamp",
- "value": "2020-09-29 17:14:56"
}
}Retrieve a list of all keys present in the specified namespace.
path Parameters
| namespace required | string Example: Sales ETL step 2 Name of namespace whose keys should be returned. |
Responses
Response samples
- 200
- 400
- 500
Content type
application/json
{- "namespace": "Sales ETL step 2",
- "keys": [
- {
- "key": "ce68c8ca-b3ff-4371-8285-7c9ce5040e42_parameter_1"
}, - {
- "key": "ce68c8ca-b3ff-4371-8285-7c9ce5040e42_parameter_2"
}
]
}Publish a message to a MQTT topic.
Request Body schema: application/jsonrequired
| topic required | string Topic to which message should be published. |
| message required | string The message is a generic text string and can thus contain anything that can be represented in a string, including JSON, key-value pairs, plain text etc. |
Responses
Request samples
- Payload
Content type
application/json
{- "topic": "qliksense/new_data_notification/sales",
- "message": "dt=20201028"
}Response samples
- 201
- 400
- 500
Content type
application/json
{- "topic": "qliksense/new_data_notification/sales",
- "message": "dt=20201028"
}Post events to New Relic.
This endpoint posts events to the New Relic event API.
Request Body schema: application/jsonrequired
| eventType required | string <= 254 characters Event type. Can be a combination of alphanumeric characters, _ underscores, and : colons. |
| timestamp | number The event's start time in Unix time. Uses UTC time zone. This field also support seconds, microseconds, and nanoseconds. However, the data will be converted to milliseconds for storage and query. Events reported with a timestamp older than 48 hours ago or newer than 24 hours from the time they are reported are dropped by New Relic. If left empty Butler will use the current time as timestamp. |
Array of objects Dimensions/attributs that will be associated with the event. |
Responses
Request samples
- Payload
Content type
application/json
{- "eventType": "relead-failed",
- "timestamp": 1642164296053,
- "attributes": [
- {
- "name": "host.name",
- "value": "dev.server.com"
}
]
}Response samples
- 202
- 400
- 500
Content type
application/json
{- "newRelicResultCode": "202",
- "newRelicResultText": "Data accepted."
}Post metrics to New Relic.
This endpoint posts metrics to the New Relic metrics API.
Request Body schema: application/jsonrequired
| name required | string <= 254 characters Metric name. |
| type required | string Value: "gauge" Metric type. |
| value required | number Value of the metric. |
| timestamp | number The metric's start time in Unix time. Uses UTC time zone. This field also support seconds, microseconds, and nanoseconds. However, the data will be converted to milliseconds for storage and query. Metrics reported with a timestamp older than 48 hours ago or newer than 24 hours from the time they are reported are dropped by New Relic. If left empty Butler will use the current time as timestamp. |
| interval | number The length of the time window (millisec). Required for count and summary metric types. |
Array of objects Dimensions that will be associated with the metric. |
Responses
Request samples
- Payload
Content type
application/json
{- "name": "memory.heap",
- "type": "gauge",
- "value": 2.3,
- "timestamp": 1642164296053,
- "interval": 0,
- "attributes": [
- {
- "name": "host.name",
- "value": "dev.server.com"
}
]
}Response samples
- 202
- 400
- 500
Content type
application/json
{- "newRelicResultCode": "202",
- "newRelicResultText": "Data accepted."
}Get all information available for existing schedule(s).
If a schedule ID is specified using a query parameter (and there exists a schedule with that ID), information about that schedule will be returned. If no schedule ID is specified, all schedules will be returned.
query Parameters
| id | string Example: id=e4b1c455-aa15-4a51-a9cf-c5e4cfc91339 Scheduld ID |
Responses
Response samples
- 200
- 400
- 500
Content type
application/json
[- {
- "id": "e4b1c455-aa15-4a51-a9cf-c5e4cfc91339",
- "created": "2020-09-29T14:29:12.283Z",
- "name": "Reload sales metrics",
- "cronSchedule": "0,30 6 * * 1-5",
- "timezone": "Europe/Stockholm",
- "qlikSenseTaskId": "210832b5-6174-4572-bd19-3e61eda675ef",
- "startupState": "started",
- "tags": [
- "tag 1",
- "tag 2"
], - "lastKnownState": "started"
}
]Create a new schedule.
Request Body schema: application/jsonrequired
| name required | string Descriptive name for the schedule. |
| cronSchedule required | string 5 or 6 position cron schedule. If 6 positions used, the leftmost position represent seconds. If 5 positions used, leftmost position is minutes. The example schedule will trigger at 00 and 30 minutes past 6:00 on Mon-Fri. |
| timezone required | string Time zone the schedule should use. Ex "Europe/Stockholm". |
| qlikSenseTaskId required | string ID of Qlik Sense task that should be started when schedule triggers. |
| startupState required | string Enum: "start" "started" "stop" "stopped" If set to "start" or "started", the schedule will be started upon creation. Otherwise it will remain in stopped state. |
| tags | Array of strings Can be used to categorise schedules. |
Responses
Request samples
- Payload
Content type
application/json
{- "name": "Reload sales metrics",
- "cronSchedule": "0,30 6 * * 1-5",
- "timezone": "Europe/Stockholm",
- "qlikSenseTaskId": "210832b5-6174-4572-bd19-3e61eda675ef",
- "startupState": "started",
- "tags": [
- "tag 1",
- "tag 2"
]
}Response samples
- 201
- 500
Content type
application/json
[- {
- "id": "e4b1c455-aa15-4a51-a9cf-c5e4cfc91339",
- "created": "2020-09-29T14:29:12.283Z",
- "name": "Reload sales metrics",
- "cronSchedule": "0,30 6 * * 1-5",
- "timezone": "Europe/Stockholm",
- "qlikSenseTaskId": "210832b5-6174-4572-bd19-3e61eda675ef",
- "startupState": "started",
- "tags": [
- "tag 1",
- "tag 2"
], - "lastKnownState": "started"
}
]Start a schedule.
Start a schedule, i.e. have the scheduler run the associated reload task according to the schedule's cron settings.
path Parameters
| scheduleId required | string Example: e4b1c455-aa15-4a51-a9cf-c5e4cfc91339 Schedule ID. |
Responses
Response samples
- 200
- 400
- 500
Content type
application/json
{- "id": "e4b1c455-aa15-4a51-a9cf-c5e4cfc91339",
- "created": "2020-09-29T14:29:12.283Z",
- "name": "Reload sales metrics",
- "cronSchedule": "0,30 6 * * 1-5",
- "timezone": "Europe/Stockholm",
- "qlikSenseTaskId": "210832b5-6174-4572-bd19-3e61eda675ef",
- "startupState": "started",
- "tags": [
- "tag 1",
- "tag 2"
], - "lastKnownState": "started"
}Start all schedules.
Start all schedules, i.e. tell the scheduler to run each schedule and start associated tasks according to each schedule's settings.
Responses
Response samples
- 200
- 500
Content type
application/json
[- {
- "id": "e4b1c455-aa15-4a51-a9cf-c5e4cfc91339",
- "created": "2020-09-29T14:29:12.283Z",
- "name": "Reload sales metrics",
- "cronSchedule": "0,30 6 * * 1-5",
- "timezone": "Europe/Stockholm",
- "qlikSenseTaskId": "210832b5-6174-4572-bd19-3e61eda675ef",
- "startupState": "started",
- "tags": [
- "tag 1",
- "tag 2"
]
}
]Stop a schedule.
Stop a schedule, i.e. tell the scheduler to no longer execute the schedule according to its cron settings.
path Parameters
| scheduleId required | string Example: e4b1c455-aa15-4a51-a9cf-c5e4cfc91339 Schedule ID. |
Responses
Response samples
- 200
- 400
- 500
Content type
application/json
{- "id": "e4b1c455-aa15-4a51-a9cf-c5e4cfc91339",
- "created": "2020-09-29T14:29:12.283Z",
- "name": "Reload sales metrics",
- "cronSchedule": "0,30 6 * * 1-5",
- "timezone": "Europe/Stockholm",
- "qlikSenseTaskId": "210832b5-6174-4572-bd19-3e61eda675ef",
- "startupState": "started",
- "tags": [
- "tag 1",
- "tag 2"
], - "lastKnownState": "started"
}Stop all schedules.
Stop all schedules, i.e. tell the scheduler to no longer execute any schedule according to its cron settings.
Responses
Response samples
- 200
- 500
Content type
application/json
[- {
- "id": "e4b1c455-aa15-4a51-a9cf-c5e4cfc91339",
- "created": "2020-09-29T14:29:12.283Z",
- "name": "Reload sales metrics",
- "cronSchedule": "0,30 6 * * 1-5",
- "timezone": "Europe/Stockholm",
- "qlikSenseTaskId": "210832b5-6174-4572-bd19-3e61eda675ef",
- "startupState": "started",
- "tags": [
- "tag 1",
- "tag 2"
]
}
]Get scheduler status.
Get basic status from the core scheduler.
No schedule metadata beyond ID, cron setting and job state will be returned, but as this comes from the core scheduler it is the authorative truth about what jobs are running (and which ones are not).
Responses
Response samples
- 200
Content type
text/plain
{ '3702cec1-b6c8-463e-bda3-58d6a94dd9ac': * */2 * * * status: Running '2d5dcebc-2440-4bd7-9aa1-fb69708715c8': */45 * * * * * status: Running 'a93ca0f3-7980-439b-9eda-723a167352e3': */10 * * * * * status: Running 'ad250f49-ffd8-45dc-9b2b-f76028e969a4': */5 * * * * * status: Running }
Do a stand-alone reload of a Qlik Sense app, without using a task.
path Parameters
| appId required | string Example: 210832b5-6174-4572-bd19-3e61eda675ef ID of Qlik Sense app. |
Request Body schema: application/json
| reloadMode | integer Reload mode that will be used. 0, 1 or 2. If not specified 0 will be used |
| partialReload | boolean Should a full (=false) or partial (=true) reload be done? If not specified a full reload will be done. |
| startQSEoWTaskOnSuccess | Array of strings Array of task IDs that should be started when the app has successfully reloaded. |
| startQSEoWTaskOnFailure | Array of strings Array of task IDs that should be started if the app fails reloading. |
Responses
Request samples
- Payload
Content type
application/json
{- "reloadMode": 0,
- "partialReload": true,
- "startQSEoWTaskOnSuccess": [
- "09b3c78f-04dd-45e3-a4bf-1b074d6572fa",
- "eaf1da4f-fd44-4cea-b2de-7b67a6496ee3"
], - "startQSEoWTaskOnFailure": [
- "09b3c78f-04dd-45e3-a4bf-1b074d6572fa",
- "eaf1da4f-fd44-4cea-b2de-7b67a6496ee3"
]
}Response samples
- 201
- 400
- 500
Content type
application/json
{- "appId": "210832b5-6174-4572-bd19-3e61eda675ef"
}Dump a Sense app to JSON.
Does the same thing as /v4/app/:appId/dump
path Parameters
| appId required | string Example: 210832b5-6174-4572-bd19-3e61eda675ef ID of Qlik Sense app. |
Responses
Response samples
- 200
- 400
- 422
- 500
Content type
application/json
{- "properties": { },
- "loadScript": "",
- "sheets": [ ],
- "stories": [ ],
- "masterobjects": [ ],
- "appprops": [ ],
- "dataconnections": [ ],
- "dimensions": [ ],
- "bookmarks": [ ],
- "embeddedmedia": [ ],
- "snapshots": [ ],
- "fields": [ ],
- "variables": [ ],
- "measures": [ ]
}Dump a Sense app to JSON.
Does the same thing as /v4/senseappdump/:appId
path Parameters
| appId required | string Example: 210832b5-6174-4572-bd19-3e61eda675ef ID of Qlik Sense app. |
Responses
Response samples
- 200
- 400
- 422
- 500
Content type
application/json
{- "properties": { },
- "loadScript": "",
- "sheets": [ ],
- "stories": [ ],
- "masterobjects": [ ],
- "appprops": [ ],
- "dataconnections": [ ],
- "dimensions": [ ],
- "bookmarks": [ ],
- "embeddedmedia": [ ],
- "snapshots": [ ],
- "fields": [ ],
- "variables": [ ],
- "measures": [ ]
}Start a Qlik Sense task.
An optional array of zero or more objects can be passed in the message body. It is used to pass additional info related to the reload task being started.
Currently supported values in this array are:
- A key-value pair that will be stored in Butler's KV store. If Butler's key-value store is not enabled, any key-value information passed in this parameter will simply be ignored.
Setting
ttl=0disables the TTL feature, i.e. the KV pair will not expire. - A task identified by its taskId that should be started.
- Tasks identified by tags set on tasks in the QMC.
- Tasks identified by custom properties set in the QMC.
This parameter uses a generic JSON/object format (type + payload). It's thus possible to add new integrations in future Butler versions.
path Parameters
| taskId required | string Example: 210832b5-6174-4572-bd19-3e61eda675ef ID of Qlik Sense task. Butler will ignore the "magic" task ID of "-" (=dash, hyphen). This ID will not be reported as invalid. |
query Parameters
| allTaskIdsMustExist | boolean Example: allTaskIdsMustExist=true If set to If set to In either case, missing/invalid taskIds will be reported in the result set back to the client calling the API. Note: Tasks started by specifying tags and/or custom properties are not affected by this. |
Request Body schema: application/json
Optional object with extra info.
Array
| type | string Enum: "keyvaluestore" "starttaskid" "starttasktag" "starttaskcustomproperty" |
| payload | object |
Responses
Request samples
- Payload
Content type
application/json
[- {
- "type": "starttaskid",
- "payload": {
- "taskId": "7552d9fc-d1bb-4975-9a38-18357de531ea"
}
}, - {
- "type": "starttasktag",
- "payload": {
- "tag": "startTask1"
}
}, - {
- "type": "starttaskcustomproperty",
- "payload": {
- "customPropertyName": "taskGroup",
- "customPropertyValue": "tasks2"
}
}, - {
- "type": "keyvaluestore",
- "payload": {
- "namespace": "MyFineNamespace",
- "key": "AnImportantKey",
- "value": "TheValue",
- "ttl": 10000
}
}
]Response samples
- 200
- 400
- 500
Content type
application/json
{- "tasksId": {
- "started": [
- {
- "taskId": "e3b27f50-b1c0-4879-88fc-c7cdd9c1cf3e",
- "taskName": "Reload task of App1"
}
], - "invalid": [
- {
- "taskId": "abc"
}
], - "denied": [
- {
- "taskId": "a1a11a11-b1c0-4879-88fc-c7cdd9c1cf3e"
}
]
}, - "tasksTag": [
- {
- "taskId": "e3b27f50-b1c0-4879-88fc-c7cdd9c1cf3e",
- "taskName": "Reload task of App1"
}
], - "tasksTagDenied": [
- {
- "tag": "startTask_invalid1"
}
], - "tasksCP": [
- {
- "taskId": "e3b27f50-b1c0-4879-88fc-c7cdd9c1cf3e",
- "taskName": "Reload task of App1"
}
], - "tasksCPDenied": [
- {
- "name": "taskGroup",
- "value": "cp_value_denied1"
}
]
}Start a Qlik Sense task.
An optional array of zero or more objects can be passed in the message body. It is used to pass additional info related to the reload task being started.
Currently supported values in this array are:
- A key-value pair that will be stored in Butler's KV store. If Butler's key-value store is not enabled, any key-value information passed in this parameter will simply be ignored.
Setting
ttl=0disables the TTL feature, i.e. the KV pair will not expire. - A task identified by its taskId that should be started.
- Tasks identified by tags set on tasks in the QMC.
- Tasks identified by custom properties set in the QMC.
This parameter uses a generic JSON/object format (type + payload). It's thus possible to add new integrations in future Butler versions.
path Parameters
| taskId required | string Example: 210832b5-6174-4572-bd19-3e61eda675ef ID of Qlik Sense task. Butler will ignore the "magic" task ID of "-" (=dash, hyphen). This ID will not be reported as invalid. |
query Parameters
| allTaskIdsMustExist | boolean Example: allTaskIdsMustExist=true If set to If set to In either case, missing/invalid taskIds will be reported in the result set back to the client calling the API. Note: Tasks started by specifying tags and/or custom properties are not affected by this. |
Request Body schema: application/json
Optional object with extra info.
Array
| type | string Enum: "keyvaluestore" "starttaskid" "starttasktag" "starttaskcustomproperty" |
| payload | object |
Responses
Request samples
- Payload
Content type
application/json
[- {
- "type": "starttaskid",
- "payload": {
- "taskId": "7552d9fc-d1bb-4975-9a38-18357de531ea"
}
}, - {
- "type": "starttasktag",
- "payload": {
- "tag": "startTask1"
}
}, - {
- "type": "starttaskcustomproperty",
- "payload": {
- "customPropertyName": "taskGroup",
- "customPropertyValue": "tasks2"
}
}, - {
- "type": "keyvaluestore",
- "payload": {
- "namespace": "MyFineNamespace",
- "key": "AnImportantKey",
- "value": "TheValue",
- "ttl": 10000
}
}
]Response samples
- 200
- 400
- 500
Content type
application/json
{- "tasksId": {
- "started": [
- {
- "taskId": "e3b27f50-b1c0-4879-88fc-c7cdd9c1cf3e",
- "taskName": "Reload task of App1"
}
], - "invalid": [
- {
- "taskId": "abc"
}
], - "denied": [
- {
- "taskId": "a1a11a11-b1c0-4879-88fc-c7cdd9c1cf3e"
}
]
}, - "tasksTag": [
- {
- "taskId": "e3b27f50-b1c0-4879-88fc-c7cdd9c1cf3e",
- "taskName": "Reload task of App1"
}
], - "tasksTagDenied": [
- {
- "tag": "startTask_invalid1"
}
], - "tasksCP": [
- {
- "taskId": "e3b27f50-b1c0-4879-88fc-c7cdd9c1cf3e",
- "taskName": "Reload task of App1"
}
], - "tasksCPDenied": [
- {
- "name": "taskGroup",
- "value": "cp_value_denied1"
}
]
}Send message to Slack.
Sends a basic message to Slack.
Request Body schema: application/jsonrequired
| channel required | string Slack channel to post message into. Prefix channel name with #. |
| from_user required | string Name of sending user, as shown in the Slack message |
| msg required | string Text going into the Slack message |
| emoji | string Emoji to shown next to Slack message |
Responses
Request samples
- Payload
Content type
application/json
{- "channel": "#reload-notification",
- "from_user": "Butler the Bot",
- "msg": "This is a message from Qlik Sense",
- "emoji": "thumbsup"
}Response samples
- 201
- 400
- 500
Content type
application/json
{- "channel": "#reload-notification",
- "from_user": "Butler the Bot",
- "msg": "This is a message from Qlik Sense",
- "emoji": "thumbsup"
}5 - Scheduler
Details about the Butler scheduler.
Scheduling overview
The scheduler is used to start Sense reloads at specific times, or at certain intervals.
The rationale for having a scheduler in Butler is mainly that Qlik Sense’s built-in scheduler doesn’t quite have the desired flexibiluty. The Sense scheduler on the other hand offer some features not included in the Butler scheduler, so they complelent each other nicely.
Put differently: Butler’s scheduler is great for kicking off the first step if reload chains (or single app reloads), while the QMC scheduler is needed to link the individual parts of reload chains together.
| Feature | Qlik Sense scheduler | Butler scheduler |
|---|---|---|
| Tasks should run same time each day. | Yes | Yes |
| Tasks should run only some days of the month/wek (ex 1st day of the month or on Wednesdays and Fridays). | Yes | Yes |
| Tasks should run only between certain hours during the day (e.g. every 5 minutes during the first 30 minutes of each hour, between 08.00 and 18.00). | Yes, with lots of manual work | Yes |
| Tasks can be chained together, with one task starting when the previous has finished. | Yes | - |
| Task definitions can be stored in Git and managed by a DevOps workflow. | Yes, with lots of work. | Yes, out of the box |
Schedules
The Butler scheduler handles zero or more schedules, where a schedule is simply a set of information about which Sense reload task should be started when.
While Butler is running it keeps the schedules in memory, but they are also stored on disk in the file specified in the Butler’s main config file. This means you can either manage the schedule file manually, or via the Butler APIs.
The following information is kept about each schdule. Note that some fields are optional and only set when managing schedules via API calls.
| Field name | Descriptions | Example |
|---|---|---|
| name | Descriptive name for the schedule. | Every 2 hours |
| cronSchedule | Cron string. 5 or 6 postitions allowed. If 5 positions used a 0 will be used for seconds. crontab.guru and similar sites are useful for creating the cron string. |
0 */2 * * * |
| timezone | Timezone in which the schedule will execute. List of available timezones here. | Europe/London |
| qlikSenseTaskId | Qlik Sense task that will be started. The task ID can be found in the task view in the QMC. | 0fe447a9-ba1f-44a9-ac23-68c3a1d88d8b |
| startupState | Controls whether this task should be started or left stopped when Butler starts up. If set to “start” or “started”, the schedule will be started upon creation. Otherwise it will remain in stopped state. |
started |
| tags | Array of tags, can be used to categorise schedules. Optional. | [ finace ETL, abc 123 ] |
| id | Schedule ID. Free text string, can be set to anything as long as it is unique. Schedules created via the REST API will have a GUID as id. | 2f7a6c38-46df-416d-83de-44bdea5cedaa |
| created | Timestamp the schedule was created. Optional when manually adding schedules to schedule file. | 2020-09-29T17:14:35.154Z |
| lastKnownState | The schedule’s last known state. Optional when manually adding schedules to schedule file. | started |
Sample schedule file
A schedule file could look like this:
butlerSchedule:
- name: Every 30 sec
cronSchedule: "*/30 * * * * *"
timezone: Europe/Stockholm
qlikSenseTaskId: 0fe447a9-ba1f-44a9-ac23-68c3a1d88d8b
startupState: started
tags:
- tag 2
- abc 123 åäö
id: c7ec214c-e9ca-40b2-acb4-54648f90dd73
created: '2020-09-29T14:29:12.283Z'
lastKnownState: started
- name: Every 2 hours
cronSchedule: "0 */2 * * *"
timezone: Europe/London
qlikSenseTaskId: 0fe447a9-ba1f-44a9-ac23-68c3a1d88d8b
startupState: started
tags:
- sales ETL
id: 2f7a6c38-46df-416d-83de-44bdea5cedaa
created: '2020-09-29T17:14:35.154Z'
lastKnownState: started
- name: 00 and 30 minutes past every hour from 06 to 18 on Mon-Fri
cronSchedule: "0,30 6-18 * * 1-5"
timezone: Europe/Paris
qlikSenseTaskId: 0fe447a9-ba1f-44a9-ac23-68c3a1d88d8b
startupState: started
tags:
- finance ETL
- weekdays
id: Manually-added-schedule-1
A few things to note:
- Schedule 1
- Uses 6 positions in the cron string. The leftmost position is seconds.
- This schedule cannot be created in the QMC scheduler, as minutes is the smallest increment there. On the other hand, it’s questionable if Sense jobs should run more often than once per minute…
- Schedule 2
- Uses 5 positions in the cron string. The leftmost position is minutes. The seconds position is implicitly set to 0, which means the schedule will fire at the top of each minute.
- This schedule can be created in the QMC scheduler.
- Schedule 3
- This schedule cannot be created in the QMC scheduler.
Manual vs API created schedules
Schedules are stored in a YAML file on disk. This file is loaded each time Butler is started.
When a new schedule is created using the APIs, all schedule definitions and their current state is written to the schedule YAML file. Thus, if Butler is stopped or restarted, it will also restart all schedules in the same state they were in before the Butler stop/restart.
There are a couple of differences between manually created schedules (i.e. ones that are added directly to the YAML file) and schedules created via the APIs:
- The
createdfield in a schedule is optional. It is set when a schedule is created using the APIs. You can set it when manually editing the schedule file, if so desired - but it’s not required. - The
lastKnownStatefield is overwritten each time the schedules are saved to disk. This means that you can change this field manually for a schedule in the schedule file on disk, but if you then change the schedule’s state (start or stop it), that will trigger a saving of the schedules to disk, and your previous (manual) change will be overwritten.
6 - Key-value store
Details about the Butler key-value store.
Key-value store overview
The API documentation is the best, most complete source of information about the key-value API endpoints.
Namespaces
Each key-value pair is associated with a “namespace”, which is simply a way to group KV pairs together in logical groups.
Examples:
- An app might have it’s own namespace to make it clear what key-value pairs belong to the app.
- A reload chain that need to pass parameters from app to app can have it’s own namespace.
The Butler.keyValueStore.maxKeysPerNamespace property in the main config file controls how many KV pairs can be created in each namespace. If no value is specified in the config file, a default value of 1000 KV pairs per namespace is used.
Time to live
When key-value pairs are created an optional time-to-live (ttl) parameter may also be passed in the call to Butler’s API.
If the ttl parameter is specified (=set to a non-zero value, unit is milliseconds) the created KV pair will only exist for that long.
If a KV pair with a running ttl clock is updated before the KV pair expires, the current ttl clock will be discarded and replaced with the ttl value from the new KV pair.
In other words: The ttl is relative to the last update of the KV pair in question.
7 - Alert template fields
List of template fields available in Butler alert messages.
Template fields
Butler uses the Handlebars library for templating work.
Handlebars offers a lot of useful features (nested template fields, evaluation context, template comments) and it’s recommended that you browse through at least the language features section of their getting started guide to get a feeling for what’s possible.
Different Butler monitoring features offer different template fields.
For example, the template fields available for reload alerts are different from the ones available for monitoring Windows services.
7.1 - Client managed Qlik Sense Enterprise on Windows
List of template fields available in alert messages for client-managed Qlik Sense Enterprise on Windows.
7.1.1 - Alert template fields for failed/aborted reload tasks
List of template fields available in Butler’s reload failed/aborted alert messages.
Template fields
Butler uses the Handlebars library for templating work.
Handlebars offers a lot of useful features (nested template fields, evaluation context, template comments) and it’s recommended that you browse through at least the language features section of their getting started guide to get a feeling for what’s possible.
Warning
Only some alert destinations support template files, namely
- Teams
- Slack
Please see the Concepts and Getting started sections for more information about which alert types support templates.
If a template field is used for an alert type where that field is not supported, the field will simply be blank. No errors will occur, but it can still be tricky to debug if you’re not aware of this.
The following template fields are available in alert messages.
Note that some fields are sometimes (often, even) empty, for example the script log for stopped messages.
This is simply how Sense works - the template fields just forward the information retrieved from the Sense APIs.
| Failed reload |
Stopped reload |
Successful reload |
Field name | Description |
|---|---|---|---|---|
| ✅ | ✅ | ✅ | hostName | Server on which a reload or other event took place. |
| ✅ | ✅ | ✅ | user | Reload failures: User ID for use doing the reload. Typically sa_scheduler.Reload stopped: User ID of user stopping the reload. |
| ✅ | ✅ | ✅ | taskId | ID of reload task. |
| ✅ | ✅ | ✅ | taskName | Name of reload task. |
| ✅ | ✅ | ✅ | appId | ID of Sense app. |
| ✅ | ✅ | ✅ | appName | Name of app. |
| ✅ | ✅ | ✅ | appDescription | Description of app. |
| ✅ | ✅ | ✅ | appFileSize | Size of app file (on disk). |
| ✅ | ✅ | ✅ | appLastSuccessfulReload | Date of last successful reload. |
| ✅ | ✅ | ✅ | appLastModifiedDate | Date/time of last modification of app. |
| ✅ | ✅ | ✅ | appLastModifiedByUserName | User who last modified the app. |
| ✅ | ✅ | ✅ | appPublishTime | Date/time when app was published. |
| ✅ | ✅ | ✅ | appPublished | Is the app published? (true/false) |
| ✅ | ✅ | ✅ | appStreamName | Name of stream where app is published. |
| ✅ | ✅ | ✅ | appCustomProperties | Custom properties that are present on the app. |
| ✅ | ✅ | ✅ | appTags | Tags that are present on the app. |
| ✅ | ✅ | ✅ | appUrl | URL to the app. |
| ✅ | ✅ | ✅ | taskCustomProperties | Custom properties that are present on the reload task. |
| ✅ | ✅ | ✅ | taskTags | Tags that are present on the reload task. |
| ✅ | ✅ | ✅ | taskIsPartialReload | Does the reload task perform a partial reload? (true/false) |
| ✅ | ✅ | ✅ | taskMaxRetries | Maximum number of retries for the reload task. |
| ✅ | ✅ | ✅ | taskModifiedByUsername | User who last modified the reload task. |
| ✅ | ✅ | ✅ | taskModifiedDate | Date when the reload task was last modified. |
| ✅ | ✅ | ✅ | taskSessionTimeout | Session timeout for the reload task, i.e how long the reload task will run before it is cancelled. |
| ✅ | ✅ | ✅ | taskNextExecution | Next scheduled execution of the reload task. |
| ✅ | ✅ | ✅ | appOwnerName | Name of app owner (if this is available in the metadata provided by the Sense server) |
| ✅ | ✅ | ✅ | appOwnerUserDirectory | App owner user’s user directory (if this is available in the metadata provided by the Sense server) |
| ✅ | ✅ | ✅ | appOwnerUserId | App owner user’s user id (if this is available in the metadata provided by the Sense server) |
| ✅ | ✅ | ✅ | appOwnerEmail | App owner email (if this is available in the metadata provided by the Sense server) |
| ✅ | ✅ | ✅ | logTimeStamp | Timestamp as recorded in the Sense logs |
| ✅ | ✅ | ✅ | logLevel | Log level of the Sense log file entry causing the alert |
| ✅ | ✅ | ✅ | logMessage | Log message from the Sense log files |
| ✅ | ✅ | ✅ | executingNodeName | Name of the server where the reload took place |
| ✅ | ✅ | ✅ | executionDuration.hours | executionDuration is a JSON object. Duration of reload (hours part) |
| ✅ | ✅ | ✅ | executionDuration.minutes | Duration of reload (minutes part) |
| ✅ | ✅ | ✅ | executionDuration.seconds | Duration of reload (seconds part) |
| ✅ | ✅ | ✅ | executionStartTime.startTimeUTC | JSON object. UTC timestamp for reload start |
| ✅ | ✅ | ✅ | executionStartTime.startTimeLocal1 | Reload start timestamp, format 1 |
| ✅ | ✅ | ✅ | executionStartTime.startTimeLocal2 | Reload start timestamp, format 2 |
| ✅ | ✅ | ✅ | executionStartTime.startTimeLocal3 | Reload start timestamp, format 3 |
| ✅ | ✅ | ✅ | executionStartTime.startTimeLocal4 | Reload start timestamp, format 4 |
| ✅ | ✅ | ✅ | executionStartTime.startTimeLocal5 | Reload start timestamp, format 5 |
| ✅ | ✅ | ✅ | executionStopTime.stopTimeUTC | JSON object. UTC timestamp for reload end |
| ✅ | ✅ | ✅ | executionStopTime.stopTimeLocal1 | Reload end timestamp, format 1 |
| ✅ | ✅ | ✅ | executionStopTime.stopTimeLocal2 | Reload end timestamp, format 2 |
| ✅ | ✅ | ✅ | executionStopTime.stopTimeLocal3 | Reload end timestamp, format 3 |
| ✅ | ✅ | ✅ | executionStopTime.stopTimeLocal4 | Reload end timestamp, format 4 |
| ✅ | ✅ | ✅ | executionStopTime.stopTimeLocal5 | Reload end timestamp, format 5 |
| ✅ | ✅ | ✅ | executionStatusNum | Final reload task status code |
| ✅ | ✅ | ✅ | executionStatusText | Final reload task status message |
| ✅ | ✅ | ✅ | executionDetails[].timestampUTC | executionDetails is an array of status updates for the reload task, similar to the one found in the QMC’s task view. UTC timestamp of the task status |
| ✅ | ✅ | ✅ | executionDetails[].timestampLocal1 | Task status timestamp, format 1 |
| ✅ | ✅ | ✅ | executionDetails[].timestampLocal2 | Task status timestamp, format 2 |
| ✅ | ✅ | ✅ | executionDetails[].timestampLocal3 | Task status timestamp, format 3 |
| ✅ | ✅ | ✅ | executionDetails[].timestampLocal4 | Task status timestamp, format 4 |
| ✅ | ✅ | ✅ | executionDetails[].timestampLocal5 | Task status timestamp, format 5 |
| ✅ | ✅ | ✅ | executionDetails[].detailsType | Task status timestamp, format 1 |
| ✅ | ✅ | ✅ | executionDetails[].message | Task status message |
| ✅ | ✅ | ✅ | scriptLogSize | Size of the reload’s script log (characters) |
| ✅ | ✅ | ✅ | scriptLogSizeRows | Size of the reload’s script log (rows) |
| ✅ | ✅ | ✅ | scriptLogSizeCharacters | Size of the reload’s script log (characters) |
| ✅ | ✅ | ✅ | scriptLogHeadCount | Number of lines extracted from the start of the script log |
| ✅ | ✅ | ✅ | scriptLogTailCount | The first y lines from the reload’s script log |
| ✅ | ✅ | ✅ | scriptLogHead | The first x lines from the reload’s script log |
| ✅ | ✅ | ✅ | scriptLogTail | Number of lines extracted from the end of the script log |
| ✅ | ✅ | ✅ | qliksSenseQMC | Links to QMC, as defined in main config file |
| ✅ | ✅ | ✅ | qliksSenseHub | Links to Hub, as defined in main config file |
| ✅ | ✅ | ✅ | genericUrls | Links to other systems, as defined in main config file |
7.1.2 - Alert template fields for Windows services events
List of template fields available in Butler’s Windows services alert messages.
Email, Slack and Teams
Butler uses the Handlebars library for templating work.
This gives a lot of flexibility in how alert messages are formatted for the destination types that support template fields.
Handlebars offers a lot of useful features (nested template fields, evaluation context, template comments) and it’s recommended that you browse through at least the language features section of their getting started guide to get a feeling for what’s possible.
Warning
Not all alert destinations (MQTT, outgoing webhooks etc) support template fields.
Please see the Getting started sections for more information on how to set up alerts for each destination.
If a template field is used for an alert type where that field is not supported, the field will simply be blank. No errors will occur.
The following alert destinations support template fields:
The following template fields are available in alert messages.
Note that some fields are usually (always?) empty, for example the script log for stopped messages.
This is simply how Sense works - the template fields just forward the information retrieved from the Sense APIs.
| Slack | Teams | Field name | Description | |
|---|---|---|---|---|
| ✅ | ✅ | ✅ | {{host}} |
The hostname of the server where the service is running |
| ✅ | ✅ | ✅ | {{serviceStatus}} |
The status of the service, e.g. RUNNING or STOPPED |
| ✅ | ✅ | ✅ | {{servicePrevStatus}} |
The previous status of the service, e.g. RUNNING or STOPPED |
| ✅ | ✅ | ✅ | {{serviceName}} |
The name of the service as defined in Windows |
| ✅ | ✅ | ✅ | {{serviceDisplayName}} |
The display name of the service as defined in Windows. Can sometimes be a bit more human readable than the serviceName. |
| ✅ | ✅ | ✅ | {{serviceFriendlyName}} |
The display name of the service as defined in the Butler config file. Used to give the service a good name when both serviceName and serviceDisplayName are unsuitable for use in for example Grafana dashboards. |
| ✅ | ✅ | ✅ | {{serviceStartType}} |
The startup mode of the service, e.g. Automatic or Manual |
| ✅ | ✅ | ✅ | {{serviceExePath}} |
The path to the executable of the service |
InfluxDB
Window service data will be stored in the InfluxDB database specified in the config file Butler.influxdb.dbName setting.
The latest status of each service will be stored in the butler_windows_services measurement every time Butler checks the status of the service, i.e. not only when the service changes status.
Fields/metrics
The following metrics will be stored in a measurement named win_service_state, for each Windows service:
| Field name | Description |
|---|---|
| state_num | A numeric representation of the Windows service’s current state. |
| state_text | The Windows service’s current state (textual representation). |
| startup_mode_num | A numeric representation of the Windows service’s startup mode. |
| startup_mode_text | The Windows service’s startup mode (textual representation). |
Mapping of state_num to state_text:
| state_num | state_text |
|---|---|
| 1 | STOPPED |
| 2 | START_PENDING |
| 3 | STOP_PENDING |
| 4 | RUNNING |
| 5 | CONTINUE_PENDING |
| 6 | PAUSE_PENDING |
| 7 | PAUSED |
Mapping of startup_mode_num to startup_mode_text:
| startup_mode_num | startup_mode_text |
|---|---|
| 0 | Automatic |
| 1 | Automatic (delayed start) |
| 2 | Manual |
| 3 | Disabled |
Tags
The following tags will be attached to all Windows service data stored in InfluxDB:
| Tag name | Description |
|---|---|
| butler_instance | The value in Butler.influxDb.instanceTag. Can be used to differentiate several Butler instances running in parallel. |
| host | Host name where the Windows service is running. Comes from Butler.serviceMonitor.monitor.<host>. |
| service_name | The name of the Windows service, as defined in Windows. |
| service_display_name | The display name of the Windows service, as defined in Windows. |
| service_friendly_name | The friendly name of the Windows service, as defined in the Butler config file Butler.serviceMonitor.monitor.<host>.services.<friendlyName>. |
New Relic
Butler will create New Relic events and/or log entries when a monitored Windows service changes state, for example from RUNNING to STOPPED.
Events
Events sent to New Relic will have these attributes attached:
| Attribute name | Description |
|---|---|
| eventType | Always set to qs_serviceStateEvent |
| butler_serviceHost | The host name where the Windows service is running. Comes from Butler.serviceMonitor.monitor.<host>. |
| butler_serviceName | The name of the Windows service, as defined in Windows. |
| butler_serviceDisplayName | The display name of the Windows service, as defined in Windows. |
| butler_serviceStatus | The status of the Windows service, e.g. RUNNING or STOPPED. |
| … | Any static and dynamic attributes defined in Butler.incidentTool.newRelic.serviceMonitor.destination.event.attribute. |
| … | Any static attributes defined in Butler.incidentTool.newRelic.serviceMonitor.sharedSettings.attribute.static. |
Log entries
Log entries sent to New Relic will have these attributes attached:
| Attribute name | Description |
|---|---|
| logType | Always set to qs_serviceStateLog |
| butler_serviceHost | The host name where the Windows service is running. Comes from Butler.serviceMonitor.monitor.<host>. |
| butler_serviceName | The name of the Windows service, as defined in Windows. |
| butler_serviceDisplayName | The display name of the Windows service, as defined in Windows. |
| butler_serviceStatus | The status of the Windows service, e.g. RUNNING or STOPPED. |
| … | Any static and dynamic attributes defined in Butler.incidentTool.newRelic.serviceMonitor.destination.log.attribute. |
| … | Any static attributes defined in Butler.incidentTool.newRelic.serviceMonitor.sharedSettings.attribute.static. |
The log message will be one of the following:
- “Windows service
on host is RUNNING.” - “Windows service
on host is STOPPED.” - “Windows service
on host is in state START_PENDING.” - “Windows service
on host is in state STOP_PENDING.” - “Windows service
on host is in state CONTINUE_PENDING.” - “Windows service
on host is in state PAUSE_PENDING.” - “Windows service
on host is in state PAUSED.”
MQTT
Similar to how InfluxDB works, Butler will send an MQTT message to the topic specified in Butler.mqttConfig.serviceStatusTopic every time it checks the status of a Windows service, i.e. not only when the service changes status.
Butler will ALSO send an MQTT message to the topics specified in Butler.mqttConfig.serviceRunningTopic and Butler.mqttConfig.serviceStoppedTopic when a Windows service is stopped or started.
Message payload for continuously sent messages is a JSON object with these properties:
| Property name | Description |
|---|---|
| host | The host name where the Windows service is running. Comes from Butler.serviceMonitor.monitor.<host>. |
| serviceName | The name of the Windows service, as defined in Windows. |
| serviceFriendlyName | The friendly name of the Windows service, as defined in the Butler config file Butler.serviceMonitor.monitor.<host>.services.<friendlyName>. |
| serviceStatus | The status of the Windows service, e.g. RUNNING or STOPPED. |
| serviceDetails | The details of the Windows service, e.g. RUNNING or STOPPED. |
Message payload for start/stop messages is a JSON object with these properties:
| Property name | Description |
|---|---|
| host | The host name where the Windows service is running. Comes from Butler.serviceMonitor.monitor.<host>. |
| serviceName | The name of the Windows service, as defined in Windows. |
| serviceFriendlyName | The friendly name of the Windows service, as defined in the Butler config file Butler.serviceMonitor.monitor.<host>.services.<friendlyName>. |
| serviceStatus | The status of the Windows service, e.g. RUNNING or STOPPED. |
| serviceDetails | The details of the Windows service, e.g. RUNNING or STOPPED. |
| prevState | The previous state of the Windows service, e.g. RUNNING or STOPPED. |
| currState | The current state of the Windows service, e.g. RUNNING or STOPPED. |
| stateChanged | true if the Windows service changed state, false if not. |
Outbound webhooks
Outbound webhooks (=http calls) are sent when a Windows service changes state, for example from RUNNING to STOPPED.
The payload for PUT and POST http calls is sent in the message body as a JSON object with these properties:
| Property name | Description |
|---|---|
| event | Always set to Windows service monitor |
| host | The host name where the Windows service is running. Comes from Butler.serviceMonitor.monitor.<host>. |
| serviceStatus | The status of the Windows service, e.g. RUNNING or STOPPED. |
| serviceName | The name of the Windows service, as defined in Windows. |
| serviceDisplayName | The display name of the Windows service, as defined in Windows. |
| serviceStartType | The startup mode of the service, e.g. Automatic or Manual |
| prevState | The previous state of the Windows service, e.g. RUNNING or STOPPED. |
| currState | The current state of the Windows service, e.g. RUNNING or STOPPED. |
| stateChanged | true if the Windows service changed state, false if not. |
The payload for GET http calls is sent as query parameters with the same properties as above, but with the property names being all lowercase letters.
A typical GET webhook URL would look like this:
https://mywebhookserver.com?event=Windows service monitor&host=MyHost&servicestatus=RUNNING&servicename=MyService&servicedisplayname=MyServiceDisplayName&servicestarttype=Automatic&prevstate=RUNNING&currstate=STOPPED&statechanged=true
7.2 - Qlik Sense Cloud
List of template fields available in alert messages for Qlik Sense Cloud.
7.2.1 - Alert template fields for failed app reloads
List of template fields available in Butler’s failed app reload alert messages.
Template fields
Butler uses the Handlebars library for templating work.
Handlebars offers a lot of useful features (nested template fields, evaluation context, template comments) and it’s recommended that you browse through at least the language features section of their getting started guide to get a feeling for what’s possible.
If a template field is used for an alert type where that field is not supported, the field will simply be blank. No errors will occur, but it can still be tricky to debug if you’re not aware of this.
The following template fields are available in alert messages.
| Failed app reload | Field name | Description |
|---|---|---|
| ✅ | tenantId | ID of the tenant where the reload took place. |
| ✅ | tenantComment | Comment for the tenant, as entered in Qlik Cloud. |
| ✅ | tenantUrl | URL to the tenant. |
| ✅ | userId | ID of the user who triggered the reload. |
| ✅ | userName | Name of the user who triggered the reload. |
| ✅ | appId | ID of Sense app. |
| ✅ | appName | Name of app. |
| ✅ | appDescription | Description of app. |
| ✅ | appUrl | URL to the app. |
| ✅ | appHasSectionAccess | Does the app have section access? (true/false) |
| ✅ | appIsPublished | Is the app published? (true/false) |
| ✅ | appPublishTime | Date/time when app was published. |
| ✅ | appThumbnail | URL to the app thumbnail. |
| ✅ | appOwnerName | Name of app owner. |
| ✅ | appOwnerUserId | App owner user’s user id. |
| ✅ | appOwnerPicture | URL to the app owner’s profile picture. |
| ✅ | appOwnerEmail | App owner email. |
| ✅ | reloadTrigger | What triggered the reload? Manually, scheduled etc. |
| ✅ | source | Source of the reload. Usually com.qlik/engine |
| ✅ | eventType | Type of QS Cloud event. For example com.qlik.v1.app.reload.finished |
| ✅ | eventTypeVersion | Version of the event type. |
| ✅ | endedWithMemoryConstraint | Did the reload end due to a memory constraint? (true/false) |
| ✅ | isDirectQueryMode | Is the app in Direct Query mode? (true/false) |
| ✅ | isPartialReload | Was the reload a partial reload? (true/false) |
| ✅ | isSessionApp | Is the app a session app? (true/false) |
| ✅ | isSkipStore | Should storing (to disk) the reloaded app be skipped? (true/false) |
| ✅ | reloadId | ID of the reload. |
| ✅ | rowLimit | Row limit for the reload. |
| ✅ | statements[] | Array of statements from the engine. |
| ✅ | status | Status of the reload. For example error. |
| ✅ | usageDuration | Duration of the reload. |
| ✅ | peakMemoryBytes | Peak memory usage during the reload. |
| ✅ | sizeMemoryBytes | Memory usage during the reload. |
| ✅ | appFileSize | Size of app file (on disk). |
| ✅ | errorCode | Error code for the reload. For example "EngineReloadScriptError". |
| ✅ | errorMessage | Error message for the reload. For example "error in app script or datasource". |
| ✅ | logMessage | Log message from the Sense log files. Example: "ReloadID: 670e1df04cf4e529c035c902\r\nStarted loading data\r\n(A detailed script progress log can be downloaded when the reload is finished)\r\nCharacters << AUTOGENERATE(26) 26 Lines fetched\r\nASCII << AUTOGENERATE(255) 191 Lines fetched\r\nTransactions << AUTOGENERATE(1000) 2,027 Lines fetched\r\n\r\nData has not been loaded. Please correct the error and try loading again.\r\nThe following error occurred:\r\nUnknown statement: ThisFailsForSure...\r\n \r\nThe engine error code: EDC_ERROR:11002\r\nThe error occurred here:\r\nThisFailsForSure...\r\n \r\n" |
| ✅ | executionDuration.hours | executionDuration is a JSON object. Duration of reload (hours part) |
| ✅ | executionDuration.minutes | Duration of reload (minutes part) |
| ✅ | executionDuration.seconds | Duration of reload (seconds part) |
| ✅ | executionStartTime.startTimeUTC | JSON object. UTC timestamp for reload start |
| ✅ | executionStartTime.startTimeLocal1 | Reload start timestamp, format 1 |
| ✅ | executionStartTime.startTimeLocal2 | Reload start timestamp, format 2 |
| ✅ | executionStartTime.startTimeLocal3 | Reload start timestamp, format 3 |
| ✅ | executionStartTime.startTimeLocal4 | Reload start timestamp, format 4 |
| ✅ | executionStartTime.startTimeLocal5 | Reload start timestamp, format 5 |
| ✅ | executionStopTime.stopTimeUTC | JSON object. UTC timestamp for reload end |
| ✅ | executionStopTime.stopTimeLocal1 | Reload end timestamp, format 1 |
| ✅ | executionStopTime.stopTimeLocal2 | Reload end timestamp, format 2 |
| ✅ | executionStopTime.stopTimeLocal3 | Reload end timestamp, format 3 |
| ✅ | executionStopTime.stopTimeLocal4 | Reload end timestamp, format 4 |
| ✅ | executionStopTime.stopTimeLocal5 | Reload end timestamp, format 5 |
| ✅ | executionStatusText | Final reload task status message, for example FAILED. |
| ✅ | scriptLogSize | Size of the reload’s script log (characters) |
| ✅ | scriptLogSizeRows | Size of the reload’s script log (rows) |
| ✅ | scriptLogSizeCharacters | Size of the reload’s script log (characters) |
| ✅ | scriptLogHeadCount | Number of lines extracted from the start of the script log |
| ✅ | scriptLogTailCount | The first y lines from the reload’s script log |
| ✅ | scriptLogHead | The first x lines from the reload’s script log |
| ✅ | scriptLogTail | Number of lines extracted from the end of the script log |
| ✅ | qliksSenseQMC | Links to QMC, as defined in main config file |
| ✅ | qliksSenseHub | Links to Hub, as defined in main config file |
| ✅ | genericUrls | Links to other systems, as defined in main config file |
8 - Information stored in InfluxDB
Butler stores a lot of information in InfluxDB. This page describes the different measurements and tags that Butler send to InfluxDB.
Failed reload tasks
Measurement: reload_task_failed
Tags
| Tag name | Description |
|---|---|
| host | Server on which the reload took place. |
| user | Full user info (directory + ID) for user doing the reload. Typically sa_scheduler for tasks started via scheduler. For manually started (from QMC) tasks the actual user starting the task is used. |
| task_id | ID of reload task that failed. |
| task_name | Name of reload task that failed. |
| app_id | ID of Sense app whose reload failed. |
| app_name | Name of Sense app whose reload failed. |
| log_level | Log level of the Sense log file entry causing the alert |
| task_executingNodeName | Name of node where the reload task was executed. |
| task_executionStatusNum | Reload task’s execution result code (numeric). |
| task_executionStatusText | Reload task’s execution result (text). |
| appTag_<app tag name 1 from Qlik Sense> | App tag defined in the QMC for the Qlik Sense app. |
| appTag_<app tag name 2 from Qlik Sense> | App tag defined in the QMC for the Qlik Sense app. |
| taskTag_<reload task tag name 1 from Qlik Sense> | Task tag defined in the QMC for the Qlik Sense reload task. |
| taskTag_<reload task tag name 2 from Qlik Sense> | Task tag defined in the QMC for the Qlik Sense reload task. |
| static-tag-1 | Static tag specified in the Butler configuration file. |
| static-tag-2 | Static tag specified in the Butler configuration file. |
Fields
| Field name | Description |
|---|---|
| log_timestamp | Timestamp of the Sense log file entry that triggered the event. |
| execution_id | Execution ID of the reload task. |
| log_message | Raw message from the Sense log file entry that triggered the alert. |
| task_executionStartTime_json | Start time of the reload task. Stringified JSON with seconds, minutes, hours. |
| task_executionStopTime_json | Stop time of the reload task. Stringified JSON with seconds, minutes, hours. |
| task_executionDuration_json | Duration of the reload task. Stringified JSON with seconds, minutes, hours. |
| task_executionDuration_sec | Duration of the reload task in seconds. |
| task_executionDuration_min | Duration of the reload task in minutes. |
| task_executionDuration_h | Duration of the reload task in hours. |
| task_scriptLogSize | Size (number of characters) of the reload script log associated with the event. |
| task_scriptLogTailCount | Number of lines from the end of the reload script log that are included in the event. |
| reload_log | Last few lines of the reload script log associated with the event. |
Successful reload tasks
Measurement: reload_task_success
Tags
| Tag name | Description |
|---|---|
| host | Server on which the reload took place. |
| user | Full user info (directory + ID) for user doing the reload. Typically sa_scheduler for tasks started via scheduler. For manually started (from QMC) tasks the actual user starting the task is used. |
| task_id | ID of reload task. |
| task_name | Name of reload task. |
| app_id | ID of Sense app that was reloaded. |
| app_name | Name of Sense app that was reloaded. |
| log_level | Log level of the Sense log file entry causing the event |
| task_executingNodeName | Name of node where the reload task was executed. |
| task_executionStatusNum | Reload task’s execution result code (numeric). |
| task_executionStatusText | Reload task’s execution result (text). |
| appTag_<app tag name 1 from Qlik Sense> | App tag defined in the QMC for the Qlik Sense app. |
| appTag_<app tag name 2 from Qlik Sense> | App tag defined in the QMC for the Qlik Sense app. |
| taskTag_<reload task tag name 1 from Qlik Sense> | Task tag defined in the QMC for the Qlik Sense reload task. |
| taskTag_<reload task tag name 2 from Qlik Sense> | Task tag defined in the QMC for the Qlik Sense reload task. |
| static-tag-1 | Static tag specified in the Butler configuration file. |
| static-tag-2 | Static tag specified in the Butler configuration file. |
Fields
| Field name | Description |
|---|---|
| log_timestamp | Timestamp of the Sense log file entry that triggered the event. |
| execution_id | Execution ID of the reload task. |
| log_message | Raw message from the Sense log file entry that triggered the event. |
| task_executionStartTime_json | Start time of the reload task. Stringified JSON with seconds, minutes, hours. |
| task_executionStopTime_json | Stop time of the reload task. Stringified JSON with seconds, minutes, hours. |
| task_executionDuration_json | Duration of the reload task. Stringified JSON with seconds, minutes, hours. |
| task_executionDuration_sec | Duration of the reload task in seconds. |
| task_executionDuration_min | Duration of the reload task in minutes. |
| task_executionDuration_h | Duration of the reload task in hours. |
Windows service info
Measurement: win_service_state
Tags
| Tag name | Description |
|---|---|
| butler_instance | Name of the Butler instance, from config file. |
| host | Server on which the reload took place. |
| service_name | Name of the Windows service. |
| display_name | Display name of the Windows service. |
| friendly_name | Friendly name of the Windows service (as defined in the Butler config file). |
Fields
| Field name | Description |
|---|---|
| state_num | State of the Windows service (numeric). |
| state_text | State of the Windows service (text). |
| startup_mode_num | Startup mode of the Windows service (numeric). |
| startup_mode_text | Startup mode of the Windows service (text). |
Qlik Sense server version info
Measurement: qlik_sense_version
Tags
| Tag name | Description |
|---|---|
| butler_instance | Name of the Butler instance, from Butler.influxDb.instanceTag in config file. |
In addition to above, all tags defined in Butler.qlikSenseVersion.versionMonitor.destination.influxDb.tag.static in the config file are added to each datapoint that is sent to InfluxDB.
Fields
| Field name | Description |
|---|---|
| content_hash | Content hash, as returned from the Sense version API. |
| sense_id | Sense ID, as returned from the Sense version API. This is usually on the form qliksenseserver:<version> |
| product_name | Sense product name, for example Qlik Sense |
| deployment_type | Sense deployment type, for example QlikSenseServer |
| version | Sense version, for example 14.173.4 |
| release_label | Sense release label, for example February 2024 Patch 1 |
| deprecated_product_version | Sense old/deprecated product version, for example 4.0.x |
| copyright_year_range | Sense copyright year range, for example 1993-2024 |
Qlik Sense server license info
Measurement: qlik_sense_server_license
Tags
| Tag name | Description |
|---|---|
| butler_instance | Name of the Butler instance, from Butler.influxDb.instanceTag in config file. |
In addition to above, all tags defined in Butler.qlikSenseLicense.serverLicenseMonitor.destination.influxDb.tag.static in the config file are added to each datapoint that is sent to InfluxDB.
Fields
| Field name | Description |
|---|---|
| license_expired | Whether the Sense license has expired. true/false. |
| expiry_date | Expiry date of the Sense server license, YYYY-MM-DD formaat. |
| days_until_expiry | Number of days until the Sense server license expires. If expiration date has passed this number will be negative. |
Qlik Sense end user access license info
The access license information returned by the Qlik Sense API is not very well documented by Qlik.
As a result it is not always clear what the different fields mean.
This is highlighted for each affected field below.
Measurement: sense_license_info
Each datapoint has a tag called license_type that can have the following values:
professionalanalyzeranalyzer_capacitytoken_logintoken_usertokens_available
The tags are the same for all license types, but the fields differ as follows.
Tags
| Tag name | Description |
|---|---|
| butler_instance | Name of the Butler instance, from Butler.influxDb.instanceTag in config file. |
In addition to above, all tags defined in Butler.qlikSenseLicense.licenseMonitor.destination.influxDb.tag.static in the config file are added to each datapoint that is sent to InfluxDB.
License types “professional” and “analyzer”
| Field name | Description |
|---|---|
| allocated | Number of allocated professional licenses. |
| available | Number of available professional licenses. |
| excess | Number of excess professional licenses. Unclear what this means. |
| quarantined | Number of quarantined professional licenses. |
| total | Total number of professional licenses included in the installed Qlik Sense license. |
| used | Number of professional licenses in use right now. |
License type “analyzer_capacity”
| Field name | Description |
|---|---|
| allocated_minutes | Total number of analyzer capacity minutes available in the installed Qlik Sense license. |
| unavailable_minutes | Number of analyzer capacity minutes that are currently unavailable. Unclear what this means. |
| used_minutes | Number of analyzer capacity minutes used so far this month. |
License type “token_login”
| Field name | Description |
|---|---|
| allocated_tokens | Unclear what this means. |
| token_cost | Unclear what this means. |
| unavailable_tokens | Unclear what this means. |
| used_tokens | Unclear what this means. |
License type “token_user”
| Field name | Description |
|---|---|
| allocated_tokens | Unclear what this means. |
| quarantined_tokens | Unclear what this means. |
| token_cost | Unclear what this means. |
| used_tokens | Unclear what this means. |
License type “tokens_available”
This license type contains aggregated token information.
| Field name | Description |
|---|---|
| available_tokens | Number of tokens available. Unclear what this means. |
| total_tokens | Total number of tokens included in the installed Qlik Sense license. Unclear what this means. |
Butler uptime info
Measurement: butler_memory_usage
Tags
| Tag name | Description |
|---|---|
| butler_instance | Name of the Butler instance, from config file. |
| version | Version of Butler. |
Fields
| Field name | Description |
|---|---|
| heap_used | Amount of heap memory used by Butler. |
| heap_total | Total amount of heap memory available to Butler. |
| external | Amount of external memory used by Butler. |
| process_memory | Amount of memory used by the Butler process. |
9 - Helper subs that make it easy to use Butler APIs
Butler exposes a REST API that can be called from many different systems and tools, just not Qlik Sense itself.
When it comes to Sense, it’s very doable but not entirely easy to call REST APIs from an app’s load script. Butler comes with a set of subs/functions that wrap the API calls and make it a lot easier and cleaner to call Butler APIs from load scripts.
This page describes those subs.
Currently available helper subs
These sub() are available in the the file butler_subs.qvs in the GitHub repo.
They are also part of the demo app Butler 5.2 demo app, which is also available in the GitHub repo.
| Sub name | Description |
|---|---|
| NiceTrace | Prints timestamped messages in the reload log. |
| ButlerInit | Initializes some data structures used by Butler. Right now there isn’t much done in this sub, but this may change in future versions. |
| GetEnabledButlerAPIEndpoints | Gets a list of which Butler API endpoints are enabled. |
| ReloadSenseApp | Does a full or partial app reload in the engine. No reload tasks are involved in the actual reload, but when the reload is done different reload tasks can be started depending on whether the reload was successful or not. |
| StartTask_KeyValue | Same as above, but a key-value pair can also be included via the extra parameters. This KV pair is saved in Butler’s KV store (assuming it’s enabled). Any Sense app (or other system) can then query the KV store for these parameters. It’s thus an easy but effective way of passing arbitrary parameters to Sense tasks (or rather to the associated app’s load scripts). |
| StartTask | Start a single or many reload tasks. Tasks can be selected using task IDs, tags and custom properties. Full freedom to use any number of task IDs, tags and custom properties to specify what tasks should be started. Any number of key-value pairs can be passed along as parameters. |
| CopyFile | Copy a file or directory in a file system that’s accessible to Butler and also approved in the config file. |
| MoveFile | Move a file or directory in a file system that’s accessible to Butler and also approved in the config file. |
| DeleteFile | Delete a file in a file system that’s accessible to Butler and also approved in the config file. |
| CreateDir | Create new directory anywhere Butler has access. Warning! If Butler is running with admin privileges there is no stopping a user from creating directories anywhere. |
| CreateDirQVD | Create a directory somewhere under the QVD folder specified in the config file’s Butler.configDirectories.qvdPath. This is less of a security concern than the previous call, of course. |
| AddKeyValue | Add a key-value pair to a namespace in the Key-Value (KV) store. |
| GetKeyValue | Get KV pair in a namespace. |
| DeleteKeyValue | Delete a KV pair in a namespace. |
| ExistsKeyValue | Tests if a KV pair exists in a namespace. |
| DeleteKeyValueNamespace | Delete a namespace and all KV pairs in it. |
| GetKeyValueNamespaces | Get all namespaces. |
| GetKeyValueNamespaces | Get all keys in a specifi namespace. |
| PostToMQTT | Post a message to a MQTT topic. |
| PostToSlack | Post a basic message to Slack. |
Helper subs on the todo list
There should be helper subs for these API endpoints too, they’re just not there yet…
| Sub name | Description |
|---|---|
| AddSchedule | Add a new task reload schedule to Butler’s scheduler. |
| GetSchedule | Get information about an existing Butler task schedule. |
| UpdateSchedule | Modify an existing Butler task schedule. |
| DeleteSchedule | Delete a Butler task schedule. |
| StartSchedule | Enable a Butler task schedule. |
| StopSchedule | Disable/pause a Butler task schedule. |
| PostToTeams | Make it easy to post basic messages to MS Teams. |
10 - Test cases
Automated test cases are used to ensure that (mainly) Butler’s REST API works as intended.
All test cases are documented in an Excel file stored in the tests directory in the main Butler repository.