This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Reference docs

Detailed, more technical descriptions of Butler’s REST API and other parts of Butler.

1 - Command line options

Description of Butler’s command line options.

Command line options

When starting Butler, you can pass command line options to customize its behavior.
Looks like this:

Usage: butler [options]

Butler gives superpowers to client-managed Qlik Sense Enterprise on Windows!
Advanced reload failure alerts, task scheduler, key-value store, file system access and much more.

Options:
  -V, --version                        output the version number
  -c, --configfile <file>              path to config file
  -l, --loglevel <level>               log level (choices: "error", "warn", "info", "verbose", "debug", "silly")
  --new-relic-account-name  <name...>  New Relic account name. Used within Butler to differentiate between different target New Relic accounts
  --new-relic-api-key <key...>         insert API key to use with New Relic
  --new-relic-account-id <id...>       New Relic account ID
  --test-email-address <address>       send test email to this address. Used to verify email settings in the config file.
  --test-email-from-address <address>  send test email from this address. Only relevant when SMTP server allows from address to be set.
  --no-qs-connection                   don't connect to Qlik Sense server at all. Run in isolated mode
  --api-rate-limit                     set the API rate limit, per minute. Default is 100 calls/minute. Set to 0 to disable rate limiting.
  --skip-config-verification           Disable config file verification (default: false)
  -h, --help                           display help for command

-V, –version

Output the version number of Butler.

-c, –configfile

Specifies the configuration file to use.

Valid values: A path to a configuration file.

Default: Whatever is specified in the NODE_ENV environment variable, with a .yaml extension added. Butler will look for that file in the ./config directory.

Example:

  • -c or --configfile are not specified. NODE_ENV is set to production. Butler will try to read settings from ./config/production.yaml.

-l, –loglevel

Specifies the log level to use.
When set, this overrides the log level specified in the configuration file.

Valid values: ’error’, ‘warn’, ‘info’, ‘verbose’, ‘debug’, ‘silly’

Default: ‘info’

When using New Relic as backend for storing logs, information about failed reloads etc, you can specify New Relic credentials in the config file - but that is not ideal from a security perspective.

To avoid that, you can specify the New Relic credentials on the command line using the following options.

–new-relic-account-name

List of New Relic account names. Used within Butler to differentiate between different target New Relic accounts to which data can be sent. This name has nothing to do with the account name used in New Relic - it’s purely for Butler’s internal use.
Specifically, it’s at multiple places in the config file where you can specificy to which New Relic account to send data.

Enclose account names in quotes if they contain spaces.
Separate multiple account names with a space.

Example: --new-relic-account-name "Account 1" "Account 2"

–new-relic-api-key

List of New Relic API keys. Used to authenticate with New Relic.

Enclose API keys in quotes if they contain spaces.
Separate multiple API keys with a space. Note that the order of the API keys must match the order of the account names, i.e. the first API key corresponds to the first account name, the second API key corresponds to the second account name, and so on.

Example: --new-relic-api-key "API key 1" "API key 2"

–new-relic-account-id

List of New Relic account IDs. Used to identify the New Relic account to which data should be sent.

Enclose account IDs in quotes if they contain spaces.
Separate multiple account IDs with a space. Note that the order of the account IDs must match the order of the account names, i.e. the first account ID corresponds to the first account name, the second account ID corresponds to the second account name, and so on.

Send test email

Butler has several alerting features for sending emails when some event happens (typoically something failing…) in Sense.

To verify that the email settings in the config file are correct, you can send a test email using the following options.

–test-email-address

Send a test email to this address, using the SMTP configuration in the config file.
Used to verify email settings in the config file.

This is the destination address for the test email.

–test-email-from-address

Send a test email to this address, using the SMTP configuration in the config file.
Used to verify email settings in the config file.

This is the sender address for the test email. Only relevant when the SMTP server allows the from address to be set.

–no-qs-connection

Don’t connect to Qlik Sense server at all. Run in isolated mode.

This is probably only relevant when you’re developing Butler and don’t have a Qlik Sense server available, or when you want to automatically exporting the API docs for Butler’s REST API (this is done during the build process).

–api-rate-limit

Set the rate limit of Butler’s REST API, per minute. Default is 100 calls/minute. Set to 0 to disable rate limiting.

Useful when you want to ensure that Butler (or the server it runs on) doesn’t get overloaded with requests.

–skip-config-verification

Disable config file verification.

By default, Butler verifies the config file when it starts. If the config file is invalid, Butler will log an error and exit.
Use this option to disable config file verification.

-h, –help

Display help for command.

2 - Config file syntax

Description of Butler’s config file.

Main Butler config file

The production_template.yaml config file looks like this (sorry for the incorrect syntax coloring, the issue is noted and is being worked on):

Info

Starting with Butler version 9.0 there is a check that the config file has the correct format.

This means that if you forget to add or change some setting in the main YAML config file, Butler will tell you what’s missing and refuse to start.
A consequence of this is that all settings are now mandatory, even if you don’t use them.

Trying to start Butler without some mandatory settings in the config file will result in an error message like this:

Starting Butler with missing setting in YAML config file

Adding the missing settings and restarting Butler will result in a successful startup.

---
Butler:
  # General notes: 
  # - File and directory paths in this sample config file use Linux/Mac syntax, i.e. using forward slashes.
  #   Windows paths work just as well, just make sure to quote them with single or double quotes.
  # - All entries in the config file are mandatory in the send that they must be present.
  #   However, if a feature is not used the corresponding config entries can contain 
  #   any value (for example the provided default ones).

  # Logging configuration
  logLevel: info          # Log level. Possible log levels are silly, debug, verbose, info, warn, error
  fileLogging: false      # true/false to enable/disable logging to disk file
  logDirectory: log       # Directory where log files are stored (no trailing / )
  anonTelemetry: true     # Can Butler send anonymous telemetry data? 
                          # More info on whata data is collected: https://butler.ptarmiganlabs.com/docs/about/telemetry/
                          # Please consider leaving this at true - it really helps future development of Butler!

  # Heartbeats can be used to send "I'm alive" messages to any other tool, e.g. an infrastructure monitoring tool
  heartbeat:
    enable: false
    remoteURL: http://my.monitoring.server/some/path/
    frequency: every 30 seconds         # https://bunkat.github.io/later/parsers.html

  # Docker health checks are used when running Butler as a Docker container. 
  # The Docker engine will call the container's health check REST endpoint with a set interval to determine
  # whether the container is alive/well or not.
  # If you are not running Butler in Docker you can safely disable this feature. 
  dockerHealthCheck:
    enable: false    # Control whether a REST endpoint will be set up to serve Docker health check messages
    port: 12398      # Port the Docker health check service runs on (if enabled)

  # Uptime monitor
  uptimeMonitor:
    enable: false                   # Should uptime messages be written to the console and log files?
    frequency: every 15 minutes     # https://bunkat.github.io/later/parsers.html
    logLevel: verbose               # Starting at what log level should uptime messages be shown?
    storeInInfluxdb: 
      enable: false                 # Should Butler memory usage be logged to InfluxDB?
    storeNewRelic:
      enable: false
      destinationAccount:
        - First NR account
        - Second NR account
      # There are different URLs depending on whther you have an EU or US region New Relic account.
      # The available URLs are listed here: https://docs.newrelic.com/docs/accounts/accounts-billing/account-setup/choose-your-data-center/
      # As of this writing the options for the New Relic metrics API are
      # https://insights-collector.eu01.nr-data.net/metric/v1
      # https://metric-api.newrelic.com/metric/v1 
      url: https://insights-collector.eu01.nr-data.net/metric/v1   # Where should uptime data be sent?
      header:                       # Custom http headers
        - name: X-My-Header
          value: Header value
      metric:
        dynamic:
          butlerMemoryUsage:
            enable: true            # Should Butler's memory/RAM usage be sent to New Relic?
          butlerUptime:
            enable: true            # Should Butler's uptime (how long since it was started) be sent to New Relic?
      attribute: 
        static:                     # Static attributes/dimensions to attach to the data sent to New Relic.
          - name: metricType
            value: butler-uptime
          - name: service
            value: butler
          - name: environment
            value: prod
        dynamic:
          butlerVersion: 
            enable: true            # Should the Butler version be included in the data sent to New Relic?

  # Credentials for third party systems that Butler integrate with.
  # These can also be specified via command line parameters when starting Butler. 
  # Command line options takes precedence over settings in this config file.
  thirdPartyToolsCredentials:
    newRelic:         # Array of New Relic accounts/insert keys. Any data sent to New Relic will be sent to both accounts. 
      - accountName: First NR account
        insertApiKey: <API key 1 (with insert permissions) from New Relic> 
        accountId: <New Relic account ID 1>
      - accountName: Second NR account
        insertApiKey: <API key 2 (with insert permissions) from New Relic> 
        accountId: <New Relic account ID 2>

  # InfluxDB settings
  influxDb:
    enable: false                   # Master switch for InfluxDB integration. If false, no data will be sent to InfluxDB.
    hostIP: influxdb.mycompany.com  # IP or FQDN of Influxdb server
    hostPort: 8086                  # Port where Influxdb is listening. Default=8086
    auth:
      enable: false                 # Does InfluxDB require login?
      username: user_joe      
      password: joesecret
    dbName: butler                  # Name of database in InfluxDB to which Butler's data is written
    instanceTag: DEV                # Tag that can be used to differentiate data from multiple Butler instances
    # Default retention policy that should be created in InfluxDB when Butler creates a new database there. 
    # Any data older than retention policy threshold will be purged from InfluxDB.
    retentionPolicy:
      name: 10d
      duration: 10d
    reloadTaskFailure:
      enable: true
      tailScriptLogLines: 20
      tag: 
        static:                 # Static tags to attach to data stored in InflixDB
          - name: butler_instance
            value: prod-1
        dynamic:
          useAppTags: true      # Should app tags be stored in InfluxDB as tags?
          useTaskTags: true     # Should task tags be stored in InfluxDB as tags?      
    reloadTaskSuccess:
      enable: true
      allReloadTasks:
        enable: false
      byCustomProperty:
        enable: true
        customPropertyName: 'Butler_SuccessReloadTask_InfluxDB'
        enabledValue: 'Yes'
      tag: 
        static:                 # Static attributes/dimensions to attach to events sent to InfluxDb
          # - name: event-specific-tag 1
          #   value: abc 123
        dynamic:
          useAppTags: true      # Should app tags be sent to InfluxDb as tags?
          useTaskTags: true     # Should task tags be sent to InfluxDb as tags?

  # Store script logs of failed reloads on disk.
  # The script logs will be stored in daily directories under the specified main directory below
  # NOTE: Use an absolute path when running Butler as a standalone executable! 
  scriptLog:
    storeOnDisk:
      reloadTaskFailure:
        enable: false
        logDirectory: /path/to/scriptlogs

  # Qlik Sense related links used in notification messages
  qlikSenseUrls:
    qmc: <Link to Qlik Sense QMC>
    hub: <Link to Qlik Sense Hub>

  # Settings for monitoring Qlik Sense licenses
  qlikSenseLicense:
    licenseMonitor:
      enable: false
      frequency: every 6 hours        # https://bunkat.github.io/later/parsers.html#text
      destination:
        influxDb:                     # Store license data in InfluxDB
          enable: true
          tag: 
            static:                   # Static attributes/tags to attach to the data sent to InflixDB
              - name: foo
                value: bar
    licenseRelease:
      enable: false                    # true/false. If true, Butler will release unused licenses according to settings below
      dryRun: true                    # true/false. If true, Butler will not actually release any licenses, just log what it would have done. 
      frequency: every 6 hours        # https://bunkat.github.io/later/parsers.html#text
      neverRelease:                   # Various ways of defining which users should never have their licenses released
        user:                         # Users who should never have their licenses released
          - userDir: 'INTERNAL'
            userId: 'sa_repository'
          - userDir: 'INTERNAL'
            userId: 'sa_api'
          - userDir: 'USERDIR'
            userId: 'qs_admin_account'
        tag:                          # Users with these tags will never have their licenses released
          - License do not release
          - some other tag
        customProperty:               # Users with these custom properties will never have their licenses released
          - name: LicenseManage
            value: do-not-release
        userDirectory:                # List of user directories whose users should never have their licenses released
          - INTERNAL
          - ADMIN
        inactive: Ignore              # Ignore/Yes/No. The value is case insensitive
                                      #   No = Don't release licenses for users marked as "Inactive=No" in the QMC
                                      #   Yes = Don't release licenses for users marked as "Inactive=Yes" in the QMC 
                                      #   Ignore = Disregard this setting
        blocked: Ignore               # Ignore/Yes/No, No = Don't release licenses for users marked as "Blocked=No" in the QMC
        removedExternally: ignore     # Ignore/Yes/No, No = Don't release licenses for users marked as "Removed externally=No" in the QMC
      licenseType:                    # License types to monitor and release
        analyzer:                     
          enable: true                # Monitor and release Analyzer licenses
          releaseThresholdDays: 30    # Number of days a license can be unused before it is released
        professional:
          enable: true                # Monitor and release Professional licenses
          releaseThresholdDays: 30    # Number of days a license can be unused before it is released
      destination:
        influxDb:                     # Store info about released licenses in InfluxDB
          enable: true
          tag: 
            static:                   # Static attributes/tags to attach to the data sent to InflixDB
              - name: foo
                value: bar

  # Settings for notifications and messages sent to MS Teams
  teamsNotification:
    enable: false
    reloadTaskFailure:
      enable: false
      webhookURL: <web hook URL from MS Teams>
      messageType: formatted     # formatted / basic. Formatted means that template file below will be used to create the message.
      basicMsgTemplate: 'Qlik Sense reload failed: "{{taskName}}"'      # Only needed if message type = basic
      rateLimit: 300             # Min seconds between emails for a given taskID. Defaults to 5 minutes.
      headScriptLogLines: 10
      tailScriptLogLines: 10
      templateFile: /path/to/teams/template/directory/failed-reload.handlebars
    reloadTaskAborted:
      enable: false
      webhookURL: <web hook URL from MS Teams>
      messageType: formatted     # formatted / basic. Formatted means that template file below will be used to create the message.
      basicMsgTemplate: 'Qlik Sense reload aborted: "{{taskName}}"'       # Only needed if message type = basic
      rateLimit: 300             # Min seconds between emails for a given taskID. Defaults to 5 minutes.
      headScriptLogLines: 10
      tailScriptLogLines: 10
      templateFile: /path/to/teams/template/directory/aborted-reload.handlebars
    serviceStopped:
      webhookURL: <web hook URL from MS Teams>
      messageType: formatted          # formatted / basic. Formatted means that template file below will be used to create the message.
      basicMsgTemplate: 'Windows service stopped: "{{serviceName}}" on host "{{host}}"'       # Only needed if message type = basic
      rateLimit: 30                   # Min seconds between messages for a given Windows service. Defaults to 5 minutes.
      templateFile: /path/to/teams/template/directory/service-stopped.handlebars
    serviceStarted:
      webhookURL: <web hook URL from MS Teams>
      messageType: formatted          # formatted / basic. Formatted means that template file below will be used to create the message.
      basicMsgTemplate: 'Windows service started: "{{serviceName}}" on host "{{host}}"'       # Only needed if message type = basic
      rateLimit: 30                   # Min seconds between messages for a given Windows service. Defaults to 5 minutes.
      templateFile: /path/to/teams/template/directory/service-started.handlebars


  # Settings for notifications and messages sent to Slack
  slackNotification:
    enable: false
    restMessage:                      
      webhookURL: <web hook URL from Slack>   # Webhook to use when sending basic Slack messages via Butler's REST API 
    reloadTaskFailure:
      enable: false
      webhookURL: <web hook URL from Slack>
      channel: sense-task-failure     # Slack channel to which task failure notifications are sent
      messageType: formatted          # formatted / basic. Formatted means that template file below will be used to create the message.
      basicMsgTemplate: 'Qlik Sense reload failed: "{{taskName}}"'      # Only needed if message type = basic
      rateLimit: 300                  # Min seconds between emails for a given taskID. Defaults to 5 minutes.
      headScriptLogLines: 10
      tailScriptLogLines: 10
      templateFile: /path/to/slack/template/directory/failed-reload.handlebars
      fromUser: Qlik Sense
      iconEmoji: ':ghost:'
    reloadTaskAborted:
      enable: false
      webhookURL: <web hook URL from Slack>
      channel: sense-task-aborted     # Slack channel to which task stopped notifications are sent
      messageType: formatted          # formatted / basic. Formatted means that template file below will be used to create the message.
      basicMsgTemplate: 'Qlik Sense reload aborted: "{{taskName}}"'       # Only needed if message type = basic
      rateLimit: 300                  # Min seconds between emails for a given taskID. Defaults to 5 minutes.
      headScriptLogLines: 10
      tailScriptLogLines: 10
      templateFile: /path/to/slack/template/directory/aborted-reload.handlebars
      fromUser: Qlik Sense
      iconEmoji: ':ghost:'
    serviceStopped:
      webhookURL: <web hook URL from Slack>
      channel: qliksense-service-alert  # Slack channel to which Windows service stopped notifications are sent
      messageType: formatted          # formatted / basic. Formatted means that template file below will be used to create the message.
      basicMsgTemplate: 'Windows service stopped: "{{serviceName}}" on host "{{host}}"'       # Only needed if message type = basic
      rateLimit: 30                   # Min seconds between messages for a given Windows service. Defaults to 5 minutes.
      templateFile: /path/to/slack/template/directory/service-stopped.handlebars
      fromUser: Qlik Sense
      iconEmoji: ':ghost:'
    serviceStarted:
      webhookURL: <web hook URL from Slack>
      channel: qliksense-service-alert  # Slack channel to which Windows service stopped notifications are sent
      messageType: formatted          # formatted / basic. Formatted means that template file below will be used to create the message.
      basicMsgTemplate: 'Windows service started: "{{serviceName}}" on host "{{host}}"'       # Only needed if message type = basic
      rateLimit: 30                   # Min seconds between messages for a given Windows service. Defaults to 5 minutes.
      templateFile: /path/to/slack/template/directory/service-started.handlebars
      fromUser: Qlik Sense
      iconEmoji: ':ghost:'

  # Settings needed to send email notifications when for example reload tasks fail.
  # Reload failure notifications assume a log appender is configured in Sense AND that the UDP server in Butler is running.
  emailNotification:
    enable: false
    reloadTaskAborted:
      enable: false
      appOwnerAlert:
        enable: true              # Should app owner get notification email (assuming email address is available in Sense user directory)
        includeOwner:
          includeAll: true                            # true = Send notification to all app owners except those in exclude list
                                                      # false = Send notification to app owners in the include list
          user:
            - directory: <Sense user directory>
              userId: <userId>
            - directory: <Sense user directory>
              userId: <userId>
        excludeOwner:
          user:
            - directory: <Sense user directory>
              userId: <userId>
            - directory: <Sense user directory>
              userId: <userId>
      # Custom property used to control which aborted tasks will cause alert emails to be sent
      # If this setting is true, alerts will not be sent for all tasks, but *only* for tasks with the CP set to the enabledValue.
      # If this setting is false, alerts will be sent for all aborted reload tasks.
      alertEnableByCustomProperty:
        enable: true
        customPropertyName: 'Butler_AbortedAlertEnableEmail'
        enabledValue: 'Yes'
      # Custom property used to say that alerts for a certain task should be sent to zero or more recipients
      # These alerts will be sent irrespective of the alertEnableByCustomProperty.enable setting.
      alertEnabledByEmailAddress:
        customPropertyName: 'Butler_AbortedAlertSendToEmail'
      rateLimit: 600                                  # Min seconds between emails for a given taskID. Defaults to 5 minutes.
      headScriptLogLines: 15                          # Number of lines from start of script to include in email
      tailScriptLogLines: 15                          # Number of lines from end of script to include in email
      priority: high                                  # high/normal/low
      subject: 'Qlik Sense reload aborted: "{{taskName}}"'  # Email subject. Can use template fields
      bodyFileDirectory: path/to/email_templates      # Directory where email body template files are stored
      htmlTemplateFile: aborted-reload                # Name of email body template file to use
      fromAdress: Qlik Sense (no-reply) <qliksense-noreply@mydomain.com>
      recipients:                                     # Array of email addresses to which the notification email will be sent
        - <Email address 1>
        - <Email address 2>
    reloadTaskFailure:
      enable: false
      appOwnerAlert:
        enable: true              # Should app owner get notification email (assuming email address is available in Sense user directory)
        includeOwner:
          includeAll: true                            # true = Send notification to all app owners except those in exclude list
                                                      # false = Send notification to app owners in the include list
          user:
            - directory: <Sense user directory>
              userId: <userId>
            - directory: <Sense user directory>
              userId: <userId>
        excludeOwner:
          user:
            - directory: <Sense user directory>
              userId: <userId>
            - directory: <Sense user directory>
              userId: <userId>
      # Custom property used to control which task failures will cause alert emails to be sent
      # If this setting is true, alerts will not be sent for all tasks, but *only* for tasks with the CP set to the enabledValue.
      # If this setting is false, alerts will be sent for all failed reload tasks.
      alertEnableByCustomProperty:
        enable: false
        customPropertyName: 'Butler_FailedAlertEnableEmail'
        enabledValue: 'Yes'
      # Custom property used to say that alerts for a certain task should be sent to zero or more recipients
      # These alerts will be sent irrespective of the alertEnableByCustomProperty.enable setting.
      alertEnabledByEmailAddress:
        customPropertyName: 'Butler_FailedAlertSendToEmail'
      rateLimit: 600                                  # Min seconds between emails for a given taskID. Defaults to 5 minutes.
      headScriptLogLines: 15                          # Number of lines from start of script to include in email
      tailScriptLogLines: 15                          # Number of lines from end of script to include in email
      priority: high                                  # high/normal/low
      subject: 'Qlik Sense reload failed: "{{taskName}}"'   # Email subject. Can use template fields
      bodyFileDirectory: path/to/email_templates      # Directory where email body template files are stored
      htmlTemplateFile: failed-reload                 # Name of email body template file to use
      fromAdress: Qlik Sense (no-reply) <qliksense-noreply@mydomain.com>
      recipients:                                       # Array of email addresses to which the notification email will be sent
        - <Email address 1>
        - <Email address 2>
    serviceStopped:
      rateLimit: 30                   # Min seconds between emails for a given service. Defaults to 5 minutes.
      priority: high                  # high/normal/low
      subject: '❌ Windows service stopped on host {{host}}: "{{serviceDisplayName}}"'
      bodyFileDirectory: path/to/email_templates/email_templates
      htmlTemplateFile: service-stopped
      fromAdress: Qlik Sense (no-reply) <qliksense-noreply@mydomain.com>
      recipients:
        - <Email address 1>
        - <Email address 2>
    serviceStarted:
      rateLimit: 30                   # Min seconds between emails for a given service. Defaults to 5 minutes.
      priority: high                  # high/normal/low
      subject: '✅ Windows service started on host {{host}}: "{{serviceDisplayName}}"'
      bodyFileDirectory: path/to/email_templates/email_templates
      htmlTemplateFile: service-started
      fromAdress: Qlik Sense (no-reply) <qliksense-noreply@mydomain.com>
      recipients:
        - <Email address 1>
        - <Email address 2>
    smtp:                                             # Email server settings. See https://nodemailer.com/smtp/ for details on the meaning of these fields.
      host: <FQDN or IP or email server, e.g. smtp.gmail.com>
      port: <port on which SMTP server is listening>
      secure: true                                    # true/false
      tls:
        serverName:                                   # If specified the serverName field will be used for TLS verification instead of the host field.
        ignoreTLS: false
        requireTLS: true
        rejectUnauthorized: false
      auth:
        enable: true
        user: <Username, email address etc>
        password: <your-secret-password>

  # Incident management tools integration
  # Used to trigger incidents in these tools when task reloads fail or are aborted.
  incidentTool:
    signl4:
      enable: false               # Enable/disable Signl4 integration as a whole
      url: https://connect.signl4.com/webhook/abcde12345
      reloadTaskFailure:
        enable: false             # Enable/disable reload failed handling in Signl4
        rateLimit: 15             # Min seconds between emails for a given taskID. Defaults to 5 minutes
        serviceName: Qlik Sense   # Signl4 "service name" to use
        severity: 1               # Signl4 severity level for failed reloads
        includeApp:
          includeAll: false
          appId:
            # - 47d38f73-628f-44e1-a62c-841604b123ff
      reloadTaskAborted:
        enable: false             # Enable/disable reload aborted handling in Signl4
        rateLimit: 15             # Min seconds between emails for a given taskID. Defaults to 5 minutes
        serviceName: Qlik Sense   # Signl4 "service name" to use
        severity: 10              # Signl4 severity level for aborted reloads
        includeApp:
          includeAll: false
          appId:
            # - 47d38f73-628f-44e1-a62c-841604b123ff
    newRelic:
      enable: false
      destinationAccount:
        event:                    # Failed/aborted reload tasks are sent as events to these New Relic accounts
          - First NR account
          - Second NR account
        log:                      # Failed/aborted reload tasks are sent as log entries to these New Relic accounts
          - First NR account
          - Second NR account
      # New Relic uses different API URLs for different kinds of data (metrics, events, logs, ...)
      # There are different URLs depending on whther you have an EU or US region New Relic account.
      # The available URLs are listed here: https://docs.newrelic.com/docs/accounts/accounts-billing/account-setup/choose-your-data-center/
      url:
        # As of this writing the valid options are
        # https://insights-collector.eu01.nr-data.net
        # https://insights-collector.newrelic.com 
        event: https://insights-collector.eu01.nr-data.net

        # Valid options are (1) EU/rest of world and 2) US)
        # https://log-api.eu.newrelic.com/log/v1
        # https://log-api.newrelic.com/log/v1 
        log: https://log-api.eu.newrelic.com/log/v1
      reloadTaskFailure:
        destination:
          event: 
            enable: false
            sendToAccount:              # Which reload task failures are sent to New Relic as events
              byCustomProperty:
                enable: false            # Control using a task custom property which reload task failures are sent as events
                customPropertyName: 'Butler_FailedTask_Event_NewRelicAccount'
              always:
                enable: false            # Controls which New Relic accounts ALL failed reload tasks are sent to (as events)
                account: 
                  - First NR account
                  - Second NR account
            attribute: 
              static:                 # Static attributes/dimensions to attach to events sent to New Relic.
                - name: event-specific-attribute 1  # Example
                  value: abc 123                    # Example
              dynamic:
                useAppTags: true      # Should app tags be sent to New Relic as attributes?
                useTaskTags: true     # Should task tags be sent to New Relic as attributes?
          log:
            enable: false
            tailScriptLogLines: 20
            sendToAccount:              # Which reload task failures are sent to New Relic as log entries
              byCustomProperty:
                enable: false            # Control using a task custom property which reload task failures are sent as log entries
                customPropertyName: 'Butler_FailedTask_Log_NewRelicAccount'
              always:
                enable: false            # Controls which New Relic accounts ALL failed reload tasks are sent to (as logs)
                account: 
                  - First NR account
                  - Second NR account
            attribute: 
              static:                 # Static attributes/dimensions to attach to events sent to New Relic.
                - name: log-specific-attribute 1    # Example
                  value: def 123                    # Example
              dynamic:
                useAppTags: true      # Should app tags be sent to New Relic as attributes?
                useTaskTags: true     # Should task tags be sent to New Relic as attributes?
        sharedSettings:
          rateLimit: 15             # Min seconds between events sent to New Relic for a given taskID. Defaults to 5 minutes.
          header:                   # Custom http headers
            - name: X-My-Header     # Example
              value: Header value 1 # Example
          attribute: 
            static:                 # Static attributes/dimensions to attach to events sent to New Relic.
              - name: service       # Example
                value: butler       # Example
              - name: environment   # Example
                value: prod         # Example
      reloadTaskAborted:
        destination:
          event: 
            enable: false
            sendToAccount:              # Which reload task aborts are sent to New Relic as events
              byCustomProperty:
                enable: false            # Control using a task custom property which reload task aborts are sent as events
                customPropertyName: 'Butler_AbortedTask_Event_NewRelicAccount'
              always:
                enable: false            # Controls which New Relic accounts ALL aborted reload tasks are sent to (as events)
                account: 
                  - First NR account
                  - Second NR account
            attribute: 
              static:                 # Static attributes/dimensions to attach to events sent to New Relic.
                - name: event-specific-attribute 2  # Example
                  value: abc 123                    # Example
              dynamic:
                useAppTags: true      # Should app tags be sent to New Relic as attributes?
                useTaskTags: true     # Should task tags be sent to New Relic as attributes?
          log:
            enable: false
            tailScriptLogLines: 20
            sendToAccount:              # Which reload task aborts are sent to New Relic as log entries
              byCustomProperty:
                enable: true            # Control using a task custom property which reload task aborts are sent as log entries
                customPropertyName: 'Butler_AbortedTask_Log_NewRelicAccount'
              always:
                enable: false          # Controls which New Relic accounts ALL aborted reload tasks are sent to (as logs)
                account: 
                  - First NR account
                  - Second NR account
            attribute: 
              static:                 # Static attributes/dimensions to attach to events sent to New Relic.
                - name: log-specific-attribute 2    # Example
                  value: def 123                    # Example
              dynamic:
                useAppTags: true      # Should app tags be sent to New Relic as attributes?
                useTaskTags: true     # Should task tags be sent to New Relic as attributes?
        sharedSettings:
          rateLimit: 15             # Min seconds between events sent to New Relic for a given taskID. Defaults to 5 minutes.
          header:                   # Custom http headers
            - name: X-My-Header     # Example
              value: Header value 2 # Example
          attribute: 
            static:                 # Static attributes/dimensions to attach to events sent to New Relic.
              - name: service       # Example
                value: butler       # Example
              - name: environment   # Example
                value: prod         # Example
      serviceMonitor:
        destination:
          event: 
            enable: false
            sendToAccount:                # Windows service events are sent to these New Relic accounts
              - First NR account
              - Second NR account
            attribute: 
              static:                     # Static attributes/dimensions to attach to events sent to New Relic.
                - name: event-specific-attribute
                  value: abc 123
              dynamic:
                serviceHost: true         # Should host where service is running be sent to New Relic as attribute?
                serviceName: true         # Should service name be sent to New Relic as attribute?
                serviceDisplayName: true  # Should service display name be sent to New Relic as attribute?
                serviceState: true        # Should service state be sent to New Relic as attribute?
          log:
            enable: false
            sendToAccount:                # Windows service log entries are sent to these New Relic accounts
              - First NR account
              - Second NR account
            attribute: 
              static:                     # Static attributes/dimensions to attach to events sent to New Relic.
                - name: log-specific-attribute
                  value: def 456
              dynamic:
                serviceHost: true         # Should host where service is running be sent to New Relic as attribute?
                serviceName: true         # Should service name be sent to New Relic as attribute?
                serviceDisplayName: true  # Should service display name be sent to New Relic as attribute?
                serviceState: true        # Should service state be sent to New Relic as attribute?
        monitorServiceState:              # Control whih service states are sent to New Relic
          running:
            enable: true
          stopped:
            enable: true
        sharedSettings:
          rateLimit: 5                    # Min seconds between events/logs sent to New Relic for a given host+service. Defaults to 5 minutes.
          header:                         # Custom http headers
            - name: X-My-Header           # Example
              value: Header value 2       # Example
          attribute: 
            static:                       # Static attributes/dimensions to attach to events sent to New Relic.
              - name: service             # Example
                value: butler             # Example
              - name: environment         # Example
                value: prod               # Example

  # Settings for notifications and messages sent using outgoing webhooks
  webhookNotification:
    enable: false
    reloadTaskFailure:
      rateLimit: 300              # Min seconds between outgoing webhook calls for a given taskID. Defaults to 5 minutes.
      webhooks:
        - description: 'This outgoing webhook is used to...'  # Informational only
          webhookURL: http://host.my.domain:port/some/path    # outgoing webhook that Butler will call
          httpMethod: POST                                    # GET/POST/PUT
        - description: 'This outgoing webhook is used to...'  # Informational only
          webhookURL: http://host.my.domain:port/some/path    # outgoing webhook that Butler will call
          httpMethod: PUT                                     # GET/POST/PUT.
        - description: 'This outgoing webhook is used to...'  # Informational only
          webhookURL: http://host.my.domain:port/some/path    # outgoing webhook that Butler will call
          httpMethod: GET                                     # GET/POST/PUT
    reloadTaskAborted:
      rateLimit: 300              # Min seconds between outgoing webhook calls for a given taskID. Defaults to 5 minutes.
      webhooks:
        - description: 'This outgoing webhook is used to...'  # Informational only
          webhookURL: http://host.my.domain:port/some/path    # outgoing webhook that Butler will call
          httpMethod: PUT                                     # GET/POST/PUT
        - description: 'This outgoing webhook is used to...'  # Informational only
          webhookURL: http://host.my.domain:port/some/path    # outgoing webhook that Butler will call
          httpMethod: POST                                    # GET/POST/PUT
        - description: 'This outgoing webhook is used to...'  # Informational only
          webhookURL: http://host.my.domain:port/some/path    # outgoing webhook that Butler will call
          httpMethod: GET                                     # GET/POST/PUT
    serviceMonitor:
      rateLimit: 15               # Min seconds between outgoing webhook calls, per Windows service that is monitored. Defaults to 5 minutes.
      webhooks:
        - description: 'This outgoing webhook is used to...'
          webhookURL: http://host.my.domain:port/some/path    # outgoing webhook that Butler will call
          httpMethod: POST                                    # GET/POST/PUT. Note that the body and URL query parameters differs depending on which method is used
        - description: 'This outgoing webhook is used to...'
          webhookURL: http://host.my.domain:port/some/path    # outgoing webhook that Butler will call
          httpMethod: PUT                                     # GET/POST/PUT. Note that the body and URL query parameters differs depending on which method is used
        - description: 'This outgoing webhook is used to...'
          webhookURL: http://host.my.domain:port/some/path    # outgoing webhook that Butler will call
          httpMethod: GET                                     # GET/POST/PUT. Note that the body and URL query parameters differs depending on which method is used

  # Scheduler for Qlik Sense tasks
  scheduler:
    enable: false                                     # Should Butler's reload task scheduler be started?
    configfile: config/schedule.yaml                  # Path to file containing task start schedules

  # Key-value store
  keyValueStore:
    enable: false                                     # Should Butler's key-value store be enabled?
    maxKeysPerNamespace: 1000                         # Max keys that can be stored per namespace. Defaults to 1000 if not specified in this file.

  mqttConfig:
    enable: false                                     # Should Qlik Sense events be forwarded as MQTT messages?
    brokerHost: <FQDN or IP of MQTT server>
    brokerPort: 1883
    azureEventGrid:
      enable: false              # If set to true, Butler will connect to an Azure Event Grid MQTT Broker, using brokerHost and brokerPort above 
      clientId: <client ID>
      clientCertFile: <path to client certificate file>
      clientKeyFile: <path to client key file>
    taskFailureSendFull: true
    taskAbortedSendFull: true
    subscriptionRootTopic: qliksense/#                                  # Topic that Butler will subscribe to
    taskStartTopic: qliksense/start_task                                # Topic for incoming messages used to start Sense tasks. Should be subtopic to subscriptionRootTopic
    taskFailureTopic: qliksense/task_failure
    taskFailureFullTopic: qliksense/task_failure_full
    taskFailureServerStatusTopic: qliksense/butler/task_failure_server
    taskAbortedTopic: qliksense/task_aborted
    taskAbortedFullTopic: qliksense/task_aborted_full
    serviceRunningTopic: qliksense/service_running
    serviceStoppedTopic: qliksense/service_stopped
    serviceStatusTopic: qliksense/service_status

  udpServerConfig:
    enable: false                                     # Should the UDP server responsible for receving task failure/aborted events be started?
    serverHost: <FQDN or IP (or localhost) of server where Butler is running>
    portTaskFailure: 9998

  restServerConfig:
    enable: false                                     # Should Butler's REST API be started? Must be true if *any* API endpoints are to be used.
    serverHost: <FQDN or IP (or localhost) of server where Butler is running>   # Use 0.0.0.0 to listen on all network interfaces (e.g. when running in Docker!).
    serverPort: 8080                                  # Port where Butler's REST is available. Any free port on the server where Butler is running can bse used.
    backgroundServerPort: 8081                        # Port used internally by Butler's REST API. Any free port on the server where Butler is running can bse used.

  # List of directories between which file copying via the REST API can be done.
  # Butler will try to clean up messy paths like this one, which resolves to /Users/goran/butler-test-dir1
  # How? First you have /Users/goran/butler-test-dir1//abc which cleans up to /Users/goran/butler-test-dir1/abc, 
  # then up one level (..).
  fileCopyApprovedDirectories:                        
    - fromDirectory: /Users/goran/butler-test-dir1//abc//..     
      toDirectory: /Users/goran/butler-test-dir2
    - fromDirectory: /Users/goran/butler-test-dir2
      toDirectory: /Users/goran/butler-test-dir1
    - fromDirectory: /from/some/directory2
      toDirectory: /to/some/directory2

  # List of directories between which file moves via the REST API can be done.
  fileMoveApprovedDirectories:                        
    - fromDirectory: /Users/goran/butler-test-dir1//abc//..
      toDirectory: /Users/goran/butler-test-dir2
    - fromDirectory: /Users/goran/butler-test-dir2
      toDirectory: /Users/goran/butler-test-dir1
    - fromDirectory: /from/some/directory2
      toDirectory: /to/some/directory2

  # List of directories in which file deletes via the REST API can be done.
  fileDeleteApprovedDirectories:                      
    - /Users/goran/butler-test-dir1
    - /Users/goran/butler-test-dir1//abc//..
    - /Users/goran/butler-test-dir2
  
  # If set to true, Butler will be started with a focus on creating an API documentation file
  # All configuration relating to outbound connetions (to Sense, email servers, MQTT broker etc) will be disabled.
  # NOTE: This setting should always be false (or just deleted), unless you want to regenerate the API doc files.
  restServerApiDocGenerate: false

  # Enable/disable individual REST API endpoints. Set config item below to true to enable that endpoint.
  restServerEndpointsEnable:
    apiListEnbledEndpoints: false
    base62ToBase16: false
    base16ToBase62: false
    butlerping: false
    createDir: false
    createDirQVD: false
    fileDelete: false
    fileMove: false
    fileCopy: false
    keyValueStore: false
    mqttPublishMessage: false
    newRelic:
      postNewRelicMetric: false
      postNewRelicEvent: false
    scheduler:
      createNewSchedule: false
      getSchedule: false
      getScheduleStatusAll: false
      updateSchedule: false
      deleteSchedule: false
      startSchedule: false
      stopSchedule: false
    senseAppReload: false
    senseAppDump: false
    senseListApps: false
    senseStartTask: false
    slackPostMessage: false 

  restServerEndpointsConfig:
    newRelic:
      postNewRelicMetric:          # Setings used by post metric to New Relic API endpoint
        destinationAccount:
          - First NR account
          - Second NR account
        # As of this writing the valid options are
        # https://insights-collector.eu01.nr-data.net/metric/v1
        # https://insights-collector.newrelic.com/metric/v1
        url: https://insights-collector.eu01.nr-data.net/metric/v1
        header:                   # Custom http headers
          - name: X-My-Header
            value: Header value
        attribute: 
          static:                   # Static attributes/dimensions to attach to the metrics data sent to New Relic.
            - name: env
              value: prod
      postNewRelicEvent:            # Setings used by post event to New Relic API endpoint
        destinationAccount:
          - First NR account
          - Second NR account
        # Note that the URL path should *not* be included in the url setting below!
        # As of this writing the valid options are
        # https://insights-collector.eu01.nr-data.net
        # https://insights-collector.newrelic.com 
        url: https://insights-collector.eu01.nr-data.net/
        header:                   # Custom http headers
          - name: X-My-Header
            value: Header value
        attribute: 
          static:                   # Static attributes/dimensions to attach to the metrics data sent to New Relic.
            - name: env
              value: prod

  # Controls which tasks can be started via Butler's REST API.
  # Enabling this feature gives Qlik Sense sysadmins a way to control which tasks can be started by third party systems and applications.
  # If this feature is disabled all tasks can be started via the API (assuming the start task API itself is enabled).
  # Note that the taskId, tag and customProperty sections below are additive. I.e. a task only has to appear in one of those sections to be on the "allowed" list.
  startTaskFilter:
    enable: false
    allowTask:
      taskId:
        # Zero or more approved/allowed task IDs
        # If Butler.startTaskFilter.enable is true, only task IDs listed below will be started by Butler 
        - e3b27f50-b1c0-4879-88fc-c7cdd9c1cf3e
        - 7552d9fc-d1bb-4975-9a38-18357de531ea
        - fb0f317d-da91-4b86-aafa-0174ae1e8c8f
      tag:
        # Zero or more tags used to indicate that a task is approved to be started by Butler.
        # Use the Qlik Sense QMC to set tags on tasks.
        # If Butler.startTaskFilter.enable is true, only tasks with the tags below will be started by Butler 
        - startTask1
        - startTask2
      customProperty:
        # Zero or more custom properties name/value pairs used to indicate that a task is approved to be started by Butler.
        # Use the Qlik Sense QMC to set custom properties on tasks.
        # If Butler.startTaskFilter.enable is true, only tasks with the custom property values below will be started by Butler 
        - name: taskGroup
          value: tasks1
        - name: taskGroup
          value: tasks2

  # Monitor Windows services.
  # This feature only works when Butler is running on Windows Server or desktop.
  # On other OSs service monitoring will be automatically disabled.
  serviceMonitor:
    enable: false                    # Main on/off switch for service monitoring
    frequency: every 30 seconds     # https://bunkat.github.io/later/parsers.html
    monitor:
      - host: <hostname or IP>      # Host name of Windows computer where services are running
        services:                   # List of services to monitor
          - name: postgresql-x64-12       # Posgress/repository db
            friendlyName: Repository DB
          - name: QlikSenseEngineService
            friendlyName: Engine
          - name: QlikSensePrintingService
            friendlyName: Printing
          - name: QlikSenseProxyService
            friendlyName: Proxy
          - name: QlikSenseRepositoryService
            friendlyName: Repository
          - name: QlikSenseSchedulerService
            friendlyName: Scheduler
          - name: QlikSenseServiceDispatcher
            friendlyName: Service Dispatcher
    alertDestination:               # Control to thich destinations service related alerts are sent
      influxDb:                     # Send service alerts to InfluxDB
        enable: true
      newRelic:                     # Send service alerts to New Relic
        enable: true
      email:                        # Send service alerts as emails
        enable: true                
      mqtt:                         # Send service alerts as MQTT messages
        enable: true
      teams:                        # Send service alerts as MS Teams messages
        enable: true
      slack:                        # Send service alerts as Slack messages
        enable: true
      webhook:                      # Send service alerts as outbound webhooks/http calls
        enable: true

  # Certificates to use when connecting to Sense. Get these from the Certificate Export in QMC.
  cert:
    clientCert: <path/to/cert/client.pem>
    clientCertKey: <path/to/cert/client_key.pem>
    clientCertCA: <path/to/cert/root.pem>
    # If running Butler in a Docker container, the cert paths MUST be the following
    # clientCert: /nodeapp/config/certificate/client.pem
    # clientCertKey: /nodeapp/config/certificate/client_key.pem
    # clientCertCA: /nodeapp/config/certificate/root.pem

  configEngine:
    engineVersion: 12.612.0         # Qlik Associative Engine version to use with Enigma.js. Works with Feb 2020 and others
    host: <FQDN or IP of Sense server where Sense Engine is running>
    port: <Port to connect to, usually 4747>
    useSSL: true
    headers:
      X-Qlik-User: UserDirectory=Internal;UserId=sa_repository
    rejectUnauthorized: false       # Set to false to ignore warnings/errors caused by Qlik Sense's self-signed certificates.
                                    # Set to true if the Qlik Sense root CA is available on the computer where Butler SOS is running.

  configQRS:
    authentication: certificates
    host: <FQDN or IP of Sense server where QRS is running>
    useSSL: true
    port: <Port to connect to, usually 4242>
    headerKey: X-Qlik-User                                      # Header used to identify what user connection to QRS is made as
    headerValue: UserDirectory=Internal; UserId=sa_repository   # What user connection to QRS is made as
    rejectUnauthorized: false       # Set to false to ignore warnings/errors caused by Qlik Sense's self-signed certificates.
                                    # Set to true if the Qlik Sense root CA is available on the computer where Butler SOS is running.

  configDirectories:
    qvdPath: <Path to folder under which QVDs are stored>

Comments

  • Note that you can enable/disable most features independently of each other. This makes it easy to have Butler do exactly what’s needed - no more - no less.

  • The default location cert/key files are found in (assuming a standard install of Qlik Sense) C:\ProgramData\Qlik\Sense\Repository\Exported Certificates\<name specified during certificate export>

    The files needed by Butler are client.pem, client_key.pem and root.pem.

3 - REST API documentation

OpenAPI/Swagger style documentation for the Butler API.

4 - REST API documentation (alternative format)

Alternative documentation for the Butler API.
Butler API documentation
API docs by Redocly

Butler API documentation (9.4.0)

Download OpenAPI specification:Download

Butler is a microservice that provides add-on features to Qlik Sense Enterprise on Windows. Butler offers both a REST API and things like failed reload notifications etc.

This page contains the API documentation. Full documentation is available at https://butler.ptarmiganlabs.com

Butler family of tools on GitHub

Get an array of all enabled API endpoints.

Get an array of all enabled API endpoints, using the key names from the Butler config file.

Note: Endpoints are enabled/disabled in the Butler main configuration file.

Responses

Response samples

Content type
application/json
[
  • "activeUserCount",
  • "activeUsers",
  • "apiListEnbledEndpoints"
]

Converts strings from base62 to base16.

Converts strings from base62 to base16.

query Parameters
base62
required
string
Example: base62=6DMW88LpSok9Z7P7hUK0wv7bF

The base62 encoded string that should be converted to base16

Responses

Response samples

Content type
application/json
{
  • "base62": "6DMW88LpSok9Z7P7hUK0wv7bF",
  • "base16": "3199af08bfeeaf5d420f27ed9c01e74370077"
}

Converts strings from base16 to base62.

Converts strings from base16 to base62.

query Parameters
base16
required
string
Example: base16=3199af08bfeeaf5d420f27ed9c01e74370077

The base16 encoded string that should be converted to base62

Responses

Response samples

Content type
application/json
{
  • "base16": "3199af08bfeeaf5d420f27ed9c01e74370077",
  • "base62": "6DMW88LpSok9Z7P7hUK0wv7bF"
}

Tests if Butler is alive and responding

Tests if Butler is alive and responding

Responses

Response samples

Content type
application/json
{
  • "response": "Butler reporting for duty",
  • "butlerVersion": "5.5.0"
}

Copy file(s) between well defined, approved locations.

Copying of files is only posttible between pre-approved directories. Defining approved source and destination directories is done in Butler's config file.

If the source directory contains subdirectories, these will be copied too.

Request Body schema: application/json
fromFile
string

Name of source file.

toFile
string

Name of destination file. Can be different from source file name, if needed.

overwrite
boolean

Controls whether destination file should be overwritten if it already exists. Note that the copy operation will silently fail if you set this to false and the destination exists. Defaults to false.

preserveTimestamp
boolean

When true, the timestamp of the source file(s) will be preserved on the destination file(s). When false, timestamp behaviour is OS-dependent. Defaults to false.

Responses

Request samples

Content type
application/json
{
  • "fromFile": "subfolder/file1.qvd",
  • "toFile": "archive/file1_20200925.qvd",
  • "overwrite": false,
  • "preserveTimestamp": false
}

Response samples

Content type
application/json
{
  • "fromFile": "subfolder/file1.qvd",
  • "toFile": "archive/file1_20200925.qvd",
  • "overwrite": false,
  • "preserveTimestamp": false
}

Move file(s) between well defined, approved locations.

Moving of files is only posttible between pre-approved directories. Defining approved source and destination directories is done in Butler's config file.

If the source directory contains subdirectories, these will be moved too.

Request Body schema: application/json
fromFile
string

Name of source file.

toFile
string

Name of destination file. Can be different from source file name, if needed.

overwrite
boolean

Controls whether destination file should be overwritten if it already exists. Defaults to false.

Responses

Request samples

Content type
application/json
{
  • "fromFile": "subfolder/file1.qvd",
  • "toFile": "archive/file1_20200925.qvd",
  • "overwrite": false
}

Response samples

Content type
application/json
{
  • "fromFile": "subfolder/file1.qvd",
  • "toFile": "archive/file1_20200925.qvd",
  • "overwrite": false
}

Delete file(s) in well defined, approved locations.

It is only possible to delete files in pre-approved directories, or subdirectories thereof. Defining approved directories is done in Butler's config file.

Request Body schema: application/json
deleteFile
string

Name of file to be deleted. Use forward/backward slashes in paths as needed, depending on whether Butler runs on Windows/non-Windows platform.

Responses

Request samples

Content type
application/json
{
  • "deleteFile": "data/qvdstore/sales/file1.qvd"
}

Response samples

Content type
application/json
{ }

Creates a directory in designated QVD directory.

Creates a directory in QVD directory (which is defined in Butler's config file).

Request Body schema: application/json
directory
string

Directory that should be created.

Responses

Request samples

Content type
application/json
{
  • "directory": "subfolder/2020-10"
}

Response samples

Content type
application/json
{
  • "directory": "subfolder/2020-10"
}

Creates a directory anywhere in the file system.

If the directory already exists nothing will happen. If permissions don't allow a directory to be created, or if the path is invalid, an error will be returned.

Request Body schema: application/json
directory
string

Path to directory that should be created. Can be a relative or absolute path.

Responses

Request samples

Content type
application/json
{
  • "directory": "/Users/joe/data/qvds/2020"
}

Response samples

Content type
application/json
{
  • "directory": "/Users/joe/data/qvds/2020"
}

List all currently defined namespaces.

Responses

Response samples

Content type
application/json
[
  • "Weekly sales app",
  • "Sales ETL step 1",
  • "Sales ETL step 2"
]

Get the value associated with a key, in a specific namespace.

path Parameters
namespace
required
string
Example: Sales ETL step 2
query Parameters
key
required
string
Example: key=Last extract timestamp

Responses

Response samples

Content type
application/json
{
  • "namespace": "Sales ETL step 2",
  • "key": "Last extract timestamp",
  • "value": "2020-09-29 17:14:56"
}

Create a new key-value pair in the specified namespace.

If the specified key already exists it will be overwritten.

If the posted data has a TTL, it will start counting when the post occur. I.e. if a previouly posted key also had a TTL, it will be replace with the most recent TTL.

path Parameters
namespace
required
string
Example: Sales ETL step 2

Name of namespace.

Request Body schema: application/json
key
string

Key to use

value
string

Value to set

ttl
number

Time to live = how long (milliseconds) the key-value pair should exist before being automatically deleted

Responses

Request samples

Content type
application/json
{
  • "key": "ce68c8ca-b3ff-4371-8285-7c9ce5040e42_parameter_1",
  • "value": "12345.789",
  • "ttl": 10000
}

Response samples

Content type
application/json
{
  • "namespace": "Sales ETL step 2",
  • "key": "Last extract timestamp",
  • "value": "2020-09-29 17:14:56",
  • "ttl": 60000
}

Delete a namespace and all key-value pairs in it.

path Parameters
namespace
required
string
Example: Sales ETL step 2

Name of namespace.

Responses

Response samples

Content type
application/json
""

Checks if a key exists in a namespace.

Returns true if the specified key exists, otherwise false.

path Parameters
namespace
required
string
Example: Sales ETL step 2
query Parameters
key
required
string
Example: key=Last extract timestamp

Responses

Response samples

Content type
application/json
{
  • "keyExists": true,
  • "keyValue": {
    }
}

Delete a key-value pair in a specific namespace.

path Parameters
namespace
required
string
Example: Sales ETL step 2

Name of namespace.

key
required
string
Example: ce68c8ca-b3ff-4371-8285-7c9ce5040e42_parameter_1

Key to use

Responses

Response samples

Content type
application/json
""

Retrieve a list of all keys present in the specified namespace.

path Parameters
namespace
required
string
Example: Sales ETL step 2

Name of namespace whose keys should be returned.

Responses

Response samples

Content type
application/json
{
  • "namespace": "Sales ETL step 2",
  • "keys": [
    ]
}

Publish a message to a MQTT topic.

Request Body schema: application/json
required
topic
required
string

Topic to which message should be published.

message
required
string

The message is a generic text string and can thus contain anything that can be represented in a string, including JSON, key-value pairs, plain text etc.

Responses

Request samples

Content type
application/json
{
  • "topic": "qliksense/new_data_notification/sales",
  • "message": "dt=20201028"
}

Response samples

Content type
application/json
{
  • "topic": "qliksense/new_data_notification/sales",
  • "message": "dt=20201028"
}

Post events to New Relic.

This endpoint posts events to the New Relic event API.

Request Body schema: application/json
required
eventType
required
string <= 254 characters

Event type. Can be a combination of alphanumeric characters, _ underscores, and : colons.

timestamp
number

The event's start time in Unix time. Uses UTC time zone. This field also support seconds, microseconds, and nanoseconds. However, the data will be converted to milliseconds for storage and query. Events reported with a timestamp older than 48 hours ago or newer than 24 hours from the time they are reported are dropped by New Relic. If left empty Butler will use the current time as timestamp.

Array of objects

Dimensions/attributs that will be associated with the event.

Responses

Request samples

Content type
application/json
{
  • "eventType": "relead-failed",
  • "timestamp": 1642164296053,
  • "attributes": [
    ]
}

Response samples

Content type
application/json
{
  • "newRelicResultCode": "202",
  • "newRelicResultText": "Data accepted."
}

Post metrics to New Relic.

This endpoint posts metrics to the New Relic metrics API.

Request Body schema: application/json
required
name
required
string <= 254 characters

Metric name.

type
required
string
Value: "gauge"

Metric type.

value
required
number

Value of the metric.

timestamp
number

The metric's start time in Unix time. Uses UTC time zone. This field also support seconds, microseconds, and nanoseconds. However, the data will be converted to milliseconds for storage and query. Metrics reported with a timestamp older than 48 hours ago or newer than 24 hours from the time they are reported are dropped by New Relic. If left empty Butler will use the current time as timestamp.

interval
number

The length of the time window (millisec). Required for count and summary metric types.

Array of objects

Dimensions that will be associated with the metric.

Responses

Request samples

Content type
application/json
{
  • "name": "memory.heap",
  • "type": "gauge",
  • "value": 2.3,
  • "timestamp": 1642164296053,
  • "interval": 0,
  • "attributes": [
    ]
}

Response samples

Content type
application/json
{
  • "newRelicResultCode": "202",
  • "newRelicResultText": "Data accepted."
}

Get all information available for existing schedule(s).

If a schedule ID is specified using a query parameter (and there exists a schedule with that ID), information about that schedule will be returned. If no schedule ID is specified, all schedules will be returned.

query Parameters
id
string
Example: id=e4b1c455-aa15-4a51-a9cf-c5e4cfc91339

Scheduld ID

Responses

Response samples

Content type
application/json
[
  • {
    }
]

Create a new schedule.

Request Body schema: application/json
required
name
required
string

Descriptive name for the schedule.

cronSchedule
required
string

5 or 6 position cron schedule.

If 6 positions used, the leftmost position represent seconds. If 5 positions used, leftmost position is minutes.

The example schedule will trigger at 00 and 30 minutes past 6:00 on Mon-Fri.

timezone
required
string

Time zone the schedule should use. Ex "Europe/Stockholm".

qlikSenseTaskId
required
string

ID of Qlik Sense task that should be started when schedule triggers.

startupState
required
string
Enum: "start" "started" "stop" "stopped"

If set to "start" or "started", the schedule will be started upon creation. Otherwise it will remain in stopped state.

tags
Array of strings

Can be used to categorise schedules.

Responses

Request samples

Content type
application/json
{
  • "name": "Reload sales metrics",
  • "cronSchedule": "0,30 6 * * 1-5",
  • "timezone": "Europe/Stockholm",
  • "qlikSenseTaskId": "210832b5-6174-4572-bd19-3e61eda675ef",
  • "startupState": "started",
  • "tags": [
    ]
}

Response samples

Content type
application/json
[
  • {
    }
]

Delete a schedule.

path Parameters
scheduleId
required
string
Example: e4b1c455-aa15-4a51-a9cf-c5e4cfc91339

Schedule ID.

Responses

Response samples

Content type
application/json
""

Start a schedule.

Start a schedule, i.e. have the scheduler run the associated reload task according to the schedule's cron settings.

path Parameters
scheduleId
required
string
Example: e4b1c455-aa15-4a51-a9cf-c5e4cfc91339

Schedule ID.

Responses

Response samples

Content type
application/json
{
  • "id": "e4b1c455-aa15-4a51-a9cf-c5e4cfc91339",
  • "created": "2020-09-29T14:29:12.283Z",
  • "name": "Reload sales metrics",
  • "cronSchedule": "0,30 6 * * 1-5",
  • "timezone": "Europe/Stockholm",
  • "qlikSenseTaskId": "210832b5-6174-4572-bd19-3e61eda675ef",
  • "startupState": "started",
  • "tags": [
    ],
  • "lastKnownState": "started"
}

Start all schedules.

Start all schedules, i.e. tell the scheduler to run each schedule and start associated tasks according to each schedule's settings.

Responses

Response samples

Content type
application/json
[
  • {
    }
]

Stop a schedule.

Stop a schedule, i.e. tell the scheduler to no longer execute the schedule according to its cron settings.

path Parameters
scheduleId
required
string
Example: e4b1c455-aa15-4a51-a9cf-c5e4cfc91339

Schedule ID.

Responses

Response samples

Content type
application/json
{
  • "id": "e4b1c455-aa15-4a51-a9cf-c5e4cfc91339",
  • "created": "2020-09-29T14:29:12.283Z",
  • "name": "Reload sales metrics",
  • "cronSchedule": "0,30 6 * * 1-5",
  • "timezone": "Europe/Stockholm",
  • "qlikSenseTaskId": "210832b5-6174-4572-bd19-3e61eda675ef",
  • "startupState": "started",
  • "tags": [
    ],
  • "lastKnownState": "started"
}

Stop all schedules.

Stop all schedules, i.e. tell the scheduler to no longer execute any schedule according to its cron settings.

Responses

Response samples

Content type
application/json
[
  • {
    }
]

Get scheduler status.

Get basic status from the core scheduler.

No schedule metadata beyond ID, cron setting and job state will be returned, but as this comes from the core scheduler it is the authorative truth about what jobs are running (and which ones are not).

Responses

Response samples

Content type
text/plain
{
'3702cec1-b6c8-463e-bda3-58d6a94dd9ac': * */2 * * * status: Running 
'2d5dcebc-2440-4bd7-9aa1-fb69708715c8': */45 * * * * * status: Running 
'a93ca0f3-7980-439b-9eda-723a167352e3': */10 * * * * * status: Running 
'ad250f49-ffd8-45dc-9b2b-f76028e969a4': */5 * * * * * status: Running 
}

Do a stand-alone reload of a Qlik Sense app, without using a task.

path Parameters
appId
required
string
Example: 210832b5-6174-4572-bd19-3e61eda675ef

ID of Qlik Sense app.

Request Body schema: application/json
reloadMode
integer

Reload mode that will be used. 0, 1 or 2. If not specified 0 will be used

partialReload
boolean

Should a full (=false) or partial (=true) reload be done? If not specified a full reload will be done.

startQSEoWTaskOnSuccess
Array of strings

Array of task IDs that should be started when the app has successfully reloaded.

startQSEoWTaskOnFailure
Array of strings

Array of task IDs that should be started if the app fails reloading.

Responses

Request samples

Content type
application/json
{
  • "reloadMode": 0,
  • "partialReload": true,
  • "startQSEoWTaskOnSuccess": [
    ],
  • "startQSEoWTaskOnFailure": [
    ]
}

Response samples

Content type
application/json
{
  • "appId": "210832b5-6174-4572-bd19-3e61eda675ef"
}

Dump a Sense app to JSON.

Does the same thing as /v4/app/:appId/dump

path Parameters
appId
required
string
Example: 210832b5-6174-4572-bd19-3e61eda675ef

ID of Qlik Sense app.

Responses

Response samples

Content type
application/json
{
  • "properties": { },
  • "loadScript": "",
  • "sheets": [ ],
  • "stories": [ ],
  • "masterobjects": [ ],
  • "appprops": [ ],
  • "dataconnections": [ ],
  • "dimensions": [ ],
  • "bookmarks": [ ],
  • "embeddedmedia": [ ],
  • "snapshots": [ ],
  • "fields": [ ],
  • "variables": [ ],
  • "measures": [ ]
}

Dump a Sense app to JSON.

Does the same thing as /v4/senseappdump/:appId

path Parameters
appId
required
string
Example: 210832b5-6174-4572-bd19-3e61eda675ef

ID of Qlik Sense app.

Responses

Response samples

Content type
application/json
{
  • "properties": { },
  • "loadScript": "",
  • "sheets": [ ],
  • "stories": [ ],
  • "masterobjects": [ ],
  • "appprops": [ ],
  • "dataconnections": [ ],
  • "dimensions": [ ],
  • "bookmarks": [ ],
  • "embeddedmedia": [ ],
  • "snapshots": [ ],
  • "fields": [ ],
  • "variables": [ ],
  • "measures": [ ]
}

Get a list of all apps in Sense environment.

Does the same thing as /v4/apps/list

Responses

Response samples

Content type
application/json
[
  • {
    }
]

Get a list of all apps in Sense environment.

Does the same thing as /v4/senselistapps

Responses

Response samples

Content type
application/json
[
  • {
    }
]

Start a Qlik Sense task.

An optional array of zero or more objects can be passed in the message body. It is used to pass additional info related to the reload task being started.

Currently supported values in this array are:

  • A key-value pair that will be stored in Butler's KV store. If Butler's key-value store is not enabled, any key-value information passed in this parameter will simply be ignored. Setting ttl=0 disables the TTL feature, i.e. the KV pair will not expire.
  • A task identified by its taskId that should be started.
  • Tasks identified by tags set on tasks in the QMC.
  • Tasks identified by custom properties set in the QMC.

This parameter uses a generic JSON/object format (type + payload). It's thus possible to add new integrations in future Butler versions.

path Parameters
taskId
required
string
Example: 210832b5-6174-4572-bd19-3e61eda675ef

ID of Qlik Sense task. Butler will ignore the "magic" task ID of "-" (=dash, hyphen). This ID will not be reported as invalid.

query Parameters
allTaskIdsMustExist
boolean
Example: allTaskIdsMustExist=true

If set to true, all specified taskIds must exist. If one or more taskIds does not exist in the Sense server, no tasks will be started.

If set to false, all existing taskIds will be started. Missing/invalid taskIds will be ignored.

In either case, missing/invalid taskIds will be reported in the result set back to the client calling the API.

Note: Tasks started by specifying tags and/or custom properties are not affected by this.

Request Body schema: application/json

Optional object with extra info.

Array
type
string
Enum: "keyvaluestore" "starttaskid" "starttasktag" "starttaskcustomproperty"
payload
object

Responses

Request samples

Content type
application/json
[
  • {
    },
  • {
    },
  • {
    },
  • {
    }
]

Response samples

Content type
application/json
{
  • "tasksId": {
    },
  • "tasksTag": [
    ],
  • "tasksTagDenied": [
    ],
  • "tasksCP": [
    ],
  • "tasksCPDenied": [
    ]
}

Start a Qlik Sense task.

An optional array of zero or more objects can be passed in the message body. It is used to pass additional info related to the reload task being started.

Currently supported values in this array are:

  • A key-value pair that will be stored in Butler's KV store. If Butler's key-value store is not enabled, any key-value information passed in this parameter will simply be ignored. Setting ttl=0 disables the TTL feature, i.e. the KV pair will not expire.
  • A task identified by its taskId that should be started.
  • Tasks identified by tags set on tasks in the QMC.
  • Tasks identified by custom properties set in the QMC.

This parameter uses a generic JSON/object format (type + payload). It's thus possible to add new integrations in future Butler versions.

path Parameters
taskId
required
string
Example: 210832b5-6174-4572-bd19-3e61eda675ef

ID of Qlik Sense task. Butler will ignore the "magic" task ID of "-" (=dash, hyphen). This ID will not be reported as invalid.

query Parameters
allTaskIdsMustExist
boolean
Example: allTaskIdsMustExist=true

If set to true, all specified taskIds must exist. If one or more taskIds does not exist in the Sense server, no tasks will be started.

If set to false, all existing taskIds will be started. Missing/invalid taskIds will be ignored.

In either case, missing/invalid taskIds will be reported in the result set back to the client calling the API.

Note: Tasks started by specifying tags and/or custom properties are not affected by this.

Request Body schema: application/json

Optional object with extra info.

Array
type
string
Enum: "keyvaluestore" "starttaskid" "starttasktag" "starttaskcustomproperty"
payload
object

Responses

Request samples

Content type
application/json
[
  • {
    },
  • {
    },
  • {
    },
  • {
    }
]

Response samples

Content type
application/json
{
  • "tasksId": {
    },
  • "tasksTag": [
    ],
  • "tasksTagDenied": [
    ],
  • "tasksCP": [
    ],
  • "tasksCPDenied": [
    ]
}

Send message to Slack.

Sends a basic message to Slack.

Request Body schema: application/json
required
channel
required
string

Slack channel to post message into. Prefix channel name with #.

from_user
required
string

Name of sending user, as shown in the Slack message

msg
required
string

Text going into the Slack message

emoji
string

Emoji to shown next to Slack message

Responses

Request samples

Content type
application/json
{
  • "channel": "#reload-notification",
  • "from_user": "Butler the Bot",
  • "msg": "This is a message from Qlik Sense",
  • "emoji": "thumbsup"
}

Response samples

Content type
application/json
{
  • "channel": "#reload-notification",
  • "from_user": "Butler the Bot",
  • "msg": "This is a message from Qlik Sense",
  • "emoji": "thumbsup"
}

5 - Scheduler

Details about the Butler scheduler.

Scheduling overview

The scheduler is used to start Sense reloads at specific times, or at certain intervals.

The rationale for having a scheduler in Butler is mainly that Qlik Sense’s built-in scheduler doesn’t quite have the desired flexibiluty. The Sense scheduler on the other hand offer some features not included in the Butler scheduler, so they complelent each other nicely.

Put differently: Butler’s scheduler is great for kicking off the first step if reload chains (or single app reloads), while the QMC scheduler is needed to link the individual parts of reload chains together.

Feature Qlik Sense scheduler Butler scheduler
Tasks should run same time each day. Yes Yes
Tasks should run only some days of the month/wek (ex 1st day of the month or on Wednesdays and Fridays). Yes Yes
Tasks should run only between certain hours during the day (e.g. every 5 minutes during the first 30 minutes of each hour, between 08.00 and 18.00). Yes, with lots of manual work Yes
Tasks can be chained together, with one task starting when the previous has finished. Yes -
Task definitions can be stored in Git and managed by a DevOps workflow. Yes, with lots of work. Yes, out of the box

Schedules

The Butler scheduler handles zero or more schedules, where a schedule is simply a set of information about which Sense reload task should be started when.

While Butler is running it keeps the schedules in memory, but they are also stored on disk in the file specified in the Butler’s main config file. This means you can either manage the schedule file manually, or via the Butler APIs.

The following information is kept about each schdule. Note that some fields are optional and only set when managing schedules via API calls.

Field name Descriptions Example
name Descriptive name for the schedule. Every 2 hours
cronSchedule Cron string. 5 or 6 postitions allowed. If 5 positions used a 0 will be used for seconds. crontab.guru and similar sites are useful for creating the cron string. 0 */2 * * *
timezone Timezone in which the schedule will execute. List of available timezones here. Europe/London
qlikSenseTaskId Qlik Sense task that will be started. The task ID can be found in the task view in the QMC. 0fe447a9-ba1f-44a9-ac23-68c3a1d88d8b
startupState Controls whether this task should be started or left stopped when Butler starts up.
If set to “start” or “started”, the schedule will be started upon creation. Otherwise it will remain in stopped state.		 started
tags Array of tags, can be used to categorise schedules. Optional. [ finace ETL, abc 123 ]
id Schedule ID. Free text string, can be set to anything as long as it is unique. Schedules created via the REST API will have a GUID as id. 2f7a6c38-46df-416d-83de-44bdea5cedaa
created Timestamp the schedule was created. Optional when manually adding schedules to schedule file. 2020-09-29T17:14:35.154Z
lastKnownState The schedule’s last known state. Optional when manually adding schedules to schedule file. started 

Sample schedule file

A schedule file could look like this:

    butlerSchedule:
    - name: Every 30 sec
        cronSchedule: "*/30 * * * * *"
        timezone: Europe/Stockholm
        qlikSenseTaskId: 0fe447a9-ba1f-44a9-ac23-68c3a1d88d8b
        startupState: started
        tags:
        - tag 2
        - abc 123 åäö
        id: c7ec214c-e9ca-40b2-acb4-54648f90dd73
        created: '2020-09-29T14:29:12.283Z'
        lastKnownState: started
    - name: Every 2 hours
        cronSchedule: "0 */2 * * *"
        timezone: Europe/London
        qlikSenseTaskId: 0fe447a9-ba1f-44a9-ac23-68c3a1d88d8b
        startupState: started
        tags:
        - sales ETL
        id: 2f7a6c38-46df-416d-83de-44bdea5cedaa
        created: '2020-09-29T17:14:35.154Z'
        lastKnownState: started
    - name: 00 and 30 minutes past every hour from 06 to 18 on Mon-Fri
        cronSchedule: "0,30 6-18 * * 1-5"
        timezone: Europe/Paris
        qlikSenseTaskId: 0fe447a9-ba1f-44a9-ac23-68c3a1d88d8b
        startupState: started
        tags:
        - finance ETL
        - weekdays
        id: Manually-added-schedule-1

A few things to note:

  • Schedule 1
    • Uses 6 positions in the cron string. The leftmost position is seconds.
    • This schedule cannot be created in the QMC scheduler, as minutes is the smallest increment there. On the other hand, it’s questionable if Sense jobs should run more often than once per minute…
  • Schedule 2
    • Uses 5 positions in the cron string. The leftmost position is minutes. The seconds position is implicitly set to 0, which means the schedule will fire at the top of each minute.
    • This schedule can be created in the QMC scheduler.
  • Schedule 3
    • This schedule cannot be created in the QMC scheduler.

Manual vs API created schedules

Schedules are stored in a YAML file on disk. This file is loaded each time Butler is started.

When a new schedule is created using the APIs, all schedule definitions and their current state is written to the schedule YAML file. Thus, if Butler is stopped or restarted, it will also restart all schedules in the same state they were in before the Butler stop/restart.

There are a couple of differences between manually created schedules (i.e. ones that are added directly to the YAML file) and schedules created via the APIs:

  • The created field in a schedule is optional. It is set when a schedule is created using the APIs. You can set it when manually editing the schedule file, if so desired - but it’s not required.
  • The lastKnownState field is overwritten each time the schedules are saved to disk. This means that you can change this field manually for a schedule in the schedule file on disk, but if you then change the schedule’s state (start or stop it), that will trigger a saving of the schedules to disk, and your previous (manual) change will be overwritten.

6 - Key-value store

Details about the Butler key-value store.

Key-value store overview

The API documentation is the best, most complete source of information about the key-value API endpoints.

Namespaces

Each key-value pair is associated with a “namespace”, which is simply a way to group KV pairs together in logical groups.

Examples:

  • An app might have it’s own namespace to make it clear what key-value pairs belong to the app.
  • A reload chain that need to pass parameters from app to app can have it’s own namespace.

The Butler.keyValueStore.maxKeysPerNamespace property in the main config file controls how many KV pairs can be created in each namespace. If no value is specified in the config file, a default value of 1000 KV pairs per namespace is used.

Time to live

When key-value pairs are created an optional time-to-live (ttl) parameter may also be passed in the call to Butler’s API.

If the ttl parameter is specified (=set to a non-zero value, unit is milliseconds) the created KV pair will only exist for that long.

If a KV pair with a running ttl clock is updated before the KV pair expires, the current ttl clock will be discarded and replaced with the ttl value from the new KV pair.

In other words: The ttl is relative to the last update of the KV pair in question.

7 - Alert template fields

List of template fields available in Butler alert messages.

Template fields

Butler uses the Handlebars library for templating work.

Handlebars offers a lot of useful features (nested template fields, evaluation context, template comments) and it’s recommended that you browse through at least the language features section of their getting started guide to get a feeling for what’s possible.

Different Butler monitoring features offer different template fields.
For example, the template fields available for reload alerts are different from the ones available for monitoring Windows services.

7.1 - Alert template fields for failed/aborted reload tasks

List of template fields available in Butler’s reload failed/aborted alert messages.

Template fields

Butler uses the Handlebars library for templating work.

Handlebars offers a lot of useful features (nested template fields, evaluation context, template comments) and it’s recommended that you browse through at least the language features section of their getting started guide to get a feeling for what’s possible.

If a template field is used for an alert type where that field is not supported, the field will simply be blank. No errors will occur.

The following template fields are available in alert messages.

Note that some fields are usually (always?) empty, for example the script log for stopped messages.
This is simply how Sense works - the template fields just forward the information retrieved from the Sense APIs.

Failed
reload		 Stopped
reload		 Field name Description
✅ ✅ hostName Server on which a reload or other event took place.
✅ ✅ user Reload failures: User ID for use doing the reload. Typically sa_scheduler.
Reload stopped: User ID of user stopping the reload.
✅ ✅ taskId ID of reload task that failed/was stopped.
✅ ✅ appName Name of Sense whose reload failed/was stopped.
✅ ✅ appId ID of Sense app whose reload failed/was stopped.
✅ ✅ appOwnerName Name of app owner (if this is available in the metadata provided by the Sense server)
✅ ✅ appOwnerUserDirectory App owner user’s user directory (if this is available in the metadata provided by the Sense server)
✅ ✅ appOwnerUserId App owner user’s user id (if this is available in the metadata provided by the Sense server)
✅ ✅ appOwnerEmail App owner email (if this is available in the metadata provided by the Sense server)
✅ ✅ logTimeStamp Timestamp as recorded in the Sense logs
✅ ✅ logLevel Log level of the Sense log file entry causing the alert
✅ ✅ logMessage Log message from the Sense log files
✅ ✅ executingNodeName Name of the server where the reload took place
✅ ✅ executionDuration.hours executionDuration is a JSON object.
Duration of reload (hours part)
✅ ✅ executionDuration.minutes Duration of reload (minutes part)
✅ ✅ executionDuration.seconds Duration of reload (seconds part)
✅ ✅ executionStartTime.startTimeUTC JSON object.
UTC timestamp for reload start
✅ ✅ executionStartTime.startTimeLocal1 Reload start timestamp, format 1
✅ ✅ executionStartTime.startTimeLocal2 Reload start timestamp, format 2
✅ ✅ executionStartTime.startTimeLocal3 Reload start timestamp, format 3
✅ ✅ executionStartTime.startTimeLocal4 Reload start timestamp, format 4
✅ ✅ executionStartTime.startTimeLocal5 Reload start timestamp, format 5
✅ ✅ executionStopTime.stopTimeUTC JSON object.
UTC timestamp for reload end
✅ ✅ executionStopTime.stopTimeLocal1 Reload end timestamp, format 1
✅ ✅ executionStopTime.stopTimeLocal2 Reload end timestamp, format 2
✅ ✅ executionStopTime.stopTimeLocal3 Reload end timestamp, format 3
✅ ✅ executionStopTime.stopTimeLocal4 Reload end timestamp, format 4
✅ ✅ executionStopTime.stopTimeLocal5 Reload end timestamp, format 5
✅ ✅ executionStatusNum Final reload task status code
✅ ✅ executionStatusText Final reload task status message
✅ ✅ executionDetails[].timestampUTC executionDetails is an array of status updates for the reload task, similar to the one found in the QMC’s task view.
UTC timestamp of the task status
✅ ✅ executionDetails[].timestampLocal1 Task status timestamp, format 1
✅ ✅ executionDetails[].timestampLocal2 Task status timestamp, format 2
✅ ✅ executionDetails[].timestampLocal3 Task status timestamp, format 3
✅ ✅ executionDetails[].timestampLocal4 Task status timestamp, format 4
✅ ✅ executionDetails[].timestampLocal5 Task status timestamp, format 5
✅ ✅ executionDetails[].detailsType Task status timestamp, format 1
✅ ✅ executionDetails[].message Task status message
✅ ✅ scriptLogHeadCount Number of lines extracted from the start of the script log
✅ ✅ scriptLogHead The first x lines from the reload’s script log
✅ ✅ scriptLogTail Number of lines extracted from the end of the script log
✅ ✅ scriptLogTailCount The first y lines from the reload’s script log
✅ ✅ qliksSenseQMC Links to QMC, as defined in main config file
✅ ✅ qliksSenseHub Links to Hub, as defined in main config file

7.2 - Alert template fields for Windows services events

List of template fields available in Butler’s Windows services alert messages.

Email, Slack and Teams

Butler uses the Handlebars library for templating work.
This gives a lot of flexibility in how alert messages are formatted for the destination types that support template fields.

Handlebars offers a lot of useful features (nested template fields, evaluation context, template comments) and it’s recommended that you browse through at least the language features section of their getting started guide to get a feeling for what’s possible.

If a template field is used for an alert type where that field is not supported, the field will simply be blank. No errors will occur.

The following alert destinations support template fields:

The following template fields are available in alert messages.

Note that some fields are usually (always?) empty, for example the script log for stopped messages.
This is simply how Sense works - the template fields just forward the information retrieved from the Sense APIs.

Email Slack Teams Field name Description
✅ ✅ ✅ {{host}} The hostname of the server where the service is running
✅ ✅ ✅ {{serviceStatus}} The status of the service, e.g. RUNNING or STOPPED
✅ ✅ ✅ {{servicePrevStatus}} The previous status of the service, e.g. RUNNING or STOPPED
✅ ✅ ✅ {{serviceName}} The name of the service as defined in Windows
✅ ✅ ✅ {{serviceDisplayName}} The display name of the service as defined in Windows. Can sometimes be a bit more human readable than the serviceName.
✅ ✅ ✅ {{serviceFriendlyName}} The display name of the service as defined in the Butler config file. Used to give the service a good name when both serviceName and serviceDisplayName are unsuitable for use in for example Grafana dashboards.
✅ ✅ ✅ {{serviceStartType}} The startup mode of the service, e.g. Automatic or Manual
✅ ✅ ✅ {{serviceExePath}} The path to the executable of the service

InfluxDB

Window service data will be stored in the InfluxDB database specified in the config file Butler.influxdb.dbName setting.

The latest status of each service will be stored in the butler_windows_services measurement every time Butler checks the status of the service, i.e. not only when the service changes status.

Fields/metrics

The following metrics will be stored in a measurement named win_service_state, for each Windows service:

Field name Description
state_num A numeric representation of the Windows service’s current state.
state_text The Windows service’s current state (textual representation).
startup_mode_num A numeric representation of the Windows service’s startup mode.
startup_mode_text The Windows service’s startup mode (textual representation).

Mapping of state_num to state_text:

state_num state_text
1 STOPPED
2 START_PENDING
3 STOP_PENDING
4 RUNNING
5 CONTINUE_PENDING
6 PAUSE_PENDING
7 PAUSED

Mapping of startup_mode_num to startup_mode_text:

startup_mode_num startup_mode_text
0 Automatic
1 Automatic (delayed start)
2 Manual
3 Disabled

Tags

The following tags will be attached to all Windows service data stored in InfluxDB:

Tag name Description
butler_instance The value in Butler.influxDb.instanceTag. Can be used to differentiate several Butler instances running in parallel.
host Host name where the Windows service is running. Comes from Butler.serviceMonitor.monitor.<host>.
service_name The name of the Windows service, as defined in Windows.
service_display_name The display name of the Windows service, as defined in Windows.
service_friendly_name The friendly name of the Windows service, as defined in the Butler config file Butler.serviceMonitor.monitor.<host>.services.<friendlyName>.

New Relic

Butler will create New Relic events and/or log entries when a monitored Windows service changes state, for example from RUNNING to STOPPED.

Events

Events sent to New Relic will have these attributes attached:

Attribute name Description
eventType Always set to qs_serviceStateEvent
butler_serviceHost The host name where the Windows service is running. Comes from Butler.serviceMonitor.monitor.<host>.
butler_serviceName The name of the Windows service, as defined in Windows.
butler_serviceDisplayName The display name of the Windows service, as defined in Windows.
butler_serviceStatus The status of the Windows service, e.g. RUNNING or STOPPED.
… Any static and dynamic attributes defined in Butler.incidentTool.newRelic.serviceMonitor.destination.event.attribute.
… Any static attributes defined in Butler.incidentTool.newRelic.serviceMonitor.sharedSettings.attribute.static.

Log entries

Log entries sent to New Relic will have these attributes attached:

Attribute name Description
logType Always set to qs_serviceStateLog
butler_serviceHost The host name where the Windows service is running. Comes from Butler.serviceMonitor.monitor.<host>.
butler_serviceName The name of the Windows service, as defined in Windows.
butler_serviceDisplayName The display name of the Windows service, as defined in Windows.
butler_serviceStatus The status of the Windows service, e.g. RUNNING or STOPPED.
… Any static and dynamic attributes defined in Butler.incidentTool.newRelic.serviceMonitor.destination.log.attribute.
… Any static attributes defined in Butler.incidentTool.newRelic.serviceMonitor.sharedSettings.attribute.static.

The log message will be one of the following:

  • “Windows service on host is RUNNING.”
  • “Windows service on host is STOPPED.”
  • “Windows service on host is in state START_PENDING.”
  • “Windows service on host is in state STOP_PENDING.”
  • “Windows service on host is in state CONTINUE_PENDING.”
  • “Windows service on host is in state PAUSE_PENDING.”
  • “Windows service on host is in state PAUSED.”

MQTT

Similar to how InfluxDB works, Butler will send an MQTT message to the topic specified in Butler.mqttConfig.serviceStatusTopic every time it checks the status of a Windows service, i.e. not only when the service changes status.

Butler will ALSO send an MQTT message to the topics specified in Butler.mqttConfig.serviceRunningTopic and Butler.mqttConfig.serviceStoppedTopic when a Windows service is stopped or started.

Message payload for continuously sent messages is a JSON object with these properties:

Property name Description
host The host name where the Windows service is running. Comes from Butler.serviceMonitor.monitor.<host>.
serviceName The name of the Windows service, as defined in Windows.
serviceFriendlyName The friendly name of the Windows service, as defined in the Butler config file Butler.serviceMonitor.monitor.<host>.services.<friendlyName>.
serviceStatus The status of the Windows service, e.g. RUNNING or STOPPED.
serviceDetails The details of the Windows service, e.g. RUNNING or STOPPED.

Message payload for start/stop messages is a JSON object with these properties:

Property name Description
host The host name where the Windows service is running. Comes from Butler.serviceMonitor.monitor.<host>.
serviceName The name of the Windows service, as defined in Windows.
serviceFriendlyName The friendly name of the Windows service, as defined in the Butler config file Butler.serviceMonitor.monitor.<host>.services.<friendlyName>.
serviceStatus The status of the Windows service, e.g. RUNNING or STOPPED.
serviceDetails The details of the Windows service, e.g. RUNNING or STOPPED.
prevState The previous state of the Windows service, e.g. RUNNING or STOPPED.
currState The current state of the Windows service, e.g. RUNNING or STOPPED.
stateChanged true if the Windows service changed state, false if not.

Outbound webhooks

Outbound webhooks (=http calls) are sent when a Windows service changes state, for example from RUNNING to STOPPED.

The payload for PUT and POST http calls is sent in the message body as a JSON object with these properties:

Property name Description
event Always set to Windows service monitor
host The host name where the Windows service is running. Comes from Butler.serviceMonitor.monitor.<host>.
serviceStatus The status of the Windows service, e.g. RUNNING or STOPPED.
serviceName The name of the Windows service, as defined in Windows.
serviceDisplayName The display name of the Windows service, as defined in Windows.
serviceStartType The startup mode of the service, e.g. Automatic or Manual
prevState The previous state of the Windows service, e.g. RUNNING or STOPPED.
currState The current state of the Windows service, e.g. RUNNING or STOPPED.
stateChanged true if the Windows service changed state, false if not.

The payload for GET http calls is sent as query parameters with the same properties as above, but with the property names being all lowercase letters.

A typical GET webhook URL would look like this:

https://mywebhookserver.com?event=Windows service monitor&host=MyHost&servicestatus=RUNNING&servicename=MyService&servicedisplayname=MyServiceDisplayName&servicestarttype=Automatic&prevstate=RUNNING&currstate=STOPPED&statechanged=true

8 - Information stored in InfluxDB

Butler stores a lot of information in InfluxDB. This page describes the different measurements and tags that Butler send to InfluxDB.

Failed reload tasks

Measurement: reload_task_failed

Tags

Tag name Description
host Server on which the reload took place.
user Full user info (directory + ID) for user doing the reload. Typically sa_scheduler for tasks started via scheduler. For manually started (from QMC) tasks the actual user starting the task is used.
task_id ID of reload task that failed.
task_name Name of reload task that failed.
app_id ID of Sense app whose reload failed.
app_name Name of Sense app whose reload failed.
log_level Log level of the Sense log file entry causing the alert
task_executingNodeName Name of node where the reload task was executed.
task_executionStatusNum Reload task’s execution result code (numeric).
task_executionStatusText Reload task’s execution result (text).
appTag_<app tag name 1 from Qlik Sense> App tag defined in the QMC for the Qlik Sense app.
appTag_<app tag name 2 from Qlik Sense> App tag defined in the QMC for the Qlik Sense app.
taskTag_<reload task tag name 1 from Qlik Sense> Task tag defined in the QMC for the Qlik Sense reload task.
taskTag_<reload task tag name 2 from Qlik Sense> Task tag defined in the QMC for the Qlik Sense reload task.
static-tag-1 Static tag specified in the Butler configuration file.
static-tag-2 Static tag specified in the Butler configuration file.

Fields

Field name Description
log_timestamp Timestamp of the Sense log file entry that triggered the event.
execution_id  Execution ID of the reload task.
log_message Raw message from the Sense log file entry that triggered the alert.
task_executionStartTime_json Start time of the reload task. Stringified JSON with seconds, minutes, hours.
task_executionStopTime_json Stop time of the reload task. Stringified JSON with seconds, minutes, hours.
task_executionDuration_json Duration of the reload task. Stringified JSON with seconds, minutes, hours.
task_executionDuration_sec Duration of the reload task in seconds.
task_executionDuration_min Duration of the reload task in minutes.
task_executionDuration_h Duration of the reload task in hours.
task_scriptLogSize Size (number of characters) of the reload script log associated with the event.
task_scriptLogTailCount Number of lines from the end of the reload script log that are included in the event.
reload_log Last few lines of the reload script log associated with the event.

Successful reload tasks

Measurement: reload_task_success

Tags

Tag name Description
host Server on which the reload took place.
user Full user info (directory + ID) for user doing the reload. Typically sa_scheduler for tasks started via scheduler. For manually started (from QMC) tasks the actual user starting the task is used.
task_id ID of reload task.
task_name Name of reload task.
app_id ID of Sense app that was reloaded.
app_name Name of Sense app that was reloaded.
log_level Log level of the Sense log file entry causing the event
task_executingNodeName Name of node where the reload task was executed.
task_executionStatusNum Reload task’s execution result code (numeric).
task_executionStatusText Reload task’s execution result (text).
appTag_<app tag name 1 from Qlik Sense> App tag defined in the QMC for the Qlik Sense app.
appTag_<app tag name 2 from Qlik Sense> App tag defined in the QMC for the Qlik Sense app.
taskTag_<reload task tag name 1 from Qlik Sense> Task tag defined in the QMC for the Qlik Sense reload task.
taskTag_<reload task tag name 2 from Qlik Sense> Task tag defined in the QMC for the Qlik Sense reload task.
static-tag-1 Static tag specified in the Butler configuration file.
static-tag-2 Static tag specified in the Butler configuration file.

Fields

Field name Description
log_timestamp Timestamp of the Sense log file entry that triggered the event.
execution_id  Execution ID of the reload task.
log_message Raw message from the Sense log file entry that triggered the event.
task_executionStartTime_json Start time of the reload task. Stringified JSON with seconds, minutes, hours.
task_executionStopTime_json Stop time of the reload task. Stringified JSON with seconds, minutes, hours.
task_executionDuration_json Duration of the reload task. Stringified JSON with seconds, minutes, hours.
task_executionDuration_sec Duration of the reload task in seconds.
task_executionDuration_min Duration of the reload task in minutes.
task_executionDuration_h Duration of the reload task in hours.

Windows service info

Measurement: win_service_state

Tags

Tag name Description
butler_instance Name of the Butler instance, from config file.
host Server on which the reload took place.
service_name Name of the Windows service.
display_name Display name of the Windows service.
friendly_name Friendly name of the Windows service (as defined in the Butler config file).

Fields

Field name Description
state_num State of the Windows service (numeric).
state_text State of the Windows service (text).
startup_mode_num Startup mode of the Windows service (numeric).
startup_mode_text Startup mode of the Windows service (text).

Qlik Sense license info

The license information returned by the Qlik Sense API is not very well documented by Qlik.
As a result it is not always clear what the different fields mean.
This is highlighted for each affected field below.

Measurement: sense_license_info

Each datapoint has a tag called license_type that can have the following values:

  • professional
  • analyzer
  • analyzer_capacity
  • token_login
  • token_user
  • tokens_available

The tags are the same for all license types, but the fields differ as follows.

License types “professional” and “analyzer”

Field name Description
allocated Number of allocated professional licenses.
available Number of available professional licenses.
excess Number of excess professional licenses. Unclear what this means.
quarantined Number of quarantined professional licenses.
total Total number of professional licenses included in the installed Qlik Sense license.
used Number of professional licenses in use right now.

License type “analyzer_capacity”

Field name Description
allocated_minutes Total number of analyzer capacity minutes available in the installed Qlik Sense license.
unavailable_minutes Number of analyzer capacity minutes that are currently unavailable. Unclear what this means.
used_minutes Number of analyzer capacity minutes used so far this month.

License type “token_login”

Field name Description
allocated_tokens Unclear what this means.
token_cost Unclear what this means.
unavailable_tokens Unclear what this means.
used_tokens Unclear what this means.

License type “token_user”

Field name Description
allocated_tokens Unclear what this means.
quarantined_tokens Unclear what this means.
token_cost Unclear what this means.
used_tokens Unclear what this means.

License type “tokens_available”

This license type contains aggregated token information.

Field name Description
available_tokens Number of tokens available. Unclear what this means.
total_tokens Total number of tokens included in the installed Qlik Sense license. Unclear what this means.

Butler uptime info

Measurement: butler_memory_usage

Tags

Tag name Description
butler_instance Name of the Butler instance, from config file.
version Version of Butler.

Fields

Field name Description
heap_used Amount of heap memory used by Butler.
heap_total Total amount of heap memory available to Butler.
external Amount of external memory used by Butler.
process_memory Amount of memory used by the Butler process.

9 - Helper subs that make it easy to use Butler APIs

Butler exposes a REST API that can be called from many different systems and tools, just not Qlik Sense itself.

When it comes to Sense, it’s very doable but not entirely easy to call REST APIs from an app’s load script. Butler comes with a set of subs/functions that wrap the API calls and make it a lot easier and cleaner to call Butler APIs from load scripts.

This page describes those subs.

Currently available helper subs

These sub() are available in the the file butler_subs.qvs in the GitHub repo.

They are also part of the demo app Butler 5.2 demo app, which is also available in the GitHub repo.

Sub name Description
NiceTrace Prints timestamped messages in the reload log.
ButlerInit Initializes some data structures used by Butler. Right now there isn’t much done in this sub, but this may change in future versions.
GetEnabledButlerAPIEndpoints Gets a list of which Butler API endpoints are enabled.
ReloadSenseApp Does a full or partial app reload in the engine. No reload tasks are involved in the actual reload, but when the reload is done different reload tasks can be started depending on whether the reload was successful or not.
StartTask_KeyValue Same as above, but a key-value pair can also be included via the extra parameters. This KV pair is saved in Butler’s KV store (assuming it’s enabled). Any Sense app (or other system) can then query the KV store for these parameters. It’s thus an easy but effective way of passing arbitrary parameters to Sense tasks (or rather to the associated app’s load scripts).
StartTask Start a single or many reload tasks. Tasks can be selected using task IDs, tags and custom properties.
Full freedom to use any number of task IDs, tags and custom properties to specify what tasks should be started.
Any number of key-value pairs can be passed along as parameters.
CopyFile Copy a file or directory in a file system that’s accessible to Butler and also approved in the config file.
MoveFile Move a file or directory in a file system that’s accessible to Butler and also approved in the config file.
DeleteFile Delete a file in a file system that’s accessible to Butler and also approved in the config file.
CreateDir Create new directory anywhere Butler has access. Warning! If Butler is running with admin privileges there is no stopping a user from creating directories anywhere.
CreateDirQVD Create a directory somewhere under the QVD folder specified in the config file’s Butler.configDirectories.qvdPath. This is less of a security concern than the previous call, of course.
AddKeyValue Add a key-value pair to a namespace in the Key-Value (KV) store.
GetKeyValue Get KV pair in a namespace.
DeleteKeyValue Delete a KV pair in a namespace.
ExistsKeyValue Tests if a KV pair exists in a namespace.
DeleteKeyValueNamespace Delete a namespace and all KV pairs in it.
GetKeyValueNamespaces Get all namespaces.
GetKeyValueNamespaces Get all keys in a specifi namespace.
PostToMQTT Post a message to a MQTT topic.
PostToSlack Post a basic message to Slack.

Helper subs on the todo list

There should be helper subs for these API endpoints too, they’re just not there yet…

Sub name Description
AddSchedule Add a new task reload schedule to Butler’s scheduler.
GetSchedule Get information about an existing Butler task schedule.
UpdateSchedule Modify an existing Butler task schedule.
DeleteSchedule Delete a Butler task schedule.
StartSchedule Enable a Butler task schedule.
StopSchedule Disable/pause a Butler task schedule.
PostToTeams Make it easy to post basic messages to MS Teams.

10 - Test cases

Automated test cases are used to ensure that (mainly) Butler’s REST API works as intended.

All test cases are documented in an Excel file stored in the tests directory in the main Butler repository.