This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Monitoring Windows services

Butler can monitor Windows services and alert if they are not running.

This is useful for monitoring services that are critical for Qlik Sense to function - or any other important service.

Messages can be sent when services stop or start, with message destinations such as Slack, Teams, email, New Relic, InfluxDB, webhooks and MQTT.

1: Sending Windows service alerts as email

2: Sending Windows service alerts to New Relic

3: Storing Windows service alerts in InfluxDB

4: Sending Windows service alerts to Slack

5: Sending Windows service alerts to Microsoft Teams

6: Sending Windows service alerts as MQTT messages

7: Sending Windows service alerts as outgoing webhooks (=http messages)

What’s this?

Qlik Sense uses Windows Services to run the Qlik Sense Engine, Qlik Sense Repository Service, Qlik Sense Scheduler Service and more.

If any of these services stop, Qlik Sense will not work.
Butler can monitor these services and alert if they are not running and when they start again.

This feature is only available when Butler is running on Windows, on other OSs a warning will be logged when Butler is starting and the feature will be disabled.

How it works

Butler will poll the Windows Service Control Manager (SCM) for the status of the services that are configured to be monitored.
The polling interval is configurable via the Butler.serviceMonitor.frequency setting, but defaults to 30 seconds.

The services to be monitored are listed in Butler.serviceMonitor.monitor section of the config file.
If firewalls etc allow it it is possible to monitor services on remote Windows machines as well.

Three pieces of information are needed for each service to be monitored:

The host name of the machine where the service is running (Butler.serviceMonitor.monitor.<host>).
This config entry is shared for all services monitored on the same host.
The name of the service (Butler.serviceMonitor.monitor.<services>.name).
This is the name of the service as it appears in the Windows Service Control Manager (SCM). Right click on a service in the Windows Services app and select Properties, then find the “Service name” on the General tab.
A “friendly name” that can be anything (Butler.serviceMonitor.monitor.<services>.friendlyName). This is useful as the Windows service name are not always very descriptive.
The friendly name is used in the alert messages sent to the various alert destinations, including InfluxDB and New Relic.

Each alert destination can be enabled or disabled via the Butler.serviceMonitor.alertDestination.<destination>.enable setting.

Settings in config file

The configuration of each alert destination is done in the destinations’ own section of the config file, for example Butler.teamsNotification.serviceStopped, Butler.emailNotification.serviceStopped, Butler.emailNotification.serviceStarted etc.

Those settings are described in sub-pages of this page.

---
Butler:
  ...
  ...
  # Monitor Windows services.
  # This feature only works when Butler is running on Windows Server or desktop.
  # On other OSs service monitoring will be automatically disabled.
  serviceMonitor:
    enable: false                    # Main on/off switch for service monitoring
    frequency: every 30 seconds     # https://bunkat.github.io/later/parsers.html
    monitor:
      - host: <hostname or IP>      # Host name of Windows computer where services are running
        services:                   # List of services to monitor
          - name: postgresql-x64-12       # Posgress/repository db
            friendlyName: Repository DB
          - name: QlikSenseEngineService
            friendlyName: Engine
          - name: QlikSensePrintingService
            friendlyName: Printing
          - name: QlikSenseProxyService
            friendlyName: Proxy
          - name: QlikSenseRepositoryService
            friendlyName: Repository
          - name: QlikSenseSchedulerService
            friendlyName: Scheduler
          - name: QlikSenseServiceDispatcher
            friendlyName: Service Dispatcher
    alertDestination:               # Control to thich destinations service related alerts are sent
      influxDb:                     # Send service alerts to InfluxDB
        enable: true
      newRelic:                     # Send service alerts to New Relic
        enable: true
      email:                        # Send service alerts as emails
        enable: true                
      mqtt:                         # Send service alerts as MQTT messages
        enable: true
      teams:                        # Send service alerts as MS Teams messages
        enable: true
      slack:                        # Send service alerts as Slack messages
        enable: true
      webhook:                      # Send service alerts as outbound webhooks/http calls
        enable: true
  ...
  ...

1 - Sending Windows service alerts as email

This page contains information on how to configure Butler to send email alerts when Windows services stop or start.

What’s this?

These config settings are specific to the email alert destination.
They are used in addition to the general Windows Service monitoring settings in Butler.serviceMonitor.

How it works

The sent emails are created from template files using the Handlebars templating engine.

The template files are located in the Butler.emailNotification.<alertType>.bodyFileDirectory directory, with the actual file name specified in Butler.emailNotification.<alertType>.htmlTemplateFile.

The template files can contain Handlebars expressions to insert values from the alert data.
The available values are:

Value	Description
`{{host}}`	The hostname of the server where the service is running
`{{serviceStatus}}`	The status of the service, e.g. `RUNNING` or `STOPPED`
`{{servicePrevStatus}}`	The previous status of the service, e.g. `RUNNING` or `STOPPED`
`{{serviceName}}`	The name of the service as defined in Windows
`{{serviceDisplayName}}`	The display name of the service as defined in Windows. Can sometimes be a bit more human readable than the serviceName.
`{{serviceFriendlyName}}`	The display name of the service as defined in the Butler config file. Used to give the service a good name when both serviceName and serviceDisplayName are unsuitable for use in for example Grafana dashboards.
`{{serviceStartType}}`	The startup mode of the service, e.g. `Automatic` or `Manual`
`{{serviceExePath}}`	The path to the executable of the service

Settings in config file

---
Butler:
  ...
  ...
  emailNotification: 
    serviceStopped:
      rateLimit: 30                   # Min seconds between emails for a given service. Defaults to 5 minutes.
      priority: high                  # high/normal/low
      subject: '❌ Windows service stopped on host {{host}}: "{{serviceDisplayName}}"'
      bodyFileDirectory: path/to/email_templates/email_templates
      htmlTemplateFile: service-stopped
      fromAdress: Qlik Sense (no-reply) <qliksense-noreply@mydomain.com>
      recipients:
        - <Email address 1>
        - <Email address 2>
    serviceStarted:
      rateLimit: 30                   # Min seconds between emails for a given service. Defaults to 5 minutes.
      priority: high                  # high/normal/low
      subject: '✅ Windows service started on host {{host}}: "{{serviceDisplayName}}"'
      bodyFileDirectory: path/to/email_templates/email_templates
      htmlTemplateFile: service-started
      fromAdress: Qlik Sense (no-reply) <qliksense-noreply@mydomain.com>
      recipients:
        - <Email address 1>
        - <Email address 2>
    smtp:                                             # Email server settings. See https://nodemailer.com/smtp/ for details on the meaning of these fields.
      host: <FQDN or IP or email server, e.g. smtp.gmail.com>
      port: <port on which SMTP server is listening>
      secure: true                                    # true/false
      tls:
        serverName:                                   # If specified the serverName field will be used for TLS verification instead of the host field.
        ignoreTLS: false
        requireTLS: true
        rejectUnauthorized: false
      auth:
        enable: true
        user: <Username, email address etc>
        password: <your-secret-password>
  ...  
  ...

2 - Sending Windows service alerts to New Relic

This page contains information on how to configure Butler to send alerts messages to New Relic when Windows services stop or start.

What’s this?

These config settings are specific to the New Relic alert destination.
They are used in addition to the general Windows Service monitoring settings in Butler.serviceMonitor.

How it works

All settings are found in the Butler.incidetTool.newRelic.serviceMonitor section of the config file.

Butler can send two kinds of messages to New Relic: events and logs entries.
New Relic events and log entries are good at different things, and you can choose to send either or both.

In general, events are good for monitoring and alerting while log entries are good for logging and troubleshooting.
If in doubt, send both - that will give you the freedom to choose later which to use in the New Relic dashboards, alerts and incidents.

New Relic events

Windows service events will be sent to New Relic with the name of qs_serviceStateEvent.

The static attributes attached to events sents to New Relic events are the ones defined in the config file.
These can be used to identify which of potentially several Butler instances the message originated from, and to filter and group messages in New Relic.

The values of dynamic attributes are determined at runtime and can be enabled or disabled in the config file:

Dynamic attribute name in New Relic	Description
butler_serviceHost	The hostname of the server where the service is running
butler_serviceName	The name of the service as defined in Windows
butler_serviceDisplayName	The display name of the service as defined in Windows. Can sometimes be a bit more human readable than the serviceName.
butler_serviceStatus	The status of the service, e.g. `RUNNING` or `STOPPED`

New Relic event for a Windows service alert message

New Relic log entries

Windows service log entries will be sent to New Relic with a log type of qs_serviceStateLog.

Static and dynamic attributes are handled in the same way as for events.

The raw data of a New Relic lg entry will look something like this:

New Relic log entry for a Windows service alert message

Settings in config file

---
Butler:
  ...
  ...
  incidentTool: 
    newRelic:
      serviceMonitor:
        destination:
          event: 
            enable: false
            sendToAccount:                # Windows service events are sent to these New Relic accounts
              - First NR account
              - Second NR account
            attribute: 
              static:                     # Static attributes/dimensions to attach to events sent to New Relic.
                - name: event-specific-attribute
                  value: abc 123
              dynamic:
                serviceHost: true         # Should host where service is running be sent to New Relic as attribute?
                serviceName: true         # Should service name be sent to New Relic as attribute?
                serviceDisplayName: true  # Should service display name be sent to New Relic as attribute?
                serviceState: true        # Should service state be sent to New Relic as attribute?
          log:
            enable: false
            sendToAccount:                # Windows service log entries are sent to these New Relic accounts
              - First NR account
              - Second NR account
            attribute: 
              static:                     # Static attributes/dimensions to attach to events sent to New Relic.
                - name: log-specific-attribute
                  value: def 456
              dynamic:
                serviceHost: true         # Should host where service is running be sent to New Relic as attribute?
                serviceName: true         # Should service name be sent to New Relic as attribute?
                serviceDisplayName: true  # Should service display name be sent to New Relic as attribute?
                serviceState: true        # Should service state be sent to New Relic as attribute?
        monitorServiceState:              # Control whih service states are sent to New Relic
          running:
            enable: true
          stopped:
            enable: true
        sharedSettings:
          rateLimit: 5                    # Min seconds between events/logs sent to New Relic for a given host+service. Defaults to 5 minutes.
          header:                         # Custom http headers
            - name: X-My-Header           # Example
              value: Header value 2       # Example
          attribute: 
            static:                       # Static attributes/dimensions to attach to events sent to New Relic.
              - name: service             # Example
                value: butler             # Example
              - name: environment         # Example
                value: prod               # Example
  ...  
  ...

3 - Storing Windows service alerts in InfluxDB

This page contains information on how to configure Butler to store alert information in InfluxDB when Windows services stop or start.

What’s this?

These config settings are specific to the InfluxDB alert destination.
They are used in addition to the general Windows Service monitoring settings in Butler.serviceMonitor.

How it works

There is no specific InfluxDB conmfiguration for Windows Service monitoring, so the general InfluxDB in Butler.influxDb settings are used.
This means that information about Windows service alerts are stored in the same InfluxDB database as other data points sent to InfluxDB from Butler (e.g. uptime metrics).

Settings in config file

---
Butler:
  ...
  ...
  # InfluxDB settings
  influxDb:
    enable: false                  # Master switch for InfluxDB integration. If false, no data will be sent to InfluxDB.
    hostIP: <IP or host name>     # Where is InfluxDB server located?
    hostPort: 8086                # InfluxDB port
    auth:
      enable: false               # Does InfluxDB require login?
      username: user_joe      
      password: joesecret
    dbName: butler                # Name of database in InfluxDB to which Butler's data is written
    instanceTag: DEV              # Tag that can be used to differentiate data from multiple Butler instances
    # Default retention policy that should be created in InfluxDB when Butler creates a new database there. 
    # Any data older than retention policy threshold will be purged from InfluxDB.
    retentionPolicy:
      name: 10d
      duration: 10d    
  ...  
  ...

4 - Sending Windows service alerts to Slack

This page contains information on how to configure Butler to send alerts messages to Slack when Windows services stop or start.

What’s this?

These config settings are specific to the Slack alert destination.
They are used in addition to the general Windows Service monitoring settings in Butler.serviceMonitor.

How it works

All settings are found in the Butler.slackNotification.serviceStopped and Butler.slackNotification.serviceStarted sections of the config file.

Butler will send a Slack message to the channel specified in the config file when a Windows service stops or starts.

Similarly to how reload-failed Slack alerts work, Butler can send two types of Slack messages:

A simple message with just the name of the service that stopped or started. This will be the case if Butler.slackNotification.serviceStopped.messageType or Butler.slackNotification.serviceStarted.messageType is set to basic.
A more detailed and better formatted message with information about the service, the server it’s running on etc. This will be the case if Butler.slackNotification.serviceStopped.messageType or Butler.slackNotification.serviceStarted.messageType is set to formatted.

Rate limiting is controlled by the Butler.slackNotification.serviceStopped.rateLimit and Butler.slackNotification.serviceStarted.rateLimit settings.

Tip

The template used to create formatted Slack messages can be customized.

Check out the handlebars documentation for more information on how to do this.

A formatted Slack message can look something like this:

Slack message when a Windows service has stopped

Information availble in formatted Slack messages

Similar to how failed-reload email notifications work, the templating engine Handlebars is used to format the Slack messages.

The following information is available in formatted Slack messages:

Handlebars variable	Description
`{{host}}`	The hostname of the server where the service is running
`{{serviceStatus}}`	The status of the service, e.g. `RUNNING` or `STOPPED`
`{{servicePrevStatus}}`	The previous status of the service, e.g. `RUNNING` or `STOPPED`
`{{serviceName}}`	The name of the service as defined in Windows
`{{serviceDisplayName}}`	The display name of the service as defined in Windows. Can sometimes be a bit more human readable than the serviceName.
`{{serviceFriendlyName}}`	The friendly name of the service as defined in the config file.
`{{serviceStartType}}`	The start type of the service, e.g. `AUTO_START` or `DEMAND_START`
`{{serviceExePath}}`	The path to the service executable

Settings in config file

---
Butler:
  ...
  ...
  # Settings for notifications and messages sent to Slack
  slackNotification:
    serviceStopped:
      webhookURL: <web hook URL from Slack>
      channel: qliksense-service-alert  # Slack channel to which Windows service stopped notifications are sent
      messageType: formatted          # formatted / basic. Formatted means that template file below will be used to create the message.
      basicMsgTemplate: 'Windows service stopped: "{{serviceName}}" on host "{{host}}"'       # Only needed if message type = basic
      rateLimit: 30                   # Min seconds between messages for a given Windows service. Defaults to 5 minutes.
      templateFile: /path/to/slack/template/directory/service-stopped.handlebars
      fromUser: Qlik Sense
      iconEmoji: ':ghost:'
    serviceStarted:
      webhookURL: <web hook URL from Slack>
      channel: qliksense-service-alert  # Slack channel to which Windows service stopped notifications are sent
      messageType: formatted          # formatted / basic. Formatted means that template file below will be used to create the message.
      basicMsgTemplate: 'Windows service started: "{{serviceName}}" on host "{{host}}"'       # Only needed if message type = basic
      rateLimit: 30                   # Min seconds between messages for a given Windows service. Defaults to 5 minutes.
      templateFile: /path/to/slack/template/directory/service-started.handlebars
      fromUser: Qlik Sense
      iconEmoji: ':ghost:'
  ...  
  ...

5 - Sending Windows service alerts to Microsoft Teams

This page contains information on how to configure Butler to send alerts messages to Microsoft Teams when Windows services stop or start.

What’s this?

These config settings are specific to the Microsoft Teams alert destination.
They are used in addition to the general Windows Service monitoring settings in Butler.serviceMonitor.

How it works

All settings are found in the Butler.teamsNotification.serviceStopped and Butler.teamsNotification.serviceStarted sections of the config file.

Butler will send a Teams message to the channel associated with Butler.teamsNotification.<serviceStopped|servierStarted>.webhookRL in the config file when a Windows service stops or starts.

Similarly to how reload-failed Teams alerts work, Butler can send two types of Teams messages:

A simple message with just the name of the service that stopped or started. This will be the case if Butler.teamsNotification.serviceStopped.messageType or Butler.teamsNotification.serviceStarted.messageType is set to basic.
A more detailed and better formatted message with information about the service, the server it’s running on etc. This will be the case if Butler.teamsNotification.serviceStopped.messageType or Butler.teamsNotification.serviceStarted.messageType is set to formatted.

Rate limiting is controlled by the Butler.teamsNotification.serviceStopped.rateLimit and Butler.teamsNotification.serviceStarted.rateLimit settings.

Tip

The template used to create formatted Teams messages can be customized.

Check out the handlebars documentation for more information on how to do this.

A formatted Teams message can look something like this:

Teams message when a Windows service has stopped

Information availble in formatted Teams messages

Similar to how failed-reload email notifications work, the templating engine Handlebars is used to format the Teams messages.

The following information is available in formatted Teams messages:

Handlebars variable	Description
`{{host}}`	The hostname of the server where the service is running.
`{{serviceStatus}}`	The status of the service, e.g. `RUNNING` or `STOPPED`.
`{{servicePrevStatus}}`	The previous status of the service, e.g. `RUNNING` or `STOPPED`.
`{{serviceName}}`	The name of the service as defined in Windows.
`{{serviceDisplayName}}`	The display name of the service as defined in Windows. Can sometimes be a bit more human readable than the serviceName.
`{{serviceFriendlyName}}`	The friendly name of the service as defined in the config file.
`{{serviceStartType}}`	The start type of the service, e.g. `AUTO_START` or `DEMAND_START`.
`{{serviceExePath}}`	The path to the service executable.

Settings in config file

---
Butler:
  ...
  ...
  # Settings for notifications and messages sent to MS Teams
  teamsNotification:
    serviceStopped:
      webhookURL: <web hook URL from MS Teams>
      messageType: formatted          # formatted / basic. Formatted means that template file below will be used to create the message.
      basicMsgTemplate: 'Windows service stopped: "{{serviceName}}" on host "{{host}}"'       # Only needed if message type = basic
      rateLimit: 30                   # Min seconds between messages for a given Windows service. Defaults to 5 minutes.
      templateFile: /path/to/teams/template/directory/service-stopped.handlebars
    serviceStarted:
      webhookURL: <web hook URL from MS Teams>
      messageType: formatted          # formatted / basic. Formatted means that template file below will be used to create the message.
      basicMsgTemplate: 'Windows service started: "{{serviceName}}" on host "{{host}}"'       # Only needed if message type = basic
      rateLimit: 30                   # Min seconds between messages for a given Windows service. Defaults to 5 minutes.
      templateFile: /path/to/teams/template/directory/service-started.handlebars
  ...  
  ...

6 - Sending Windows service alerts as MQTT messages

This page contains information on how to configure Butler to send alerts as MQTT messages when Windows services stop or start.

What’s this?

These config settings are specific to the MQTT alert destination.
They are used in addition to the general Windows Service monitoring settings in Butler.serviceMonitor.

How it works

All settings are found in the Butler.mqttConfig section of the config file.

Butler will send two kinds of MQTT messages:

A state message indicating that a service has changed its state, for example from RUNNING to STOPPED.
- When a service stops or starts, Butler will send a message to the topic defined in Butler.mqttConfig.serviceStoppedTopic, with /<hostname>/<serviceName> appended to the topic. The payload will be a JSON with information about the service (name, display name, current state, previous state, dependencies, EXE path etc.).)
- When a service starts the same thing happens, but the base topic used is defined in Butler.mqttConfig.serviceStartedTopic.
A message containing the current state of a service. These messages are sent when Butler starts up and when the state of a service changes.
- The base MQTT topic for these messages are defined in the Butler.mqttConfig.serviceStateTopic setting. To this topic, Butler will append /<hostname>/<serviceName> before sending the message.
- These messages are sent every time Butler checks the status of the Windows services, i.e. every Butler.serviceMonitor.frequency seconds.
- The MQTT message will be sent as a JSON with information about the service (name, display name, current state, dependencies, EXE path etc.).

A few MQTT message can look like this when viewed in MQTT Explorer:

MQTT messages related to Windows services

Settings in config file

---
Butler:
  ...
  ...
  mqttConfig:
    enable: false                                     # Should Qlik Sense events be forwarded as MQTT messages?
    brokerHost: <FQDN or IP of MQTT server>
    brokerPort: 1883
    serviceRunningTopic: qliksense/service_running
    serviceStoppedTopic: qliksense/service_stopped
    serviceStatusTopic: qliksense/service_status  
  ...  
  ...

7 - Sending Windows service alerts as outgoing webhooks (=http messages)

This page contains information on how to configure Butler to send alerts as outbound http calls, also known as “outbound webhooks”.

What’s this?

These config settings are specific to the outbound webhook alert destination.
They are used in addition to the general Windows Service monitoring settings in Butler.serviceMonitor.

How it works

All settings are found in the Butler.webhookNotification section of the config file.

Butler can send three kinds of http messages: POST, PUT and GET.
Some services only support one/some of these, so you need to check the documentation for the service you want to send the message to.

It is possible to define any number of webhook, and each destination can have its own settings such as http method and URL.
It is for example possible to send POST messages to different URLs if needed.

The rate limit defined in Butler.webhookNotification.rateLimit is calculated against each state change of the monitored Windows service.
There is no check with respect to rate limits how manu URLs are defined (and thus outbound http messages are sent).

Payload of outbound http calls

The same webhooks/URLs are used for both Windows service start and stop events.
The defails of the Windows service events is sent in the payload of the http message - exactly how depends on the http method used.

POST

The payload is sent as JSON in the body of the http message.

Here Node-RED is used to receive the http message and display it in a debug window:

POST http call when Windows service has stopped

PUT

The message payload is sent in the body, exactly as for POST messages.

The same fields are used as for POST messages:

PUT http call when Windows service has stopped

GET

The message payload is sent as URL query parameters rather than in the body.

The fields are the same as for POST and PUT messages, except that the field names are in lower case.

GET http call when Windows service has stopped

Settings in config file

---
Butler:
  ...
  ...
  # Settings for notifications and messages sent using outgoing webhooks
  webhookNotification:
    enable: false
    serviceMonitor:
      rateLimit: 15               # Min seconds between outgoing webhook calls, per Windows service that is monitored. Defaults to 5 minutes.
      webhooks:
        - description: 'This outgoing webhook is used to...'
          webhookURL: http://host.my.domain:port/some/path    # outgoing webhook that Butler will call
          httpMethod: POST                                    # GET/POST/PUT. Note that the body and URL query parameters differs depending on which method is used
        - description: 'This outgoing webhook is used to...'
          webhookURL: http://host.my.domain:port/some/path    # outgoing webhook that Butler will call
          httpMethod: PUT                                     # GET/POST/PUT. Note that the body and URL query parameters differs depending on which method is used
        - description: 'This outgoing webhook is used to...'
          webhookURL: http://host.my.domain:port/some/path    # outgoing webhook that Butler will call
          httpMethod: GET                                     # GET/POST/PUT. Note that the body and URL query parameters differs depending on which method is used
  ...
  ...