Butler offers a lot of flexibility when it comes to alerts when reloads fail or are aborted in Qlik Sense Enterprise on Windows (QSEoW).
Learn how to set up the desired features, the alert layout, formatting and more.
Reload task failure. Send alerts when reload tasks fail, no matter if they were started on schedule or manually from the QMC.
Reload task aborted. Send alerts when reload tasks are manually aborted in the QMC.
Alert destinations and options
Alerts can be sent to these destinations, with different options available for each destination.
Each destination can be individually enabled/disabled in the config file.
The failed reload’s script log is available in InfluxDb.
New Relic
✅
✅
✅
-
✅
-
The failed reload’s script log is available in New Relic.
Signl4
✅
✅
✅
-
✅
-
Alerts are presented in Signl4’s own format in their mobile app.
Slack
✅
✅
✅
✅
MS Teams
✅
✅
✅
✅
Outgoing webhook
✅
✅
-
-
Formatting is not relevant for webhooks
MQTT
✅
✅
-
-
Formatting is not relevant for MQTT messages
How it works
In order for Butler initiated alerts to become a reality, Butler must somehow be notified that the event of interest (for example a failed reload task) has occurred.
This is achieved by adding a log appender to Qlik Sense Enterprise on Windows.
Log appenders offer a way to hook into Qlik Sense’s logging subsystem, which is called log4net.
By adding a carefully crafted .xml file in the right location on the Sense server(s), you can make Sense notify Butler by means of UDP messages when the events of interest occur. Conceptually it looks like this:
So what happens when a scheduled reload task fails?
Let’s look at the steps:
A reload task is started by the Sense scheduler, either on a time schedule, as a result of some other task(s) finishing or manually by a user in the QMC or from the Hub.
When the task’s state changes, entries are written to the Sense scheduler’s log files using log4net (which is built into Qlik Sense). If the filter defined in the log appender (= the .xml file on the Sense server) matches the log entry at hand, the associated action in the log appender will be carried out.
Log appenders can do all kinds of things, everything from writing custom log files, sending basic emails, writing to databases and much more.
Here we’re interested in the log appender sending a UDP message from Qlik Sense to Butler.
The log appender provided as part of Butler will make log4net send a UDP message to Butler, including various info about the reload task that just failed or was stopped/aborted.
Butler will look at the incoming event and determine what it is about.
For example: Is the event about a reload task failure, a reload that has been aborted/stopped, or something else?
Butler thus first works as a dispatcher. In a second step, after the initial dispatch, the event is sent to the relevant handler function within Butler.
Response times are usually very good - Butler will typically get the UDP message within a few seconds after (for example) the reload failing, with alerts going out shortly thereafter.
Warning
The log appenders that catch failed and aborted reloads in the Qlik Sense engine and scheduler must be set up on all Qlik Sense servers where reloads are happening for this feature to work.
Failing to do so will result in Butler not being notified about some reload failures/aborted reloads.
Adding a log appender
This is possibly the trickiest part to get right when it comes to setting up log4net based alerts.
Still, if you start from the sample .xml file provided in the Butler repository on GitHub it’s not too hard.
Those sample .xml files are also included in the release Zip files available on the Butler releases page.
The steps are:
In this case you want to be notified when certain events occur in the scheduler log files.
This is important: Qlik Sense Enterprise on Windows consists of many different subsystems (engine, proxy, scheduler, printing etc) - here we’re interested in log events from the scheduler subsystem.
Add a file LocalLogConfig.xml in the C:\ProgramData\Qlik\Sense\Scheduler folder on the Sense server whose scheduler you want to get events from. If you have multiple Sense servers with schedulers running on them, the .xml file should be deployed on each server (assuming you want events from all the servers).
The contents of LocalLogConfig.xml will determine what events are forwarded to Butler, or what other actions will be taken by log4net. See below for examples.
Sense will eventually detect and load the new xml file, but it might take a while (minutes). Restarting the Qlik Sense Scheduler Windows service will make the changes take effect immediately.
Forwarding task reload events to Butler
Here’s the XML that should go into C:\ProgramData\Qlik\Sense\Scheduler\LocalLogConfig.xml to enable the various kinds of Butler task reload alerts.
The remoteAddress property should be set to the host name or IP where Butler is running.
The remotePort property should match the port number specified in Butler’s config file. Note that Butler uses different ports for task related and user activity related events.
The first appender looks for the text “Max retries reached” in the System.Scheduler.Scheduler.Master.Task.TaskSession log stream. That log entry will be created when a reload task has failed and also carried out all its retries. Once the search string is found a UDP message will be sent to port 9998 on IP 10.11.12.13.
The second appender looks for “Execution State Change to Aborting” in the System.Scheduler.Scheduler.Master.Task.TaskSession log stream. That log entry occurs when a user stops a running reload from the QMC’s task view, or using the Sense APIs. When the search string is found a UDP message is once again sent to 10.11.12.13:9998, but with a different messsage (as specified in the conversionpattern property of the appender).
The third appender looks for “Reload complete” in the System.Scheduler.Scheduler.Slave.Tasks.ReloadTask log stream.
That log entry occurs when a reload task has completed successfully.
Here is an XML file that would forward log events as UDP messages to Butler:
<?xml version="1.0" encoding="UTF-8"?><configuration><!-- Appender for detecting reload task failures. Only the last of potentially several retries is reported --><appendername="TaskFailureLogger"type="log4net.Appender.UdpAppender"><filtertype="log4net.Filter.StringMatchFilter"><paramname="stringToMatch"value="Max retries reached"/></filter><filtertype="log4net.Filter.DenyAllFilter"/><paramname="remoteAddress"value="<IP of server where Butler is running>"/><paramname="remotePort"value="9998"/><paramname="encoding"value="utf-8"/><layouttype="log4net.Layout.PatternLayout"><converter><paramname="name"value="hostname"/><paramname="type"value="Qlik.Sense.Logging.log4net.Layout.Pattern.HostNamePatternConverter"/></converter><paramname="conversionpattern"value="/scheduler-reload-failed/;%hostname;%property{TaskName};%property{AppName};%property{User};%property{TaskId};%property{AppId};%date;%level;%property{ExecutionId};%message"/></layout></appender><!-- Appender for detecting aborted reloads --><appendername="AbortedReloadTaskLogger"type="log4net.Appender.UdpAppender"><filtertype="log4net.Filter.StringMatchFilter"><paramname="stringToMatch"value="Execution State Change to Aborting"/></filter><filtertype="log4net.Filter.DenyAllFilter"/><paramname="remoteAddress"value="<IP of server where Butler is running>"/><paramname="remotePort"value="9998"/><paramname="encoding"value="utf-8"/><layouttype="log4net.Layout.PatternLayout"><converter><paramname="name"value="hostname"/><paramname="type"value="Qlik.Sense.Logging.log4net.Layout.Pattern.HostNamePatternConverter"/></converter><paramname="conversionpattern"value="/scheduler-reload-aborted/;%hostname;%property{TaskName};%property{AppName};%property{User};%property{TaskId};%property{AppId};%date;%level;%property{ExecutionId};%message"/></layout></appender><!-- Appender for detecting successful reload tasks --><appendername="ReloadTaskSuccessLogger"type="log4net.Appender.UdpAppender"><filtertype="log4net.Filter.StringMatchFilter"><paramname="stringToMatch"value="Reload complete"/></filter><filtertype="log4net.Filter.DenyAllFilter"/><paramname="remoteAddress"value="<IP of server where Butler is running>"/><paramname="remotePort"value="9998"/><paramname="encoding"value="utf-8"/><layouttype="log4net.Layout.PatternLayout"><converter><paramname="name"value="hostname"/><paramname="type"value="Qlik.Sense.Logging.log4net.Layout.Pattern.HostNamePatternConverter"/></converter><paramname="conversionpattern"value="/scheduler-reloadtask-success/;%hostname;%property{TaskName};%property{AppName};%property{User};%property{TaskId};%property{AppId};%date;%level;%property{ExecutionId};%message"/></layout></appender><!-- Send message to Butler on task failure --><!-- Send message to Butler on task abort --><loggername="System.Scheduler.Scheduler.Master.Task.TaskSession"><appender-refref="TaskFailureLogger"/><appender-refref="AbortedReloadTaskLogger"/></logger><!-- Send message to Butler on reload task success --><loggername="System.Scheduler.Scheduler.Slave.Tasks.ReloadTask"><appender-refref="ReloadTaskSuccessLogger"/></logger></configuration>
The above configuration is enough to support all task reload alerts currently supported by Butler.
Sending basic alert emails from log4net
If you are happy with the more basic/limited reload-failed alert emails provided by log4net, you can add a SMTP appender like this (the example below is for sending emails using Google GMail, customise as needed):
<?xml version="1.0"?><configuration><!-- Mail appender--><appendername="MailAppender"type="log4net.Appender.SmtpAppender"><filtertype="log4net.Filter.StringMatchFilter"><paramname="stringToMatch"value="Message from ReloadProvider"/></filter><filtertype="log4net.Filter.DenyAllFilter"/><evaluatortype="log4net.Core.LevelEvaluator"><paramname="threshold"value="ERROR"/></evaluator><paramname="to"value="<email address to send failed task notification emails to>"/><paramname="from"value="<sender email address used in notification emails>"/><paramname="subject"value="Qlik Sense failed task (server <servername>)"/><paramname="smtpHost"value="smtp.gmail.com"/><paramname="port"value="587"/><paramname="EnableSsl"value="true"/><paramname="Authentication"value="Basic"/><paramname="username"value="<Gmail username>"/><paramname="password"value="<Gmail password>"/><paramname="bufferSize"value="0"/><!-- Set this to 0 to make sure an email is sent on every error --><paramname="lossy"value="true"/><layouttype="log4net.Layout.PatternLayout"><paramname="conversionPattern"value="%newline%date %-5level %newline%property{TaskName}%newline%property{AppName}%newline%message%newline%newline%newline"/></layout></appender><!--Send mail on task failure--><loggername="System.Scheduler.Scheduler.Slave.Tasks.ReloadTask"><appender-refref="MailAppender"/></logger></configuration>
References
Qlik’s documenation around log appenders and how to hook into the Sense logs is somewhat brief, but does provide a starting point if you want to dive deeper into this topic.
The main log4net documentation (log4net is the logging framework used by Qlik Sense Enterprise) can also be useful.
These links describe how emails can be sent from the log4net logging framework itself, directly to the recipient. Butler includes sameple XML files for this use case too, but Butler takes things further by using the data in the Sense logs to pull in more data around the failed or stopped reload.
In other words - Butler’s alert emails are significantly more flexible and contain information (such as script logs) that are not availble using purely log4net.
1 - Reload alerts for client-managed Qlik Sense
Butler offers a lot of flexibility when it comes to alerts when reloads fail, are aborted or succeed in Qlik Sense Enterprise on Windows (QSEoW).
Learn how to set up the desired features, the alert layout, formatting and more.
Alert types
These alert types are available:
Reload task failure. Send alerts when reload tasks fail, no matter if they were started on schedule or manually from the QMC.
Reload task aborted. Send alerts when reload tasks are manually aborted in the QMC.
Reload task success. Send alerts when reload tasks complete successfully.
Alert destinations and options
Alerts can be sent to these destinations, with different options available for each destination.
Each destination can be individually enabled/disabled in the config file.
The failed reload’s script log is available in InfluxDb.
New Relic
✅
✅
✅
✅
The failed reload’s script log is available in New Relic.
Signl4
✅
✅
✅
✅
Alerts are presented in Signl4’s own format in their mobile app.
Slack
✅
✅
✅
✅
MS Teams
✅
✅
✅
✅
Outgoing webhook
✅
✅
Formatting is not relevant for webhooks
MQTT
✅
✅
Formatting is not relevant for MQTT messages
How it works
In order for Butler initiated alerts to become a reality, Butler must somehow be notified that the event of interest (for example a failed reload task) has occurred.
This is achieved by adding a log appender to Qlik Sense Enterprise on Windows.
Log appenders offer a way to hook into Qlik Sense’s logging subsystem, which is called log4net.
By adding a carefully crafted .xml file in the right location on the Sense server(s), you can make Sense notify Butler by means of UDP messages when the events of interest occur. Conceptually it looks like this:
So what happens when a scheduled reload task fails?
Let’s look at the steps:
A reload task is started by the Sense scheduler, either on a time schedule, as a result of some other task(s) finishing or manually by a user in the QMC or from the Hub.
When the task’s state changes, entries are written to the Sense scheduler’s log files using log4net (which is built into Qlik Sense). If the filter defined in the log appender (= the .xml file on the Sense server) matches the log entry at hand, the associated action in the log appender will be carried out.
Log appenders can do all kinds of things, everything from writing custom log files, sending basic emails, writing to databases and much more.
Here we’re interested in the log appender sending a UDP message from Qlik Sense to Butler.
The log appender provided as part of Butler will make log4net send a UDP message to Butler, including various info (reload task ID, timestamp, host name etc) about the reload task that just failed or was stopped/aborted.
Butler will look at the incoming event and determine what it is about.
For example: Is the event about a reload task failure, a reload that has been aborted/stopped, or something else?
Butler thus first works as a dispatcher. In a second step, after the initial dispatch, the event is sent to the relevant handler function within Butler.
Response times are usually very good - Butler will typically get the UDP message within a few seconds after (for example) the reload failing, with alerts going out shortly thereafter.
Warning
The log appenders that catch failed and aborted reloads in the Qlik Sense engine and scheduler must be set up on all Qlik Sense servers where reloads are happening for this feature to work.
Failing to do so will result in Butler not being notified about some reload failures/aborted reloads.
Tip
The concept above is the same also for aborted and successful reload tasks.
Adding a log appender
This is possibly the trickiest part to get right when it comes to setting up log4net based alerts.
Still, if you start from the sample .xml file provided in the Butler repository on GitHub it’s not too hard.
Those sample .xml files are also included in the release Zip files available on the Butler releases page.
The steps are:
In this case you want to be notified when certain events occur in the scheduler log files.
This is important: Qlik Sense Enterprise on Windows consists of many different subsystems (engine, proxy, scheduler, printing etc) - here we’re interested in log events from the scheduler subsystem.
Add a file LocalLogConfig.xml in the C:\ProgramData\Qlik\Sense\Scheduler folder on the Sense server whose scheduler you want to get events from. If you have multiple Sense servers with schedulers running on them, the .xml file should be deployed on each server (assuming you want events from all the servers).
The contents of LocalLogConfig.xml will determine what events are forwarded to Butler, or what other actions will be taken by log4net. See below for examples.
Sense will eventually detect and load the new xml file, but it might take a while (minutes). Restarting the Qlik Sense Scheduler Windows service will make the changes take effect immediately.
Forwarding reload task events to Butler
Here’s the XML that should go into C:\ProgramData\Qlik\Sense\Scheduler\LocalLogConfig.xml to enable the various kinds of Butler task reload alerts.
The remoteAddress property should be set to the host name or IP where Butler is running.
The remotePort property should match the port number specified in Butler’s config file. Note that Butler uses different ports for task related and user activity related events.
The first appender looks for the text “Max retries reached” in the System.Scheduler.Scheduler.Master.Task.TaskSession log stream. That log entry will be created when a reload task has failed and also carried out all its retries. Once the search string is found a UDP message will be sent to port 9998 on IP 10.11.12.13.
The second appender looks for “Execution State Change to Aborting” in the System.Scheduler.Scheduler.Master.Task.TaskSession log stream. That log entry occurs when a user stops a running reload from the QMC’s task view, or using the Sense APIs. When the search string is found a UDP message is once again sent to 10.11.12.13:9998, but with a different messsage (as specified in the conversionpattern property of the appender).
The third appender looks for “Reload complete” in the System.Scheduler.Scheduler.Slave.Tasks.ReloadTask log stream.
That log entry occurs when a reload task has completed successfully.
Here is an XML file that would forward log events as UDP messages to Butler:
<?xml version="1.0" encoding="UTF-8"?><configuration><!-- Appender for detecting reload task failures. Only the last of potentially several retries is reported --><appendername="TaskFailureLogger"type="log4net.Appender.UdpAppender"><filtertype="log4net.Filter.StringMatchFilter"><paramname="stringToMatch"value="Max retries reached"/></filter><filtertype="log4net.Filter.DenyAllFilter"/><paramname="remoteAddress"value="<IP of server where Butler is running>"/><paramname="remotePort"value="9998"/><paramname="encoding"value="utf-8"/><layouttype="log4net.Layout.PatternLayout"><converter><paramname="name"value="hostname"/><paramname="type"value="Qlik.Sense.Logging.log4net.Layout.Pattern.HostNamePatternConverter"/></converter><paramname="conversionpattern"value="/scheduler-reload-failed/;%hostname;%property{TaskName};%property{AppName};%property{User};%property{TaskId};%property{AppId};%date;%level;%property{ExecutionId};%message"/></layout></appender><!-- Appender for detecting aborted reloads --><appendername="AbortedReloadTaskLogger"type="log4net.Appender.UdpAppender"><filtertype="log4net.Filter.StringMatchFilter"><paramname="stringToMatch"value="Execution State Change to Aborting"/></filter><filtertype="log4net.Filter.DenyAllFilter"/><paramname="remoteAddress"value="<IP of server where Butler is running>"/><paramname="remotePort"value="9998"/><paramname="encoding"value="utf-8"/><layouttype="log4net.Layout.PatternLayout"><converter><paramname="name"value="hostname"/><paramname="type"value="Qlik.Sense.Logging.log4net.Layout.Pattern.HostNamePatternConverter"/></converter><paramname="conversionpattern"value="/scheduler-reload-aborted/;%hostname;%property{TaskName};%property{AppName};%property{User};%property{TaskId};%property{AppId};%date;%level;%property{ExecutionId};%message"/></layout></appender><!-- Appender for detecting successful reload tasks --><appendername="ReloadTaskSuccessLogger"type="log4net.Appender.UdpAppender"><filtertype="log4net.Filter.StringMatchFilter"><paramname="stringToMatch"value="Execution State Change to FinishedSuccess"/></filter><filtertype="log4net.Filter.DenyAllFilter"/><paramname="remoteAddress"value="<IP of server where Butler is running>"/><paramname="remotePort"value="9998"/><paramname="encoding"value="utf-8"/><layouttype="log4net.Layout.PatternLayout"><converter><paramname="name"value="hostname"/><paramname="type"value="Qlik.Sense.Logging.log4net.Layout.Pattern.HostNamePatternConverter"/></converter><paramname="conversionpattern"value="/scheduler-reloadtask-success/;%hostname;%property{TaskName};%property{AppName};%property{User};%property{TaskId};%property{AppId};%date;%level;%property{ExecutionId};%message"/></layout></appender><!-- Send message to Butler on task failure --><!-- Send message to Butler on task abort --><!-- Send message to Butler on reload task success --><loggername="System.Scheduler.Scheduler.Master.Task.TaskSession"><appender-refref="TaskFailureLogger"/><appender-refref="AbortedReloadTaskLogger"/><appender-refref="ReloadTaskSuccessLogger"/></logger></configuration>
The above configuration is enough to support all task reload alerts currently supported by Butler.
Sending basic alert emails from Qlik Sense/log4net
If you are happy with the more basic/limited reload-failed alert emails provided by log4net, you can add a SMTP appender like this (the example below is for sending emails using Google GMail, customise as needed).
Note
If sending alert emails from Log4Net you will not get any of the nice formatting, script logs or other features that Butler provides in its alerts.
The email will instead just tell you that a task failed, and include some basic information about the task (task name, specifically).
<?xml version="1.0"?><configuration><!-- Mail appender--><appendername="MailAppender"type="log4net.Appender.SmtpAppender"><filtertype="log4net.Filter.StringMatchFilter"><paramname="stringToMatch"value="Message from ReloadProvider"/></filter><filtertype="log4net.Filter.DenyAllFilter"/><evaluatortype="log4net.Core.LevelEvaluator"><paramname="threshold"value="ERROR"/></evaluator><paramname="to"value="<email address to send failed task notification emails to>"/><paramname="from"value="<sender email address used in notification emails>"/><paramname="subject"value="Qlik Sense failed task (server <servername>)"/><paramname="smtpHost"value="smtp.gmail.com"/><paramname="port"value="587"/><paramname="EnableSsl"value="true"/><paramname="Authentication"value="Basic"/><paramname="username"value="<Gmail username>"/><paramname="password"value="<Gmail password>"/><paramname="bufferSize"value="0"/><!-- Set this to 0 to make sure an email is sent on every error --><paramname="lossy"value="true"/><layouttype="log4net.Layout.PatternLayout"><paramname="conversionPattern"value="%newline%date %-5level %newline%property{TaskName}%newline%property{AppName}%newline%message%newline%newline%newline"/></layout></appender><!--Send mail on task failure--><loggername="System.Scheduler.Scheduler.Slave.Tasks.ReloadTask"><appender-refref="MailAppender"/></logger></configuration>
References
Qlik’s documenation around log appenders and how to hook into the Sense logs is somewhat brief, but does provide a starting point if you want to dive deeper into this topic.
The main log4net documentation (log4net is the logging framework used by Qlik Sense Enterprise) can also be useful.
These links describe how emails can be sent from the log4net logging framework itself, directly to the recipient. Butler includes sameple XML files for this use case too, but Butler takes things further by using the data in the Sense logs to pull in more data around the failed or stopped reload.
In other words - Butler’s alert emails are significantly more flexible and contain information (such as script logs) that are not availble using purely log4net.
1.1 - Reload alerts sent as emails
Description of the various kinds of alert emails Butler can send.
What’s this?
Butler can send two kinds of alert emails:
When a reload task fails during execution.
When a running reload task is somehow stopped/aborted.
When a reload task completes successfully.
Butler has a de-duplication feature that ensure each email address that has qualified for an alert email only gets ONE email per alert.
See the Concepts section for additional details and sample alert emails.
Basic vs formatted email alerts
If you want Butler to send email alerts you must provide an email template file.
For some other alert destinations (Slack and Teams) Butler offers a “basic” option. A fixed format alert is then sent by Butler.
The closest thing available for emails is to use the mail log appender described here, but if you set up a log appender AND have Butler running, you might as well use the formatted email option as it provides much more flexibility than log4net’s email appender.
Rate limiting
Butler has rate limiting feature to ensure alert recipients are not spammed with too many alert emails.
The rate limit is configured (in seconds) in the main config file and can be set independently for reload-failed and reload-aborted emails.
The corresponding config settings are Butler.emailNotification.reloadTaskFailure.rateLimit, Butler.emailNotification.reloadTaskAborted.rateLimit and Butler.emailNotification.reloadTaskSuccess.rateLimit.
Rate limiting is done based on task ID + email address.
Sending test emails to verify correct settings
It can be tricky to find the correct settings to use Butler with email servers.
Butler itself uses a very generic email components to send emails, but corporate email servers may impose restrictions on from where/what servers emails will be accepted, encryption may be used together with non-standard network ports etc.
Butler offers a command line option that when used will send a simple test email to the specified email address.
This makes is very easy to test if the email settings in Butler’s config file are working or not.
When this command line option is used Butler will start normally, but also send a test email during startup.
The command line option is --test-email-address <address>.
The sender of the test email can be specified with --test-email-from-address <address>.
If the settings in the config file’s Butler.emailNotification.smtp section are valid and correct a command like this can be used: butler.exe -c ./config/production.yaml --test-email-address myname@somedomain.com. Adapt config file location and email address as needed.
The resulting email looks like this:
Sending alert emails to app owners
Butler can optionally send alert emails to the owner of apps that failed reloading/were aborted.
Note
App owner notification email can only be sent to app owners that have an email stored in their Qlik Sense user profile.
This is typically the case if the Qlik Sense user directory has been synced from a Microsoft Active Directory - but there is no guarantee this is the case.
If there is no email available for an app owner, he/she will simply not receive a notification email.
This feature is controlled by the config file properties Butler.emailNotification.reloadTaskAborted.appOwnerAlert.enable and Butler.emailNotification.reloadTaskFailure.appOwnerAlert.enable.
If set to true the app owner will be added to the send list of alert emails, in addition to the recipients specied in Butler.emailNotification.reloadTaskAborted.recipients and Butler.emailNotification.reloadTaskFailure.recipients.
The sections of the config file dealing with app owner notification emails looks like this:
appOwnerAlert:enable:true# Should app owner get notification email (assuming email address is available in Sense user directory)includeOwner:includeAll:true# true = Send notification to all app owners except those in exclude list# false = Send notification to all app owners in the include listuser:- directory:<Sense user directory>userId:<userId>- directory:<Sense user directory>userId:<userId>excludeOwner:user:- directory:<Sense user directory>userId:<userId>- directory:<Sense user directory>userId:<userId>
It works like this:
If appOwnerAlert.enable is set to false no app owner emails will be sent. If it’s set to true the rules below apply.
If appOwnerAlert.includeOwner.includeAll is set to true all app owners will get notification emails when apps the own fail/are aborted…
… except those app owners listed in the appOwnerAlert.excludeOwner.user array.
That array thus provides a way to exclude some app owners (e.g. system accounts) to receive notifcation emails.
If appOwnerAlert.includeOwner.includeAll is set to false it’s still possible to add individual app owners to the appOwnerAlert.includeOwner.user array.
Those users will then receive notification emails for apps they own.
Send alerts only for some reload tasks
Some reload tasks may be more important than others.
I.e. some tasks should generate alert emails when they fail/abort/succeed, but others not.
Butler controls which tasks to send alerts for by looking at a specific Qlik Sense custom property.
Note
The concept described below is the same for failed, aborted and successful reload tasks.
Each of these three types of tasks have their own settings in the config file.
If the config file setting Butler.emailNotification.reloadTaskFailure.alertEnableByCustomProperty.enable is set to false, all failed reload tasks will cause alert emails.
If that setting is true only some tasks will cause alert emails:
If a task has the value specified in Butler.emailNotification.reloadTaskFailure.alertEnableByCustomProperty.enabledValue set for the custom property named as specified in Butler.emailNotification.reloadTaskFailure.alertEnableByCustomProperty.customPropertyName, the alert will be sent.
If a task does not have that custom property set, no alert will be sent for that task.
A task can still cause an alert to be sent if a specific email address is specified for the task, see below for details.
Some configuration is needed to make this work:
Make changes to the config file. Specifically the three settings mentioned above needs to be reviewed and updated as needed.
Create a custom property in Sense.
The name and value of the custom property must match the one in the config file, Butler.emailNotification.reloadTaskFailure.alertEnableByCustomProperty.customPropertyName and Butler.emailNotification.reloadTaskFailure.alertEnableByCustomProperty.enabledValue.
The custom property should be available on reload tasks.
Set the custom property for reload tasks for which alert emails should be sent.
Aborted reload tasks (as compared to the failed reload tasks described above) are handled the same way, with their own settings in the config file.
In the QMC the custom property can look like this:
Send alerts to specific people, for some tasks
It’s possible to send alert emails to specific email addresses and control this on a per-task basis.
This is achieved by using a Sense custom property that contains the email addresses alerts should be sent to, for the task in question.
Note
The concept described below is the same for failed, aborted and successful reload tasks.
Each of these three types of tasks have their own settings in the config file.
These config setting Butler.emailNotification.reloadTaskFailure.alertEnableByEmailAddress.customPropertyName controls which custom property is used to store email addresses for failed reload tasks.
Email specific alert recpients is independent from the feature where alerts can be switched on/off for individual tasks (see above).
In other words: If an email address has been designated as recipient of alert emails, that address will always receive alert emails for all failed reload tasks.
Having set two different (blurred out) recipients of alert emails for a reload task:
Settings in config file
Warning
Don’t forget to create the log appender .xml files on the Sense server(s). This page describes how.
Those xml files are the foundation on top of which all Butler reload task alerts are built - without them the alerts described on this page won’t work.
---Butler:......# Qlik Sense related links used in notification messagesqlikSenseUrls:qmc:<Link to Qlik Sense QMC>hub:<Link to Qlik Sense Hub>......# Settings needed to send email notifications when for example reload tasks fail.# Reload failure notifications assume a log appender is configured in Sense AND that the UDP server in Butler is running.emailNotification:enable:falsereloadTaskSuccess:enable:false# Custom property used to control which task successes will cause alert emails to be sent# If this setting is true, alerts will not be sent for all tasks, but *only* for tasks with the CP set to the enabledValue.# If this setting is false, alerts will be sent for all failed reload tasks.alertEnableByCustomProperty:enable:falsecustomPropertyName:'Butler_SuccessAlertEnableEmail'enabledValue:'Yes'# Custom property used to say that alerts for a certain task should be sent to zero or more recipients# These alerts will be sent irrespective of the alertEnableByCustomProperty.enable setting.alertEnabledByEmailAddress:customPropertyName:'Butler_SuccessAlertSendToEmail'rateLimit:60# Min seconds between emails for a given taskID. Defaults to 5 minutes.headScriptLogLines:15tailScriptLogLines:25priority:high # high/normal/lowsubject: '✅ Qlik Sense reload success:"{{taskName}}"'
bodyFileDirectory: path/to/email_templates
htmlTemplateFile: success-reload-qseow
fromAddress: Qlik Sense (no-reply) <qliksense-noreply@ptarmiganlabs.com>
recipients:
- <Email address 1>
- <Email address 2>
reloadTaskAborted:
enable: false
appOwnerAlert:
enable: true # Should app owner get notification email (assuming email address is available in Sense user directory)
includeOwner:
includeAll: true # true = Send notification to all app owners except those in exclude list
# false = Send notification to app owners in the include list
user:
- directory: <Sense user directory>
userId: <userId>
- directory: <Sense user directory>
userId: <userId>
excludeOwner:
user:
- directory: <Sense user directory>
userId: <userId>
- directory: <Sense user directory>
userId: <userId>
# Custom property used to control which aborted tasks will cause alert emails to be sent
# If this setting is true, alerts will not be sent for all tasks, but *only* for tasks with the CP set to the enabledValue.
# If this setting is false, alerts will be sent for all aborted reload tasks.
alertEnableByCustomProperty:
enable: true
customPropertyName: 'Butler_AbortedAlertEnableEmail'enabledValue:'Yes'# Custom property used to say that alerts for a certain task should be sent to zero or more recipients# These alerts will be sent irrespective of the alertEnableByCustomProperty.enable setting.alertEnabledByEmailAddress:customPropertyName:'Butler_AbortedAlertSendToEmail'rateLimit:600# Min seconds between emails for a given taskID. Defaults to 5 minutes.headScriptLogLines:15# Number of lines from start of script to include in emailtailScriptLogLines:15# Number of lines from end of script to include in emailpriority:high # high/normal/lowsubject: 'Qlik Sense reload aborted:"{{taskName}}"' # Email subject. Can use template fields
bodyFileDirectory: path/to/email_templates # Directory where email body template files are stored
htmlTemplateFile: aborted-reload # Name of email body template file to use
fromAddress: Qlik Sense (no-reply) <qliksense-noreply@mydomain.com>
recipients: # Array of email addresses to which the notification email will be sent
- <Email address 1>
- <Email address 2>
reloadTaskFailure:
enable: false
appOwnerAlert:
enable: true # Should app owner get notification email (assuming email address is available in Sense user directory)
includeOwner:
includeAll: true # true = Send notification to all app owners except those in exclude list
# false = Send notification to app owners in the include list
user:
- directory: <Sense user directory>
userId: <userId>
- directory: <Sense user directory>
userId: <userId>
excludeOwner:
user:
- directory: <Sense user directory>
userId: <userId>
- directory: <Sense user directory>
userId: <userId>
# Custom property used to control which task failures will cause alert emails to be sent
# If this setting is true, alerts will not be sent for all tasks, but *only* for tasks with the CP set to the enabledValue.
# If this setting is false, alerts will be sent for all failed reload tasks.
alertEnableByCustomProperty:
enable: false
customPropertyName: 'Butler_FailedAlertEnableEmail'enabledValue:'Yes'# Custom property used to say that alerts for a certain task should be sent to zero or more recipients# These alerts will be sent irrespective of the alertEnableByCustomProperty.enable setting.alertEnabledByEmailAddress:customPropertyName:'Butler_FailedAlertSendToEmail'rateLimit:600# Min seconds between emails for a given taskID. Defaults to 5 minutes.headScriptLogLines:15# Number of lines from start of script to include in emailtailScriptLogLines:15# Number of lines from end of script to include in emailpriority:high # high/normal/lowsubject: 'Qlik Sense reload failed:"{{taskName}}"' # Email subject. Can use template fieldsbodyFileDirectory:path/to/email_templates # Directory where email body template files are storedhtmlTemplateFile:failed-reload # Name of email body template file to usefromAddress:Qlik Sense (no-reply) <qliksense-noreply@mydomain.com>recipients:# Array of email addresses to which the notification email will be sent- <Email address 1>- <Email address 2>......smtp:# Email server settings. See https://nodemailer.com/smtp/ for details on the meaning of these fields.host:<FQDN or IP or email server, e.g. smtp.gmail.com>port:<port on which SMTP server is listening>secure:true# true/falsetls:serverName:# If specified the serverName field will be used for TLS verification instead of the host field.ignoreTLS:falserequireTLS:truerejectUnauthorized:falseauth:enable:trueuser:<Username, email address etc>password:<your-secret-password>......udpServerConfig:enable:false# Should the UDP server responsible for receving task failure and session events be started? true/falseserverHost:<FQDN or IP (or localhost) of server where Butler is running>portTaskFailure:9998......
Templates: Configuring email appearance
Alert emails use standard HTML formatting. Inline CSS can be used (if so desired) for fine tuning the visual look of the alert email.
Butler’s process for sending alert emails is
Figure out which email body template file should be used. This is determine by two set of fields in the main config file:
For reload failure emails these config file properties are used:
Butler.emailNotification.reladTaskFailure.bodyFileDirectory and
Butler.emailNotification.reladTaskFailure.htmlTemplateFile
For aborted reload emails these config file properties are used:
Butler.emailNotification.reloadTaskAborted.bodyFileDirectory and
Butler.emailNotification.reloadTaskAborted.htmlTemplateFile
For email subjects, these config properties are used:
Butler.emailNotification.reladTaskFailure.subject and
Butler.emailNotification.reloadTaskAborted.subject
Process the body template, replacing template fields with actual values.
Process the email subject template, replacing template fields with actual values.
Send the email.
A couple of sample template files are found in the src/config/email_templates directory of the GitHub repository.
Remember
You can use template fields in email subjects too!
Template fields reference
A complete list of template fields - including descriptions - is available in the Reference section.
1.2 - Reload alerts in InfluxDB
Description of how information of how successful and failed reload tasks can be stored in InfluxDB.
What’s this?
Butler can store information about both successful and failed reload tasks in InfluxDB.
If enabled, Butler will store information about all failed reload tasks to InfluxDB.
For successful reload tasks, there are two options:
Store information about all successful reload tasks to InfluxDB.
Store information about some successful reload tasks to InfluxDB.
Which tasks to store information about is controlled using a custom property on the reload task.
Once the information about the reload task is in InfluxDB it can be used in Grafana dashboards.
This way it is possible to get a good, continuous overview of the reload activity in your Qlik Sense environment.
You can also use the information to create alerts in Grafana using it’s comprehensive alerting capabilities, including alerting to Slack, Teams, email, etc.
Please note that InflixDB must be enabled and correctly configured in the Butler config file for the below features to work.
Monitor failed reload tasks
If enabled using the Butler.influxDb.reloadTaskFailure.enable setting, Butler will store information about all failed reload tasks in InfluxDB.
The information stored includes (among other things):
The name and ID of the app that the failed reload task was reloading.
The name and ID of the reload task.
The name of the Qlik Sense node/server that the task was running on.
User who started the reload task. This will be the service account when the task was started by a schedule or via a task chain/trigger.
Execution ID of the reload. This is a unique ID that is generated by Qlik Sense for each reload task execution, it can be used to cross-reference the reload task with related entries in the Qlik Sense log files.
Last Butler.influxDb.reloadTaskFailure.tailScriptLogLines lines of the Sense log file for the reload task.
Static tags defined in the config file’s Butler.influxDb.reloadTaskFailure.tag.static section.
Dynamic app tags, i.e. Sense tags for the app being reloaded, if enabled in the config file Butler.influxDb.reloadTaskFailure.tag.dynamic.useAppTags section.
Dynamic reload task tags, i.e. Sense tags for the reload task being executed, if enabled in the config file Butler.influxDb.reloadTaskFailure.tag.dynamic.useTaskTags section.
A complete definition of all information sent to InfluxDB is available in the reference section.
Monitor successful reload tasks
Butler can monitor all reload tasks for successful completion, or only some of them.
Monitor all successful reload tasks
If enabled using the Butler.influxDb.reloadTaskSuccess.allReloadTasks.enable setting, Butler will store information about all successful reload tasks in InfluxDB.
The information stored is almost the same as for failed reload tasks, except that the Sense script log file is not included.
Monitor only some successful reload tasks
If enabled using the Butler.influxDb.reloadTaskSuccess.byCustomProperty.enable setting, Butler will store information about only some successful reload tasks in InfluxDB.
Which tasks to store information about is controlled using a custom property on the reload task.
The name of the custom property is defined in the Butler.influxDb.reloadTaskSuccess.byCustomProperty.customPropertyName setting.
The value of the custom property that will be used to indicate that the reload task should be monitored is defined in the Butler.influxDb.reloadTaskSuccess.byCustomProperty.enabledValue setting.
Static vs dynamic tags
Butler offers two kinds of tags: Static and dynamic.
Static tags are defined in the config file and are the same for all messages stored in InfluxDB.
An example of a static tag could be the name of the Qlik Sense server that Butler is running on, or whether the message related to a production or test Qlik Sense environment.
Dynamic attributes are determined at run-time when the message is stored in InfluxDB.
Settings in config file
---Butler:......influxDb:......reloadTaskFailure:enable:truetailScriptLogLines:20tag:static:# Static tags to attach to data stored in InflixDB- name:butler_instancevalue:prod-1dynamic:useAppTags:true# Should app tags be stored in InfluxDB as tags?useTaskTags:true# Should task tags be stored in InfluxDB as tags? reloadTaskSuccess:enable:trueallReloadTasks:enable:falsebyCustomProperty:enable:truecustomPropertyName:'Butler_SuccessReloadTask_InfluxDB'enabledValue:'Yes'tag:static:# Static attributes/dimensions to attach to events sent to InfluxDb# - name: event-specific-tag 1# value: abc 123dynamic:useAppTags:true# Should app tags be sent to InfluxDb as tags?useTaskTags:true# Should task tags be sent to InfluxDb as tags?......
1.3 - Reload alerts via New Relic
Description of how reload alerts can be sent to New Relic as events and log messages.
What’s this?
Butler can send two kinds of messages to New Relic:
When a scheduled or started from the QMC reload task fails.
When a scheduled or started from the QMC reload task is somehow stopped/aborted.
See the Concepts section for examples on what a New Relic alert can look like.
This page has additional info on how to set up Butler to work with New Relic.
A complete reference to the config file format is found here.
Different kinds of New Relic messages
Two kinds of messages can be sent to New Relic: Events and log messages.
The difference between them is that New Relic events are meant to be used for alerting, while New Relic log messages are meant to be used for troubleshooting.
Events are more flexible in terms of what data can be included in them, whereas log messages are just that - parts of Sense log files sent to New Relic.
Together they provide a powerful combination of alerting and troubleshooting capabilities, but they can also be enabled independently of each other.
Destination accounts
New Relic does not have very good access control capabilities for their dashboards, so if you want certain people to see only some reload alerts, and other people to see other alerts, you need to create multiple New Relic accounts.
Butler supports this scenario and can send messages to one or more New Relic accounts.
It is possible to specify per reload task which New Relic account(s) to send alerts to.
Three pieces of information is needed for each New Relic account that Butler should send messages to:
The name of the New Relic account. This is just a name that you choose, it is not used for anything other than to identify the account in Butler’s config file and in the custom properties of Qlik Sense reload tasks.
The New Relic account ID.
The New Relic insert/API key. This is basically a secret key that is used to authenticate Butler with New Relic.
Account numbers and insert keys are available in the New Relic UI, under “Account settings” > “Data sharing”.
Authentication and credentials
Butler looks for New Relic account names, account ID and API keys in two places:
The command line, using the --new-relic-account-name, --new-relic-account-id and --new-relic-api-key options.
If you have multiple New Relic accounts they should be listed in sequence, separated by space.
Account names can include spaces, but should then be enclosed in double quotes.
Example: --new-relic-account-name "First New Relic account" "Second New Relic account" --new-relic-api-key 1234567890abcdef 0987654321fedcba --new-relic-account-id 1234567 7654321
The config file, in the Butler.thirdPartyToolsCredentials.newRelic section.
Standard attributes
When sending messages to New Relic you can include “attributes”.
Attributes are key/value pairs that can be used to provide additional information about the message.
They can be added to both events and log messages.
Attributes can be used in New Relic dashboards to filter and group messages in various ways.
Static vs dynamic attributes
Butler offers two kinds of attributes: Static and dynamic.
Static attributes are defined in the config file and are the same for all messages sent to New Relic.
An example of a static attribute could be the name of the Qlik Sense server that Butler is running on, or whether the message related to a production or test Qlik Sense environment.
Dynamic attributes are determined at run-time when the message is sent to New Relic.
Examples include:
Sense tags that are assigned to the reload task that failed. Their names are qs_appTag_<tag name>
App tags of the app that failed to reload. Their names are qs_taskTag_<tag name>
Shared settings
Some settings are shared between events and log messages, these are found in the sharedSettings sections of the config file.
Values there will be used for both events and log messages, unless they are overridden in the respective events or logMessages sections of the config file.
Settings in config file
---Butler:......thirdPartyToolsCredentials:newRelic:# Array of New Relic accounts/insert keys. Any data sent to New Relic will be sent to both accounts. - accountName:First NR accountinsertApiKey:<API key 1 (with insert permissions) from New Relic> accountId:<New Relic account ID 1>- accountName:Second NR accountinsertApiKey:<API key 2 (with insert permissions) from New Relic> accountId:<New Relic account ID 2>......incidentTool:newRelic:enable:falsedestinationAccount:event:# Failed/aborted reload tasks are sent as events to these New Relic accounts- First NR account- Second NR accountlog:# Failed/aborted reload tasks are sent as log entries to these New Relic accounts- First NR account- Second NR account# New Relic uses different API URLs for different kinds of data (metrics, events, logs, ...)# There are different URLs depending on whther you have an EU or US region New Relic account.# The available URLs are listed here: https://docs.newrelic.com/docs/accounts/accounts-billing/account-setup/choose-your-data-center/url:# As of this writing the valid options are# https://insights-collector.eu01.nr-data.net# https://insights-collector.newrelic.com event:https://insights-collector.eu01.nr-data.net# Valid options are (1) EU/rest of world and 2) US)# https://log-api.eu.newrelic.com/log/v1# https://log-api.newrelic.com/log/v1 log:https://log-api.eu.newrelic.com/log/v1reloadTaskFailure:destination:event:enable:falsesendToAccount:# Which reload task failures are sent to New Relic as eventsbyCustomProperty:enable:false# Control using a task custom property which reload task failures are sent as eventscustomPropertyName:'Butler_FailedTask_Event_NewRelicAccount'always:enable:false# Controls which New Relic accounts ALL failed reload tasks are sent to (as events)account:- First NR account- Second NR accountattribute:static:# Static attributes/dimensions to attach to events sent to New Relic.- name:event-specific-attribute 1 # Examplevalue:abc 123 # Exampledynamic:useAppTags:true# Should app tags be sent to New Relic as attributes?useTaskTags:true# Should task tags be sent to New Relic as attributes?log:enable:falsetailScriptLogLines:20sendToAccount:# Which reload task failures are sent to New Relic as log entriesbyCustomProperty:enable:false# Control using a task custom property which reload task failures are sent as log entriescustomPropertyName:'Butler_FailedTask_Log_NewRelicAccount'always:enable:false# Controls which New Relic accounts ALL failed reload tasks are sent to (as logs)account:- First NR account- Second NR accountattribute:static:# Static attributes/dimensions to attach to events sent to New Relic.- name:log-specific-attribute 1 # Examplevalue:def 123 # Exampledynamic:useAppTags:true# Should app tags be sent to New Relic as attributes?useTaskTags:true# Should task tags be sent to New Relic as attributes?sharedSettings:rateLimit:15# Min seconds between events sent to New Relic for a given taskID. Defaults to 5 minutes.header:# Custom http headers- name:X-My-Header # Examplevalue:Header value 1# Exampleattribute:static:# Static attributes/dimensions to attach to events sent to New Relic.- name:service # Examplevalue:butler # Example- name:environment # Examplevalue:prod # ExamplereloadTaskAborted:destination:event:enable:falsesendToAccount:# Which reload task aborts are sent to New Relic as eventsbyCustomProperty:enable:false# Control using a task custom property which reload task aborts are sent as eventscustomPropertyName:'Butler_AbortedTask_Event_NewRelicAccount'always:enable:false# Controls which New Relic accounts ALL aborted reload tasks are sent to (as events)account:- First NR account- Second NR accountattribute:static:# Static attributes/dimensions to attach to events sent to New Relic.- name:event-specific-attribute 2 # Examplevalue:abc 123 # Exampledynamic:useAppTags:true# Should app tags be sent to New Relic as attributes?useTaskTags:true# Should task tags be sent to New Relic as attributes?log:enable:falsetailScriptLogLines:20sendToAccount:# Which reload task aborts are sent to New Relic as log entriesbyCustomProperty:enable:true# Control using a task custom property which reload task aborts are sent as log entriescustomPropertyName:'Butler_AbortedTask_Log_NewRelicAccount'always:enable:false# Controls which New Relic accounts ALL aborted reload tasks are sent to (as logs)account:- First NR account- Second NR accountattribute:static:# Static attributes/dimensions to attach to events sent to New Relic.- name:log-specific-attribute 2 # Examplevalue:def 123 # Exampledynamic:useAppTags:true# Should app tags be sent to New Relic as attributes?useTaskTags:true# Should task tags be sent to New Relic as attributes?sharedSettings:rateLimit:15# Min seconds between events sent to New Relic for a given taskID. Defaults to 5 minutes.header:# Custom http headers- name:X-My-Header # Examplevalue:Header value 2# Exampleattribute:static:# Static attributes/dimensions to attach to events sent to New Relic.- name:service # Examplevalue:butler # Example- name:environment # Examplevalue:prod # ExampleserviceMonitor:destination:event:enable:falsesendToAccount:# Windows service events are sent to these New Relic accounts- First NR account- Second NR accountattribute:static:# Static attributes/dimensions to attach to events sent to New Relic.- name:event-specific-attributevalue:abc 123dynamic:serviceHost:true# Should host where service is running be sent to New Relic as attribute?serviceName:true# Should service name be sent to New Relic as attribute?serviceDisplayName:true# Should service display name be sent to New Relic as attribute?serviceState:true# Should service state be sent to New Relic as attribute?log:enable:falsesendToAccount:# Windows service log entries are sent to these New Relic accounts- First NR account- Second NR accountattribute:static:# Static attributes/dimensions to attach to events sent to New Relic.- name:log-specific-attributevalue:def 456dynamic:serviceHost:true# Should host where service is running be sent to New Relic as attribute?serviceName:true# Should service name be sent to New Relic as attribute?serviceDisplayName:true# Should service display name be sent to New Relic as attribute?serviceState:true# Should service state be sent to New Relic as attribute?monitorServiceState:# Control whih service states are sent to New Relicrunning:enable:truestopped:enable:truesharedSettings:rateLimit:5# Min seconds between events/logs sent to New Relic for a given host+service. Defaults to 5 minutes.header:# Custom http headers- name:X-My-Header # Examplevalue:Header value 2 # Exampleattribute:static:# Static attributes/dimensions to attach to events sent to New Relic.- name:service # Examplevalue:butler # Example- name:environment # Examplevalue:prod # Example......
1.4 - Reload alerts via Slack
Description of how reload alerts can be sent as Slack messages.
What’s this?
Butler can send two kinds of alert messages via Slack:
A complete reference to the config file format is found here.
Basic vs formatted Slack alerts
Slack alerts come in two forms:
Customizable formatting using a template concept. A standard template that will fit most use cases is included with Butler. Using this option the first and last parts of the script log can be included in the message, allowing you to tell from the Slack message what caused the reload to fail.
You can also add buttons to the message that can be used to open any URL you want, or open the app that failed reloading.
A fixed, more basic format that is built into Butler. No template file needed.
Which option to go for depends on whether you want just a notification that something went wrong, or if you want as much detail as possible in the Slack message.
Sample message with custom formatting
Note
The concept described below is the same for failed and aborted reload tasks.
Each of these have their own settings in the config file.
A “reload task failed” Slack message using the custom formatting option could look like this:
Here’s how to set this up:
Create an incoming webhook in Slack, take note of its URL (you will need it in step 2 below).
Edit the Slack section of the config file, i.e. the settings in Butler.slackNotification.reloadTaskFailure.
The messageType property should be set to formatted.
The basicMsgTemplate property is not used with formatted messages and can thus be left empty,
Edit the template file if/as needed, the file is specified in Butler.slackNotification.reloadTaskFailure.templateFile. It uses the Handlebars templating engine, to which Butler provides template fields with actual values.
Sample template files are included in the release Zip file, and are also available in the GitHub repository’s src/config/slack_templates directory.
Restart Butler if it’s already running.
Sample message with basic formatting
Note
The concept described below is the same for failed and aborted reload tasks.
Each of these have their own settings in the config file.
A “reload task failed” Slack message with basic formatting could look like this:
To set it up:
Create an incoming webhook in Slack if you don’t already have one, take note of its URL (you will need it in step 2 below).
Edit the Slack section of the config file, i.e. in Butler.slackNotification.reloadTaskFailure.
The messageType property should be set to basic.
The basicMsgTemplate property is the message that will be sent via Slack. Template fields can be used.
Restart Butler if it’s already running.
Customizing Slack messages
When using the formatted Slack alerts you have full freedom to create the alert you need.
Behind the scenes Slack messages are constructed from blocks defined in a JSON object. Each block can then contain either plain text, Markdown, images, buttons etc.
The Slack documentation is the best place for learning how to customize messages.
When it comes to Butler, it uses the Handlebars templating engine to render the template files into Slack JSON objects that are then sent to Slack via their APIs.
A few things to keep in mind when creating custom Slack messages:
The handlebars syntax itself must be correct. If incorrect no Slack JSON object will be created. And no Slack messages sent.
The handlebars template must result in a JSON object that adheres to Slack’s API specifications.
If the JSON syntax is somehow invaid the Slack API will return errors and no messages sent. JSON can be pretty sensitive to details, there should for example not be any trailing commas in properly formatted JSON objects.
Block Kit Builder: Great sandbox wtih readily available examples of different message layouts, syntax and more. Note: You must be logged into your Slack account to use this tool.
How it works
Warning
Don’t forget to create the log appender .xml files on the Sense server(s).
Those xml files are the foundation on top of which all Butler alerts are built - without them the alerts described on this page won’t work.
The concept is the same for for all alert types, see the email alerts for details.
Settings in config file
---Butler:......# Settings for notifications and messages sent to SlackslackNotification:enable:falserestMessage:webhookURL:<web hook URL from Slack> # Webhook to use when sending basic Slack messages via Butler's REST API reloadTaskFailure:# Reload task failed in QSEoWenable:falsewebhookURL:<web hook URL from Slack>channel:sense-task-failure # Slack channel to which task failure notifications are sentmessageType:formatted # formatted / basic. Formatted means that template file below will be used to create the message.basicMsgTemplate: 'Qlik Sense reload failed:"{{taskName}}"' # Only needed if message type = basic
rateLimit: 300 # Min seconds between emails for a given taskID. Defaults to 5 minutes.
headScriptLogLines: 10
tailScriptLogLines: 10
templateFile: /path/to/slack/template/directory/failed-reload-qseow.handlebars
fromUser: Qlik Sense
iconEmoji: ':ghost:'reloadTaskAborted:# Reload task aborted in QSEoWenable:falsewebhookURL:<web hook URL from Slack>channel:sense-task-aborted # Slack channel to which task stopped notifications are sentmessageType:formatted # formatted / basic. Formatted means that template file below will be used to create the message.basicMsgTemplate: 'Qlik Sense reload aborted:"{{taskName}}"' # Only needed if message type = basic
rateLimit: 300 # Min seconds between emails for a given taskID. Defaults to 5 minutes.
headScriptLogLines: 10
tailScriptLogLines: 10
templateFile: /path/to/slack/template/directory/aborted-reload-qseow.handlebars
fromUser: Qlik Sense
iconEmoji: ':ghost:'......udpServerConfig:enable:false# Should the UDP server responsible for receving task failure and session events be started? true/falseserverHost:<FQDN or IP (or localhost) of server where Butler is running>portTaskFailure:9998......
1.5 - Reload alerts via Microsoft Teams
Description of how reload alerts can be sent as Microsoft Teams messages.
What’s this?
Butler can send two kinds of alert messages via Teams:
A complete reference to the config file format is found here.
Basic vs formatted Teams alerts
Teams alerts come in two forms:
Customizable formatting using a template concept. A standard template that will fit most use cases is included with Butler. With this option the first and last parts of the script log can be included in the message, allowing you to tell from the Teams message what caused the reload to fail.
You can also add buttons to the message that can be used to open any URL you want, or open the app that failed reloading.
A fixed, more basic format that is built into Butler. No template file needed.
Which option to go for depends on whether you want just a notification that something went wrong, or if you want as much detail as possible in the Teams message.
Sample message with custom formatting
Note
The concept described below is the same for failed and aborted reload tasks.
Each of these have their own settings in the config file.
A “reload task failed” Teams message using the custom formatting option could look like this:
Here’s how to set it up:
Create a workflow in Teams, take note of its URL (you will need it in step 2 below). More information on how to create a workflow further down this page.
Edit the Teams section of the config file, i.e. the settings in Butler.teamsNotification.reloadTaskFailure.
The messageType property should be set to formatted.
The basicMsgTemplate property is not used with formatted messages and can thus be left empty,
Edit the template file if/as needed, the file is specified in Butler.teamsNotification.reloadTaskFailure.templateFile.It uses the Handlebars templating engine, to which Butler provides template fields with actual values.
Sample template files are included in the release Zip file, and are also available in the GitHub repository’s src/config/teams_templates directory.
Restart Butler if it’s already running.
Sample message with basic formatting
Note
The concept described below is the same for failed and aborted reload tasks.
Each of these have their own settings in the config file.
A “reload task failed” Teams message with basic formatting could look like this:
To set it up:
Create an incoming webhook in Teams if you don’t already have one, take note of its URL (you will need it in step 2 below).
Edit the Teams section of the config file i.e. the settings in Butler.teamsNotification.reloadTaskFailure and/or Butler.teamsNotification.reloadTaskAborted sections of the confi file.
The messageType property should be set to basic.
The basicMsgTemplate property is the message that will be sent via Teams. Template fields can be used.
Restart Butler if it’s already running.
Customizing Teams messages
When using the formatted Teams alerts you have full freedom to create the alert you need.
Behind the scenes Teams messages are constructed as “Adaptive Cards”, which is standardised JSON format that Teams understands.
More information on Adaptive Cards can be found here, here and here.
When it comes to Butler, it uses the Handlebars templating engine to render a template file into an adaptive card JSON object that is then sent to the workflow webhook.
A few things to keep in mind when creating custom Teams messages:
The handlebars syntax itself must be correct. If incorrect no Teams JSON object will be created. And no Teams message sent.
The handlebars template must result in a JSON object that adheres to Teams’s specifications for JSON payloads.
If the JSON syntax is somehow invaid the Teams API will return errors and no messages sent. JSON can be pretty sensitive to details, there should for example not be any trailing commas in properly formatted JSON objects.
How it works
Warning
Don’t forget to create the log appender .xml files on the Sense server(s).
---Butler:......# Settings for notifications and messages sent to MS TeamsteamsNotification:enable:falsereloadTaskFailure:enable:falsewebhookURL:<web hook URL from MS Teams>messageType:formatted # formatted / basic. Formatted means that template file below will be used to create the message.basicMsgTemplate: 'Qlik Sense reload failed:"{{taskName}}"' # Only needed if message type = basic
rateLimit: 300 # Min seconds between emails for a given taskID. Defaults to 5 minutes.
headScriptLogLines: 10
tailScriptLogLines: 10
templateFile: /path/to/teams/template/directory/failed-reload-qseow.handlebars
reloadTaskAborted:
enable: false
webhookURL: <web hook URL from MS Teams>
messageType: formatted # formatted / basic. Formatted means that template file below will be used to create the message.
basicMsgTemplate: 'Qlik Sense reload aborted:"{{taskName}}"' # Only needed if message type = basicrateLimit:300# Min seconds between emails for a given taskID. Defaults to 5 minutes.headScriptLogLines:10tailScriptLogLines:10templateFile:/path/to/teams/template/directory/aborted-reload-qseow.handlebars......udpServerConfig:enable:false# Should the UDP server responsible for receving task failure and session events be started? true/falseserverHost:<FQDN or IP (or localhost) of server where Butler is running>portTaskFailure:9998......
1.6 - Reload alerts via MQTT
Description of how reload alerts can be sent as MQTT messages.
What’s this?
Butler can send two kinds of alert messages as MQTT messages:
When a scheduled, running reload task fails.
When a scheduled, running reload task is somehow stopped.
How it works
Basic message
The MQTT message will be sent on the MQTT topic defined in the config file property Butler.mqttConfig.taskAbortedTopic or Butler.mqttConfig.taskFailureTopic, depending on the event type.
The task name will be sent in the message body.
The basic message looks like this when viewed in the MQTTLens app:
Complete message
Optionally a larger, more complete message is also sent if Butler.mqttConfig.taskFailureSendFull or Butler.mqttConfig.taskFailureSendFull are set to true.
This message contains a stringified JSON of all available information about the failed/aborted task.
The message is sent on the Butler.mqttConfig.taskFailureFullTopic or Butler.mqttConfig.taskAbortedFullTopic topics.
That message can look like this:
Remember
Don’t forget to create the log appender .xml files on the Sense server(s). This page describes how.
Those xml files are the foundation on top of which all Butler alerts are built - without them the alerts described on this page won’t work.
The concept is more or less the same as for alert emails.
Settings in config file
---Butler:......mqttConfig:enable:false# Should Qlik Sense events be forwarded as MQTT messages?brokerHost:<FQDN or IP of MQTT server>brokerPort:1883azureEventGrid:enable:false# If set to true, Butler will connect to an Azure Event Grid MQTT Broker, using brokerHost and brokerPort above clientId:<client ID>clientCertFile:<path to client certificate file>clientKeyFile:<path to client key file>taskFailureSendFull:truetaskAbortedSendFull:truesubscriptionRootTopic:qliksense/# # Topic that Butler will subscribe totaskStartTopic:qliksense/start_task # Topic for incoming messages used to start Sense tasks. Should be subtopic to subscriptionRootTopictaskFailureTopic:qliksense/task_failuretaskFailureFullTopic:qliksense/task_failure_fulltaskFailureServerStatusTopic:qliksense/butler/task_failure_servertaskAbortedTopic:qliksense/task_abortedtaskAbortedFullTopic:qliksense/task_aborted_full......udpServerConfig:enable:false# Should the UDP server responsible for receving task failure and session events be started? true/falseserverHost:<FQDN or IP (or localhost) of server where Butler is running>portTaskFailure:9998......
1.7 - Reload alerts via outgoing webhooks
Description of how reload alerts can be sent via outgoing webhooks.
What’s this?
Butler can send two kinds of alert messages as outgoing webhooks:
When a scheduled, running reload task fails.
When a scheduled, running reload task is somehow stopped/aborted.
How it works
Outgoing webhooks is a concept where Butler will do a GET, POST or PUT HTTP call to a specific URL when a task fails or is aborted/stopped.
The use case is to interface with currently unkown third party systems in a generic way.
Both http and https calls are supported, including the use of self-signed certificates and untrusted certificates.
As the call will include information about the failed/aborted task, the typical (and arguably most correct) way of doing this would be via a PUT call.
But some systems only handle GET calls - and Butler should still be able to notify them using webhooks.
The chosen solution is to offer full flexibility for outgoing webhooks and support both GET, PUT and POST calls.
Webhook notifications can be turned off all together with the Butler.webhookNotification.enable property in the config file.
If that property is true both task fail and abort webhooks are enabled.
If you don’t need any outgoing webhooks you should keep the Butler.webhookNotification.reloadTaskFailure.webhooks and Butler.webhookNotification.reloadTaskAborted.webhooks arrays empty.
There are also rate limiting properties that are used to ensure that webhooks are not sent too often.
Certificates and https
Outgoing webhooks can use http or https.
If https is used and the server being called uses a publicly trusted certificate, no additional configuration is needed.
If the server uses a self-signed certificate, the corresponding root CA certificate must be provided to Butler in order to avoid certificate validation errors.
Each webhook has its own certificate configuration, so Butler can be integrated with many systems, each using their own publicly verified or self-signed certificates - or just plain http without any certificates at all.
The certificate configuration is done in the Butler config file and looks like this for each webhook:
...cert:enable:true# Set to true to use a custom CA certificate when calling the webhookURLrejectUnauthorized:true# Set to false to ignore warnings/errors caused by self-signed certificates used on the webhooks server.certCA:/path/to/ca-certificate.pem # Path to the CA certificate file...
If ...cert.enable is set to true Butler will use the certificate specified in ...cert.certCA when calling the webhook.
If ...cert.rejectUnauthorized is set to false Butler will ignore warnings/errors caused by self-signed certificates being used on the webhook server.
Data included in outgoing webhooks
This information is included in all outgoing webhook calls:
Field
Description
event
Type of event, for example Qlik Sense reload failed.
hostName
Name of server where the event took place.
user
User directory/userId of user causing the event. For task failures this will be the user account used to do the reload. For aborts it will be the user stopping/aborting the task.
taskName
Task name
taskId
Task ID
appName
App name
appId
App ID
logTimeStamp
Timestamp entry in the Qlik Sense log files when the event took place.
logLevel
Log level used in the Qlik Sense log file in which the event was detected by the log appender.
executionId
Execution ID of the failed/aborted task.
logMessage
Message in Qlik Sense log file that triggered the event.
GET call
When doing GET calls all the data fields will be passed as search parameters in the URL.
For example, a failed task GET call to a remote URL could look like this:
The received/remote system can then unpack the URL parameters and use them as needed.
PUT and POST calls
PUT and POST calls work the same when it comes to Butler’s outgoing webhooks:
A stringified JSON is created based on the event’s data fields.
The string is sent in the POST/PUT call’s body.
The same event as above looks like this:
{"event":"Qlik Sense reload failed","hostName":"pro2-win1","user":"LAB\\goran","taskName":"Manually triggered reload of Test failing reloads 2","taskId":"dec2a02a-1680-44ef-8dc2-e2bfb180af87","appName":"Test failing reloads 2","appId":"e7af59a0-c243-480d-9571-08727551a66f","logTimeStamp":"2021-02-16 09:24:59,099","logLevel":"INFO","executionId":"14a81bf5-f81c-4047-b1a1-193b0920de28","logMessage":"Max retries reached"}
Settings in config file
---Butler:......# Settings for notifications and messages sent using outgoing webhookswebhookNotification:enable:falsereloadTaskFailure:rateLimit:300# Min seconds between outgoing webhook calls for a given taskID. Defaults to 5 minutes.webhooks:- description:'This outgoing webhook is used to...'# Informational onlywebhookURL:http://host.my.domain:port/some/path # Outgoing webhook that Butler will callhttpMethod:POST # GET/POST/PUTcert:enable:false# Set to true to use a custom CA certificate when calling the webhookURLrejectUnauthorized:true# Set to false to ignore warnings/errors caused by self-signed certificates used on the webhooks server.certCA:/path/to/ca-certificate.pem # Path to the CA certificate file- description:'This outgoing webhook is used to...'# Informational onlywebhookURL:http://host.my.domain:port/some/path # Outgoing webhook that Butler will callhttpMethod:PUT # GET/POST/PUTcert:enable:false# Set to true to use a custom CA certificate when calling the webhookURLrejectUnauthorized:true# Set to false to ignore warnings/errors caused by self-signed certificates used on the webhooks server.certCA:/path/to/ca-certificate.pem # Path to the CA certificate file- description:'This outgoing webhook is used to...'# Informational onlywebhookURL:http://host.my.domain:port/some/path # Outgoing webhook that Butler will callhttpMethod:GET # GET/POST/PUTcert:enable:false# Set to true to use a custom CA certificate when calling the webhookURLrejectUnauthorized:true# Set to false to ignore warnings/errors caused by self-signed certificates used on the webhooks server.certCA:/path/to/ca-certificate.pem # Path to the CA certificate filereloadTaskAborted:rateLimit:300# Min seconds between outgoing webhook calls for a given taskID. Defaults to 5 minutes.webhooks:- description:'This outgoing webhook is used to...'# Informational onlywebhookURL:http://host.my.domain:port/some/path # Outgoing webhook that Butler will callhttpMethod:PUT # GET/POST/PUTcert:enable:false# Set to true to use a custom CA certificate when calling the webhookURLrejectUnauthorized:true# Set to false to ignore warnings/errors caused by self-signed certificates used on the webhooks server.certCA:/path/to/ca-certificate.pem # Path to the CA certificate file- description:'This outgoing webhook is used to...'# Informational onlywebhookURL:http://host.my.domain:port/some/path # Outgoing webhook that Butler will callhttpMethod:POST # GET/POST/PUTcert:enable:false# Set to true to use a custom CA certificate when calling the webhookURLrejectUnauthorized:true# Set to false to ignore warnings/errors caused by self-signed certificates used on the webhooks server.certCA:/path/to/ca-certificate.pem # Path to the CA certificate file- description:'This outgoing webhook is used to...'# Informational onlywebhookURL:http://host.my.domain:port/some/path # Outgoing webhook that Butler will callhttpMethod:GET # GET/POST/PUTcert:enable:false# Set to true to use a custom CA certificate when calling the webhookURLrejectUnauthorized:true# Set to false to ignore warnings/errors caused by self-signed certificates used on the webhooks server.certCA:/path/to/ca-certificate.pem # Path to the CA certificate file......
2 - Reload alerts for Qlik Sense Cloud
Butler offers a lot of flexibility when it comes to alerts when reloads fai in Qlik Sense Cloud.
Learn how to set up the desired features, the alert layout, formatting and more.
Alert types
These alert types are available:
Reload task failure. Send alerts when reload tasks fail, no matter if they were started on schedule or manually from the QMC.
Reload task aborted. Send alerts when reload tasks are manually aborted in the QMC.
Reload task success. Send alerts when reload tasks complete successfully.
Alert destinations and options
Alerts can be sent to these destinations, with different options available for each destination.
Each destination can be individually enabled/disabled in the config file.
The failed reload’s script log is available in InfluxDb.
New Relic
✅
✅
✅
✅
The failed reload’s script log is available in New Relic.
Signl4
✅
✅
✅
✅
Alerts are presented in Signl4’s own format in their mobile app.
Slack
✅
✅
✅
✅
MS Teams
✅
✅
✅
✅
Outgoing webhook
✅
✅
Formatting is not relevant for webhooks
MQTT
✅
✅
Formatting is not relevant for MQTT messages
How it works
In order for Butler initiated alerts to become a reality, Butler must somehow be notified that the event of interest (for example a failed reload task) has occurred.
This is achieved by adding a log appender to Qlik Sense Enterprise on Windows.
Log appenders offer a way to hook into Qlik Sense’s logging subsystem, which is called log4net.
By adding a carefully crafted .xml file in the right location on the Sense server(s), you can make Sense notify Butler by means of UDP messages when the events of interest occur. Conceptually it looks like this:
So what happens when a scheduled reload task fails?
Let’s look at the steps:
A reload task is started by the Sense scheduler, either on a time schedule, as a result of some other task(s) finishing or manually by a user in the QMC or from the Hub.
When the task’s state changes, entries are written to the Sense scheduler’s log files using log4net (which is built into Qlik Sense). If the filter defined in the log appender (= the .xml file on the Sense server) matches the log entry at hand, the associated action in the log appender will be carried out.
Log appenders can do all kinds of things, everything from writing custom log files, sending basic emails, writing to databases and much more.
Here we’re interested in the log appender sending a UDP message from Qlik Sense to Butler.
The log appender provided as part of Butler will make log4net send a UDP message to Butler, including various info (reload task ID, timestamp, host name etc) about the reload task that just failed or was stopped/aborted.
Butler will look at the incoming event and determine what it is about.
For example: Is the event about a reload task failure, a reload that has been aborted/stopped, or something else?
Butler thus first works as a dispatcher. In a second step, after the initial dispatch, the event is sent to the relevant handler function within Butler.
Response times are usually very good - Butler will typically get the UDP message within a few seconds after (for example) the reload failing, with alerts going out shortly thereafter.
Warning
The log appenders that catch failed and aborted reloads in the Qlik Sense engine and scheduler must be set up on all Qlik Sense servers where reloads are happening for this feature to work.
Failing to do so will result in Butler not being notified about some reload failures/aborted reloads.
Tip
The concept above is the same also for aborted and successful reload tasks.
Adding a log appender
This is possibly the trickiest part to get right when it comes to setting up log4net based alerts.
Still, if you start from the sample .xml file provided in the Butler repository on GitHub it’s not too hard.
Those sample .xml files are also included in the release Zip files available on the Butler releases page.
The steps are:
In this case you want to be notified when certain events occur in the scheduler log files.
This is important: Qlik Sense Enterprise on Windows consists of many different subsystems (engine, proxy, scheduler, printing etc) - here we’re interested in log events from the scheduler subsystem.
Add a file LocalLogConfig.xml in the C:\ProgramData\Qlik\Sense\Scheduler folder on the Sense server whose scheduler you want to get events from. If you have multiple Sense servers with schedulers running on them, the .xml file should be deployed on each server (assuming you want events from all the servers).
The contents of LocalLogConfig.xml will determine what events are forwarded to Butler, or what other actions will be taken by log4net. See below for examples.
Sense will eventually detect and load the new xml file, but it might take a while (minutes). Restarting the Qlik Sense Scheduler Windows service will make the changes take effect immediately.
Forwarding reload task events to Butler
Here’s the XML that should go into C:\ProgramData\Qlik\Sense\Scheduler\LocalLogConfig.xml to enable the various kinds of Butler task reload alerts.
The remoteAddress property should be set to the host name or IP where Butler is running.
The remotePort property should match the port number specified in Butler’s config file. Note that Butler uses different ports for task related and user activity related events.
The first appender looks for the text “Max retries reached” in the System.Scheduler.Scheduler.Master.Task.TaskSession log stream. That log entry will be created when a reload task has failed and also carried out all its retries. Once the search string is found a UDP message will be sent to port 9998 on IP 10.11.12.13.
The second appender looks for “Execution State Change to Aborting” in the System.Scheduler.Scheduler.Master.Task.TaskSession log stream. That log entry occurs when a user stops a running reload from the QMC’s task view, or using the Sense APIs. When the search string is found a UDP message is once again sent to 10.11.12.13:9998, but with a different messsage (as specified in the conversionpattern property of the appender).
The third appender looks for “Reload complete” in the System.Scheduler.Scheduler.Slave.Tasks.ReloadTask log stream.
That log entry occurs when a reload task has completed successfully.
Here is an XML file that would forward log events as UDP messages to Butler:
<?xml version="1.0" encoding="UTF-8"?><configuration><!-- Appender for detecting reload task failures. Only the last of potentially several retries is reported --><appendername="TaskFailureLogger"type="log4net.Appender.UdpAppender"><filtertype="log4net.Filter.StringMatchFilter"><paramname="stringToMatch"value="Max retries reached"/></filter><filtertype="log4net.Filter.DenyAllFilter"/><paramname="remoteAddress"value="<IP of server where Butler is running>"/><paramname="remotePort"value="9998"/><paramname="encoding"value="utf-8"/><layouttype="log4net.Layout.PatternLayout"><converter><paramname="name"value="hostname"/><paramname="type"value="Qlik.Sense.Logging.log4net.Layout.Pattern.HostNamePatternConverter"/></converter><paramname="conversionpattern"value="/scheduler-reload-failed/;%hostname;%property{TaskName};%property{AppName};%property{User};%property{TaskId};%property{AppId};%date;%level;%property{ExecutionId};%message"/></layout></appender><!-- Appender for detecting aborted reloads --><appendername="AbortedReloadTaskLogger"type="log4net.Appender.UdpAppender"><filtertype="log4net.Filter.StringMatchFilter"><paramname="stringToMatch"value="Execution State Change to Aborting"/></filter><filtertype="log4net.Filter.DenyAllFilter"/><paramname="remoteAddress"value="<IP of server where Butler is running>"/><paramname="remotePort"value="9998"/><paramname="encoding"value="utf-8"/><layouttype="log4net.Layout.PatternLayout"><converter><paramname="name"value="hostname"/><paramname="type"value="Qlik.Sense.Logging.log4net.Layout.Pattern.HostNamePatternConverter"/></converter><paramname="conversionpattern"value="/scheduler-reload-aborted/;%hostname;%property{TaskName};%property{AppName};%property{User};%property{TaskId};%property{AppId};%date;%level;%property{ExecutionId};%message"/></layout></appender><!-- Appender for detecting successful reload tasks --><appendername="ReloadTaskSuccessLogger"type="log4net.Appender.UdpAppender"><filtertype="log4net.Filter.StringMatchFilter"><paramname="stringToMatch"value="Execution State Change to FinishedSuccess"/></filter><filtertype="log4net.Filter.DenyAllFilter"/><paramname="remoteAddress"value="<IP of server where Butler is running>"/><paramname="remotePort"value="9998"/><paramname="encoding"value="utf-8"/><layouttype="log4net.Layout.PatternLayout"><converter><paramname="name"value="hostname"/><paramname="type"value="Qlik.Sense.Logging.log4net.Layout.Pattern.HostNamePatternConverter"/></converter><paramname="conversionpattern"value="/scheduler-reloadtask-success/;%hostname;%property{TaskName};%property{AppName};%property{User};%property{TaskId};%property{AppId};%date;%level;%property{ExecutionId};%message"/></layout></appender><!-- Send message to Butler on task failure --><!-- Send message to Butler on task abort --><!-- Send message to Butler on reload task success --><loggername="System.Scheduler.Scheduler.Master.Task.TaskSession"><appender-refref="TaskFailureLogger"/><appender-refref="AbortedReloadTaskLogger"/><appender-refref="ReloadTaskSuccessLogger"/></logger></configuration>
The above configuration is enough to support all task reload alerts currently supported by Butler.
Sending basic alert emails from Qlik Sense/log4net
If you are happy with the more basic/limited reload-failed alert emails provided by log4net, you can add a SMTP appender like this (the example below is for sending emails using Google GMail, customise as needed).
Note
If sending alert emails from Log4Net you will not get any of the nice formatting, script logs or other features that Butler provides in its alerts.
The email will instead just tell you that a task failed, and include some basic information about the task (task name, specifically).
<?xml version="1.0"?><configuration><!-- Mail appender--><appendername="MailAppender"type="log4net.Appender.SmtpAppender"><filtertype="log4net.Filter.StringMatchFilter"><paramname="stringToMatch"value="Message from ReloadProvider"/></filter><filtertype="log4net.Filter.DenyAllFilter"/><evaluatortype="log4net.Core.LevelEvaluator"><paramname="threshold"value="ERROR"/></evaluator><paramname="to"value="<email address to send failed task notification emails to>"/><paramname="from"value="<sender email address used in notification emails>"/><paramname="subject"value="Qlik Sense failed task (server <servername>)"/><paramname="smtpHost"value="smtp.gmail.com"/><paramname="port"value="587"/><paramname="EnableSsl"value="true"/><paramname="Authentication"value="Basic"/><paramname="username"value="<Gmail username>"/><paramname="password"value="<Gmail password>"/><paramname="bufferSize"value="0"/><!-- Set this to 0 to make sure an email is sent on every error --><paramname="lossy"value="true"/><layouttype="log4net.Layout.PatternLayout"><paramname="conversionPattern"value="%newline%date %-5level %newline%property{TaskName}%newline%property{AppName}%newline%message%newline%newline%newline"/></layout></appender><!--Send mail on task failure--><loggername="System.Scheduler.Scheduler.Slave.Tasks.ReloadTask"><appender-refref="MailAppender"/></logger></configuration>
References
Qlik’s documenation around log appenders and how to hook into the Sense logs is somewhat brief, but does provide a starting point if you want to dive deeper into this topic.
The main log4net documentation (log4net is the logging framework used by Qlik Sense Enterprise) can also be useful.
These links describe how emails can be sent from the log4net logging framework itself, directly to the recipient. Butler includes sameple XML files for this use case too, but Butler takes things further by using the data in the Sense logs to pull in more data around the failed or stopped reload.
In other words - Butler’s alert emails are significantly more flexible and contain information (such as script logs) that are not availble using purely log4net.
2.1 - Reload alerts sent as emails
Description of the various kinds of alert emails Butler can send.
What’s this?
Butler can send two kinds of alert emails:
When a reload task fails during execution.
When a running reload task is somehow stopped/aborted.
When a reload task completes successfully.
Butler has a de-duplication feature that ensure each email address that has qualified for an alert email only gets ONE email per alert.
See the Concepts section for additional details and sample alert emails.
Basic vs formatted email alerts
If you want Butler to send email alerts you must provide an email template file.
For some other alert destinations (Slack and Teams) Butler offers a “basic” option. A fixed format alert is then sent by Butler.
The closest thing available for emails is to use the mail log appender described here, but if you set up a log appender AND have Butler running, you might as well use the formatted email option as it provides much more flexibility than log4net’s email appender.
Rate limiting
Butler has rate limiting feature to ensure alert recipients are not spammed with too many alert emails.
The rate limit is configured (in seconds) in the main config file and can be set independently for reload-failed and reload-aborted emails.
The corresponding config settings are Butler.emailNotification.reloadTaskFailure.rateLimit, Butler.emailNotification.reloadTaskAborted.rateLimit and Butler.emailNotification.reloadTaskSuccess.rateLimit.
Rate limiting is done based on task ID + email address.
Sending test emails to verify correct settings
It can be tricky to find the correct settings to use Butler with email servers.
Butler itself uses a very generic email components to send emails, but corporate email servers may impose restrictions on from where/what servers emails will be accepted, encryption may be used together with non-standard network ports etc.
Butler offers a command line option that when used will send a simple test email to the specified email address.
This makes is very easy to test if the email settings in Butler’s config file are working or not.
When this command line option is used Butler will start normally, but also send a test email during startup.
The command line option is --test-email-address <address>.
The sender of the test email can be specified with --test-email-from-address <address>.
If the settings in the config file’s Butler.emailNotification.smtp section are valid and correct a command like this can be used: butler.exe -c ./config/production.yaml --test-email-address myname@somedomain.com. Adapt config file location and email address as needed.
The resulting email looks like this:
Sending alert emails to app owners
Butler can optionally send alert emails to the owner of apps that failed reloading/were aborted.
Note
App owner notification email can only be sent to app owners that have an email stored in their Qlik Sense user profile.
This is typically the case if the Qlik Sense user directory has been synced from a Microsoft Active Directory - but there is no guarantee this is the case.
If there is no email available for an app owner, he/she will simply not receive a notification email.
This feature is controlled by the config file properties Butler.emailNotification.reloadTaskAborted.appOwnerAlert.enable and Butler.emailNotification.reloadTaskFailure.appOwnerAlert.enable.
If set to true the app owner will be added to the send list of alert emails, in addition to the recipients specied in Butler.emailNotification.reloadTaskAborted.recipients and Butler.emailNotification.reloadTaskFailure.recipients.
The sections of the config file dealing with app owner notification emails looks like this:
appOwnerAlert:enable:true# Should app owner get notification email (assuming email address is available in Sense user directory)includeOwner:includeAll:true# true = Send notification to all app owners except those in exclude list# false = Send notification to all app owners in the include listuser:- directory:<Sense user directory>userId:<userId>- directory:<Sense user directory>userId:<userId>excludeOwner:user:- directory:<Sense user directory>userId:<userId>- directory:<Sense user directory>userId:<userId>
It works like this:
If appOwnerAlert.enable is set to false no app owner emails will be sent. If it’s set to true the rules below apply.
If appOwnerAlert.includeOwner.includeAll is set to true all app owners will get notification emails when apps the own fail/are aborted…
… except those app owners listed in the appOwnerAlert.excludeOwner.user array.
That array thus provides a way to exclude some app owners (e.g. system accounts) to receive notifcation emails.
If appOwnerAlert.includeOwner.includeAll is set to false it’s still possible to add individual app owners to the appOwnerAlert.includeOwner.user array.
Those users will then receive notification emails for apps they own.
Send alerts only for some reload tasks
Some reload tasks may be more important than others.
I.e. some tasks should generate alert emails when they fail/abort/succeed, but others not.
Butler controls which tasks to send alerts for by looking at a specific Qlik Sense custom property.
Note
The concept described below is the same for failed, aborted and successful reload tasks.
Each of these three types of tasks have their own settings in the config file.
If the config file setting Butler.emailNotification.reloadTaskFailure.alertEnableByCustomProperty.enable is set to false, all failed reload tasks will cause alert emails.
If that setting is true only some tasks will cause alert emails:
If a task has the value specified in Butler.emailNotification.reloadTaskFailure.alertEnableByCustomProperty.enabledValue set for the custom property named as specified in Butler.emailNotification.reloadTaskFailure.alertEnableByCustomProperty.customPropertyName, the alert will be sent.
If a task does not have that custom property set, no alert will be sent for that task.
A task can still cause an alert to be sent if a specific email address is specified for the task, see below for details.
Some configuration is needed to make this work:
Make changes to the config file. Specifically the three settings mentioned above needs to be reviewed and updated as needed.
Create a custom property in Sense.
The name and value of the custom property must match the one in the config file, Butler.emailNotification.reloadTaskFailure.alertEnableByCustomProperty.customPropertyName and Butler.emailNotification.reloadTaskFailure.alertEnableByCustomProperty.enabledValue.
The custom property should be available on reload tasks.
Set the custom property for reload tasks for which alert emails should be sent.
Aborted reload tasks (as compared to the failed reload tasks described above) are handled the same way, with their own settings in the config file.
In the QMC the custom property can look like this:
Send alerts to specific people, for some tasks
It’s possible to send alert emails to specific email addresses and control this on a per-task basis.
This is achieved by using a Sense custom property that contains the email addresses alerts should be sent to, for the task in question.
Note
The concept described below is the same for failed, aborted and successful reload tasks.
Each of these three types of tasks have their own settings in the config file.
These config setting Butler.emailNotification.reloadTaskFailure.alertEnableByEmailAddress.customPropertyName controls which custom property is used to store email addresses for failed reload tasks.
Email specific alert recpients is independent from the feature where alerts can be switched on/off for individual tasks (see above).
In other words: If an email address has been designated as recipient of alert emails, that address will always receive alert emails for all failed reload tasks.
Having set two different (blurred out) recipients of alert emails for a reload task:
Settings in config file
Warning
Don’t forget to create the log appender .xml files on the Sense server(s). This page describes how.
Those xml files are the foundation on top of which all Butler reload task alerts are built - without them the alerts described on this page won’t work.
---Butler:......# Qlik Sense related links used in notification messagesqlikSenseUrls:qmc:<Link to Qlik Sense QMC>hub:<Link to Qlik Sense Hub>......# Settings needed to send email notifications when for example reload tasks fail.# Reload failure notifications assume a log appender is configured in Sense AND that the UDP server in Butler is running.emailNotification:enable:falsereloadTaskSuccess:enable:false# Custom property used to control which task successes will cause alert emails to be sent# If this setting is true, alerts will not be sent for all tasks, but *only* for tasks with the CP set to the enabledValue.# If this setting is false, alerts will be sent for all failed reload tasks.alertEnableByCustomProperty:enable:falsecustomPropertyName:'Butler_SuccessAlertEnableEmail'enabledValue:'Yes'# Custom property used to say that alerts for a certain task should be sent to zero or more recipients# These alerts will be sent irrespective of the alertEnableByCustomProperty.enable setting.alertEnabledByEmailAddress:customPropertyName:'Butler_SuccessAlertSendToEmail'rateLimit:60# Min seconds between emails for a given taskID. Defaults to 5 minutes.headScriptLogLines:15tailScriptLogLines:25priority:high # high/normal/lowsubject: '✅ Qlik Sense reload success:"{{taskName}}"'
bodyFileDirectory: path/to/email_templates
htmlTemplateFile: success-reload-qseow
fromAddress: Qlik Sense (no-reply) <qliksense-noreply@ptarmiganlabs.com>
recipients:
- <Email address 1>
- <Email address 2>
reloadTaskAborted:
enable: false
appOwnerAlert:
enable: true # Should app owner get notification email (assuming email address is available in Sense user directory)
includeOwner:
includeAll: true # true = Send notification to all app owners except those in exclude list
# false = Send notification to app owners in the include list
user:
- directory: <Sense user directory>
userId: <userId>
- directory: <Sense user directory>
userId: <userId>
excludeOwner:
user:
- directory: <Sense user directory>
userId: <userId>
- directory: <Sense user directory>
userId: <userId>
# Custom property used to control which aborted tasks will cause alert emails to be sent
# If this setting is true, alerts will not be sent for all tasks, but *only* for tasks with the CP set to the enabledValue.
# If this setting is false, alerts will be sent for all aborted reload tasks.
alertEnableByCustomProperty:
enable: true
customPropertyName: 'Butler_AbortedAlertEnableEmail'enabledValue:'Yes'# Custom property used to say that alerts for a certain task should be sent to zero or more recipients# These alerts will be sent irrespective of the alertEnableByCustomProperty.enable setting.alertEnabledByEmailAddress:customPropertyName:'Butler_AbortedAlertSendToEmail'rateLimit:600# Min seconds between emails for a given taskID. Defaults to 5 minutes.headScriptLogLines:15# Number of lines from start of script to include in emailtailScriptLogLines:15# Number of lines from end of script to include in emailpriority:high # high/normal/lowsubject: 'Qlik Sense reload aborted:"{{taskName}}"' # Email subject. Can use template fields
bodyFileDirectory: path/to/email_templates # Directory where email body template files are stored
htmlTemplateFile: aborted-reload # Name of email body template file to use
fromAddress: Qlik Sense (no-reply) <qliksense-noreply@mydomain.com>
recipients: # Array of email addresses to which the notification email will be sent
- <Email address 1>
- <Email address 2>
reloadTaskFailure:
enable: false
appOwnerAlert:
enable: true # Should app owner get notification email (assuming email address is available in Sense user directory)
includeOwner:
includeAll: true # true = Send notification to all app owners except those in exclude list
# false = Send notification to app owners in the include list
user:
- directory: <Sense user directory>
userId: <userId>
- directory: <Sense user directory>
userId: <userId>
excludeOwner:
user:
- directory: <Sense user directory>
userId: <userId>
- directory: <Sense user directory>
userId: <userId>
# Custom property used to control which task failures will cause alert emails to be sent
# If this setting is true, alerts will not be sent for all tasks, but *only* for tasks with the CP set to the enabledValue.
# If this setting is false, alerts will be sent for all failed reload tasks.
alertEnableByCustomProperty:
enable: false
customPropertyName: 'Butler_FailedAlertEnableEmail'enabledValue:'Yes'# Custom property used to say that alerts for a certain task should be sent to zero or more recipients# These alerts will be sent irrespective of the alertEnableByCustomProperty.enable setting.alertEnabledByEmailAddress:customPropertyName:'Butler_FailedAlertSendToEmail'rateLimit:600# Min seconds between emails for a given taskID. Defaults to 5 minutes.headScriptLogLines:15# Number of lines from start of script to include in emailtailScriptLogLines:15# Number of lines from end of script to include in emailpriority:high # high/normal/lowsubject: 'Qlik Sense reload failed:"{{taskName}}"' # Email subject. Can use template fieldsbodyFileDirectory:path/to/email_templates # Directory where email body template files are storedhtmlTemplateFile:failed-reload # Name of email body template file to usefromAddress:Qlik Sense (no-reply) <qliksense-noreply@mydomain.com>recipients:# Array of email addresses to which the notification email will be sent- <Email address 1>- <Email address 2>......smtp:# Email server settings. See https://nodemailer.com/smtp/ for details on the meaning of these fields.host:<FQDN or IP or email server, e.g. smtp.gmail.com>port:<port on which SMTP server is listening>secure:true# true/falsetls:serverName:# If specified the serverName field will be used for TLS verification instead of the host field.ignoreTLS:falserequireTLS:truerejectUnauthorized:falseauth:enable:trueuser:<Username, email address etc>password:<your-secret-password>......udpServerConfig:enable:false# Should the UDP server responsible for receving task failure and session events be started? true/falseserverHost:<FQDN or IP (or localhost) of server where Butler is running>portTaskFailure:9998......
Templates: Configuring email appearance
Alert emails use standard HTML formatting. Inline CSS can be used (if so desired) for fine tuning the visual look of the alert email.
Butler’s process for sending alert emails is
Figure out which email body template file should be used. This is determine by two set of fields in the main config file:
For reload failure emails these config file properties are used:
Butler.emailNotification.reladTaskFailure.bodyFileDirectory and
Butler.emailNotification.reladTaskFailure.htmlTemplateFile
For aborted reload emails these config file properties are used:
Butler.emailNotification.reloadTaskAborted.bodyFileDirectory and
Butler.emailNotification.reloadTaskAborted.htmlTemplateFile
For email subjects, these config properties are used:
Butler.emailNotification.reladTaskFailure.subject and
Butler.emailNotification.reloadTaskAborted.subject
Process the body template, replacing template fields with actual values.
Process the email subject template, replacing template fields with actual values.
Send the email.
A couple of sample template files are found in the src/config/email_templates directory of the GitHub repository.
Remember
You can use template fields in email subjects too!
Template fields reference
A complete list of template fields - including descriptions - is available in the Reference section.
2.2 - Reload alerts via Slack
Description of how reload alerts can be sent as Slack messages.
What’s this?
Butler can send two kinds of alert messages via Slack:
A complete reference to the config file format is found here.
Basic vs formatted Slack alerts
Slack alerts come in two forms:
Customizable formatting using a template concept. A standard template that will fit most use cases is included with Butler. Using this option the first and last parts of the script log can be included in the message, allowing you to tell from the Slack message what caused the reload to fail.
You can also add buttons to the message that can be used to open any URL you want, or open the app that failed reloading.
A fixed, more basic format that is built into Butler. No template file needed.
Which option to go for depends on whether you want just a notification that something went wrong, or if you want as much detail as possible in the Slack message.
Sample message with custom formatting
Note
The concept described below is the same for failed and aborted reload tasks.
Each of these have their own settings in the config file.
A “reload task failed” Slack message using the custom formatting option could look like this:
Here’s how to set this up:
Create an incoming webhook in Slack, take note of its URL (you will need it in step 2 below).
Edit the Slack section of the config file, i.e. the settings in Butler.slackNotification.reloadTaskFailure.
The messageType property should be set to formatted.
The basicMsgTemplate property is not used with formatted messages and can thus be left empty,
Edit the template file if/as needed, the file is specified in Butler.slackNotification.reloadTaskFailure.templateFile. It uses the Handlebars templating engine, to which Butler provides template fields with actual values.
Sample template files are included in the release Zip file, and are also available in the GitHub repository’s src/config/slack_templates directory.
Restart Butler if it’s already running.
Sample message with basic formatting
Note
The concept described below is the same for failed and aborted reload tasks.
Each of these have their own settings in the config file.
A “reload task failed” Slack message with basic formatting could look like this:
To set it up:
Create an incoming webhook in Slack if you don’t already have one, take note of its URL (you will need it in step 2 below).
Edit the Slack section of the config file, i.e. in Butler.slackNotification.reloadTaskFailure.
The messageType property should be set to basic.
The basicMsgTemplate property is the message that will be sent via Slack. Template fields can be used.
Restart Butler if it’s already running.
Customizing Slack messages
When using the formatted Slack alerts you have full freedom to create the alert you need.
Behind the scenes Slack messages are constructed from blocks defined in a JSON object. Each block can then contain either plain text, Markdown, images, buttons etc.
The Slack documentation is the best place for learning how to customize messages.
When it comes to Butler, it uses the Handlebars templating engine to render the template files into Slack JSON objects that are then sent to Slack via their APIs.
A few things to keep in mind when creating custom Slack messages:
The handlebars syntax itself must be correct. If incorrect no Slack JSON object will be created. And no Slack messages sent.
The handlebars template must result in a JSON object that adheres to Slack’s API specifications.
If the JSON syntax is somehow invaid the Slack API will return errors and no messages sent. JSON can be pretty sensitive to details, there should for example not be any trailing commas in properly formatted JSON objects.
Block Kit Builder: Great sandbox wtih readily available examples of different message layouts, syntax and more. Note: You must be logged into your Slack account to use this tool.
How it works
Warning
Don’t forget to create the log appender .xml files on the Sense server(s).
Those xml files are the foundation on top of which all Butler alerts are built - without them the alerts described on this page won’t work.
The concept is the same for for all alert types, see the email alerts for details.
Settings in config file
---Butler:......# Settings for notifications and messages sent to SlackslackNotification:enable:falserestMessage:webhookURL:<web hook URL from Slack> # Webhook to use when sending basic Slack messages via Butler's REST API reloadTaskFailure:# Reload task failed in QSEoWenable:falsewebhookURL:<web hook URL from Slack>channel:sense-task-failure # Slack channel to which task failure notifications are sentmessageType:formatted # formatted / basic. Formatted means that template file below will be used to create the message.basicMsgTemplate: 'Qlik Sense reload failed:"{{taskName}}"' # Only needed if message type = basic
rateLimit: 300 # Min seconds between emails for a given taskID. Defaults to 5 minutes.
headScriptLogLines: 10
tailScriptLogLines: 10
templateFile: /path/to/slack/template/directory/failed-reload-qseow.handlebars
fromUser: Qlik Sense
iconEmoji: ':ghost:'reloadTaskAborted:# Reload task aborted in QSEoWenable:falsewebhookURL:<web hook URL from Slack>channel:sense-task-aborted # Slack channel to which task stopped notifications are sentmessageType:formatted # formatted / basic. Formatted means that template file below will be used to create the message.basicMsgTemplate: 'Qlik Sense reload aborted:"{{taskName}}"' # Only needed if message type = basic
rateLimit: 300 # Min seconds between emails for a given taskID. Defaults to 5 minutes.
headScriptLogLines: 10
tailScriptLogLines: 10
templateFile: /path/to/slack/template/directory/aborted-reload-qseow.handlebars
fromUser: Qlik Sense
iconEmoji: ':ghost:'......udpServerConfig:enable:false# Should the UDP server responsible for receving task failure and session events be started? true/falseserverHost:<FQDN or IP (or localhost) of server where Butler is running>portTaskFailure:9998......
2.3 - Reload alerts via Microsoft Teams
Description of how reload alerts can be sent as Microsoft Teams messages.
What’s this?
Butler can send two kinds of alert messages via Teams:
A complete reference to the config file format is found here.
Basic vs formatted Teams alerts
Teams alerts come in two forms:
Customizable formatting using a template concept. A standard template that will fit most use cases is included with Butler. With this option the first and last parts of the script log can be included in the message, allowing you to tell from the Teams message what caused the reload to fail.
You can also add buttons to the message that can be used to open any URL you want, or open the app that failed reloading.
A fixed, more basic format that is built into Butler. No template file needed.
Which option to go for depends on whether you want just a notification that something went wrong, or if you want as much detail as possible in the Teams message.
Sample message with custom formatting
Note
The concept described below is the same for failed and aborted reload tasks.
Each of these have their own settings in the config file.
A “reload task failed” Teams message using the custom formatting option could look like this:
Here’s how to set it up:
Create a workflow in Teams, take note of its URL (you will need it in step 2 below). More information on how to create a workflow further down this page.
Edit the Teams section of the config file, i.e. the settings in Butler.teamsNotification.reloadTaskFailure.
The messageType property should be set to formatted.
The basicMsgTemplate property is not used with formatted messages and can thus be left empty,
Edit the template file if/as needed, the file is specified in Butler.teamsNotification.reloadTaskFailure.templateFile.It uses the Handlebars templating engine, to which Butler provides template fields with actual values.
Sample template files are included in the release Zip file, and are also available in the GitHub repository’s src/config/teams_templates directory.
Restart Butler if it’s already running.
Sample message with basic formatting
Note
The concept described below is the same for failed and aborted reload tasks.
Each of these have their own settings in the config file.
A “reload task failed” Teams message with basic formatting could look like this:
To set it up:
Create an incoming webhook in Teams if you don’t already have one, take note of its URL (you will need it in step 2 below).
Edit the Teams section of the config file i.e. the settings in Butler.teamsNotification.reloadTaskFailure and/or Butler.teamsNotification.reloadTaskAborted sections of the confi file.
The messageType property should be set to basic.
The basicMsgTemplate property is the message that will be sent via Teams. Template fields can be used.
Restart Butler if it’s already running.
Customizing Teams messages
When using the formatted Teams alerts you have full freedom to create the alert you need.
Behind the scenes Teams messages are constructed as “Adaptive Cards”, which is standardised JSON format that Teams understands.
More information on Adaptive Cards can be found here, here and here.
When it comes to Butler, it uses the Handlebars templating engine to render a template file into an adaptive card JSON object that is then sent to the workflow webhook.
A few things to keep in mind when creating custom Teams messages:
The handlebars syntax itself must be correct. If incorrect no Teams JSON object will be created. And no Teams message sent.
The handlebars template must result in a JSON object that adheres to Teams’s specifications for JSON payloads.
If the JSON syntax is somehow invaid the Teams API will return errors and no messages sent. JSON can be pretty sensitive to details, there should for example not be any trailing commas in properly formatted JSON objects.
How it works
Warning
Don’t forget to create the log appender .xml files on the Sense server(s).
---Butler:......# Settings for notifications and messages sent to MS TeamsteamsNotification:enable:falsereloadTaskFailure:enable:falsewebhookURL:<web hook URL from MS Teams>messageType:formatted # formatted / basic. Formatted means that template file below will be used to create the message.basicMsgTemplate: 'Qlik Sense reload failed:"{{taskName}}"' # Only needed if message type = basic
rateLimit: 300 # Min seconds between emails for a given taskID. Defaults to 5 minutes.
headScriptLogLines: 10
tailScriptLogLines: 10
templateFile: /path/to/teams/template/directory/failed-reload-qseow.handlebars
reloadTaskAborted:
enable: false
webhookURL: <web hook URL from MS Teams>
messageType: formatted # formatted / basic. Formatted means that template file below will be used to create the message.
basicMsgTemplate: 'Qlik Sense reload aborted:"{{taskName}}"' # Only needed if message type = basicrateLimit:300# Min seconds between emails for a given taskID. Defaults to 5 minutes.headScriptLogLines:10tailScriptLogLines:10templateFile:/path/to/teams/template/directory/aborted-reload-qseow.handlebars......udpServerConfig:enable:false# Should the UDP server responsible for receving task failure and session events be started? true/falseserverHost:<FQDN or IP (or localhost) of server where Butler is running>portTaskFailure:9998......
