This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Start Sense tasks using Butler APIs

1 - Start Sense tasks using REST API

Use Butler’s REST API to start Sense tasks

If the Butler config file is properly set up it’s possible to start Sense tasks by doing a PUT or POST call to /v4/reloadtask/{taskId}/start endpoint.

A great use case is to have upstream systems that feed Qlik Sense with data trigger a Sense task when new data is available.
That way Sense doesn’t have to poll for new data, with less system resources used in both upstream system and in Sense.

AND users get the new data as quickly as possible!

Requirements

These config file settings must be set up before Butler can use the REST API to start tasks:

  • Configure Butler’s REST server:
    • Butler.restServerConfig.enable: true
    • Butler.restServerConfig.serverHost: <IP or hostname where Butler’s REST server is running>
    • Butler.restServerConfig.serverPort:
    • Butler.restServerConfig.backgroundServerPort:
  • Enable the start task API endpoint
    • Butler.restServerEndpointsEnable.senseStartTask: true

Seeing is believing

The video below is available at Ptarmigan Labs’ YouTube channel and also in the Butler playlist.

The video gives a quick demo of what calling the APIs can look like when using macOS.


There are many tools that can be used to call REST APIs.

Postman is cross platform and works in the browser, Paw is outstanding if you’re using macOS - and many others.

In the examples below we keep it simple and just use curl to call the API.

Note:

  • This API requires an empty array to be passed in the body even when no tags, custom properties or similar are used.
  • In the examples Butler is exposing its API on 192.168.1.168:8080

Start a single task using task ID

Using a PUT call to start task with ID e3b27f50-b1c0-4879-88fc-c7cdd9c1cf3e:

➜  ~ curl -X "PUT" "http://192.168.1.168:8080/v4/reloadtask/e3b27f50-b1c0-4879-88fc-c7cdd9c1cf3e/start" \
     -H 'Content-Type: application/json; charset=utf-8' \
     -d $'[]'
{
  "tasksId": {
    "started": [
      {
        "taskId": "e3b27f50-b1c0-4879-88fc-c7cdd9c1cf3e",
        "taskName": "Reload task of App1"
      }
    ],
    "invalid": [],
    "denied": []
  },
  "tasksTag": [],
  "tasksTagDenied": [],
  "tasksCP": [],
  "tasksCPDenied": []
}
➜  ~

The response tells us:

  • One task was started.
  • No task IDs were invalid.
  • No task IDs were denied based on task filtering.
  • No tasks were started or denied using tags or custom properties.

Start a single task using an invalid task ID

The task ID abc123 is invalid. This will be detected and reported in the response:

➜  ~ curl -X "PUT" "http://192.168.1.168:8080/v4/reloadtask/abc123/start" -H 'Content-Type: application/json; charset=utf-8' -d $'[]'
{
  "tasksId": {
    "started": [],
    "invalid": [
      {
        "taskId": "abc123"
      }
    ],
    "denied": []
  },
  "tasksTag": [],
  "tasksTagDenied": [],
  "tasksCP": [],
  "tasksCPDenied": []
}
➜  ~

Start multiple tasks using valid task IDs

In this example all task IDs are valid. One of them is passed in the URL and the other two in the message body.

➜  ~ curl -X "PUT" "http://192.168.1.168:8080/v4/reloadtask/-/start" \
     -H 'Content-Type: application/json; charset=utf-8' \
     -d $'[
  {
    "type": "starttaskid",
    "payload": {
      "taskId": "7552d9fc-d1bb-4975-9a38-18357de531ea"
    }
  },
  {
    "type": "starttaskid",
    "payload": {
      "taskId": "fb0f317d-da91-4b86-aafa-0174ae1e8c8f"
    }
  }
]'
{
  "tasksId": {
    "started": [
      {
        "taskId": "7552d9fc-d1bb-4975-9a38-18357de531ea",
        "taskName": "Reload task of App2"
      },
      {
        "taskId": "fb0f317d-da91-4b86-aafa-0174ae1e8c8f",
        "taskName": "Reload task of App3"
      }
    ],
    "invalid": [],
    "denied": []
  },
  "tasksTag": [],
  "tasksTagDenied": [],
  "tasksCP": [],
  "tasksCPDenied": []
}
➜  ~

The response tells us:

  • The magic task ID “-” will be ignored.
  • Two tasks, specified in the request body, were started.

Start multiple tasks using task IDs, all task IDs must exist, task filtering ON

Here two task IDs are valid and on the list of approved task IDs. One task ID is invalid (too short).
As allTaskIdsMustExist=true we expect that no task is started (all task IDs must exist for any task to be started based on task ID!). Task filtering is turned on in the config file’s Butler.startTaskFilter.enable entry.

➜  ~ curl -X "PUT" "http://192.168.1.168:8080/v4/reloadtask/e3b27f50-b1c0-4879-88fc-c7cdd9c1cf3e/start?allTaskIdsMustExist=true" \
     -H 'Content-Type: application/json; charset=utf-8' \
     -d $'[
  {
    "type": "starttaskid",
    "payload": {
      "taskId": "7552d9fc-d1bb-4975-9a38-18357de531ea"
    }
  },
  {
    "type": "starttaskid",
    "payload": {
      "taskId": "fb0f317d-da91-4b86-aafa-0174ae1e8c8"
    }
  }
]'
{
  "tasksId": {
    "started": [],
    "invalid": [
      {
        "taskId": "fb0f317d-da91-4b86-aafa-0174ae1e8c8"
      }
    ],
    "denied": [
      {
        "taskId": "e3b27f50-b1c0-4879-88fc-c7cdd9c1cf3e"
      },
      {
        "taskId": "7552d9fc-d1bb-4975-9a38-18357de531ea"
      }
    ]
  },
  "tasksTag": [],
  "tasksTagDenied": [],
  "tasksCP": [],
  "tasksCPDenied": []
}
➜  ~

The response tells us:

  • No tasks were started based on task IDs.
  • One invalid (too short!) task is returned in the response.
  • As there was one or more invalid task IDs, the two valid and approved task IDs were not started. Their task IDs are returned in the denied array in the response.

Start multiple tasks using task IDs, all task IDs must exist, task filtering OFF

Here two task IDs are valid and on the list of approved task IDs. One task ID is invalid (too short).
As allTaskIdsMustExist=true we expect that no task is started (all task IDs must exist for any task to be started based on task ID!). Task filtering is turned off in the config file’s Butler.startTaskFilter.enable entry.

➜  ~ curl -X "PUT" "http://192.168.1.168:8080/v4/reloadtask/e3b27f50-b1c0-4879-88fc-c7cdd9c1cf3e/start?allTaskIdsMustExist=true" \
     -H 'Content-Type: application/json; charset=utf-8' \
     -d $'[
  {
    "type": "starttaskid",
    "payload": {
      "taskId": "7552d9fc-d1bb-4975-9a38-18357de531ea"
    }
  },
  {
    "type": "starttaskid",
    "payload": {
      "taskId": "fb0f317d-da91-4b86-aafa-0174ae1e8c8"
    }
  }
]'
{
  "tasksId": {
    "started": [],
    "invalid": [
      {
        "taskId": "fb0f317d-da91-4b86-aafa-0174ae1e8c8"
      }
    ],
    "denied": [
      {
        "taskId": "e3b27f50-b1c0-4879-88fc-c7cdd9c1cf3e"
      },
      {
        "taskId": "7552d9fc-d1bb-4975-9a38-18357de531ea"
      }
    ]
  },
  "tasksTag": [],
  "tasksTagDenied": [],
  "tasksCP": [],
  "tasksCPDenied": []
}
➜  ~

The response and its message is the same as in the previous example:

  • No tasks were started based on task IDs.
  • One invalid (too short!) task is returned in the response.
  • As there was one or more invalid task IDs, the two valid and approved task IDs were not started. Their task IDs are returned in the denied array in the response.

Start multiple tasks using task IDs, task filtering ON

  • Two task IDs are valid and on the list of approved task IDs.
  • One task ID is valid but not on list of approved task IDs.
  • One task ID is invalid (too short).
  • Task filtering is turned on in the config file’s Butler.startTaskFilter.enable entry.
➜  ~ curl -X "PUT" "http://192.168.1.168:8080/v4/reloadtask/e3b27f50-b1c0-4879-88fc-c7cdd9c1cf3e/start" \
     -H 'Content-Type: application/json; charset=utf-8' \
     -d $'[
  {
    "type": "starttaskid",
    "payload": {
      "taskId": "7552d9fc-d1bb-4975-9a38-18357de531ea"
    }
  },
  {
    "type": "starttaskid",
    "payload": {
      "taskId": "fb0f317d-da91-4b86-aafa-0174ae1e8c8"
    }
  },
  {
    "type": "starttaskid",
    "payload": {
      "taskId": "8b4fe424-d90c-493f-a61d-0ce91cd485c9"
    }
  }
]'
{
  "tasksId": {
    "started": [
      {
        "taskId": "e3b27f50-b1c0-4879-88fc-c7cdd9c1cf3e",
        "taskName": "Reload task of App1"
      },
      {
        "taskId": "7552d9fc-d1bb-4975-9a38-18357de531ea",
        "taskName": "Reload task of App2"
      }
    ],
    "invalid": [
      {
        "taskId": "fb0f317d-da91-4b86-aafa-0174ae1e8c8"
      }
    ],
    "denied": [
      {
        "taskId": "8b4fe424-d90c-493f-a61d-0ce91cd485c9"
      }
    ]
  },
  "tasksTag": [],
  "tasksTagDenied": [],
  "tasksCP": [],
  "tasksCPDenied": []
}
➜  ~

The response tells us:

  • Two tasks were started, as their task IDs were approved in the config file.
  • One task ID was invalid (too short!).
  • One task ID had a valid format but was not on the list of approved task IDs.

Start tasks using tags

The underlying Qlik Sense system has two tags associated with tasks: startTask1 and startTask2.

The QMC shows which tasks have these tags set:

Qlik Sense QMC tasks with tags

Starting the three tasks tagged with startTask1:

➜  ~ curl -X "PUT" "http://192.168.1.168:8080/v4/reloadtask/-/start" -H 'Content-Type: application/json; charset=utf-8' -d $'[
  {
    "type": "starttasktag",
    "payload": {
      "tag": "startTask1"
    }
  }
]'
{
  "tasksId": {
    "started": [],
    "invalid": [],
    "denied": []
  },
  "tasksTag": [
    {
      "taskId": "e3b27f50-b1c0-4879-88fc-c7cdd9c1cf3e",
      "taskName": "Reload task of App1"
    },
    {
      "taskId": "7552d9fc-d1bb-4975-9a38-18357de531ea",
      "taskName": "Reload task of App2"
    },
    {
      "taskId": "fb0f317d-da91-4b86-aafa-0174ae1e8c8f",
      "taskName": "Reload task of App3"
    }
  ],
  "tasksTagDenied": [],
  "tasksCP": [],
  "tasksCPDenied": []
}
➜  ~

The response tells us:

  • Three tasks were started because they had a tag matching the one specified in the call to the API.
  • One invalid task ID was specified. This is the one in the URL - if needed it’s ok to provide a dummy task ID, as done here.

Start tasks using custom properties

A custom property taskGroup available on reload tasks have the following possible values:

Qlik Sense QMC custom property

Here’s a call that will start all tasks that have the custom property taskGroup set to either tasks1 or tasks2:

➜  ~ curl -X "PUT" "http://192.168.1.168:8080/v4/reloadtask/-/start?allTaskIdsMustExist=false" \
     -H 'Content-Type: application/json; charset=utf-8' \
     -d $'[
  {
    "type": "starttaskcustomproperty",
    "payload": {
      "customPropertyName": "taskGroup",
      "customPropertyValue": "tasks1"
    }
  },
  {
    "type": "starttaskcustomproperty",
    "payload": {
      "customPropertyName": "taskGroup",
      "customPropertyValue": "tasks2"
    }
  }
]'
{
  "tasksId": {
    "started": [],
    "invalid": [],
    "denied": []
  },
  "tasksTag": [],
  "tasksTagDenied": [],
  "tasksCP": [
    {
      "taskId": "e3b27f50-b1c0-4879-88fc-c7cdd9c1cf3e",
      "taskName": "Reload task of App1"
    },
    {
      "taskId": "7552d9fc-d1bb-4975-9a38-18357de531ea",
      "taskName": "Reload task of App2"
    },
    {
      "taskId": "fb0f317d-da91-4b86-aafa-0174ae1e8c8f",
      "taskName": "Reload task of App3"
    },
    {
      "taskId": "7552d9fc-d1bb-4975-9a38-18357de531ea",
      "taskName": "Reload task of App2"
    }
  ],
  "tasksCPDenied": []
}
➜  ~

The response tells us:

  • 3 unique tasks were started.
  • As the task “Reload task of App2” had both values set for the custom property, this task was started twice.

Sending parameters to apps

Sometimes there is a need to send parameters from outside of Sense to an app that should be reloaded.
This is supported by Butler as follows:

  1. Start a task and pass in one or more key-value pairs (=the parameters that should be sent to the app(s)) in the body of the call.
  2. Have the app being reloaded read the key-value pairs from within the load script, using the Butler APIs.
  3. Optional: Clear up (delete KV pairs or the namespace used) the key-value store when done.

Here a single task, identified by its ID in the URL, is started.
Two key-value pairs are passed along as parameters to the app. One has a TimeToLive of 10 seconds, the other has no TTL (=it will not be automatically deleted).

Task filtering is off, i.e. any task can be started using this API.

➜  ~ curl -X "PUT" "http://192.168.1.168:8080/v4/reloadtask/fbf645f0-0c92-40a4-af9a-6e3eb1d3c35c/start" -H 'Content-Type: application/json; charset=utf-8' -d $'[
  {
    "type": "keyvaluestore",
    "payload": {
      "value": "TheValue",
      "namespace": "MyFineNamespace",
      "key": "AnImportantKey",
      "ttl": 10000
    }
  },
  {
    "type": "keyvaluestore",
    "payload": {
      "value": "Bar",
      "namespace": "MyFineNamespace",
      "key": "Foo"
    }
  }
]'
{
  "tasksId": {
    "started": [
      {
        "taskId": "fbf645f0-0c92-40a4-af9a-6e3eb1d3c35c",
        "taskName": "Reload Operations Monitor"
      }
    ],
    "invalid": [],
    "denied": []
  },
  "tasksTag": [],
  "tasksTagDenied": [],
  "tasksCP": [],
  "tasksCPDenied": []
}
➜  ~

A bit of everything

Combining all of the above can look like this:

➜  ~ curl -X "PUT" "http://192.168.1.168:8080/v4/reloadtask/e3b27f50-b1c0-4879-88fc-c7cdd9c1cf3e/start?allTaskIdsMustExist=true" -H 'Content-Type: application/json; charset=utf-8' -d $'[
  {
    "type": "starttaskid",
    "payload": {
      "taskId": "7552d9fc-d1bb-4975-9a38-18357de531ea"
    }
  },
  {
    "type": "starttaskid",
    "payload": {
      "taskId": "fb0f317d-da91-4b86-aafa-0174ae1e8c8f"
    }
  },
  {
    "type": "starttasktag",
    "payload": {
      "tag": "startTask1"
    }
  },
  {
    "type": "starttasktag",
    "payload": {
      "tag": "startTask2"
    }
  },
  {
    "type": "starttaskcustomproperty",
    "payload": {
      "customPropertyName": "taskGroup",
      "customPropertyValue": "tasks1"
    }
  },
  {
    "type": "starttaskcustomproperty",
    "payload": {
      "customPropertyName": "taskGroup",
      "customPropertyValue": "tasks2"
    }
  },
  {
    "type": "keyvaluestore",
    "payload": {
      "value": "TheValue",
      "namespace": "MyFineNamespace",
      "key": "AnImportantKey",
      "ttl": 10000
    }
  },
  {
    "type": "keyvaluestore",
    "payload": {
      "namespace": "MyFineNamespace",
      "key": "Foo",                                                                                                                                                                                               <....
{
  "tasksId": {
    "started": [
      {
        "taskId": "e3b27f50-b1c0-4879-88fc-c7cdd9c1cf3e",
        "taskName": "Reload task of App1"
      },
      {
        "taskId": "7552d9fc-d1bb-4975-9a38-18357de531ea",
        "taskName": "Reload task of App2"
      },
      {
        "taskId": "fb0f317d-da91-4b86-aafa-0174ae1e8c8f",
        "taskName": "Reload task of App3"
      }
    ],
    "invalid": [],
    "denied": []
  },
  "tasksTag": [
    {
      "taskId": "e3b27f50-b1c0-4879-88fc-c7cdd9c1cf3e",
      "taskName": "Reload task of App1"
    },
    {
      "taskId": "7552d9fc-d1bb-4975-9a38-18357de531ea",
      "taskName": "Reload task of App2"
    },
    {
      "taskId": "fb0f317d-da91-4b86-aafa-0174ae1e8c8f",
      "taskName": "Reload task of App3"
    },
    {
      "taskId": "fb0f317d-da91-4b86-aafa-0174ae1e8c8f",
      "taskName": "Reload task of App3"
    }
  ],
  "tasksTagDenied": [],
  "tasksCP": [
    {
      "taskId": "e3b27f50-b1c0-4879-88fc-c7cdd9c1cf3e",
      "taskName": "Reload task of App1"
    },
    {
      "taskId": "7552d9fc-d1bb-4975-9a38-18357de531ea",
      "taskName": "Reload task of App2"
    },
    {
      "taskId": "fb0f317d-da91-4b86-aafa-0174ae1e8c8f",
      "taskName": "Reload task of App3"
    },
    {
      "taskId": "7552d9fc-d1bb-4975-9a38-18357de531ea",
      "taskName": "Reload task of App2"
    }
  ],
  "tasksCPDenied": []
}
➜  ~

2 - Start Sense tasks from load script of Sense apps

Helper functions included

It is very much possible to call Butler’s REST API from the load script of Sense apps.
Create a REST connector in the Sense editor, configure it for the endpoint you want to call and use it from the load script.

This works but is tedious and quickly leads to lots of script code - especially if you need to make many calls to the Butler API.

To make things a bit easier the Butler GitHub repository includes a set of helper .qvs files.
These contain functions/subs that encapsulate various Butler APIs (include starting tasks) and make them easier to use.

Just include the butler_subs.qvs file from the GitHub release package and you get (among many other things) a helper function that’s called StartTask.

Requirements for starting tasks via REST API

These config file settings must be set up before Butler can use the REST API to start tasks:

  • Connection to Qlik Sense:
    • Butler.configQRS.*
  • Configure Butler’s REST server:
    • Butler.restServerConfig.enable: true
    • Butler.restServerConfig.serverHost: <IP or hostname where Butler’s REST server is running>
    • Butler.restServerConfig.serverPort:
    • Butler.restServerConfig.backgroundServerPort:
  • Enable the start task API endpoint
    • Butler.restServerEndpointsEnable.senseStartTask: true
  • Sense data connections as described in the Getting started section.

Helper functions

There are two helper functions/sub for starting tasks:

  • StartTask(...) is a generic function that can be called with a single task ID, or with complex combinations of task IDs, tags, custom properties and key-value pairs.
  • StartTask_KeyValue(...) makes it easy to start a single task and pass along one key-value pair as parameter. This function is essentially a specialized version of the more generic StartTask sub.

Start a single task

The function(=sub in Sense lingo) StartTask takes a single taskId parameter, which means that starting a reload task from an app’s load script is as simple as

Call StartTask(<TaskId>)

The demo app Butler 8.4 demo app.qvf (link) contains such a demo (and many others).

Need to pass along parameters to a task? There’s a Sub for that!

Sometimes you need to send parameters to a reload task (or rather to the load script of the app associated with the task).
This can be done by using the StartTask_KeyValue helper function/Sub.

That Sub takes a taskId as parameter (similarly to its StartTask sibling), but it also takes parameters for a full key-value pair:

Call StartTask_KeyValue('fbf645f0-0c92-40a4-af9a-6e3eb1d3c35c', 'MyNamespace', 'An important key', 'The value', 3000)

The parameters are

  • The namespace to store the key-value pair in (required).
  • The key (required).
  • The value (required).
  • An time-to-live valud in milliseconds (optional). When the ttl times out the key-value pair is automatically deleted.

Documentation about Butler’s key-value store is available here.

An example showing how task chaining with parameters can be done using key-values is found here.

Start several tasks using task IDs

If several tasks should be started using task IDs, those IDs need to be passed into the StartTask sub.
This is done by storing the task IDs in a separate table whose name is passed as a parameter into StartTask:

These tables can be called anything as long as

  1. They are qualified (i.e. keep the “Qualify *;” statement!).
  2. The table names are passed as parameters to the StartTask function.
  3. The table MUST have a field called TaskId that contains the IDs of reload tasks to be started.

Regarding parameters to StartTask:

  1. Trailing, unused parameters can be omitted.
  2. Unused parameters that are followed by used parameters should be set to Null().

Example 1

The script below will start tasks fbf645f0-0c92-40a4-af9a-6e3eb1d3c35c (via the first parameter), 7552d9fc-d1bb-4975-9a38-18357de531ea (via second parameter, i.e. a table) and fb0f317d-da91-4b86-aafa-0174ae1e8c8f (via second parameter too).

Qualify *;

ButlerTaskIDs:
Load * Inline [
TaskId
7552d9fc-d1bb-4975-9a38-18357de531ea
fb0f317d-da91-4b86-aafa-0174ae1e8c8f
];

Call StartTask('fbf645f0-0c92-40a4-af9a-6e3eb1d3c35c', 'ButlerTaskIDs')

Unqualify *;

Example 2

Same as previous example, except that the first parameter is not used.
It must still be specified though! Set to Null() to indicate it isn’t used.

The script below will thus start tasks 7552d9fc-d1bb-4975-9a38-18357de531ea and fb0f317d-da91-4b86-aafa-0174ae1e8c8f.

Qualify *;

ButlerTaskIDs:
Load * Inline [
TaskId
7552d9fc-d1bb-4975-9a38-18357de531ea
fb0f317d-da91-4b86-aafa-0174ae1e8c8f
];

Call StartTask(Null(), 'ButlerTaskIDs')

Unqualify *;

Start tasks using tags

Similar to how multiple tasks can be started using a table of task IDs (see above), tasks can also be started using a table containing tag names.

Example 1

The script below will start all reload tasks that have the startTask1 or startTask2 tag set.

Qualify *;

ButlerTags:
Load * Inline [
Tag
startTask1
startTask2
];

Call StartTask(, Null(), 'ButlerTags')

Unqualify *;

Start tasks using custom properties

Similar to how multiple tasks can be started using a table of task IDs (see above), tasks can also be started using a table containing custom property names and values.

Example 1

The script below will start all reload tasks that have the taskGroup custom property set to a value of tasks1.

Qualify *;

ButlerCustomProperties:
Load * Inline [
Name, Value
taskGroup, tasks1
];

Call StartTask(, Null(), Null(), 'ButlerKeyValues')

Unqualify *;

Seeing is believing

The video below is available at Ptarmigan Labs’ YouTube channel and also in the Butler playlist.

3 - Start Sense tasks using MQTT

Use MQTT to start Sense tasks

Butler can be configured to listen to a specific MQTT topic (specified in config file property Butler.mqttConfig.taskStartTopic) and use any message received in that topic as a Sense task ID, which is then started.

For example:

  • A Sense app, used by end users, relies on data in a source system that talks MQTT (there are lots of MQTT libraries available, covering most operating systems).
  • The data in the source system can be updated at any time.

In order to update the Sense app with data the most common approach is to schedule reloads of the Qlik Sense app at certain intervals, i.e. polling the source system.

But if the source system instead posts a MQTT message on a well defined topic when new data is available, theat message will trigger the Sense app’s reload.

This way the Sense app will be updated as quickly as possible after new data is availabe in the source system.

I.e. the end user will have access to more up-to-date data, compared to the polling based solution.

Requirements for starting tasks via MQTT

These config file settings must be set up before Butler will use MQTT messages to start tasks:

  • Connection to MQTT broker (=server):
    • Butler.mqttConfig.enable: true
    • Butler.mqttConfig.brokerHost:
    • Butler.mqttConfig.brokerPort:
  • MQTT topics that Butler should subscribe to
    • Butler.mqttConfig.subscriptionRootTopic: <Root topic that Butler should subscribe to. Something like qliksense/#>
    • Butler.mqttConfig.taskStartTopic: <Topic used to start Sense tasks. MUST be a suptopic to the root topic above!>

Seeing is believing

The video below is available at Ptarmigan Labs’ YouTube channel and also in the Butler playlist.