Butler was developed with InfluxDB version 1.x in mind.
If you intend to use Butler together with InfluxDB you need to be aware of the following:
InfluxDB is currently available in version 2.x and while this version brings lots of new goodies, it’s not out-of-the-box compatible with Butler.
For that reason you should use the latest 1.x version of InfluxDB, which at the time of this writing is 1.8.4.
If you do not intend to use any InfluxDB related features of Butler you can simply disregard this warning.
In due time Butler will be updated to support InfluxDB 2.x too.
Given the cross platform nature of Node.js (which is the language Butler is written in), Butler can run on lots of different hardware platforms and operating systems.
It is therefore difficult to give detailed installation instructions for each possible installation scenario. This site thus tries explain how to get started with Butler in some of the most common scenarios.
Pre-built binaries are available for Windows, macOS and Linux. When using these there is no need to install Node.js, as the Node.js runtime is bundled into the binaries.
Using these binaries is the easiest - and thus recommended - way of using Butler.
…unless you want to use Docker, which is also a great option.
Getting started
Sorry - there is no installer for Butler.
The pre-built binaries for Windows, macOS, Linux and Docker simply work as-is when combined with a properly set up configuration file.
If you still want to run Butler as Node.js app you will first need to install Node.js.
The instructions on the pages below should provide good guidance, if you still run into troubles you can always reach out via the GitHub discussion forums.
What’s required to use Butler
A Butler config file adapted to your specific Qlik Sense environment
Certificates exported from Qlik Sense Enterprise.
What
Comment
Qlik Sense Enterprise on Windows
Mandatory. Butler is developed with client-managed Qlik Sense Enterprise on Windows (QSEoW) in mind. While some Butler features might also work with Sense Desktop or Sense cloud, you are on your own there.
Butler executable
Mandatory. A Butler executable of some kind. This would be a) a stand-alone binary for the operating system you plan to use, b) a Docker image from which a Butler container can be created or c) the Butler source code plus Node.js installed.
MQTT broker
Optional. MQTT is used for both in- and out-bound pub-sub messaging. Butler assumes a working MQTT broker is available, the IP of which is defined in the Butler config file. Mosquitto is a great open source broker. It requires very little hardware to run, even the smallest (usually free) Amazon/Google/Microsoft/… instance is enough, if you want a dedicated MQTT server. If you don’t care about the pubsub features of Butler, you don’t need a MQTT broker. In this case you can disable the MQTT features in the config YAML file.
Optional. A commercial online service offering a vast set of observability features of which Butler uses just a few. Reload failure alerts are for example very nicely handled in New Relic as you get access to the script logs (similar to what can be done with InfluxDB + Grafana) right in the New Relic UI. New Relic’s free tier usually goes a long way towards the need of SenseOps and Butler use cases, so it’s easy to try out New Relic.
Optional. A smaller but very nice, mobible-first incident management service. Using Signl4 it’s easy to get failed reload alerts to your phone. The service also makes it easy to set up on-call schedules, escalate incidents if needed etc.
1 - Decide how to run Butler
On what platforms does Butler run?
The short answer is: Almost anywhere.
The pre-built binaries for Windows, macOS, Linux and Docker should cater for most use cases.
If you have some other, more exotic platform or operating system you want to run Butler on that’s probably possible too.
Butler is built on Node.js and as long as Node.js is available on the platform/operating system of your choice there is a good chance Butler will run there.
Butler has been successfully used on Windows Server, Windows 10, various Linux distributions, in Docker, Kubernetes, on Mac OS and even on Raspberry Pis. And a Raspberry Pi based Kubernetes cluster.
Your platform options thus typically fall into three categories:
Butler as a stand-alone executable
Here you will be using the pre-built Butler binaries (Windows, Linux, Mac OS) that are available for Butler 7.2 and later.
When using third party tools these binaries can be started as services.
For example, on Windows the free NSSM tool is a great way to run Butler as a Windows service.
Another good tool is PM2 which works well on Linux-ish platforms.
If you have access to or can set up a container runtime environment, this is a great way to running Butler.
Installation is less error prone compared to installing Butler as a native Node.js app, you get all the benefits from the Docker ecosystem (monitoring of running containers etc), and upgrades to future Butler versions become trivial.
If you have access to a Kubernetes cluster, that is usually an even better option than Docker. Kubernetes can be daunting when first approached, but will give you superb reliability, failover and restarts if a server goes down or becomes unresponsive etc. All major cloud providers (Microsoft Azure, Google, Amazon etc) offer Kubernetes services.
Rancher’s K3s is a very good way to get started with self hosted Kubnernetes. Fully featured, well supported and a vibrant developer community.
Butler as a Node.js application
This option means you will first install Node.js on your server of choice, then Butler and it’s dependencies.
It works perfectly well but is the most demanding when it comes to amount of work needed to get started.
2 - Running Butler as a native, pre-built application
How to install the pre-built, stand alone Butler applications.
Downloading the app
Download Butler for your preferred operating systym.
Decide where to install Butler
It is usually a good starting point to run Butler on the Sense server. If there are more than one server in the Sense cluster, Butler can be placed on the reload server (as the /createDir endpoint then can be used to create folders in which QVD and other files can be stored).
On the other hand, you might want to keep the Sense servers as clean as possible (with respect to software running on them). If that is a priority you should install Butler on some other server.
The bottom line is that Butler can run on any server, as long as there is network connectivity to the Sense server(s).
It’s usually a good idea to keep 3rd party tools installed in the same directory tree, to maintenance as easy as possible.
A good place for Butler could be c:\tools\butler or d:\tools\butler on Windows, for example.
Download Butler
Download the latest version from the releases page.
Make sure to get the binary file for your preferred operating system.
Unzip the downloaded file, then copy or move the butler binary to the desired directory (e.g. c:\tools\butler) and that’s it.
Tip
On Windows you must “unblock” the ZIP file before extracting the Butler binary from it.
This is basically a way to tell Windows that the ZIP is safe even though it was downloaded from Internet.
Right click on the ZIP file, then select Properties.
If there is an “Unblock” check box in the lower right part of the properties window you should click that box and hit OK.
Then unpack the ZIP file.
The macOS version of Butler is signed using Apple’s official app signing process.
This means you may see a warning the first time you start Butler, but after that there should be no more warnings.
3 - Running Butler in Docker
How to install Butler as a Docker container.
Installation steps
The following steps give some guidance on how to get Butler running on Docker.
Here Mac OS was used, things will look different on Linux and Windows.
Note: While the console logs below refer to an older version of Butler’s Docker image, the steps involved are the same also for current/most recent version of Butler.
proton:~ goran$ mkdir /Users/goran/butler
proton:~ goran$ cd /Users/goran/butler
proton:butler goran$ mkdir -p config/certificate
proton:butler goran$
proton:butler goran$ wget https://raw.githubusercontent.com/ptarmiganlabs/butler/master/src/docker-compose.yaml
--2021-10-25 17:07:23-- https://raw.githubusercontent.com/ptarmiganlabs/butler/master/src/docker-compose.yaml
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 660[text/plain]Saving to: ‘docker-compose.yaml’
docker-compose.yaml 100%[=====================================================================================================================================>]660 --.-KB/s in 0s
2021-10-25 17:07:23 (42.0 MB/s) - ‘docker-compose.yaml’ saved [660/660]proton:butler goran$ cat docker-compose.yaml
# docker-compose.ymlversion: '3.3'services:
butler:
image: ptarmiganlabs/butler:6.1.0
container_name: butler
restart: always
ports:
- "8080:8080"# REST API available on port 8180 to services outside the container - "9998:9998/udp"# UDP port for task failure events volumes:
# Make config file accessible outside of container - "./config:/nodeapp/config" - "./log:/nodeapp/log" environment:
- "NODE_ENV=production" logging:
driver: json-file
options:
max-file: "5" max-size: "5m"proton:butler goran$
At this point you should
Export certificates from the Qlik Sense QMC. Export a full set of certificates in PEM format, no psasword on the certificates.
Copy the certificates to the ./config/certificate directory.
Copy the template config file from the GitHub repository to the ./config directory, modify it as needed based on your system(s) and which Butler features you want enabled, and rename it to for example production.yaml.
You can name the config file anything, but its name has to match the NODE_ENV environment variable, as set it the docker-compose.yaml file.
Optional. Copy the template schedule file to the location specified in Butler’s config file. This is only needed if you manually want to add schedules. If using the API to create schedules, there is no need to first manually create a schedules file (the schedule file will be created by Butler in this case).
In the terminal where you ran docker-compose, you will see a new line saying that a app list was retrieved:
butler | 2021-10-25T19:20:50.356Z info: /v4/senselistapps called from 192.168.176.1
4 - Running Butler as a Node.js application
How to install Butler as a Node.js application.
Selecting an OS
While Qlik Sense Enterprise is a Windows only system, Butler should be able to run on any OS where Node.js is available.
Butler has been succesfully used - during development and production - on Windows, Linux (Debian and Ubuntu tested) and mac OS.
Installation steps
The steps below outline the steps needed to install Butler as a native Node.js application on for example Windows Server.
Install node.js
Butler has been developed and tested using the 64 bit version of Node.js. The most recent LTS (Long Term Support) version is usually a good choice.
Decide where to install Butler
It is usually a good starting point to run Butler on the Sense server. If there are more than one server in the Sense cluster, Butler can be placed on the reload server (as the /createDir endpoint then can be used to create folders in which QVD and other files can be stored).
On the other hand, you might want to keep the Sense servers as clean as possible (with respect to software running on them). If that is a priority you should install Butler on some other server.
The bottom line is that Butler can run on any server, as long as there is network connectivity to the Sense server(s).
Download Butler
Download the repository zip from the releases page.
Do not just clone the Butler repository as that will give you the latest development version, which may not yet be fully tested and packaged.
The exception is of course if you want to contribute to Butler development - then forking and cloning the repository is the right thing to do.
Install node dependencies
From a Windows command prompt (assuming the Butler ZIP file/repository was saved to d:\node\butler):
d:
cd \node\butler\src
npm install
This will download and install all Node.js modules used by Butler.
On some OSs you’ll get some warnings during the installation - they are usually harmless. Try to run Butler even if you got some warnings, chances are good that things will work just fine. This is common on especially Windows Server and is a result of some Butler dependencies being primarily developed on Linux rather than Windows.