In this tutorial, learn how to load data from remote files into Qlik Associative Engine with the built-in File Connectivity Service.
This tutorial shows an example where data is loaded from data files that are located on Dropbox, but the solution is similar for loading data from other remote sources, such as Google Drive or Microsoft OneDrive.
To follow along in this tutorial, you should have basic understanding of Docker.
You need the following software installed:
Shell commands should be run in a Bash shell. If you are using Windows, we recommend using Git Bash.
The File Connectivity Service¶
The File Connectivity Service is an example of a service that you can use to load data from remote files in a Qlik Core stack. You can configure this service to provide built-in data connectivity to connect to OAuth 2.0-protected data sources like Dropbox, OneDrive, and Google Drive.
The data connection service in this tutorial works by defining a unique HTTP endpoint for each registered connection provider. The Qlik Associative Engine can then access different data sources by making calls to the service-defined HTTP endpoints.
To run the example code, clone the qlik-oss/core-file-connectivity-service Git repository. Before you continue, look at the documentation to get familiar with the content and structure of the repository.
In your shell, make sure current directory is at the repository root.
First, install the dependencies:
Next, copy the data/airports.csv file into your Dropbox root folder.
Now you need to create an OAuth2.0 application by following the Dropbox OAuth guide. Copy the App key and App secret as these are needed in the next steps.
When you create your application, the
Redirect URI should be the address of the callback that is running the
http://[host]:[port]/oauth2/callback. For example:
Before you start the Docker container, you need to accept the EULA, and you need to set the App Key and App Secret as environment variables so the engine container can communicate with the OAuth2.0 app.
Run the following commands:
cd examples ACCEPT_EULA=<yes/no> \ DROPBOX_CLIENT_ID=<Dropbox App key> \ DROPBOX_CLIENT_SECRET=<Dropbox App secret> \ docker-compose up -d --build
core-data-loading examples script logging is enabled to get the log messages in
The script logging verbosity can be set in e.g. a docker-compose file like this.
Further description regarding the verbosity levels and log message details can be found in the logging chapter.
Loading data from Dropbox¶
After you complete the steps above, you can now load data from the file on Dropbox.
Run the following:
The expected output printed to the console is a list of 10 airport entries from the airports.csv file.
More runnable load script examples, including more advanced ones, are available in the qlik-oss/core-data-loading Git repository.