sFTP integration instructions

An sFTP connection allows data to be uploaded to Thematic directly for processing.


Initial setup

During the initial setup phase Thematic will set up an account on Thematic’s sFTP server, secured using the customers public/private key. There will also be an option for setting up a second account for the purposes of testing.


The following process will be used for initial setup:

  1. The customer will provide their public key and IP addresses to whitelist to Thematic
  2. Thematic will provision an account on sftp.getthematic.com and set up the directories needed
  3. The customer can test access using their private key and upload a test file
  4. Thematic will verify that they can access the test file

Credentials

We will use a public/private key pair where the public key is shared by the customer and used to authenticate their access to the sFTP server.


Folder structure

There will be two directories provisioned on the server:

  • uploaded : this is the destination directory for uploads by the customer
  • processed : when a file has been picked up from ‘uploaded’ and sent for processing successfully it will be moved to the processed directory. This can be used for tracing which files have been handled and for re-processing in the event that an error is determined. Files will be removed from this folder after 30 days.

Staging / Sandboxing

If the customer requires, Thematic will set up a second sFTP account for the purposes of testing. This account can use a different key pair. The setup and process will otherwise be the same.

Adding a dataset

This section details the steps necessary to set up a new dataset in Thematic once the initial setup has been completed.


The following process will be used for setting up a new dataset:

  1. The customer will determine a unique file pattern that differentiates the source of the data, for example nps_daily_{date}.csv 
  2. The customer will upload an initial test file into uploaded and notify Thematic of the pattern
    1. This format should be discussed prior to upload to reduce back and forwards about any needed changes to optimise the analysis
  3. Thematic will setup a new destination dataset internally and the process to pick up files matching the pattern. 
  4. Thematic will use the format of the file to configure the dataset and have it show in the platform. 
  5. Thematic will notify the customer and continue to monitor for files matching the given pattern.

Errors

Thematic will monitor for uploads that haven’t occurred and uploads that do not match the schema of the initial files uploaded. If these occur Thematic will notify the customer to help resolve issues


Changes to file formats

If the format of the files being uploaded needs to change the customer will notify Thematic so that we can make sure to change the setup to accommodate.