SFTP

Detailed technical documentation on the Lemnisk SFTP to get data from SFTP to Lemnisk

SFTP is a network protocol that allows you to securely transfer and manage large sizes of confidential data. You can setup SFTP as a data source to get data from SFTP to Lemnisk.

Steps to configure SFTP as Source

Go to -> Data Pipeline -> Sources

Step 1: Click on + Create New Source

Step 2: Select source type as SFTP and click Proceed

Step 3: Set up the source

Source name: Add a unique name to the source

Authentication: Select basic authentication and click on Create Source

Encrypting the SFTP file with GPG2 Encryption

To encrypt the SFTP file with GPG2 encryption, you can download the Public Key from the SFTP source as shown below. You need to encrypt the SFTP file with the public key provided.

The public key will end with .key extension and the contents would look something like this:

-----BEGIN PGP PUBLIC KEY BLOCK-----
<public key contents here>
-----END PGP PUBLIC KEY BLOCK-----

Once the key is downloaded, you can perform the below steps to encrypt the key with GPG2 encryption. You can run these commands in a terminal.

1. List the public keys available

gpg2 --list-public-keys

The above command should ideally produce an output like this:

gpg: directory `/home/ubuntu/.gnupg' created
gpg: new configuration file `/home/ubuntu/.gnupg/gpg.conf' created
gpg: WARNING: options in `/home/ubuntu/.gnupg/gpg.conf' are not yet active during this run
gpg: keyring `/home/ubuntu/.gnupg/pubring.gpg' created
gpg: /home/ubuntu/.gnupg/trustdb.gpg: trustdb created

2. Import the downloaded public key. Replace lemnisk_public_key with the name of the actual downloaded key.

gpg2 --import lemnisk_public_key.key

The above command should produce an output like this:

gpg: keyring `/home/ubuntu/.gnupg/secring.gpg' created
gpg: key 4E1CC4EA: public key "Lemnisk <[e-mail]>" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

3. Encrypt the tsv file that you wlll upload. Please take care of the --output format to align with your format so as to not overwrite older files in SFTP. Ideally add a date or timestamp of the filename.

gpg2 --encrypt --recipient [e-mail]  --output content-3Jan2024.gpg2 content.tsv

You can replace the content.tsv with the actual file name that you want to upload to Lemnisk. The above command should produce an output like this:

gpg: ADAA0048: There is no assurance this key belongs to the named user
pub  3072R/ADAA0048 2024-01-03 Lemnisk <[e-mail]>
 Primary key fingerprint: F2BE 5CE4 BF4A 896A 1CA8  0968 8E48 F843 4E1C C4EA
      Subkey fingerprint: E211 899C 75D7 1DA5 9EED  7B5E D943 F469 ADAA 0048
It is NOT certain that the key belongs to the person named
in the user ID.  If you *really* know what you are doing,
you may answer the next question with yes.
Use this key anyway? (y/N)

4. Enter y and press Enter to continue.

5. Place the encrypted file (content-3Jan2024.gpg2) in the SFTP folder shared with you.