Website Blog

AWS S3

Getting started with AWS S3 and setting it up

A cloud-based object storage service, Amazon S3 (Simple Storage Service) allows businesses and individuals to store their data securely within buckets. They can store and protect any amount of data such as data lakes, websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics into the bucket.

A bucket is a container for objects.

Its easy to use features like interface and management helps you optimize and organize the data according to business-specific requirements.

Features of AWS S3

  • Store data with changing or unknown access patterns with the S3 Intelligent-Tiering feature.

  • Various storage managenment features used manage costs, meet regulatory requirements, reduce latency, and save multiple distinct copies of your data for compliance requirements.

  • Amazon S3 provides features for auditing and managing access to your buckets and objects.

You can refer to the AWS S3 doc to know more about the features.

Getting started

The AWS S3 destination is a secure method of connecting to your S3 buckets. Lemnisk uses it's own IAM Roles to access your S3 buckets.

Connection

ConnectionWebMobile

​Device Mode

🚫

🚫

Cloud Mode

✅

✅

Configuration

In order to set up Amazon S3 as a destination in Lemnisk, you will have to login to your Amazon AWS S3 console and then create a new S3 bucket (you can also choose an existing one, if any).

Irrespective of the region, the raw logs of data from AWS S3, received by Lemnisk are put into your S3 bucket.

The data is processed from your sources and events are collected in batches by Lemnisk. Once every hour, a process is initiated where the batches are uploaded to a Lemnisk S3 bucket, and then securely copied to your own S3 bucket. However, there are no limitations to the data size.

For Lemnisk to access your S3 bucket, you will have to add the below bucket policy for a smooth transition of data.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowLemniskUser",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::821664204980:user/s3-copy"
            },
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::<s3-bucket-name>/lemnisk-logs/*"
        }
    ]
}

The data batching is done every one hour and there are no limitations on the data size.

Data format

Logs are stored as gzipped, newline-separated JSON containing the full call information in Lemnisk.

During the day, Lemnisk groups logs and then names them using the below format:

s3://{bucket}/lemnisk-logs/{source-id}/{received-day}/filename.gz

Methods

When a track event is triggered, it is sent to AWS S3 as a track event.

{
   "anonymousId":"viz_62200c405e6de",
   "traits":{
      "phone":"9433057867",
      "email":"paul.pinaki@gmail.com"
   },
   "messageId":"c855a52e-ff43-46fd-b99a-665fc3c96a3d",
   "sentAt":"2022-03-03T00:35:06.000Z",
   "receivedAt":"2022-03-03T00:35:07.000Z",
   "type":"track",
   "originalTimestamp":"2022-03-03T00:35:06.000Z",
   "userId":"CX1192900",
   "srcid":"6",
   "context":{
      "library":{
         "name":"smarttag",
         "version":"0.30v"
      },
      "ip":"42.110.172.44",
      "userAgent":"Mozilla/5.0 (Linux; Android 11; SM-M515F Build/RP1A.200720.012; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/98.0.4758.101 Mobile Safari/537.36 Instagram 224.0.0.20.116 Android (30/11; 450dpi; 1080x2174; samsung; SM-M515F; m51; qcom; en_GB; 353744237)",
      "page":{
         "path":"/user/verify-mobile",
         "referrer":"https://accounts.livspace.com/",
         "title":"Verify Mobile",
         "url":"https://accounts.livspace.com/user/verify-mobile"
      },
      "locale":"en-GB"
   },
   "event":"accounts_lead-form:otp_verified",
   "integrations":{
      
   },
   "properties":{
      "name":"Pinaki Damian Paul",
      "customerId":"1192900",
      "event_name":"accounts_lead-form:otp_verified",
      "userid":"CX1192900",
      "email":"paul.pinaki@gmail.com",
      "phoneNo":"9433057867",
      "leadId":"1343802"
   },
   "timestamp":"2022-03-03T00:35:07.000Z"
}

Setting up AWS S3 destination in Lemnisk

You will have to perform the following steps in the Lemnisk app to add AWS S3 as a destination.

Navigate to CONNECT > Connections > Destinations and click +Create Destination.

Step 1: A Catalog pop-up appears. Select Amazon S3 and click Continue2

Step 2: Select a source of your choice by checking the box adjacent to the source and then click Continue.

Step 3: Type a unique name that acts as your destination identifier. Click Create Destination.

Step 4: The Overview tab appears with the details of the destination you just created.

Step 5: Toggle the Status button to activate/ deactivate the destination.

Step 6: Click +Add Sources and select a source(s) of your choice by checking the box adjacent to the source and then click Continue.

Step 7: Type a unique name that acts as your bucket identifier. Click Update Destination.

You can add a custom prefix path before typing the bucket name in the Bucket Name text box. For example, if AbcdS3 is your bucket name, and AWS is the prefix. The bucket name along with the prefix would be AWS AbcdS3.

FAQs

​Q1. What connection does the AWS S3 work on?

The AWS S3 is available in Cloud mode for both web and mobile.

Q2. How do I configure AWS S3?

You will have to log in to your Amazon AWS S3 console and then create a new S3 bucket (you can also choose an existing one, if any).

Q3. How is the data processed in Lemnisk?

The data is processed from your sources and events are collected in batches by Lemnisk. Once every hour, a process is initiated where the batches are uploaded to a Lemnisk S3 bucket, and then securely copied to your own S3 bucket. However, there are no limitations to the data size.

Q4. How do I set up AWS S3 as a destination in Lemnisk?

You will have to perform the following steps in the Lemnisk app to add AWS S3 as a destination.

  • Navigate to CONNECT > Connections > Destinations and click +Create Destination.

  • A Catalog pop-up appears. Select Amazon S3 and click Continue.

  • Select a source and then click Continue.

  • Type a unique name and click Create Destination.

  • The Overview tab appears with the details of the destination you just created.

  • Toggle the Status button to activate/ deactivate the destination.

  • Click +Add Sources. Select a source(s) of your choice and then click Continue.

  • Type a unique name and click Update Destination.

Q5. How to add a prefix path to the destination name?

You can add a custom prefix path before typing the bucket name in the Bucket Name text box. For example, if AbcdS3 is your bucket name, and AWS is the prefix. The bucket name along with the prefix would be AWS AbcdS3.

PreviousAppsFlyerNextCriteo

Last updated