Skip to main content

Ingestion by S3 Bucket

This procedure describes the prerequisites for configuring an AWS S3 Bucket for data ingestion.

Prerequisites​

Permissions​

  • Administrator role or equivalent in an AWS account dedicated to monitoring

Functioning​

Architectural Diagram - AWS MonitoringArchitectural Diagram - AWS Monitoring

Diagram of how some AWS connectors work

Steps to configure an S3 Bucket​

Variables to be determined​

Here is the list of variables that you will encounter during the procedure:

  • {{S3AuditBucketName}}: The name of the S3 bucket to create


    caution

    The name is unique for the entire AWS environment

  • {{Region}}: The region in which the bucket must be created

  • {{MondataOrgID}}: organization ID provided by Mondata to assign reading rights in the Bucket

  • {{KMSDataEncryptionKeyARN}} (Optional): ARN of the key used to encrypt the logs

Creation of the S3 bucket​

caution

If the S3 bucket is already configured, ensure that the Object Ownership configuration has the value BucketOwnerEnforced.

aws s3api create-bucket --bucket {{S3AuditBucketName}} --region {{Region}} --object-ownership BucketOwnerEnforced --create-bucket-configuration LocationConstraint={{Region}}

Block public access​

aws s3api put-public-access-block --bucket {{S3AuditBucketName}} --public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

Enable encryption​

aws s3api put-bucket-encryption --bucket {{S3AuditBucketName}} --server-side-encryption-configuration '{
     "Rules": [
         {
             "ApplyServerSideEncryptionByDefault": {
                 "SSEAlgorithm": "AES256"
             }
         }
     ]
}

Activate the lifecycle (90 day retention)​

aws s3api put-bucket-lifecycle --bucket {{S3AuditBucketName}} --lifecycle-configuration '{
    "Rules": [
        {
            "Expiration": {
                "Days": 90
            },
            "Prefix":"",
            "ID": "90 days retention",
            "Status": "Enabled",
            "NoncurrentVersionExpiration": {
                "NoncurrentDays": 90
            }
        }
    ]
}'

Add a basic policy​

Allows Mondata to collect the logs, to which will be added the configuration for each service which will carry logs into the bucket.

{
    "Statement": [
        {
            "Sid": "Allow mondata to fetch objects from the bucket",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::{{MondataOrgID}}:role/mondata-log-fetching-role"
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::{{S3AuditBucketName}}/*",
                "arn:aws:s3:::{{S3AuditBucketName}}"
            ]
        }
    ]
}

Optional: Personalized KMS key​

Added a statement in the key policy to allow Mondata to decrypt the contents of the bucket when key management is done with company-managed keys. Not required if keys are managed by AWS.

GuardDuty can use this configuration.

{
    "Sid": "Allow mondata to use the key",
    "Effect": "Allow",
    "Principal": {
        "AWS": [
            "arn:aws:iam::{{MondataOrgID}}:role/mondata-log-fetching-role"
        ]
    },
    "Action": [
        "kms:Decrypt"
    ],
    "Resource": "{{KMSDataEncryptionKeyARN}}"
}

Take notes of information to configure integrations​

  • {{S3AuditBucketName}}