Use manifest files with EI data file retrieval

If you need to identify when your Enterprise Insights (EI) data exports are ready, you have two options: Amazon SNS notifications or manifest files. This article explains how to use manifest files as a polling-based, file-driven alternative to SNS, and provides guidance for retrieving and interpreting your export data.

On this page

• When to use manifest files
• How manifest files work
• How to use a manifest file
• Sample manifest file
• Notes

When to use manifest files

Enterprise Insights supports two primary methods for export identification:

• SNS notifications: Event-driven, requires AWS service setup
• Manifest files: Polling-based, requires no AWS notification configuration

While SNS notifications are ideal for automated, event-driven workflows, manifest files offer a flexible, self-contained alternative. By regularly checking for a manifest file in your S3 export directory, you can reliably determine when new data exports are available, without subscribing to SNS topics or managing AWS notification services.

Manifest files are especially useful if you:

• Prefer a file-based workflow
• Need persistent audit trails
• Operate in environments where SNS integration is not feasible

How manifest files work

Each time an export is generated, a manifest file, manifest.json, is placed in your S3 export directory. This file contains all the metadata you need to identify available data files, including:

• Export type (full or incremental)
• ETL job IDs
• Timestamps
• Direct S3 paths to exported data

The manifest file is always in the customer root folder and overwrites with each delta or full load export. It retains critical information from the previous run. By polling for the manifest file, you can track both full and incremental exports and retrieve the latest data as soon as it’s available.

How to use a manifest file

1. Navigate to your S3 export bucket
   Example path:

   s3://align-ei-CUSTOMERNAME-export/manifest.json

2. Download or read the manifest.json file
3. Parse the JSON to extract key information:
   • last_generated_utc: Last full export time, in coordinate universal time
   • latest_full_load: Details of the latest full export, including the S3 path and ETL job ID
   • deltas_since_latest_full_load: S3 path and ETL job IDs for all incremental exports since the last full load
   • objects: List of exported tables and their S3 paths
4. Use the S3 paths to retrieve your data files

Sample manifest file structure

Manifest file code sample

{
  "last_generated_utc": "2025-10-20 06:34:59",
  "latest_full_load": {
    "path": "s3://align-ei-CUSTOMERNAME-export/2025/10/20/full-export/",
    "etl_id": 50085
  },
  "deltas_since_latest_full_load": {
    "path": "s3://align-ei-CUSTOMERNAME-export/2025/10/20/delta-export/",
    "etl_ids": [50086, 50087, ..., 50098]
  },
  "objects": [
    {
      "object_name": "About",
      "s3_path": "s3://align-ei-CUSTOMERNAME-export/2025/10/20/daily-export/delta/About/etl_id=50098"
    }
    // ... more objects
  ],
  "export_objects_list_manifest": "2025-10-20 19:21:50"
}

Key fields

• last_generated_utc: UTC timestamp of the last full export.
• latest_full_load: S3 path and ETL job ID for the latest full export.
• deltas_since_latest_full_load: S3 path and an array of ETL job IDs for all incremental exports since the last full load.
  • Note: The last ETL ID of the array is the latest delta load ID.
• objects: Array of exported tables/entities and their S3 locations.
• export_objects_list_manifest: UTC timestamp when the latest ETL job was completed.

Notes

• The manifest file always resides in the customer root folder and is overwritten with each export.
• All object_name entries in the objects array reference the latest ETL ID, representing the most recent delta export.

• S3 paths follow a date-based hierarchy:

  s3://align-ei-CUSTOMERNAME-export/YYYY/MM/DD/daily-export/export-type/object/etl_id=XXXXX

• After a new full export, the deltas_since_latest_full_load array resets.