AWS Kinesis

The AWS Kinesis connector is a specialized integration within the Netmera platform, falling under the Data Streaming category. Its core purpose is to enable the real-time export of customer events and engagement data from Netmera directly into Amazon Kinesis Data Streams for high-throughput processing and integration with the wider Amazon Web Services (AWS) ecosystem.

Overview

Attribute	Details
Connector Name	Kinesis
Category	Data Streaming / Data Flow Connector
Provider	Amazon Web Services (AWS)
Primary Function	Real-time transmission of Netmera events (Hooks) to Kinesis Data Streams.
Netmera Component	KinesisHookSender class
Data Flow	Outbound (Netmera > AWS Kinesis Stream)
Connection Type	AWS SDK (REST API using configured AWS credentials)

The Kinesis integration is activated and configured via the Netmera Connectors panel, but its operational use is tied directly to the Hooks module, allowing clients to define specific triggers (events, message logs) that should be streamed immediately to AWS.

Use Cases and Benefits

The primary benefit of integrating AWS Kinesis is facilitating high-volume, real-time data movement, enabling immediate consumption by external systems.

1. Real-Time Analytics Pipelines: Events captured by Netmera (e.g., app opens, purchases, location updates) can be streamed instantly into Kinesis, where they can be consumed by downstream processing systems for immediate analysis.

2. Data Warehouse/Data Lake Feeding: Kinesis is commonly used to feed massive streams of operational data directly into centralized storage solutions (like Amazon S3, Redshift, or other data warehouses) for long-term storage and complex reporting.

3. AWS Lambda Function Triggering: The data stream can directly trigger AWS Lambda functions based on event occurrences, enabling real-time automation outside the Netmera platform.

4. High-Throughput Data Transfer: Kinesis is architecturally designed to handle vast amounts of data volume, making it suitable for Netmera clients that generate significant messaging or usage event payloads.

5. Big Data Integration: It provides a necessary layer for clients engaged in comprehensive Big Data processing and enterprise-level AWS ecosystem integration.

Data Flow

The AWS Kinesis connector relies on the Outbound Hook mechanism to transfer data.

1. Hook Processing: The Hook module's consumers intercept the event. If the configured hook type is KINESIS, and the Kinesis connector is active, the process proceeds.

2. Payload Generation: Netmera converts the event data, user attributes, and identities into a JSON format payload.

3. API Submission: The dedicated KinesisHookSender uses the configured apiKey, secretKey, and region to make an authenticated call to the AWS Kinesis API using the PutRecords method.

4. Partitioning: Crucially, Netmera uses the User ID (userId) of the affected user as the Partition Key when writing records to the stream. This ensures that records belonging to the same user are generally directed to the same shard, maintaining transactional ordering for individual users within Kinesis.

$$\text{Netmera Event / Message Log} \xrightarrow[\text{JSON payload; partition key: UserId}]{\text{KinesisHookSender}} \text{AWS Kinesis Stream}$$

Configuration Reference

The connector configuration is brief but requires sensitive AWS credentials.

Parameter	Type	Description	AWS Context
region	String	The AWS region where the Kinesis stream is hosted (e.g., eu-central-1).	Defines the target API endpoint.
apiKey	String	The AWS IAM user's Access Key ID.	Required for programmatic API authentication.
secretKey	String	The AWS IAM user's Secret Access Key.	Required for secure API authentication.

These credentials must belong to an IAM user granted permissions to perform PutRecord actions (or full access) on the target Kinesis stream.

Setup Instructions

The setup process requires actions on both the AWS side and the Netmera Control Panel.

Step 1: AWS Prerequisites

1. Access the AWS Console and navigate to the IAM service.

2. Create a dedicated IAM user with Programmatic Access and assign a policy allowing access to Kinesis (e.g., AmazonKinesisFullAccess or a custom policy for specific stream writing).

3. Obtain and securely store the Access Key ID (apiKey) and Secret Access Key (secretKey).

4. Navigate to the Kinesis service and create a Data Stream, noting its AWS Region and Stream Name.

Step 2: Configure in Netmera Connector

1. Navigate to the Netmera Control Panel and go to the Connectors section.

2. Select the Kinesis connector (via the /connector/kinesis endpoint).

3. Input the obtained Region, API Key, and Secret Key.

4. Click Save Configuration.

Step 3: Configure the Hook

1. Navigate to Settings>Hooks and create a New Hook.

2. Set the Hook Type to Kinesis.

3. Specify the Stream Name (obtained in Step 1) in the Hook configuration details.

4. Define the Trigger Type (e.g., EVENT, MESSAGE_LOG) and the specific event code that should trigger the stream.

Usage Notes & Considerations

1. AWS Credential Rotation: The integration relies on static API keys. Any required rotation of the AWS IAM user credentials must be handled manually by updating the configuration in the Netmera Control Panel.

2. Connector Dependency: The Kinesis integration functions exclusively as a Hook Type. If the underlying Kinesis connector configuration is set to inactive, no hook submissions will occur, even if the individual hooks are correctly configured.

3. API Quota and Cost: AWS charges based on shard usage and data volume. Clients must actively monitor their Kinesis usage and shard configuration to manage costs and avoid potential rate limiting imposed by AWS.

4. Payload Handling: For particularly large event payloads, the architecture encourages the use of batch processing (defined in the Hook configuration) to optimize performance and adhere to Kinesis record size limits.

5. Stream Name Management: While the apiKey and region are set at the connector level, the crucial Stream Name is defined and managed separately within each individual Hook configuration in the Netmera UI.