# Advanced Usage Reporter Data Transformation

The goal is to leverage the use of Amazon Kinesis Firehose Data Transformation with AWS Lambda, to process our incoming data streams and convert it into our desired format, before delivering it to the target destination.

[toc]

## Record Processing
Create a Firehose processing Lambda function that will process the incoming data.

## Record Format Conversion

## Deserializer
The deserializer used to read the JSON of the input data stream.
- **OpenX JSON SerDe** (works with our timestamp formats)

## Schema
Create **AWS Glue Table**.

## Serializer
The serializer used to convert the data to the target format.
- **Parquet SerDe** (for Apache Parquet)

## Backups
We should store backups of our raw (unprocessed) data streams in S3 for future processing.
- Enable source record backups

## Sources
- https://aws.amazon.com/blogs/compute/amazon-kinesis-firehose-data-transformation-with-aws-lambda/
- https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html#console-to-s3
- https://docs.aws.amazon.com/firehose/latest/dev/create-transform.html

###### tags: `shift` `advanced-usage-reporter`