# Time-Series Notes

## Database

- DynamoDB
- Timestream

## DynamoDB


- Primary Key = Partition Key + Sort Key.
- Sort Key is optional.
- Primary must unique.
- Query Example:
- Query movie's data
- PartitionKey = TheDarkKnight AND SortKey = Year#2005


- No Join.
- A single table design.
- Schemaless.
- Item:
- Maximum size of Item: 4 KB.
- Query Example:
- Query movies's data and actors
- PartitionKey = TheDarkKnight
- Only query movies's actors
- PartitionKey = TheDarkKnight AND BEGINS_WITH(SortKey, 'Actor#')


- Query Example:
- Query Christopher Nolan's movies.
- IndexName = GlobalSecondIndex AND PartitionKey = ChristopherNolan


- Maximum Number of Global Second Index: 20.
- Maximum Number of Local Second Index: 5.
- Same Partition Key.
- Maximum size of Item Collection: 10 GB.
- Eventually consistent vs Strongly consistent.
- Query Example:
- Query Christopher Nolan's movies have Christian Bale.
- IndexName = GlobalSecondIndex2 AND PartitionKey = ChristopherNolan AND BEGINS_WITH(SortKey, 'Actor#ChristianBale#')

### Key Condition Expression

- Key Operator: =, <, <=, >, >=, BETWEEN, begins_with
- Query by Date Range

| PK | SK |
| --- | ------------------------ |
| 123 | 2022-04-20T08:18:23.351Z |

- Key Condition Expression
- PK = 123 AND SK < '2022-04-20T08:00:00.000Z'
- PK = 123 AND SK > '2022-04-01T08:00:00.000Z'
- PK = 123 AND SK BETWEEN '2022-04-01T00:00:00.000Z' AND '2022-04-20T08:00:00.000Z'

| PK | SK |
| --- | ------------------------------ |
| 123 | ADMIN#2022-04-20T08:18:23.351Z |

- Key Condition Expression
- PK = 123 AND SK < 'ADMIN#2022-04-20T08:00:00.000Z'
- PK = 123 AND SK > 'ADMIN#2022-04-01T08:00:00.000Z'
- PK = 123 AND SK BETWEEN 'ADMIN#2022-04-01T00:00:00.000Z' AND 'ADMIN#2022-04-20T08:00:00.000Z'

### Filter expressions

- Filter on any attributes.
- Key Operator: =, <, <=, >, >=, BETWEEN, begins_with
- Filter Operator<> (not-equals), OR, CONTAINS, IN

---

1. Read items from table (a maximum of 1MB of data)
2. If FilterExpression remove itmes that don't match
3. Return items

| PK | SK | Attributes: Name | Attributes: Value |
| --- | ------------------------ | ---------------- | ----------------- |
| 123 | 2022-04-20T08:18:23.351Z | Hats | 100 |
| 123 | 2022-04-21T08:18:23.351Z | Gloves | 200 |

- Filter Condition Expression
- Name <> Hats AND Value > 99
- No Limit.
- N + 1 Query Problem.
- 10 ms * n MB / 1 MB
- Example:
- 10ms(Query Latency) * 10 MB / 1 MB = 100 ms + (Network Latency)

### Pagination

- Offset Pagination vs Cursor Pagination

### Performance at scale

- Serverless.
- Single-digit millisecond read and write performance.
- Microsecond latency with DynamoDB Accelerator(DAX).
- Time-to-live (TTL), DynamoDB Streams


## Timestream (Time Series Database)

- use cases
- IoT applications
- DevOps analysis
- App analysis



### Dimensions and measures

- Dimensions
- Dimension names and values cannot be updated, measure value can be.
- The maximum number: 128.
- Maximum size of dimension value: 2 KB.
- Support datatype: VARCHAR.
- Measures
- Modeling the ID attribute as a measure value will result in significantly better query latency.
- The maximum number of measures: 8192.
- Support datatype: IGINT, BOOLEAN, DOUBLE, VARCHAR, MULTI, TIMESTAMP.

### Limit

- Timestamp: No more than 35 minutes into the future.
- No record deletion.
- Only upsert measure value.
- No way to change the schema datatype.
- No backup / restore functionality.
- Eventual consistency for reads.

### Duplicate Records

- Amazon Timestream has detected duplicate records. Records are marked as duplicate, when there are multiple records with the same dimensions, timestamps, and measure names but:
- Measure values are different.
- Version is not present in the request or the value of version in the new record is equal to or lower than the existing value. If Amazon Timestream rejects data for this reason, the ExistingVersion field in the RejectedRecords will contain the record’s current version as stored in Amazon Timestream. To force an update, you can resend the request with a version for the record set to a value greater than the ExistingVersion.

### Architecture



### Storage
- Memory Store
- high throughput data writes.
- fast point-in-time queries.
- Magnetic Store
- lower throughput late arrival data writes
- long term data storage
- fast analytical queries.
- Retention policies
- move data from the memory store to the magnetic store.
- When data expires out of the magnetic store, it is permanently deleted.

### Performance and scale

- You can run queries that analyze tens of gigabytes of time-series data from the memory store within milliseconds.
- Analytical queries that analyze terabytes of time-series data from the magnetic store within seconds.