---
# System prepended metadata

title: data exam notes

---

# data exam notes

### outlier analysis in time series

when accidents, high traffic, or some annual events, there may be lots of data which is abnormal. you can find such an abnormal samples from time series analysis.

https://pro.arcgis.com/en/pro-app/latest/tool-reference/space-time-pattern-mining/understanding-outliers-in-time-series-analysis.htm

### Apache Parquet

parquet is a data format that can be used for column-based data rather than row-based data like csv.

> why it's speriror when column-based data?
> this format is intended to use for aggregated query for large databases. each node can skip task when they don't have related data.


https://databricks.com/glossary/what-is-parquet

https://learning.oreilly.com/library/view/mastering-hadoop-3/9781788620444/9e6d6a8c-a4fe-4d25-8e1a-8c6aafe024f2.xhtml

### Avro

Avro is a serialized file format that remove dependency of specific language of Hadoop. which mean the Avro data which written in one languate, it can be read in the other languages.

Avro provides as followes:

- Language portability
  - the Avro data which is written in one languate, it can be read by the other languages.
- schema evolution
    - you don't need to change query or pipeline after change schema for the next ingestion. 
- splitting and compression