# Fast Data Pipeline Notes https://lucid.app/lucidchart/43f8ab92-1eab-417f-8869-0fddc2111353/edit?page=0_0# https://hevodata.com/learn/data-pipelines/#dp_vs_etl https://hazelcast.com/glossary/stream-processing/ Going Forward, do we have an opinion on the technology used for a Fast Data Pipeline What do I need to Monitor for Fast Data Pipelines? System Performance/ Availability / Applicvation performance / Throughput Data Stream Processing:  In a streaming data pipeline, data from the point of sales system would be processed as it is generated. The stream processing engine could feed outputs from the pipeline to data stores, marketing applications, and CRMs, among other applications, as well as back to the point of sale system itself.  Tooling and Technologies - ETL Tools: Hevo, Informatica PowerCenter, Talend Open Studio, Apache Spark - Data Warehouses: Amazon Redshift, Google BigQuery - Stream Data Processing: Flink, Apache Spark, Apache Kafka, et - Batch Schedulers: Luigi, Airflow, Azkaban, Oozie - Data Lakes: IBM Data Lake, MongoDB Atlas Data Lake, etc. --- ## Overvierw Slide --- ## Attention Getter --- ## Use Cases & Industries: - Financial Market Analysis - Hospitality & Travel - Connected Device Telemetry - Medical Devices - Anomaly Detection - Retail (Brick/ Mortar & e-Commerce) - Marketing, Advertising --- ## Streaming Data Pipeline on Kubernetes Requirements - Easy to Deploy and Manage - Secure - Observable - Scaleable - Resilient - Financiually visible --- At D2iQ, we sinplify and Kubernetes and other Cloud-Native Technologies in production, on any infrastructure. Today, I would like to show you how automated ands simple it is to deploy a Real-Time Data Pipeline on Kubernetes