# DBT operator
Add data quality tests in our pipelines using custom dbt operators

## Current view of usual pipelines

This schema describes simplified view of regular pipelines that we usually build.


Try to sense source file, convert data to parquet format, validate it using GE tool (staging and validation can be swapped), load data to DV RAW, and apply transformation using dbt tool.

## Problem statement
We don't do data quality tests explicitely and we don't know do we have correct data in our views.

We alredy have Monte Carlo checks and alerts.This system helps us with monitoring data: we check the granularity, fresheness, etc. But we don't do tests against the source files such as sums of aggregated data for the specific periods, the amount of data in the business view.

## How it will look like
*The circle represents the first steps of the pipeline*


One of the possible solutions is to add a dbt test operator in the platform.

CDNA team already do it in their pipelines. Thanks our college for share [code of dbt test operator](https://github.com/pepsico-ecommerce/dpide-az/blob/142cadf7e11f293a8b2ff9edade0c2fe992fa297/dags/common/dbt/dbt_operator_extended.py#L60).

**Be aware: `trigger dbt operator` task is not task that triggers `dbt for change models pipeline`**

- `trigger dbt operator` under hook runs the command `dbt run -m model_name`.

- `dbt test operator` runs the test command
`dbt test --select tag:test_id:__unique_test_id__`

DBT supports [tagging](https://docs.getdbt.com/reference/resource-configs/tags) and very useful feature that may help to run the specific sets of tests/models.

With dbt we can do [custom](https://docs.getdbt.com/guides/best-practices/writing-custom-generic-tests) flexible tests.

## Alternatives

We can use Monte Carlo tool for testing but I tend to think that this tool is more about monitoring not quality testing. And monitoring should be differentiated from testing.

Use separate pipeline for do quality testing for all models.

Create bash script that will run dbt test command on the schedule.