<!-- .slide: data-background="https://go.justmiles.io/f/686f794e?download"--> # CloudQuery Sync any source to any destination. --- ## What is this thing?  cloudquery.io Note: - Like many software projects, they're capabale of more than they're useful for. - We're not going to rewrite all our ETL to be used with cloudquery. ---- - software that helps simplify data access - particularly useful for extracting resources from APIs Note: - Take an API, any API, iterate over the resources, store the results in a DB --- ## Why use CloudQuery? ---- **Asset Inventory** - Ingest AWS resources - anything with an ARN, huck it into postgres <!-- .element: class="fragment" --> - Ingest CrowdStrike resources (vulnerabilities, devices, etc) <!-- .element: class="fragment" --> - JumpCloud? Sophos? <!-- .element: class="fragment" --> - Unified view across multiple resources <!-- .element: class="fragment" --> ---- **Compliance reporting** - Unencrypted EBS volumes? <!-- .element: class="fragment" --> - S3 buckets with public acccess? <!-- .element: class="fragment" --> - Standard Cloud best-practices in one view <!-- .element: class="fragment" --> --- ## Sources ----  ----  ----  ----  ----  --- ## Destinations ----  ----  --- ## How CloudQuery Works - Connects to a source - variety of API connections - Extracts to a destination - handles the schema - Written in Go; plugin architecture for sources and destinations - When writing a plugin, only need to translate the resource into the SDK's struct - Simple YAML configs ---- ```yaml kind: source spec: name: "crowdstrike" registry: "github" path: "justmiles/crowdstrike" version: "v0.0.3" destinations: ["sqlite"] --- kind: destination spec: name: sqlite path: cloudquery/sqlite version: "v2.2.0" spec: connection_string: ./db.sql ``` --- ## CloudQuery DEMO - metabase.ops.gofigg.net --- ## Honorable Mention ---- SteamPipe - https://steampipe.io ```sql select runtime, count(*) from aws_lambda_function group by runtime; +------------+-------+ | runtime | count | +------------+-------+ | nodejs12.x | 1 | | python3.7 | 1 | | python3.8 | 2 | +------------+-------+ ``` ---- - SteamPipe creates a PostgreSQL plugin for APIs - Instead of storing the data, it pulls it realtime - via SQL! - Why not use this?? - API Rate Limits - Ability to join across data sources - CloudQuery provides historical context --- ### Challenge 1. Install steampipe: https://steampipe.io/downloads 2. Install AWS Plugin ``` steampipe plugin install aws ``` 3. Launch the query editor ``` steampipe query ``` 4. Solve the Challenge and Slack me the answer and the query you used to solve it. First correct answer wins. ---- How many ECR images do we have in the FI account? Hint: `aws_ecr_?` --- Which ECR repository has the most images? --- ## Challenge Answers ---- How many ECR images do we have in the FI account?  ---- Which ECR repository has the most images?  --- ## Resources - https://www.cloudquery.io/docs/developers/creating-new-plugin