---
# System prepended metadata

title: Client Configuration Auditing
tags: [APM, SRE]

---

---
title: 'Client Configuration Auditing'
disqus: hackmd
---


<!-- ![downloads](https://img.shields.io/github/downloads/atom/atom/total.svg)
![build](https://img.shields.io/appveyor/ci/:user/:repo.svg)
![chat](https://img.shields.io/discord/:ido.svg)
 -->
Client Configuration Auditing 
===
[TOC]

# What is it?

configuration auditing is about recording any client I/O configuration changes via the APM Agent for data analysis and alerting. 

# Architecture 

```plantuml
!define Background   #2d2d2d
!define CurrentLine  #393939
!define Selection    #515151
!define Foregound    #cccccc
!define Comment      #999999
!define Red          #f2777a
!define Orange       #f99157
!define Yellow       #ffcc66
!define Green        #99cc99
!define Aqua         #66cccc
!define Blue         #6699cc
!define Purple       #cc99cc

skinparam Shadowing true
skinparam handwritten true

skinparam backgroundColor #2d2d2d
skinparam Arrow {
  Color Foregound
  FontColor Foregound
  FontStyle Bold
}
skinparam Default {
  FontName Menlo
  FontColor #fdfdfd
}
skinparam package {
  FontColor Purple
  BackgroundColor CurrentLine
  BorderColor Selection
}
skinparam node {
  FontColor Yellow
  BackgroundColor CurrentLine
  BorderColor Selection
}
skinparam component {
  BackgroundColor Selection
  BorderColor Blue
  FontColor Blue
  Style uml2
}
skinparam database {
  BackgroundColor CurrentLine
  BorderColor Selection
  FontColor Orange
}

skinparam folder {
  BackgroundColor Selection
  BorderColor Blue
  FontColor Blue
  Style uml2
}

skinparam cloud {
  BackgroundColor CurrentLine
  BorderColor Selection
}

skinparam interface {
  BackgroundColor CurrentLine
  BorderColor Selection
  FontColor Green
}

skinparam actor {
  BackgroundColor CurrentLine
  BorderColor Blue
  FontColor Orange
}

title Configuration Auditing Architecture 
database "kafka" {
    folder "topics" {
        folder "Auditing" as auditing
    }
}


node "Common Operational Picture" {
    database "Audit History DB" as db_audit_history {
      folder "entities" {
        Folder http
        Folder "cosmos DB" as cosmos_db
      }
    }
    
    database "Dependency Graph" as db_dependency_graph
}

node "WCNP" {
    folder "WCNP App"  {
        [Agent] as wcnp_agent
    }
    
    folder "Functions" {
      [Audit Consumer] as audit_consumer
      [Auditing http endpoint] as http_endpoint_auditing
    }
    
}

node "One Ops" {
    folder "One Ops App"  {
        [Agent] as oneops_agent
    }
}

actor "Data Scienctist" as actor_data_scienctist

component reports


wcnp_agent --> http_endpoint_auditing : https:443
oneops_agent--> http_endpoint_auditing : https:443
actor_data_scienctist --> db_audit_history : port?
actor_data_scienctist --> reports : port?
http_endpoint_auditing --> auditing : kafka port ?
auditing --> audit_consumer : port?
audit_consumer --> cosmos_db : port?
audit_consumer --> http : port?

@enduml
```

## Components
### WCNP
### One Ops
### Kafka/Messaging
### Common Operational Picture
### Agent
### Auditing Http Endpoint
### Auditing Consumer
### Audit History DB 
### Dependency Graph

## Open questions

- Q: Agent: was also considered something on lower level like [eBPF](https://ebpf.io/what-is-ebpf)? - @Miroslav 
  A: I'm interested in that. I think this can be used later on... I have taken a look at https://github.com/redsift/redbpf - @isamuelson 

- Q: Kubernetes supports sidecars which can intercept the traffic. They can be used to solve this part. Example: https://developers.redhat.com/blog/2019/02/27/sidecars-analyze-debug-network-traffic-kubernetes-pod/
  A: @ameterko can you explain which requirements this will answer? I might explained it wrong, we do not need to analyze network traffic...more of how the connection was configured. - @isamuelson 
  UPD: Seems that I did not capture that correctly - we would not need it for configuration audit (@ameterko)
  
- Q: Java has a ton of different HTTP client libraries with their own implementations. Should we just pick the most popular ones and track only them in the beginning? @ameterko   

- Q: Kafka configuration audit - which info should it scrape? Insights on the brokers, partitions and other server related data might be already available for us via Lenses. @ameterko 

# Requirements 

## Functional
- Able to query history of last 24 hours or 7 days 
- Data science will look at the database, created detection rules will need to run in realtime and on the database
- History Database is for reports
- Agent need to hook to function to understand the usage, configuration, etc
- Agent is now reported structured, should not depends on any lib. Most likely web hook that is highly scalable. Webhook can now write to Kafka for processing. If web hook isn’t available an retry with backoff of minutes.
- Agent Web hook function and agent caller must be lean 
- Outliers - how do we report outliers? Data science will look at the database, created detection rules will need to run in realtime and on the database For example, http client, if a call takes 10 seconds , it’s an outlier. We would want to report it as structured and not metric.
- Timestamp needs to be accurate from client side 
- Defensive against abuse of the client (dedup, remove duplications) , how to detect anti pattern of creating a http client per request , configuration should be reported only once but client creation should be a metric
- Agent configuration via dns ? audit.sre.walmart.com , how do we handle env, maybe env variable? What is the most lean way?
- Java, nodeJS, .NET support 

## Non Functional
- Millions of records per entity (entity can be Kafka,  cosmos DB, etc) 
- Rather query then update a current - no upsert 
- Soft realtime for [Common Operational Picture](https://en.wikipedia.org/wiki/Common_operational_picture) 
- Elements of a data warehouse (map reduce? Analytics?)
- Day to day , search should be the current configuration of any of the services
- Reliability 99%
- Push model (to start)

# Architecture Leads

| Component        | Lead                | Advisors            |
|:---------------- |:------------------- |:------------------- |
| Agent            | Slava Lapitski      |                     |
| Dependency Graph | Dmytro Diukariev    |                     |
| Audit History DB | Vladislav Zablotsky | Christoph Leinemann |
| Kafka            | Dmytro Diukariev    |                     |
| Node JS Agent    | ?                   |                     |
| .NET Agent       | ?                   |                     |


## Decomposition proposal by @Miroslav - need review
```plantuml

package "Processing and data aggregation" {
    component "Dependency Graph" {
        component "Dependency Graph Feed" as dg_feed
    }
    component "Playbook Profiles" {
        component "Active Profiles Feed" as ap_feed
    }
    package "Runtime Auditing" {
        package Agents as agent {
            component "Jvm Agent"
            component ".Net Agent"
            component "Node.js Agent"
        }
    
        component "Auditing Endpoint" {
            component "Http endpoint" as ahp {
              component "Auditing protocol" as protocol
            }
            component "Runtime Audit Feed" as ra_feed
        }
        agent -u-> ahp: uses
    }
}

package "Common Operational Picture" as cop {
    component "Audit History DB"
}

cop --> dg_feed: consumes
cop --> ap_feed: consumes
cop --> ra_feed: consumes
```

| Module | Description |
|--------|-------------|
| **Common operational Picture** | TODO |
| **Dependency Graph** | TODO |
| **Playbook Profiles** | TODO |
| **Runtime Auditing** | TODO |

Enterprise Architect Lead - Ido Samuelson
Stake Holder: Yoseph Reuveni

###### tags: `APM` `SRE`
