High-Level Design: Azure Key Vault Secret & Certificate Rotation Management

# High-Level Design: Azure Key Vault Secret & Certificate Rotation Management **Version:** 1.4 **Date:** May 1, 2025 ## 1. Introduction ### 1.1. Purpose This document outlines the high-level design for a comprehensive system to manage the rotation of secrets and certificates stored within Azure Key Vaults across multiple tenants, implemented through a strategic phased approach. ### 1.2. Problem Statement Manual tracking and rotation of secrets and certificates is error-prone, time-consuming, and increases the risk of service disruptions due to expirations. The lack of clear ownership, inconsistent documentation standards, and scattered information across various platforms (JIRA, Confluence, Emails, tribal knowledge) hinders efficient and timely rotation, potentially impacting business operations. ### 1.3. Goals The primary goal is to establish a robust, increasingly automated framework that ensures timely and efficient rotation of Key Vault items. This will be achieved incrementally by: #### Phase 1 Clarifying ownership, standardizing processes and documentation, and centralizing foundational information. #### Phase 2 Introducing automation for monitoring, visibility, and basic alerting, providing a Tactical/Strategic view. #### Phase 3 Implementing advanced workflow automation to support engineers during active rotation, providing an Operational view. #### Phase 4 Enabling end-to-end automated rotation for suitable secrets and certificates, minimizing manual intervention. Overall, reducing operational risk, improving efficiency, and establishing a sustainable, highly automated management practice. ## 2. Objectives * SRE is a first-class citizen, and SREX is in priority. * We need to Establish Clear Ownership and define and maintain an accurate, accessible record of the owner/SME for each Key Vault and its critical contents. * Preferrably, we need to clarify architectural dependencies (e.g. that could be just a placeholder secret sitting in KV for notification purposes only, or vise-versa be a part of automation pipelines, etc.) * And Centralize Rotation Information to create a single source of truth identifying items needing rotation, their owners, expiration dates, dependencies, and standardized rotation instructions (manual and automated). We already have Confluence Page for that purpose, we can start working on improvements from there. * Clarify and Manage Tenant Access as we need to Track and Manage engineer *and service principal* access permissions across different Azure tenants required for performing rotation tasks. * Need to Streamline Rotation Execution and Provide engineers with automated tools offering both Operational (task-specific) and Tactical/Strategic (overview) views to expedite the rotation process, and implement fully automated rotation where possible. * Need to Implement Proactive Monitoring and Deploy a dashboard and alerting system to visualize upcoming expirations based on configurable thresholds and automatically trigger notifications, remediation workflows (e.g., JIRA ticket creation), or fully automated rotation attempts. * Need to Reduce Risk and Minimize the likelihood of service disruptions caused by expired secrets or certificates through proactive and automated rotations. * And Improve Efficiency and Significantly decrease the manual effort required to track, prepare for, and execute rotations. ## 3. Proposed Solution Overview The solution is designed around core pillars, delivered across multiple phases: #### 1. Foundation - Information & Access Management Establishing the baseline data structures and processes for tracking Key Vaults, their contents (secrets/certificates), designated owners, engineer/service principal access levels, and links to standardized rotation procedures (+ preferrably demistifying and clarifying architectural dependencies) (Primarily Phase 1). #### 2. Proactive Monitoring & Alerting An automated dashboard coupled with a notification system to provide continuous visibility into upcoming expirations and trigger preventative actions based on predefined rules. This primarily enables the **Tactical/Strategic View**. (Primarily Phase 2) #### 3. Operational Workflow Automation An automated system designed to assist engineers *during* the active rotation phase by aggregating and presenting relevant information on demand. This primarily enables the **Operational View**. (Primarily Phase 3) #### 4. Deep Automation - End-to-End Rotation Implementing automated scripts and workflows capable of performing the entire rotation process (generation, storage in KV, updating consuming applications) for specific, well-understood secrets/certificates. (Primarily Phase 4) ### 3.1. Engineer Perspectives: Operational vs. Tactical/Strategic Views The system is designed to provide engineers and managers with different levels of information depending on their current task: * **Operational View (Immediate Action Focus):** * **Purposed** to support the Engineer during the *active rotation* of a specific item, whether manual or overseeing an automated attempt. Focuses on answering "What do I need to do *right now* for *this specific* secret/certificate?", "Who are contact points / stakeholders" or "What is the status of the automated rotation for this item?", "Do I have all the necessary permissions to proceed". * **Provided by** primarily the **Operational Workflow Interface (Phase 3)**, triggered by an alert, ticket, or direct query. Enhanced with status monitoring of automated jobs in Phase 4. * **Key characteristics:** * Highly contextual, detailed information for a single item (owner, exact expiry, direct link to instructions, related tickets, dependencies, automation status/logs). * Aims for rapid execution or monitoring of the immediate task. * **Tactical/Strategic View (Planning & Overview Focus):** * **Purposed** to provide a broader overview of the rotation landscape for planning, prioritization, and status reporting. Focuses on answering "What is the overall health?", "What's coming up soon?", "Which items are automated?", "Are there any problem areas? (e.g. non-clarified items, new items on the landscape, etc.)", "Requires downtime?", "Has special procedural requirements?", etc.. * **Provided by** primarily the **Monitoring Dashboard (Phase 2)** and aggregated reports/metrics. Enhanced with automation status reporting in Phase 4. * **Key characteristics:** * Aggregated data, trends, lists of expiring items across various filters (timeframe, environment, owner, automation status), overall compliance status. * Supports planning upcoming work, identifying risks, and reporting to management. ## 4. Implementation Phases A phased approach will be adopted to deliver value incrementally and manage complexity. ### Phase 1. Foundation & Process Improvement * It's **Goal** is establish the single source of truth, clarify ownership, standardize procedures, and improve manual processes. * **Key Activities & Deliverables:** * Define and set up the **Central Information Repository** structure (e.g., Azure SQL, SharePoint Lists, Confluence Pages, etc.). * Populate **Table #1 (Item Inventory)** and **Table #2 (Engineer Access)**. *These tables serve as the primary, human-readable references in this phase, detailing what needs rotation, who owns it, how to rotate it (Table 1), and who has access to which environments (Table 2). See Section 6.1.1 for details.* This involves discovery (e.g., leveraging Azure Resource Graph for initial KV/Subscription lists), targeted SME interviews (prioritizing critical production vaults), and structured manual data entry. * Formally assign and document **Key Vault/Item Owners (SMEs)** in Table #1. * Establish the **Rotation Instructions Hub** (e.g., dedicated Confluence space). * Develop **standardized templates** for rotation playbooks/guides. * Populate the Hub with initial **rotation instructions** for critical items (collaborating with SMEs). * Define and document the **manual process** for keeping the repository and instructions updated. * Create the initial **Rotation Checklist** deliverable. * Define architectural depndencies graphs / diagrams (in narrative or/and visual formats). * Expected **Outcome** is a centralized inventory, clear ownership, standardized instructions, clear architecture, and improved (though still manual) tracking capability. Provides the foundational data but lacks automated operational or strategic views. ### Phase 2. Monitoring, Visibility & Basic Automation * It's **Goal** is automating data collection, providing visibility (**Tactical/Strategic View**), and implementing basic alerting. * **Key activities & deliverables:** * Implement the **KV Scanning Engine** to automatically discover KVs/items and update expiration dates in the Central Repository (Table #1). * Develop and deploy the **Monitoring Dashboard** (e.g. Power BI, Grafana, etc.) providing Tactical/Strategic views based on repository data. * Implement the basic **Alerting Engine** to send email notifications based on configurable expiry thresholds stored in the repository or configuration. * (Optional Stretch Goal) Implement basic **JIRA ticket creation** from the Alerting Engine. * Major expected **Outcome** is reduced manual effort for tracking expirations, proactive visibility via the dashboard (Tactical/Strategic view), and automated basic notifications triggering manual operational workflows. ### Phase 3. Workflow Automation & Context Aggregation * It's **Goal** is to streamline the active rotation process for engineers (**Operational View**) by automatically gathering context from multiple sources. * **Key activities & deliverables** are as follows: * Develop and deploy the **Context Aggregation Service** with integrations (API calls) to the Central Repository, Confluence, and JIRA. * Develop and deploy the **Operational Workflow Interface** (Web App, CLI, Chat Bot) providing the Operational View for engineers to query expiring items and view aggregated context. * Refine **JIRA integration** (linking tickets, potentially updating status). * (Optional Stretch Goal) Implement **Email archive searching** via API if deemed high value. * Primary **Outcome** is to be significant reduction in time spent by engineers gathering information via the Operational View, leading to faster and more consistent rotations. A more fully automated support system complementing the Tactical/Strategic view from Phase 2. ### Phase 4. Deep Automation (End-to-End Rotation) * It's **Goal** is to automate the entire rotation lifecycle for suitable, well-defined secrets and certificates, minimizing manual steps. * **Key activities & deliverables:** * Identify candidate secrets/certificates suitable for full automation (e.g., those with well-defined generation processes and automatable application updates). * Develop **Automated Rotation Scripts/Workflows** (e.g., using Azure Functions, Logic Apps, DevOps Pipelines) capable of: * Generating the new secret/certificate. * Storing the new version securely in Key Vault. * Updating consuming applications/services (e.g., App Service connection strings, AKS secrets) via APIs or deployment processes. * Performing validation checks post-rotation. * Logging results and updating status in the Central Repository and/or JIRA. * Enhance the **Central Repository (Table #1)** to track automation status (`Automated`, `Manual`, `Automation Eligible`) and link to automation scripts/logs. * Enhance the **Alerting Engine** to trigger automated rotation workflows directly for eligible items, potentially with an approval gate. * Enhance the **Operational Workflow Interface** and **Monitoring Dashboard** to display automation status, logs, and success/failure metrics. * Define clear **error handling and fallback procedures** for failed automated rotations (e.g., automatically create a high-priority JIRA ticket for manual intervention). * Update **Table #2** to include permissions required for service principals executing automated rotations. * Expected **Outcome** is fully automated, "zero-touch" rotation for a subset of secrets/certificates, drastically reducing manual effort and risk for those items. Improved overall efficiency and reliability. ## 5. High-Level Architecture (Target State - Achieved post-Phase 4) ```mermaid graph TB subgraph AzureEnvironment["Azure Environment"] subgraph AzureTenants["Azure Tenants"] KV1[("🔐 Key Vault 1")] KV2[("🔐 Key Vault 2")] KV3[("🔐 Key Vault N")] APP1[("💻 App Service / VM / AKS")] end end subgraph InformationSources["Information Sources"] JIRA[("📋 JIRA")] CONF[("📚 Confluence")] EMAIL[("📧 Email Archives")] TEAMS[("👥 Team Knowledge")] OTHER_DB[("🗄️ Existing DBs")] end subgraph RotationManagementSystem["Rotation Management System"] subgraph Foundation["Foundation (Phase 1+)"] DB[("🗃️ Central Repository Item Inventory (T1) Access Management (T2) Automation Status")] end subgraph AutomationTools["Automation & Tools (Phase 2, 3 & 4)"] SCAN[["🔍 KV Scanning Engine (P2)"]] AGG[["🔗 Context Aggregation Service (P3 - Operational)"]] ALERT[["🚨 Alerting Engine (P2+) Triggers Manual/Auto"]] UI[["📊 Monitoring Dashboard (P2 - Tactical/Strategic)"]] WF[["💻 Workflow Interface (P3 - Operational)"]] AUTO_ROT[["⚙️ Automated Rotation Workflows (P4)"]] end end subgraph UsersNotifications["Users & Notifications"] ENG[("👨💻 Engineers")] NOTIF[("📬 Email Notifications (P2)")] TICKETS[("🎫 JIRA Tickets (P2/P3/P4 Fallback)")] end %% Connections indicating primary build phase & interactions KV1 -->|P2| SCAN KV2 -->|P2| SCAN KV3 -->|P2| SCAN SCAN -->|P2| DB TEAMS -->|P1| DB JIRA -->|P3| AGG CONF -->|P3| AGG EMAIL -->|P3-Opt| AGG OTHER_DB -->|P1/P3| AGG DB -->|P1| AGG DB -->|P2| UI DB -->|P2| ALERT DB -->|P4| AUTO_ROT ENG -->|P3| WF WF -->|P3| AGG AGG -->|P3| WF WF -->|P3/P4| ENG ALERT -->|P2| NOTIF ALERT -->|P2/P3| TICKETS ALERT -->|P4| AUTO_ROT AUTO_ROT -->|P4-Updates KV| KV1 AUTO_ROT -->|P4-Updates App Config| APP1 AUTO_ROT -->|P4-Updates Status| DB AUTO_ROT -->|P4 Fallback-Creates Failure Ticket| TICKETS UI -->|P2| ENG ENG -->|P1 Manual/P4 Oversee| KV1 ENG -->|P1 Manual/P4 Oversee| KV2 ENG -->|P1 Manual/P4 Oversee| KV3 %% Styling style AzureEnvironment fill:#e6f3ff,stroke:#0078d4,stroke-width:3px style AzureTenants fill:#f0f8ff,stroke:#333,stroke-width:2px style InformationSources fill:#fff4e6,stroke:#ff8c00,stroke-width:3px style RotationManagementSystem fill:#f0fff0,stroke:#32cd32,stroke-width:3px style Foundation fill:#f5f5f5,stroke:#333,stroke-width:2px style AutomationTools fill:#fff0f5,stroke:#333,stroke-width:2px style UsersNotifications fill:#f0f0ff,stroke:#4169e1,stroke-width:3px classDef keyComponent fill:#ffe6cc,stroke:#d35400,stroke-width:3px classDef dataStore fill:#d5e8d4,stroke:#82b366,stroke-width:2px classDef service fill:#dae8fc,stroke:#6c8ebf,stroke-width:2px classDef user fill:#f8cecc,stroke:#b85450,stroke-width:2px classDef automation fill:#e6e6fa,stroke:#9370db,stroke-width:2px class DB,JIRA,CONF,EMAIL,OTHER_DB dataStore class SCAN,AGG,ALERT,UI,WF service class AUTO_ROT automation class ENG,NOTIF,TICKETS user class KV1,KV2,KV3,APP1 keyComponent ``` *Diagram: High-Level Architecture of the Key Vault Rotation Management System (Target State). Annotations indicate primary build phase and primary view supported. Phase 4 adds Automated Rotation Workflows.* ## 6. Key Components (Aligned with Phases) ### 6.1. Central Information Repository (Phase 1) **Description** A structured data store (e.g., Azure SQL Database, SharePoint Lists) holding the core information required for managing rotations. This repository serves as the single source of truth, initially populated and maintained manually, and later updated via automation. It logically contains the two key informational structures: Table #1 and Table #2. **Maintenance** Initial population in Phase 1. Manual update processes defined in Phase 1. Expiration dates automated in Phase 2. Automation status and script links added in Phase 4. Regular audits are required to ensure data accuracy. #### 6.1.1. Table #1: Item Inventory (The (e.g./i.e.) "What, Where, Who, When, How, Auto?") * It's **Purpose** is to provide a logical table acting as the central registry for *every* secret and certificate that the rotation team is responsible for managing. Its primary goal is to provide engineers with the essential information needed to perform or oversee a rotation: * **What** the specific secret or certificate name (e.g., `DB-Password`, `AppCert`). * **Where** the precise location within Azure (Tenant, Subscription, Resource Group, Key Vault name). * **Who** as the designated Owner/SME responsible for the item, who understands its usage and dependencies. * **When** the item's Expiration Date. * **"How"** containing a direct link or reference to the standardized rotation instructions (manual or automated) located in the Rotation Instructions Hub or code repository. * **"Auto?"** indicating if the item is configured for automated rotation (Phase 4). * **Dependencies** reflecting graph of dendencies. * Some **Example Content Columns** could be: `Tenant`, `Subscription`, `Resource Group`, `Key Vault`, `Item Type`, `Item Name`, `Owner (SME)`, `Expiration Date`, `Instructions Link`, `Automation Status` (Manual/Automated/Eligible - P4), `Automation Script Link` (P4), `Last Rotation Date`, `Last Rotation Status`, `Dependencies Graph`, (Success/Failure - P4). * It's **Role** in Phase 1, this is the primary manual reference. In Phases 2 & 3, it feeds monitoring and workflow tools. In Phase 4, it tracks and drives automated rotations. #### 6.1.2. Table #2: Engineer & Service Principal Access (The "Who/What Can Do What Where") (potentially, can be entirely skipped for now (on Phases 1-3)) * Its major **Purpose** is to serve as a logical table focusing specifically on mapping the *engineers* and *service principals* (for Phase 4 automation) performing rotations to the *Azure Tenants* and specific resources (like Key Vaults or target applications) they are authorized to access for rotation tasks. * Some **Example Content Columns** could be: `Tenant`, `Principal Name` (Engineer UPN or SPN AppID), `Principal Type` (User/ServicePrincipal), `Access Status` (e.g., Yes/No, ✓/✗, Role Assigned), `Target Resource Scope` (Optional, for finer-grained permissions). * Its **Role** is to be used for assigning manual tasks, pre-verifying access, supporting access requests/audits, and defining permissions needed for automated rotation service principals. ### 6.2. Ownership & Access Management (Phase 1+) * The **Process** can be as follows: KV Owners/SMEs formally identified and recorded in Table #1. Engineer and Service Principal access tracked in Table #2. Processes for requesting/auditing access established and refined for automation in Phase 4. * And its **Responsibility** is rotation management team ensures data accuracy through defined review cadences and processes triggered by personnel changes or application onboarding/offboarding/automation enablement. ### 6.3. Rotation Instructions Hub (Phase 1+) Designated Confluence space (or similar) for detailed, standardized *manual* rotation guides. Links to automated script repositories added in Phase 4. * **Linkage:** Table #1 provides direct links. * For **Standardization** - templates developed and enforced in Phase 1. Content populated iteratively, maintained by **?**SMEs**?** or **?**SREs**?**. ### 6.4. Key Vault Scanning Engine (Phase 2) Automated process (Azure Function, Logic App, etc.) running periodically (e.g., daily). * It's primary **Functionality** is using Managed Identity with read permissions (e.g., Reader role across relevant subscriptions) to discover KVs/items and update expiration dates in the Central Repository (Table #1). ### 6.5. Context Aggregation Service (Phase 3) Backend service (e.g., API hosted on App Service or Functions) triggered by the Workflow Interface. * It's primary **Functionalit:** is filtering repository data based on engineer queries, retrieves linked information from Confluence and JIRA via their respective APIs, and consolidates this context to support the **Operational View**. Enhanced in Phase 4 to include automation status/logs. ### 6.6. Operational Workflow Interface (Phase 3+) Tool (Web App, CLI, Chat Bot) used by engineers to interact with the system for active rotations. * Its **Functionality** provides the **Operational View**. Allows querying for specific items or filtered lists, presents the consolidated context from the Aggregation Service with actionable links. Enhanced in Phase 4 to display automation status, trigger manual overrides, and view automation logs. ### 6.7. Monitoring Dashboard (Phase 2+) Visualization tool (Power BI, Grafana, Azure Monitor Workbook). * Its **Functionality** provides the **Tactical/Strategic View**. Displays near-expiration items, metrics, and status based on data in the Central Repository. Allows filtering, drill-down, and reporting. Enhanced in Phase 4 to include automation status metrics (e.g., % automated, success/failure rates). ### 6.8. Alerting Engine (Phase 2+) Automated component (e.g., Logic App, Function) checking repository data against configurable thresholds. * Its **Functionality** (Phase 2) Sends email alerts. Phase 2/3: Creates JIRA tickets. Phase 4: Can be configured to *directly trigger* **Automated Rotation Workflows** for eligible items, or create tickets for manual oversight/fallback. ### 6.9. Automated Rotation Workflows (Phase 4) Scripts or orchestration workflows (e.g., Azure Functions, Logic Apps, Azure Automation Runbooks, DevOps Pipelines) designed to perform end-to-end rotation. * Its **Functionality** executes the specific steps required for a given secret/certificate type: generate new value, update Key Vault, update consuming application(s), validate, log results, update status in the Central Repository. Requires appropriate service principal permissions defined in Table #2. Includes robust error handling and notification/ticketing upon failure. ## 7. Data Management *(Data Flow Diagram remains largely the same conceptually for the target state, but components are built/activated across phases, with Phase 4 adding automated updates)* ```mermaid flowchart TD subgraph DataSources["Data Sources"] A[("🔑 Azure Key Vaults")] B[("📝 Manual Input (P1+)")] C[("📚 Confluence API (P3)")] D[("📋 JIRA API (P3)")] E[("📧 Email API (P3 Opt)")] F[("🗄️ Internal DBs (P1/P3)")] S[("⚙️ Automation Scripts (P4)")] end subgraph DataProcessing["Data Processing"] G[["🔍 Scanning Engine (P2)"]] H[["📊 Repository Updates (P1 Manual, P2/P4 Auto)"]] I[["🔗 Context Aggregation (P3)"]] end subgraph CentralRepository["Central Repository (P1)"] J[("Table #1 Item Inventory + Auto Status (P4)")] K[("Table #2 Engineer & SP Access")] end subgraph DataConsumers["Data Consumers"] L[["📊 Monitoring Dashboard (P2)"]] M[["🚨 Alerting Engine (P2+)"]] N[["💻 Workflow Interface (P3)"]] T[["⚙️ Automated Rotation Workflows (P4)"]] end subgraph OutputActions["Output & Actions"] O[("📈 Tactical/Strategic View (P2)")] P[("📬 Email Alerts (P2)")] Q[("🎫 JIRA Tickets (P2/P3/P4)")] R[("📋 Operational View (P3)")] U[("🔄 Automated Rotation (P4)")] end A -->|"KV Metadata"| G G -->|"Extract & Process"| H H -->|"Update Expiry Data"| J B -->|"Ownership Info"| J B -->|"Access Info"| K B -->|"Rotation Links"| J B -->|"Set Automation Status"| J J --> L J --> M J --> I J --> T K --> I K --> T C -->|"Documentation"| I D -->|"Tickets/Issues"| I E -->|"Email Context"| I F -->|"Additional Data"| I I --> N N --> R L --> O M --> P M --> Q M --> T T --> U S --> T T -->|"Automation updates status in Repo"| H %% Styling style A fill:#e1d5e7,stroke:#9673a6,stroke-width:2px style B fill:#e1d5e7,stroke:#9673a6,stroke-width:2px style C fill:#dae8fc,stroke:#6c8ebf,stroke-width:2px style D fill:#dae8fc,stroke:#6c8ebf,stroke-width:2px style E fill:#dae8fc,stroke:#6c8ebf,stroke-width:2px style F fill:#dae8fc,stroke:#6c8ebf,stroke-width:2px style S fill:#e6e6fa,stroke:#9370db,stroke-width:2px style G fill:#fff2cc,stroke:#d6b656,stroke-width:2px style H fill:#fff2cc,stroke:#d6b656,stroke-width:2px style I fill:#fff2cc,stroke:#d6b656,stroke-width:2px style J fill:#d5e8d4,stroke:#82b366,stroke-width:3px style K fill:#d5e8d4,stroke:#82b366,stroke-width:3px style L fill:#f8cecc,stroke:#b85450,stroke-width:2px style M fill:#f8cecc,stroke:#b85450,stroke-width:2px style N fill:#f8cecc,stroke:#b85450,stroke-width:2px style T fill:#e6e6fa,stroke:#9370db,stroke-width:2px style O fill:#ffe6cc,stroke:#d79b00,stroke-width:2px style P fill:#ffe6cc,stroke:#d79b00,stroke-width:2px style Q fill:#ffe6cc,stroke:#d79b00,stroke-width:2px style R fill:#ffe6cc,stroke:#d79b00,stroke-width:2px style U fill:#e6e6fa,stroke:#9370db,stroke-width:2px ``` *On the Diagram the Data Flow indicating primary build phase for processing/consuming components and output views, including Phase 4 automation.* * As **Key data management aspects** areL Sources, Storage, and Flow remain as previously described, but are realized incrementally across phases. Data quality and consistency, especially from manual inputs in Phase 1, are critical. Phase 4 introduces automated updates to the repository based on rotation job outcomes. ## 8. Workflows *(Workflow diagrams remain valid for the target state. The implementation phase determines when each workflow becomes fully operational or automated).* ### 8.1. Workflow Diagrams **1. Onboarding & Maintenance Workflow** ```mermaid flowchart TB subgraph W1["1. Onboarding & Maintenance Workflow"] A1[("🆕 New KV/Item Created or App Onboarded")] --> B1[["📝 Identify Owner/SME"]] B1 --> C1[["✍️ Record in Table #1 (Central Repository)"]] C1 --> D1[["📄 Create/Update Rotation Instructions (Confluence Hub / Script Repo P4)"]] D1 --> E1[["🔗 Link Instructions to Item (Update Table #1)"]] E1 --> F1[["⚙️ Assess Automation Eligibility (P4)"]] F1 --> G1[["Set Automation Status in Table #1 (P4)"]] G1 --> H1[["👥 Update Engineer/SP Access (Table #2 / Access Request)"]] H1 --> I1[["📅 Schedule Regular Reviews (Data Accuracy Audit)"]] I1 --> J1[("✅ Maintenance Complete")] end style W1 fill:#f5f5f5,stroke:#333,stroke-width:3px style A1 fill:#ffe6cc,stroke:#d79b00,stroke-width:2px style J1 fill:#d5e8d4,stroke:#82b366,stroke-width:2px style F1 fill:#e6e6fa,stroke:#9370db,stroke-width:2px style G1 fill:#e6e6fa,stroke:#9370db,stroke-width:2px ``` **2. Proactive Monitoring & Alerting Workflow (Target State)** ```mermaid flowchart TB subgraph W2["2. Proactive Monitoring & Alerting Workflow"] A2[("⏰ Scheduled Scan Trigger (P2)")] --> B2[["🔍 Scan Key Vaults (P2)"]] B2 --> C2[["📊 Update Repository (P2)"]] C2 --> D2{{"⚠️ Check Expiration Thresholds (P2)"}} D2 -->|"Within Threshold"| E2{{"🤖 Check Automation Status (P4)"}} D2 -->|"Outside Threshold"| F2[("📊 Update Dashboard (P2) (Tactical/Strategic View)")] E2 -->|"Automated"| G2[["🚀 Trigger Automated Rotation (P4)"]] E2 -->|"Manual/Eligible"| H2[["📧 Send Email Alert (P2)"]] E2 -->|"Manual/Eligible"| I2[["🎫 Create JIRA Ticket (P2/P3)"]] G2 --> J2{{"✅ Automation Success? (P4)"}} J2 -->|"Yes"| K2[("✨ Rotation Complete (Auto)")] J2 -->|"No"| L2[["🚨 Create Failure Ticket (P4 Fallback)"]] H2 --> M2[("🔔 Alert Sent")] I2 --> M2[("🎫 Ticket Created")] L2 --> M2 F2 --> N2[("📈 Dashboard Updated")] M2 --> O2[("Operational Workflow Triggered (Manual)")] end style W2 fill:#e6f3ff,stroke:#0078d4,stroke-width:3px style A2 fill:#ffe6cc,stroke:#d79b00,stroke-width:2px style B2 fill:#fff2cc,stroke:#d6b656,stroke-width:2px style C2 fill:#fff2cc,stroke:#d6b656,stroke-width:2px style D2 fill:#f8cecc,stroke:#b85450,stroke-width:2px style E2 fill:#e6e6fa,stroke:#9370db,stroke-width:2px style G2 fill:#e6e6fa,stroke:#9370db,stroke-width:2px style J2 fill:#e6e6fa,stroke:#9370db,stroke-width:2px style K2 fill:#d5e8d4,stroke:#82b366,stroke-width:2px style M2 fill:#d5e8d4,stroke:#82b366,stroke-width:2px style N2 fill:#d5e8d4,stroke:#82b366,stroke-width:2px style O2 fill:#f0fff0,stroke:#32cd32,stroke-width:2px ``` **3. Engineer-Driven Rotation Workflow (Target State)** ```mermaid flowchart TB subgraph W3["3. Engineer-Driven Rotation Workflow"] A3[("👨‍💻 Engineer Initiates (Via Alert/Ticket or Proactive Check using Tactical View)")] --> B3{{"🤖 Check Automation Status"}} B3 -->|"Manual"| C3[["🔎 Use Operational View (P3 Interface Query)"]] B3 -->|"Automated (Oversee/Retry)"| D3[["👁️ Monitor Auto Job / Review Logs (P4 Interface)"]] C3 --> E3[["🔗 Review Aggregated Context (Owner, Links, Tickets)"]] E3 --> F3[["📚 Access Instructions (Direct Link)"]] F3 --> G3[["🔐 Execute Rotation in Azure"]] G3 --> H3{{"✅ Verify Success"}} H3 -->|"Success"| I3[["📝 Update JIRA/Status"]] H3 -->|"Issues"| J3[["🚨 Report Issues"]] I3 --> K3[("✨ Rotation Complete")] J3 --> K3 D3 --> L3{{"✅ Auto Success?"}} L3 -->|"Yes"| K3 L3 -->|"No / Needs Manual Retry"| C3 end style W3 fill:#f0fff0,stroke:#32cd32,stroke-width:3px style A3 fill:#ffe6cc,stroke:#d79b00,stroke-width:2px style B3 fill:#e6e6fa,stroke:#9370db,stroke-width:2px style C3 fill:#fff2cc,stroke:#d6b656,stroke-width:2px style D3 fill:#e6e6fa,stroke:#9370db,stroke-width:2px style E3 fill:#fff2cc,stroke:#d6b656,stroke-width:2px style G3 fill:#fff2cc,stroke:#d6b656,stroke-width:2px style H3 fill:#f8cecc,stroke:#b85450,stroke-width:2px style L3 fill:#f8cecc,stroke:#b85450,stroke-width:2px style K3 fill:#d5e8d4,stroke:#82b366,stroke-width:2px ``` ### 8.2. Sequence Diagrams **Secret/Certificate Rotation Process (Target State - Manual)** ```mermaid sequenceDiagram title Secret/Certificate Rotation Process (Target State - Manual) actor Engineer participant WF as Workflow Interface (P3) participant AGG as Context Aggregation (P3) participant DB as Central Repository (P1) participant JIRA as JIRA (P3 Integration) participant Confluence as Confluence (P1 Hub, P3 Integration) participant Azure as Azure Key Vault rect rgb(240, 248, 255) note right of Engineer: 1. Initiate Rotation Process (P3 - Operational View) Engineer->>WF: Request details for specific item (e.g., via Ticket link) WF->>AGG: Query for item context AGG->>DB: Get item details (Owner, Expiry, Links, Status=Manual) DB-->>AGG: Return item details end rect rgb(240, 255, 240) note right of Engineer: 2. Aggregate Context (P3 - Operational View) AGG->>JIRA: Get related tickets JIRA-->>AGG: Return ticket info AGG->>Confluence: Get documentation link Confluence-->>AGG: Return docs link AGG-->>WF: Unified context view for the item WF-->>Engineer: Display results (links to KV, Confluence, JIRA) end rect rgb(255, 248, 240) note right of Engineer: 3. Execute Rotation (P1+) Engineer->>Confluence: Access Instructions (via link) Engineer->>Azure: Access Key Vault Engineer->>Azure: Rotate secret/cert Azure-->>Engineer: Confirm rotation end rect rgb(240, 240, 255) note right of Engineer: 4. Update Status (P1 Manual, P2/P3 Ticket) Engineer->>JIRA: Update ticket status alt P1 Manual Update Engineer->>DB: (Optional) Update rotation record DB-->>Engineer: Confirm update end end ``` **Automated Monitoring & Rotation (Target State - Automated)** ```mermaid sequenceDiagram title Automated Monitoring & Rotation (Target State - Automated) participant Timer as Scheduled Timer (P2) participant Scanner as KV Scanning Engine (P2) participant Azure as Azure Resources (KV, Apps) participant DB as Central Repository (P1) participant Alert as Alerting Engine (P2) participant AutoRot as Automated Rotation Workflow (P4) participant JIRA as JIRA (P2/P3/P4 Integration) participant Dashboard as Monitoring Dashboard (P2) rect rgb(250, 235, 215) note right of Timer: 1. Scheduled Scan (P2) Timer->>Scanner: Trigger scan Scanner->>Azure: Query KV metadata Azure-->>Scanner: Return items & expiry end rect rgb(240, 255, 255) note right of Timer: 2. Update Repository (P2) Scanner->>DB: Update expiration data DB-->>Scanner: Confirm update Scanner->>Dashboard: Trigger data refresh / Dashboard polls DB end rect rgb(255, 228, 225) note right of Timer: 3. Check Thresholds & Automation Status (P2+) Timer->>Alert: Trigger alert check / Alert runs on schedule Alert->>DB: Query items by threshold DB-->>Alert: Return expiring items list loop For each expiring item Alert->>DB: Get Automation Status for item DB-->>Alert: Return Status (e.g., Automated) alt Item is Automated Alert->>AutoRot: Trigger Rotation Workflow for item else Manual Alert->>JIRA: Create Manual Rotation Ticket end end end rect rgb(230, 230, 250) note right of Timer: 4. Execute Automated Rotation (P4) AutoRot->>Azure: Generate New Secret/Cert AutoRot->>Azure: Update Key Vault AutoRot->>Azure: Update Consuming App Config AutoRot->>Azure: Validate Rotation alt Rotation Successful AutoRot->>DB: Update Last Rotation Status/Date AutoRot->>JIRA: (Optional) Close/Comment on Triggering Ticket else Rotation Failed AutoRot->>JIRA: Create High Priority Failure Ticket AutoRot->>DB: Update Last Rotation Status (Failed) end end ``` ### 8.3. Key Workflows Summary (Phased Implementation) * For **Onboarding/Maintenance** manual processes established in Phase 1, enhanced in Phase 4 to include assessment and configuration for full automation. * **Proactive Monitoring & Alerting** are semi-manual checks in Phase 1 (PS1 script). Automated scanning, dashboard views (**Tactical/Strategic**), and email alerts in Phase 2. JIRA integration in Phase 2/3. Phase 4 enables alerts to directly trigger automated rotation workflows or fallback ticketing. * **Engineer-Driven Rotation** are fully manual in Phase 1. Supported by dashboard/alerts in Phase 2. Enhanced with **Operational View** in Phase 3. Phase 4 allows engineers to oversee automated jobs, review logs via the interface, and handle automated failures or trigger manual retries. ## 9. Security Considerations *(Security principles remain crucial, with increased emphasis on service principal security in Phase 4)* * For **Azure Access** automation components (Scanning Engine, Auto-Rotation Workflows) must use secure identities (e.g., Managed Identity, Service Principals with certificates) with least-privilege RBAC roles. Scanning needs Reader. Auto-Rotation needs specific write permissions (e.g., `Key Vault Secrets Officer`, `App Configuration Data Owner`, specific App Service/AKS deployment roles) scoped as tightly as possible. **Never** grant broad contributor roles. * For **API Access** we shall securely manage API keys/tokens for JIRA, Confluence, etc., using a dedicated, securely managed Key Vault (MGMTKV in architecture). * For **Data Access** we shall implement access control on the Central Information Repository and Monitoring Dashboard. * As for **Rotation Permissions** - engineers need appropriate RBAC roles for manual rotations or overseeing automated ones. Service Principals for Phase 4 need carefully scoped permissions. Consider Privileged Identity Management (PIM) for both user and service principal roles where applicable. * And we need **Audit Trails** to ensure comprehensive logging for both manual and automated actions in Azure Activity Logs, application logs, and the Central Repository. ## 10. System Components Overview *(This diagram shows the target state relationship between logical components, including Phase 4)* ```mermaid graph TB %% Root node with styling ROOT["Key Vault Rotation Management System"] %% Main components subgraph Foundation["Foundation Components (P1+)"] REPO["Central Repository"] REPO_TB1["Table 1: Item Inventory"] REPO_TB2["Table 2: Access Management"] REPO --> REPO_TB1 REPO --> REPO_TB2 end subgraph DataProcessing["Data Processing (P2-P3)"] SCAN["KV Scanning Engine (P2)"] AGG["Context Aggregation (P3)"] SCANNER_IMPL["Azure Functions/Logic Apps"] AGG_IMPL["JIRA/Confluence/Email Integration"] SCAN --> SCANNER_IMPL AGG --> AGG_IMPL end subgraph Presentation["Presentation & Alerting (P2-P3)"] DASH["Monitoring Dashboard (P2)"] ALERT["Alerting Engine (P2+)"] WUI["Workflow Interface (P3)"] DASH_IMPL["Power BI/Azure Workbooks"] ALERT_IMPL["Email/JIRA Ticket Creation"] WUI_IMPL["Web App/Teams Integration"] DASH --> DASH_IMPL ALERT --> ALERT_IMPL WUI --> WUI_IMPL end subgraph Automation["Automation (P4)"] AUTOROT["Automated Rotation"] ROT_IMPL1["Functions/Logic Apps"] ROT_IMPL2["Secret Generation"] ROT_IMPL3["KV & App Updates"] ROT_IMPL4["Validation & Logging"] AUTOROT --> ROT_IMPL1 AUTOROT --> ROT_IMPL2 AUTOROT --> ROT_IMPL3 AUTOROT --> ROT_IMPL4 end subgraph ExternalSystems["External Systems & Sources"] AZURE["Azure Resources"] COLLAB["Collaboration Tools"] INFO["Information Sources"] AZURE1["Target Key Vaults"] AZURE2["Target Applications"] COLLAB1["JIRA/Confluence"] COLLAB2["Email Systems"] INFO1["Team Knowledge"] INFO2["Existing Documentation"] AZURE --> AZURE1 AZURE --> AZURE2 COLLAB --> COLLAB1 COLLAB --> COLLAB2 INFO --> INFO1 INFO --> INFO2 end subgraph Security["Security & Identity"] AUTH["Authentication"] AUTHZ["Authorization"] SECMGMT["Secrets Management"] AUTH1["Managed Identity"] AUTH2["Service Principals"] AUTHZ1["Azure RBAC Roles"] SECMGMT1["Management Key Vault"] AUTH --> AUTH1 AUTH --> AUTH2 AUTHZ --> AUTHZ1 SECMGMT --> SECMGMT1 end %% Connections between main groups ROOT --> Foundation ROOT --> DataProcessing ROOT --> Presentation ROOT --> Automation ROOT --> ExternalSystems ROOT --> Security %% Data Flow between components SCAN -->|Extracts Data| REPO_TB1 AGG -->|Enriches With Context| REPO_TB1 REPO_TB1 -->|Provides Data| DASH REPO_TB1 -->|Triggers Alerts| ALERT REPO_TB1 -->|Informs| WUI REPO_TB1 -->|Drives| AUTOROT REPO_TB2 -->|Provides Access Info| AGG REPO_TB2 -->|Security Context| AUTOROT ALERT -->|Triggers| AUTOROT %% External Interactions SCAN -->|Reads From| AZURE1 AUTOROT -->|Updates| AZURE1 AUTOROT -->|Updates| AZURE2 AGG -->|Integrates With| COLLAB1 ALERT -->|Creates Tickets In| COLLAB1 ALERT -->|Sends Notifications Via| COLLAB2 %% Security Connections AUTH1 -->|Secures| SCAN AUTH1 -->|Secures| AGG AUTH2 -->|Secures| AUTOROT SECMGMT1 -->|Stores Secrets For| AGG SECMGMT1 -->|Stores Secrets For| AUTOROT %% Styling style ROOT fill:#663399,stroke:#333,stroke-width:4px,color:white style Foundation fill:#f5f5dc,stroke:#333,stroke-width:2px style DataProcessing fill:#add8e6,stroke:#333,stroke-width:2px style Presentation fill:#f0e68c,stroke:#333,stroke-width:2px style Automation fill:#e6e6fa,stroke:#9370db,stroke-width:2px style ExternalSystems fill:#fafad2,stroke:#333,stroke-width:2px style Security fill:#ffe4e1,stroke:#333,stroke-width:2px ``` ## 11. Deployment Architecture *(This diagram shows a potential physical deployment including Phase 4 components)* ```mermaid graph TB subgraph EnterpriseEnvironment["Enterprise Environment"] subgraph AzureSubscriptions["Azure Subscriptions (Managed Tenants)"] KV1["Key Vault 1 Production"] KV2["Key Vault 2 Test"] KVN["Key Vault N Dev"] APP1["Target App 1 (e.g., App Svc)"] APP2["Target App 2 (e.g., AKS)"] end subgraph RotationManagementSystem["Rotation Management System (Dedicated Subscription/RG)"] SCANNER["KV Scanning Engine (Azure Function App - P2)"] DB["Central Repository (Azure SQL DB - P1)"] AGG["Context Aggregation (App Service / Function API - P3)"] ALERT["Alerting Engine (Logic App / Function - P2)"] DASHBOARD["Monitoring Dashboard (Power BI Service - P2)"] WORKFLOW["Workflow Interface (Web App / Teams Bot - P3)"] AUTOROT["Automated Rotation (Function App / Logic App - P4)"] MGMTKV["Management KV (Stores API Keys, SP Certs)"] end subgraph SecurityIdentity["Security & Identity (Azure AD)"] MI["Managed Identity (For Scanner, AGG, Alert)"] SPN["Service Principal (For AutoRot - P4)"] RBAC["RBAC Roles (User & SPN Permissions)"] end end subgraph ExternalSystems["External Systems"] JIRA["JIRA Cloud/Server"] CONF["Confluence Cloud/Server"] EMAIL["Email Service (O365/SendGrid)"] end subgraph Users["Users"] ENGINEER["Engineer"] OWNER["Owner/SME"] end %% Data/Interaction Flow SCANNER -->|"Scans (Azure API via MI)"| KV1 SCANNER -->|"Scans (Azure API via MI)"| KV2 SCANNER -->|"Scans (Azure API via MI)"| KVN SCANNER -->|"Updates (SQL Auth via MI)"| DB AGG -->|"Reads (SQL Auth)"| DB AGG -->|"Reads (API Key from MGMTKV)"| JIRA AGG -->|"Reads (API Key from MGMTKV)"| CONF AGG -->|"Reads (API Key from MGMTKV)"| EMAIL WORKFLOW -->|"Requests (HTTP/S via AAD Auth)"| AGG ENGINEER -->|"Uses (Operational View) (HTTPS/Teams)"| WORKFLOW ENGINEER -->|"Rotates (Manual) (Azure Portal/CLI/API via RBAC)"| KV1 ALERT -->|"Monitors (SQL Auth)"| DB ALERT -->|"Sends (SMTP/API Key from MGMTKV)"| EMAIL ALERT -->|"Creates (API Key from MGMTKV)"| JIRA ALERT -->|"Triggers (Queue/HTTP)"| AUTOROT AUTOROT -->|"Rotates (Azure API via SPN)"| KV1 AUTOROT -->|"Updates App (Azure API via SPN)"| APP1 AUTOROT -->|"Updates App (Azure API via SPN)"| APP2 AUTOROT -->|"Updates Status (SQL Auth via MI/SPN)"| DB AUTOROT -->|"Creates Failure Ticket (API Key from MGMTKV)"| JIRA DASHBOARD -->|"Reads (SQL Auth/AAD)"| DB ENGINEER -->|"Views (Tactical/Strategic View) (HTTPS/PowerBI App)"| DASHBOARD OWNER -->|"Updates Info (Manual Process/UI TBD)"| DB MGMTKV -->|"Provides Secrets"| SCANNER MGMTKV -->|"Provides Secrets"| AGG MGMTKV -->|"Provides Secrets"| ALERT MGMTKV -->|"Provides Secrets/Certs"| AUTOROT %% Styling style AzureSubscriptions fill:#e6f3ff,stroke:#0078d4,stroke-width:2px style RotationManagementSystem fill:#f0fff0,stroke:#32cd32,stroke-width:2px style SecurityIdentity fill:#fff0f5,stroke:#db7093,stroke-width:2px style ExternalSystems fill:#fffacd,stroke:#f0e68c,stroke-width:2px style KV1 fill:#e1d5e7,stroke:#9673a6,stroke-width:2px style KV2 fill:#e1d5e7,stroke:#9673a6,stroke-width:2px style KVN fill:#e1d5e7,stroke:#9673a6,stroke-width:2px style MGMTKV fill:#e1d5e7,stroke:#9673a6,stroke-width:2px style APP1 fill:#dae8fc,stroke:#6c8ebf,stroke-width:2px style APP2 fill:#dae8fc,stroke:#6c8ebf,stroke-width:2px style DB fill:#d5e8d4,stroke:#82b366,stroke-width:3px style ENGINEER fill:#f8cecc,stroke:#b85450,stroke-width:2px style OWNER fill:#f8cecc,stroke:#b85450,stroke-width:2px style AUTOROT fill:#e6e6fa,stroke:#9370db,stroke-width:2px ``` ## 12. Technology Stack (Potential Options) *(Choices made during detailed design for each phase)* * For **KV Scanning/Automation** e.g. Azure Functions (e.g. Python / NodeJS recommended for Azure SDK use), Azure Logic Apps, Azure DevOps Pipelines, n8n. * For **Central Repository** e.g. Azure SQL/PSQL Database (Recommended for relational data), Azure Cosmos DB (If schema flexibility is key), SharePoint Lists / Confluence (Viable only for very small scale/complexity in Phase 1). * For **Dashboard/UI** e.g. Power BI (Strong Azure integration), Grafana (Good for time-series, requires hosting), Azure Monitor Workbooks (Good for Azure-native metrics, less flexible layout). Custom Web App (React/Angular/Vue + Backend API - Most flexible, highest effort). * For **Alerting/Integration** e.g. Azure Monitor Alerts (Native Azure), Logic Apps (Visual workflow, connectors), Power Automate (Similar to Logic Apps, O365 focus), Custom code (Functions, etc. using SDKs/REST APIs), Grafana (has built-in capabilities). * For **Collaboration/Docs** e.g. Confluence, JIRA, Teams. * For **Operational Workflow Interface** e.g. Custom Web App (Hosted on App Service), CLI Tool (Python/PowerShell/NodeJS), ChatOps Bot Framework (Teams). * For **Automated Rotation Workflows (P4)** e.g. Azure Functions, Logic Apps, Azure Automation Runbooks, Azure DevOps Pipelines (YAML), custom scripts invoked by orchestration tools. Choice depends heavily on the complexity of the rotation and application update process. ## 13. Deliverables (Aligned with Phases) * **Phase 1:** * Rotation Checklist (Initial Version). * Populated Central Information Repository (Tables #1 & #2). * Established Rotation Instructions Hub & Templates. * Initial set of Rotation Instructions/Playbooks for critical items. * Documented manual maintenance processes (SOPs for updates). * **Phase 2:** * Deployed KV Scanning Engine (Code, ARM/Bicep templates). * Deployed Monitoring Dashboard (Report file/configuration) - Providing Tactical/Strategic View. * Deployed Alerting Engine (Logic App definition/Function code). * (Optional) Basic JIRA Ticket Creation capability (Code/Configuration). * Updated documentation & SOPs for automated components. * **Phase 3:** * Deployed Context Aggregation Service (API code, deployment templates). * Deployed Operational Workflow Interface (App code/CLI script/Bot config) - Providing Operational View. * Enhanced JIRA Integration (Updated code/configuration). * (Optional) Email API Integration (Code/Configuration). * Updated SOPs incorporating workflow tools. * User guides for the Workflow Interface. * **Phase 4:** * Deployed Automated Rotation Workflows for selected items (Code, deployment templates, test plans). * Updated Central Repository schema/data to track automation. * Updated Alerting Engine to trigger automated workflows. * Updated Dashboard and Workflow Interface to reflect automation status/logs. * Documented procedures for managing automated rotations (including failure handling). * Updated SOPs and Rotation Checklist. ## 14. Assumptions & Dependencies *(Assumptions evolve with phases)* * As for **API Availability & Access** we need stable and accessible APIs for Azure (Resource Graph, Key Vault, App Service, AKS, etc.), JIRA, Confluence, Email. Licenses/permissions secured per phase (not sure if any licenses would be needed though, not likely). * As for **Azure Permissions** we need sufficient permissions for discovery (P1/2), monitoring (P2), workflow support (P3), and *automated modifications* (P4 - requires careful scoping). * Ofcourse, **Team Commitment & SME Availability** is crucial from Phase 1. And, SMEs needed to validate automation logic in Phase 4. * As for **Tooling Access & Decisions** - supposed to be made per phase. Phase 4 requires decisions on automation/orchestration tools. * When approaching to Phase 4 we need to ensure and improve **Suitability for Automation** which assumes that a subset of secrets/certificates have rotation processes that *can* be fully automated, including updating consuming applications without manual intervention. This requires investigation per item. ## 15. Conclusion This High-Level Design presents a comprehensive, **four-phased approach** to managing Azure Key Vault secret and certificate rotations. **Phase 1** focuses on building a solid foundation of information and process. **Phase 2** introduces automation for visibility (**Tactical/Strategic View**) and basic alerting. **Phase 3** delivers advanced workflow automation (**Operational View**) to significantly streamline the rotation task for engineers performing manual rotations. **Phase 4** introduces **Deep Automation**, enabling end-to-end, zero-touch rotation for suitable items, further reducing manual effort and risk. This incremental strategy allows the organization to realize benefits early, manage implementation risks, and adapt based on learnings from each phase. By systematically addressing ownership, documentation, monitoring (providing both strategic and operational perspectives), workflow support, and finally, targeted end-to-end automation, this solution will significantly reduce the risk of service disruptions due to expired credentials while maximizing operational efficiency. The modular architecture supports scalability and future enhancements as the organization's needs evolve.

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.