OpenAI:Isk-proj-akFX-LdEQKgRnjr5lbHYJLLaasR6ADHGm7rAp15THNbwIMlVc-x-3-YiG-Ww99Sfwe26Qk1OLBT3BlbkFJQHbAnYaRvDowf0S0dksRYlR5oa473UxRoRxjy4GdIoLp3oL3Qlpi7Mu1pBljbaaqL2zGf-KPMA

Re-established Microsoft Transcription Azure Function Using New Flexible Consumption Plan

The Microsoft Transcription Azure Function was previously deleted and is now successfully re-established using a new Flexible Consumption Plan.

This function is responsible for handling the subscription and retrieval of online meeting transcriptions from Microsoft Graph API.

ctions Taken:
• Provisioned a new Flexible Consumption Plan on Azure.
• Re-deployed the transcription Azure Function app.
• Re-linked it to the webhook and Graph API subscription pipeline.
• Verified event handling and transcript flow works as expected.

Industry: Technology Sector: Cybersecurity
Competency Area: Project Management, Cybersecurity education, Security and Privacy
Lead Technical Program Manager Consultant; 11 months.
Facebook Inc.
Mohamed worked crossed-functionally with Facebook engineers and program
managers to establish technical interview preparation and cybersecurity
project-based training to 25 colleges TA's leading to an interest stimulation in
cybersecurity and career at Facebook. He performed a production support and monitoring of their Google Cloud Platform based application through which students run their assignment.
In addition, he managed, planned and travelled to with students to Cybersecurity conferences sponsored by Facebook (Black Hat, Defcon, Enigma) leading to their increase in security awareness, regulation and
industries challenges.

Industry: Technology Sector: Cybersecurity
Competency Area: Project Management, Cybersecurity education, Security and Privacy
Lead Technical Program Manager Consultant; 11 months.
Facebook Inc.
Mohamed collaborated with Facebook engineers and program managers to deliver technical interview prep and cybersecurity training to 25 college TAs, boosting interest in cybersecurity careers. He supported a GCP-based student application and organized student trips to major cybersecurity conferences, enhancing awareness of industry challenges and regulations

Hey team, quick question — I’ve built an Azure Function that pulls Teams transcripts via Graph API and stores them in MongoDB (using delegated auth + token refresh).

Wondering if Power Automate could replicate this:
• Can it handle delegated auth + refresh tokens?
• Can it poll the Graph API efficiently?
• Any way to store data directly to MongoDB?

Curious if it’s worth exploring Power Automate for this or sticking with the function. Would love your thoughts!

Exploring Power Automate as an Alternative to Azure Function for Transcript Fetching

Explored Power Automate + Graph API for meeting transcripts. It’s feasible but not a clear upgrade:
• Auth: Refresh token support for server use still unclear.
• Triggers: No real-time events—manual polling needed.
• Logic: Would need to replicate our current process, including filtering duplicates.
• Effort: Similar complexity to our Azure Function—no major time savings.
• Pros: Possible visibility gains (~3K executions) and predictable costs.

Is this considering voice transcripts only? If so, do any opinions change considering Teams messages as well? Whatever is easiest to create and maintain is best. If neither has a particular advantage, I'll leave it to what you both decide is best for your development. Let's just have our efforts be consistent.


This application is a serverless Azure Function designed to interact with the Microsoft Graph API using delegated permissions via Azure AD( Entra). It enables automatic retrieval and storage of Microsoft Teams meeting transcripts, which are saved to a MongoDB collection. Currently deployed as a POC in the PlutodogHouse tenant, the application requires enhancements for production readiness.

Hey team,

I’ve explored using Power Automate with the Graph API to fetch meeting transcripts. Here’s the rundown:

Feasibility: It’s doable, but we need to check if refresh tokens work for server setups (still waiting on dev feedback).
Event Triggers: No real-time transcript event connectors exist, so we’d rely on manual polling.
Logic: We’d need to recreate our current process—polling for events, grabbing metadata and content, and filtering to avoid duplicate API calls.
Effort: It’s about the same workload as our current setup—no time or complexity savings.
Pros:
Visibility: Could improve tracking (TBD with ~3K executions).
Cost: Pricing is predictable (excluding API metering).
Bottom Line: Power Automate is an option, but it’s not a quick win. Should we dig deeper into the visibility and cost benefits, or stick with what we’ve got?

I performed an extended research on the topic of interactions between Power Automate and GraphAPI in context of the transcript fetching task and here are the main points:

The task seems to be feasible, conditioned on ability of using refresh tokens on server deployments (to be decided, after talk with dev cohort)
There are no built-in connectors which allow us to get event data (i.e. transcript created) while getting around permission model
So it seems that porting to Power Automate would require replication of the complete logic flow from the application function:
Pooling events
Polling transcription metadata
Pooling transcript content based on metadata
Optional filtering to avoid fetching already seen data from metered endpoints
So there is no indication that porting will reduce development and maintenance effort at the moment.
However there might be some advantages:
Better visibility (though unclear with ~3K executions and much higher number of individual tasks)
Fixed cost (metered graph API calls not included)

Mohamed is a Government & Public Services (GPS) Strategy & Analytics consultant with 5 years of experience in GenAI, NLP, Data Science/ Data Engineering and Big data analytics. He possesses a broad functional expertise and is passionate about building data-driven decision processes.
He has experience in applied quantitative and qualitative research design, data cleaning, exploratory data analysis, and machine learning engineering and deploying model to production. He has proven his ability to manage projects and teams, translate business requirements to scalable solutions, and communicate effectively with stakeholders. He possesses programming skills in Python, SQL, Java, and JavaScript, Data Science/Machine learning skills, Big Data skills, Operating system skills, and understanding of cybersecurity and privacy framework.
He is fluent in English and French, and intermediate in Spanish. He has a master’s in data science (MIDS) from UC Berkeley, MIT's applied Data Science program, an executive data science Master from North Carolina State, and a bachelor’s degree in computer science from the City College of New York. Mohamed is focused on research design, machine learning, data engineering, mining and exploring, data visualization, ethics and privacy, statistical analysis, and communicating results.

He led the development of RAG following for a Question Answering for health Agency domain insights. He streamlined best development practices, RAG architecture, ETL Pipeline and the deployment of a Streamlit app bot along with an API for consumption by other systems.

Mike is a skilled Data Engineer and Analytics Consultant with 5 years of experience in building scalable data pipelines, transforming complex datasets, and supporting machine learning and GenAI initiatives. He brings deep expertise in Python, SQL, Spark, and cloud data platforms, with a strong focus on data quality, performance, and privacy.

He excels at translating business needs into efficient data solutions, leading cross-functional teams, and delivering insights that drive decision-making. Mike holds advanced degrees in Data Science from UC Berkeley and NC State, and is fluent in English and French, with working knowledge of Spanish.

Technical Leadership & System Design
1. Can you walk us through the architecture of the Quid royalties and payments system you built at Amazon Publishing? What trade-offs did you have to make?
2. How would you design a scalable, low-latency video metadata management system for a global streaming service?
3. What would be your approach to refactoring a monolithic streaming app backend into a microservices architecture?

Technical Deep Dive
4. You mentioned experience with Spring and Hibernate—how have you used these in production, and what are the performance pitfalls you’ve encountered?
5. You’ve worked with both SQL (MySQL, Oracle) and NoSQL (DynamoDB) systems—how do you decide which to use in different parts of a system like a streaming backend?
6. How have you implemented observability (logging, tracing, monitoring) in the systems you’ve built or maintained?

Cloud & DevOps (AWS-focused)
7. How have you used AWS Step Functions and Lambda in orchestration flows? Would you use them in a high-throughput streaming ingestion system?
8. Can you describe a time you optimized cost or performance of a service using AWS services (e.g., S3, Athena, EC2, etc.)?
9. What’s your experience with CI/CD pipelines, and how have you ensured reliability in production deployments?

🧠 Data Pipelines & Analytics
10. At Amazon Catalog BI, you supported Spark data pipelines and Elasticsearch—how would you apply that to support content discovery in a streaming service?
11. How would you design a data analytics pipeline to generate personalized recommendations using user activity logs

Collaboration, Mentorship & Leadership
12. You’ve mentored developers at Amazon—what’s your approach to technical mentorship and team growth?
13. Tell us about a time when you had to resolve a conflict between engineering and product teams over feature scope or timelines.
14. How do you lead without micromanaging? What techniques do you use to empower engineers while ensuring delivery?

We need support for a Gen AI project with Minnesota Department of Employment & Economic Development (DEED). The ideal candidate(s) should have the following skills:

1. Strong experience developing AI algorithms coding in Python
2. Experience conducting prompt engineering on Large Language Models (LLMs)
3. Understanding of transformer model architectures (e.g., GPT, BERT, Llama)
4. Ability to conduct chunking and preprocessing of large documents
5. Experience with information retrieval using vector databases and understanding the use of similarity metrics
6. Natural Language Processing skills for text cleaning / parsing and metadata tagging
7. Practical knowledge of backend integration using Langchain / LlamaIndex and performing API integration
8. Experience using Docker or relevant cloud services for deployment of LLM capabilities

Mohamed is a Government & Public Services (GPS) Strategy & Analytics consultant with 5 years of experience in GenAI, NLP, Data Science/ Data Engineering and Big data analytics. He possesses a broad functional expertise and is passionate about building data-driven decision processes.
He has experience in applied quantitative and qualitative research design, data cleaning, exploratory data analysis, and machine learning engineering and deploying model to production. He has proven his ability to manage projects and teams, translate business requirements to scalable solutions, and communicate effectively with stakeholders. He possesses programming skills in Python, SQL, Java, and JavaScript, Data Science/Machine learning skills, Big Data skills, Operating system skills, and understanding of cybersecurity and privacy framework.
He is fluent in English and French, and intermediate in Spanish. He has a master’s in data science (MIDS) from UC Berkeley, MIT's applied Data Science program, an executive data science Master from North Carolina State, and a bachelor’s degree in computer science from the City College of New York. Mohamed is focused on research design, machine learning, data engineering, mining and exploring, data visualization, ethics and privacy, statistical analysis, and communicating results.

With over five years of experience in Generative AI, NLP, and Data Science, I bring a strong background in developing AI solutions using Python and deploying LLMs for public sector clients. I have hands-on expertise in prompt engineering, transformer architectures (GPT, BERT, LLaMA), document chunking, and semantic search using vector databases. I’ve built end-to-end GenAI pipelines with LangChain, LlamaIndex, and Docker, and I’m skilled in NLP tasks like text cleaning and metadata tagging. My strong communication skills, project leadership experience, and deep understanding of data privacy frameworks make me well-suited to support Minnesota DEED’s GenAI initiatives.

Thank you for your message. I’m currently out of the office and will return on [return date]. During this time, I will have limited access to email.

If your matter is urgent, please contact [alternate contact name] at [email/phone].

Best regards

https://meet.jit.si/OrganizedActorsMarkArticulately

Summary architecture.


Quick update on the Teams Transcription Bot Azure function deployment:

* 		Basics Azure functions are deployed Successfully(POC), but for a production use,  it will require further improvements.
* 		The API usability and security need refinement, especially around authorization.
* 		Currently, authorization uses a device flow instead of an interactive login process, allowing the function to act on behalf of the user.
* 		The approach enables access to transcripts without requiring user login, but we need to assess security risks.


I explored some options for polling events from graph API, but it seems like these are pretty limited.

n particular onlineMeetings the endpoints don't accept filter on createdDateTime.
Delta endpoints, same as webhooks, require application permissions https://learn.microsoft.com/en-us/graph/delta-query-overview#optional-query-parameters.

Maybe it is possible to create event hub subscriptions, but it doesn't seem I can create one on plutodoghouse. I'll double check that tomorrow.

In the worst case scenario we can try to pool everything and filter on the client side, but this might be pretty expensive and not scale to our use case.
If none of the above works, we might have to require explicit registration of the meeting as object to be tracked, once it is created.



===================

Discussion around Delegated permission: 
Delegated permission are primarily use for interactive flow. This might complicate things, as something will be needed to mimic authentication for a user.

There are part of the graph API that can't be access using delegated permission at all.

Examples of limitations
* 		Event driven and Webhook can't be registered with delegated permission.

* 		Meeting bot cannot be access as well with delegated permission.

It will be hard to get a notification. As we work around, maybe. We can try different workarounds (pooling, service accounts), but it might create serious technological debt, with no guarantee of success in case another "deploy into production now" event.







Today, When trying to get content of transcripts for the meetings (actual content, not metadata) using the plutodoghouse account I started getting 402 errors (Payment required).


HTTPStatusError: Client error '402 Payment Required' for url

After further investigation it seems like it is metered endpoint

https://learn.microsoft.com/en-us/graph/teams-licenses

with only limited free trial (not free tier) which seems to be over. The pricing is 0.75 per 1000 requests if read this correctly.

According to the Cohort team, it was hard to get a subscription on plutodoghouse.

Empowering Disaster Response with Data
• Natural disasters are unpredictable, but our response doesn’t have to be.
• The Emergency Response Dashboard transforms disaster management by providing:
Real-time data for faster decision-making
Prioritized rescue operations for vulnerable populations
Optimized resource allocation for hospitals, shelters, and aid distribution
AI-driven insights to improve preparedness and response

“With data-driven insights, we can save more lives, allocate resources efficiently, and ensure help reaches those who need it most.”

The Emergency Response Dashboard is a real-time data visualization tool designed to aid emergency responders, governments, and relief agencies.
• It provides real-time disaster tracking, rescue prioritization, and resource allocation insights to improve emergency response efficiency.
• By integrating weather data (NOAA), earthquake reports (USGS), and disaster declarations (FEMA), the dashboard enhances situational awareness and decision-making.
• Key objectives:
• 📍 Track disasters using live geospatial data.
• 🚑 Prioritize vulnerable populations (children, elderly, pregnant women).
• 🏥 Optimize hospital & shelter access for rescue operations.
• 🔍 Use AI-driven insights to improve disaster forecasting and relief efforts.

I’d like to sync with you after your call to go over the missing Azure Subscription on the Plutodog account.

Currently, my Azure Function application is running on our main Azure account, utilizing services like Key Vault, Storage, etc.. However, on Plutodog, I have a separate application registered under extra, specifically responsible for making Graph queries within the Plutodog environment.

There’s no direct connection between these two parts—they don’t share permissions or access controls. If we proceed with setting up a service account on the main Disney tenant, it will not have access to Plutodog resources using those credentials.

OK. So update for today is as follows:

I kept exploring options for polling events from graph API, but it seems like these are pretty limited. In particular onlineMeetings endpoints don't accept filter on createdDateTime.
Delta endpoints, same as webhooks, require application permissions https://learn.microsoft.com/en-us/graph/delta-query-overview#optional-query-parameters.
Maybe it is possible to create event hub subscriptions, but it doesn't seem I can create one on plutodoghouse. I'll double check that tomorrow.
In the worst case scenario we can try to pool everything and filter on the client side, but this might be pretty expensive and not scale to our use case.
If none of the above works, we might have to require explicit registration of the meeting as object to be tracked, once it is created.

import pandas as pd
from geopy.distance import geodesic
from google.cloud import bigquery

# Initialize BigQuery Client
client = bigquery.Client()

# Query to get hospital locations
hospital_query = """
SELECT name, latitude, longitude, city, state
FROM `bigquery-public-data.cms_physician.compare`
WHERE type = 'HOSPITAL'
"""
# Fetch hospital data
hospital_df = client.query_and_wait(hospital_query).to_dataframe()

# Your dataset (already fetched from previous SQL)
df = pd.read_csv("test_bimaharana.csv")  # Load your saved dataset

# Function to find the nearest hospital
def find_nearest_hospital(lat, lon):
    min_distance = float("inf")
    nearest_hospital = None

    for _, hosp in hospital_df.iterrows():
        hosp_location = (hosp["latitude"], hosp["longitude"])
        person_location = (lat, lon)
        distance = geodesic(person_location, hosp_location).miles  # Calculate distance in miles

        if distance < min_distance:
            min_distance = distance
            nearest_hospital = hosp["name"]

    return nearest_hospital, min_distance

# Apply the function to each row in the dataset
df[["nearest_hospital", "hospital_distance_miles"]] = df.apply(
    lambda row: pd.Series(find_nearest_hospital(row["latitude"], row["longitude"])), axis=1
)

# Save results
df.to_csv("nearest_hospitals.csv", index=False)

# Show first few rows
df.head()

import plotly.express as px

fig_hospital_distance = px.scatter_mapbox(
df, lat="latitude", lon="longitude",
size="estimated_max_driving_distance_hospital",
color="estimated_min_driving_distance_hospital",
hover_data=["estimated_max_driving_distance_hospital", "estimated_min_driving_distance_hospital"],
title="Min & Max Distance to Hospitals",
mapbox_style="open-street-map"
)

! pip3 install --upgrade --quiet google-cloud-aiplatform

user_id = "bimaharana"

pip install pandas geopy dash

import pandas as pd
import numpy as np
from geopy.distance import geodesic
import dash
from dash import dcc, html, Input, Output
import plotly.express as px

# Run a query against the table we created earlier

from google.cloud import bigquery

client = bigquery.Client()

# Perform a query.
sql = ('SELECT address,city,county,county_fips_code,latitude,longitude,number_of_adults,number_of_people,population ,state_abbreviation,state,estimated_driving_distance_hospital,estimated_min_driving_distance_hospital,estimated_max_driving_distance_hospital,children_age_group,fam_age_elderly,fam_age_midage,fam_age_youngsters,income_level_category,income_level_category_index,zip_income_decile,children_age_group_00_to_02_ind,children_age_group_00_to_02_ind,children_age_group_03_to_05_ind,children_age_group_06_to_10_ind,children_age_group_11_to_15_ind,respiratory_risk_raw_decile,respiratory_risk_decile,respiratory_risk_raw_centile,respiratory_risk_centile,estimated_driving_distance_urgent_care,estimated_min_driving_distance_urgent_care,estimated_max_driving_distance_urgent_care,financial_profile,is_pregnant,pregnancy_ind,children_age_group_16_to_18_ind,age FROM `us-con-gcp-npr-0000182-072619.prism_training_20250311.predrisk_data_2501_nation_sample_temp` LIMIT 10000')

# Save results to a pandas dataframe in memory
df = client.query_and_wait(sql).to_dataframe()

# Save results to csv
df.to_csv('test_'+user_id+'.csv')

# Define up file to gcs function

from google.cloud import storage

def upload_blob(bucket_name, source_file_name, destination_blob_name):
"""Uploads a file to the bucket."""

storage_client = storage.Client()
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(destination_blob_name)
generation_match_precondition = 0

blob.upload_from_filename(source_file_name, if_generation_match=generation_match_precondition)

print(
f"File {source_file_name} uploaded to {destination_blob_name}."
)

# Run upload to gcs function
upload_blob('prism_training_20250311', './test_'+user_id+'.csv', user_id + '/test_'+user_id+'.csv')

file_path = "test_bimaharana.csv" # Update the path to actual dataset
df = pd.read_csv(file_path)

df.dropna(inplace=True)
df.drop_duplicates(inplace=True)

df['latitude'] = pd.to_numeric(df['latitude'], errors='coerce')
df['longitude'] = pd.to_numeric(df['longitude'], errors='coerce')

print("Data cleaned and loaded successfully!")

def find_nearest_hospitals(df, user_lat, user_lon, num_hospitals=3):
user_location = (user_lat, user_lon)
df['distance_from_user'] = df.apply(
lambda row: geodesic(user_location, (row['latitude'], row['longitude'])).km, axis=1)
nearest_df = df.sort_values(by='distance_from_user').head(num_hospitals)
return nearest_df[['address', 'city', 'state',
'estimated_driving_distance_hospital',
'estimated_min_driving_distance_hospital',
'estimated_max_driving_distance_hospital',
'distance_from_user']]

try:
user_lat = float(input("Enter your latitude: "))
user_lon = float(input("Enter your longitude: "))
except ValueError:
print("Invalid latitude or longitude entered.")
exit()

nearest_hospitals = find_nearest_hospitals(hospital_df, user_lat, user_lon)

majorly we need to visualize 3 thing

Min and max distance to hospital
priority of people to be rescued first i.e. old, children, pregnant
rescued to prioritize for those who are financially not stable.

End of Day Summary

Accomplishments:
• Successfully got the application up and running.
• Pulled transcript data and exposed it through a simple REST API.
• Verified that live updates can be retrieved during an active meeting, though it requires polling the server and manually tracking changes.

Next Steps:
• Research the deployment process for hosting this application internally on Disney’s Azure tenant.
• Explore integration with the Teams chatbot developed by Todd.

Open Questions:
• Clarify whether application permissions allow access in all cases.

Summary for today:

Got application up and running.
Was able to pull transcript and provide it through simple rest API
Tested that it is possible to get live updates while meeting is running, but it requires pooling from the server and tracking the changes manually
Next steps:

Researching steps required to deploy this application internally on Disney Azure tenant
Then seeing if it can be connected to teams chat bot developed by Todd
Still unresolved:

Query if access to application permissions is allowed in any case.

Got approval for transcript and recording permissions, and these seem to work fine.
Were able to invoke transcript API, but don't have internal meeting id (different than join id)
Wanted to add OnlineMeetings.read permission to get details including internal meeting id, but got stuck with "Approval Required" popup. Asked for clarifications from Ryan and Lilli.
Did additional research on meetings bots with Speech capabilities. These look very promising (audio can be captured in real time and processed with speech to text API) but require application only permissions (Calls.JoinGroupCall.All, Calls.AccessMedia.All) which seem to be off-limits for us. Asked for confirmation from Ryan and Lilli.

Summary for today:

Finally got enough permissions approved to be able to access meeting metadata and call transcripts and recording.
Was able to record a meeting and fetch the recording through API.
Wasn't able to test transcripts, because they're disabled at the organization level. We might be able to do transcripts manually on the app side, but I"d prefer to avoid that unless necessary.
Things pending:

Still didn't get response on the question about application level permissions. These are required for most of the advanced interactions, including call bots, webhook notifications.
If there is no chance to get these approved because of lack of compartmentalization we will be severely limited in terms of what can be done on our side. It might mean that microsoft will be able to access things, that we can't within the company.

https://meet.jit.si/jake_interview_prep1


https://wormhole.app/kMeLy#66BlK_IVSdPN6vtlIOeTDA

https://wormhole.app/nR85z#D0yI_aU46FXbRG7wN19JQQ

https://wormhole.app/WlyeB#av0AE_LE_vOF3WN6GqH-aQ

https://drive.proton.me/urls/CVBKBSXJG8#UZBW14QEdv2X

https://meet.jit.si/OrganizedActorsMarkArticulately

https://drive.proton.me/urls/H31MCW4MX0#xwBSGGQ8ndTA
Mohamed: https://drive.proton.me/urls/G1JKP3DJEW#xV2YVPnV5ejD


=========================

• 1 Paragraph regarding client work role and impact
• 1 Paragraph regarding initiative role(s) and impact(s)
• Bullets highlighting key accomplishments for client work
• Bullets highlighting key accomplishments for firm initiatives
• Names and email addresses of Team Leads and Managers

Actually I have some work for you to create a training dataset for the Incident Channel bot but that's still a work in progress. I was hoping to have the bulk of the demo done by the end of next week and the team has been working hard on it. My goal was to have it done before Christmas so we'll see if that happens. What I can do is make some time to show you the data and preliminary LLM results. Note that after the demo, the requirements will change because the point of the demo is to stimulate stakeholder feedback, of which, we have barely any. So the training dataset and validation and testing will all be an evolving work in progress.

If the knowledge base content is largely shared and the queries are similar, use a shared knowledge base with separate APIs and query filtering.
• If the new team’s use case requires specialized data, sensitive access, or introduces significant load, create a separate knowledge base

Hi Team,

To enable the other team to deploy their new Lambda function independently without impacting the existing chatbot, we will:
1. Use the same repository but organize the code into separate folders:
• /chatbot-lambda/ → Existing chatbot code
• /new-team-lambda/ → New team’s Lambda code
2. Create a separate CI/CD pipeline for the new Lambda function:
• Each Lambda will have its own pipeline for building, testing, and deployment.
• The chatbot pipeline triggers on changes to /chatbot-lambda/.
• The new pipeline triggers on changes to /new-team-lambda/.
3. Ensure isolation:
• Pipelines are independent to avoid conflicts or downtime.
• Different environment variables, permissions, and logs will be configured.

This approach allows parallel development and deployments, ensuring no disruption to the chatbot. Let me know if you have any questions!

Actually I have some work for you to create a training dataset for the Incident Channel bot but that's still a work in progress. I was hoping to have the bulk of the demo done by the end of next week and the team has been working hard on it. My goal was to have it done before Christmas so we'll see if that happens. What I can do is make some time to show you the data and preliminary LLM results. Note that after the demo, the requirements will change because the point of the demo is to stimulate stakeholder feedback, of which, we have barely any. So the training dataset and validation and testing will all be an evolving work in progress.

I need two methods that allow me to capture two things:

Cypher query given user prompt, system prompt (with ability to fix part that depends on the data / schema) and parameters
Natural language response given user prompt, system prompt (and whatever goes in there), dummy data (that simulates database response) and parameters
I want these to be stable over time
and possible to run within CI/CD pipeline.
So I believe that it means interacting with the system at a different level that you have in mind.
But I'm open to different options, if they're more convenient for you.
However, please note that I want to be able to fix / set as constant parts of the input that user shouldn't be able to control
Including mock schema and mock response from cypher query.

what is the best way to accomplish this?
I want to ingest all tickets from src_jira.src_tickets once. Is it to push a MR for stg_tickets with this one line commented out (see screenshot), then push a MR with it un-commented out after stg_tickets refreshes?
The context: we started ingesting the column PILLAR on 9/30/2024. However, there are tickets with PILLAR != null that were last loaded BEFORE 9/30/2023, so they never had pillar updated in our datawarehouse.

Summary Workflow

1. PDFs are uploaded to the backend.
2. Backend processes the documents, splits text, and builds a vector store.
3. Users ask questions via the React app.
4. The backend retrieves relevant chunks, generates answers, and sends them back to React.
5. React displays the response, references, and follow-up questions dynamically.

I am a seasoned professional with experience in designing and deploying AI-powered solutions, including large language models (LLMs), Retrieval-Augmented Generation (RAG) systems, and knowledge graphs. My expertise spans across various domains such as safety, technical education, customer service, and generative AI application development. I bring a hands-on approach to building innovative systems, optimizing model parameters, and creating governance structures to ensure quality and reliability in AI solutions. Additionally, my experience in fostering collaboration among cross-functional teams positions me to lead projects effectively and deliver impactful results

My long-term goal is to develop and scale a robust GenAI ecosystem for AARP that enhances member engagement, accessibility, and personalized services. By leveraging advanced generative AI models and a structured approach to system testing, monitoring, and continuous improvement, I aim to ensure the reliability, fairness, and scalability of AI applications. Additionally, I aspire to establish AARP as a leader in ethical AI adoption, delivering meaningful innovations that align with its mission to empower and enrich the lives of its members, especially as they age. Over time, I would also focus on mentoring teams, building strong AI governance frameworks, and driving strategic initiatives that align with AARP’s broader goals of advocacy and service.

As the current solution is being build
I think it will have been good to have streamlined solution to connect Airtable with Tableau using a Web Data Connector (WDC) custom scrupt. With This approach we can still fetch only the specific columns we need directly from Airtable via its API, which minimizes data load and speeds up processing.

How It will Works:

• We use a lightweight HTML script to connect Airtable to Tableau.
• You can dynamically input your Airtable Base ID, Table Name, and desired columns through a simple form.
• Tableau will pull just those columns directly from Airtable—no need for manual CSV exports or additional tools.

Dear Brandon,

I hope you’re doing well. I’m reaching out to express my interest in any available roles within our AI initiative. I recently rolled off my project, where I led the LLM as Judge initiative, focusing on enhancing LLM evaluation and accuracy through innovative frameworks and integrations.

I’m eager to bring this expertise to a new challenge and contribute to the team’s success. Please let me know if there’s an opportunity to discuss how I can add value to the initiative.

Best regards,

Hi Prad,

I’m [Your Name], and I’m excited to step into Charlie’s role and take on the responsibilities he’s been managing. Charlie shared a bit about the position and the amazing work being done here, and I’m eager to contribute.

I have a strong background in AI and leadership, with experience in [briefly mention relevant experience or skills]. I’m looking forward to working with you and the team to continue driving impactful AI initiatives.

Please let me know how I can hit the ground running!

Best regards,

Hi Everyone,

Happy Monday! I hope you are doing well. Not everyone could joined the scheduled governance call. I will reschedule this to maintain momentum on this initiative with the goal to discuss our data governance / management objectives

As of today, I’ll be wrapping up my time on the project. I just wanted to say how grateful I am for the opportunity to work with each of you. It’s been a privilege to contribute to such a dedicated and talented team, and I truly appreciate all the support, collaboration, and learning along the way.

Thank you all for everything – I look forward to staying in touch and seeing the continued success of this project!

============
Confirm that we both have access to the same VPN
- Communicate that this application is not design to faceweb and should be secure.
- We can connect this appliction from from VPN.
- We need range of IP that the VPN uses.
- Which IP range should be whitelisted.
- Once we have Ressource group, I will create Virtual machine, Disable all connection not comming from the VPN IP range.
- Why do we need that? Also because such application has keys for OpenAI and could be abuve on the internet.Keys can leak if someone find vulnerability.

Session Topic,Description/Objective,Key Tools/Methods,Owner/Presenter,Date,Status
Test Case Management,Overview of managing and organizing test cases for LLM evaluation and testing.,"Test case tools, CSV trackers, Jira/Confluence",[Your Name],[Date],Not started
Accuracy Testing,"Discussing methods for measuring LLM accuracy, comparing ground truth vs model responses.","RAGAS, evaluation metrics (Precision, Recall, F1)",[Your Name],[Date],In progress
Policy Scraping,Overview of techniques for extracting policies from documents for testing purposes.,"Scraping libraries (BeautifulSoup), regex",[Your Name],[Date],Not started
File Hashing Comparison,Explaining file integrity checks using hashing to compare different versions of LLM responses.,Hashing algorithms (SHA256),[Your Name],[Date],Not started
LLM as a Judge,Exploring the use of LLMs to evaluate or grade answers based on given policies and criteria.,"LLM outputs, RAG for evaluation",[Your Name],[Date],Not started
Question Generation,Techniques for generating relevant and follow-up questions using topics from PDFs.,"RAG models, custom question generation pipelines",[Your Name],[Date],In progress
Connecting to the Sandbox,Guidelines for setting up and connecting LLM testing infrastructure to a sandbox environment.,"AWS Bedrock, Claude, RAGAS, API connections",[Your Name],[Date],Not started
Performance Testing,"Methods for testing LLM performance (response time, throughput) when handling large datasets or complex queries.","Load testing tools (e.g., Locust), logging setups",[Your Name],[Date],In progress
Pinging API with Multiple Questions,Automating and managing multiple API requests for varied question sets to stress test response accuracy.,"API automation scripts, Postman collections",[Your Name],[Date],Not started
Infrastructure Change Accuracy Testing,Methods for retesting LLM accuracy after infrastructure or environment changes.,"Infrastructure monitoring, comparison pipelines",[Your Name],[Date],Not started
Chunks Spell Checker,Using spell checkers and language tools to validate the accuracy of LLM-generated text chunks.,"Spell checking libraries, context validation tools",[Your Name],[Date],Not started
Non-English Text Validator,Testing and validating LLM responses for accuracy in non-English languages.,"Multilingual LLMs, language detection tools",[Your Name],[Date],In progress

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for traits
  Running setup.py clean for traits
Successfully built configobj
Failed to build traits
ERROR: Could not build wheels for traits, which is required to install pyproject.toml-based projects

import os
import fitz  # PyMuPDF
import re
import random

# Regular expressions for finding PII
EMAIL_REGEX = r'[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+'
PHONE_REGEX = r'\(?\b[0-9]{3}[-.)\s]*[0-9]{3}[-.\s]*[0-9]{4}\b'
NAME_REGEX = r'\b[A-Z][a-z]+(?:\s[A-Z][a-z]+)*\b'
ADDRESS_REGEX = r'\d{1,5}\s\w+(\s\w+)*,\s\w+(\s\w+)*,\s[A-Z]{2}\s\d{5}'

# Generate dummy-like data
def generate_dummy_email():
    domains = ["example.com", "dummy.org", "test.net"]
    username = ''.join(random.choices('abcdefghijklmnopqrstuvwxyz', k=8))
    return f"{username}@{random.choice(domains)}"

def generate_dummy_phone():
    return f"({random.randint(100, 999)}) {random.randint(100, 999)}-{random.randint(1000, 9999)}"

def generate_dummy_name():
    first_names = ["John", "Jane", "Alex", "Chris", "Sam", "Taylor"]
    last_names = ["Smith", "Doe", "Johnson", "Brown", "Davis", "Miller"]
    return f"{random.choice(first_names)} {random.choice(last_names)}"

def generate_dummy_address():
    street_names = ["Main St", "Broadway", "Elm St", "Maple Ave", "Oak St"]
    cities = ["Springfield", "Rivertown", "Lakeview", "Hilltop", "Greenville"]
    state_codes = ["CA", "NY", "TX", "FL", "IL"]
    return f"{random.randint(1, 9999)} {random.choice(street_names)}, {random.choice(cities)}, {random.choice(state_codes)} {random.randint(10000, 99999)}"

# Replace PII in text with dummy-like data
def replace_pii(text):
    # Replace emails
    text = re.sub(EMAIL_REGEX, lambda _: generate_dummy_email(), text)
    # Replace phone numbers
    text = re.sub(PHONE_REGEX, lambda _: generate_dummy_phone(), text)
    # Replace names
    text = re.sub(NAME_REGEX, lambda _: generate_dummy_name(), text)
    # Replace addresses
    text = re.sub(ADDRESS_REGEX, lambda _: generate_dummy_address(), text)
    return text

# Process each PDF and replace PII
def process_pdf(file_path, output_path):
    # Open the PDF
    document = fitz.open(file_path)
    
    # Iterate through each page
    for page_number in range(len(document)):
        page = document.load_page(page_number)
        text = page.get_text("text")
        
        # Replace PII in text
        new_text = replace_pii(text)
        
        # Redraw the page with new text
        page.clear_contents()  # Remove original content
        page.insert_text((72, 72), new_text, fontsize=12)  # Insert updated content at default margin
    
    # Save the modified PDF to output path
    document.save(output_path)
    document.close()

# Process all PDFs in a folder
def process_folder(folder_path, output_folder):
    # Ensure the output folder exists
    if not os.path.exists(output_folder):
        os.makedirs(output_folder)
    
    # Iterate over all PDF files in the folder
    for filename in os.listdir(folder_path):
        if filename.lower().endswith('.pdf'):
            file_path = os.path.join(folder_path, filename)
            output_path = os.path.join(output_folder, filename)
            process_pdf(file_path, output_path)
            print(f"Processed {filename}")

# Specify the input folder containing PDFs and output folder for processed files
input_folder = 'path/to/your/input/folder'
output_folder = 'path/to/your/output/folder'

process_folder(input_folder, output_folder)

AttributeError: 'Page' object has no attribute 'clear_contents'

# Process each PDF and replace PII
def process_pdf(file_path, output_path):
    # Open the PDF
    document = fitz.open(file_path)
    
    # Iterate through each page
    for page_number in range(len(document)):
        page = document.load_page(page_number)
        text = page.get_text("text")

        # Replace PII in text
        new_text = replace_pii(text)

        # Redact the existing text (mark the text to be erased) and insert new text
        page.delete_text()  # This deletes all the text on the page
        
        # Insert updated text
        page.insert_text((72, 72), new_text, fontsize=12)  # Insert the updated text at the default margin
    
    # Save the modified PDF to output path
    document.save(output_path)
    document.close()

# Process all PDFs in a folder
def process_folder(folder_path, output_folder):
    # Ensure the output folder exists
    if not os.path.exists(output_folder):
        os.makedirs(output_folder)
    
    # Iterate over all PDF files in the folder
    for filename in os.listdir(folder_path):
        if filename.lower().endswith('.pdf'):
            file_path = os.path.join(folder_path, filename)
            output_path = os.path.join(output_folder, filename)
            process_pdf(file_path, output_path)
            print(f"Processed {filename}")

AttributeError: 'Page' object has no attribute 'delete_text'

# Process each PDF and replace PII
def process_pdf(file_path, output_path):
# Open the PDF
document = fitz.open(file_path)

# Iterate through each page
for page_number in range(len(document)):
page = document.load_page(page_number)
text_instances = []

# Search for PII in the text and store their positions
for regex, dummy_func in [(EMAIL_REGEX, generate_dummy_email),
(PHONE_REGEX, generate_dummy_phone),
(NAME_REGEX, generate_dummy_name),
(ADDRESS_REGEX, generate_dummy_address)]:
matches = re.finditer(regex, page.get_text("text"))
for match in matches:
dummy_text = dummy_func()
text_instances.append((match.start(), match.end(), match.group(), dummy_text))

# Redact and replace with dummy text
for start, end, original_text, dummy_text in text_instances:
text_rects = page.search_for(original_text)
for rect in text_rects:
page.add_redact_annot(rect, fill=(1, 1, 1)) # Add a white redaction
page.apply_redactions()
page.insert_text((rect.x0, rect.y0), dummy_text, fontsize=12)

# Save the modified PDF to output path
document.save(output_path)
document.close()

# Process all PDFs in a folder
def process_folder(folder_path, output_folder):
# Ensure the output folder exists
if not os.path.exists(output_folder):
os.makedirs(output_folder)

# Iterate over all PDF files in the folder
for filename in os.listdir(folder_path):
if filename.lower().endswith('.pdf'):
file_path = os.path.join(folder_path, filename)
output_path = os.path.join(output_folder, filename)
process_pdf(file_path, output_path)
print(f"Processed {filename}")

import os
import fitz  # PyMuPDF
import re
import random

# Regular expressions for finding PII
EMAIL_REGEX = r'[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+'
PHONE_REGEX = r'\(?\b[0-9]{3}[-.)\s]*[0-9]{3}[-.\s]*[0-9]{4}\b'
NAME_REGEX = r'\b[A-Z][a-z]+(?:\s[A-Z][a-z]+)*\b'
ADDRESS_REGEX = r'\d{1,5}\s\w+(\s\w+)*,\s\w+(\s\w+)*,\s[A-Z]{2}\s\d{5}'

# Generate dummy-like data
def generate_dummy_email():
    domains = ["example.com", "dummy.org", "test.net"]
    username = ''.join(random.choices('abcdefghijklmnopqrstuvwxyz', k=8))
    return f"{username}@{random.choice(domains)}"

def generate_dummy_phone():
    return f"({random.randint(100, 999)}) {random.randint(100, 999)}-{random.randint(1000, 9999)}"

def generate_dummy_name():
    first_names = ["John", "Jane", "Alex", "Chris", "Sam", "Taylor"]
    last_names = ["Smith", "Doe", "Johnson", "Brown", "Davis", "Miller"]
    return f"{random.choice(first_names)} {random.choice(last_names)}"

def generate_dummy_address():
    street_names = ["Main St", "Broadway", "Elm St", "Maple Ave", "Oak St"]
    cities = ["Springfield", "Rivertown", "Lakeview", "Hilltop", "Greenville"]
    state_codes = ["CA", "NY", "TX", "FL", "IL"]
    return f"{random.randint(1, 9999)} {random.choice(street_names)}, {random.choice(cities)}, {random.choice(state_codes)} {random.randint(10000, 99999)}"

# Replace PII in text with dummy-like data
def replace_pii(text):
    # Replace emails
    text = re.sub(EMAIL_REGEX, lambda _: generate_dummy_email(), text)
    # Replace phone numbers
    text = re.sub(PHONE_REGEX, lambda _: generate_dummy_phone(), text)
    # Replace names
    text = re.sub(NAME_REGEX, lambda _: generate_dummy_name(), text)
    # Replace addresses
    text = re.sub(ADDRESS_REGEX, lambda _: generate_dummy_address(), text)
    return text

# Process each PDF and replace PII
def process_pdf(file_path, output_path):
    # Open the PDF
    document = fitz.open(file_path)
    
    # Iterate through each page
    for page_number in range(len(document)):
        page = document.load_page(page_number)
        text_instances = []

        # Search for PII in the text and store their positions
        for regex, dummy_func in [(EMAIL_REGEX, generate_dummy_email), 
                                   (PHONE_REGEX, generate_dummy_phone),
                                   (NAME_REGEX, generate_dummy_name), 
                                   (ADDRESS_REGEX, generate_dummy_address)]:
            matches = re.finditer(regex, page.get_text("text"))
            for match in matches:
                text = match.group()
                text_rects = page.search_for(text)
                for rect in text_rects:
                    text_instances.append((rect, dummy_func()))

        # Apply redactions and insert dummy text
        for rect, dummy_text in text_instances:
            page.add_redact_annot(rect, fill=(1, 1, 1))  # Add a white redaction to cover original text

        page.apply_redactions()  # Apply all redactions

        for rect, dummy_text in text_instances:
            page.insert_text((rect.x0, rect.y0), dummy_text, fontsize=12, color=(0, 0, 0))  # Insert dummy text at the same position
    
    # Save the modified PDF to output path
    document.save(output_path)
    document.close()

# Process all PDFs in a folder
def process_folder(folder_path, output_folder):
    # Ensure the output folder exists
    if not os.path.exists(output_folder):
        os.makedirs(output_folder)
    
    # Iterate over all PDF files in the folder
    for filename in os.listdir(folder_path):
        if filename.lower().endswith('.pdf'):
            file_path = os.path.join(folder_path, filename)
            output_path = os.path.join(output_folder, filename)
            process_pdf(file_path, output_path)
            print(f"Processed {filename}")

# Specify the input folder containing PDFs and output folder for processed files
input_folder = 'path/to/your/input/folder'
output_folder = 'path/to/your/output/folder'

process_folder(input_folder, output_folder)

===============

# Regular expressions for finding PII
EMAIL_REGEX = r'[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+'
PHONE_REGEX = r'\(?\b[0-9]{3}[-.)\s]*[0-9]{3}[-.\s]*[0-9]{4}\b'
NAME_REGEX = r'\b[A-Z][a-z]+(?:\s[A-Z][a-z]+)*\b'
ADDRESS_REGEX = r'\d{1,5}\s\w+(\s\w+)*,\s\w+(\s\w+)*,\s[A-Z]{2}\s\d{5}'

# Generate dummy-like data
def generate_dummy_email():
    domains = ["example.com", "dummy.org", "test.net"]
    username = ''.join(random.choices('abcdefghijklmnopqrstuvwxyz', k=8))
    return f"{username}@{random.choice(domains)}"

def generate_dummy_phone():
    return f"({random.randint(100, 999)}) {random.randint(100, 999)}-{random.randint(1000, 9999)}"

def generate_dummy_name():
    first_names = ["John", "Jane", "Alex", "Chris", "Sam", "Taylor"]
    last_names = ["Smith", "Doe", "Johnson", "Brown", "Davis", "Miller"]
    return f"{random.choice(first_names)} {random.choice(last_names)}"

def generate_dummy_address():
    street_names = ["Main St", "Broadway", "Elm St", "Maple Ave", "Oak St"]
    cities = ["Springfield", "Rivertown", "Lakeview", "Hilltop", "Greenville"]
    state_codes = ["CA", "NY", "TX", "FL", "IL"]
    return f"{random.randint(1, 9999)} {random.choice(street_names)}, {random.choice(cities)}, {random.choice(state_codes)} {random.randint(10000, 99999)}"

# Replace PII in text with dummy-like data
def replace_pii(text):
    # Replace emails
    text = re.sub(EMAIL_REGEX, lambda _: generate_dummy_email(), text)
    # Replace phone numbers
    text = re.sub(PHONE_REGEX, lambda _: generate_dummy_phone(), text)
    # Replace names
    text = re.sub(NAME_REGEX, lambda _: generate_dummy_name(), text)
    # Replace addresses
    text = re.sub(ADDRESS_REGEX, lambda _: generate_dummy_address(), text)
    return text

# Process each PDF and replace PII while trying to retain the original structure
def process_pdf(file_path, output_path):
    # Open the PDF
    document = fitz.open(file_path)
    
    # Iterate through each page
    for page_number in range(len(document)):
        page = document.load_page(page_number)
        text_instances = []

        # Get the text and rectangles containing PII
        text = page.get_text("text")

        for regex, dummy_func in [
            (EMAIL_REGEX, generate_dummy_email), 
            (PHONE_REGEX, generate_dummy_phone),
            (NAME_REGEX, generate_dummy_name), 
            (ADDRESS_REGEX, generate_dummy_address)
        ]:
            matches = re.finditer(regex, text)
            for match in matches:
                original_text = match.group()
                text_rects = page.search_for(original_text)
                for rect in text_rects:
                    dummy_text = dummy_func()
                    text_instances.append((rect, original_text, dummy_text))

        # Redact and replace PII in the original positions
        for rect, original_text, dummy_text in text_instances:
            # Create a redaction annotation over the PII
            page.add_redact_annot(rect, fill=(1, 1, 1))  # White redaction to mask the original text

        # Apply all redactions
        page.apply_redactions()

        # Insert the dummy text in place of redacted areas
        for rect, original_text, dummy_text in text_instances:
            page.insert_text((rect.x0, rect.y0), dummy_text, fontsize=10, color=(0, 0, 0))

    # Save the modified PDF to output path
    document.save(output_path)
    document.close()

# Process all PDFs in a folder
def process_folder(folder_path, output_folder):
    # Ensure the output folder exists
    if not os.path.exists(output_folder):
        os.makedirs(output_folder)
    
    # Iterate over all PDF files in the folder
    for filename in os.listdir(folder_path):
        if filename.lower().endswith('.pdf'):
            file_path = os.path.join(folder_path, filename)
            output_path = os.path.join(output_folder, filename)
            process_pdf(file_path, output_path)
            print(f"Processed {filename}")

    with st.expander("Show", expanded=False, icon=None):
        for question in follow_up_questions:
            if st.button(question):
                st.session_state['prompt_bar'] = question
                st.session_state["follow_up_question"] = question
                print(f"Follow up is {question}")
                # st.stop()

Hi Nigel,

I wanted to inform you that I’ve published the Confluence page regarding the concerns and risks related to the design of the team knowledge graph with the cyberbot system. You can review it using the following link: [Insert Confluence Page Link].

Could we schedule a brief meeting in about an hour to go over the content and address any questions you might have?

Looking forward to your feedback.

Best regards,
Mohamed

VPN Access and Security for the domain insight application Setup

I’d like to confirm that we both have access to the same VPN, as it’s essential for securing this application. Since this app is not designed for external web access, we need to ensure it’s securely accessed only via VPN.

Here’s a brief outline of our security steps:

• The application will be accessible only from within the VPN.
• We’ll need the range of IPs that the VPN uses, as these will be whitelisted for access.
• Once the resource group is ready, I’ll create a virtual machine and disable all connections not originating from the VPN IP range.

This approach is crucial because the application stores sensitive keys, including OpenAI keys, which could be at risk if vulnerabilities are exploited on an open network. Limiting access to VPN IPs will greatly reduce this risk.

Please let me know if we’re aligned on the VPN and IP range details to proceed securely.

### Todd meeting

Goals for the meeting with Todd:

1. Getting his input about possible places where any type of capturing logic can be placed (CI/CD pipeline, DEV?QA deployment, etc.)
2. Getting information how environment from 1. is set up (is the data the same for example)

For things that can be captured. Any parameters for the model

1. That includes system prompt if it changes and / or is augmented with live queried data
2. User prompt / conversation chain
3. LLM version
4. ...
5. Generated cypher query and (optionally) neo4j execution plan
6. Data returned from the datatabase
7. Generated description of data

For things that can be measured

1. "Drift" of the query (possibly low hanging fruit, as this should be restricted)
2. "Drift" of the description / answer (hard, especially with changing data).

Note that code generation (text -> cypher) and summarizer (data -> response) are two different "AI" applications and should be evaluated separately.

=================

Here is my thoughts a solution for capturing and monitoring the cypher bot model drift and other critical system metrics.
The main idea here is to separate data collection and evaluation as well as introduce higher granularity of data capturing for evaluation and monitoring.
Below is a concise outline of our goals to ensure we achieve this effectively:
Short-Term Goals
To enable robust drift and metric monitoring, we need to capture various aspects of the system:
• Code version
• Model(s) version and parameters
• Data version (i.e. versioned test database)
• Inputs and outputs
To ensure comprehensive monitoring, we should separate and test individual components, such as:
1. (system-prompt, user-prompt) -> cypher-query
2. (system-prompt, user-prompt, database-version) -> neo4j-execution-plan
3. (user-prompt, data) -> summary
These metrics can be captured via a CI/CD pipeline using programmatic APIs or through QA deployments with REST APIs triggered by web-hooks.
• All data should be stored persistently (e.g., MongoDB, S3-compatible storage) for further inspection and analysis.
• Data-related risks can be mitigated by using synthetic or anonymized data for logged information.
The choice of a particular approach is mostly irrelevant, as long as other goals are satisfied, so let's choose approach that is the least intrusive.
Mid-Term Goals
Extend question “coverage” to better understand system changes over time, including:
• Questions meaningful to stakeholders.
• Expert-written Cypher queries.
Long-Term Goals, starting ASAP
• Include SME level descriptions of the data for given questions, where possible.
• Continuously evaluate and improve tools and metrics with persistent logs, enabling retrospective quality analysis.
• Adapt to the evolving landscape of tools and frameworks, ensuring our solution remains future-proof.
We will need a set of APIs to facilitate the functionalities described above:
User Question to Cypher Query
An API endpoint that takes a user’s question as input and returns the corresponding Cypher query.
2 . Cypher Query to Execution Plan
An API endpoint that receives a Cypher query as input and retrieves the execution plan from a fixed, controlled database designed specifically for testing purposes.
3. User Question to Natural Language Answer Response
An API endpoint that takes a user’s question, combines it with example data, and returns a natural language answer generated by the system.

==========================

Jira Ticket 2: Develop Component Tests and CI/CD Integration

Ticket Title:
Develop Component Testing and CI/CD Pipeline for Drift Monitoring

Description:
To ensure comprehensive drift monitoring for the Text-to-Cyber Chatbot model, separate and test individual components. Integrate these tests into a CI/CD pipeline to automate metric tracking and trigger drift detection workflows programmatically.

Acceptance Criteria:
1. Component Tests:
• Create unit tests for the following:
1. (system-prompt, user-prompt) -> cypher-query: Validate the correctness of generated Cypher queries.
2. (system-prompt, user-prompt, database-version) -> neo4j-execution-plan: Ensure execution plans are accurate and efficient.
3. (user-prompt, data) -> summary: Verify generated summaries align with input prompts and data.
2. CI/CD Integration:
• Develop a pipeline to run component tests automatically during deployments.
• Enable metric tracking for:
• Query accuracy.
• Execution success rates.
• Summary quality metrics (e.g., BLEU or ROUGE scores).
• Configure REST APIs or webhooks to trigger QA deployments for additional testing.
3. Alert Mechanisms:
• Implement alerting for drift detection when metrics deviate from defined baselines.
4. Reporting and Dashboards:
• Provide logs and metrics for analysis in a visual dashboard (e.g., Grafana or Streamlit).

Dependencies:
• Access to QA deployment environment for testing pipeline integration.
• Agreement on CI/CD tool and workflow design (e.g., GitHub Actions, Jenkins).

Priority:
High

==========

Jira Ticket 1: Implement Versioning and Persistent Logging Framework

Ticket Title:
Implement Versioning and Persistent Logging for Drift Monitoring

Description:
To enable robust drift and metric monitoring for the Text-to-Cyber Chatbot model, implement a framework to capture and store system data persistently. This includes versioning of code, model, and data, as well as logging inputs and outputs for future inspection and analysis.

Acceptance Criteria:
1. Versioning:
• Track code version via Git commit hashes.
• Store model version and parameters in a version control tool (e.g., MLflow or Weights & Biases).
• Maintain a versioned test database with snapshots of schema and data.
2. Persistent Storage:
• Log inputs and outputs systematically in a centralized, persistent storage system (e.g., MongoDB or S3-compatible storage).
• Design a schema to capture:
• System prompt.
• User prompt.
• Generated outputs (queries, execution plans, summaries).
• Error logs and metrics.
• Mitigate data risks by using synthetic or anonymized data where applicable.
3. Documentation:
• Document the storage structure and versioning strategy in the project wiki.

Dependencies:
• Decision on the choice of persistent storage (e.g., MongoDB vs. S3-compatible storage).

Priority:
High

===============

am a seasoned professional with experience in designing and deploying AI-powered solutions, including large language models (LLMs), Retrieval-Augmented Generation (RAG) systems, and knowledge graphs. My expertise spans across various domains such as safety, technical education, customer service, and generative AI application development. I bring a hands-on approach to building innovative systems, optimizing model parameters, and creating governance structures to ensure quality and reliability in AI solutions. Additionally, my experience in fostering collaboration among cross-functional teams positions me to lead projects effectively and deliver impactful results.

=====================

Team, I’ve reviewed the Gemini code merge with Neo4j for evaluating model drift in our chatbot. The code is workable but needs additional capabilities for:
• Extract prompts, parameters, etc., for better inspection and logging.
• Generalize model access (e.g., consider LiteLLM) to handle breaking changes seamlessly.
• Abstract the Chat class API to decouple it from implementation details.

Additionally, guardrails should enforce instructions (e.g., “do not generate/execute DELETE queries”) rather than relying solely on documentation.

@valentina
Questions to address:
1. Which storage option for storing the results can be easily accessed within CICD, and how?
2. Which database already has connectivity established within CICD—S3 or MongoDB?

========================

Jira Ticket
Project: Membership Renewal System

Issue Type: Bug

Summary: Incorrect Membership Renewal Price Displayed by LLM

Description: The LLM (Language Learning Model) provided an incorrect price for a 5-year membership renewal to a consumer. The LLM stated that the renewal rate is $9 per year, which is outdated and incorrect due to a recent membership price increase. The correct rate is $79 for 5 years, averaging $15.80 per year. This discrepancy caused confusion and required manual correction by a contact center agent.

Steps to Reproduce:

Initiate a chat with the LLM (AARPBOT).
Ask for the discount rate for renewing a membership for 5 years.
Observe the response provided by the LLM.
Expected Result: The LLM should provide the correct renewal rate of $79 for 5 years, averaging $15.80 per year.

Actual Result: The LLM incorrectly states that the 5-year renewal rate is $9 per year.

Attachments:

Transcript from 12:53 PM PT showing the incorrect information provided by the LLM and the subsequent correction by the agent.
Priority: High

Assignee: [Assign to appropriate team member]

Labels: LLM, Pricing, Membership Renewal, Bug

Environment:

LLM (AARPBOT) in the customer support chat system.
Additional Notes:

Ensure the LLM is updated with the latest membership pricing information.
Implement stricter controls to prevent outdated pricing from being displayed.
Train agents to handle such discrepancies effectively and monitor LLM responses regularly for accuracy.
Reporter: [Your Name]

This Jira ticket outlines the issue, expected and actual results, and provides steps for resolution to ensure the LLM provides accurate pricing information in the future.

Copy

Regenerate

Type a message. Use Shift+Enter for a new line

Deloitte data

======================================================

Here is a summarized version of the task: Once we choose one, we will do the research. That's the first step

Task 1: Teams CoPilot Transcription & Incident Management Integration
• Goal: Capture Teams CoPilot voice transcriptions from P1/P2 incident calls and integrate them into Slack (soon to be Teams) Incident Management threads and ServiceNow.
• Next Steps:
1. Research Teams CoPilot transcription capabilities & APIs.
2. Test automation using Microsoft Graph API, Power Automate, or webhooks.
3. Prototype extraction & integration into Slack/ServiceNow.

Task 2: Teams AI Bot Development & Migration
• Goal: Investigate how to build a Teams AI bot and identify development challenges, security, and cost considerations.
• Next Steps:
1. Review Microsoft Bot Framework & Teams Bot APIs.
2. Test creating a simple Teams bot.
3. Analyze security, authentication (Azure AD), and cost.
4. Plan migration of existing chatbots into Teams.

https://learn.microsoft.com/en-us/microsoftteams/platform/graph-api/meeting-transcripts/overview-transcripts?utm_source=chatgpt.com