# HENM NextAudit Meeting Notes

## Team Meeting Notes (2020-10-30)

- Recommendation for findings
- Recommendation for measure
- Non-Conformity recommendation may be immediately
- measures are usually added later (no real time requirement)

### Target Next Meeting
- Technical Details
- Code Hosting
- Issue Management
- Goal: Written Definition of Goals Project should achieve
- Send our questions to nextAudit 4.11/5.11?

## Team Meeting II (2020-11-06)

### Project Goal Proposal

The development and further implementation into the existing BE/FE system of:
+ a recommender engine suggesting for any new question per audit type an ordered list of potential findings based on previous findings for context-similar questions.
+ a recommender engine suggesting for any new finding per (audit type, question) an ordered list of potential measures based on previous measures for context-similar (question, finding) pairs.
+ The possibility to update the training dataset and retrain by user feedback on the recommended suggestion (by selection)

## Customer Meeting Notes (2020-11-06)

### Deliverables
- Implementation (commented code) + sufficient documentation to follow
- Introduction to NLP is of interest
- Shipable Prototype

### Technical Details
- Code Hosting
- Gitlab Projects will be set up
- Issue Management
- Gitlab Issues
- Deployment
- Gitlab CI/CD
- GCP Based
- Probably Cloud Functions but not fixed
- App Engine in Evaluation
- Model Training on RWTH Ressources, just deployment to GCP
- Target Plattform
- TypeScript + express based REST-API
- Data Sources
- Based on firestore
- Training data probably provided as json export
- Code Quality
- https://gitlab.com/nextaudit/audit-cloud-ai
- Has eslint config
- Look for test implementation

### Review Requirements
- Can use Arbeitsplan
- Can get slides for market analysis
- Personas not available
- Think of something else
- Possibility for interview to analyse target group

### Remote Project
- No measures yet, just quetions and findings
- not iso19011
- Images not important yet
- but keep possibility to extend in the future in mind (OCR/speech-to-text) probable project outlook
- Audit Tempates
- Questions will become editable (e.g. add free questions)
- Every question refers to a chapter in the standard --> use this information
- Questions may focus on individual subsystems impacted by the chapter
- Measure Types
- Can be ignored
- Probably still include them (DISCUSS)

### Data Structure

* Almost everything included in measureitem
* Should be available someday next week

### Project Goal

The development and further implementation into the existing BE/FE system of:
+ a recommender engine suggesting for any new question per audit type an ordered list of potential findings based on previous findings for context-similar questions.
+ a recommender engine suggesting for any new finding per (audit type, question) an ordered list of potential measures based on previous measures for context-similar (question, finding) pairs.
+ The possibility to update the training dataset and retrain by user feedback on the recommended suggestion
+ Feedback: Star Rating of Effectiveness (to be introduced)

--> Agreed

## Followup Team Meeting Notes (2020-11-06)

### ToDos
- [x] Log-In to test instance
- [x] (monday) Fill out experience template
- [x] (monday) Discuss Business Case Slides
- [x] (wednesday) Get familiar with (frontend) Code
- [x] (wednesday) Prepare User Stories
- [x] (wednesday) Review Review Material
- [x] Transfer Tasks to issues

## Team-Meeting (2020-11-09)

### Agenda
- Define Scrum Roles
- Create User Stories

### Notes
- Scrum Roles: Master: Lisa, Rest: Members

## Team-Meeting (2020-11-11)

### Agenda
- Select User Stories from suggested Input
- Clarify which testing we should run (maybe to implement e2e is not worth it within our timeframe)

### Notes
- Scrum Ceremony Schedule
- Daily
- On weekly basis
- Friday evening
- Communicate via Slack
- Dynamic additional meetings
- Planning
- Before Review
- Customer Review
- Every 2nd Friday in recurring meeting
- Retro
- In Customer Review Weeks

## Customer-Meeting (2020-11-13)

### Agenda
- Agree on User Stories
- Market Analysis

### Notes
- Story discussion:
- 1. story: ok, findings, measure: just list of highest rated.
- create wireframe or sketch to agree on
- 2. story:
- idea: good, all questions have unique id + chapter => simple context/meta data matching.
- 3. story: good, measures not in wireframe atm.
- 4. story: good.
- 5. story: good. Baseline: selection/frequency, extended: rating
- 6. story sounds good, but no priority, add later.

- REST APIs
- see nextAudit slides

- Slides Feedback:
- seems good so far, share with Alex.

## Team-Meeting (2020-11-13)

### Agenda
- (Discuss Components of FE?) (ok)
- Refine User Stories (ok)
- Tasks
- Assign Tasks to Team Members
- Discuss Deliverables for 1st Review (X)

### Notes
- Story 1
- Create sketches for recommendation UI flow (mohammad)
- Agree on sketches with customer (team)
- Implement Vue component with mock data (open pool)
- Story 2:
- Get better understanding of communication flow.
- analyse question and finding data for non-nlp recommendation system
- baseline matching design (target data structure)
- data preprocessing pipeline
- Wire frontend component with suggestions from recommendation service

- Git Flow Model
- Include in final review

## Team Meeting (2020-11-18)

### Agenda
- Recover from emotional shock (ok)
- Look into Backend Code as a next task (ok)
- Prepare questions to reflect how 1st review could get so out of hand (ok)
- Prepare UI sketch for Friday (Mohammad, ok)
- Discuss Communication Flow (ok)
- Visualize Communication Flow (ok)
- Maybe: prepare to discuss data input - data output for models with customer, only 2 options (Fred) ? (ok)
- look for code samples we could provide (if necessary)
- Vue.js very very simple integration
- Data Preprocess pipeline for json data (needed anyways)
- https://jupyter.rwth-aachen.de/hub/login or colab or ... [how does jupyter hub handles data uploads, colab resets all 12h and forces a new dataset upload I think...] (ok)

### Notes
+ Not real code received yet -> discuss on friday.
+ Email Frowin for Friday. (He is on the invitation list for the recurring meeting)
+ Mail to Alex "Supervisor"
+ "git" jupyter notebooks
+ https://gitlab.com/henm-project/data-processing

## Customer Meeting (2020-11-20)

### Agenda
- Ask Frowin whether he is on the mailing list (yes)
- Discuss component flow (yes)
- Discuss Sketches (yes)
- Discuss Model structure (top level view) (yes)

### Design
- Proposal 1:
- Suggestion: When typing something suggestions should be limited to similarity matched
- Suggestion: Button should show the number of recommendations
- Proposal 2: Start Typing, recommendations start to show
- Could do, maybe later
- could be more interesting for measures

### Data Analysis
- Probably try to detect names an obfuscate them
- Long run idea: Have placeholders and introduce input fields in the recommendation field

### Component Flow
- Firebase document size is limited
- Consider storage efficient recommendation storage layout
- Do not recompute measure recommendations

## Team Meeting (2020-11-20)

- Preparation for Monday?
- Prepare frontend and preproc-pipeline
- Question for Monday: Presentation Content Distribution

## Customer Meeting (2020-11-27)

### Agenda
+ Question: Question ID's in JSON always the same across audits? (yes)
+ Discuss Vue Component with customer (yes)
+ Discuss anonymization on real client data (yes)
+ Discuss BE issues (yes)
+ Short Note on Model Structure (Fred) (yes)
+ Note if you are interested in the offer on Entrepreneurship (yes)

### Notes
+ Only select one finding per question, what type of finding (major, minor),
- remove right selection window
- Update sketches
- Filter by deviation type (major/minor)
+ Question IDs globally same
+ Use for recommendation association
+ Other Personal Data Found (focus on personal information first)
+ emails
+ telephone numbers
+ locations
+ company names
+ register numbers
+ domains/urls
+ file names and paths
+ Custom List for tokens to anonymize
+ intgrate into general stopword removal process
+ store somewhere in json, readin in process

## Team Meeting (2020-11-27)
Out of scope note: remove title from names as well

### Agenda
+ Collect prospects for fireside chat (yes)
+ Sprint Review/Retro? (discussed, monday)
+ Discuss Baseline model (Fred) (yes)
+ Collect new isses for next sprint (some already listed)

### Notes
+ Entrepreneurship -> Slack
+ Sprint Review
+ Tasks mainly in review
+ All test backend installation/setup
+ Sprint Planning
+ Extend same stories
+ More anonymization
+ Redesign UI according to feedback
+ Implement backend endpoint
+ Wire backend endpoint to frontend
+ Sprint Retro
+ Everyone fills out and votes in https://easyretro.io/publicboard/GG6C68wXsLS3VhITgQF2EiKQKlI3/6f7b5aa0-adff-454b-9bfd-1a9d1dc47578
+ Discussion on Monday 17:15

#### Sprint 2 Tasks

- Question - Question Matching
- Question - Finding Matching

## Team Meeting (2020-11-30)

### Agenda
- Discuss retro notes

### Action Items Sprint Retro
- Create Slackbot reminder for Wednesday in #midweek-updates
- Quickly feedback missing information or non-working code to nextAudit
- Account 45 minutes for retro next time
- Kepp notes in Hackmd
- Encourage everyone to update the agenda before meetings
- Try to create task pools and primary assignees in sprint planning sessions.
- Agree on agenda mutually (and stick to it) for all meetings to avoid context mixing.
- Keep pair programming in mind when creating tasks
- When a feature is ready for review create a mr and notify on slack
- Share Current Status via Screenshare
- Notify on slack when scheduling issues occur as soon as possible.

## Customer Meeting (2020-12-04)

### Agenda
- Presentation of new FE component (Johannes)
- Presentation of Anonymizer (Fred)
- Presentation of BERT Idea (Xuefeng)
- Presentation of Working Example without classification (Fred)
- Hint for upcoming review

### Notes
- Measure in FE now working
- "Nicht Zutreffend" and "I.O." not to recommend!
- BE has to provide major/minor deviation type
- Anonymizer needs predefined token list
- Mail for anonymizer instructions (Fred)

## Team Meeting (2020-12-04)

### Agenda
- Prepare Review II
- come up with questions
- Friday Meeting Schedule
- Merge Requests (Fred)
- Titles in anonymizer
- FE/BE Wiring

### Notes

- Review
- Wait and see
- Hope for requirements to be sent out on Monday
- Open Issues
- FE: Next PR by Johannes
- BE: Install issues still unresolved
- yarn fix now in master
- use json import tool if firestore import fails
- BE: First simple endpoint implemented
- BE/FE: Store action can be implemented to wire both
- Swapping might make sense start time will be 2pm either way
- Titles
- Keep for now, measure accuracy
- Consider if possible to replace in a postprocessing step

## Customer Meeting (2020-12-11)

### Agenda
- very short FE update (filter) (Johannes)
- Prepare Review II
- come up with questions:
* How are the endpoints supposed to communicate with FE?
* How are the recommendations going to be stored and updated
* What approaches are you currently pursueing for the Recommender?
* Are personal information included in the training data?
* What were your largest challenges concerning teamwork?
* reusable components
* Agile appearing tasks
- Export firestore fixture

### Notes
- Last meeting 2020: next week
- First meeting 2021: 8.1.

## Team Meeting (2020-12-11)

### Agenda
- Discuss all points on TODO for 2nd review
- distribute tasks
- decide who presents
- Project Backlog (Thore)
- DoD (Thore)
- Demo (Johannes)
- Team Progress (Johannes)
- Conclusion/Outlook (Mohammad)
- Collect issues for the third sprint

## Customer Meeting (2020-12-18)

### Agenda
- Short Update on status of semantic matcher (Fred)
- Short Feedback on model evaluation idea (Fred)
- Doublecheck endpoint changes with nextAudit (Thore?)

### Notes
- Filter findings also per recommendation in data model
- 1 doc per Recommendation is ok
- Calculate with
- 100 questions per audit
- 100 audits
- 150 items per audit
- documents can't be larger than 1MiB
- Cost Calculation
- Meeting on Data stuff 21. 14:30 -> Mail
- Vacation
- Frowin from 21st to 4th.
- Next Meeting on the 8th
- Contact Alex if necessary
-

## Team Meeting (2020-12-18)
### Agenda
- Data Proc repo again huge (350MB) :(
- Will always stay in history -> true, ok
- Short question on storing recommendations (Fred)
- Anything to prepare for monday?
- Most likely some kind of code review (each one a sample of project related code?)
- Architecture documentation document
- NLP method evaluation discussion (Fred)
- Anyone else experiencing lags in Hackmd (yes)?
- Plan for the rest of this year?
- Any standups/synchronization
- (OT) Anyone attending RC3?

### Notes
- is_no_deviation in parser (Fred)
- Ask Alex on Monday whether we have to create a project architecture documentation for the final product
- Split document up
- Team Coordination next 2 weeks
- Wednesday updates will continue
- Meeting 28th 2PM

## Review Review Meeting (2020-12-21)

### Notes
- Include a questions slide in the next slideset
- The third review is going to be very similar to the second
- A bit more focus on technical stuff
- testing
- ci/cd
- Final presentation is going to be a 5 minute business oriented pitch.

## Team Meeting (2020-12-28)

### Agenda

- Last Sprint Retro
- How did our action items work out?
- Architecture Documentation for NA?
- UMLs
- Cost Consideration
- Outlook
- Continuous learning
- Recommendation Updates
- Deploying and updating models
- Prepare for third review
- Elaborate on CI pipelines (linting, testing, extend testing where possible)
- Code Quality (Xuefeng) (Fred)
- Short update on Recommender (Fred)
- Split up baseline model for ingestion
- Sprint Planning
- Already plan data ingestion (recommender -> firestore)

### Notes

- Project Handbook
- https://mettmd.selfnet.de/pNPHLiHiRWqVH5gYHO4Nlg?edit
- Centered to be used by nextaudit
- might be handed in at the end of the project
- focussed to keep our thoughts and ideas alive
- Third review
- Include gitlab-ci pipeline (probably as question from NA)
- Next Meeting:
- Firday if necessary
- if necessary is decided on Thursday 8pm (on slack)
- otherwise 8th

### Next Sprint

- Unit Tests for frontend
- Build storage for association of finding recommendations to questions
- ingestion feature to pass the final storage model to firebase
- finding-feedback endpoint
- add new issue specific to feedback
- measures five star rating + feedback (fe/be)
- rating (five star)
- feedback (has been selected)
- present recommendations in the frontend which have been fetched from the backend

## Customer Meeting (2021-01-08)

### Agenda
- Short status update (all)
- Discuss Storage Model (again) (Fred, Thore)
- Again Collection of Questions for Review [e.g. Testing] (all)

### Notes
- No unique ids for questions, findings
- GCloud capacity free
- Questions to prepare for review

## Team Meeting (2021-01-08)

### Agenda
- Discuss Storage Model (again) (Fred, Thore)
- Cost Evaluation for review?
- Discuss Use of Master Branches
- Exchange firestore data of BE (including recommendations)
- Model Evaluation (Lisa)
- Sprint Planning

### Notes

- Discuss issues to discuss with frowin
- unique auditItemId + findingId (hashing ourselfs needed)
- deviation types different across customers (please not)
- final agreement on storage model (query-cost update)
- New finding added by linker (Fred)
- Probably add to outlook (with continuous learning model)
- Merge dev into master after each sprint
- json export for firestore
- idiots approach: set in fetch endpoint

### Sprint Planning

## Team Meeting (2021-01-13)

For Project Documents see https://hackmd.io/aGV2bU42QfeIwqNowGNQbA?edit

### Agenda
- Progress on sprint (e.g. realistic to get generated finding recommendations to be provided by BE until review?)
- Build recommendation ingester into firebase emulator
- possibly skip hashing for now
- copy finding recommendation to measure recommendation
- add similar recommendation behaviour for measures
- What should be included feature-wise for the review?
- no five star-rating (mark as future feature)
- recommendations for findings and measures
- create video on sunday
- Prepare review III
- come up with questions to be asked by nextAudit
- distribute tasks for review material

## Customer Meeting (2021-01-15)

### Agenda
- 5-star-rating as future feature (in current process no effectiveness check)
- Update on FE (measures are not yet stored in BE)
- Discuss review material
- Discuss Q&A
- Questions on how CI/Testing works with us?
- No questions regarding measure integration!
- Questions regarding model evaluation?
- Questions on information Storage and performance?
- Effort for each subproject (model, model evaluation, backend, frontend)
- Ideas for continuous learning

## Team Meeting (2021-01-15)

### Agenda
- Recommendations should include type (-> Fred includes that)
- Status of progress towards third review
- integration of recommendation script in BE (who is doing exactly what)?
- different documents
- Handbook (Outlook needs work -> Fred), then merge it
- End User Guide (Johannes, ready for review)
- Video for running system demo (Johannes, ready for review)
- Reflection (Lisa, ready for review)

- Discuss Presentation Distribution
- present ratio of effort between FE/BE
- Status of Model Eval -> German retraining? include in PPTX?

## Presenter List:
- Project Description : Fred
- System Test : Xuefeng
- CI/CD : Wenyu
- Reflection / Outlook : Lisa

## Customer Meeting (2021-01-22)

### Agenda
- Short update on project? (e.g. has Frowin seen the FE/our system demo?)
- Two functions runtimes in the emulator
- Future of the Project
- Handover preparation
- Remarks for the last review

### Notes

## Team Meeting (2021-01-22)

### Agenda
- Database loader, Connection to Recommender
- (Last) Sprint Planning
- Prepare for review review

### Notes
- Data transformation difficult
- where, how to make it efficient
- discuss later on slack

### Sprint planning
- Data ingestion
- Prepare final presentation

### Cardinality to Score Integration

- loadRecommendations sums up cardinality and score
- Norm at retrieve time before returning to fe
- May be prone to spamming/frontend bugs
- Propose a "feedback token" approach in the outlook
- Each auditItem recommendation includes a feedback token which has to be sent to the feedback endpoint and is one time use only.

## Review Review Meeting (2021-01-25)

- For final presentation use NA pitch slides/preferences as orientation

## Customer Meeting (2021-01-29)

### Agenda
- Discuss pitch slides: (see other hackMD for details)
- nextAudit Accelerator / Thinktank Programme (prototype)
- 0x00: Team Slide + Company description (B2C)
- 1. Short fancy problem pitch -> past info not used, Often slow audits, not streamlined -> inefficient audit process, "excel graveyard"
- 2. Benefit: short and easy case [time limits!] Time saving, effectiveness as summary, easier to align with compliance
- 3. Solution Slide -> how we do it at nextAudit customer example
- 4. Why us -> already developed solution, specialized in AI MoDeLz...
- 5. nextAudit Offer (fix payment for product + Abo model for service)
- Who, specifically, is the customer? Do they have money? How do they pay? (question from requirements) -> nextAudit, yes, see above? (Fred)

### Questions from RWTH

"The following five slides are expected to be shown by you:"

"Please act as a company or a startup presenting **to possible customers.**"

- What is the PROBLEM solved by the product or service you plan to provide?
- What is the quantifiable economic benefit of solving that problem?
- Who, specifically, is your customer? Do they have money? How do they pay you?
- What is your, your teams, or your company's distinct competitive advantage in providing this product or service to the customer?
- What is the PRODUCT or service you plan to provide?

### Notes
- Data from nextAudit for slides (3h/audit saving)
- Page 5/6 Doc -> Screenshots check
- Handover meeting?
- Wednesday 1-5
- Thursday 9-12 -> Mail, 10

## Team Meeting (2021-01-29)

### Agenda
- Handover meeting, time/duration -> Mail to Alex (Thursday 9-12 -> Mail, 10)
- CoP reflection (until 31.01.)
- Preparation of Review Docs Distribution (Fred, see other HMD for details)
- Poster (Lisa)
- Slides (Fred)
- System Demo (Old Video)
- Test Cases -> separate doc (check for Updates) (Thore)
- Documentation Update (all)
- Trigger Service (Thore)
- List of implemented endpoints (Thore)
- Feedback Token Idea (Thore)
- Models (Xuefeng)
- Outlook (Fred)
- Add a section about pagination (Thore)
- Final Reflection 1 PAGE (Lisa)
- Update on Progress / Task Distribution (Fred)

### Notes
- Talk on Issues 5PM or later

### Firestore Ingestion

#### Encoding
- Init Function
- DONE: Gets all documents and generates embeddings and stores these
- Needs to be run initially and when updating/changing the encoder
- HALF: use loader again and write update function to write back into database
- Trigger function
- updates embeddings for new/changed items
- could be a rest call to encoding endpoint in trigger service

#### Linking (Finding Recommendation)
- Init function
- DONE: create link table
- Update:
- Check if exists
- update score
- hash if doesn't
- DONE: Linker directly returns the final format

#### Measure Recommender
- Pulls all data again (can use existing firestore-loader)
- can use the loader again
- needs linker then recommender call

### Final

- Init (called when new audit created)
- DONE: All embeddings
- DONE: builds finding recommendation (by detetcting similar questions)
- finding id (hash) + score[auditItemId]
- DONE: write data back to firestore
- New finding
- HALF call trigger service for embedding (Trash this)
- HALF trigger measure recommender
- done until ingestion

## Customer Handover Meeting (2021-02-04)

### Notes

- https://next-audit.de/sind-audits-immer-zeitraubend-und-unwirksam/
- Put more focus on highlighting recommendation feature
- Plan steps
- 15
- 50
- unlimitied (get in contact
- Linkedin Posting?

## Customer Meeting (2021-02-05)

### Agenda
- Questions from Yesterday?
- Updated slides

## Team Meeting (2021-02-05)

### Agenda
- Finalizing Steps
- Presentation Planning
- Presentation: (Fred if noone wants to)
- Demonstration: Johannes

### Notes
- Deadlines
- Saturday 8pm: Finished; Sunday 8pm: Feedback
- Presentation
- Documentation
- Separate document for Test Cases
- Reflection
- Questions for nextAudit (?)
- Monday
- Working Demo Code
- Remember to also create a release on the master branch

### Questions for nextAudit:
- foobar