---
# System prepended metadata

title: HENM NextAudit Meeting Notes

---

# HENM NextAudit Meeting Notes


## Team Meeting Notes (2020-10-30)

- Recommendation for findings
- Recommendation for measure
- Non-Conformity recommendation may be immediately
- measures are usually added later (no real time requirement)

### Target Next Meeting
- Technical Details
    - Code Hosting
    - Issue Management
- Goal: Written Definition of Goals Project should achieve
- Send our questions to nextAudit 4.11/5.11?

## Team Meeting II (2020-11-06)

### Project Goal Proposal

The development and further implementation into the existing BE/FE system of: 
+ a recommender engine suggesting for any new question per audit type an ordered list of potential findings based on previous findings for context-similar questions. 
+ a recommender engine suggesting for any new finding per (audit type, question) an ordered list of potential measures based on previous measures for context-similar (question, finding) pairs.
+ The possibility to update the training dataset and retrain by user feedback on the recommended suggestion (by selection)


## Customer Meeting Notes (2020-11-06)

### Deliverables
- Implementation (commented code) + sufficient documentation to follow
- Introduction to NLP is of interest
- Shipable Prototype

### Technical Details
- Code Hosting
    - Gitlab Projects will be set up
- Issue Management
    - Gitlab Issues
- Deployment
    - Gitlab CI/CD
    - GCP Based
        - Probably Cloud Functions but not fixed
        - App Engine in Evaluation
    - Model Training on RWTH Ressources, just deployment to GCP
- Target Plattform
    - TypeScript + express based REST-API
- Data Sources
    - Based on firestore
    - Training data probably provided as json export
- Code Quality
    - https://gitlab.com/nextaudit/audit-cloud-ai
    - Has eslint config
    - Look for test implementation

### Review Requirements
- Can use Arbeitsplan
- Can get slides for market analysis
- Personas not available
    - Think of something else
    - Possibility for interview to analyse target group

### Remote Project
- No measures yet, just quetions and findings
- not iso19011
- Images not important yet
    - but keep possibility to extend in the future in mind (OCR/speech-to-text) probable project outlook
- Audit Tempates
    - Questions will become editable (e.g. add free questions)
    - Every question refers to a chapter in the standard --> use this information
        - Questions may focus on individual subsystems impacted by the chapter
- Measure Types
    - Can be ignored
    - Probably still include them (DISCUSS)

### Data Structure

* Almost everything included in measureitem
* Should be available someday next week

### Project Goal

The development and further implementation into the existing BE/FE system of: 
+ a recommender engine suggesting for any new question per audit type an ordered list of potential findings based on previous findings for context-similar questions. 
+ a recommender engine suggesting for any new finding per (audit type, question) an ordered list of potential measures based on previous measures for context-similar (question, finding) pairs.
+ The possibility to update the training dataset and retrain by user feedback on the recommended suggestion
    + Feedback: Star Rating of Effectiveness (to be introduced)

--> Agreed

## Followup Team Meeting Notes (2020-11-06)

### ToDos
- [x] Log-In to test instance
- [x] (monday) Fill out experience template
- [x] (monday) Discuss Business Case Slides
- [x] (wednesday) Get familiar with (frontend) Code
- [x] (wednesday) Prepare User Stories
- [x] (wednesday) Review Review Material
- [x] Transfer Tasks to issues

## Team-Meeting (2020-11-09)

### Agenda
- Define Scrum Roles
- Create User Stories

### Notes
- Scrum Roles: Master: Lisa, Rest: Members

## Team-Meeting (2020-11-11)

### Agenda
- Select User Stories from suggested Input
- Clarify which testing we should run (maybe to implement e2e is not worth it within our timeframe)

### Notes
- Scrum Ceremony Schedule
    - Daily
        - On weekly basis
        - Friday evening
        - Communicate via Slack
        - Dynamic additional meetings
    - Planning
        - Before Review
    - Customer Review
        - Every 2nd Friday in recurring meeting
    - Retro
        - In Customer Review Weeks

## Customer-Meeting (2020-11-13)

### Agenda
- Agree on User Stories
- Market Analysis

### Notes
- Story discussion: 
    - 1. story: ok, findings, measure: just list of highest rated. 
        - create wireframe or sketch to agree on
    - 2. story: 
        - idea: good, all questions have unique id + chapter => simple context/meta data matching.
    - 3. story: good, measures not in wireframe atm. 
    - 4. story: good. 
    - 5. story: good. Baseline: selection/frequency, extended: rating
    - 6. story sounds good, but no priority, add later.

- REST APIs
    - see nextAudit slides

- Slides Feedback:
    - seems good so far, share with Alex.

## Team-Meeting (2020-11-13)

### Agenda
- (Discuss Components of FE?) (ok)
- Refine User Stories (ok)
- Tasks
- Assign Tasks to Team Members
- Discuss Deliverables for 1st Review (X)

### Notes
- Story 1
    - Create sketches for recommendation UI flow (mohammad)
    - Agree on sketches with customer (team)
    - Implement Vue component with mock data (open pool)
- Story 2:
    - Get better understanding of communication flow.
    - analyse question and finding data for non-nlp recommendation system
        - baseline matching design (target data structure)
        - data preprocessing pipeline
    - Wire frontend component with suggestions from recommendation service

- Git Flow Model
    - Include in final review


## Team Meeting (2020-11-18)

### Agenda
- Recover from emotional shock (ok)
- Look into Backend Code as a next task (ok)
- Prepare questions to reflect how 1st review could get so out of hand (ok)
- Prepare UI sketch for Friday (Mohammad, ok)
- Discuss Communication Flow (ok)
- Visualize Communication Flow (ok)
- Maybe: prepare to discuss data input - data output for models with customer, only 2 options (Fred) ? (ok)
- look for code samples we could provide (if necessary)
    - Vue.js very very simple integration
    - Data Preprocess pipeline for json data (needed anyways)
        - https://jupyter.rwth-aachen.de/hub/login or colab or ... [how does jupyter hub handles data uploads, colab resets all 12h and forces a new dataset upload I think...] (ok)

### Notes
+ Not real code received yet -> discuss on friday.
+ Email Frowin for Friday. (He is on the invitation list for the recurring meeting)
+ Mail to Alex "Supervisor"
+ "git" jupyter notebooks
+ https://gitlab.com/henm-project/data-processing


## Customer Meeting (2020-11-20)

### Agenda
- Ask Frowin whether he is on the mailing list (yes)
- Discuss component flow (yes)
- Discuss Sketches (yes)
- Discuss Model structure (top level view) (yes)

### Design
- Proposal 1: 
    - Suggestion: When typing something suggestions should be limited to similarity matched
    - Suggestion: Button should show the number of recommendations
- Proposal 2: Start Typing, recommendations start to show
    - Could do, maybe later
    - could be more interesting for measures

### Data Analysis
- Probably try to detect names an obfuscate them
- Long run idea: Have placeholders and introduce input fields in the recommendation field

### Component Flow
- Firebase document size is limited
- Consider storage efficient recommendation storage layout
- Do not recompute measure recommendations

## Team Meeting (2020-11-20)

- Preparation for Monday?
    - Prepare frontend and preproc-pipeline
- Question for Monday: Presentation Content Distribution

## Customer Meeting (2020-11-27)

### Agenda
+ Question: Question ID's in JSON always the same across audits? (yes)
+ Discuss Vue Component with customer (yes)
+ Discuss anonymization on real client data (yes)
+ Discuss BE issues (yes)
+ Short Note on Model Structure (Fred) (yes)
+ Note if you are interested in the offer on Entrepreneurship (yes)

### Notes
+ Only select one finding per question, what type of finding (major, minor),
    - remove right selection window
    - Update sketches
    - Filter by deviation type (major/minor)
+ Question IDs globally same
    + Use for recommendation association
+ Other Personal Data Found (focus on personal information first)
    + emails
    + telephone numbers
    + locations
    + company names
    + register numbers
    + domains/urls
    + file names and paths
+ Custom List for tokens to anonymize
    + intgrate into general stopword removal process
    + store somewhere in json, readin in process

## Team Meeting (2020-11-27)
Out of scope note: remove title from names as well

### Agenda
+ Collect prospects for fireside chat (yes)
+ Sprint Review/Retro? (discussed, monday)
+ Discuss Baseline model (Fred) (yes)
+ Collect new isses for next sprint (some already listed)


### Notes
+ Entrepreneurship -> Slack
+ Sprint Review
    + Tasks mainly in review
    + All test backend installation/setup
+ Sprint Planning
    + Extend same stories
    + More anonymization
    + Redesign UI according to feedback
    + Implement backend endpoint
    + Wire backend endpoint to frontend
+ Sprint Retro
    + Everyone fills out and votes in https://easyretro.io/publicboard/GG6C68wXsLS3VhITgQF2EiKQKlI3/6f7b5aa0-adff-454b-9bfd-1a9d1dc47578
    + Discussion on Monday 17:15

#### Sprint 2 Tasks

- Question - Question Matching
- Question - Finding Matching

## Team Meeting (2020-11-30)

### Agenda
- Discuss retro notes

### Action Items Sprint Retro
- Create Slackbot reminder for Wednesday in #midweek-updates
- Quickly feedback missing information or non-working code to nextAudit
- Account 45 minutes for retro next time
- Kepp notes in Hackmd
- Encourage everyone to update the agenda before meetings
- Try to create task pools and primary assignees in sprint planning sessions.
- Agree on agenda mutually (and stick to it) for all meetings to avoid context mixing.
- Keep pair programming in mind when creating tasks
- When a feature is ready for review create a mr and notify on slack
- Share Current Status via Screenshare
- Notify on slack when scheduling issues occur as soon as possible.

## Customer Meeting (2020-12-04)

### Agenda
- Presentation of new FE component (Johannes)
- Presentation of Anonymizer (Fred)
- Presentation of BERT Idea (Xuefeng)
- Presentation of Working Example without classification (Fred)
- Hint for upcoming review

### Notes
- Measure in FE now working
- "Nicht Zutreffend" and "I.O." not to recommend!
    - BE has to provide major/minor deviation type
- Anonymizer needs predefined token list 
- Mail for anonymizer instructions (Fred)

## Team Meeting (2020-12-04)

### Agenda
- Prepare Review II
    - come up with questions
- Friday Meeting Schedule
- Merge Requests (Fred)
- Titles in anonymizer
- FE/BE Wiring

### Notes

- Review
    - Wait and see
    - Hope for requirements to be sent out on Monday
- Open Issues
    - FE: Next PR by Johannes
    - BE: Install issues still unresolved
        - yarn fix now in master
        - use json import tool if firestore import fails
    - BE: First simple endpoint implemented
    - BE/FE: Store action can be implemented to wire both
- Swapping might make sense start time will be 2pm either way
- Titles
    - Keep for now, measure accuracy
    - Consider if possible to replace in a postprocessing step

## Customer Meeting (2020-12-11)

### Agenda
- very short FE update (filter) (Johannes)
- Prepare Review II
    - come up with questions:
        * How are the endpoints supposed to communicate with FE?
        * How are the recommendations going to be stored and updated
        * What approaches are you currently pursueing for the Recommender?
        * Are personal information included in the training data?
        * What were your largest challenges concerning teamwork?
        * reusable components
        * Agile appearing tasks
- Export firestore fixture

### Notes
- Last meeting 2020: next week
- First meeting 2021: 8.1. 

## Team Meeting (2020-12-11)

### Agenda
- Discuss all points on TODO for 2nd review
- distribute tasks
- decide who presents
    - Project Backlog (Thore)
    - DoD (Thore)
    - Demo (Johannes)
    - Team Progress (Johannes)
    - Conclusion/Outlook (Mohammad)
- Collect issues for the third sprint

## Customer Meeting (2020-12-18)

### Agenda
- Short Update on status of semantic matcher (Fred)
- Short Feedback on model evaluation idea (Fred)
- Doublecheck endpoint changes with nextAudit (Thore?)

### Notes
- Filter findings also per recommendation in data model
- 1 doc per Recommendation is ok
- Calculate with
	- 100 questions per audit
	- 100 audits
	- 150 items per audit
    - documents can't be larger than 1MiB
- Cost Calculation
- Meeting on Data stuff 21. 14:30 -> Mail
- Vacation
    - Frowin from 21st to 4th.
    - Next Meeting on the 8th
    - Contact Alex if necessary
    - 

## Team Meeting (2020-12-18)
### Agenda
- Data Proc repo again huge (350MB) :(
    - Will always stay in history -> true, ok
- Short question on storing recommendations (Fred)
- Anything to prepare for monday?
    - Most likely some kind of code review (each one a sample of project related code?)
- Architecture documentation document
- NLP method evaluation discussion (Fred)
- Anyone else experiencing lags in Hackmd (yes)?
- Plan for the rest of this year?
    - Any standups/synchronization
    - (OT) Anyone attending RC3?

### Notes
- is_no_deviation in parser (Fred)
- Ask Alex on Monday whether we have to create a project architecture documentation for the final product
- Split document up
- Team Coordination next 2 weeks
    - Wednesday updates will continue
    - Meeting 28th 2PM


## Review Review Meeting (2020-12-21)

### Notes
- Include a questions slide in the next slideset
- The third review is going to be very similar to the second
- A bit more focus on technical stuff
    - testing
    - ci/cd
- Final presentation is going to be a 5 minute business oriented pitch.

## Team Meeting (2020-12-28)

### Agenda

- Last Sprint Retro
    - How did our action items work out?
- Architecture Documentation for NA?
    - UMLs
    - Cost Consideration
    - Outlook
        - Continuous learning
        - Recommendation Updates
        - Deploying and updating models
- Prepare for third review
    - Elaborate on CI pipelines (linting, testing, extend testing where possible)
- Code Quality (Xuefeng) (Fred)
- Short update on Recommender (Fred)
    - Split up baseline model for ingestion
- Sprint Planning
    - Already plan data ingestion (recommender -> firestore)



### Notes

- Project Handbook
    - https://mettmd.selfnet.de/pNPHLiHiRWqVH5gYHO4Nlg?edit
    - Centered to be used by nextaudit
    - might be handed in at the end of the project
    - focussed to keep our thoughts and ideas alive
- Third review
    - Include gitlab-ci pipeline (probably as question from NA)
- Next Meeting:
    - Firday if necessary
        - if necessary is decided on Thursday 8pm (on slack)
    - otherwise 8th

### Next Sprint

- Unit Tests for frontend
- Build storage for association of finding recommendations to questions
- ingestion feature to pass the final storage model to firebase
- finding-feedback endpoint
    - add new issue specific to feedback
- measures five star rating + feedback (fe/be)
    - rating (five star)
    - feedback (has been selected)
- present recommendations in the frontend which have been fetched from the backend


## Customer Meeting (2021-01-08)

### Agenda
- Short status update (all)
- Discuss Storage Model (again) (Fred, Thore)
- Again Collection of Questions for Review [e.g. Testing] (all)

### Notes
- No unique ids for questions, findings
- GCloud capacity free
- Questions to prepare for review


## Team Meeting (2021-01-08)

### Agenda
- Discuss Storage Model (again) (Fred, Thore)
- Cost Evaluation for review?
- Discuss Use of Master Branches
- Exchange firestore data of BE (including recommendations)
- Model Evaluation (Lisa)
- Sprint Planning

### Notes

- Discuss issues to discuss with frowin
    - unique auditItemId + findingId (hashing ourselfs needed)
    - deviation types different across customers (please not)
    - final agreement on storage model (query-cost update)
- New finding added by linker (Fred)
    - Probably add to outlook (with continuous learning model)
- Merge dev into master after each sprint
- json export for firestore
    - idiots approach: set in fetch endpoint

### Sprint Planning


## Team Meeting (2021-01-13)

For Project Documents see https://hackmd.io/aGV2bU42QfeIwqNowGNQbA?edit


### Agenda
- Progress on sprint (e.g. realistic to get generated finding recommendations to be provided by BE until review?)
    - Build recommendation ingester into firebase emulator
        - possibly skip hashing for now
    - copy finding recommendation to measure recommendation
    - add similar recommendation behaviour for measures
- What should be included feature-wise for the review?
    - no five star-rating (mark as future feature)
    - recommendations for findings and measures
    - create video on sunday
- Prepare review III
    - come up with questions to be asked by nextAudit
    - distribute tasks for review material

## Customer Meeting (2021-01-15)

### Agenda
- 5-star-rating as future feature (in current process no effectiveness check)
- Update on FE (measures are not yet stored in BE)
- Discuss review material
- Discuss Q&A
	- Questions on how CI/Testing works with us?
	- No questions regarding measure integration!
	- Questions regarding model evaluation?
	- Questions on information Storage and performance?
	- Effort for each subproject (model, model evaluation, backend, frontend)
	- Ideas for continuous learning

## Team Meeting (2021-01-15)

### Agenda
- Recommendations should include type (-> Fred includes that)
- Status of progress towards third review
	- integration of recommendation script in BE (who is doing exactly what)?
    - different documents
        - Handbook (Outlook needs work -> Fred), then merge it
        - End User Guide (Johannes, ready for review)
        - Video for running system demo (Johannes, ready for review)
        - Reflection (Lisa, ready for review)

- Discuss Presentation Distribution
	- present ratio of effort between FE/BE
- Status of Model Eval -> German retraining? include in PPTX?

## Presenter List:
- Project Description : Fred
- System Test : Xuefeng
- CI/CD : Wenyu
- Reflection / Outlook : Lisa

## Customer Meeting (2021-01-22)

### Agenda
- Short update on project? (e.g. has Frowin seen the FE/our system demo?)
- Two functions runtimes in the emulator
- Future of the Project
    - Handover preparation
- Remarks for the last review

### Notes


## Team Meeting (2021-01-22)

### Agenda
- Database loader, Connection to Recommender
- (Last) Sprint Planning
- Prepare for review review

### Notes
- Data transformation difficult
    - where, how to make it efficient
    - discuss later on slack

### Sprint planning
- Data ingestion
- Prepare final presentation


### Cardinality to Score Integration

- loadRecommendations sums up cardinality and score
- Norm at retrieve time before returning to fe
- May be prone to spamming/frontend bugs
    - Propose a "feedback token" approach in the outlook
        - Each auditItem recommendation includes a feedback token which has to be sent to the feedback endpoint and is one time use only.

## Review Review Meeting (2021-01-25)

- For final presentation use NA pitch slides/preferences as orientation

## Customer Meeting (2021-01-29)

### Agenda
- Discuss pitch slides: (see other hackMD for details)
- nextAudit Accelerator / Thinktank Programme (prototype)
	- 0x00: Team Slide + Company description (B2C)
	- 1. Short fancy problem pitch -> past info not used, Often slow audits, not streamlined -> inefficient audit process, "excel graveyard"
	- 2. Benefit: short and easy case [time limits!] Time saving, effectiveness as summary, easier to align with compliance
	- 3. Solution Slide -> how we do it at nextAudit customer example
	- 4. Why us -> already developed solution, specialized in AI MoDeLz...
	- 5. nextAudit Offer (fix payment for product + Abo model for service)
- Who, specifically, is the customer? Do they have money? How do they pay? (question from requirements) -> nextAudit, yes, see above? (Fred)


### Questions from RWTH

"The following five slides are expected to be shown by you:"

"Please act as a company or a startup presenting **to possible customers.**"

- What is the PROBLEM solved by the product or service you plan to provide?
- What is the quantifiable economic benefit of solving that problem?
- Who, specifically, is your customer? Do they have money? How do they pay you?
- What is your, your teams, or your company's distinct competitive advantage in providing this product or service to the customer?
- What is the PRODUCT or service you plan to provide?

### Notes
- Data from nextAudit for slides (3h/audit saving)
- Page 5/6 Doc -> Screenshots check
- Handover meeting?
    - Wednesday 1-5
    - Thursday 9-12 -> Mail, 10

## Team Meeting (2021-01-29)

### Agenda
- Handover meeting, time/duration -> Mail to Alex (Thursday 9-12 -> Mail, 10)
- CoP reflection (until 31.01.)
- Preparation of Review Docs Distribution (Fred, see other HMD for details)
	- Poster (Lisa)
	- Slides (Fred)
	- System Demo (Old Video)
	- Test Cases -> separate doc (check for Updates) (Thore)
	- Documentation Update (all)
	    - Trigger Service (Thore)
	    - List of implemented endpoints (Thore)
	    - Feedback Token Idea (Thore)
	    - Models (Xuefeng)
	    - Outlook (Fred)
	        - Add a section about pagination (Thore)
	- Final Reflection 1 PAGE (Lisa)
- Update on Progress / Task Distribution (Fred)

### Notes
- Talk on Issues 5PM or later

### Firestore Ingestion

#### Encoding
- Init Function
    - DONE: Gets all documents and generates embeddings and stores these
    - Needs to be run initially and when updating/changing the encoder
    - HALF: use loader again and write update function to write back into database
- Trigger function
    - updates embeddings for new/changed items
    - could be a rest call to encoding endpoint in trigger service


#### Linking (Finding Recommendation)
- Init function
    - DONE: create link table
- Update:
    - Check if exists
        - update score
        - hash if doesn't
- DONE: Linker directly returns the final format



#### Measure Recommender
- Pulls all data again (can use existing firestore-loader)
- can use the loader again
- needs linker then recommender call


### Final

- Init (called when new audit created)
    - DONE: All embeddings
    - DONE: builds finding recommendation (by detetcting similar questions)
        - finding id (hash) + score[auditItemId]
    - DONE: write data back to firestore
- New finding
    - HALF call trigger service for embedding (Trash this)
    - HALF trigger measure recommender
        - done until ingestion

## Customer Handover Meeting (2021-02-04)

### Notes

- https://next-audit.de/sind-audits-immer-zeitraubend-und-unwirksam/
- Put more focus on highlighting recommendation feature
- Plan steps
    - 15
    - 50
    - unlimitied (get in contact
- Linkedin Posting?

## Customer Meeting (2021-02-05)

### Agenda
- Questions from Yesterday?
- Updated slides


## Team Meeting (2021-02-05)

### Agenda
- Finalizing Steps
- Presentation Planning
    - Presentation: (Fred if noone wants to)
    - Demonstration: Johannes

### Notes
- Deadlines
    - Saturday 8pm: Finished; Sunday 8pm: Feedback
        - Presentation
        - Documentation
        - Separate document for Test Cases
        - Reflection
        - Questions for nextAudit (?)
    - Monday
        - Working Demo Code
        - Remember to also create a release on the master branch

### Questions for nextAudit:
- foobar