Data Science and AI Educators' Programme: Cohort Call Seven notes
===

###### tags: `cohort-call-7` `DS-AI-Educators'-Programme`

## CC7: Continuous assessment and feedback: self-guided exercises

Due to the cancellation of this cohort call, the breakout room activities have been adapted into self-guided activities for you to complete. There will be **three** activities, each taking around 10-15 minutes to complete.

### Is YOUR feedback fit for purpose?

Feedback may sometimes feel like nothing more than an expected ritual within the cycle of assessment processes that are a part of academic life. However, it can be more than this and can be seen to perform multiple functions for us as educators:

- correct student errors;
- develop understanding through educator explanations;
- generate more learning by suggesting further specific study tasks;
- promote the development of generic skills by focusing on the evidence of the use of the skills rather than the content;
- promote meta-cognition by encouraging students' reflection and awareness of learning processes involved in the assignment;
- encourage students to continue studying.

Feedback, if effective, can have multiple positive benefits:

- promote an individual's learning journey and encourage greater achievement;
- enable students to reflect and develop their strengths and weaknesses;
- foster greater levels of self-esteem and motivation;
- enable educators to realign their teaching content and methods in response to learner's needs; encourage a more effective dialogue between educators and students;
- develop core skills around assessment and peer-to-peer evaluation.

:::info
### ACTIVITY 1:

Think of a time, recently, where you gave feedback to a student. This could have been verbal or written feedback. Consider the following three questions, noting down your answers in the HackMD below.
1. What were your overall aims of the piece of feedback?
2. Were different parts of the feedback trying to do different things?
3. Do you think your aims will have been clear to the student: what will they have taken away from your feedback?

Now, consider how you might **improve** your feedback:

Identify any ways in which you could change, re-word or add to the feedback to further achieve those purposes;

4. Draft those changes, adding to your notes below.
:::

### ACTIVITY 1 REFLECTIONS:

**1. What were your overall aims of the piece of feedback?**
* Georgina Dangerfield - Aim was to provide constructive criticism in an interview setting
* Jake Marshall - Aim was to get across how the whole panel (not just myself) felt about the performance at the interview and provide constructive criticism.
* Ogerta Elezaj- The aim it was to provide feedback to the students about their coursework...
* [Cutillo-Lu-Ranathunga-Kedia] The main points discussed are: feedback is an occasion of clarification, the students could understand what they did not express clearly and in full details, not only wrongly.
* Greta Timaite - my most recent feedback was given to a colleague about a written piece of work.
* Kwong-cheong Wong - The overall aim is to tell the students how good their work is and to correct their mistakes.
* Anastasis Georgoulas - Explain performance on coursework, suggest improvements.
* Maarya Sharif: The aim of the feedback was to improve report writing as well as providing feedback on the analysis carried out.

**2. Were different parts of the feedback trying to do different things?**
* Georgina Dangerfield - I used the 'feedback sandwich' technique where a positive is stated/discussed, followed by a negative and ending on a positive note. This allows the student to see what was done well as well as areas of improvement without focusing too much on one.
* Jake Marshall - Similarly to Georgina, I use the 'feedback sandwich' approach where possible, ensuring to highlight positives and negatives in a similarly objective way.
* Ogerta Elezaj- My feedback was "constructive criticism" discussing specific problem areas that require change, making room for discussions about actions to improve the work.
* _María Navas_ - My feedback was aimed to let the student know the main poitive points of their work and how to improve or continue it.
* [Cutillo-Lu-Ranathunga-Kedia]The feedback highlights: achivements so to appreciate the students' work, mistakes to point out area of lack of knoledge, discussion about overall improvements.
* Greta Timaite - I used "feedback sandwich" technique and also suggested how I would change certain parts of the text.
* Kwong-cheong Wong - The part on marks tell the students how many marks their work get according to the marking scheme and the other part gives comments on their work for improvement.
* Anastasis Georgoulas - Parts of it were simply accompanying the grade, explaining with reference to the marking scheme. Other parts described issues with the submission or touched on interesting choices, discussing other aspects it would be good for the students to consider.
* _[name=Maarya Sharif]_ Not really. Most of it directly dealt with the grading scheme i.e. where marks were not awarded. I realise that I did not provide much constuctive feedback.

**3. Do you think your aims will have been clear to the student: what will they have taken away from your feedback?**
* Georgina Dangerfield - I was clear on the techniques they utilised that worked well and also things to consider for their next interview, be it here or elsewhere.
* Jake Marshall - The student/interviee will have gained useful insight into their approach when it comes to civil service interviews. They will be able to cross reference future interview prep with the feedback provided here.
* Ogerta Elezaj- The feedback provided was meaningful and specific enough for student to understand how to improve and gain a better mark.
* _María Navas_ - The student may have taken away that I value the work done and the effort and think that the work is valuable to be continued.
* [Cutillo-Lu-Ranathunga-Kedia] The student may have taken away the assessment criteria, sense of achivement, areas to improve and clarify.
* Greta Timaite - based on the response, I think the feedback was well-taken.
* Kwong-cheong Wong - The students usually pay more attention to the part on marks. They want more to know why marks were deducted. Usually I will them a detailed marking scheme so that they can check my marking.
* Anastasis Georgoulas - Like Kwong-cheong says, I fear they may have focused on the marks and ignored the overall comments and suggestions, or the reasoning for specific marks.
* _[name=Maarya Sharif]_ In agreement with other i suspect that given the limited feedback provided students would largely focus on the marks awarded rather than the feedback. In addition the fact that the report was only allocated 5% of the overall grade i doubt students paid much attention to the feedback.

**4. Drafted changes to my feedback:**
* Georgina Dangerfield - Do a 'feedback sandwich' for each interview 'section' that is clearly defined. This way the feedback is more in depth and allows the student to plan for future interviews with feedback alongside them.
* Jake Marshall - Make it more personal and less 'robotic', utilising the candidates name but keeping the panel anonymous, rather than anonymising both in my response.
* Ogerta Elezaj- When suggesting what can be improved, to identify no many areas of the work, to avoid confussion.
* _María Navas_ - Maybe divide the feedback into sections.
* [Cutillo-Lu-Ranathunga-Kedia] Have clear sections (positive,negative, to improve) as mentioned before and adding minor an dmajor in each section.
* Greta Timaite - I like the idea of dividing feedback into sections. This is something I will try to do next time.
* Anastasis Georgoulas - I would make sure there is an overall statement on the submission rather than only individual points (whether negative or positive).
* _[name=Maarya Sharif]_ I would included specific comments on where exactly to improve on rather than focussing on the grading scheme alone.

---

:::success
### Feedback: efficiency for us, learning payoff for students.

### ACTIVITY 2:

**Consider the list of feedback methods, below.**

- live feedback in class
- individual written feedback
- ad hoc verbal feedback e.g. in a seminar
- written feedback, unreadable or too short
- peer group discussion
- exam marks, no comment
- peer assessment, assuming fairly
- generic written report for all students
- recorded audio feedback for individuals
- self feedback/assessment
- talking to small groups about common problems
- face-to-face feedback one-to-one
- recorded generic audio feedback to a whole group
- criteria sheets - rubrics
- email feedback
- track changes
- hand-written feedback on end-of-semester major assignments

**Pick 6-10 of the methods and decide where they sit on the following matrix. Make sure to include methods that you currently use and please add any additional methods to the list. Click [here](https://miro.com/welcomeonboard/U0cyTE0wbG4xcExRZlJTQXFDaUFtY0JhMDB3WkVmZTV3R1I2U3dUT0ljTjdpUG1nVENqY1NsdkpzNGEzT0ZzVHwzNDU4NzY0NTI4ODA3NzExMzM2?share_link_id=1479430232) to add your methods to the shared matrix. These can be anonymous and you may have differing opinions so please do not worry about what has already been entered into the matrix.**

### Questions for reflection:

1. Where do your current feedback methods sit? What does this suggest about continuing to set these methods or changing to other methods?
2. Are you able to find methods which could work in your practice and which are both highly efficient for educators and highly beneficial for students?
3. Are there ways in which some of the methods can be adapted slightly to bring them closer to the top-left quadrant?
:::

### ACTIVITY 2 REFLECTIONS:

**1. Where do your current feedback methods sit? What does this suggest about continuing to set these methods or changing to other methods?**
* [Cutillo-Lu-Ranathunga-Kedia] A mixture of written individual in chats or hw, interacive group discussion, smaller groups discussion
* _[name=Anastasis Georgoulas]_ I tend towards methods that involve more effort. This does not scale well for larger classes and I would like to move to more efficient ones.
* _[name=Maarya Sharif]_ My feedback methods are spread across the four quadrants. This process is helping me to evaluate whether i should encourage for example, face-to-face questions on a one-to-one basis
* _[name=enter name here]_ ...

**2. Are you able to find methods which could work in your practice and which are both highly efficient for educators and highly beneficial for students?**
* [Cutillo-Lu-Ranathunga-Kedia] small groups of students interactive discussions on common and individual probelms are a good tradeoff and more effective.
* _[name=Anastasis Georgoulas]_ I am tempted to try recording audio feedback, because I already write comments, and this way might be faster, feel more personal, and be harder to ignore for the students.
* _[name=Maarya Sharif]_ Technically methods may fall into this quadrant but i guess i havent researched whether the methods are indeed benefical for students. For example at my institution we use an online forum (Piazza) for students to ask questions (anonymously) but i have moticed that not all students in the course sign up to this service and therefore miss out on highly beneficial feedback.
* _[name=enter name here]_ ...

**3. Are there ways in which some of the methods can be adapted slightly to bring them closer to the top-left quadrant?**
* [Cutillo-Lu-Ranathunga-Kedia] subgrouping and adding more interactivity
* _[name=Anastasis Georgoulas]_ A generic report for all students could have a higher educational payoff if students are encouraged to reflect on which of the comments apply to their own work, and therefore self-assess and identify possible improvements themselves. Peer assessment is more useful when the criteria are made clear from the start (or if the students have to first come up with the criteria themselves, which encourages them to think of what good work looks like).
* _[name=Maarya Sharif]_ A face-to-face/one-to-one question can be answered on a course wide basis hence moving it into the top-left quadrant.
* _[name=enter name here]_ ...

---

:::danger

### Feedback to students would work much better for me if only I...

Below are responses from educators on things which would make feedback better from their own perspectives. This activity is designed to help you reflect on challenges that you yourself face with respect to feedback, the source of the challenge, the factors which are perhaps within your control and the areas where you would like to explore new strategies.

### ACTIVITY 3:

1. Read through the statements and, in the first column 'true for me', tick the statements that you feel apply to you. Please do not put your responses directly into the table, instead, use the section further below to jot down your responses.
2. Consider the responses you ticked as 'true for you' and, in the 'source of challenge' column, jot down the source of the challenge(s) for you. You might find the following categories helpful but, if not, use some of your own: 'the institutional system', 'student attitude', 'my attitude', 'time', 'my skills or abilities' etc.
3. Pick three or four of these challenges which feel the most pressing and explore potential strategies to counteract these. You may want to make use of the Slack channel here to get advice or startegies from your fellow educators.

| Staff responses... </br> </br> _Feedback to students would work much better for me if only I..._ | True for me | Source of challenge |
| ------------------------ | ---- | --------- |
| Thought they'd read and digest it. | | | Complete
| Could give them back their essays to keep. | | | Complete
| Thought it would make a difference. | | | Complete
| Was able to do it more quickly. | | | Complete
| Could be in the right frame of mind when meeting them face to face. | | | Complete
| Could be sure they would understand what I'm trying to tell them. | | | Complete
| Could get them to turn up to receive feedback. | | | Complete
| Had some expectations that they would use it to improve their next grade/mark. | | | Complete
| Cared less about the judgements I make and how to articulate these in writing. | | | Complete
| Could get more of them to attend sessions and engage in the learning activities. | | | Complete
| Knew the students better. | | | Complete
| Could discuss this with my students. | | | Complete
| Knew them personally, and their profiles. | | | Complete
| Knew how to phrase it in terms that students understand. | | | Complete
| Saw evidence of improvement as a result of that feedback. | | | Complete
| Knew what to say to them. | | | Complete
| Could be clearer about why they got the grade/mark that they did. | | | Complete
| Made more time for giving effective feedback. | | | Complete
| Knew that they would read the comments and not just look at the grade/mark. | | | Complete
| Could identify better what the student needs to do to improve the next piece of work and feedforward. | | | Complete
| Could do this as a dialogue. | | | Complete
| Didn't take so long over it. | | | Complete
| Had the chance to discuss it and explain it. | | | Complete
:::

### ACTIVITY 3 REFLECTIONS:

**1. The statements that resonate with me are...**
* [Cutillo-Lu-Ranathunga-Kedia]
* Was able to do it more quickly.
* Could give them back their essays to keep.
* Knew them personally, and their profiles.
* Made more time for giving effective feedback.
* Could identify better what the student needs to do to improve the next piece of work and feedforward.
* _[name=Anastasis Georgoulas]_
* Thought they'd read and digest it
* Was able to do it more quickly
* Knew the students better
* Saw evidence of improvement as a result of that feedback
* Knew that they would read the comments and not just look at the grade/mark
* _[name=Maarya Sharif]_
* Thought they’d read and digest it.
* Was able to do it more quickly.
* Could get them to turn up to receive feedback.
* Could get more of them to attend sessions and engage in the learning activities.
* Knew the students better.
* Knew what to say to them.
* _[name=enter name here]_ ...

**2. For me, the challenge(s) associated with these statements is/are...**
* [Cutillo-Lu-Ranathunga-Kedia]_ too many students and not much time
* _[name=Anastasis Georgoulas]_ institutional system (class size, structure that doesn't allow frequent feedback), time
* _[name=Maarya Sharif]_
* Thought they’d read and digest it: student attitude
* Was able to do it more quickly: time
* Could get them to turn up to receive feedback: student attitude
* Could get more of them to attend sessions and engage in the learning activities: student attitude (low numbers attending) and the institutional system (not enough scheduled contact time with students)
* Knew the students better: the institutional system, too many students and not enough time
* Knew what to say to them: my skills or abilities
* _[name=enter name here]_ ...
* _[name=enter name here]_ ...

**3. Potential strategies for the challenges I encounter are...**
* [Cutillo-Lu-Ranathunga-Kedia] Having multiple staff members teaching as a team and supporting each others work
* _[name=Anastasis Georgoulas]_ Better organisation/prioritisation on my side to get feedback out earlier (+ aim for more efficient methods). More opportunities for formative feedback through the term, including from peers. Separate feedback from grades when possible.
* _[name=Maarya Sharif]_ With experience some of these things will get easier. Nonetheless i belive developing a rapport with students and encouraging a growth mindset in them will help in them engaging with the learning process and feedback process. In addition learning more deeply about the subject matter should boost my confidence as well.
* _[name=enter name here]_ ...

### Useful Links/references:

- **Systematic review of automated assessment**
- Paiva, José Carlos, José Paulo Leal, and Álvaro Figueira. "Automated Assessment in Computer Science Education: A State-of-the-Art Review." ACM Transactions on Computing Education (TOCE) 22.3 (2022): 1-40. [[link]](https://dl.acm.org/doi/abs/10.1145/3513140)
- **Automated assessment in Jupyer Notebooks**
- There is growing literature exploring approaches to automated grading in Jupyter notebooks, which are commonly used in data science teaching. Here are a couple of interesting starter links:
- Manzoor, H., Naik, A., Shaffer, C. A., North, C., & Edwards, S. H. (2020, February). Auto-grading jupyter notebooks. In Proceedings of the *51st ACM Technical Symposium on Computer Science Education* (pp. 1139-1144). [[link]](https://dl.acm.org/doi/abs/10.1145/3328778.3366947)