# [Internal] Invited review process

Last updated: 2023-06-01
URL for this page: https://hackmd.io/@investinopen/invited-review
URL for this site: https://hackmd.io/@investinopen/how-we-work

## What is this?
An **internal** process to faciltiate the invited review process for research outputs, including [preliminary investigations](https://hackmd.io/@investinopen/research-products#Preliminary-investigation), [analytic memos](https://hackmd.io/@investinopen/research-products#Analytic-memo), and other works.

Not to be confused with the [Reviewer guide](https://hackmd.io/@investinopen/reviewer-guide), which is a complementary piece of documentation shared with reviewers with instructions and guidelines for reviewers participating in this reviewing process.

## Background
- The invited review process aims to improve the accuracy and reliability of our research, before we publish/share that research with a wider audience.
- Typically, a research report draft is made available for invited reviewers' comments for 2 weeks; this time could be shortened when quick turnaround is required, although we try to have at least two weekends within the review period.

## Timeline

### 5-6 weeks ahead of desired publication date
1. **Determine the timeline for the invited review and publication**
- Determine this with the Director of Research
- Typically, a research report draft is made available for invited reviewers' comments for 2 weeks; this time could be shortened when quick turnaround is required, although we try to have at least two weekends within the review period.
- After review is closed, allow at least 1 week (preferably 2-3 weeks) for report author to respond to reviewers' comments
- Make sure the review's open, close, and report publication dates are added to the Communications Calendar *(only accessible by IOI staff)*

2. **Identify invited reviewers**
- Aim to identify 6 reviewers to invite
- Invited reviewers are usually:
- Researchers in the research domain of the report
- Practitioners with lived experience in field that the research is studying
- In the case of reporting out from user research, the interviewees/users involved
- The above include those who are already in our networks, but depending on the subject matter of the work to be reviewed, we also sometimes reach out to new, "cold" contacts. In those cases, we try to schedule an introductory call prior to inviting.
- Solicit team contributions to the list of invited reviewers during a Research weekly meeting; and/or on Asana
- Deprioritize reviewers who have just been engaged to review another piece of research from us

#### Recording reviewers on CiviCRM
Once you have identified reviewers to invite, you should record this information in CiviCRM.

To do this:
1. Log onto CiviCRM. On the top menu, hover over `Contacts` -> click `New Activity`
1. Fill in the form as follows; for the fields not specified below, leave them as the default/blank:
- Activity Type: Invited Review
- With Contact: search for and select the identified reviewers by typing in their names; if you can't find the name in the dropdown, it means they does not exist in our CiviCRM yet. You'd need to first add a new individual (you can find this option on the dropdown).
- Activity Separation: Choose `Create separate activities for each contact`
- Subject: put in the name of the research report/product being reviewed
- Date: choose the date when you invite the reviewers; you'd need to input a time but it doesn't have to be accurate
- Activity status: Invited
- Details: if you can, put a link to the draft open for comments.
1. Click `Save`.
1. When the reviewers reviewed/declined the invitation, change their activity status to "commented" or "declined" respectively.
- In CiviCRM, search for the reviewer's profile.
- On the panel on the left, select `Activities`.
- Find the appropriate review activity and select "Edit" towards the right hand side of the screen
- Update the activity status to "Commented" or "Declined"
- Click `Save`.

> :bulb: Tip: you can also use Search Kit to batch update reviewer status.
> 1. Go to [this template](https://investinopen.civicrm.org/civicrm/admin/search#/edit/36)
> 2. Change the subject keyword to an appropriate one for the invited review
> 
> 3. Click `Invited Reviewers - Table 1` on the left hand panel, then `View display` on the top right corner. Statuses can be edited in-line in the table.
> 

### At the start of an invited review process
1. Ensure that the draft is ready to be reviewed - that most internal comments have been addressed and that the author is OK for the report to be sent out for review
2. Prepare a copy of the draft for reviewers
- make a copy of the report (usually a Google Document); put this copy in the `Team Drive > Communications > Public Content > DraftsOpenForComments`; make sure that everyone with the link has commenting abilities
- In the file name, put `[Draft open for comments]` before the rest of the name
- In the header, put the following text in blue:
> This is a draft currently open for invited reviews, up to and including [review closing date].
> Please be reminded that we will publish a version of the report with all reviewers comments, on our public Zenodo community collection. Your name may be recorded in this version if you choose to comment while signed in your Google account. More information can be found on our [Reviewer Guide](https://hackmd.io/@investinopen/reviewer-guide).

> 

- In the acknowledgement section (usually towards the end of the report), put the following text:
> We would also like to thank [Name and Name] for providing feedback to the complete draft of this report.
- Highlight this text in acknowledgement and put the following comment:
> Note to reviewers: we'd love to acknowledge your contributions to this report. Please let us know by replying to this comment or by email if you'd rather NOT be publicly acknowledged for your effort. Thank you!
3. Record a short video (3-5 minutes) introducing the work to be reviewed. The video might include the rationale for the research, an overview of the research output, and it might point to some particular sections we'd like reviewers to examine closely.

> Here is an example video (recorded via [Jumpshare](https://hackmd.io/@investinopen/tools-we-use#Jumpshare)):
> <div style="position: relative; padding-bottom: 56.25%; height: 0;"><iframe src="https://jumpshare.com/embed/klBVLgHYm5IHRXhOGMoA" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen style="position: absolute; top: 0; left: 0; width: 100%; height: 100%;"></iframe></div>

.

4. Send invitation emails to invited reviewers
- Adapt [this template](###Reviewer-invitation-email)

### During the invited review period
1. Address questions from reviewers; direct them to the IOI research team where necessary
2. Thank reviewers who have commented
3. Note reviewers who have already commented/declined in [this spreadsheet](https://docs.google.com/spreadsheets/d/1pJL0WfD3mTDvhcnuE0bRp8GZ1cAktL2eFiHf0Y8UXVs/edit#gid=1387591992)
4. **2 working days before the invited review closes** send a reminder to invited reviewers who haven't commented/declined; adapt [this template](###Reviewer-reminder-email)

### After the invited review period
1. Thank reviewers who have commented; adapt [this template](###Reviewer-thank-you-email)
2. Change the header text in the draft to the following:
> This draft is currently being revised - thank you for your comments and input.
> 

3. The author of the report/director of research will acknowledge the receipt of the individual comments and answer questions by replying to them directly. They will make bigger changes to the text on a separate, internal copy of the report.
4. The project coordinator would deposit a public copy of the report with the comments on Zenodo in [IOI's community collection](https://zenodo.org/communities/investinopen/).
5. **When the report is published**: inform the reviewers and thank them once again. Adapt [this template](###Post-publication-email-to-inform-reviewers).

## Templates

### Reviewer invitation email
Subject: `A concise title of the report`

Dear `first name`,

We have conducted a `title of the report, linked to the draft open for comments` and would like to invite your questions, comments and feedback to our draft report, by `invited review closing date`.

`2-3 sentences summarizing the report`.

Given your expertise in`their area of expertise/experience`, `name of person who recommended we reach out to them` has suggested reaching out to you for early comments and review on this draft. **We would be really grateful for your questions, feedback and suggestions, up to and including`invited review closing date`: `link to the draft open to comments`**

Your input will help advance our collective understanding of `area of investigation of the report` For more information and tips on this process, please refer to [our reviewer guide](https://hackmd.io/@investinopen/reviewer-guide).

`(if this is reporting out from user interviews, include this paragraph)` Please note that we’re simultaneously asking all interview participants involved to review this draft and collecting consent for public disclosure of their identities. All participants contribute authoritative insight into this space.

After this review process, we will revise the report. We aim to publish the final version later in `expected publication month for the report`.

Please don’t hesitate to contact us if you have any questions, and we look forward to receiving your comments!

### Reviewer reminder email

*(As a reply to the first invitation email)*

Dear `first name`,

Hoping to follow up on this – we'd be extremely grateful for your comments and feedback on this draft report on `subject matter of the report` by `report closing date`: `link to draft open for comments`

Please do let me know if I can help in any way!

### Reviewer thank you email

Dear `first name`,

Thank you so much for taking the time to read the report and for your comments, which are very helpful. We are currently in the process of revising the report, including adding acknowledgements. `Report author first name` will endeavor to respond in-line to your comments shortly, and we will be in touch again when we release the revised report.

### Post-publication email to inform reviewers

Dear `first name`,

The report on `subject of report` is now live: `link to the blog post on the IOI website`.

Thank you so much for your early feedback on this work!

## See also
- [Reviewer guide](https://hackmd.io/@investinopen/reviewer-guide)
- Archived copy of this page (via the Internet Archive's Wayback Machine): https://web.archive.org/web/*/https://hackmd.io/@investinopen/invited-review

---
This page first published: 2022-11-17

###### tags: `about-us` `process`

---
<a rel="license" href="http://creativecommons.org/licenses/by/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by/4.0/88x31.png" /></a><br />This work is made available under a <a rel="license" href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>. Users are free to share, remix, and adapt this work. (Please attribute [Invest in Open Infrastructure](https://investinopen.org/) in any derivative work).