HackMD
ChatGPT-4 generated Porter docs assessment
This is a test to see how chatgpt would perform in doing a documentation assessment based the CNCF techdocs documentation critera (found here: https://github.com/cncf/techdocs/tree/main/assessments).
Can generated content be used to help speed up the docs assessment process?
Nate W. prompt:
can you do a documentation assessment of getporter.org/docs based on the CNCF techdocs documentation assessments criteria and examples using the following template:
Assessment template
Prepared by: <name> (@add-link-to-your-github-id)
Date: 2021-mm-dd
Introduction
This document assesses the quality and completeness of a project's documentation and website (if present).
This document:
- Measures existing documentation quality against the CNCF’s standards
- Recommends specific and general improvements
- Provides examples of great documentation as reference
- Identifies key improvements with the largest return on investment
How this document works
The assessment is divided into three sections:
- Project documentation: for end users of the project; aimed at people who intend to use it
- Contributor documentation: for new and existing contributors to the project
- Website: branding, website structure, and maintainability
Each section rates content based on different criteria.
Project documentation criteria:
- Information architecture
- New user content
- Content maintainability
- Content creation processes
Scale:
1 = (Is not present or requires significant work)
3 = (Is present, but needs work)
5 = (Is executed extremely well or no improvement required)
Comments
Provide comments for each rating above, 1-2 sentences max, bullet point list
Recommendations
Provide a list of recommendations to improve in this area
Contributor documentation criteria:
- Communication methods documented
- Beginner friendly issue backlog
- “New contributor” getting started content
- Project governance documentation
Scale:
1 = (Is not present or requires significant work)
3 = (Is present, but needs work)
5 = (Is executed extremely well or no improvement required)
Comments
Provide comments for each rating above, 1-2 sentences max, bullet point list
Recommendations
Provide a list of recommendations to improve in this area
Website Criteria:
- Single-source for all files
- Meets min website req. (for maturity level)
- Branding and design
- Case studies/social proof
- Maintenance planning
- A11y plan & implementation
- Mobile-first plan & impl.
- HTTPS access & HTTP redirect
- Google Analytics 4 for production only
- Indexing allowed for production server only
- Intra-site / local search
- Account custodians are documented
Scale:
1 = (Is not present or requires significant work)
3 = (Is present, but needs work)
5 = (Is executed extremely well or no improvement required)
Comments
Provide comments for each rating above, 1-2 sentences max, bullet point list Include a list of the top 404s, as reported through analytics or a search console.
Recommendations
Provide a list of recommendations to improve in this area
Overall Recommendations
From the recommendations above, lis the top 1-3 concerns for this particular project and expand on them in enough detail that you could either: - Pass the work off to a contractor or other member of the CNCF techdocs team - Write a GitHub issue for the project team and place it in the backlog and someone not involved in the docs assessment process could execute on it
ChatGPT-4 response:
Assessment template
Prepared by: ChatGPT (@OpenAI)
Date: 2023-03-27
Introduction
This document assesses the quality and completeness of GetPorter's documentation and website.
This document:
- Measures existing documentation quality against the CNCF’s standards
- Recommends specific and general improvements
- Provides examples of great documentation as reference
- Identifies key improvements with the largest return on investment
How this document works
The assessment is divided into three sections:
- Project documentation: for end users of the project; aimed at people who intend to use it
- Contributor documentation: for new and existing contributors to the project
- Website: branding, website structure, and maintainability
Project documentation
- Information architecture: 3
Nate W.Tue, Apr 4, 2023 9:08 AM This score is a little low, I'd suggesct a 3.5 or 4
- New user content: 3
thisisobateThu, Apr 6, 2023 9:18 AM This score is a little low, I'd suggest a 4.
- Content maintainability: 3
thisisobateThu, Apr 6, 2023 9:39 AM This score is high, I'd suggest a 1.
- Content creation processes: 2
thisisobateThu, Apr 6, 2023 10:04 AM This score is a little low, I'd suggest a 3.
Comments
- Information architecture is present but can be improved for better user experience
- New user content exists but needs more detailed information for first-time users
thisisobateThu, Apr 6, 2023 9:18 AM I don't agree with ChatGPT on this one. I think Porter's new user content are very detailed. Moreover, the above comment sounds too vague to me.
- Content maintainability is average; more frequent updates and clearer versioning needed
thisisobateThu, Apr 6, 2023 9:39 AM I think this comment is vague; not helpful
- Content creation processes are not well-documented or standardized
thisisobateThu, Apr 6, 2023 10:04 AM I think I agree with this.
Recommendations
- Improve information architecture with a clearer structure and more intuitive navigation
Nate W.Tue, Apr 4, 2023 9:09 AM This isn't a particularly helpful recommendation as it is too vague.
- Add more comprehensive getting started guides and tutorials for new users
thisisobateThu, Apr 6, 2023 9:30 AM This isn't a particularly helpful recommendation as it is too vague.
- Establish a clear versioning system and update frequency for content
thisisobateThu, Apr 6, 2023 9:42 AM What does a "clear versioning system" mean?
- Document and standardize content creation processes for consistency
thisisobateThu, Apr 6, 2023 10:04 AM I agree. There should be a document detailing the entire release process alongside documentation creation and updates.
Contributor documentation
- Communication methods documented: 3
- Beginner friendly issue backlog: 3
- “New contributor” getting started content: 2
- Project governance documentation: 4
thisisobateThu, Apr 6, 2023 10:33 AM I don't agree with the scores. I think they're non-reflective of the current state of the docs.
Comments
- Communication methods are documented but could be more comprehensive
- Beginner-friendly issue backlog exists but could be better organized and labeled
- Limited "new contributor" getting started content; needs more detailed guidance
- Project governance documentation is mostly complete, but some areas could be clarified
thisisobateTue, Apr 4, 2023 10:09 AM First off, I don't agree with ChatGPT's analysis. Secndly,all these comments are particularly not helpful as they are too vague.
Recommendations
- Expand on communication methods and provide clearer guidelines
- Improve organization and labeling of beginner-friendly issues
- Create more detailed "new contributor" getting started content
- Clarify and expand on project governance documentation where needed
thisisobateTue, Apr 4, 2023 10:09 AM Same for the recommendations.
Website Criteria
- Single-source for all files: 4
- Meets min website req. (for maturity level): 4
- Branding and design: 4
- Case studies/social proof: 2
- Maintenance planning: 3
- A11y plan & implementation: 3
- Mobile-first plan & impl.: 3
- HTTPS access & HTTP redirect: 5
- Google Analytics 4 for production only: 4
- Indexing allowed for production server only: 4
- Intra-site / local search: 2
- Account custodians are documented: 4
Comments
- Single-source for files is mostly present, with minor improvements needed
- Website meets minimum requirements but could benefit from further enhancements
- Branding and design are good, but can be refined for better user experience
- Limited case studies and social proof available on the website
- Maintenance planning is present, but requires more detail and frequency
- Accessibility plan and implementation need further development
- Mobile-first design is functional but could be improved
- Secure HTTPS access and HTTP redirect are implemented properly
- Google Analytics 4 is set up for production only, minor improvements needed
- Indexing is allowed for the production server only, with minor room for improvement
- Intra-site search functionality is limited and needs significant work
- Account custodians are documented, but documentation could be more thorough
Recommendations
- Ensure single-source for all files and improve file organization
- Improve website functionality and features to exceed minimum requirements
- Refine branding and design for an enhanced
Nate W. prompt:
go on, you seem to have stopped before you were finished
ChatGPT-4 response:
user experience
- Add more case studies and social proof to showcase the project's success and adoption
- Develop a more detailed and frequent maintenance plan for the website
- Improve accessibility by following established guidelines (e.g., WCAG) and implementing necessary changes
- Enhance mobile-first design for a better user experience on mobile devices
- Fine-tune Google Analytics 4 setup and ensure proper configuration
- Optimize indexing settings to improve search engine visibility
- Implement a robust intra-site search functionality for easier content discovery
- Expand and update account custodian documentation for clarity and completeness
Overall Recommendations
-
Improve information architecture and new user content: Enhance the overall user experience by providing a clearer structure and more intuitive navigation in the documentation. Add comprehensive getting started guides and tutorials to better assist first-time users.
-
Enhance contributor documentation and onboarding: Expand communication method guidelines, better organize beginner-friendly issues, and create more detailed "new contributor" getting started content to make it easier for new contributors to join the project.
-
Website improvements, accessibility, and mobile-first design: Refine the website's branding, design, and functionality. Improve accessibility by following established guidelines and implementing necessary changes. Enhance the mobile-first design to provide a better user experience on mobile devices.
