# User guide for CoPED
## About
The [Catalogue of Projects on Energy Data (CoPED)](https://coped.energy/) portal captures and unifies metadata from various information stores and makes them accessible to energy actors using open source business intelligence tools.
### Data sources
Currently, the data is obtained solely from the [UKRI gateway to publicly funded research and innovation](https://gtr.ukri.org/), but the aim is for further repositories to be added soon. We are also open to suggestions so if you have any interesting or useful sources then please raise it as an issue on our [GitLab repository](https://gitlab.com/es-catapult-public/coped).
### Identifying Energy projects
The complete collection of projects is filtered to focus on energy related project using various glossaries of energy search terms developed from existing knowledge in the field. This inlcudes using [Ofgem's Energy Terms](https://www.ofgem.gov.uk/energy-terms-explained), the US Office of Energy Efficiency and Renewable Energy [Glossary](https://www.energy.gov/eere/fuelcells/glossary), EDF's [Glossary](https://www.edfenergy.com/large-business/glossary), and the US Energy Information Administration's [Glossary](https://www.eia.gov/tools/glossary/).
The list of terms can be amended and updated to include other terms and identify other relevant projects. If you notice any relevant UKRI projects which are not included in our database then please raise it as an issue on our [GitLab repository](https://gitlab.com/es-catapult-public/coped). This will help us identify all relevant projects not identified in the original filter.
### Subject extraction
To enable searching of the projects, the corpus of collected projects including their abstract is processed with a third party Natural Language Processing service to extract the most relevant keywords and subjects. Further information on the NLP packages used can be found via the [GitLab repository](https://gitlab.com/es-catapult-public/coped).
### CoPED database
At the heart of the platform there is a Django project setup which manages CoPED energy project data scraping using [Scrapy](https://docs.scrapy.org/en/latest/) and the definition of the PostgreSQL database. Further information on the code is available via the [GitLab repository](https://gitlab.com/es-catapult-public/coped).
The database is then linked to the CoPED Metabase platform, which provides built-in, intuitive data exploration, visualisation, analysis and sharing capabilities. For further information about Metabase a skim trough their extensive [documentation](https://www.metabase.com/docs/latest/) is advised.
### CoPED Git platform
If you like the project and want to contribute to CoPED, we would love your input! We want to make contributing to this project as easy and transparent as possible, whether it's:
* Reporting a bug
* Discussing the current state of the code
* Submitting a fix
* Proposing new features
* Becoming a maintainer
Have a look at the [GitLab repository](https://gitlab.com/es-catapult-public/coped) and continue reading our User's Guide!
# CoPED's Features
CoPED has the following main features:
* Project, People and Organisation Search (See Section on [Searching for People, Organisations and Projects](#Searching-for-People,-Organisations-and-Projects)).
* Visualisations through plots and Dashboards (See Section on [Data and Visualisations](#Data-and-Visualisations)).
* Messaging and Announcement Service (See Section on [How to use the messaging system](#How-to-use-the-messaging-system)).
The remainder of this guide will describe how to use these features and others within the CoPED platform in more detail.
# Signing Up and Searching
## Signing up and creating a profile
A profile is required to do the following:
1. Message another user
2. Add, edit, or claim a project on the [CoPED website](https://coped.energy/)
3. View full data tables or create visualisations on [CoPED's Metabase platform](https://coped.energy/metabase/).
To create a profile, first go to the [CoPED homepage](https://coped.energy/).
On the homepage, click the "Sign up" button in the upper right-hand corner. That will redirect you to the [CoPED registration page](https://coped.energy/accounts/register/).
Enter details here, which includes a username, email address, first name, last name, password, and password confirmation. Click the Captcha tick box to confirm you are human and complete the task on screen. Then click "Submit".
You should then receive an email asking you to confirm your account registration. This link will be valid for 7 days.
## Searching for People, Organisations and Projects
The most basic feature of CoPED is the search function. This is available directly from the [CoPED homepage](https://coped.energy/). Further filters can be applied to the initial search including information on the start or end year of the project, minimum and maximum funding amount and whether the projects are active or closed. Similarly people can be searched via their name, or organisation by name.
Alternatively, you can begin your search with more information directly by clicking the relevant link (Project, People or Organisation) at the top of the homepage, and then entering the details in the boxes on the side of the search window. You can also sort your searches by various information such as name (ascending or descending) or Start Data if relevant to the information.
The example below shows the results from searching for the EnergyREV project from the homepage.
You may find that some information is not available which you would expect to be found. This may be because:
* The project, person or organisation is not on the current repositories extracted by CoPED (currently just the UKRI gateway).
* The information does not contain our current energy project keywords used to extract the data.
* The information has not yet been extracted (data is updated monthly).
If the data should be in the list and is not explained by the above list then you can either add the project manually by becoming a contributor (See Section [Request contributor permission](#Request-contributor-permission)) or by creating an issue in the CoPED [GitLab repository](https://gitlab.com/es-catapult-public/coped).
## Finding Similar Projects
Project similarity is based on the text similarity metric provided by the [Finto AI](https://github.com/NatLibFi/FintoAI) API, which compares the project descriptions to suggest similar projects.
If you are in a project details page, you can acces similar projects clicking on the 'Find Similar' button just below the title.
You should now be prompted with the list of the top 100 similar projects to the starting one.
## Saving Favourites
This feature allows users to easily access database entries they frequently visit saving them in a personal list of favourites. This can be done for projects, people and organisations in the CoPED database.
To save a project in your list of favourites start from the [CoPED homepage](https://coped.energy/) and go to the projects page.
CLick on the project(s) you are interested in to access the project details page, from here click the 'Add Favourite' button (the one with the star symbol just below the project title).
In this example we will repeat the process for the first three entries in the projects, people and organisations pages respectively.
Once you have added all you entries of interest, you can quickly access them from the 'Favourites' page under your username dropdown menu in the top right corner.
# Messaging and Announcement
The messaging system can be used to do the following:
1. Send a message to another user or administrator.
2. Request ownership of a project, person, or organization.
3. Request a data change on a project, person, or organization.
4. Request contributor permission.
In the sections below, we will demonstrate the use of this system for sending messages to others, claiming ownership of a project, requesting data changes on an existing project, and requesting permission to contribute.
### Messaging another user or administrator
If you wish to send a message to another user or the administrator, starting from the [CoPED homepage](https://coped.energy/), click your username in the top right corner. This will open a drop-down menu with multiple options including "Messages".
After clicking the "Messages" option, you will be redirected to your messaging system, shown below. Here you can view unread or read messages and send a new message. There are two separate new message buttons: one to contact another user and one to contact the administrator.
Clicking "New Message" will allow you to send a message to another user, including the administrator "copedadmin". Clicking the "Contact Administrator" button will only allow you to message the administrator. Both buttons open a similar page, shown below, where you can type your message.
After clicking "Send", the user will be notified of a new message in their inbox.
### Receiving a message
If you receive a message, you will receive a notification next to your username in the top right corner.
If you click your username with the red notification icon, a drop-down menu will appear.
Click "Messages" to access your inbox, shown below.
Here we see one unread message. If you click "View", you are able to read the full message and reply as appropriate.
### Submit an announcement to the platform administrator
To communicate with the platform administrator you can use the Announcement section. For example you may want to notify that a new project has been added to the database. From the Homepage, go to the [Announcement page](https://coped.energy/announcements/) clicking on the Announcement button, then click submit announcement and fill in the announcement details to inform the administrator about the announcement proposal.
This action will start a conversation with the admin that you can access through your messages section.
Once reviewed and accepted by the admin the announcement will appear in the Announcement section.
# CoPED Roles and Project Ownership
The following roles exist for using and managing the CoPED platform:
* Contributors: These have full contributor status to edit, delete and add projects to the data within CoPED. This is currently reserved for the CoPED adminsitrators.
* Owner: These can claim ownership over a project and may edit the project information contained within CoPED.
* Normal User: These users cans search, and save projects but have no editing capabilities. However, they can request changes through the messaging systems.
### Claim ownership of a project
Ownership of a project means that the user will have the write permissions to edit and update the project details. They will not have permissions to change other projects which they are not owners of. This capability is reserved for administrators who will have "contributor" status.
Starting on the [CoPED homepage](https://coped.energy/), there are two options to view an existing project, as the blue arrows indicate in the image below:
1. Type the name into the "Search for projects" bar, or
2. Click the "Projects" button on the top of the page.
In the below example I am searching for the EnergyREV project, the results of this search are shown below.
The third project down is the correct one, so click the blue project title text to redirect to the project page. On this page, one of the options under the title is to "Claim Ownership" via the green button, as shown below.
If the "Claim Ownership" button is not appearing, it is possible that someone else has already claimed ownership. Please see the [Contacting a project owner](#Contacting-a-project-owner) section for further details.
Clicking the "Claim Ownership" button will redirect you to the messaging system, shown below which will create a message to send to the CoPED administrators.
The "To user", "Subject", and some of the "Content" fields are pre-filled with relevant information, but the Content box has the text prompts:
1. What is your involvement with the project?
2. Your name and contact details (optional).
This information should be filled in by the user to help the administrator verify whether ownership should be approved. Once the relevant information is added, click "Send" to send your request to the administrator.
The administrator will then evaluate and reply to your request to confirm or deny your ownership of the project.
### Contacting a project owner
If a project has already been claimed, you will see a "Contact Owner" button in place of the "Claim Ownership" one, as shown below.
(Note that the owner should be someone else who has worked on the project and therefore should be known to the user. If there is an anomaly or the project owner is incorrect then please contact the administator through the messaging system (see Section [Messaging another user or administrator](#Messaging-another-user-or-administrator)) to resolve this).
Clicking the "Claim Ownership" button will redirect you to the messaging system, shown below, where you can see the username of the owner.
You are then able to contact the owner by typing your message in the "Content" section and pressing "Send".
### Request contributor permission
You will need contributor permission to add a new project, person, or organisation to CoPED. This is reserved for administrators since it will allow editing capabilities across the platform. The contributor request can be done in two ways.
The first way is to go to your profile. From the [CoPED homepage](https://coped.energy/), click your username in the top right corner. This opens a drop-down menu with the option "Profile" at the top.
Clicking "Profile" redirects you to your profile details, as shown below. Here you can click the yellow button for "Request contributor permissions".
The second way to make this request is through the Projects page. Starting from the [CoPED homepage](https://coped.energy/), click the "Projects" button in the top menu bar.
Clicking "Projects" will open the page shown below. Here you can click the yellow "Request Permission to Contribute" button.
Both approaches will open the page shown below.
Here you can detail why you are making the request along with your name and details. After this is done, click "Submit" to send your request to the administrator.
# Editing or Changing CoPED Information
## Request a data change to a project
Following the previous example, you may have noticed an error or typo in the description on the EnergyREV project page.
If you notice an error, or want to request another data change, this can be done in a number of ways depending on whether you own the project or not. If you are a project owner then you can directly make the changes on the project (See Section [Claim ownership of a project](#Claim-ownership-of-a-project) to see how to become a project owner.). Alternatively, data changes can be done by clicking the yellow "Request Data Change" button at the top of the corresponding project page.
Clicking the "Request Data Change" will redirect you to the messaging system to the CoPED administrators. The content field is pre-filled with relevant information and inlcudes text prompts, for details on:
1. What needs to be changed?
2. What is your involvement with the project?
Once the relevant information is added, click "Send" to send your request to the administrator.
### Adding a project and corresponding information
To add a project that is not already present in the CoPED database go to the [projects page](https://coped.energy/projects/). If you have [contributor permission](#Request-contributor-permission) the 'Create New+' button should be visible in green at the top of the projects table.
Click it and access the project creation page, fill in all the project details and click save at bottom of the page.
You will land on the project details page which will look like any other existing project in the CoPED database.
### Editing/updating a project
To edit an existing project go to the [projects page](https://coped.energy/projects/) and find the project you want to modify. In the project details page click on the green 'Update' button, change the information theat needs to be updated and click save.
Note that only owner of the project and administrator can edit or delete a project. If you are not in one of these categories you can request a project change or contact the owner of the project.
To verify if the update was saved open the history page and check if the latest modification is there.
# CoPED's Metabase platform
The CoPED Metabase platform can be accessed either directly at [https://coped.energy/metabase/](https://coped.energy/metabase/) or through the [CoPED homepage](https://coped.energy/) by clicking the 'Data & Analysis' button and selecting 'Home' from the dropdown menu.
Please note that a login is required to access to the Metabase platform.
## Investigating the data in the CoPED database
To learn more about the data in the CoPED database start from the [CoPED homepage](https://coped.energy/) by clicking the 'Data & Analysis' button and selecting 'Home' from the dropdown menu. Select 'CoPED Database' and then the 'Learn about our data' button in the top right corner. Select 'CoPED Database' again to access the details about the database as well as its tables.
Select 'Tables in CoPED Database' to access details about the tables, select a table to know more about the fields it contains and questions using the table. Here is what the 'Organisation' table details look like:
It is a good idea to familiarise whit the tables and their content before attempting a visualisation.
# Data and Visualisations
## Downloading data from a table or visualisation
<iframe width="560" height="315" margin= 0 auto src="https://www.youtube.com/embed/uqbTTK0EifA?controls=0" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" display= block border-style= none allowfullscreen></iframe>
If you are interested in downloading data from the visualisations on the [CoPED homepage](https://coped.energy/), put your cursor over the visualisation you are interested in. A cloud button will appear in the top right-hand corner of the visualisation, as shown in the "Project funders" visualisation below.
Click the cloud button to download full results. You will then be given the option of downloading three different file types (csv, xlsx, or json).
If you are interested in dowloading data from the [CoPED Metabse platform](https://coped.energy/metabase/), you will need to be logged in. From the homepage, we can view all the data tables under 'Base Views'. After selecting the table you are interested in, you will be able to view the data.
In the example below we chose 'Project funds'. To download the data from the table, click the cloud button in the bottom right-hand corner and choose the desired file type.
## Creating a simple visualisation on a single table
<iframe width="560" height="315" src="https://www.youtube.com/embed/jiT4KyisjFw?controls=0" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
To create a simple visualisation from a single table, start on the [CoPED Metabase platform](https://coped.energy/metabase/), click on CoPED Database under OUR DATA and chose the desired database table. In the example below we choose 'Project Subject'. Click on the 'Show editor' button in the top right corner and summarise the table using a count of rows by subject label. You may also want to narrow the results using a filter that returns only the rows that have a count value above let us say 500. Eventually, it is possible to sort the data by count value.
Once you are satisfied with the result (check the preview using the play button beside the last instruction), you can move on to the visualisation section pressing the 'Visualize' button.
For a new visualisation, you will be prompted with the 'Table' type of visualisation. Click on the 'Visualization' button on the bottom left corner to change the type to 'Row'.
## Combining tables to visualise linked data
To link data from multiple objects in the database, we strongly reccomend to look at the CoPED database relationships table, which defines how to join the different tables depending on the type of relatioship between them (one-to-one, one-to-many, many-to-many).
In this example we will combine the 'Organisation' table with the 'Subject' table.
To combine tables and visualise linked data, start on the [CoPED Metabase platform](https://coped.energy/metabase/), click on CoPED Database under OUR DATA and chose the table 'Organisation'. Looking at the relationship table you can soon realise that it is not possible to join it directly with the subject table, you need join first on the 'Project Organisation' for that. Click the join button and select the second table. As for the join key, select the prinary key 'ID' for the Organisation table and the foreign key 'Organisation' for the Project Organisation table.
The following step will be to join 'Project Organisation' and 'Project Subject' linking tables on the foreign key 'Project'. Once done that, you can finally join the previous results with the 'Subject' table on the 'Subject' key. When satisfied with the result, you can clean up you resulting data by unchecking undesired columns using the pick columns button.
## Creating a visualisation from linked data
Starting from the previous example on [Combining tables to visualise linked data](#Combining-tables-to-visualise-linked-data), we now want to visualise the data in a graph. Before doing that, in this example we will filter the data including only the rows whose 'Project Organisation - Role' value is 'LEAD_ORG' to narrow down the results to leading organisation. As last thing before plotting, you might want to summarise the data using a count of rows and group them by 'Subject -> Label' to quantify the number organisations leading on projects with a certain subject. Order by count and limit the result to 20 rows to make it more readable.
Click the 'Visualize' button and choose 'Pie' as a visualisation type. Make sure in the Dimension and Measure fields, 'Subject -> Label' and 'Count' are selected respectively.
If you want to be able to use this visualisation for further analisys make sure you save it in your personal collection using the 'Save' button in the top right corner.
## Building a new visualisation from an existing one
To build a visualisation from an existing one, start from the [CoPED Metabase platform](https://coped.energy/metabase/), click on 'Browse all items' under OUR ANALYTICS and then go to 'Your personal collection'.
Open the question from the [Creating a visualisation from linked data](#Creating-a-visualisation-from-linked-data) example and switch to the editor page, in the 'Summarize' section change the 'by' value from 'Subject -> Label' to the 'Name' column of the 'Organisation' table. This in order to quantify the number subject leading organisations have on their projects.
Visualise the result as a bar chart, you can also change the graph appearance in the settings if you are not happy with the default ones.
Make sure you save it with a different name without replacing the previous question before closing it, as we will use it in the next example to create our first dashboard.
## Existing Dashboards
Dashboards provide high-level, centralised insights on the landscape of energy projects. They can cover everything from general overview of funding and knowledge exchange to detailed metrics focused on stakeholders.
The [CoPED Metabase platform](https://coped.energy/metabase/) contains already several predeveloped dashboards, aimed at promoting collective management of data resources.
To access them go to the 'Main Dashboards' collection:
* Homepage Dashboard - Overview of project counts in each DNO region, project funders across all projects, and top 10 subject categories.
* Funding over time - The total funds allocated to projects by start and end year and their combination.
* Organisation Collaborations - Who collaborates most on funded energy projects?
* Overview of projects - A percentage breakdown of funders by the number of projects funded.
* Organisation Dashboard - A broad picture of the energy projects landscape filterable by organisation.
* Person Dashboard - A broad picture of the energy projects landscape filterable by individual.
* Funding Body Dashboard - A focus on the funding of energy projects filterable by funding body.
* Collaborations by subject - A focus on the number of collaborations filterable by subject.
* Funding by subject - A focus on the funding of projects filterable by subject.
* Projects by subject - A focus on the number of projects filterable by subject.
We encourage every user to use them as they are, as well as to duplicate them in your collection and craft them to a specific use case.
## Creating your own dashboard
Dashboards are useful for grouping a set of related visualisations in the same page.
To create your own dashboard, start from the [CoPED Metabase platform](https://coped.energy/metabase/), click on the plus symbol in the top right corner and select create new dashboard, fill in the name and description fields and click create.
At this point you should see an empty dashboard. To populate it, click the edit dashboard button in the top right corner and then the plus symbol to add a question. You should be prompted with saved question from your personal collection, add them both and click save. You may want to add a short description of the dashboard content in a Markdown formatted text box as well. Remember to save the changes you made in edit mode to make them permanent.
Every time you open a dashboard, it will fetch data directly from the CoPED database tables. This happens according to the instructions contained in the saved questions. In other words the data will be always up to date.
To access a single question from the dashboard click on a card's title, a copy of the main question will be opened with settings inherited from the dashboard (if any). You can notice it because of the 'originated from' message at the right of the derived question.
NOTE: If the main question has multiple summarise sections, the last one will be lost when opening a derived question from a dashboard (e.g. when viewing the results from a particular bar in a bar plot).
## Adding filters to make dynamic dashboards
It is often the case that a user wants to see the same visualisation for a slightly diffent value of a variable. In this case you can add a filter to a dashboard to make it dynamic.
Starting from the dashboard created in the previous section, click the pencil icon to enter dashboard editing mode, then click the 'Add a filter' button, select 'Other categories', give a descriptive name to the filter in the 'Label' section, select 'Subject.Label' as column to filter on for both visualisation and click save.
Unless you specify a default value for the filter in edit mode the dashboard will appear the same as before because it will be using all the entries of the filtered column. The main difference is that now in the top left corner you can see a Subject filter box, which you can use to select only the variables of interest for that column.
In this example we used 'batteries', 'wind energy', 'solar energy' and 'artificial intelligence' as values for the 'Subject.Label' column.
Try different subject and see how the visualisation changes accordingly. You can also define other types of filters depending on the data types available in the question columns.
NOTE: The dynamic filter works adding a temporary filter section just before the last summarise in the main question. You can notice it by opening the editor from the derived question. Make sure the variable you want to filter on is at least in the penultimate summarise section!
## Adding filters to make searchable tables
To make a searchable table, in this example we will start from the question in section [Building a new visualisation from an existing one](#Building-a-new-visualisation-from-an-existing-one). Open the quesiton from your personal collection or directly from the dashboard. Add 'Subject.Label' and 'Organisation.Role' in the Summarize section, remove the row limit and modify visualisation type to table. Save it as a new question and add it to a dashboard (the one we created in the previous section for example).
The table card will behave like any other visualisation in the dashboard, so you can assign a filter by clicking on the 'Edit dashboard' button, then on the filter field to edit the 'Column to filter on' field for the table. It is easy to notice that other variables are available, for which you might define a new filter. In this example we added a filter on the 'Organisation.Name' column.
You can combine both filters to narrow the search when looking for specific information in the table. In this case we used the same values from the previous section for the subject filter and restricted the 'Organisation.Name' value to be 'Offshore Renewable Energy Catapult'.
## How to share or embed a visualisation or dashboard
The easiest way to share a question or a dashboard is to use the public link sharing option. In the dashboard page, click on the sharing icon (the arrow pointing up and to the right) in the top righ corner, choose sharing and embedding and you will be presented with the sharing option of a public link or a code snippet for a public embed.
This type of sharing will work also for people lacking a CoPED account and it is the less restrictive one, for this reasons only administrators can enable it.
If you are an administrator you are fine, otherwise you must ask an admin to share or enable sharing in the dashboard or question for you. Another option is to ask an admin to assign you to a group with curation permission on a collection and save your work in that collection or to move (or duplicate) your work in an public CoPED collection.
For a complete overview of the multiple sharing options available it is worth to have a look at [Metabase Guide to Sharing data](https://www.metabase.com/learn/administration/guide-to-sharing-data) and [Metabase Publishing data visualization to the web](https://www.metabase.com/learn/customer-facing-analytics/embedding-charts-and-dashboards) article.
Sign in with Wallet
Connect another wallet