# ODP Issues

## OD-1557 Parameterize SQL statements
### Time spent: 11h
### Description
Static code analysis tool “bandit” finds the issues below in the ckanext-ed code. All the issues are related to constructing SQL queries using string composition functions, which can be addressed by switching to using bound parameters in the SQL query strings.

Additional locations:
Location: ckanext-ed\ckanext\ed\actions.py:1959:10
Location: ckanext-ed\ckanext\ed\actions.py:1966:11
Location: ckanext-ed\ckanext\ed\actions.py:1973:11
Location: ckanext-ed\ckanext\ed\actions.py:2033:11
Location: ckanext-ed\ckanext\ed\actions.py:2070:15
Location: ckanext-ed\ckanext\ed\blueprints\ed_stats.py:116:14
Location: ckanext-ed\ckanext\ed\blueprints\ed_stats.py:163:14
Location: ckanext-ed\ckanext\ed\blueprints\ed_stats.py:456:14
Location: ckanext-ed\ckanext\ed\commands\fix_extras.py:49:16
Location: ckanext-ed\ckanext\ed\commands\fix_extras.py:69:17
Location: ckanext-ed\ckanext\ed\commands\populate_recordsdb.py:27:18
Location: ckanext-ed\ckanext\ed\dbutil.py:66:14
Location: ckanext-ed\ckanext\ed\dbutil.py:82:14
Location: ckanext-ed\ckanext\ed\helpers.py:840:10
Location: ckanext-ed\ckanext\ed\helpers.py:899:10
Location: ckanext-ed\ckanext\ed\model.py:165:14
Location: ckanext-ed\ckanext\ed\model.py:179:27
Location: ckanext-ed\ckanext\ed\model.py:189:21
Location: ckanext-ed\ckanext\ed\model.py:195:20
Location: ckanext-ed\ckanext\ed\model.py:200:20
Location: ckanext-ed\ckanext\ed\model.py:205:20
Location: ckanext-ed\ckanext\ed\model.py:212:33
Location: ckanext-ed\ckanext\ed\model.py:229:33
Location: ckanext-ed\ckanext\ed\model.py:260:14
Location: ckanext-ed\ckanext\ed\model.py:289:10
Location: ckanext-ed\ckanext\ed\model.py:354:10
Location: ckanext-ed\ckanext\ed\model.py:378:11
Location: ckanext-ed\ckanext\ed\model.py:409:10
Location: ckanext-ed\ckanext\ed\model.py:427:10
Location: ckanext-ed\ckanext\ed\model.py:440:10
Location: ckanext-ed\ckanext\ed\model.py:448:14
Location: ckanext-ed\ckanext\ed\model.py:459:14
Location: ckanext-ed\ckanext\ed\model.py:470:18
Location: ckanext-ed\ckanext\ed\model.py:485:10
Location: ckanext-ed\ckanext\ed\model.py:495:14
Location: ckanext-ed\ckanext\ed\model.py:510:18
Location: ckanext-ed\ckanext\ed\model.py:521:22
Location: ckanext-ed\ckanext\ed\model.py:534:10
Location: ckanext-ed\ckanext\ed\model.py:544:14
Location: ckanext-ed\ckanext\ed\model.py:566:10
Location: ckanext-ed\ckanext\ed\model.py:576:14
Location: ckanext-ed\ckanext\ed\model.py:596:10
Location: ckanext-ed\ckanext\ed\model.py:656:10
Location: ckanext-ed\ckanext\ed\model.py:669:10
Location: ckanext-ed\ckanext\ed\model.py:708:14
Location: ckanext-ed\ckanext\ed\model.py:730:14
Location: ckanext-ed\ckanext\ed\model.py:750:10

Location: ckanext-ed\ckanext\ed\model.py:772:14
Location: ckanext-ed\ckanext\ed\model.py:793:14

### Tasks
- [x] Check the Bandidt output - is this a security concent?
- [x] Do an investigation into the LOE that would be needed to refactor all the string values into parameterised SQL statements
- [x] Provide the LOE that would be needed to the team

## OD-1556 Clarify email hash purpose
### Time spent: 4.75h (investigation + fix for forgot password)
### Description
Static code analysis tool “bandit” finds the issue below in the ckanext-ed code.

>> Issue: [B324:hashlib] Use of weak MD4, MD5, or SHA1 hash for security. Consider usedforsecurity=False

Severity: High Confidence: High

CWE: CWE-327 (https://cwe.mitre.org/data/definitions/327.html)

Location: ckanext-ed\ckanext\ed\blueprints\ed_forgot_password.py:47:25

More Info: https://bandit.readthedocs.io/en/1.7.4/plugins/b324_hashlib.html

46 email_hash = email.strip().lower().encode('utf8')

### Tasks
- [x] 1.)Understand Bandit
> * Based on investigation / discussion we could potentially use a security flag to disable security flag
> * BUT based on internal discussion it might very well be a security risk
- [x] 2.) Investigate whether is really a security
- [x] 3.)What will it take to correct it if it’s a security concern

## OD-1553 Visibility toggle not visible for users utilizing higher contrast
### Time spent: 2.25h
### Description

Using both Edge and Chrome from my Ed Laptop > logged into Prod > selected Hidden Data Profile > visibility toggle is not visible using higher contrast.


Notes:
Elements can't rely on background color for Windows high contrast mode to work properly. Borders were conditionally added to the toggles using [this](https://developer.mozilla.org/en-US/docs/Web/CSS/@media/forced-colors) CSS media query

## OD-1547 data.json internal endpoint config does not appear to work

TIMEBOX 2 days / 16h

### Time Spent:10h

### Description
While attempting to fix the data.json functionality, I found that the internal endpoint configurable for the datajson plugin (ckanext.datajson.path) is not working properly. It always returns a 404 from ckan no matter how Apache is set up to use it (even if nothing special is done for routing or permissions).

We have a workaround for building a cache file, but this should be investigated if it’s desirable to use this endpoint instead.

### To Do
* Set up Apache → verify on local then testing
* Endpoints
* Cache

## OD-1538 Lists: `li` Elements & List Structure
### Time spent: 2h

### Description
1.) The following Cypress test error Ensures <li> elements are used semantically (listitem) was found on the following 2 routes:

## OD-1536 Duplicated IDs

### Time spent: 4h

### Description
The following Cypress test errors `Ensure every id attribute value of active elements is unique (duplicate-id-active)` and `Ensures every id attribute value used in ARIA and in labels is unique` were found. Related to ticket:**OD-1510: File inputs have duplicated ids, causing screen readers to read wrong label when input is focused**

### Tasks:

- [x] Run tests
- [x] Document which routes are being flagged
- [x] Fix errors
- [x] Test the entire route for errors when testing with Cypress Axe to make sure all automated tests are now fixed

### NOTES:
* Can affect functionality
* Ensure you are using the latest version of the code

### Implementation notes
Routes:
- "/category",
- "/ckan-admin/config",
- "/collection/",
- "/dashboard/groups",
- "/dashboard/topics",
- "/data_explorer",
- "/dataset/",
- "/dataset/<id>/resource/<resource_id>/edit",
- "/dataset/new",
- "/ed_source/",
- "/harvest/",
- "/organization/<id>",
- "/organization/bulk_process/<id>",

## OD-1523 Dependabot: (High) Alert #56 minimatch ReDoS vulnerability

### Time spent: 0.5h

### Description
Based on the work in **OD-1478: Dependabot issues review**, Dependabot alert #56, high priority, remains open. This ticket is to track the fix.

└─┬ chromedriver@2.43.1
└─┬ del@3.0.0
└─┬ globby@6.1.0
└─┬ glob@7.1.3
└── minimatch@3.0.4

Chromedrives sub dependency.

Dependabot creates a PR for this and it can be safely merged.

Also Dependabot points out that the higher version is 99% compatible.

**ORIGINAL IMAGE IN THE TICKET:**

## OD-1524 Dependabot: (High) Alert #27 glob-parent before 5.1.2 vulnerable to regular expression ...
### Time spent: 0h

### Description
Based on the work in **OD-1478: Dependabot issues review**, Dependabot alert #56, high priority, remains open. This ticket is to track the fix.


## OD-1531 Dependabot: (High) Alert #60 qs vulnerable to Prototype Pollution

### Time spent: 0.5h

### Description
Based on the work in **OD-1478: Dependabot issues review**, Dependabot alert #60, high priority, is new. This ticket is to track the fix. Also to confirm how to respond, in general, to the “prototype pollution” tickets

**ORIGINAL IMAGE IN JIRA TICKET:**

└─┬ chromedriver@2.43.1
└─┬ request@2.88.0
└── qs@6.5.2

Chromedriver sub dependency (we don't even use that, in the future we can clean this up).

Dependabot creates a PR for this and it can be safely merged.

## OD-1525 Dependabot: (Low) Alert #58 decode-uri-component vulnerable to DoS
### Time spent: 0.5h

### Description
Based on the work in **OD-1478: Dependabot issues review**, Dependabot alert #58, low priority, is new. This ticket is to track the fix.

**ORIGINAL IMAGE IN THE JIRA TICKET:**


└─┬ gulp@4.0.2
└─┬ glob-watcher@5.0.5
└─┬ anymatch@2.0.0
└─┬ micromatch@3.1.10
└─┬ snapdragon@0.8.2
└─┬ source-map-resolve@0.5.3
└── decode-uri-component@0.2.0

Gulp sub dependency.
Compatibility is high.

Tested Gulp with `make assets` after upgrading (by mergin the PR) and it's still working fine.

## OD-1514 Fix integration tests for Cypress Axe Testing

### Time spent: 1.75h

### Description
508 Issues have passed testing on various compliance extensions but the integration Cypress Axe tests are failing

Contrast is failing - need to know if it’s failing for the same reasons and testing the same elements

**Investigation timebox: 4h**

## OD-1509 Dropdown values are read twice

### Time spent: 1.75h

### Description
When using a screen reader, the values listed in a select2 dropdown are read twice. To resolve this, we can add `aria-hidden="true"` to the select2 autogen list of options. The issue here is that we need to find a way to intervene.

An initial attempt to resolve this failed. You can’t get/override the select2 option element IDs until you’ve interacted with the dropdown because they don’t exist on the page until the dropdown is interacted with. We have the original HTML options in some cases (such as level of data), but select2 takes the values on the fly and returns the final element.

#### Investigation Timebox: 4h

***ASSOCIATED TICKET: OD-1506 Select 2 Issues: Form Label***

**DESCRIPTION**
|Group|Route|Issue|
|-----------------|
|Dataset|/new/<id>/edit |‘Ensures every form element has a label (label)’|
|CKAN Admin |/ckan-admin/config |Ensures every id attribute value used in ARIA and in labels is unique (duplicate-id-aria)’|
|Collection |/collection | ‘Ensures every form element has a label (label)’ (change owner user selection)|
|Documentaion |/documentation/resources/<id> |‘Ensures every form element has a label (label)’ (reorder resources)|
|Sources |/ed_source |‘Ensures every form element has a label (label)’ (change owner user selection)|
|Data Explorers|/data_explorer |‘Ensures every form element has a label (label)’ (change owner user selection)|
| | |‘An aria-labelledby or aria-describedby reference exists, but the target for the reference does not exist.’ (data explorer cards)|
|Approval Dashboard

/dashboard/approval/resources

‘Ensures every form element has a label (label)’ (resource approval select checkbox)

Organisation

/organization/<id>

‘Ensures every form element has a label (label)’ (change owner user selection)

/organization/member_new/<id>

‘Ensures every form element has a label (label)’ (coordinator toggle)

/organization/bulk_process/acsfa

‘Ensures every form element has a label (label)’ (Data Profile selection checkbox)

Close buttons on multi-select elements

Collection

/collection/<id>

‘A heading contains no content.’

/collection/documentation/<id>

‘A heading contains no content.’

/collection/edit/<id>

‘A heading contains no content.’

Sources

/ed_source/<id>

‘A heading contains no content.’

/ed_source/edit/<id>

‘A heading contains no content.’

/ed_source/members/<id>

‘A heading contains no content.’

**COMMENTS**
Michael Polidori
3 days ago
So far, all attempts to get screen readers working properly with select2 multiselect have failed. Changing/adding roles, labels, etc., to the currently selected elements doesn’t get picked up for some reason. I’ve reached out to a few other Datopian devs (along with the CKAN community via gitter), so I might have more ideas/examples tomorrow to help make progress here.

Michael Polidori
November 23, 2022 at 4:55 PM
There are additional label fixes needed for accessibility. Specifically, I still need to connect the label for the user defaults toggle, and more importantly, need to programmatically generate labels (and add tabbable navigation) for value boxes within select fields (e.g. tags, relationships, etc.). Currently, tab navigation and screen readers are unable to read or see these selected values.

## OD-1528 508 Compliance Toggle Issues

### Time spent: 2.5h

### Description
Accessibility-related issues are being flagged.

* Ensure that the form label contains text that describes the function of the associated form control. Labels are not required for image, submit, reset, button, or hidden form controls. If a label is not necessarily visually, a descriptive title attribute may be added to the form control.

* Ensure that at most one label element is associated to the form control. If multiple form labels are necessary, use aria-labelledby.

#### List of toggles on the platform:
* View/view (visibility toggle)
* Coordinator toggle
* Default values toggles

### Tasks
Confirm all the places toggles occur

Fix compliance issues

Test with compliance tools

Create 508 Cypress Axe tests if needed

### Notes:
Pages:
- Data profile tabs (description, resources, docs, usage, admin)
- Organizations/new member
- Bulk update data profile, edit data profile (auto fill toggle)
- User edit page

Relevant routes already included in our ed-508.js automated tests

## BUG (Internal work 1)- Data explorers Losing metadata

### Description
Data Explorers. The owner organization isn't placed in the field when you edit, so it gets removed on update, and if there's an image, it breaks on update.

### Tasks
- [x] Replicate the issue
- [x] Fix selected org on initial state
- [x] Fix image

## BUG (Internal work 2)- Sources - unable to add multiple data profiles

### Description
If you add more than one dataprofile to a source all data profiles get removed

### Tasks
- [x] Replicate the issue
- [x] Fix

## OD-1510 - File inputs have duplicated ids, causing screenreaders to read wrong label when input is focused

### Time spent: 3.5h

### Description
This issue was initially found on `/dataset/new_resource/<dataset_id>from_button=`. When navigating on this page with a screen reader, when the data dictionary "Upload" input is focused the screen reader is reading "Resource".
After some analysis, it was found out that because this page has two file upload inputs, and the inputs have the same id, what's being read by the screen reader is the first label that points to this id, which is always "Resource".

<div class="form-group control-full" style="">
    <!-- note this "for" -->
    <label class="control-label" for="field-image-upload">Resource</label>
    <div class="controls ">
        <input id="field-image-upload" type="file" name="upload" value="" placeholder="" accept="" title="Upload a file on your computer" style="width: 91.5312px;"><a href="javascript:;" id="file-upload-button" class="btn btn-default" style=""><i class="fa fa-cloud-upload"></i>Upload</a><a href="javascript:;" class="btn btn-default" title="Link to a URL on the internet (you can also link to an API)" style=""><i class="fa fa-globe"></i>Link</a>
    </div>
</div>

...

<div class="form-group control-full" style="">
    <label class="control-label" for="field-image-upload">Data Dictionary</label>
    <div class="controls ">
        <!-- note the "id" of the input below -->
        <input id="field-image-upload" type="file" name="data_dictionary_res_file" value="" placeholder="" accept="" title="Upload a file on your computer" style="width: 91.5312px;">
        <a href="javascript:;" id="file-upload-button" class="btn btn-default" style=""><i class="fa fa-cloud-upload"></i>Upload</a>
        <a href="javascript:;" class="btn btn-default" title="Link to a URL on the internet (you can also link to an API)" style=""><i class="fa fa-globe"></i>Link</a>
    </div>
 </div>

In order to fix this issue and to follow best practices we have to adapt the file input component to accept different ids.

The fix can be tested by editing the "id" and the "form" of the data dictionary input/label using the browser devtools.

### Tasks
- [x] Replicate the issue
- [x] Fix the issue on the resource page
- [x] Do a global search for "form.image_upload" to find all the places that currently have upload fields
- [ ] Test each of the forms that were found on previous search, using a screen reader to ensure the field is being read correctly

___
## OD-1499 Modal focus order

### Time spent: 6h

### Description
1. Modal focus order
Issue Description: Keyboard focus does not move to the “Copy Data Profile” modal after activation of the button.

URL(s): https://stagedata.lab.ed.gov/dataset/testing#content

**Source Code:**

<div id="copy-data-profile-form">

  <div class="form-popup" id="copy-page-1" style="display: block;">

    <form class="form-container" action="/dataset/new">

      <h1>Copy Data Profile</h1>

 

      <b>Hierarchy (select one):</b><br><br>

 

      <input type="radio" name="copy-relationship-choice" value="STANDALONE" checked="checked"> Standalone (no relationships) <br>

      <input type="radio" name="copy-relationship-choice" value="CHILD_OF"> Child of this Data Profile <br>

      <input type="radio" name="copy-relationship-choice" value="SIBLING_OF"> Sibling of this Data Profile <br><br>

 

      <b>Fields to copy (select multiple—all other fields will be copied by default):</b><br><br>

 

      <input type="checkbox" name="copy-fields-choice" value="DEPENDENCIES"> Dependencies <br>

            <input type="checkbox" name="copy-fields-choice" value="DERIVED_FROM"> Derived from <br>

      <input type="checkbox" name="copy-fields-choice" value="DATA_DICTIONARY"> Data Dictionary <br><br>

      <input type="hidden" name="data" value="42f100a7-789e-4e54-9a74-1dcff5b3e6f5">

     

      <button type="submit" class="btn" id="submit-copy">Copy</button>

      <button type="button" class="btn cancel" onclick="toggleElement('copy-page-1');">Close</button>

    </form>

  </div>

</div>

**Recommendation:** Follow the expected [keyboard interaction and semantics for a modal dialog](https://www.w3.org/WAI/ARIA/apg/patterns/dialogmodal/) as outlined within the ARIA Authoring Practices.

Success Criteria: 2.4.3 Focus Order

Impact: Serious

### TASKS

- [x] See if the same issues are present somewhere else - like the survey
- [x] Give HTML box tab-index
- [x] Alternatively → Aria focus attribute might be needed
- [x] Testing (identification on the 508 platforms)

## OD-1493

### Time spent: 3h

### Description


**Issue Description**: Upload button receives twice keyboard focus.

- On first time keyboard focus button is selectable but there is not visual focus indicator
- On second time keyboard focus links is not selectable and there is a visual focus indicator

URL(s): https://stagedata.lab.ed.gov/dataset/new 

**Source Code**:

<div class="form-group control-full" style="">

    <label class="control-label" for="field-image-upload">Data Dictionary</label>

    <div class="controls ">

    <input id="field-image-upload" type="file" name="data_dictionary_pkg_file" value="" placeholder="" class="form-control" title="Upload a file on your computer" style="width: 91.5312px;"><a href="javascript:;" id="file-upload-button" class="btn btn-default" style=""><i class="fa fa-cloud-upload"></i>Upload</a><a href="javascript:;" class="btn btn-default" title="Link to a URL on the internet (you can also link to an API)" style=""><i class="fa fa-globe"></i>Link</a>

</div>

</div>

**Recommendation**: Use the native input type of file and remove the custom link.

**Success Criteria**: 2.1.1 Keyboard, 2.4.7 Focus Visible

**Impact**: Serious

This needs some investigation: Investigation [2h]

- Look at Upload buttons
- Most upload fields have the same structure
- Can have something to do with labelfor
- Might have to override CKAN upload HTML (code already available in the extension)

## OD-1491 Group labels not programmatically associated
### Time spent: 5h

#### Related ticket: OD=1498
##### Description



**Issue description:** Group labels are not programmatically associated with the radio buttons and check boxes.

**URL(s):** https://stagedata.lab.ed.gov/dataset/new#content

**Source Code:**

<div class="form-group control-full">

    <label class="control-label" for="access_level">Access Level <span data-module="ed_descrp_popover" data-module-field="accessLevel" class="dcat-class" data-original-title="" title="">DCAT</span></label>

    <div class="controls">

      <div class="radio-container">

       

        <div class="radio-group">

          <input type="radio" name="access_level_radios" value="public" id="public_id">

          <label for="public_id" class="custom29">Public</label>

        </div>

       

        <div class="radio-group">

          <input type="radio" name="access_level_radios" value="restricted-public" id="restricted-public_id">

          <label for="restricted-public_id" class="custom29">Restricted Public</label>

        </div>

       

        <div class="radio-group">

          <input type="radio" name="access_level_radios" value="non-public" id="non-public_id">

          <label for="non-public_id" class="custom29">Non-public</label>

        </div>

       

      </div>

      <select id="access_level" name="access_level" class="disable-actions" tabindex="-1">

       

        <option value="public">Public</option>

       

        <option value="restricted-public">Restricted Public</option>

       

        <option value="non-public">Non-public</option>

       

      </select>

**Recommendation:** Associate group label by wrapping the contents in a <fieldset> element, add a <legend> element as the first child of the <fieldset> and group the radio buttons/check boxes together.

**Success Criteria**: 1.3.1 Info and Relationships

Impact: Serious

### TASKS

Associate group label by wrapping the contents in a <fieldset> element, add a <legend> element as the first child of the <fieldset> and group the radio buttons/checkboxes together.

Test with a screen reader

### NOTES:

This will have to be grouped on the following pages

Copy data profiles

resources

Bulk updates

Mostly HTML, so wrapping should be simple

## OD-1464 Reproduce all existing Cypress tests (on Dev) to Stage
### Time spent:

#### Preceding related ticket: OD-1394 QUESTION: Is there any existing automated testing?
##### Description
For the Datopian Team: We believe there is automated testing on Dev. Can you confirm, please, if this is the case? If so, what tests are available and how current are they?

Are there any tests for Stage or Prod?

The purpose of this ticket is simply to garner a current inventory on automated testing.
##### Comment / Reply
Yes, we have integration tests (Cypress) for every feature to verify that they are working.
Tests are run and must pass before code is merged, so tests are always current.

As far as we are aware there is no plan to have integration tests on Staging, what the devs are awaiting, is the approval of Github actions so that no code would be allowed to be merged unless all the tests pass.

In terms of tests in the production environment, that would be entirely up to the DevOps team to decide whether they want to run tests in the live environment.

We do not recommend running Cypress tests on a live environment - but rather only on a clean, test-only instance

We will provide a list of tests (But will need some more information on the exact needs)

### OD-1464 Description
Hello @Osahon Okungbowa and @Susan Botha `In OD-1394: QUESTION: Is there any existing automated testing?` you provided, I believe, a list of all current Cypress tests used in Dev. {I dont think we did}

**For this ticket/ SPIKE, the ODP team would like to learn the expected LOE to enable the same tests on Stage.**

**-Software required
-Workflow changes required**

### TASKS
- [ ] Provide a list of requirements to run the tests (from Datopian to CivicActions)
- [ ] Point the team to the applicable documentation (from Datopian to CivicActions)
- [ ] ED to make sure the software is approved (to run on Stage)

### comments

## OD-1478
### Time spent:
### Description
https://github.com/CivicActions/ckanext-ed/security/dependabot
We want to analyse the security issues identified by the Dependabot and triage them.
This would involve investigating each issue and having some recommended steps to be taken - and the additional tickets that are needed

TASKS


***Linked issues***

relates to **OD-1447:Provide access to the vulnerabilities reported by dependabot on ckanext-ed**

**Description:**

After dependabot was enabled on the ckanext-ed repository it reported 55 vulnerabilities.

**Security Considerations**

each of these vulnerabilities needs to be assessed and remediated as necessary.

## OD-1379 WebInspect Found: 'Often Misused: File Upload' (Vuln ID: 909)(Low)
### Time spent:
### Description

WebInspect Found: 'Often Misused: File Upload' (Vuln ID: 909)

Finding Instance Count: 2

Affected Asset(s):
https://data.ed.gov:443/webassets/base/266988e1_ckan.js
https://stagedata.lab.ed.gov:443/webassets/base/266988e1_ckan.js

Instance Detail:
HTML Status: HTTP/1.1 200 OK

Recommendation:

*For Security Operations:*
This check is part of unknown application testing. Unknown application testing seeks to uncover new vulnerabilities in both custom and commercial software. Because of this, there are no specific patches or descriptions for this issue. If there is no apparent file upload capability on the page, this check may be safely ignored. You can instruct the scanner to ignore this vulnerability by right-clicking the vulnerability node on the displayed results tree and click "Ignore Vulnerability."

*For QA:*
This issue will need to be resolved in the production code. Notify the appropriate developer of this issue.

### Tasks
- [ ] Ensure that the following steps are taken to sanitize the file being received:
- [ ] Limit the types of files that can be uploaded. For instance, on an image upload page, any file other than a .jpg should be refused.
- [ ] Ensure that the web user has no control whatsoever over the name and location of the uploaded file on the server.
- [ ] Never use the name that the user assigns it.
- [ ] Never derive the filename from the web user's username or session ID.
- [ ] Do not place the file in a directory accessible by web users. It is preferable for this location to be outside of the webroot.
- [ ] Ensure that strict permissions are set on both the uploaded file and the directory it is located in.
- [ ] Do not allow execute permissions on uploaded files. If possible, deny all permission for all users but the web application user.
- [ ] Verify that the uploaded file contains appropriate content. For instance, an uploaded JPEG should have a standard JPEG file header.

Threat Description:

The exact implications depend upon the nature of the files an attacker would be able to upload. Implications range from unauthorized content publishing to aid in phising attacks, all the way to full compromise of the web server.

### Comments

>Heidi Harting-Rex
July 5, 2022 at 4:38 PM
>
>Hello @Michael Polidori Can you kindly confirm the following for us, please:
1/ Is our CKAN app a 1:1, namely only one file per dataset?
2/ Do we have a limit on the number of files that can be uploaded?
>
>
>We can work through this list together:
>
>Limit the types of files that can be uploaded. For instance, on an image upload page, any file other than a .jpg should be refused.
:question_mark: to Datopian Team
>
>Ensure that the web user has no control whatsoever over the name and location of the uploaded file on the server.
:question_mark: to Datopian Team
>
>Never use the name that the user assigns it
:question_mark: to Datopian Team
>
>Never derive the filename from the web user's username or session ID
:question_mark: to Datopian Team
>
>Do not place the file in a directory accessible by web users. It is preferable for this location to be outside of the webroot
:question_mark: to Datopian Team
>
>Ensure that strict permissions are set on both the uploaded file and the directory it is located in
:question_mark: to Datopian Team > believed to be N/A due to database
>
>Do not allow execute permissions on uploaded files. If possible, deny all permission for all users but the web application user
:question_mark: to Datopian Team
>
>Verify that the uploaded file contains appropriate content. For instance, an uploaded JPEG should have a standard JPEG file header
:question_mark: to Datopian Team > can images be uploaded?
>
>@Michael Polidori If you could add your input, it would help us. Thank you!

>Michael Polidori
July 6, 2022 at 3:13 PM
>
>@Heidi Harting-Rex Regarding the first 2: there can be any number of files associated with a dataset. There's no limit on how many beyond the server storage size as far as I know.
>
>Some of the others here are most likely not a problem, but I'll have to go through this list tomorrow to confirm.

NOTE: This got pushed out of priority, but we had a meeting a few weeks later and discussed further. The next comment is the output:

>Heidi Harting-Rex
July 28, 2022 at 1:47 PM
>
>Refinement notes: This scan result is in contract to the very basic functionality of the CKAN app. We will need, like in OD-1378: WebInspect Found: 'HTML5: Misconfigured Content Security Policy' (Vuln ID: 11609)TECHNICAL REVIEW, to create a POAM and the Risk Acceptance plan.
>
>What we need to document:
A/ Our workflow is required in order to release the file live
B/ We have virus checkers running on all of the stored files
>
>As per the ODP Team, we do not want to limit the types of files that can be uploaded.
>
>Regarding sanitization of the received file, input as per ODP Team:
>
>No • Limit the types of files that can be uploaded. For instance, on an image upload page, any file other than a .jpg should be refused.
>
>No • Ensure that the web user has no control whatsoever over the name and location of the uploaded file on the server.
>
>No • Never use the name that the user assigns it.
>
>This is already in place • Never derive the filename from the web user's username or session ID.
>
>This is already in place • Do not place the file in a directory accessible by web users. It is preferable for this location to be outside of the webroot.
>
>This is already in place • Ensure that strict permissions are set on both the uploaded file and the directory it is located in.
>
>This is already in place • Do not allow execute permissions on uploaded files. If possible, deny all permission for all users but the web application user.
>
>This is already in place • Verify that the uploaded file contains appropriate content. For instance, an uploaded JPEG should have a standard JPEG file header.
### NOTES:

## OD-1463 WebInspect Found: 'Web Server Misconfiguration: Server Error Message' (Vuln ID: 10932) (LOW)
### Time spent: 4.5h

### Description
Issue found 09/14/22

WebInspect Found: 'Web Server Misconfiguration: Server Error Message' (Vuln ID: 10932)

Finding Instance Count: 6

Affected Asset(s):
https://data.ed.gov:443/user/logged_in?came_from=%00&__logins=0
https://stagedata.lab.ed.gov:443/user/logged_in?came_from=%00&__logins=0

Instance Detail:
HTML Status: HTTP/1.1 500 INTERNAL SERVER ERROR

--------------------------------------------------

Affected Asset(s):
https://data.ed.gov:443/_tracking
https://data.ed.gov:443/login_generic?came_from=/user/logged_in
https://stagedata.lab.ed.gov:443/_tracking
https://stagedata.lab.ed.gov:443/login_generic?came_from=/user/logged_in

Instance Detail:
HTML Status: HTTP/1.1 500 Internal Server Error

***Threat Description:***

The server has issued a 500 error response. While the body content of the error page may not expose any information about the technical error, the fact that an error occurred is confirmed by the 500 status code. Knowing whether certain inputs trigger a server error can aid or inform an attacker of potential vulnerabilities.

***Reccomendations:***

**For Security Operations:**

Server error messages, such as "File Protected Against Access", often reveal more information than intended. For instance, an attacker who receives this message can be relatively certain that file exists, which might give him the information he needs to pursue other leads, or to perform an actual exploit. The following recommendations will help to ensure that a potential attacker is not deriving valuable information from any server error message that is presented. • Uniform Error Codes: Ensure that you are not inadvertently supplying information to an attacker via the use of inconsistent or "conflicting" error messages. For instance, don't reveal unintended information by utilizing error messages such as Access Denied, which will also let an attacker know that the file he seeks actually exists. Have consistent terminology for files and folders that do exist, do not exist, and which have read access denied.

* Informational Error Messages: Ensure that error messages do not reveal too much information. Complete or partial paths, variable and file names, row and column names in tables, and specific database errors should never be revealed to the end user. Remember, an attacker will gather as much information as possible, and then add pieces of seemingly innocuous information together to craft a method of attack.

* Proper Error Handling: Utilize generic error pages and error handling logic to inform end users of potential problems. Do not provide system information or other data that could be utilized by an attacker when orchestrating an attack.

**Removing Detailed Error Messages**

Find instructions for turning off detailed error messaging in IIS at this link:

http://support.microsoft.com/kb/294807

**For Development:**

From a development perspective, the best method of preventing problems from arising from server error messages is to adopt secure programming techniques that prevent problems that might arise from an attacker discovering too much information about the architecture and design of your web application. The following recommendations can be used as a basis for that. • Stringently define the data type (for instance, a string, an alphanumeric character, etc) that the application will accept.

* Use what is good instead of what is bad. Validate input for improper characters.
* Do not display error messages to the end user that provide information (such as table names) that could be utilized in orchestrating an attack.
* Define the allowed set of characters. For instance, if a field is to receive a number, only let that field accept numbers.
* Define the maximum and minimum data lengths for what the application will accept.
* Specify acceptable numeric ranges for input.

**For QA:**
The best course of action for QA associates to take is to ensure that the error handling scheme is consistent. Do you receive a different type of error for a file that does not exist as opposed to a file that does? Are phrases like "Permission Denied" utilized which could reveal the existence of a file to an attacker? Inconsistent methods of dealing with errors gives an attacker a very powerful way of gathering information about your web application.

Reference(s):
**Apache: **
Security Tips for Server Configuration - [Security Tips](https://httpd.apache.org/docs/current/misc/security_tips.html) - Apache HTTP Server Version 2.4
Protecting Confidential Documents at Your Site - [WWW Security FAQ: Running a Secure Server](https://www.w3.org/Security/faq/wwwsf3.html)
Securing Apache - Access Control - [Linux.com Article DB: Securing Apache - Access Control - By Brad Marshall - 1/1 ](http://linux.omnipotent.net/article.php?article_id=3667)

**Microsoft:**
How to set required NTFS permissions and user rights for an IIS 5.0 Web server - [How to set minimum NTFS permissions and user rights for IIS 5.x or IIS 6.0 ](https://support.microsoft.com/en-gb/topic/how-to-set-minimum-ntfs-permissions-and-user-rights-for-iis-5-x-or-iis-6-0-263194ff-bf28-9dc2-c3da-aee109d6cc73)
Default permissions and user rights for IIS 6.0 - http://support.microsoft.com/kb/812614/
Description of Microsoft Internet Information Services (IIS) 5.0 and 6.0 status codes - [HTTP status code overview - Internet Information Services](https://learn.microsoft.com/en-GB/troubleshoot/developer/webapps/iis/www-administration-management/http-status-code)
### Acceptance Criteria
- [ ]

### Tasks
- [ ]

### NOTES:

### Comments:

>Heidi Harting-Rex
>September 22, 2022 at 1:16 PM
>
>Refinement notes: No internal system errors can be given. For these assets:
https://data.ed.gov:443/_tracking
https://data.ed.gov:443/login_generic?came_from=/user/logged_in
https://stagedata.lab.ed.gov:443/_tracking
https://stagedata.lab.ed.gov:443/login_generic?came_from=/user/logged_in
>
>We can customize Apache to return different errors, e.g., 404.

>Alan Moss
October 5, 2022 at 8:27 AM
Edited
>
>@Heidi Harting-Rex @Jeff Schmidt @Pete Cornell
>
>Prod results
https://data.ed.gov:443/_tracking - bogus URL - The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.
https://data.ed.gov:443/login_generic?came_from=/user/logged_in - brings >you to home page with message: “Login failed. Bad username or password.”
>
>Same results in stage
https://stagedata.lab.ed.gov:443/_tracking - bogus URL - The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.
https://stagedata.lab.ed.gov:443/login_generic?came_from=/user/logged_in - brings you to home page with message: “Login failed. Bad username or >password.”
>
>I do not see any information being returned that could be used for an attack. The replies seem reasonable to me.
>
>What would be the suggested response?

>Heidi Harting-Rex
October 6, 2022 at 1:46 PM
>
>Refinement notes: Adjust the errors as follows:
>
>FROM
>
>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.
>
>TO
>
>The requested URL is unavailable.
>
>FROM
>Login failed. Bad username or password.
>
>TO
>
>Login failed.

>Alan Moss
>last week
>
>@Heidi Harting-Rex
>
>in fact it looks like both are coming from code.
>
>I was unable to locate any 40x or 50x errors in the apache logs. So I tried to reach a bogus URL called my404test.html. I did get bac a message >saying:
>
>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.
>
>But when I looked in the logs I saw the request with a “200” (All is good) response from apache.
>
>10.23.12.151 - - [19/Oct/2022:09:41:47 -0400] "POST /_tracking HTTP/1.1" 200 - "https://stagedata.lab.ed.gov/my404test.html" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36 Edg/106.0.1370.34"
So apache is not producing those messages.

>Jeff Schmidt
4 days ago
>
>Looks like this is the case, likely here:
https://github.com/CivicActions/ckanext-ed/blob/master/ckanext/ed/blueprints/user.py#L26

### Related/helpful previous ticket? OD-1206 Remediate: Server Error Message - internal server error, HTTP code 500

### Description

Low priority finding from August 2021 WebInspect scan:

https://data.ed.gov/user/logged_in?__logins=0&came_from=%2Fuser%2Flogged_in%0d%0aSPIHeader:%20SPIValue

returns a internal server error, HTTP code 500.

WebInspect Found: 'HTML5: Misconfigured Content Security Policy' (Vuln ID: 11609)

Finding Instance Count: 2

Affected Asset(s):
https://data.ed.gov:443/
https://stagedata.lab.ed.gov:443/

Instance Detail:
HTML Status: HTTP/1.1 200 OK
Access link / through a proxy and notice the unsafe-inlineand/or unsafe-evalin the CSP header in the response. By default, WebInspect flags only one instance of this vulnerability per host because it is typical to set this header at the host level in a server configuration. Perform the following steps to flag all instances of this issue:

- Create a new policy with the selection of checks that you want to include in a rescan. We recommend using the Blank or Passive policy as a base.
- Select this check and uncheck the “FlagAtHost” check input from standard description.
- Save the policy.
- Rescan with this new custom policy.

### Comments:

>Tom Wood (not a request participant)
December 28, 2021 at 9:22 AM
>
>Hi Susan. If you click on the link in the description, you’ll see that Welcome - Department of Education Open Data Platform returns an internal server error instead of something like a 404 not found. This is considered a security issue (it reveals the fact that our system was not prepared for that input, which an attacker might be able to use to guess other ways in which the system is not prepared or is not validating input.) Does this help? @Susan Botha

>
>Osahon Okungbowa
>December 30, 2021 at 3:52 AM
>Edited
>
>Hi @Tom Wood . We’ve tried to recreate the bug in our test environment, but we couldn’t recreate it.
>
>For us, visit this URL: https://us-ed-testing.ckan.io/user/logged_in?__logins=0&came_from=%2Fuser%2Flogged_in SPIHeader: SPIValue results in either of the following.
>
>If a valid user session is still active, the URL redirects to the Dashboard page because the came_from parameter references an invalid page
>
>if no valid user session exists, the URL redirects to the default login page with the failed login error message displayed (see screenshot)
>
>To investigate this further from our end, we would need to see the error logs for the internal server error generated.
>
>Cc @Alan Moss

>Alan Moss
December 30, 2021 at 10:58 AM
>
>@Osahon Okungbowa @Michael Polidori
I just checked in stage and production, and it does not matter whether I am logged in or not, I consistently receive a 500 error.
>
>So my question is, what version of CKAN and Postgres are you running in your test environment?

>Isaac Aderonmu
January 3, 2022 at 4:28 AM
>
>@Alan Moss
>This has been tested on both ckan 2.8.8 and ckan 2.9.4. it would be helpful if you could provide the ckan error logs from your environment to troubleshoot this further

>Alan Moss
January 4, 2022 at 8:53 AM
Edited
>
>@Isaac Aderonmu
>
>what I have noticed is that this uri works as you have described:
Prod-data-ed-gov/user/login?no_cache=True&came_from=
>
>However this uri causes an Internal System Error everytime:
Prod-data-ed-gov/user/logged_in?__logins=0&came_from=%2Fuser%2Flogged_in%0d%0aSPIHeader:%20SPIValue
>
>Here is what shows up in the prod.ckan.log:
>
>[root@bpvhzlvcka001 httpd]# tail -100 prod.ckan.log | grep ' 500'
>
>10.3.122.228 - - [04/Jan/2022:08:40:17 -0500] "GET /user/logged_in?__logins=0&came_from=%2Fuser%2Flogged_in%0d%0aSPIHeader:%20SPIValue HTTP/1.1" 500 527 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"

>Isaac Aderonmu
>January 7, 2022 at 6:22 AM
>
>@Alan Moss
>
>From analysis of our code base, this is where we're handling the came_from parameter: https://github.com/CivicActions/ckanext-ed/blob/045ae965821e103356158340e81350cde9ea6898/ckanext/ed/controllers/user.py#L12-L14 we are checking that it is a local url here: https://github.com/CivicActions/ckanext-ed/blob/045ae965821e103356158340e81350cde9ea6898/ckanext/ed/controllers/user.py#L13 which is basically just verifying that it starts with /: ckan/helpers.py at ckan-2.8.8 · ckan/ckan It then redirects to the url in the came_from value, which in this case is /user/logged_in%0d%0aSPIHeader:%20SPIValue Going to https://us-ed-testing.ckan.io/user/logged_in%0d%0aSPIHeader:%20SPIValue gives User not found if logged in, and Not authorized to see this page if you're not logged in. There doesn't seem to be any security issue in ckan, and the code in both ckan and our extension hasn’t changed between 2.8 and 2.9. My guess at this point is that your environment might be handling the url a little differently from ours. This is what the url decodes to:
>
>/user/logged_in
SPIHeader: SPIValue
(%0d is a return character). We don't believe the bug is code-related, rather it might be from the load balancer or reverse proxy or whatever is handling the requests before it gets to ckan

>Alan Moss
February 15, 2022 at 9:06 AM
Edited
>
>@Isaac Aderonmu
>
>to my knowledge there is no load balancer nor a proxy in front of our apache process.
>
>Also we do not know how long this issue has existed.
>
>We also have not made any changes to how urls are handled and requests are served in our environment.

>Alan Moss
March 1, 2022 at 10:16 AM
>
>@Osahon Okungbowa
>
>Just tested in prod, received the Internal Server Error and saw this in prod.ckan.log:
>
>
>10.3.122.229 - - [01/Mar/2022:09:44:33 -0500] "GET /user/logged_in?__logins=0&c
ame_from=%2Fuser%2Flogged_in%0d%0aSPIHeader:%20SPIValue HTTP/1.1" 500 527 "-" "
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/9
8.0.4758.102 Safari/537.36"
185.191.171.38 - - [01/Mar/2022:09:44:37 -0500] "GET /dataset?tags=ANNH&tags=FT
E&tags=HEA&tags=HSI&tags=PBI HTTP/1.1" 200 34350 "-" "Mozilla/5.0 (compatible;
SemrushBot/7~bl; +http://www.semrush.com/bot.html)"
185.191.171.19 - - [01/Mar/2022:09:44:49 -0500] "GET /organization/519e6cdf-301
1-4b75-9e12-34a45cb17fd8?tags=advanced-placement&tags=course-taking&tags=school
-counselors&tags=sworn-law-enforcement-officers&tags=teacher-experience HTTP/1.
1" 200 31994 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush
.com/bot.html)"
185.191.171.44 - - [01/Mar/2022:09:45:00 -0500] "GET /organization/519e6cdf-301
1-4b75-9e12-34a45cb17fd8?_tags_limit=0&tags=advanced-placement&tags=special-edu
cation-services&tags=student-chronic-absenteeism&tags=teacher-absenteeism HTTP/
1.1" 200 47980 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semru
sh.com/bot.html)"
52.45.92.94 - - [01/Mar/2022:09:45:02 -0500] "HEAD / HTTP/1.1" 200 - "-" "curl/
7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.44 zlib/1.2.3 libidn/1.18
libssh2/1.4.2""

>Alan Moss
>March 2, 2022 at 4:53 PM
>
>This does still occur in production as well as in stage
>
>Did some co-working today with Isaac and Michael. We discovered that it is a carriage return ‘\r’ embedded in the uri that is causing a parsing error. This error does not happen in the datopian environment, but it does in the DOE environment. Why?
>
>I will be sending some questions to Pivot H about the front end of the env.
>
>If you remove the ‘\r’ represented as ‘%0a’ in the uri, it will not result in a 500 error.

>Jason Heaster
>March 21, 2022 at 9:14 AM
>
>Update: There is a known letsencrypt issue that causes this. Alan has sent comms to PIVOT-H and has not heard back. He will follow up.

>Alan Moss
May 2, 2022 at 9:54 AM
>Edited
>
>@Elizabeth Basirico @Christian Anderson @Heidi Harting-Rex
>
>Wording describing the issue:
>We discovered that it is a carriage return ‘\r’ embedded in the uri that is causing a parsing error. This error does not happen in the datopian >environment, but it does in the DOE environment. Why?
>
>There should be some front end processing to remove such characters that do not belong.
>
>I sent some questions to Pivot H about the front end of the env.
>
>If you remove the ‘\r’ represented as ‘%0a’ in the uri, it will not result in a 500 error.
>
>Just as there are specifications for designing with CSS, HTML, and JavaScript, there are specifications for working with URIs/URLs. The Internet Engineering Task Force (IETF) clearly defines these specifications in RFC 3986: Uniform Resource Identifier (URI): Generic Syntax. Within that document, there are guidelines regarding which characters may be used safely within URIs. This post summarizes the information, and encourages developers to understand and implement accordingly.
>
>

>Elizabeth Basirico
>last week
>
>Hello ODP Team,
>
>I am following up on an action item assigned to PIVOT-H during our meeting on 10/5. The issue is regarding a vulnerability found by WebInspect: “WebInspect Found: 'HTML5: Misconfigured Content Security Policy' (Vuln ID: 11609)”. The CSP was modified after the PaaS team created it. We believe your team is better equipped to modify the header as you know what information is needed for your application to work properly. However, we are happy to make the changes if you provide us with the needed >information.
>
>The PaaS team includes the following header information for all Apache webservers:
>
>Header is always set to Content-Security-Policy "frame-ancestors 'self';"
>
>When we looked at the headers in the stage/prod Apache servers we found the following information (highlighted in red are the unsafe information in >the CSP Header).
>
>Header always set Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline' www.google-analytics.com; font-src 'self' fonts.gstatic.com; style-src 'self' 'unsafe-inline' fonts.googleapis.com; img-src 'self' gravatar.com; frame-ancestors 'self'; >form-action 'self'"
>
>Furthermore, we compared the old Apache Configuration on ODP PROD Server to the current CSP and we noticed the following differences:
>
>

<         Header always set Content-Security-Policy "frame-ancestors 'self'"

---

>       Header always set Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline' dap.digitalgov.gov www.google-analytics.com; font-src 'self' fonts.gstatic.com; style-src 'self' 'unsafe-inline' fonts.googleapis.com; frame-ancestors 'self'; connect-src 'self' www.google-analytics.com; img-src 'self' gravatar.com; form-action 'self'"

### Implementation notes:
https://data.ed.gov:443/user/logged_in?came_from=invalidroute&__logins=0
- Was causing this error:

werkzeug.routing.BuildError
werkzeug.routing.BuildError: Could not build url for endpoint '\x00'. Did you mean 'ed_organization.new' instead?

- It happens when CKAN is trying to build an URL based on the value of the parameter `came_from`. Any value passed that's not a valid route will make the page crash, not only necessarily `%00`.
- Fixed with a CKAN patch.
- Now if the user got to the login page with this parameter set to an invalid route, after succesfully logging in he's just going to be redirected to the default dashboard.
- This patch also fixes it on any page that has a similar behavior (where a redirect happens based on dynamic parameters that can be set by the user)

https://data.ed.gov:443/_tracking
- This page is just outputting a 404
- I tried adding randim parameters to that but still the error didn't occur
- E.g. https://data.ed.gov/_tracking?came_from=%00 (there's a hidden %00 in this link, if you hover your mouse over it it's going to show up)

https://data.ed.gov:443/login_generic?came_from=/user/logged_in
- Same issue as the first URL
- The `came_from` parameter can be manipulated by the users, and the same fix as for the first URL also applies here
- I also noticed that a 500 error would occur when navigating to a user that didn't exist while logged in, e.g.: https://data.ed.gov/user/this_user_doesnt_exist
- Also fixed that

## OD-1323 User Control over Notification Subscriptions

### Time spent:

### Description
*Current state:*

* No control for certain notifications:
> * Creating a Resource (to admins - approval pending)
> * Approval of Resource (to the creator - approval rec’d)
* A checkbox “subscribe to notifications” in user settings

*Desired state:*
* Ability to control whether a notification is rec’d for either of the above situations
* A more granular set of subscribe checkboxes that breaks down possible notifications (ticket separately)

### Acceptance Criteria
- [ ] admin user gets notified when new resources are awaiting approval
- [ ] data stewards get notified when resources are approved or rejected
- [ ] users can successfully opt-in to notifications in the UI

### Tasks
- [x] Go through all the possible email notifications that exist (list)
- [x] Create a new table for the notifications that each user receives
- [x] Handle the storage of the notification values (e.g. new object attached to the user, new table in DB etc)
- [x] How to pass the current state of the notification to the new UI design for the notification
- [x] Screenshots of the UI implementation for review and approval
- [ ] Write tests for this use case


### Time spent: 16.5h + 3.75h

### Notes/Comments
- Regarding the existing “Subscribe to notifications” checkbox, this is enabled/disabled via this ckan.ini env variable:

`ckan.activity_streams_email_notifications = True`

If this variable isn’t present or it’s not set to True, users won’t receive any activity emails and this checkbox won’t be available to enable/disable emails:

ss of the UI:

## OD-1459 Data Profile DCAT Requirement Validator script

### Time spent:

### Description
As an ODP sysadmin, I want to have a script that validates data profiles to determine if they meet the minimum dcat requirement, so that I can easily identify such data profiles and have them as private, draft etc

* Starting point:The Data profile creation point on the platfom
* US standard for Opendata to share metadata on your profile = DCAT standard
* Our implemetation of the DCAT stabdard was very recent
* ODP has their own set of fields
* They now want to unify the DCAT standard and the ODP metadata fields

### Acceptance Criteria
- [ ] A script that identifies already existing legacy dataprofiles where the default/required DCAT fields are absent and need to be updates

### Tasks
- [x] Script to check the ODP fields against DCAT items for all profiles (public and private)
- [x] Set Script variables to run against specific organisations (Argument for single Org or txt file for bulk)
- [x] If a required field is empty set the data profile as draft and hidden

### Notes/Comments
- Spent 9h
- In-depth documentation provided on the scripts docs
- Added ability to export the execution log as a file
- Added the ability to run the script without hidding the data profiles, for preview purposes
- Running the script with `--help` argument provides full usage info

## OD-1430 Data Explorers Reporting 0 Data Profiles Found

Browser, version and device : Edge , Chrome both on ED Laptop

Description : Data Explorers is showing “0 Profiles found” intermittingly

URL : https://stagedata.lab.ed.gov/data_explorer

Steps to reproduce:

Viewed “Data Explorers” tab

Received “0 Data Profiles Found”

Refreshed page; 19 Data Profiles found

Screenshot(s):


**COMMENT: May be related to user traffic, happens intermittently**

### Acceptance Criteria
- [ ] TBC

### Tasks
- [x] Change data profile label to data explorer

### Notes
- Fixed the label
- Was able to see the issue only once after starting up the environment and it went away after refreshing the page. Tried causing the issue by:
- Randomly navigating between pages
- Creating data explorers
- Deleting data explorers
- Logging in and out
- Deleting cookies
- Throttling the internet speed
- Recreating the
- Deleting all the data explorers and importing it again with the seed script
- Tried it in incognito mode
- Tried with the query parameters that appear on the screenshot (http://ckan-dev:5000/data_explorer?q=&sort=&page=1)
- Tried stressing the server/Solr up with: `ab -n 100000 -c 50 ckan-dev:5000/data_explorer`
- Spent ~3.5h

## OD-1429 500 Error - Deleting Data Explorer

Browser, version and device : Edge, ED Laptop

### Description:
When Deleting a Data Explorer; I receive a 500 error. Data Explorer does successfully delete.

URL : https://stagedata.lab.ed.gov/data_explorer/edit/hiking

**Steps to reproduce:**

1. Viewed Data Explorers from “Data Explorers” tab
2. Navigated to Data Explorer I wanted to delete
3. Selected “Manage”
5. Selected “Delete”
7. Confirmed Deletion
9. Received 500 Error
11. Navigated back to check if Data Explorer was deleted successfully; it was

Screenshot(s):
![Uploading file..._4v6srh1od]() <br>

<br>

<br>

<br>

### Acceptance Criteria
- [ ] TBC

### Tasks
- [ ] TBC

### Notes
Spent ~4h