ticket 67

the content has been moved here: github.com/OSMPH/papercut_fix/issues/67

Draft update to the ticket

This ticket is for tracking the roll-back of several changesets which added admin_level relations from a source with an incompatible license.

This is a community initiative to clean-up the data, please add your name if you want to help.

Context

Someone from the mailinglist posted a query about admin boundaries in Alaminos City being dissimilar compared against their knowledge, and was inquiring for more information.

The question triggered a wider review of related edits, and after a re-examination and the contributor's corroboration, it is confirmed that the data source has an incompatible license for addition to OpenStreetmap. The contributor provided us a list of changesets related to these edits and requested for help with rolling back said edits.

Next actions

Simulate roll-back for several changeset using the revert.pl script from osm-revert-scripts. @maning
Share this plan to talk-ph mailing list, OSMPH LoCos telegram and DWG contact email.
Incorporate any comments/suggestions from the community after review of this plan.
Roll-back each changeset starting from the most recent in descending order. With the following changeset comment: Rolled-back because source data license is incompatible with OSM. For details: github.com/OSMPH/papercut_fix/issues/67

List of changeset ids

Post-roll-back manual cleanup.

FAQ

Why was this data uploaded to OpenStreetMap?

The data is available for "free" from various government websites, and states that they are free to use by the public under non-commercial terms. This is a commonly misinterpreted as "OK for importing to OpenStreetMap" by users. As a community, we decided in the past that unless there is explicit permission or a waiver of rights from data owners under such license, we should not add them in OSM.

We wish to emphasize that we found no malicious intent by the contributor, as determined in our private conversations with them. The technical aspect of the edits is good, but the misunderstanding about the license is unfortunate.

Rolling-back the data is a lot of work, why not ask the OSM Data Working Group (DWG) for assistance?

The DWG will be advised of the on-going effort, however, as a commmunity, we prefer this review and clean-up process to be done by the local volunteers. As a local community, we should be able to respond to such incidents, and avoid burdening the DWG unecessarily, especially if issues can be addressed at the local level.

A separate OSM account (OSMPilipinas-PaperCut-Fix) will be used for the roll-back.

Will rolling-back these changesets affect my own contributions?

We did a couple of simulated roll-back, in most cases, there will be a clean roll-back.

However, in the small chance that any other contributions are inadvertently affected, we hope to ensure the data integrity through a manual review of changesets by volunteers after the roll-back operations.

Please use the changeset discussions, to discuss specific issues related to any affected changeset.

2020-07-07

tl;dr We go manual this time.

Preparation

create a spreadsheet for tasking by municipality maning
extract admin relations from March via overpass Erwin
Create a TMS in studio as guide for re-alinging to previous borders ???

Phase 1

Review all admin relations by municipality/city. Manually delete relations and its members that came from incompatable source.

Go to this tracker spreadhseet, pick a municipality/city to work on by adding your name in the User column. Update the status to In progress. Refer to the map, and make sure to select a task with no shared boundary to a task in progress
Open JOSM, make sure JOSM Remote Control is enabled.
Click the JOSM-RC link in your assigned municipality/city. This will download all relations (parent/child) within the municipality/city.
Find and delete barangay boundary relations using the JOSM search filter type:relation admin_level=10.
Check the relation history to ensure that you only delete relation that was originally added by user VMPanes containing incompatble source (Select relation > Ctrl + h).
Image Not Showing Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
v1 of this relation contains incompatible source, this should be deleted.
Find and delete orphaned/untagged ways using JOSM search filter: untagged -child
Review and modify/delete place nodes added from an invalid source. There may be cases where you only need to delete tags, and not the element itself. Common scenarios are the following:

Image	Scenario/Action
Image Not Showing Possible Reasons The image file may be corrupted The server hosting the image is unavailable The image path is incorrect The image format is not supported Learn More →	v1 came from incompatible source. Action: Delete the place node.
Image Not Showing Possible Reasons The image file may be corrupted The server hosting the image is unavailable The image path is incorrect The image format is not supported Learn More →	v8 and later contain incompatible source. Action: Fix tags to retain v7 information, such as: - adding back `is_in` (red); - removing `admin_level`, `admin_type:ph`, `ref` (green); - modify `source` (beige).
Image Not Showing Possible Reasons The image file may be corrupted The server hosting the image is unavailable The image path is incorrect The image format is not supported Learn More → Image Not Showing Possible Reasons The image file may be corrupted The server hosting the image is unavailable The image path is incorrect The image format is not supported Learn More →	Place node moved to a new location. Action: The Coordinates within the History dialog shows the difference of location for each changeset. Move back the location to desired version (v7) by: - dragging the node in the main map or; - use the Move node tool (Tool > Move) by copying the coordinates from v7.

Run JOSM validator and fix any errors related to the boundary relations.
Upload your changes using the changeset comment:
Rolled-back because source data license is incompatible with OSM. For details: github.com/OSMPH/papercut_fix/issues/67
Mark your task in the spreadsheet as complete. Add notes in the spreadsheet, as needed. If you cannot complete the whole municipality, upload the data to OSM anyway. Do not keep updates local to avoid data conflict with other mappers. When you want to start again, always download fresh data via the spreadsheet tracker.

Phase 2

Work on provincial and regional boundaries. Replicate the boundaries using Wikimedia Maps

Select a geographic area from Phase 2 tracker spreadhseet. Make sure that provincial relations has been fixed, before tackling the region relation. Update the status to In progress. Refer to a map, and ensure that you are working on a relation with no shared boundary to a task in progress.
With JOSM open, and Remote Control enabled, click on the "RC for PCs" link to load the whole relation. Avoid using the older link that loads the relation via Overpass Turbo, which may not be a fresh version.
Find and delete nodes and ways wholy added by VMPanes using the following JOSM search filters. Expect relations to break when you do this, which you need to fix afterwards.
- For nodes: type:node version:1 user:VMPanes.
- For ways: type:way user:VMPanes
Use the Wikipedia Maps layer to reconstruct the boundary geometry to the previous version.
Find ways that were part of the original relation before March, and recycle them.
- You will need to extend/split/add/merge these older ways to fix the gaps in the relation, and add new nodes to match the boundaries found on the Wikipedia Map layer.
- Click and watch Ian Lopez's demo on how to "recyle" these ways:
  
  Image Not Showing Possible Reasons
  The image file may be corrupted
  The server hosting the image is unavailable
  The image path is incorrect
  The image format is not supported
  Learn More →
Once the relation is whole again, run validation and fix and resolve any validation issues you may find. Resolve conflicts with elements deleted in your changeset, in favor of your version. Expect other relations to break. Take note of these relations and work on them afterwards.
Upload your changes using the changeset comment:
Replicate pre-March 2020 boundary geometry, due to rolled back elements coming from OSM incompatible data source. May break relations‼ For details, see: github.com/OSMPH/papercut_fix/issues/67
Mark your task in the spreadsheet as Fixed. Add notes in the spreadsheet, as may be needed.

Todo - For QA review

Other notes

rollback first and upload to keep the changeset separate from any additional you may want to add.
extract data from March, create a background layer as guide for JOSM
- OPTIONAL: Use available OSMaPaaaralan data to re-add place nodes.
  - If you do this, or any other edits not related to the rollback, make sure to upload it as different changeset.
get relation id for the municipality you are reviewing and load in JOSM via Overpass

[out:xml]
rel(3488359); // relation id of municipality
(._;>>;);
out meta;

Select all barangay relation.
prepare query for municipality parent/child relations
- we'll work at the muni level, to remove affected barangay relations and/or place nodes.
- make sure to run JOSM validator before uploading
after completion of all muni-level boundaries, proceed with provinces, and then the regional boundary.
do not delete ways with other tags
- JOSM search filter: place=village type:node timestamp:2020-03-01/
- ~~JOSM search filter: type:node admin_level=10~~
  
  Needs refinement. This isn't a clean-cut approach for place nodes that was added by others and later modifed with incompatible data. Needs manual review of history, per element. Erwin
- External: View OSM changes in time

Win Olario

2020/07/08 00:30:36

extract admin relations from March via overpass [name=Erwin] - [ ] Create a TMS in studio as guide for re-alinging to previous borders [nam

I wonder if these are necessary. Wouldn't this issue be resolved by: (1) restore the relation to the clean prior version, (2) load the relation boundary and child relations in JOSM and run the validator to check for relation-related issues. If errors do happen, wouldn't it be more of new elements from child relations not being part of the parent relations? Or am I missing something? (Edited)

2020/07/08 00:38:26

OPTIONAL: Use available OSMaPaaaralan data to re-add place nodes. - If you do this, or any other edits not related to the rollback, make sure to upload it as different changeset.

This is probably better off as an off-shoot challenge from OSMaPaaralan: **find all schools without place nodes within X km radius of the school** (Edited)

2020/07/08 03:51:01

1. Select a geo

I wonder if this has been thought out correctly. A roll-back at the provincial or high level boundaries will affect muni-level shared ways modified in phase 1 🤔 When the relation is loaded to JOSM in step 4, those muni-level edits has to be thoroughly checked. Need to simulate this once a province is ready, before finalizing workflow (Edited)

2020/07/10 07:30:41

untagged -child

Use `type:way -child` instead (Edited)