# ticket 67

the content has been moved here: github.com/OSMPH/papercut_fix/issues/67

*Draft update to the ticket*

**This ticket is for tracking the roll-back of several changesets which added admin_level relations from a source with an incompatible license. </p> This is a community initiative to clean-up the data, please add your name if you want to help.**

#### Context

Someone from the mailinglist posted a query about admin boundaries in [Alaminos City being dissimilar](https://lists.openstreetmap.org/pipermail/talk-ph/2020-June/006591.html) compared against their knowledge, and was inquiring for more information.

The question triggered a wider review of related edits, and after a re-examination and the contributor's [corroboration](https://user-images.githubusercontent.com/5821688/86066157-00db5b00-baa4-11ea-953c-d27636575dc4.png), it is confirmed that the data source has an incompatible license for addition to OpenStreetmap. The contributor provided us a list of changesets related to these edits and [requested](https://user-images.githubusercontent.com/5821688/86083276-5f1c3400-bacc-11ea-8227-3e5e3064e451.png) for help with rolling back said edits.

#### Next actions

- [x] Simulate roll-back for several changeset using the `revert.pl` script from [osm-revert-scripts](https://github.com/woodpeck/osm-revert-scripts/). @maning
- [ ] Share this plan to talk-ph mailing list, OSMPH LoCos telegram and DWG contact email.
- [ ] Incorporate any comments/suggestions from the community after review of this plan.
- [ ] Roll-back each changeset starting from the most recent in descending order. With the following changeset comment: `Rolled-back because source data license is incompatible with OSM. For details: github.com/OSMPH/papercut_fix/issues/67`

<details>
<summary>List of changeset ids</summary>

- [ ] 87152111
- [ ] 87100379
- [ ] 87010401
- [ ] 87009673
- [ ] 86987045
- [ ] 86975870
- [ ] 86915142
- [ ] 86913549
- [ ] 86746044
- [ ] 86614562
- [ ] 86614381
- [ ] 86602290
- [ ] 86602275
- [ ] 86494412
- [ ] 86446593
- [ ] 86446461
- [ ] 86303678
- [ ] 86299952
- [ ] 86276331
- [ ] 86216486
- [ ] 86216475
- [ ] 86088389
- [ ] 86057045
- [ ] 85998097
- [ ] 85960420
- [ ] 85929498
- [ ] 85924531
- [ ] 85919675
- [ ] 85813405
- [ ] 85799174
- [ ] 85704886
- [ ] 85700976
- [ ] 85687592
- [ ] 85686941
- [ ] 85677781
- [ ] 85663404
- [ ] 85656925
- [ ] 85621337
- [ ] 85619142
- [ ] 85616252
- [ ] 85612272
- [ ] 85511906
- [ ] 85480318
- [ ] 85475436
- [ ] 85454525
- [ ] 85371078
- [ ] 85338851
- [ ] 85326172
- [ ] 85324918
- [ ] 85324346
- [ ] 85323702
- [ ] 85232023
- [ ] 85154377
- [ ] 85098335
- [ ] 85037190
- [ ] 84972593
- [ ] 84958110
- [ ] 84957660
- [ ] 84957655
- [ ] 84833159
- [ ] 84769060
- [ ] 84726872
- [ ] 84705774
- [ ] 84632966
- [ ] 84611240
- [ ] 84550088
- [ ] 84524802
- [ ] 84491542
- [ ] 84383431
- [ ] 84343990
- [ ] 84282833
- [ ] 84282022
- [ ] 84260671
- [ ] 84260367
- [ ] 84199566
- [ ] 84142073
- [ ] 84130470
- [ ] 84117507
- [ ] 84107804
- [ ] 84068372
- [ ] 84002703
- [ ] 84002450
- [ ] 83989307
- [ ] 83940542
- [ ] 83918481
- [ ] 83918276
- [ ] 83890802
- [ ] 83884845
- [ ] 83850579
- [ ] 83835478
- [ ] 83826009
- [ ] 83825894
- [ ] 83825853
- [ ] 83791033
- [ ] 83779056
- [ ] 83767197
- [ ] 83762210
- [ ] 83759319
- [ ] 83759084
- [ ] 83743953
- [ ] 83740587
- [ ] 83736892
- [ ] 83717222
- [ ] 83697403
- [ ] 83694350
- [ ] 83680621
- [ ] 83679510
- [ ] 83678186
- [ ] 83675202
- [ ] 83661522
- [ ] 83645668
- [ ] 83642426
- [ ] 83640512
- [ ] 83632956
- [ ] 83624611
- [ ] 83622072
- [ ] 83590691
- [ ] 83588571
- [ ] 83587555
- [ ] 83585935
- [ ] 83572156
- [ ] 83569239
- [ ] 83562502
- [ ] 83557328
- [ ] 83547233
- [ ] 83466315
- [ ] 83457554
- [ ] 83451709
- [ ] 83434719
- [ ] 83425075
- [ ] 83424667
- [ ] 83407176
- [ ] 83405124
- [ ] 83393323
- [ ] 83392318
- [ ] 83389843
- [ ] 83363603
- [ ] 83363566
- [ ] 83354829
- [ ] 83354614
- [ ] 83320130
- [ ] 83320103
- [ ] 83261655
- [ ] 83236432
- [ ] 83230462
- [ ] 83224931
- [ ] 83214936
- [ ] 83175311
- [ ] 83173815
- [ ] 83100518
- [ ] 83090941
- [ ] 82879606
- [ ] 82772881
- [ ] 82764007
- [ ] 82759416
- [ ] 82746472
- [ ] 82740833
- [ ] 82739670
- [ ] 82733174
- [ ] 82652855
- [ ] 82652635
- [ ] 82638779
- [ ] 82535887

</details>

- [ ] Post-roll-back manual cleanup.

### FAQ
#### Why was this data uploaded to OpenStreetMap?

The data is available for "free" from various government websites, and states that they are ++free to use by the public under non-commercial terms++. This is a commonly misinterpreted as "OK for importing to OpenStreetMap" by users. As a community, we decided in the past that unless there is explicit permission or a waiver of rights from data owners under such license, we should not add them in OSM.

==We wish to emphasize that we found no malicious intent by the contributor, as determined in our private conversations with them. The technical aspect of the edits is good, but the misunderstanding about the license is unfortunate.==

#### Rolling-back the data is a lot of work, why not ask the OSM Data Working Group (DWG) for assistance?

The DWG will be advised of the on-going effort, however, as a commmunity, we prefer this review and clean-up process to be done by the local volunteers. As a local community, we should be able to respond to such incidents, and avoid burdening the DWG unecessarily, especially if issues can be addressed at the local level.

A separate OSM account (`OSMPilipinas-PaperCut-Fix`) will be used for the roll-back.

#### Will rolling-back these changesets affect my own contributions?

We did a couple of simulated roll-back, in most cases, there will be a clean roll-back.

However, in the small chance that any other contributions are inadvertently affected, we hope to ensure the data integrity through a manual review of changesets by volunteers after the roll-back operations.

Please use the changeset discussions, to discuss specific issues related to any affected changeset.

### 2020-07-07
tl;dr We go manual this time.

### Preparation
- [x] create a spreadsheet for tasking by municipality [name=maning]
- [ ] extract admin relations from March via overpass [name=Erwin]
- [ ] Create a TMS in studio as guide for re-alinging to previous borders [name=???]

#### Phase 1

Review all admin relations by municipality/city. Manually delete relations and its members that came from incompatable source.

1. Go to this **[tracker spreadhseet](https://docs.google.com/spreadsheets/d/1Jxh6Mg6mG07WGY3doy1L5gktue2n7oEAXIOrvT7aFcA/edit#gid=0)**, pick a municipality/city to work on by adding your name in the **User** column. Update the status to **In progress**. Refer to the map, and make sure to select a task with ++no shared boundary++ to a task in progress
2. Open JOSM, make sure [JOSM Remote Control](https://wiki.openstreetmap.org/wiki/JOSM/RemoteControl) is enabled.
3. Click the JOSM-RC link in your assigned municipality/city. This will download all relations (parent/child) within the municipality/city.
4. Find and delete barangay boundary relations using the JOSM search filter `type:relation admin_level=10`.
5. Check the relation history to ensure that you only delete relation that was originally added by user VMPanes containing incompatble source (Select relation > Ctrl + h).
<br>*v1 of this relation contains incompatible source, this should be deleted.*
6. Find and delete orphaned/untagged ways using JOSM search filter: `untagged -child`
7. Review and modify/delete `place` nodes added from an invalid source. There may be cases where you only need to delete tags, and not the element itself. Common scenarios are the following:

|Image|Scenario/Action
---|---
 | v1 came from incompatible source.<br> **Action**: Delete the place node.
 | v8 and later contain incompatible source.<br> **Action**: Fix tags to retain v7 information, such as: <br>- adding back `is_in` (red); <br>- removing `admin_level`, `admin_type:ph`, `ref` (green); <br>- modify `source` (beige).
<img width="400" alt="Screenshot 2020-07-08 10 55 54" src="https://user-images.githubusercontent.com/353700/86870542-2bb05980-c10b-11ea-827a-546b73f4d76f.gif"><br><img width="400" alt="Screenshot 2020-07-08 10 55 54" src="https://user-images.githubusercontent.com/353700/86871055-1ab41800-c10c-11ea-90c5-21c1581e7add.png"> | Place node moved to a new location. <br>**Action**: The **Coordinates** within the History dialog shows the difference of location for each changeset. Move back the location to desired version (v7) by: <br> - dragging the node in the main map or; <br> - use the **Move node** tool (Tool > Move) by copying the coordinates from v7.

8. Run JOSM validator and fix any errors related to the boundary relations.
9. Upload your changes using the changeset comment:
`Rolled-back because source data license is incompatible with OSM. For details: github.com/OSMPH/papercut_fix/issues/67`
11. Mark your task in the spreadsheet as complete. Add notes in the spreadsheet, as needed. If you cannot complete the whole municipality, upload the data to OSM anyway. Do not keep updates local to avoid data conflict with other mappers. **When you want to start again, always download fresh data via the spreadsheet tracker.**

#### Phase 2

Work on provincial and regional boundaries. Replicate the boundaries using Wikimedia Maps

1. Select a geographic area from **[Phase 2 tracker spreadhseet](https://docs.google.com/spreadsheets/d/1Jxh6Mg6mG07WGY3doy1L5gktue2n7oEAXIOrvT7aFcA/edit#gid=255232387)**. Make sure that provincial relations has been fixed, ++before++ tackling the region relation. Update the status to **In progress**. Refer to a map, and ensure that you are working on a relation with **no shared boundary** to a task in progress.
2. With JOSM open, and [Remote Control](https://wiki.openstreetmap.org/wiki/JOSM/RemoteControl) enabled, click on the "RC for PCs" link to load the whole relation. Avoid using the older link that loads the relation via Overpass Turbo, which may not be a fresh version.
3. Find and delete nodes and ways wholy added by VMPanes using the following JOSM search filters. Expect relations to break when you do this, which you need to fix afterwards.
* For nodes: `type:node version:1 user:VMPanes`.
* For ways: `type:way user:VMPanes`
5. Use the Wikipedia Maps layer to reconstruct the boundary geometry to the previous version.
6. Find ways that were part of the original relation before March, and recycle them.
* You will need to extend/split/add/merge these older ways to fix the gaps in the relation, and add new nodes to match the boundaries found on the Wikipedia Map layer.
* Click and watch Ian Lopez's demo on how to "recyle" these ways:
[](https://youtu.be/2JVASEp6qYk)

6. Once the relation is whole again, run validation and fix and resolve any validation issues you may find. Resolve conflicts with elements deleted in your changeset, in favor of **your version**. Expect other relations to break. Take note of these relations and work on them afterwards.
9. Upload your changes using the changeset comment:
`Replicate pre-March 2020 boundary geometry, due to rolled back elements coming from OSM incompatible data source. May break relations‼ For details, see: github.com/OSMPH/papercut_fix/issues/67`
11. Mark your task in the spreadsheet as **Fixed**. Add notes in the spreadsheet, as may be needed.

### Todo - For QA review
- [ ] Resolving conflicts
- [ ] Validation
- [ ] Untagged and orphaned ways - overpass-turbo.eu/s/WAi
- [ ] admin_level ways - overpass-turbo.eu/s/WAj
- [ ] Broken relations
- [ ] Un-restored geometry/elements prior to March edits
- [ ] Osmose issues in AOI
- [ ] [Geometry, high severity](http://osmose.openstreetmap.fr/en/map/#country=philippines&zoom=8&lat=16.783&lon=122.59&item=xxxx&level=1&tags=geom&fixable=)
- [ ] [Boundary, normal to high severity](http://osmose.openstreetmap.fr/en/map/#country=philippines&zoom=8&lat=16.783&lon=122.59&item=xxxx&level=1%2C2&tags=boundary&fixable=)
- [ ] Elements/fragments to review that indicate refutable sources
- [ ] [NOAH, PSA, PSGC, NAMRIA](http://overpass-turbo.eu/s/WAr)

### Other notes
- rollback first and upload to keep the changeset separate from any additional you may want to add.
- extract data from March, create a background layer as guide for JOSM
- OPTIONAL: Use available OSMaPaaaralan data to re-add place nodes.
- If you do this, or any other edits not related to the rollback, make sure to upload it as different changeset.
- get relation id for the municipality you are reviewing and load in JOSM via Overpass

[out:xml]
rel(3488359); // relation id of municipality
(._;>>;);
out meta;

- Select all barangay relation.
- prepare query for municipality parent/child relations
- we'll work at the muni level, to remove affected barangay relations and/or place nodes.
- make sure to run JOSM validator before uploading
- after completion of all muni-level boundaries, proceed with provinces, and then the regional boundary.
- do not delete ways with other tags

- JOSM search filter: `place=village type:node timestamp:2020-03-01/`
- ~~JOSM search filter: `type:node admin_level=10`~~
> :heavy_exclamation_mark: Needs refinement. This isn't a clean-cut approach for place nodes that was added by others and later modifed with incompatible data. Needs manual review of history, per element. [name=Erwin]
- External: [View OSM changes in time](/XfrY334rS7CV0tnPzx8Wvw)