# GRB working notes
###### tags: `projects osm be`
## Documentation
* Landing Page: https://wiki.openstreetmap.org/wiki/AIV_GRB_building_import. Has links to everywhere EXCEPT
* PDF: https://matrix.org/_matrix/media/r0/download/matrix.org/xeWAIiNCPOTKYpkwovRvjaIv (version 3, with notes by Joost integrated by Tim)
* This page
## Procedure
1. Join the Element (formerly: Riot) GRB group
2. Read all the docs
3. Find the URL for the tool in this document
4. Make your first changeset and report back to Element before continuing
5. Learn from the feedback and continue
## Todo
* TOEVOEGEN AAN PDF: Voeg de speciale validatieregels toe voor GRB. Ga naar: JOSM-instellingen → tab met vinkje ("Gegevensvalidator")→ ga naar tweede tabblad "Tag checker rules" (Regels voor tagcontrole) → plus → URL: https://framagit.org/Midgard/osm/raw/master/josm_validation_grb.validator.mapcss
* Automated edit missed buildings in Kaprijke
* add documentation about handling "verdieping"
* add to documentation: gebruik gekleurde straten/gebouwen kaartstijl, en let op bestaande straatnamen
* nakijken A127_GRB https://www.openstreetmap.org/changeset/77998598#map=18/51.13388/4.44960
~~* nakijken ralpha https://www.openstreetmap.org/changeset/71982023~~
* nakijken Hopperpop https://www.openstreetmap.org/user/Hopperpop
* ~~contacteren Ercole Savio: https://www.openstreetmap.org/user/Ercole%20Savio (done: see https://www.openstreetmap.org/changeset/74197325)~~ (done by Seppe)
* document https://framagit.org/Midgard/osm/raw/master/josm_style_grb.mapcss
* add a page for non-importers "what are these weird external id's, what should I do now?", and add a link to it on the project overview. Everyone who does importing should have a link to that page on their osm.org profile.
* Add specific PICC support information to WIKI (Since it doesn't use entity's as grb does, not a lot to explain). ref + date system is compatible with GRB
* Add specific URBIS support information to WIKI . Urbis data is in there too now, but does not support 'date' part in ref + date logic. A version number is present but tool does not support it yet. Consider Urbis alpha level
* In general, expand GRB to a general tool (buttons renamed in grbosm.site to reflect)
## Workflow
Basic flow is:
### survey
Good mapping work starts with proper surveying: know what you're mapping, map as much as possible of what's there, and avoid adding thing that are not (or no longer) there.
> Buildings: Please try to distinguish between house, appartments, garages, sheds, industrial buildings, ... as much as you can, to check or add this info later on when mapping.
House numbers: Often the tool will give multiple buildings with a same adress point, often on touching buildings. Keep an eye out for 'strange' housenumber patterns, or buildings that seemingly lack an adress.
POI's: keep track of businesses, government instances, schools, etc..
### copy data from tool to JOSM
>(1) select an area you want to work in (just a bit wider then the part where you want buildings) and hit the 'open area in JOSM' button.
>(2) zoom in a bit closer untill the buildings appear and hit 'export GRB'
>(3) in JOSM you'll see two layers: one with a very random name, the 'working layer' and one called 'filtered-sourcelayer'. The latter contains the data you can now work with.
### integrate in OSM using JOSM
- Copy data from 'filtered-sourcelayer' to the working layer
> There's basically two approaches, which can both work:
> (A) Delete in the 'filtered-sourcelayer' what you don't need, so you only have left what you actually want to copy.
> (B) Manually select (hand pick) the buildings you want from the 'filtered-sourcelayer' by SHIFT + selecting them.
>Once you have the desired selecting, copy using Ctrl + C, set the working layer to active (the green checkmark) and paste objects by using Ctrl + Alt + V (to keep the exact coördinates)
- attention for multipolygon buildings (!!!)
> Indeed somehow I need:
> (1) to select both building lines
> (2) add the relation to the selection,
> (3) copy
> (4) switch to target layer
> (5) switch back to source layer
> (6) copy again, switch back to target layer
> (7) paste
If I paste after the first switch to target layer, I had cases where the relation was not copied. With this elaborate flow, I know for sure the relation is copied as well.
switching layers is done with Shift-A + number
NOTE: I think the above description is by Escada. Personally I copy both building lines and build the inner/outer relation (with the building tag) afterwards in the working layer.
- Merge with existing building data
> Sometimes you'll end up editing existing buildings, or to at least make it 'fit' to existing buildings next to it. Ideally, we want to keep the 'way' number the original building had (in case the link to it is referenced somewhere externally) but we do want the new geometry (and tags) added there.
> The most convenient way is to use the 'utilsplugin2' info. This allows to:
select both the 'old' and 'new' geometries (using the shift key). The key-combination Ctrl+Shift+G merges all tags (old and new) to the most recent geometry, and give a list where there's possibly conflicting tag choices. Please select carefully which tag you'll proceed with. As a rule of thumb, survey data > old OSM data > tool suggestion.
- check/adapt the building types
> As stated before in the 'survey' section: while the tool makes a 'best possible estimation' on what type of building it is (house, apartment, industrial, garage, shed, ...) the mapper is responsible for the proper pick.
- check the geometries
> FYI, if you compare the geometry from the tool with the GRB background, you will see in some cases that the data from the tool is different.
you can then use aerial imagery and streetview photos (survey / Mapillary / OSC) to decide which is the correct one. I had encountered such a case recently. While we try and keep the source dataset up-to-date (by periodically recalculation it all with 'fresh' GRB data), bear in mind even 'recent' data can be outdate. Ground truth always takes precendence, but in some cases comparison with GRB tiles in a background layer can show differences.
- add surveyed stuff (on or around the buildings)
> Add the POI's you surveyed. Add parking lots, hedges, walls, fences, .. there's very little limit to what you can add: sewer manholes, lighting posts, letterboxes, ...
- adapt landuse/landcover/parkings/... to fit the new building geometries
>To point out the obvious: there's no use in replace a building geometry with a more accurate version, if you don't touch what's directly against it.
-Check the 'fixme=*' tags. For certain cases (like overhanging 'first floors', carports, ... ),which are in the 'GBA' layer in the sourcedata, there's fixme tags explaining you need to verify which case it is, and edit the affected building polygon accordingly.
TIP: type Ctrl + F, and search for 'fixme = *'. If expert mode is enabled in JOSM settings, on the middle left of the search dialog there will be a checkbox to 'add a toolbar button for this search'. In the future, when looking for these 'fixme' tags, all one needs to do is this button.
### upload
* mention GRB source
> Those changesets should not be that different from other changesets. So besides the normal validation step, I simply add "; Buildings via GRB"
> Current scheme is as follows:
source:geometry:ref Gbg/3470483
source:geometry:date 2012-04-18
(with Gbg/Gba/.. refering to the GRB sourcelayer, and OIDN the geometry has within the source data )
> It's a deliberate choice not to add a 'source' tag to the objects itself, we do however add a 'contracted' reference to the object ID's and dates in the source data.
## Stats
* Where have you been working?
TODO: the tool no longer adds a "source:geometry:entity" tag. Older data does not have the new source:geometry:ref tag. Both have source:geometry:date. Therefore, you can find all the buildings using a query like this: http://overpass-turbo.eu/s/HF6
> As an example, here's what happened in Oost-Vlaanderen in 2019: http://overpass-turbo.eu/s/H3f.
> This is just buildings edited since 1/1/2019 with a tag related to the import. So this is not everything that happened and it also includes buildings that are older and happen to have been updated recently.
> This takes the same query but generates a CSV, which helps you identify the big workers: http://overpass-turbo.eu/s/H3g
> Similar to the first, but querys Vlaams-Brabant by it's relation ID: http://overpass-turbo.eu/s/H3n.
>
> Edits since 1/1/2019:
> * West-Vlaanderen: [csv](http://overpass-turbo.eu/s/ZmC), (following links outdated!) [json](http://overpass-turbo.eu/s/H3w)
> * Oost-Vlaanderen: [csv](http://overpass-turbo.eu/s/H3x), [json](http://overpass-turbo.eu/s/H3y)
> * Antwerpen: [csv](http://overpass-turbo.eu/s/H3A), [json](http://overpass-turbo.eu/s/H3z)
> * Vlaams Brabant: [csv](http://overpass-turbo.eu/s/H3t), [json](http://overpass-turbo.eu/s/H3q)
> * Limburg: [csv](http://overpass-turbo.eu/s/H3B), [json](http://overpass-turbo.eu/s/H3r)
* Give me the basic stats ([how?](http://overpass-turbo.eu/s/HFb))
> 1/1/2017: +/- 80.000 objects
> 1/1/2018: +/- 130.000
> 17/3/2019: 151.503
> 4/4/2019: 176.035
> 6/5/2019: 176.905
> 19/6/2019: 196.539
> 4/7/2019: 207.411
> 8/7/2019: 209.031
> 27/7/2019: 214.897
> 5/8/2019: 217.085
> 5/9/2019: 237.603
> 12/9/2019: 247.699
> 4/10/2019: 262.143
> 27/10/2019: 275.409
> 4/11/2019: 279.188
> 3/12/2019: 314.170
> 27/1/2020: 386.767
> 01/05/2020: 481.501
> 29/05/2020: 496.172
> 08/06/2020: 502.487
> 14/07/2020: 564.075
> 11/08/2020: 605.426
> 25/08/2020: 626.971
> 22/09/2020: 665.935
> 24/10/2020: 727.675
> 1/11/2020: 736.281
> 29/11/2020: 771.557
> 21/01/2021: 809.802
> 10/05/2021: 900.255
> 05/09/2021: 944.031
> 10/12/2021: 975.479
> 08/02/2022: 1.064.777 (VL), 44.808 ([WA](https://overpass-turbo.eu/s/1fSo)), 29.175 ([BR](https://overpass-turbo.eu/s/1fSr))
> 08/02/2023: 1.440.266 (VL), 129.247 (WA), 29.766 (BR)
> - [Oost-Vlaanderen](https://overpass-turbo.eu/s/1r8p): 604.410
> - [West-Vlaanderen](https://overpass-turbo.eu/s/1r8r): 388.444
> - [Vlaams-Brabant](https://overpass-turbo.eu/s/1r8s): 174.170
> - [Limburg](https://overpass-turbo.eu/s/1r8t): 203.151
> - [Antwerpen](https://overpass-turbo.eu/s/1r8u): 70.091
> --> Total Flanders: 1.440.266 ways or relations
> - [Wallonia](https://overpass-turbo.eu/s/1fSo): 129.247 ways or relations
> - [Brussels Capital Region](https://overpass-turbo.eu/s/1fSr): 29.766 ways or relations
> --> Total Belgium: 1.599.279 ways or relations
>
>24/11/2023: 1.687.955 (VL), 217.185 (WA), 30.278 (BR)
>
>- [Oost-Vlaanderen](https://overpass-turbo.eu/s/1r8p): 743.379
>- [West-Vlaanderen](https://overpass-turbo.eu/s/1r8r): 426.998
>- [Vlaams-Brabant](https://overpass-turbo.eu/s/1r8s): 211.629
>- [Limburg](https://overpass-turbo.eu/s/1r8t): 214.208
>- [Antwerpen](https://overpass-turbo.eu/s/1r8u): 91.741
>–> Total Flanders: 1.687.955 ways or relations
>- [Wallonia](https://overpass-turbo.eu/s/1fSo): 217.185 ways or relations
>- [Brussels Capital Region](https://overpass-turbo.eu/s/1fSr): 30.278 ways or relations
>–> Total Belgium: 1.935.418 ways or relations
>
> extrapolated from rate second half 2019:
> *1-01-2020: 300.997*
> *1-01-2021: 490.238*
## Monitoring
* [Example OSMCHA filter](https://osmcha.mapbox.com/filters?filters=%7B%22comment%22%3A%5B%7B%22label%22%3A%22grb%22%2C%22value%22%3A%22grb%22%7D%5D%2C%22in_bbox%22%3A%5B%7B%22label%22%3A%222.9508%2C49.9777%2C5.3172%2C51.4755%22%2C%22value%22%3A%222.9508%2C49.9777%2C5.3172%2C51.4755%22%7D%5D%2C%22reasons%22%3A%5B%7B%22label%22%3A%22possible%20import%22%2C%22value%22%3A2%7D%5D%7D) (note that you need to connect your OSM account for it to work)
* All the buildings of a single user (in a certain area) : http://overpass-turbo.eu/s/LlW
* All the recenlty imported buildings that are version 1 and are just a way; return just the centroid to avoid crashing your browser: [link](http://overpass-turbo.eu/?Q=%5Bout%3Ajson%5D%5Btimeout%3A25%5D%3B%0A(%0A%20%20%2F%2F%20query%20part%20for%3A%20%E2%80%9Camenity%3Dtoilet%E2%80%9D%0A%20%20way%5B%22building%22%5D%5B%22source%3Ageometry%3Adate%22%5D(newer%3A%222022-01-01T00%3A00%3A00Z%22)(if%3Aversion()%3C2)(%7B%7Bbbox%7D%7D)%3B%0A)%3B%0Aout%20meta%20center%3B&C=51.04512;4.03224;14)
* [List of "recent" building importers](http://overpass-turbo.eu/?q=W291dDpjc3YodXNlcix0b3RhbCxub2Rlcyx3YXnEmHJlbGF0aW9ucyldW8SibWXEgcSDMTgwXTsKe3tnxKxjxJVlQcSeYTpWxKBhbsSWxJ5ufX0tPi7EimFyY2jEvGVhxLMoCiAgbndyWyJidWlsZGluZyLEqCJzxIHFjmU6xLdvxKt0cnk6ZMShZcWmKMWNxZLFi8WSxY7FkMS9KShuZXfEizoiMjAyMS0xMC0wMVQwMDrGlMaWMFoiKcSzxpwKZm9yIMSIxIpyKCkpe8WWxZcgbWFrZSBzxJB0IMarxqwixInEiyI9Xy52xJEsxrfFl8SUxJZzPcS5dW50xoLEusSmx4LGrMWXxJrEnMeIxIHHiyjHk8ePx4MgxJ7EoMSixKTHh8eJx5fHncShxKPEpSnHkMeRxI7EkGwgPSDHoseMx4XElykgK8ewx5bHjMeZx7XHt8exKMekx5_Hp8SzIAnErcSzfcSz&c=BLlvAq7RDK). Copy the result to a spreadsheet to sort. Note that it is based on last edit, so it could be people updating a lot of buildings, not importing them.
* [Comparison GRB Flanders with OSM (on Mapbox)](https://play.osm.be/historischekaart.html#17/51.16308/4.75393/BasiskaartGRBVL-OSMbuildings)
* [Map of self-declared areas we're working on (umap)](http://umap.openstreetmap.fr/en/map/building-import-status_714473#9/50.6295/4.6225)
* [Map of GRB with blue imported buildings on top, green organicly mapped buildings](https://umap.openstreetmap.fr/nl/map/grb-import-organic_804007#18/50.72356/4.23717) (slow! Triggers a query on each move of the map)
## (F)AQ
* Does the tool always have the latest version of GRB? Should the mapper check for this? What if the current version of GRB contains better data than the GRB version in the tool?
> There's always a chance the tool's data is behind reality or live GRB data. The reason behind it is that the process to generate the tool's database process the entire OSM-buildingdatabase and the entire GRB-building database. It takes alot of time/resources to setup/process the data (6 to 8 hours) which means the effort taken to do this process at a higher frequency doesn't outweigh the limited benefits. The difference is negligible, and the most likely scenario (building demolished since last data update) should be obvious from survey anyway.
> This delay is similar to the update frequency of the CRAB data, where it has never caused issues.
> You can download the data used at https://bitless.be/grb/
* Say a building's geometry has been changed (e.g. part of it has been demolished) and this is not yet reflected in GRB. Wich tags should be changed when changing the geometry (e.g. based on aerial imagery)?
> For completely demolished buildings:
landuse = brownfield if left just like that, 'landuse = construction' + 'construction = *' if it's clearly being used to build something new.
for partially demolished buildings, edit based on a 'best possible estimate' and add a 'fixme = geometry' tag to the remaining building section.
* If you edit the building geometry (due to parts being added or removed), can you keep the GRB-ref tags?
> It's not intrinsically wrong to keep them, as long as there's a way they get flagged for others working/mapping in that area (with the 'fixme' tag). Should there be more recent GRB data at that time to reflect the actual geometry, the tag merging should point out the difference. Should it be needed, can the ref date be used to figure out which is the recent one.
>
* what to do with the GRB-specific tags that were imported, like the ref. I don't know whether this ref will stay the same when GRB is updated to reflect the new situation, so can you just leave the ref as is? Or is it no longer relevant when the geometry is changed?
> Ideally, you would leave a note there, also, you add the source of the information (survey?) and the survey data. These tags would be able to co-exist on an object. We probably need to work this out in detail together. I have a building across me that has been gone for months. still GRB has it in it's data, I can't do anything about that. Ideally you would just remove the GRB building. That was the whole idea on writing this toolset, it's to merge data. The big risk in this is that mappers will start using GRB as the authorative source, while often OSM is more accurate, maybe not in terms of geometries, but when we talk about those cases , construction/demolished , often OSM is boss.
* What to do with 'verdiepingen' when you cannot survey the building? Should you leave them out? Or should you map them as a building part?
> Those are easy, often there is a road running underneath, like in cities they give access to private parking behind the building, there is almost always another object it covers. This one is actually the easiest one. There are also written instructions on how to go about it, in fact, think the wiki has some clues already now. IT will be documented and exampled properly. There are just plain buildings usually. Sometimes building parts can help mapping it properly. It depends on the other objects in close vicinity
* What to do with buildings that were already (or seem to have been) imported from GRB? How can one determine whether the building can be left alone or whether it's geometry and/or tags should be updated?
> at export time you can actually take a difference , that is why I add the source:geometry:* namespace tags , as long as overpass works, you can filter these out before export to JOSM. It could probably be enhanced a bit to support the source date on the object. Or the version number, but we kinda decided the version number is probably not handy, the date is changed everytime the versions of an object changes. So manually now, you can already visually check the data of an object and determine it's updated. Usually , you'll see that this is just a matter of new parts being added and constructed or new 'landmeters' making more accurate measurements, both will visually show what's lacking when you put satellite pics in the background like I do all the time. It's quite noticable
>
* I found a bug in the tool!
>Please post an Issue at https://github.com/gplv2/grbtool/issues
* So where's the tool?
> I thought you'd never ask. Now that you've read everything here it is: https://buildings.osm.be
> But don't forget to respect the procedure mentioned at the very top of this page.
* What is the toolchain that does all this? Can I contribute, where is the source code?
> Glad you asked! Let's break it down.
The site and API itself is written in PHP on the Laravel framework. The visible part in the above mentioned URL , the source code can be found here: https://github.com/gplv2/grbtool in the master branch.
There is quite a large portion of the code in javascript. Laravel provides the API backend at the same time as the frontend website. Angular.js is used to provide authentication / JWT token implementation. Most of the heavy lifting is done by YOUR computer since javascript runs local.
When you follow the steps from the slippy map vector data to the export to JOSM and the filtering is all done in JS.
There is a small postgresql database that handles the site metadata, users, export list etc.
Next to this, there is a seperate postgresql data with postgis extension , this is where the GRB processed data lives in, and this is the database that gets updated whenever we reparse data.
This reparsing is done use gcloud + terraform and that is a fully automated process , the code for that is in the tileserver branch here : https://github.com/gplv2/grb-postgis/tree/tileserver
With the exception of downloading fresh GRB data and importing this in postgis, the process is 100% automated including setting up a google cloud node with terraform. A full processing run costs about 3 dollars if you're interested. Once the data export is present, a manual copy to the live server is done , after this the google computing node can be destroyed again.
There are a few other smaller repo/tools including some I've had to make performance patches for that are required. But unless you're actually want to delve into this deeper, you probably won't be interested. This can all be found in the main script that sets up everything after the google compute node is present. The real bulk of the logic is located in the following script : https://github.com/gplv2/grb-postgis/blob/tileserver/scripts/install.sh
For questions, remarks, idea's or critisism , feel free to contact me in private or use the matrix GRB channel.
# Preparing data update
Current site: https://bitless.be/grb/
To be: similar site that some of us can update. Downloadable via wget/curl
## GRB
### Main dataset
* Download GRB/GBA/KNW per province via download.vlaanderen.be (at this moment entire GRBGis is used, but could possibly work also on the three separate layers per province). A so-called Geosecure account is needed for a download (or an access via a municipality, utility company...)
* Border = 500meter (this will reflect in the filename containing B500)
* DO NOT clip
* Download as shapefile (not GML)
* Keep the original name (keep the date)
* When uploading, keep two versions: the one you just downloaded and the one that was last processed
### 3D data
No update needed as Digitaal Vlaanderen doesn't update this dataset for the moment (i.e. the data is based on the Digital Height Model of Flanders II or DHMV II (Lidar-data acquired between 2013 and 2015) and GRB-CRAB data of spring 2016).
## URBIS
...
Addresses: download the Postgresql dump
## PICC
Permalink would be easiest probably
Download per province
The download contains the addresses from ICAR already
Sign in with Wallet
Connect another wallet