(2) - HERE Data Hub CLI / API

Introducing HERE CLI、Data Hub API 與 Data Hub Console

What is HERE CLI

CLI stands for Command Line Interface, which means using commands to ask your computer to do something, so it's pure method with texts input, manually or automatically by programming.

Text mode looks like a bit clumsy, but it's eventually more convenient with programming or batch file, so you can run the command in different ways, or even in different timing, criteria or condition, it's more customizable way to do things.

However, HERE CLI needs Node JS environment to run, so please download and install it beforehand: https://nodejs.org/en/download

What is Data Hub API

API stands for Application Programming Interface, looks like hard to understand, but you and imagine your TV as a program, it has power button, audio button, channel button, settings button, HDML port, and last but not least, a IR sensor to receive the signal from remote controller. These things, that receive inputs or deliver outputs, are APIs.

API is a designed for interacting with application software, with specific way. Such as, you have to push power button to turn the power of TV on, and the logic behind the button is not really meaningful for you, you don't need to take care the technical detail to turn on the power, just push the button, that's all. The thing you only know is once the button pushed, the TV will be turned on, maybe you will see some logo in seconds, then the show will go on, the procedure or sequence is designed logic, you don't need to understand them, just be sure to push the correct button.

API not only can be used to acquire results, but also accept input. Such as HDMI port behind the TV is an API of a application (TV). HDMI port can only be plugged with HDMI cable, and once the HDMI cable was well connected, the video signal from the other end (can be DVD player or video game) can be shown on TV if the source fit the specifications of HDMI. There are a bunch of standards, specifications, certifications, programs, protocols, firmware, and copyrights behind the scene, but no need to worry about them, you just need to make sure the cable are connected, and settings are correct, that's it.

API is something that make things easier.

How API works, image source: https://bytenbit.com/best-guidelines-design-restful-api/

If an API is deployed on the internet and opened for specific client, usually you will see HTTP GET/POST/PUT requests:

• Input commands and parameters (like pushing buttons of TV) then get the results (sounds and images), which is like HTTP GET request.
• Input commands, parameters and some contents (like plugging HDMI cable) then get the results (sounds and images), which is like HTTP POST or PUT request.

HTTP requests：https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods

This is why HERE Data Hub is useful, because any map data stored on HERE Data Hub can be accessed with API, which means: any static map file will be enabled for dynamic update, or you can use Data Hub API to build your own map data from scratch. You can do whatever you want with your map data with Data Hub API.

Take the example of Mask Map we just built, you can use Data Hub API to search by locations (such as pharmacies near a place), or by attributes (pharmacies of a city), or statistics, or download your map data and draw it on a map.

HERE CLI also utilizes HERE Data Hub APIs, so you can use simple commands to do things you like. Because the usage of API somehow needs abilities of programming, so it's a bit difficult to experience the potential of what API can do.

In terms of difficulty, CLI is much easier than API.

What is Data Hub Console

Putting things on internet is extremely dangerous! No matter you want to use the data only for yourself, or open to specific users, even open for public, you will need to manager the permissions.

For example, there is a map on the internet, maybe someone are there to maintain it, then the permissions of READ and WRITE need to open for them; if only want to display the data on map, then maybe only READ is just fine to avoid someone has the ability to change or even delete the map data by accident or bad intentions. So the management of permissions is a critical step of providing services, we must make sure the safety of map data to deliver seamless map experience to the end user. HERE Data Hub uses Token to manage the permissions.

HERE offers webpage to manage permissions, which is HERE Data Hub Console. Please open this page:

https://xyz.api.here.com/token-ui/index.html

You need to login to continue, press "Sign In" and finish login it. The first page is "Space Management", you can see the map file "scidm-maskdata_geo.csv" is in it already.

Now enter "Access Management", you can see some "Tokens" in it, because you've used HERE Studio to read the map data, so corresponding token has been generated. If you delete these tokens, the map you just made might not work properly anymore, so please be careful of any changes of tokens, do things only you have 100% confidence.

Besides, you can see the types of token are "TEMPORARY" and "PERMANENT", you can choose between them while generating tokens according to the use cases. Also, please make sure your tokens are well kept, it's harmful for your data if tokens with unnecessary permissions leaked. Once you found a token has leaked, you can delete it then generate a new one, however it might seriously impact existing services and users, which is very critical, you will never want to see it happens. Protect your token.

By the way, the "TEMPORARY" token you saw was generated automatically while login HERE Studio, valid period is 1 day only, so HERE Studio will ask you to login again from time to time.

Generate a Token

First of all, press "Generate Token" on the top-left corner of the page "Access Management".

You can see 2 options, the first is "ALL", second is "scidm-maskdata_geo.csv" we just uploaded. "ALL" means all, the permission will apply to all the "Spaces" under your account, so be most careful with it. You can choose to grant permission to one or more "Spaces". "Space" is map data file you generated or uploaded.

The permissions we can choose are:

1. manageSpaces: manage "Spaces", such as add or delete Spaces.
2. readFeatures: read the Features of Spaces.
3. createFeatures: create new Features in Spaces.
4. deleteFeatures: delete Features in Spaces.
5. updateFeatures: modify/update Features in Spaces.
6. ALL: all permissions.

But how to choose between them? It depends on the usage and the purpose of the space, we can take examples to understand it better:

• You've developed an APP to track packages, and there are GPS trackers on it. You've created a Space on HERE Data Hub to store the trace of GPS trackers, you also write a program on GPS tracker to upload GPS location and timestamp every minute, so the permission of "createFeatures" should be granted to the program, because it just upload data to HERE Data Hub without reading or delete something. So if this token leaked, there will be no risk of losing data (but there could be incorrect or junk data uploaded to your Space).
• This APP of tracking packages will need to connect to HERE Data Hub to read a Space of GPS traces, then draw them on the map, so this APP should have permission of "readFeatures" which only for reading data and no need to change anything a Space. So even this token leaked, there will be no risk of losing data. Maybe the confidential data on it will be visible for others, yet data itself is still safe.
• Maybe the company of delivery want to have functions like Geofencing to know if the packages are delivered, so they will need a Space to store the polygon of geofences, then grant ALL permissions of this space to this company to create new geofences, deleted unwanted and change existing ones.

You can image more user scenarios, HERE Data Hub Console is flexible for you to set different permissions for different purposes. Now we will create a "ALL" permission for "ALL", which is a highest permission to please be cautious. Press "ALL" of "ALL", then all options will be enabled which it affects all permissions of all spaces. Press "Next", it will ask if you want to use paid functions (Add-on), just press "Next".

Finally, you can choose between "TEMPORARY" and "PERMANENT", please choose "PERMANENT". You can type in some notes in "DESCRIPTION" field. "ALLOWED ORIGINS / HOSTS ( ENTER ONE IN A LINE )" is used to limit the token to specific domain or IP address, and we don't need this now, however if you are building mapping services for restricted clients (such as a service only open for specific IP), then you might need it.

Your account and APP ID is at the bottom.

Press "Submit" if you are all set.

Then you can see this "ALL of ALL" token is generated, we will use it in the next course.

Next, please generate a token of reading features only for "scidm-maskdata_geo.csv", we will use it when discussing Data Hub API.

Experience HERE CLI

Install HERE CLI

HERE CLI Website：https://developer.here.com/documentation/studio/map_customization_suite_cli_hlp/dev_guide/index.html

After setting up Node JS environment, open your Command Prompt mode (Windows) or Terminal (Mac OS/Linux), in terms of Windows OS, the fastest way is pressing WIN + R to open command input, input "cmd" then press Enter, you will do this a lot in the future. If you are using Mac OS, press CMD + Space then input "term" and press ENTER/RETURN. Now we take Windows OS as examples.

You will see a windows with black background, the blinking cursor is where you can type texts in, please input the command below, then press ENTER:

npm -v

This step is just to make sure Node JS and npm (package management of Nod JS) are successfully installed. If it shows a series of numbers then that's it.

If everything is fine, please input the command below:

npm install -g @here/cli

npm will automatically download necessary files and install them, it shows added xxx packages from xxx contributors in xx.xxs when finished.

HERE CLI basic commands

Now we need to configure our account with HERE CLI, usually you only need it at the 1st time. Input the command below:

here configure account

After pressing ENTER, it will ask for your account information, enter your email and password (password will be hidden), CLI will login your account and store information, so you won't need to login again.

Now we are ready to go! Because we've uploaded a map file, we can show it with HERE CLI. Input the command below:

here xyz list

You will see a table shown on the screen, with my example, the ID of "scidm-maskdata_geo.csv" is "z4nk5sMV", which is just like we saw with HERE Studio.

This ID (Space ID) is important, every map file you uploaded will have a unique ID, the same data file upload by different user will have different ID, so the ID of yours won't be the same as mine. We can show the content of this Space ID with HERE CLI, input the command below:

here xyz show z4nk5sMV (Change "z4nk5sMV" with your own Space ID)

You can see all the basic attributes of this Space was printed, however it's difficult to read, so please input the command below:

here xyz show -w z4nk5sMV (Change "z4nk5sMV" with your own Space ID)

After the command executed, it will open your browser and display the content of this Space "z4nk5sMV". You can pan and zoom the map, switch map modes and click markers on the map (which is from the map data) to check the attributes. The texts on the right side is the source code of this Space, in GeoJSON format.

Using HERE CLI, we don't need to login Studio to see our map data, the URL of this webpage is also possible to share with others.

Besides simply display them, we can use some condition to filter those we want, for example we can choose only to display attribute of 南投縣埔里鎮衛生所, using this command:

here xyz show -w z4nk5sMV -s "p.醫事機構名稱=南投縣埔里鎮衛生所" (Change "z4nk5sMV" with your own Space ID)

Parameter "-s" means "Search", this command is to search "醫事機構名稱" (name of pharmacy) equals "南投縣埔里鎮衛生所", if the condition is not valid, an empty data will be returned. Make sure adding "p." before the name of attribute.

You can use operators such as = (equal), != (not equal), > (greater), >= (greater than or equal), < (less than), <= (less than or equal). If the type of attribute is string, then only = (equal) and != (not equal) are supported, proximate matching is not supported natively, nor wildcards and Regex.

Thus computing quantity of masks of this Space won't be successful, because you can see all the types of attributes are string (this is a problem with CSV, you can't define the type of each column), even you run the command below to search pharmacies have more then 3000 children masks, all feature will still be returned:

here xyz show -w z4nk5sMV -s "p.兒童口罩剩餘數>3000" (Change "z4nk5sMV" with your own Space ID)

To overcome problem, you will need to convert the type attribute before uploading them, change the type of "兒童口罩剩餘數" (quantity of children mask) and "成人口罩剩餘數" (quantity of adult mask) to integer then it's possible to use operators to calculate between them. But we will ignore this now.

You can delete or clear a space, but thing twice before doing it because it's not reversible, you won't be able to undo if it's done.

• Delete a space：here xyz delete SPACE_ID
• Clear a space：here xyz clear SPACE_ID

Reference of all commands: https://developer.here.com/documentation/studio/map_customization_suite_cli_hlp/dev_guide/topics/commands.html

Use HERE CLI to update map dynamically

There is a trick to dynamically update your map without writing a line of code.

Open browser, and open the webpage we downloaded mask data: https://scidm.nchc.org.tw/en/dataset/nhi-maskdata

Find "scidm-maskdata_geo.csv" and press "Explore" then right click on "Download", choose "Copy link" or "Copy link address".

Then the link of file is already in your clipboard. Now open cmd, input the command below:

cd Desktop

It will switch working directory to your Desktop folder. Input the command below:

curl -o mask_data.csv https://scidm.nchc.org.tw/en/dataset/nhi-maskdata/resource/6116b06b-9c88-4b63-80f9-3f65c7da1370/nchcproxy

This command will download "scidm-maskdata_geo.csv" and save as "mask_data.csv" in your Desktop folder, you can check the content of the file now.

Now we need to upload the fresh file with HERE CLI to replace everything of the Space, but if we just upload it, the existing content won't be replaced but adding new records, which will cause lots of duplication and this is not we desired. We need to clear this Space first before we upload new file.

Run this command in cmd:

here xyz list

Confirm your Space ID, then run this command:

here xyz clear z4nk5sMV (Change "z4nk5sMV" with your own Space ID)

It will ask you to confirm the action, press "y" and press ENTER.

Now the Space "z4nk5sMV" is still there, but nothing in it. We will upload "mask_data.csv" to this Space "z4nk5sMV". Make sure you are under Desktop, then run this command:

here xyz upload z4nk5sMV -f mask_data.csv (Change "z4nk5sMV" with your own Space ID)

The progress will be on the screen, total number of record will be shown after upload is completed.

Now we open the map we just made with HERE Studio, you can see the data is updated to the fresh version of mask data.

So, we have 3 critical commands:

1. Download fresh mask data:
   curl -o mask_data.csv https://scidm.nchc.org.tw/en/dataset/nhi-maskdata/resource/6116b06b-9c88-4b63-80f9-3f65c7da1370/nchcproxy

2. Clear the Space, we can add "–force" to this command to force it proceed without confirmation:
   here xyz clear {SPACE_ID} --force

3. Upload the file we just got:
   here xyz upload {SPACE_ID} -f mask_data.csv

You can now update your mask map by running these 3 command repeatedly.

Experience HERE Data Hub API

Data Hub API Document: https://developer.here.com/documentation/studio/api-reference-swagger.html

Introducing Data Hub API

We can use Data Hub API to show the content of a Space, please open this URL with your browser:

https://xyz.api.here.com/hub/spaces/{SPACE_ID}/iterate?access_token={TOKEN}

If you inputted correct Space ID and Token, you should be able to see something like the image below, it's returned result from Data Hub API with "iterate" API, the function is to return everything of a Space. You will need to wait longer if you have larger data in it. The response is in GeoJSON format.

We can use API to get the statistics of a Space, using "statistics" API:

https://xyz.api.here.com/hub/spaces/{SPACE_ID}/statistics?access_token={TOKEN}

Some browser shows JSON with a beautified format which is easier to read, if your browser doesn't, you can use some free service to do so, such as https://jsonformatter.org/, you can copy and paste all of the texts to the field on the left, the beautified version will show up on the right.

We can use Data Hub API to search "南投縣埔里鎮衛生所" like we just did with HERE CLI, using "search" API:

https://xyz.api.here.com/hub/spaces/{SPACE_ID}/search?p.醫事機構名稱=南投縣埔里鎮衛生所&access_token={TOKEN}

Instead of searching with attribute, we can perform spatial search as well, for example we can search pharmacies near Taipei Main Station (longitude: 121.5170534, latitude: 25.0478554) within 1000 meters, using "spatial" API:

https://xyz.api.here.com/hub/spaces/{SPACE_ID}/spatial?lon=121.5170534&lat=25.0478554&radius=1000&access_token={TOKEN}

We can use Data Hub API to get the content of a specific tile of a map. We usually use X, Y and Z to present specific tile of web map.

Ref: https://en.wikipedia.org/wiki/Tiled_web_map

Such as X = 856, Y = 440, Z = 10 is approximately Shei-Pa National Park: https://a.tile.openstreetmap.org/10/856/440.png

We can see if there is any pharmacy within the boundary of this tile, using "tile" API:

https://xyz.api.here.com/hub/spaces/{SPACE_ID}/tile/web/10_856_440?access_token={TOKEN}

Yes it is! it's "臺中市和平區梨山衛生所".

We will use Data Hub API and Javascript map framework to make a web map in the coming course.