Opendata - HackMD

# Opendata ## datapusher by appending `/etc/ckan/datapusher/datapusher-uwsgi.ini` with: ``` workers = 3 threads = 3 lazy-apps = true ``` the hanging issue of updating resource to datastore got fixed. For details, refer [this](https://github.com/ckan/datapusher#high-availability-setup) ## cdcregistration porting to CKAN 2.9 ### 前情回顧插件 `basiccharts` 目前是涼了，問題出在 Javascript 裡，CKAN error log 以及 browser console 都沒錯誤訊息。統計圖無法在建立時以預覽方式呈現、basic grid 建立時也無法預覽。在看看能不能找出這種 silent bug 吧。 ### cdcregistration HTML 檔 `ckanext/cdcregistration/templates/user/read_base.html` 中使用的 `{% set user = c.user %}` 其中的 `c.user` 為 Pylons 專用的物件，現在必須改為 `g.user`。同份檔案中的 `{{ h.build_nav('user_index', _('Users')) }}` 中的 `'user_index'` 於執行時期跳出找不到這個 menu_ item 的錯誤，原先沒看清楚，以為 `build_nav` 在 2.9 已經沒有了，要自己實做，結果文檔/issue/PR 看了一圈，覺得不妙，因為有看到相關實做，才驚覺其實 `build_nav` 是有實做的，只是 `'user_index'` 這個 menu item (官方名字) 在 2.9 被拿掉了，原因還在等 gitter 回覆。經討論區人員回覆後得知，使用 Flask 時需要用 `user.index` 才對。做完上述修改後接著遇到 `user_datasets` 找不到的問題，嘗試改成 `user.datasets` 沒用，正在詢問討論區是否有相關文檔可參考，因為個人是沒找到，同時詢問了 `user_datasets` 的替代。經討論區人員回覆後得知，此處需使用 `user.read`。文檔的部份官方則表示，因為這是 CKAN internals，所以大部分都修改都不會紀錄到文檔中。此外，官方建議直接去參考對應要修改的 template 檔 (此處為 `templates/user/read_base.html`)，以得知目前版本如何使用相關物件。根據官方建議 (參考對應要修改的 template...)，陸續修改了幾處，十分受用。目前卡在建凱的 cdcmainlib (cdcregistration 的 dep.) 要存取 DB (postgresql) 時會發生錯誤，目前已確定欲存取的欄位是存在的 (擔心版本不一造成欄位名稱有更改/刪除)。錯誤如下： ``` File "/usr/lib/ckan/default/lib/python3.8/site-packages/ckanext/cdcmainlib/helpers.py", line 159, in getUserState return data[0]["state"] IndexError: list index out of range ``` 此處為存取特定使用者的 state，state 表示該使用者的帳號目前是否啟用。其他還有非 state 欄位的存取，均失敗收場，最後都是出現 `IndexError: list index out of range` 錯誤。找問題過程中發現設定檔少加了 `cdcregistration` 需要使用的 config: `ckan.cdcmainlib.psqlUrl = postgresql://ckan_default:THISISPASSWORD@localhost:5432/ckan_default` 原先加入時沒有給 port (因為觀察到設定檔中其他 DB 相關設定 URL 並沒有給 port)，但後來發現 cdcmainlib 中使用 regex 解析從設定檔拿來的字串，其中有假設有給 port，因此將其加上。相關 regex 程式碼片段: ``` pattern = re.compile('\S+://(\S+):(\S+)@(\S+):(\d+)/(\S+)') match = pattern.match(url) ``` #### 4 (week1) 原先於使用者簡介頁面提供的**修改次數**資訊所使用的 ckan internal 已於[此 commit](https://github.com/ckan/ckan/commit/e6401d17806bccbef45e272a3d49c3de83171d5f) 移除： ![](https://i.imgur.com/iOrGP36.png) 除了按照剪開的 README 魔改 ckan internal，其中還至少新增三份魔改 HTML 檔 ## ckan 2.5 -> 2.9 移植工作項目 - [ ] 使用 Py3 - [ ] plugin - [ ] [王建凱插件](https://github.com/jiankaiwang?tab=repositories&q=ckanext&type=&language=) - [ ] [ckanext-cdcmainlib](https://github.com/jiankaiwang/ckanext-cdcmainlib) - [ ] [ckanext-cdcframe](https://github.com/jiankaiwang/ckanext-cdcframe) - [ ] [ckanext-cdcregistration](https://github.com/jiankaiwang/ckanext-cdcregistration) - [ ] [ckanext-cdctondc](https://github.com/jiankaiwang/ckanext-cdctondc) - [ ] [ckanext-cdccushomepage](https://github.com/jiankaiwang/ckanext-cdccushomepage) - [ ] 王建凱客製化的官方插件 - [ ] [ckanext-pages](https://github.com/jiankaiwang/ckanext-pages/commit/a16c79556f0c2f781e2e5a5124d51173d6f35ab7) - [ ] [ckanext-basiccharts](https://github.com/jiankaiwang/ckanext-basiccharts) - [ ] [ckanext-scheming](https://github.com/jiankaiwang/ckanext-scheming) - [ ] [scheming_organizations](https://github.com/jiankaiwang/ckanext-scheming/blob/master/setup.py#L28) - [ ] [scheming_test_subclass](https://github.com/jiankaiwang/ckanext-scheming/blob/master/setup.py#L29) - [ ] [scheming_groups](https://github.com/jiankaiwang/ckanext-scheming/blob/master/setup.py#L27) - [ ] [scheming_datasets](https://github.com/jiankaiwang/ckanext-scheming/blob/master/setup.py#L26) - [ ] [resourceproxy](https://github.com/ckan/ckan/tree/ckan-2.5.7/ckanext/resourceproxy) - [ ] [ckanext-basiccharts](https://github.com/jiankaiwang/ckanext-basiccharts) - [ ] [ckanext-geoview](https://github.com/jiankaiwang/ckanext-geoview) - [ ] [ckanext-download](https://github.com/jiankaiwang/ckanext-download) - [ ] 官方插件 - [ ] [stats](https://github.com/ckan/ckan/tree/5ec941bb108cdae9ff15981ff2d6f329f365fb9b/ckanext/stats) - [ ] [textview](https://github.com/ckan/ckan/tree/5ec941bb108cdae9ff15981ff2d6f329f365fb9b/ckanext/textview) - [ ] [imageview](https://github.com/ckan/ckan/tree/5ec941bb108cdae9ff15981ff2d6f329f365fb9b/ckanext/imageview) - [ ] [reclineview](https://github.com/ckan/ckan/tree/5ec941bb108cdae9ff15981ff2d6f329f365fb9b/ckanext/reclineview) - [ ] [datastore](https://github.com/ckan/ckan/tree/5ec941bb108cdae9ff15981ff2d6f329f365fb9b/ckanext/datastore) - [ ] [datapusher](https://github.com/ckan/ckan/tree/5ec941bb108cdae9ff15981ff2d6f329f365fb9b/ckanext/datapusher) - [x] Solr 升級至版本 X.Y (==指定 v8.6.3 ?==) - [ ] 小心魔改文件 ## 嘗試於 CKAN2.9 w/ Python2 安裝王建凱 plugin 原先以為使用 Python2 的話就只會用到 Pylons (因為大量判斷式的關係)，但從 errorlog 觀察到還是有執行 flask 的痕跡： ``` 2021-03-04 16:10:42,980 ERROR [ckan.config.middleware.flask_app] There is a broken url being created {'action': 'search', 'controller': 'package'} Traceback (most recent call last): File "/usr/lib/ckan/default/lib/python2.7/site-packages/flask/app.py", line 1949, in full_dispatch_request rv = self.dispatch_request() ... ``` ### [cdccushomepage](https://github.com/jiankaiwang/ckanext-cdccushomepage) 安裝完 dependency [cdcmainlib](https://github.com/jiankaiwang/ckanext-cdcmainlib) 後未有 crash。接著安裝[cdccushomepage](https://github.com/jiankaiwang/ckanext-cdccushomepage)，安裝完啟動時 crash，錯誤指出找不到 `customized_promoted.html`： ``` TemplateNotFound: home/snippets/customized_promoted.html ``` 其中 `home/...` 這段路徑是 CKAN 的原始碼中的 `templates/` 目錄下的路徑，看到這邊，大家應該發現了一件事...，==不是王建凱搞魔改就是接手過來後有人魔改了==。為什麼可以確信有魔改發生呢？首先先回到官方 [2.5.9](https://github.com/ckan/ckan/tree/5ec941bb108cdae9ff15981ff2d6f329f365fb9b/ckan/templates/home/snippets)，我們可以看到 `home/snippets` 下並沒有 `customized_promoted.html` 這份 HTML，但進入 willo 提供的機器 (運行 CKAN 2.5.9) 後，我們可以在 CKAN 原始碼目錄下看到該檔案： ```htmlembedded= {% set intro = g.site_intro_text %} <div class="box" style="padding-bottom: 20px;"> <header class="hp-header-bg"> {% if intro %} {{ h.render_markdown(intro) }} {% else %} <h3 class="page-heading module-content"> {{ h.getLangLabel("Latest News","最新消息") }} </h3> {% endif %} </header> {# cdc #} {% block home_image %} {% snippet 'snippets/latest_item.html' %} {% endblock %} </div> ``` 繼續往下，發現還有兩份魔改檔案在同目錄中，`customized_popular.html`, `customized_stats.html`。在直接把魔改檔案 (過程中還有另一份魔改檔案位於 `snippets/latest_item.html`) 放到新機器後，下一個 crash 是因為一支 helper function `get_recent_blog_posts` 沒有定義。在去 willo 的機器 grep 完後，發現其實 `cdccushomepage` 這支 plugin 還有一個==隱藏== dependency，[ckanext-pages](https://github.com/jiankaiwang/ckanext-pages)： ```shell= grep -rn 'get_recent_blog_posts' . Binary file ./ckanext-pages/ckanext/pages/plugin.pyc matches ./ckanext-pages/ckanext/pages/plugin.py:72:def get_recent_blog_posts(number=5, exclude=None): ./ckanext-pages/ckanext/pages/plugin.py:121: 'get_recent_blog_posts': get_recent_blog_posts ./ckanext-pages/ckanext/pages/theme/templates_main/ckanext_pages/blog.html:30:{% set posts = h.get_recent_blog_posts(number=5, exclude=c.page.name) %} ./ckan/ckan/templates/snippets/latest_item.html:1:{% set posts = h.get_recent_blog_posts(number=3) %} ./ckan/ckan/templates/home/snippets/customized_stats.html:36: {% set posts = h.get_recent_blog_posts() %} ``` 驚奇不斷 :smile: 在安裝完 [ckanext-pages](https://github.com/jiankaiwang/ckanext-pages) 後，再次 crash: ``` ERROR [ckan.config.middleware.flask_app] There is a broken url being created {'action': 'search', 'controller': 'package'} Traceback (most recent call last): File "/usr/lib/ckan/default/lib/python2.7/site-packages/flask/app.py", line 1949, in full_dispatch_request . . . File "/usr/lib/ckan/default/src/ckan/ckan/templates/home/snippets/customized_stats.html", line 15, in block "stats_group" <a href="{{ h.url_for(controller='package', action='search') }}"> File "/usr/lib/ckan/default/src/ckan/ckan/lib/helpers.py", line 371, in url_for return _local_url(my_url, locale=locale, **kw) File "/usr/lib/ckan/default/src/ckan/ckan/lib/helpers.py", line 618, in _local_url raise ckan.exceptions.CkanUrlException(error) ``` 從最後 raise exception 的位置猜測，可能是 2.5 -> 2.9 的變動造成，暫時無解。暫時把 `{'action': 'search', 'controller': 'package'}` 修改為 `{'action': 'organization', 'controller': 'dataset'}` 之後 (頁面中出現的超連結會是錯的)，又遇到缺少魔改檔案的問題，缺少 `snippets/popular_data.html`。補上 `snippets/popular_data.html` 後因為其原始碼造成新 crash: ```htmlembedded= {# customized : list 5 popular datasets #} {% for post in range(0,4,1) %} <div class="blog-title dataset-item module-content"> {% snippet 'snippets/package_item.html', package=c.datasets[post], banner=true %} </div> {% endfor %} ``` crash 資訊： ``` UndefinedError: list object has no element 0 ``` 看來是 `c.datasets[post]` 的 `post` 用 0 去索引元素時元素不存在。 :::info jinja 如何做 `catch`？ ::: ![](https://i.imgur.com/QUWr6L5.png) ![](https://i.imgur.com/h3pKFX1.png) ## CKAN plugin 撰寫注意，ckan 的 plugin 讀取方式為從第一個指定的 plugin 開始，讀取到最後一個 plugin。因此，倘若有兩個插件修改了同一份 HTML 檔 (或同樣檔案中同名的 block)，則最後繪製出的頁面會是==較前面==的 plugin 所修改的結果。換句話說，插件 A、插件 B 均修改了 `templates/aaa.html`，並且設定檔 ((`production.ini` 或是 `ckan.ini`)) 所啟用的 plugin 如下: ``` ckan.plugins = A B ``` 則最後呈現的頁面會是插件 A 的結果。當然，一般不會有多個 plugin 修改同樣檔案或 block 的情況發生... ### plugin 安裝很多地方(包含官方文檔)都指示只要在 plugin 的根目錄下執行: ``` python setup.py develop ``` 即可安裝 plugin，但執行此命令後，並未有在對應 `site-package/` 目錄中找到新安裝的 plugin。而是，執行： ``` python setup.py install ``` 才可成功安裝 plugin 至 `site-package/` 目錄下。 :::warning 值得注意的是，執行命令 `$ python setup.py install` 後，只有新的檔案會新增至 `site-package/` 下，而在新修改中==刪除的檔案將不會被從 `site-package/` 中移除==，因此如果有刪除檔案的話要注意到這點。此外...在執行命令 `$ python setup.py install` 後，會有一名為 `build/` 的目錄生成於 plugin 的目錄下，倘若在新修改中有移除檔案，建議直接把這個目錄刪除，讓安裝命令自己重新建立新的，否則也需要如剛才所述，需手動刪除。 python 3 的情況可能有所不同，屆時需再觀察。 ::: :::danger 此段落均假設 shell 已在 virtualenv 環境中，也就是使用命令： ``` . /usr/lib/ckan/default/bin/activate ``` 所進入的環境。 ::: ### 使 CKAN 載入修改過的 plugin #### CKAN 2.5.9 ``` sudo service ckan restart ``` ## 系統升級保險起見(不確定過程中會踩到多深的坑)，將現有 extension 移植至 python3、flask 為基礎的系統預估**至少一個月** ## 2/19 ### 碰到的困難對 Ckan 所使用的程式語言 (Python, HTML...) 的掌握度太低 ### 近期目標了解 Ckan extension 之撰寫，以能夠寫出像樣的 extension，例如：符合「於新增 dataset 時上傳檔案」功能 (要求功能之一) 的 Ckan extension。 ### 目前進度正在「[舉燭](http://dict.revised.moe.edu.tw/cgi-bin/cbdic/gsweb.cgi?ccd=FDKgSP&o=e0&sec1=1&op=sid=%22Z00000158282%22.&v=-2)」[網路上](https://extensions.ckan.org/)關於 Ckan extension 的程式碼，並搭配[官方文檔](https://docs.ckan.org/en/2.9/theming/templates.html#template-helper-functions)解讀。 ## 從頭建構 CDC 網站在 ckan 設定檔中新增 locale 以始頁面預設使用 zh_TW: (不確定是否影響模組行為) ``` ckan.locale_default = zh_TW ckan.locales_offered = en zh_TW ``` ----- ### cdcmainlib `plugin.py`: ```diff= - from helpers import * + from .helpers import * ``` 直接使用 `from helpers` 無法找到同樣目錄下的 `helpers.py`，需用 `.` 表示當前目錄 (多個 `.` 表示向外多移動幾層的目錄)。注意 `helpers.py` 中也需做相關修改。此外，#L4 的 `py2sql` 也需加 `.` `helpers.py`: ```diff= - from pylons import config + from flask import config ``` 看來王建凱時代的插件是針對 pylons 寫的，現在到 flask 可能有些潛在的相容性問題。至此，Ckan 已可於 `cdcmainlib` 開啟時正常啟動。 ### cdccushomepage 直接從 willo 提供之 portal 機將 extension 壓縮後傳送至新機器安裝，其中未來需注意要修正的部份請[見此](https://github.com/jiankaiwang/ckanext-cdccushomepage#preparation)，需修改的部份主要為 domain name 相關。啟動 CKAN 沒問題，但存取首頁時 crash 了： ```= ERROR [ckan.config.middleware.flask_app] Registry initialized. Traceback (most recent call last): File "/usr/lib/ckan/default/lib/python3.8/site-packages/flask/app.py", line 1949, in full_dispatch_request rv = self.dispatch_request() File "/usr/lib/ckan/default/lib/python3.8/site-packages/flask/app.py", line 1935, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/usr/lib/ckan/default/src/ckan/ckan/config/middleware/../../views/home.py", line 67, in index return base.render(u'home/index.html', extra_vars={}) File "/usr/lib/ckan/default/src/ckan/ckan/lib/base.py", line 151, in render return flask_render_template(template_name, **extra_vars) File "/usr/lib/ckan/default/lib/python3.8/site-packages/flask/templating.py", line 137, in render_template return _render( File "/usr/lib/ckan/default/lib/python3.8/site-packages/flask/templating.py", line 120, in _render rv = template.render(context) File "/usr/lib/ckan/default/lib/python3.8/site-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/ckan/default/lib/python3.8/site-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/ckan/default/lib/python3.8/site-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/ckan/default/lib/python3.8/site-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/usr/lib/ckan/default/src/ckan/ckan/templates/home/index.html", line 2, in <module> {% set homepage_style = ( g.homepage_style or '1' ) %} File "/usr/lib/ckan/default/src/ckan/ckan/templates/page.html", line 1, in <module> {% extends "base.html" %} File "/usr/lib/ckan/default/lib/python3.8/site-packages/ckanext/cdccushomepage/templates/base.html", line 1, in <module> {% ckan_extends %} File "/usr/lib/ckan/default/src/ckan/ckan/templates/base.html", line 68, in <module> {%- block styles %} File "/usr/lib/ckan/default/lib/python3.8/site-packages/ckanext/cdccushomepage/templates/base.html", line 25, in <module> {% resource 'cdccushomepage/combined.js' %} File "/usr/lib/ckan/default/src/ckan/ckan/lib/jinja_extensions.py", line 334, in _call h.include_resource(args[0], **kwargs) File "/usr/lib/ckan/default/src/ckan/ckan/lib/helpers.py", line 2049, in include_resource import ckan.lib.fanstatic_resources as fanstatic_resources File "/usr/lib/ckan/default/src/ckan/ckan/lib/fanstatic_resources.py", line 252, in <module> create_library('vendor', os.path.join(base_path, 'vendor'), depend_base=False) File "/usr/lib/ckan/default/src/ckan/ckan/lib/fanstatic_resources.py", line 243, in create_library registry.add(library) File "/usr/lib/ckan/default/lib/python3.8/site-packages/fanstatic/registry.py", line 18, in add self[item.name] = item File "/usr/lib/ckan/default/lib/python3.8/site-packages/fanstatic/registry.py", line 72, in __setitem__ raise ValueError('Registry initialized.') ValueError: Registry initialized. ``` 主因為 ckan.config.middleware.==flask_app==，由 extension ([cdccushomepage](https://github.com/jiankaiwang/ckanext-cdccushomepage)) 的 [templates/base.html](https://github.com/jiankaiwang/ckanext-cdccushomepage/blob/master/ckanext/cdccushomepage/templates/base.html#L25) 引入 [combined.js](https://github.com/jiankaiwang/ckanext-cdccushomepage/blob/master/ckanext/cdccushomepage/fanstatic/combined.js) 時間接 (上述錯誤訊息 L33) 造成。推測此問題為 Flask 取代 Pylons 後造成。或許在新版本環境 (Flask, Ckan2.9) 先了解 jinja 的使用，再去探索王建凱的 cdccushomepage 等 extension 的運作，最終再嘗試移植模組？否則現在連程式碼 (例如： `{% resource 'cdccushomepage/combined.js' %}`) 都看不懂. ## Ckan - extension 撰寫 (以 v2.5.7 為例) ### 介紹 [擴充 Ckan 功能](https://tech.datopian.com/ckan-v3/#ckan-v2-ckan-v3-and-why-v3) Ckan 建構於 [Jinja](https://jinja.palletsprojects.com/en/2.11.x/) 之上，所以開發其 extension 時我們會常用到 Jinja 的 syntax，Jinja 相關 syntax 主要以 `{% <statement> %}` 鑲嵌於 [HTML](https://en.wikipedia.org/wiki/HTML) 中。Jinja 範例可參見 [Wiki](https://en.wikipedia.org/wiki/Jinja_(template_engine)#Example) ### extension 撰寫 #### 行為修改篇首先，我們需要先進入 Ckan 的 [virtualenv](https://virtualenv.pypa.io/en/latest/): ``` $ . /usr/lib/ckan/default/bin/activate ``` 進入開發 extension 的目錄 (通常為 `/usr/lib/ckan/default/src`) 後，執行： ``` $ paster --plugin=ckan create -t ckanext ckanext-iauthfunctions ``` 執行上述命令後，開發者需輸入 extension 相關簡介、開發者名字、信箱等等： ``` Selected and implied templates: ckan#ckanext CKAN extension project template Variables: egg: ckanext_uploadDialog package: ckanextuploaddialog project: ckanext-uploadDialog Enter description (a one-line description of the extension, for example: "A simple blog extension for CKAN") ['']: A extension which uploads data (.csv, .ppt and etc.) with a prompted dialog at dataset creation page Enter author (for example: "Guybrush Threepwood") ['']: foxhoundsk ... ``` 輸入完成後預期可在當前目錄中見到至少以下內容： ``` ckanext-uploadDialog/ ckanext/ __init__.py uploadDialog/ __init__.py plugin.py templates/ ckanext_uploadDialog.egg-info/ setup.py ``` 其中 `ckanext/uploadDialog` 目錄用於存放 extension 的主要程式碼。在 `ckanext/uploadDialog/plugin.py` 中，可見： ```python class UploaddialogPlugin(plugins.SingletonPlugin): plugins.implements(plugins.IConfigurer) ``` 其中 `plugins.implements()` 表示此 extension 修改了 Ckan 哪些功能 (對應功能的程式碼以 interface 為單位，而 interface 中帶有多個 method)，此處 `plugins.IConfigurer` 即為被修改的功能 (`class`)。 :::warning 此處需注意，大部分 interface 都包含多個 method，倘若僅修改某個 interface，則該 interface 的所有 method 均需實做，也就是從原始程式碼複製貼上的概念。除非[v2.9 文件](https://docs.ckan.org/en/2.9/extensions/tutorial.html#id2)漏更新，否則以下屬實： > 雖然在 v2.5 時有 TODO 在討論要省去這多餘的步驟，但截至 v2.9 都還沒有結果。 ::: 以下以修改 authentication 的行為為例，將建立群組的操作修改為只有管理員才有權限執行： ```python= import ckan.plugins as plugins def group_create(context, data_dict=None): return {'success': False, 'msg': 'No one is allowed to create groups'} class ExampleIAuthFunctionsPlugin(plugins.SingletonPlugin): plugins.implements(plugins.IAuthFunctions) def get_auth_functions(self): return {'group_create': group_create} ``` 此處因為 `IAuthFunctions` 只有一個 method，所以我們只看到 `get_auth_functions` 單獨一個 method。此範例改寫 `get_auth_functions` 內，名為 `group_create` 的 function，將其替換為第 4 行名為 `group_create` 的 function。替換過的 `group_create` function 將使得當有使用者建立群組時一律拒絕，除非使用者為管理員 (管理原在 Ckan 中預設略過 authentication 機制) :::danger 其實 `get_auth_functions` 這個 method 還有一個名為 `user_create` 的 function。但範例中沒有對該 function 實做，因此推斷在修改 interface 時，其 method 中並非所有 function 都必須實做，或是説，其實是文件漏更新了，現在連 method 都不用對沒有修改到的 method 做複製貼上的實做。 ::: #### 頁面 (HTML) 修改篇同於 Ckan extension 的目錄中，我們可找到名為 `templates` 的目錄，此目錄將映射至 Ckan 原始程式碼 (通常位於 `/usr/lib/ckan/default/src/ckan/`) 中的 `ckan/templates/` 目錄。換句話說，如果 extension 有成功啟用的話，我們修改過並存放於 extension 中的 HTML, CSS 等檔案會優先被讀取。以 CDC 首頁頁面為例，其使用的 [extension](https://github.com/jiankaiwang/ckanext-cdccushomepage) 名為 `ckanext-cdccushomepage`，我們可以在其 templates 中找到 Ckan 首頁的 [HTML 檔](https://github.com/jiankaiwang/ckanext-cdccushomepage/blob/master/ckanext/cdccushomepage/templates/home/layout1.html)。接下來是 Jinja 施魔法的環節，以下介紹 Jinja 兩個功能： - `{% snippet 'xxx.html' %}` HTML 檔中此 tag 用於將特定區塊的 HTML 以其他 HTML 檔案做替換，可避免替換/開關 extension 時需要修改相關 HTML 檔的必要。範例可參見[此處](https://github.com/jiankaiwang/ckanext-cdccushomepage/blob/master/ckanext/cdccushomepage/templates/home/layout1.html#L81)。 - `{% ckan_extends %}` 此 tag 用於擴充對應的 Ckan 原生檔案，對其做小幅度的修改/新增，可避免小幅度修改就需要複製整份檔案的需要。新增範例可見[此處](https://github.com/davidread/ckanext-downloadall/blob/master/ckanext/downloadall/templates/package/read.html)，可搭配對照其擴充的[原始 HTML 檔](https://github.com/ckan/ckan/blob/master/ckan/templates/package/read.html)。關於修改，可見此[文件](https://docs.ckan.org/en/2.9/theming/templates.html?highlight=jinja#replacing-template-blocks-with-block) #### 啟用 extension 無論是修改行為或是頁面，除非直接修改 Ckan 原始程式碼，否則都必須在首次 (後續修改僅需重啟 Ckan 即可) 經過以下步驟以使修改生效。於 extension 目錄中，執行命令： ``` $ python setup.py install ``` 並且於 Ckan 設定檔 (通常位於 `/etc/ckan/default/production.ini`，較新版本檔名為 `ckan.ini`) 中修改以下設定： ``` ckan.plugins = <extension 名稱> ``` 將 extension 新增至此處後， Ckan 將會於啟動時載入 extension。 :::warning extension 名稱由 extension 根目錄下的 `setup.py` 中的 `ckan.plugins` 的欄位中的被賦值的變數決定，例如有一 `setup.py`，其片段內容為： ```python= entry_points=''' [ckan.plugins] uploadDialog=ckanext.uploadDialog.plugin:UploaddialogPlugin [babel.extractors] ckan = ckan.lib.extract:extract_ckan ''', ``` 第 3 行的 `uploadDialog` 即為 extension 名稱。 ::: 最後重啟 Ckan 即可，依安裝方式不同，重啟方式有： - `$ sudo service ckan restart` - `$ sudo service apache2 reload` - `$ uwsgi --ini-paste /etc/ckan/default/production.ini` - `$ sudo systemctl restart ckan.service` ### 參考資料 - [templating Ckan](https://docs.ckan.org/en/2.5/contributing/frontend/templating.html) - [writing extension](https://docs.ckan.org/en/2.5/extensions/tutorial.html) ## Booting CDC image ### portal Solr has successfully started (manual IP update in the config file is required, the IP is currently GCP's), whereas Ckan hasn't. Ckan error log: ``` Dec 30 00:51:55 cdc-provide-portal systemd[1]: ckan.service: Main process exited, code=exited, status=22/n/a Dec 30 00:51:55 cdc-provide-portal systemd[1]: ckan.service: Failed with result 'exit-code'. Dec 30 00:51:55 cdc-provide-portal systemd[1]: ckan.service: Scheduled restart job, restart counter is at 2. Dec 30 00:51:55 cdc-provide-portal systemd[1]: Stopped Taiwan CDC Open Data. Dec 30 00:51:55 cdc-provide-portal systemd[1]: Started Taiwan CDC Open Data. Dec 30 00:51:55 cdc-provide-portal ckan.sh[1718]: [uWSGI] getting INI configuration from /etc/ckan/default/production.ini ``` ``` Dec 30 00:49:07 cdc-provide-portal postgresql@9.3-main[938]: psql: error: fe_sendauth: no password supplied Dec 30 00:49:07 cdc-provide-portal systemd[1]: Started PostgreSQL Cluster 9.3-main. Dec 30 00:49:07 cdc-provide-portal systemd[1]: Starting PostgreSQL RDBMS... Dec 30 00:49:07 cdc-provide-portal systemd[1]: Finished PostgreSQL RDBMS. Dec 30 00:49:10 cdc-provide-portal google_oslogin_nss_cache[852]: oslogin_cache_refresh[852]: Failure getting groups, quitting Dec 30 00:49:10 cdc-provide-portal google_oslogin_nss_cache[852]: oslogin_cache_refresh[852]: Produced empty group cache file, removing /etc/oslog. Dec 30 00:49:10 cdc-provide-portal systemd[1]: google-oslogin-cache.service: Succeeded. Dec 30 00:49:10 cdc-provide-portal systemd[1]: Finished NSS cache refresh. Dec 30 00:49:11 cdc-provide-portal named[927]: managed-keys-zone: Unable to fetch DNSKEY set '.': timed out Dec 30 00:49:11 cdc-provide-portal named[927]: resolver priming query complete Dec 30 00:49:27 cdc-provide-portal systemd[1]: systemd-fsckd.service: Succeeded. Dec 30 00:50:13 cdc-provide-portal instance-setup: ERROR GET request error retrieving metadata. <urlopen error timed out>. Dec 30 00:50:30 cdc-provide-portal systemd[1]: ckan.service: Main process exited, code=exited, status=22/n/a Dec 30 00:50:30 cdc-provide-portal systemd[1]: ckan.service: Failed with result 'exit-code'. Dec 30 00:50:31 cdc-provide-portal systemd[1]: ckan.service: Scheduled restart job, restart counter is at 1. Dec 30 00:50:31 cdc-provide-portal systemd[1]: Stopped Taiwan CDC Open Data. Dec 30 00:50:31 cdc-provide-portal systemd[1]: Started Taiwan CDC Open Data. Dec 30 00:50:31 cdc-provide-portal ckan.sh[1698]: [uWSGI] getting INI configuration from /etc/ckan/default/production.ini Dec 30 00:51:55 cdc-provide-portal systemd[1]: ckan.service: Main process exited, code=exited, status=22/n/a Dec 30 00:51:55 cdc-provide-portal systemd[1]: ckan.service: Failed with result 'exit-code'. Dec 30 00:51:55 cdc-provide-portal systemd[1]: ckan.service: Scheduled restart job, restart counter is at 2. Dec 30 00:51:55 cdc-provide-portal systemd[1]: Stopped Taiwan CDC Open Data. Dec 30 00:51:55 cdc-provide-portal systemd[1]: Started Taiwan CDC Open Data. Dec 30 00:51:55 cdc-provide-portal ckan.sh[1718]: [uWSGI] getting INI configuration from /etc/ckan/default/production.ini Dec 30 00:52:43 cdc-provide-portal kernel: [ 227.424892] [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:12:34:56:52:55:0a:00:02:02:08:00 SRC=169.254.169.2 Dec 30 00:52:45 cdc-provide-portal kernel: [ 229.145959] [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:12:34:56:52:55:0a:00:02:02:08:00 SRC=169.254.169.2 Dec 30 00:52:57 cdc-provide-portal kernel: [ 241.241917] [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:12:34:56:52:55:0a:00:02:02:08:00 SRC=169.254.169.2 Dec 30 00:52:59 cdc-provide-portal kernel: [ 242.742272] [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:12:34:56:52:55:0a:00:02:02:08:00 SRC=169.254.169.2 Dec 30 00:53:04 cdc-provide-portal kernel: [ 248.451941] [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:12:34:56:52:55:0a:00:02:02:08:00 SRC=169.254.169.2 Dec 30 00:53:12 cdc-provide-portal kernel: [ 256.354990] [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:12:34:56:52:55:0a:00:02:02:08:00 SRC=169.254.169.2 Dec 30 00:53:18 cdc-provide-portal kernel: [ 262.512966] [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:12:34:56:52:55:0a:00:02:02:08:00 SRC=169.254.169.2 ``` and more: ``` Dec 30 00:53:21 cdc-provide-portal systemd[1]: ckan.service: Main process exited, code=exited, status=22/n/a Dec 30 00:53:21 cdc-provide-portal systemd[1]: ckan.service: Failed with result 'exit-code'. Dec 30 00:53:22 cdc-provide-portal systemd[1]: ckan.service: Scheduled restart job, restart counter is at 3. Dec 30 00:53:22 cdc-provide-portal systemd[1]: Stopped Taiwan CDC Open Data. Dec 30 00:53:22 cdc-provide-portal systemd[1]: Started Taiwan CDC Open Data. Dec 30 00:53:22 cdc-provide-portal ckan.sh[1739]: [uWSGI] getting INI configuration from /etc/ckan/default/production.ini Dec 30 00:53:25 cdc-provide-portal kernel: [ 268.897802] [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:12:34:56:52:55:0a:00:02:02:08:00 SRC=169.254.169.2 Dec 30 00:53:27 cdc-provide-portal kernel: [ 270.817146] [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:12:34:56:52:55:0a:00:02:02:08:00 SRC=169.254.169.2 Dec 30 00:53:39 cdc-provide-portal kernel: [ 282.701115] [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:12:34:56:52:55:0a:00:02:02:08:00 SRC=169.254.169.2 Dec 30 00:53:41 cdc-provide-portal kernel: [ 284.749024] [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:12:34:56:52:55:0a:00:02:02:08:00 SRC=169.254.169.2 Dec 30 00:53:47 cdc-provide-portal kernel: [ 291.032823] [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:12:34:56:52:55:0a:00:02:02:08:00 SRC=169.254.169.2 Dec 30 00:53:54 cdc-provide-portal kernel: [ 298.328584] [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:12:34:56:52:55:0a:00:02:02:08:00 SRC=169.254.169.2 Dec 30 00:54:06 cdc-provide-portal kernel: [ 310.509735] [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:12:34:56:52:55:0a:00:02:02:08:00 SRC=169.254.169.2 Dec 30 00:54:27 cdc-provide-portal kernel: [ 331.352047] [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:12:34:56:52:55:0a:00:02:02:08:00 SRC=169.254.169.2 Dec 30 00:54:45 cdc-provide-portal kernel: [ 349.588129] [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:12:34:56:52:55:0a:00:02:02:08:00 SRC=169.254.169.2 Dec 30 00:54:46 cdc-provide-portal systemd[1]: ckan.service: Main process exited, code=exited, status=22/n/a Dec 30 00:54:46 cdc-provide-portal systemd[1]: ckan.service: Failed with result 'exit-code'. Dec 30 00:54:47 cdc-provide-portal systemd[1]: ckan.service: Scheduled restart job, restart counter is at 4. Dec 30 00:54:47 cdc-provide-portal systemd[1]: Stopped Taiwan CDC Open Data. Dec 30 00:54:47 cdc-provide-portal systemd[1]: Started Taiwan CDC Open Data. Dec 30 00:54:47 cdc-provide-portal ckan.sh[1766]: [uWSGI] getting INI configuration from /etc/ckan/default/production.ini Dec 30 00:55:04 cdc-provide-portal kernel: [ 367.691972] [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:12:34:56:52:55:0a:00:02:02:08:00 SRC=169.254.169.2 Dec 30 00:55:28 cdc-provide-portal kernel: [ 392.440466] [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:12:34:56:52:55:0a:00:02:02:08:00 SRC=169.254.169.2 Dec 30 00:55:44 cdc-provide-portal kernel: [ 407.617691] [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:12:34:56:52:55:0a:00:02:02:08:00 SRC=169.254.169.2 ``` ### data Successfully booted, is there any config that need to be done manually? E.g, database IP ## Boot up the VM on GCP Follow the [official doc](https://cloud.google.com/compute/docs/import/import-existing-image) to add existing VM image to the cloud. [Setting up a VM instance](https://cloud.google.com/compute/docs/instances/create-start-instance#creating_a_vm_from_a_custom_image) ### Logging into the VM through SSH If you don't have [GCP guest environment package](https://cloud.google.com/compute/docs/images/install-guest-environment) installed in your VM image, you would need to add you SSH public key manually into the VM prior to boot up the VM on GCP. If you have [GCP guest environment package](https://cloud.google.com/compute/docs/images/install-guest-environment) installed in your VM image, you can connect to the VM with SSH through simply a button click at GCP dashboard. Nevertheless, you can make GCP to perform the setup [during the import](https://cloud.google.com/compute/docs/import/importing-virtual-disks#importing_virtual_disks) automatically. ### Related questions - [RTFM -- SSH public key login](https://stackoverflow.com/questions/65296459/import-existing-ubuntu-cloud-image-to-gcp#comment115455134_65296459) ## Tweak VM images for Solr and Ckan It is no longer needed to have another file (a.k.a snapshot, [referred as backing file in QEMU](https://techpiezo.com/linux/use-and-implementation-of-backing-file-and-snapshot-in-qemu-kvm/)) to co-exist with these two images, that is, they are both now standalone image. [Improved image usage document](https://docs.google.com/document/d/1drdzuN_ilbW7OKRoxSLWi2bCEwgqcgzy4M-49-gxEbQ/edit) ## Dockerize (Solr 6.6.5, Ckan 2.9.1 currently) After setting up the environment (docker and docker-compose), follow the instructions [here](https://docs.ckan.org/en/2.9/maintaining/installing/install-from-docker-compose.html). Since docker flavored Ckan is [currently](https://github.com/ckan/ckan/issues/5572#issuecomment-685453697) in the process of a complete overhaul, hence performing docker installation instruction against Ckan 2.9.1 would eventually fail to start the service. [Here](https://github.com/ckan/ckan/pull/5381) is the fix which has already been [merged](https://github.com/ckan/ckan/commit/9abeaa1b7d2f6539ade946cc3f407878f49950eb) and is expected to be rolled out at 2.9.2. ### Result ``` ❯ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS S d0ddf660defa docker_ckan "/ckan-entrypoint.sh…" 5 seconds ago Up 4 seconds 0.0.0.0:5000->500n 2c999949a53f docker_solr "docker-entrypoint.s…" 19 minutes ago Up 19 minutes 8983/tcp r 2bb65832962b redis:latest "docker-entrypoint.s…" 19 minutes ago Up 19 minutes 6379/tcp s 7359d4df756b docker_db "docker-entrypoint.s…" 19 minutes ago Up 19 minutes (healthy) 5432/tcp b 47a2b1462201 clementmouchet/datapusher "python datapusher/m…" 19 minutes ago Up 19 minutes 0.0.0.0:8800->880r ``` PS: Seems that if we want to separate nginx from Ckan, it is required to do some modifications to the source, because the official doesn't separate the installation of nginx even in [installation from source](https://docs.ckan.org/en/2.9/maintaining/installing/install-from-source.html). ==PPS: What version of Solr are they using for now?== ## Ckan ### `site_url` should be: (it is not necessary to specify the port) ``` ckan.site_url = http://140.121.198.214 ``` ### Import data to Ckan Prior to perform the import, an organization must exist, ## Misc ### Host machine port forwarding (for QEMU) Simply add the following options to the startup command: ``` -net user,hostfwd=tcp::<host_port>-:<guest_port> -net nic ``` where `<host_port>` is the port to be forwarded from the host machine, whereas `<guest_port>` is the port at guest machine to receive the traffic. Repeat `hostfwd...` with `,` as delimiter if you have multiple ports require forwarding. ## Solr ### Fresh install (Solr v7.4 specific) #### ckan.ini 設定確認 CKAN 設定檔 `/etc/ckan/default/ckan.ini` 中 `solr_url` 欄位設定正確，預期應為： ``` solr_url = http://localhost:8983/solr/ckan ``` 或是： ``` solr_url = http://10.10.0.1:8983/solr/ckan ``` 其中 `10.10.0.1` 應改為 Solr 主機的對外 IP 位置。倘若單純在本機端測試，則使用 localhost 也無妨，因為在 CKAN 網站中操作時，localhost 將直接導向執行 Solr 的機器。 #### Solr schema 設定 (假設直接在 solr 解壓縮後的目錄操作) 進入執行檔資料夾 `bin/` 後，執行以下命令： ``` $ ./solr start ``` 啟動後，執行以下命令建立 CKAN 所需之 core： ``` $ ./solr create -c ckan ``` :::danger 倘若是安裝 Solr 後才建立 core，請使用以下命令建立 core: ``` sudo su - solr -c "/opt/solr/bin/solr create -c ckan" ``` 其中 `-c ckan` 為建立一名為 ckan 的 core、`su - solr -c` 為切換為使用者 Solr，再執行雙引號內的命令 (由於建 core 所需寫入的目錄的擁有者為 Solr，因此需要切換使用者) ::: 建立成功後，預期可在 `../server/solr/ckan/conf/` (`bin/` 目錄的相對路徑) 目錄中看到 Solr 預設給 CKAN 使用的 schema,`managed-schema`，這邊先使用以下命令將其備份： ``` $ mv ../server/solr/ckan/conf/managed-schema ../server/solr/ckan/conf/managed-schema.bak ``` :::danger 倘若是使用已經安裝完成的 Solr 建立 core，則建立完的 core 預期會在 `/var/solr/data/` 這個目錄下。此外，安裝完的 Solr 會位於 ` /opt/solr-<版本號>` 目錄中。 ::: 接著將在該資料夾建立專為 CKAN2.9--Solr7.4 調整過的 schema： ``` $ gedit ../server/solr/ckan/conf/managed-schema ``` 將[此處](#Solr-74---Ckan-29)提供的 schema 完整複製貼上並存檔。接著我們需要編輯此 core 的設定檔，首先開啟同樣目錄下的 `solrconfig.xml` 檔案： ``` $ gedit ../server/solr/ckan/conf/solrconfig.xml ``` 在檔案中尋找 `update.autoCreateFields`，將其設定 `true` 改為 `false`: ```diff=1192 - <updateRequestProcessorChain name="add-unknown-fields-to-the-schema" default="${update.autoCreateFields:false}" + <updateRequestProcessorChain name="add-unknown-fields-to-the-schema" default="${update.autoCreateFields:true}" ``` 修改完成後存檔退出。重新啟動 Solr: ``` $ ./solr restart ``` 至此 CKAN 理當可以開始使用。 :fire_engine: ##### 備註 - Solr log 位置： `/var/solr/logs/solr.log` and `/var/solr/logs/solr-8983-console.log`, or directory under Solr's source `server/logs/` ### plugin installation #### mmseg4j This plugin has a Solr specific version, which can be located [here](https://github.com/chenlb/mmseg4j-solr/tree/dependabot/maven/solr.version-8.4.1) (note that the master branch cannot compile with maven currently ([commit 8f281b](https://github.com/chenlb/mmseg4j-solr/commit/8f281b128369c02d5c690c1c9bf9f216a7450309))) master branch compilation failure: ``` [ERROR] Failed to execute goal on project mmseg4j-solr: Could not resolve dependencies for project com.chenlb.mmseg4j:mmseg4j-solr:jar:2.4.0: Coul] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project mmseg4j-solr: Could not resolve dependencies for projectr at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:269) at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147) . . . Caused by: org.apache.maven.project.DependencyResolutionException: Could not resolve dependencies for project com.chenlb.mmseg4j:mmseg4j-solr:jar:r at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve (DefaultProjectDependenciesResolver.java:209) at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:243) ``` Turns out the other [branch](https://github.com/chenlb/mmseg4j-solr/tree/dependabot/maven/solr.version-8.4.1) is also not compile-able with maven: ``` [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.3.1:compile (default-compile) on project mmseg4j-solr: [ERROR] /home/ubuntu/mmseg4j-solr/src/main/java/com/chenlb/mmseg4j/analysis/TokenUtils.java:[5,33] error: cannot find symbol . . . /home/ubuntu/mmseg4j-solr/src/main/java/com/chenlb/mmseg4j/analysis/TokenUtils.java:[5,33] error: cannot find symbol ``` However, `mmseg4j-core-1.10.0.jar` and `mmseg4j-solr-2.3.0.jar`, which are provided from the Google drive, can both successfully installed into Solr 7.4. ### 7.4 [workable schema](https://hackmd.io/6gveBw5OR2Sgmz1DtiCAbg?view#Solr-74---Ckan-29) (paired with Ckan 2.9) Except [this](https://stackoverflow.com/questions/48533541/can-ckan-support-solr-7-0/51063193#51063193), section [here](https://lucene.apache.org/solr/guide/7_4/field-type-definitions-and-properties.html#field-type-definitions-in-schema-xml) should be added to the schema. Nevertheless, and the following: (note that some of these may not be required, I just add them all, cause the procedure is annoying) ```xml <fieldType name="pdate" class="solr.DatePointField" docValues="true"/> <fieldType name="pdates" class="solr.DatePointField" docValues="true" multiValued="true"/> <fieldType name="plongs" class="solr.LongPointField" docValues="true" multiValued="true"/> <fieldType name="pdoubles" class="solr.DoublePointField" docValues="true" multiValued="true"/> ``` Then the modification for solrconfig.xml which mentioned in the first paragraph [here](https://stackoverflow.com/a/31721587/8559609). Alternatively, you can change its field `update.autoCreateFields` from true to false: > Remove the AddSchemaFieldsUpdateProcessorFactory section from the updateRequestProcessorChain config in your solrconfig.xml In case you encounter other schema related problems, [here](https://www.mail-archive.com/dev@nutch.apache.org/msg26272.html) is additional reference. ### Known supported Solr-CKAN pair - [Solr 7.4 -- CKAN 2.8](https://stackoverflow.com/a/51063193/8559609) - [Solr 7.2.1 -- CKAN 2.7.5 (from official Gitter)](https://gitter.im/ckan/chat) > Hello CKAN/SOLR experts! > > I am looking for advice on how to improve search performance on our [BC Data Catalogue](https://catalogue.data.gov.bc.ca). We are using CKAN 2.7.5 and the SOLR 7.2.1 version. > > We are not defining a specific query parser in our solrconfig.xml file; from what I've read this means that the Lucene parser will be used. In our solrconfig.xml file we've defined the default search field to be "text". > > ``` > <requestHandler name="/select" class="solr.SearchHandler"> >  > <lst name="defaults"> > <str name="echoParams">explicit</str> > <int name="rows">10</int> > <str name="df">text</str> > <str name="q.op">AND</str> > </lst> > </requestHandler> > ``` > > Our schema,xml file is the one that's packaged with CKAN 2.7.5: > https://github.com/ckan/ckan/blob/2.7/ckan/config/solr/schema.xml > > I'm wondering then why the following 3 CKAN API URL's and 1 SOLR API URL's do not return the same records: > > https://catalogue.data.gov.bc.ca/api/3/action/package_search?q=text:data%20innovation%20program --> __23__ matches > https://catalogue.data.gov.bc.ca/api/3/action/package_search?q=data%20innovation%20program --> __855__ matches > https://catalogue.data.gov.bc.ca/api/3/action/package_search?q=data%20innovation%20program&defType=lucene --> __23__ matches > > http://apps.bcgov:8093/solr/ckan-m/select?q=data%20innovation%20program --> 23 matches > > Curious as well about whether anyone has experiences with any of the following that they could share: > > - using different query parsers (e.g., edismax) > - in schema.xml, setting store=false for fields that don't appear in the search results page > - in schema.xml, setting index=false for fields that will never be searched on > - in schema.xml, removing fields that are not used > - in schema.xml, using the Stop filter and stopwords.txt > - Using a newer SOLR version > > What's the most recent SOLR version compatible with CKAN 2.9 > > Many thanks! > Mike Dunham-Wilkie ### Copy ckan-specific schema to Solr core directory Solr will fail to initialize the core without modifications within the schema file. #### Modification - Remove deprecated configs, e.g. `solr.TrieIntField`, `solr.TrieFloatField`, `solr.SynonymFilterFactory`, etc. - Unknown data type error, e.g. `int`, `float`, `date`, etc. - `coreFirst: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Could not load conf for core coreFirst: Can't load schema /home/ubuntu/opendata/solr/solr-8.6.3/server/solr/coreFirst/conf/schema.xml: Setting defaultSearchField in schema not supported since Solr 7` - `coreFirst: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Could not load conf for core coreFirst: Can't load schema /home/ubuntu/opendata/solr/solr-8.6.3/server/solr/coreFirst/conf/schema.xml: Setting default operator in schema (solrQueryParser/@defaultOperator) not supported` - After fixing the error above and restarting Solr, the schema got renamed to `schema.xml.bak`, and the default schema (a.k.a `managed-schema`) is generated automatically. The following error is triggered by using default schema: `coreFirst: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: fieldType 'text_general' not found in the schema` ### Cautions :warning: - Even Solr authentication plugin is enabled, it's not recommended to expose Solr to the outside world ### Create new core with authentication enabled ([doc](https://lucene.apache.org/solr/guide/8_6/basic-authentication-plugin.html#using-the-solr-control-script-with-basic-auth)) With authentication enabled, we can't directly create a new core. Therefore, you can add the admin username/password into the file named `solr.in.sh`, then you can go create new core as if there is no authentication required. Configs required to add to `solr.in.sh` are: ``` SOLR_AUTH_TYPE="basic" SOLR_AUTHENTICATION_OPTS="-Dbasicauth=username:password" ``` ### Account management ([doc](https://lucene.apache.org/solr/guide/8_6/basic-authentication-plugin.html#basic-authentication-plugin)) #### Add new account / Change password - Through API Simply use this command: ``` curl --user opendata:SolrRocks \ https://localhost:8983/api/cluster/security/authentication \ -H 'Content-type:application/json' \ -d '{"set-user": {"opendata":"505ntouLab"}}' --http0.9 -k ``` where: - `--user opendata:SolrRocks` stands for username:password, which should be the credential for user who intend to login and make the modification - `https://localhost...` stands for [the entry URL](https://lucene.apache.org/solr/guide/8_6/basic-authentication-plugin.html#api-entry-point) - `-H 'Content-type:application/json'` stands for extra header to include in the HTTP request - `-d '{"set-user": {"opendata":"505ntouLab"}}'` stands for the data in the HTTP request - `--http0.9` stands for accept non-http-stuff response from the server - `-k` stands for accept self-signed SSL certificate What the curl command output is: ``` { "responseHeader":{ "status":0, "QTime":4} } ``` Actually, you can modify the config file `security.json` ([located at $SOLR_HOME directory](https://lucene.apache.org/solr/guide/8_6/taking-solr-to-production.html#solr-home-directory)) directly, but, this way you need to generate the key hash on your own ==Note that same command above can also be used for adding new account.== ### Enable SSL for Solr ([doc](https://lucene.apache.org/solr/guide/8_6/enabling-ssl.html)) First, create a self-signed certificate: ``` $ keytool -genkeypair -alias solr-ssl -keyalg RSA -keysize 2048 -keypass secret \ -storepass secret -validity 9999 -keystore solr-ssl.keystore.p12 -storetype PKCS12 \ -ext SAN=DNS:localhost,IP:140.121.198.214,IP:127.0.0.1 -dname "CN=localhost, \ OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country" ``` Modify `solr.in.sh` (located at the same directory with Solr main executable `solr`, i.e. `bin/`) to enable SSL feature for Solr: (these options are commented out by default) ``` # Enables HTTPS. It is implicitly true if you set SOLR_SSL_KEY_STORE. Use this config # to enable https module with custom jetty configuration. SOLR_SSL_ENABLED=true # Uncomment to set SSL-related system properties # Be sure to update the paths to the correct keystore for your environment SOLR_SSL_KEY_STORE=etc/solr-ssl.keystore.p12 SOLR_SSL_KEY_STORE_PASSWORD=secret SOLR_SSL_TRUST_STORE=etc/solr-ssl.keystore.p12 SOLR_SSL_TRUST_STORE_PASSWORD=secret # Require clients to authenticate SOLR_SSL_NEED_CLIENT_AUTH=false # Enable clients to authenticate (but not require) SOLR_SSL_WANT_CLIENT_AUTH=false # SSL Certificates contain host/ip "peer name" information that is validated by default. Setting # this to false can be useful to disable these checks when re-using a certificate on many hosts SOLR_SSL_CHECK_PEER_NAME=true ``` ### solrconfig.xml ([doc](https://lucene.apache.org/solr/guide/8_6/configuring-solrconfig-xml.html)) A per-core conf. file which can be located under `conf/` with core directory as root directory. This file has the most parameters affecting Solr itself. Some important parameters are: - Request handlers Request processing, e.g. requests to add documents to the index or requests to return results for a query - Listeners Listen for particular query-related events, can be used to trigger the execution of special code, e.g. invoking some common queries to warm-up caches - Request dispatcher HTTP communication management - Admin web interface For plugin related conf., detailed introduction can be found [here](https://lucene.apache.org/solr/guide/8_6/libs.html#lib-directives-in-solrconfig). ### Solr plugin installation ([doc](https://lucene.apache.org/solr/guide/8_6/libs.html#lib-directories)) Say you have the plugin prepared (packaged to `.jar` file), ### Solr installation log #### [Install from package](https://docs.ckan.org/en/2.9/maintaining/installing/install-from-package.html) ##### Package - solr-tomcat ![](https://i.imgur.com/XcQRqgp.png) ###### Result ![](https://i.imgur.com/O6PX3S5.png) ![](https://i.imgur.com/8QPxTQX.png) #### Install from source (v8.6.3) ##### Creation of the core - command: ``` sudo su - solr -c "/opt/solr/bin/solr create -c your_collection -n data_driven_schema_configs" ``` successfully created a new core named `your_collection`: ``` Created new core 'your_collection' ``` Same command except `-V` (i.e. verbose mode) option added: ``` Copying configuration to new core instance directory: /var/solr/data/test_collection Creating new core 'test_collection' using command: http://localhost:8983/solr/admin/cores?action=CREATE&name=test_collection&instanceDir=test_collection { "responseHeader":{ "status":0, "QTime":288}, "core":"test_collection"} ``` Same command as above except `-n` option not provided: ``` $ sudo su - solr -c "/opt/solr/bin/solr create -c test_collection_second -V" WARNING: Using _default configset with data driven schema functionality. NOT RECOMMENDED for production use. To turn off: bin/solr config -c test_collection_second -p 8983 -action set-user-property -property update.autoCreateFields -value false Copying configuration to new core instance directory: /var/solr/data/test_collection_second Creating new core 'test_collection_second' using command: http://localhost:8983/solr/admin/cores?action=CREATE&name=test_collection_second&instanceDir=test_collection_second { "responseHeader":{ "status":0, "QTime":250}, "core":"test_collection_second"} ``` ==Seems that using arbitrary configuration filename (i.e. `-n aswad`) __won't__ incur any warnings, but a warning is incured without specifying the conf. filename.== ## Solr 7.4 - Ckan 2.9 Following is the schema used on this pairing: ```xml= <?xml version="1.0" encoding="UTF-8" ?>    <schema name="ckan" version="2.9"> <types> <fieldType name="pdoubles" class="solr.DoublePointField" docValues="true" multiValued="true"/> <fieldType name="plongs" class="solr.LongPointField" docValues="true" multiValued="true"/> <fieldType name="pdate" class="solr.DatePointField" docValues="true"/> <fieldType name="pdates" class="solr.DatePointField" docValues="true" multiValued="true"/> <fieldType name="text_general" class="solr.TextField" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />  <filter class="solr.LowerCaseFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" /> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/> <filter class="solr.LowerCaseFilterFactory"/> </analyzer> </fieldType> <fieldType name="string" class="solr.StrField" sortMissingLast="true" omitNorms="true"/> <fieldType name="boolean" class="solr.BoolField" sortMissingLast="true" omitNorms="true"/> <fieldtype name="binary" class="solr.BinaryField"/> <fieldType name="int" class="solr.TrieIntField" precisionStep="0" omitNorms="true" positionIncrementGap="0"/> <fieldType name="float" class="solr.TrieFloatField" precisionStep="0" omitNorms="true" positionIncrementGap="0"/> <fieldType name="long" class="solr.TrieLongField" precisionStep="0" omitNorms="true" positionIncrementGap="0"/> <fieldType name="double" class="solr.TrieDoubleField" precisionStep="0" omitNorms="true" positionIncrementGap="0"/> <fieldType name="tint" class="solr.TrieIntField" precisionStep="8" omitNorms="true" positionIncrementGap="0"/> <fieldType name="tfloat" class="solr.TrieFloatField" precisionStep="8" omitNorms="true" positionIncrementGap="0"/> <fieldType name="tlong" class="solr.TrieLongField" precisionStep="8" omitNorms="true" positionIncrementGap="0"/> <fieldType name="tdouble" class="solr.TrieDoubleField" precisionStep="8" omitNorms="true" positionIncrementGap="0"/> <fieldType name="date" class="solr.TrieDateField" omitNorms="true" precisionStep="0" positionIncrementGap="0"/> <fieldType name="tdate" class="solr.TrieDateField" omitNorms="true" precisionStep="6" positionIncrementGap="0"/> <fieldType name="tdates" class="solr.TrieDateField" precisionStep="7" positionIncrementGap="0" multiValued="true"/> <fieldType name="booleans" class="solr.BoolField" sortMissingLast="true" multiValued="true"/> <fieldType name="tints" class="solr.TrieIntField" precisionStep="8" positionIncrementGap="0" multiValued="true"/> <fieldType name="tfloats" class="solr.TrieFloatField" precisionStep="8" positionIncrementGap="0" multiValued="true"/> <fieldType name="tlongs" class="solr.TrieLongField" precisionStep="8" positionIncrementGap="0" multiValued="true"/> <fieldType name="tdoubles" class="solr.TrieDoubleField" precisionStep="8" positionIncrementGap="0" multiValued="true"/> <fieldType name="text" class="solr.TextField" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/> <filter class="solr.ASCIIFoldingFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/> <filter class="solr.ASCIIFoldingFilterFactory"/> </analyzer> </fieldType>  <fieldType name="textgen" class="solr.TextField" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="0"/> <filter class="solr.LowerCaseFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="0"/> <filter class="solr.LowerCaseFilterFactory"/> </analyzer> </fieldType> <fieldType name="text_ngram" class="solr.TextField" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.NGramTokenizerFactory" minGramSize="2" maxGramSize="10"/> <filter class="solr.LowerCaseFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> </analyzer> </fieldType> </types> <fields> <field name="index_id" type="string" indexed="true" stored="true" required="true" /> <field name="id" type="string" indexed="true" stored="true" required="true" /> <field name="site_id" type="string" indexed="true" stored="true" required="true" /> <field name="title" type="text" indexed="true" stored="true" /> <field name="title_ngram" type="text_ngram" indexed="true" stored="true" /> <field name="entity_type" type="string" indexed="true" stored="true" omitNorms="true" /> <field name="dataset_type" type="string" indexed="true" stored="true" /> <field name="state" type="string" indexed="true" stored="true" omitNorms="true" /> <field name="name" type="string" indexed="true" stored="true" omitNorms="true" /> <field name="name_ngram" type="text_ngram" indexed="true" stored="true" /> <field name="revision_id" type="string" indexed="true" stored="true" omitNorms="true" /> <field name="version" type="string" indexed="true" stored="true" /> <field name="url" type="string" indexed="true" stored="true" omitNorms="true" /> <field name="ckan_url" type="string" indexed="true" stored="true" omitNorms="true" /> <field name="download_url" type="string" indexed="true" stored="true" omitNorms="true" /> <field name="notes" type="text" indexed="true" stored="true"/> <field name="author" type="textgen" indexed="true" stored="true" /> <field name="author_email" type="textgen" indexed="true" stored="true" /> <field name="maintainer" type="textgen" indexed="true" stored="true" /> <field name="maintainer_email" type="textgen" indexed="true" stored="true" /> <field name="license" type="string" indexed="true" stored="true" /> <field name="license_id" type="string" indexed="true" stored="true" /> <field name="ratings_count" type="int" indexed="true" stored="false" /> <field name="ratings_average" type="float" indexed="true" stored="false" /> <field name="tags" type="string" indexed="true" stored="true" multiValued="true"/> <field name="groups" type="string" indexed="true" stored="true" multiValued="true"/> <field name="organization" type="string" indexed="true" stored="true" multiValued="false"/> <field name="capacity" type="string" indexed="true" stored="true" multiValued="false"/> <field name="permission_labels" type="string" indexed="true" stored="false" multiValued="true"/> <field name="res_name" type="textgen" indexed="true" stored="true" multiValued="true" /> <field name="res_description" type="textgen" indexed="true" stored="true" multiValued="true"/> <field name="res_format" type="string" indexed="true" stored="true" multiValued="true"/> <field name="res_url" type="string" indexed="true" stored="true" multiValued="true"/> <field name="res_type" type="string" indexed="true" stored="true" multiValued="true"/>  <field name="text" type="text" indexed="true" stored="false" multiValued="true"/> <field name="urls" type="text" indexed="true" stored="false" multiValued="true"/> <field name="depends_on" type="text" indexed="true" stored="false" multiValued="true"/> <field name="dependency_of" type="text" indexed="true" stored="false" multiValued="true"/> <field name="derives_from" type="text" indexed="true" stored="false" multiValued="true"/> <field name="has_derivation" type="text" indexed="true" stored="false" multiValued="true"/> <field name="links_to" type="text" indexed="true" stored="false" multiValued="true"/> <field name="linked_from" type="text" indexed="true" stored="false" multiValued="true"/> <field name="child_of" type="text" indexed="true" stored="false" multiValued="true"/> <field name="parent_of" type="text" indexed="true" stored="false" multiValued="true"/> <field name="views_total" type="int" indexed="true" stored="false"/> <field name="views_recent" type="int" indexed="true" stored="false"/> <field name="resources_accessed_total" type="int" indexed="true" stored="false"/> <field name="resources_accessed_recent" type="int" indexed="true" stored="false"/> <field name="metadata_created" type="date" indexed="true" stored="true" multiValued="false"/> <field name="metadata_modified" type="date" indexed="true" stored="true" multiValued="false"/> <field name="indexed_ts" type="date" indexed="true" stored="true" default="NOW" multiValued="false"/>  <field name="title_string" type="string" indexed="true" stored="false" /> <field name="data_dict" type="string" indexed="false" stored="true" /> <field name="validated_data_dict" type="string" indexed="false" stored="true" /> <field name="_version_" type="string" indexed="true" stored="true"/> <dynamicField name="*_date" type="date" indexed="true" stored="true" multiValued="false"/> <dynamicField name="extras_*" type="text" indexed="true" stored="true" multiValued="false"/> <dynamicField name="res_extras_*" type="text" indexed="true" stored="true" multiValued="true"/> <dynamicField name="vocab_*" type="string" indexed="true" stored="true" multiValued="true"/> <dynamicField name="*" type="string" indexed="true" stored="false"/> </fields> <uniqueKey>index_id</uniqueKey> <df>text</df> <solrQueryParser q.op="AND"/> <copyField source="url" dest="urls"/> <copyField source="title" dest="title_ngram"/> <copyField source="name" dest="name_ngram"/> <copyField source="ckan_url" dest="urls"/> <copyField source="download_url" dest="urls"/> <copyField source="res_url" dest="urls"/> <copyField source="extras_*" dest="text"/> <copyField source="res_extras_*" dest="text"/> <copyField source="vocab_*" dest="text"/> <copyField source="urls" dest="text"/> <copyField source="name" dest="text"/> <copyField source="title" dest="text"/> <copyField source="text" dest="text"/> <copyField source="license" dest="text"/> <copyField source="notes" dest="text"/> <copyField source="tags" dest="text"/> <copyField source="groups" dest="text"/> <copyField source="organization" dest="text"/> <copyField source="res_name" dest="text"/> <copyField source="res_description" dest="text"/> <copyField source="maintainer" dest="text"/> <copyField source="author" dest="text"/> </schema> ``` ## 把cdc portal跟data移到GCP ### portal 在GCP上建好cdc portal後 1.進去裡面打" vi /etc/ckan/default/production.ini" 會開始編輯檔案 2.![](https://i.imgur.com/8aN6Yb6.png) 如果看到這個畫面,按E就好了 3.![](https://i.imgur.com/itSs8Iy.png) 成功進到這個畫面就可以開始編輯了 4.輸入"/35"可以搜尋35(35是舊的IP的開頭),按下enter後,再按下鍵盤上的"I"鍵就可以開始修改內容了(注意輸入法可能會導致無法進入修改模式),把舊IP換成GCP的portal機的外部IP,接著按下鍵盤上的"esc"鍵跳出,再進行上述的步驟,直到所有的舊ip全部換成GCP上的portal機的外部IP即可 5.輸入":wq"後存檔離開 ### data 在GCP上建好cdc data後 #### 修改line.txt 1.輸入"sudo vi ../tatung/tcdc_opendata_visualization_data/config/line.txt "開始編輯檔案(ps:sudo vi 是用最高權限編輯) 2.![](https://i.imgur.com/wUZO0tb.png) 把除了上圖以外的東西都刪掉即可 #### 修改index.js 1.輸入"sudo vi /home/jkw/opendata-redis-backend/index.js"開始編輯檔案 2.![](https://i.imgur.com/asOH3zo.png) 把文件內除了上圖以外的所有IP改成GCP上的dat機的外部IP,如果是https記得把s去掉即可 #### 修改enterovirus.py 1.輸入"sudo vi /home/tatung/tcdc_opendata_visualization_data/enterovirus.py"開始編輯檔案 2.![](https://i.imgur.com/qXT9UsU.png) 將" soup = Soup(urllib.urlopen('https://nidss.cdc.gov.tw/old/ch/Default.aspx?op=1'))" 這一列修改成"soup = Soup(urllib.urlopen('https://nidss.cdc.gov.tw/PivotChart/ChartOpdCDCWNH08'))" (如下圖所示) ![](https://i.imgur.com/3YNexH3.png) 3.將在剛剛改的那行下面的 "res = select(soup, 'li#chart4 script')"改成 " # res = select(soup, 'li#chart4 script') res = select(soup, 'div')" (如下圖所示) ![](https://i.imgur.com/WOJ8CSK.png) ## Reference - [Official Solr documentation](https://lucene.apache.org/solr/guide/8_6/) - [Official Ckan documentation](https://docs.ckan.org/en/2.9/) - [Difference between the term core and collection](https://stackoverflow.com/questions/17044640/solr-collection-vs-cores) - [cloud-init - setting `lock_passwd` to false to enable password login](https://cloudinit.readthedocs.io/en/latest/topics/examples.html) - [Solr installation](https://tecadmin.net/install-apache-solr-on-linuxmint/) ###### tags: `ntou`

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.