Mohamed and Dragan Pairing Material
===

[toc]

## Installing CKAN for development using docker-compose

It is recommended that you create a dedicates folder for this, and inside run the following commands:

$ git clone https://github.com/okfn/docker-ckan.git

This will result in a `./docker-ckan` directory available where you cloned the repo, where which you can configure parameters and start the containers neccessary for running your local CKAN portal.

### Configuration

#### Modifying docker-compose.dev.yml file to suit our needs

version: "3"

services:

  ckan-dev:
    container_name: ckan-dev-sigma2
    build:
      context: ckan/
      dockerfile: Dockerfile.dev
      args:
        - TZ=${TZ}
    env_file:
      - .env
    links:
      - db
      - solr
      - redis
      # - datapusher
    ports:
      - "0.0.0.0:${CKAN_PORT}:5000"
    volumes:
      - ./src:/srv/app/src_extensions
      - ckan_storage-sigma2:/var/lib/ckan
    
  # datapusher:
  #   container_name: datapusher-sigma2
  #   build:
  #     context: datapusher/
  #   ports:
  #     - "8800:8800"

  db:
    container_name: db-sigma2
    env_file:
      - .env
    build:
      context: postgresql/
    environment:
      - PGDATA=/var/lib/postgresql/data/db
    volumes:
      - pg_data-sigma2:/var/lib/postgresql/data
    ports:
      - "5432:5432"

  solr:
    container_name: solr-sigma2
    image: ckan/ckan-solr:2.9-solr8
    ports:
      - "8983:8983"
    volumes:
      - solr_data-sigma2:/var/solr

  redis:
    container_name: redis-sigma2
    image: redis:alpine

  # minio:
  #   image: minio/minio
  #   container_name: minio-sigma2
  #   ports:
  #     - "9000:9000"
  #     - "9001:9001"
  #   environment:
  #     MINIO_ROOT_USER: minioadmin
  #     MINIO_ROOT_PASSWORD: minioadmin
  #   command: server /data --console-address ":9001"
  #   volumes:
  #     - minio_data-sigma2:/data

volumes:
  ckan_storage-sigma2:
  pg_data-sigma2:
  solr_data-sigma2:
  # minio_data-sigma2: 

#### Modifying Dockerfile.dev

FROM openknowledge/ckan-dev:2.10

MAINTAINER Your Name Here <[e-mail]>

# Set timezone
ARG TZ
RUN cp /usr/share/zoneinfo/$TZ /etc/localtime
RUN echo $TZ > /etc/timezone

ARG GITHUB_TOKEN
ENV GITHUB_TOKEN=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

# Install any extensions needed by your CKAN instance
# (Make sure to add the plugins to CKAN__PLUGINS in the .env file)
# For instance:

# RUN pip install -e git+https://$[e-mail]/datopian/ckanext-sigma2.git#egg=ckanext-sigma2 && \
# RUN pip install -e git+https://{GIHUB_TOKEN}@github.com/datopian/ckanext-datopian@18a9e12aeb55f2703f5829e7782451ebe9611c24#egg=ckanext-datopian  && \ 
RUN pip install -e git+https://github.com/datopian/ckanext-oauth2.git@sigma2-version#egg=ckanext-oauth2 && \
    pip install -e git+https://github.com/ckan/ckanext-scheming#egg=ckanext-scheming

COPY licenses.json /ext/licenses.json

# ENV CKAN__PLUGINS envvars image_view text_view recline_view oauth2 sigma2 datopian 
ENV TZ=UTC
RUN ckan config-tool /srv/app/ckan.ini "ckan.plugins = ${CKAN__PLUGINS}"
# RUN pip3 install -r /srv/app/src/ckan/dev-requirements.txt
RUN pip3 install flask_debugtoolbar
RUN chown -R 1001:1001 /srv/app/
RUN mkdir -p /var/lib/ckan/storage
RUN chown -R 1001:1001 /var/lib/ckan/storage
USER root

COPY setup/oauth2_config.yaml /srv/app/setup/files/oauth2_config.yaml

# Clone the extension(s) your are writing for your own project in the `src` folder
# to get them mounted in this image at runtime

# Apply any patches needed to CKAN core or any of the built extensions (not the
# runtime mounted ones)
# See https://github.com/okfn/docker-ckan#applying-patches
COPY patches ${APP_DIR}/patches

RUN for d in $APP_DIR/patches/*; do \
        if [ -d $d ]; then \
            for f in `ls $d/*.patch | sort -g`; do \
                cd $SRC_DIR/`basename "$d"` && echo "$0: Applying patch $f to $SRC_DIR/`basename $d`"; patch -p1 < "$f" ; \
            done ; \
        fi ; \
    done

#### Modifying .env file

You configure this `docker-compose` setup via environment variables. The way to do so is to copy the template `.env.example` file to `.env` and configure whatever customization you'd like to introduce as values to the environment variable already introduced in this file:

$ cd docker-ckan
$ cp .env.example .env
$ vi .env

You can use whatever editor you're comfortable with, here I'm using *vim*.

Some explanation of those configuration variables is availabe inline in the config file snapshot below:

# DB image settings
POSTGRES_PASSWORD=ckan
DATASTORE_READONLY_PASSWORD=datastore

# Basic
CKAN_SITE_ID=default
CKAN_SITE_URL=http://ckan:5000
CKAN_PORT=5000
CKAN___BEAKER__SESSION__SECRET=CHANGE_ME
# See https://docs.ckan.org/en/latest/maintaining/configuration.html#api-token-settings
CKAN___API_TOKEN__JWT__ENCODE__SECRET=
CKAN___API_TOKEN__JWT__DECODE__SECRET=
CKAN_SYSADMIN_NAME=ckan_admin
CKAN_SYSADMIN_PASSWORD=test1234
CKAN_SYSADMIN_EMAIL=[e-mail]
TZ=UTC

# Database connections (TODO: avoid duplication)
CKAN_SQLALCHEMY_URL=postgresql://ckan:ckan@db/ckan
CKAN_DATASTORE_WRITE_URL=postgresql://ckan:ckan@db/datastore
CKAN_DATASTORE_READ_URL=postgresql://datastore_ro:datastore@db/datastore

# Test database connections
TEST_CKAN_SQLALCHEMY_URL=postgres://ckan:ckan@db/ckan_test
TEST_CKAN_DATASTORE_WRITE_URL=postgresql://ckan:ckan@db/datastore_test
TEST_CKAN_DATASTORE_READ_URL=postgresql://datastore_ro:datastore@db/datastore_test

# Other services connections
CKAN_SOLR_URL=http://solr:8983/solr/ckan
CKAN_REDIS_URL=redis://redis:6379/1
CKAN_DATAPUSHER_URL=http://datapusher:8800
CKAN__DATAPUSHER__CALLBACK_URL_BASE=http://ckan:5000

# If the "expire_api_token" plugin is enabled we need to define
# "expires_in" and "unit" parameters
DATAPUSHER_TOKEN_EXPIRES_IN=
DATAPUSHER_TOKEN_EXPIRES_UNIT=
# example for 1 year
# DATAPUSHER_TOKEN_EXPIRES_IN=365
# unit=1:Second;unit=2:Minute;unit=3:Hour;unit=4:Day
# DATAPUSHER_TOKEN_EXPIRES_UNIT=4

TEST_CKAN_SOLR_URL=http://solr:8983/solr/ckan
TEST_CKAN_REDIS_URL=redis://redis:6379/1

# Core settings
CKAN__STORAGE_PATH=/var/lib/ckan

# Extensions

CKAN__PLUGINS="envvars image_view text_view recline_view activity scheming_datasets dataset_approval"
CKAN__HARVEST__MQ__TYPE=redis
CKAN__HARVEST__MQ__HOSTNAME=redis
CKAN__HARVEST__MQ__PORT=6379
CKAN__HARVEST__MQ__REDIS_DB=1

# ckanext-cloudstorage
# CKANEXT__CLOUDSTORAGE__CONTAINER_NAME="sigma2"
# CKANEXT__CLOUDSTORAGE__DRIVER="S3"
# CKANEXT__CLOUDSTORAGE__DRIVER_OPTIONS='{"key": "minioadmin", "secret": "minioadmin", "secure": false, "host": "minio-sigma2", "port": 9000}'
# CKANEXT__CLOUDSTORAGE__USE_SECURE_URLS=1
# CKANEXT__CLOUDSTORAGE__MAX_MULTIPART_LIFETIME=7

CKAN___LICENSES_GROUP_URL=file:///ext/licenses.json
CKAN__CORS__ORIGIN_ALLOW_ALL=True
CKAN__DATASETS_PER_PAGE=25
CKAN__DATASETS__CREATE_ON_UI_REQUIRES_RESOURCES=False
# CKAN__HIDE_ACTIVITY_FROM_USERS=default
CKAN__OAUTH2__CONFIG_PATH=/srv/app/setup/files/oauth2_config.yaml
CKAN__SITE_TITLE=Sigma 2
CKAN__TRACKING_ENABLED=True
CKAN_SITE_ID=sigma2
CKAN_SMTP_MAIL_FROM=[e-mail]
CKAN_SMTP_PASSWORD=FjXKZAn1H3ExzYOa
CKAN_SMTP_SERVER=smtp-relay.sendinblue.com:587
CKAN_SMTP_USER=[e-mail]
CKAN__ACTIVITY_STREAMS_ENABLED=True
CKAN__ACTIVITY_STREAMS_EMAIL_NOTIFICATIONS=True
CKAN__EMAIL_NOTIFICATIONS_SINCE=2 days
CKANEXT__OAUTH2__ACCOUNT_APPROVAL=True
CKANEXT__OAUTH2__PROFILE_UPDATE_ON_REGISTRATION=True
CKANEXT__SIGMA2__ARCHIVE_MANAGER_EMAIL=[e-mail]
CKANEXT__VALIDATION__PASS_AUTH_HEADER_VALUE=Secret
CKANEXT__VALIDATION__PASS_AUTH_HEADER=True
OAUTHLIB_INSECURE_TRANSPORT=False

# Ckanext-scheming
CKAN___SCHEMING__DATASET_SCHEMAS="ckanext.datasetapproval:dataset_schema.yaml"
CKAN___SCHEMING__PRESETS="ckanext.scheming:presets.json"

Once you're satisfied with the settings, it's time to move to the next section.

### Installing extension for development

We are working mainly on two extension [ckanext-sigma2](https://github.com/datopian/ckanext-sigma2/tree/master) and [ckanext-datasetapproval](https://github.com/datopian/ckanext-datasetapproval/tree/master). For the simplicuty we'll focus only on the second one now.

So, clone the secon extension into the `src` folder in your `docker-ckan` folder. When the extension is cloned checkout `sigma2` branch.

The extension will be installed when we boot up our containers. It is enabled in `.env` file, like all other extensions in line 51 with `dataset_approval` plugin.

### Booting Up Containers

Before we proceed with setup, let's remove this plugin from the line 51 in `.env` file. This way we'll see basic CKAN and try to get more familiar with some user workflows that we need. This way we can compare the changes that this extension is making in CKAN.

$ docker-compose -f docker-compose.dev.yaml up

If this is the first time you are executing this, you'd see a lot of interesting and verbose message of building the dependent continaers (see a list of software in the *Preamble*) until you reach a state where the setup has finished building and the webapp is accepting HTTP connections, as an example it could similar to this:

$ docker-compose -f docker-compose.dev.yml up
Docker Compose is now in the Docker CLI, try `docker compose up`

Creating network "docker-ckan_default" with the default driver
Building datapusher
[+] Building 395.7s (10/10) FINISHED 
=> [internal] load build definition from Dockerfile                                                                                                                                                       0.0s
 => => transferring dockerfile: 1.62kB                                                                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                                                          0.0s
 => => transferring context: 2B                                                                                                                                                                            0.0s
 => [internal] load metadata for docker.io/library/alpine:3.13                                                                                                                                             3.5s
 => [1/6] FROM docker.io/library/alpine:3.13@sha256:f51ff2d96627690d62fee79e6eecd9fa87429a38142b5df8a3bfbb26061df7fc                                                                                       0.0s
 => => resolve docker.io/library/alpine:3.13@sha256:f51ff2d96627690d62fee79e6eecd9fa87429a38142b5df8a3bfbb26061df7fc                                                                                       0.0s
 => => sha256:f51ff2d96627690d62fee79e6eecd9fa87429a38142b5df8a3bfbb26061df7fc 1.64kB / 1.64kB                                                                                                             0.0s
 => => sha256:def822f9851ca422481ec6fee59a9966f12b351c62ccb9aca841526ffaa9f748 528B / 528B                                                                                                                 0.0s
 => => sha256:6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec 1.47kB / 1.47kB                                                                                                             0.0s
 => [2/6] WORKDIR /srv/app                                                                                                                                                                                 0.0s
 => [3/6] RUN apk add --no-cache     python3     py3-pip     py3-wheel     libffi-dev     libressl-dev     libxslt     uwsgi     uwsgi-http     uwsgi-corerouter     uwsgi-python     && apk add --no-ca  34.4s
 => [4/6] RUN mkdir /srv/app/src && cd /srv/app/src &&     git clone -b 0.0.17 --depth=1 --single-branch https://github.com/ckan/datapusher.git &&     cd datapusher &&     python3 setup.py install &&  352.2s 
 => [5/6] RUN apk del .build-deps &&     cp /srv/app/src/datapusher/deployment/*.* /srv/app &&     sed -i '/http/d' /srv/app/datapusher-uwsgi.ini &&     sed -i '/wsgi-file/d' /srv/app/datapusher-uwsgi.  0.5s 
 => [6/6] RUN addgroup -g 92 -S www-data &&     adduser -u 92 -h /srv/app -H -D -S -G www-data www-data
............
.........
.....
..

## Getting familiar with CKAN workflows

Here we will create our first organization, one dataset and learn create and edit dataset workflows.

## Enabling `datasetapproval` extension

Now we will enable this extension, learn a basic things about CKAN's templating (jinja), CSS, webassets...

### Links

1. CKAN
* https://github.com/ckan/ckan/tree/master

2. CKAN templates
* https://github.com/ckan/ckan/tree/master/ckan/templates

3. ckanext-datasetapproval
* https://github.com/datopian/ckanext-datasetapproval/tree/master

4. Figma
* https://www.figma.com/file/lCeOuYv3youqUDnE1qfzwi/NIRD---Mockup?type=design&node-id=480-2950&mode=design&t=zuPNb684fgNLc0wp-0

5. CKAN Theming guide
* https://docs.ckan.org/en/2.10/theming/index.html

## Licenses

[
    {
        "domain_content": false,
        "domain_data": false,
        "domain_software": false,
        "family": "",
        "id": "notspecified",
        "is_generic": true,
        "maintainer": "",
        "od_conformance": "not reviewed",
        "osd_conformance": "not reviewed",
        "status": "active",
        "title": "License Not Specified",
        "url": ""
    },
    {
        "domain_content": false,
        "domain_data": true,
        "domain_software": false,
        "family": "",
        "id": "ODC-PDDL-1.0",
        "maintainer": "",
        "od_conformance": "approved",
        "osd_conformance": "not reviewed",
        "status": "active",
        "title": "Open Data Commons Public Domain Dedication and Licence 1.0",
        "url": "http://www.opendefinition.org/licenses/odc-pddl"
    },
    {
        "domain_content": false,
        "domain_data": true,
        "domain_software": false,
        "family": "",
        "id": "ODbL-1.0",
        "maintainer": "",
        "od_conformance": "approved",
        "osd_conformance": "not reviewed",
        "status": "active",
        "title": "Open Data Commons Open Database License 1.0",
        "url": "http://www.opendefinition.org/licenses/odc-odbl"
    },
    {
        "domain_content": false,
        "domain_data": true,
        "domain_software": false,
        "family": "",
        "id": "ODC-BY-1.0",
        "maintainer": "Open Data Commons",
        "od_conformance": "approved",
        "osd_conformance": "not reviewed",
        "status": "active",
        "title": "Open Data Commons Attribution License 1.0",
        "url": "http://www.opendefinition.org/licenses/odc-by"
    },
    {
        "domain_content": true,
        "domain_data": true,
        "domain_software": true,
        "family": "",
        "id": "CC0-1.0",
        "maintainer": "Creative Commons",
        "od_conformance": "approved",
        "osd_conformance": "not reviewed",
        "status": "active",
        "title": "CC0 1.0",
        "url": "https://creativecommons.org/publicdomain/zero/1.0/"
    },
    {
        "domain_content": true,
        "domain_data": true,
        "domain_software": false,
        "family": "",
        "id": "CC-BY-4.0",
        "maintainer": "Creative Commons",
        "od_conformance": "approved",
        "osd_conformance": "not reviewed",
        "status": "active",
        "title": "Creative Commons Attribution 4.0",
        "url": "https://creativecommons.org/licenses/by/4.0/"
    },
    {
        "domain_content": true,
        "domain_data": true,
        "domain_software": false,
        "family": "",
        "id": "cc-by-sa",
        "maintainer": "Creative Commons",
        "od_conformance": "approved",
        "osd_conformance": "not reviewed",
        "status": "active",
        "title": "Creative Commons Attribution Share-Alike 4.0",
        "url": "https://creativecommons.org/licenses/by-sa/4.0/"
    },
    {
        "domain_content": true,
        "domain_data": false,
        "domain_software": false,
        "family": "",
        "id": "GFDL-1.3-no-cover-texts-no-invariant-sections",
        "maintainer": "Free Software Foundation",
        "od_conformance": "approved",
        "osd_conformance": "not reviewed",
        "status": "active",
        "title": "GNU Free Documentation License 1.3 with no cover texts and no invariant sections",
        "url": "http://www.opendefinition.org/licenses/gfdl"
    },
    {
        "domain_content": true,
        "domain_data": false,
        "domain_software": false,
        "family": "",
        "id": "other-open",
        "is_generic": true,
        "maintainer": "",
        "od_conformance": "approved",
        "osd_conformance": "not reviewed",
        "status": "active",
        "title": "Other (Open)",
        "url": ""
    },
    {
        "domain_content": true,
        "domain_data": false,
        "domain_software": false,
        "family": "",
        "id": "other-pd",
        "is_generic": true,
        "maintainer": "",
        "od_conformance": "approved",
        "osd_conformance": "not reviewed",
        "status": "active",
        "title": "Other (Public Domain)",
        "url": ""
    },
    {
        "domain_content": true,
        "domain_data": false,
        "domain_software": false,
        "family": "",
        "id": "other-at",
        "is_generic": true,
        "maintainer": "",
        "od_conformance": "approved",
        "osd_conformance": "not reviewed",
        "status": "active",
        "title": "Other (Attribution)",
        "url": ""
    },
    {
        "domain_content": true,
        "domain_data": true,
        "domain_software": true,
        "family": "",
        "id": "uk-ogl",
        "is_generic": false,
        "maintainer": "UK Government",
        "od_conformance": "approved",
        "osd_conformance": "not reviewed",
        "status": "active",
        "title": "Open Government Licence 2.0 (United Kingdom)",
        "url": "https://www.nationalarchives.gov.uk/doc/open-government-licence/version/2/"
    },
    {
        "domain_content": true,
        "domain_data": true,
        "domain_software": true,
        "family": "",
        "id": "OGL-UK-3.0",
        "is_generic": false,
        "maintainer": "UK Government",
        "od_conformance": "approved",
        "osd_conformance": "not reviewed",
        "status": "active",
        "title": "Open Government Licence 3.0 (United Kingdom)",
        "url": "https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/"
    },
    {
        "domain_content": true,
        "domain_data": true,
        "domain_software": false,
        "family": "Creative Commons",
        "id": "CC-BY-NC-4.0",
        "maintainer": "Creative Commons",
        "od_conformance": "rejected",
        "osd_conformance": "not reviewed",
        "status": "active",
        "title": "Creative Commons Attribution-NonCommercial 4.0",
        "url": "https://creativecommons.org/licenses/by-nc/4.0/"
    },
    {
        "domain_content": false,
        "domain_data": false,
        "domain_software": false,
        "family": "",
        "id": "other-nc",
        "is_generic": true,
        "maintainer": "",
        "od_conformance": "not reviewed",
        "osd_conformance": "not reviewed",
        "status": "active",
        "title": "Other (Non-Commercial)",
        "url": ""
    },
    {
        "domain_content": false,
        "domain_data": false,
        "domain_software": false,
        "family": "",
        "id": "other-closed",
        "is_generic": true,
        "maintainer": "",
        "od_conformance": "not reviewed",
        "osd_conformance": "not reviewed",
        "status": "active",
        "title": "Other (No License)",
        "url": ""
    }
]