# CKAN Python 3 support roadmap

## Overview

With the end of official support for Python 2.7 coming up on the end of this year we need to have a clear roadmap of what needs to happen to get full Python 3 support before that. It is unlikely that we will be fully migrated (as in the whole project being Python 3 only) but we should aim to have a release in place by then that can run both Python 2 and 3. As discussed previously, we will aim for a transition release where the same code base supports both versions, before dropping all Python 2 support in the next one:

* CKAN 2.8 (current) - Python 2 only
* CKAN 2.9 - Python 2 and 3
* CKAN 3.0 (*) - Python 3 only

(*) Major version changes are the time to break compatibilities, so if all the changes are internal py3 ones we might want to save the major number change for future changes on API, frontend, etc. Something to think about.

### How this roadmap works

The following is a collection of tasks that need to happen to get us to these releases. They are different in size, they need to happen at different points in time and some are blocked by others. This will be reflected as precisely as possible on an appropriate tool like GitHub projects, but the goal is to have a GitHub issue for each of this tasks, that can be assigned and tracked.

All issues will be clearly labeled, including these that are Blocked, Good for Contribution (there are several of these) or even Beginner Friendly (there are some of these).

The number of tasks might look daunting but the truth is that only three or four of them are really big and complex tasks, with only a couple of major blockers. The rest are self-contained, and most of them can be tackled by developers with different degrees of experience. I've tried to give as much detail as possible on the Approach section so it's clear what needs to be done to implement them. But of course, the further we go into the future the more unknowns we will find.

A quick word on estimates. Estimating is hard and with a project like this involving different stages and multiple tasks even more. What's more we want to get as much help on this as possible so we need to consider different degrees of familiarity with CKAN (eg same task might take X to a core dev but Y to someone with less experience) I know that ideally people would like to see hours or days put down but I don't think is realistic at this stage. Instead I've borrowed the concept of Story Points from agile methodologies. They are a scale of Fibonnaci-like values that take into account aspects like Amount of Work, Risk and Complexity. There are different scales and interpretations, I just picked one an stuck with it:


Here are [some](https://www.mountaingoatsoftware.com/blog/what-are-story-points) [resources](https://www.atlassian.com/agile/project-management/estimation) on [story points](https://medium.com/@mdalmijn/12-common-mistakes-made-when-using-story-points-f0bb9212d2f7).

## Stages

As mentioned in the previous section the process can be roughly split in these three phases:

1. **Enabling Python 3**

If the first goal is to be able to run CKAN in Python 3 the priority is to remove any Python 2 only requirement. Most of the work to remove the biggest one (Pylons) is done, but there's still vdm and other minor requirements that block the transition. These should be the top priority over the next months.

Main risks for this stage:

* Removing VDM: just because of its pervasiveness
* Replacing/Upgrading pyutilib: as it touches the critical feature of plugins interfaces, which needs extra care to avoid breaking compatibility

2. **Transition**

Once we can run CKAN with a set of requirements that supports Python 3 we need to make sure that the core codebase can run on it and that all tests pass. There has been extensive work done on that front already, but quick preliminary tests show that there are still issues to be ironed out and no doubt new ones will come up. Also we need to focus on extensions, developing a clear upgrade path and documenting changes.

Main risks for this stage:

* Extensions that need to support multiple CKAN (and Python) versions: we haven't explored much the options there

3. **Cleanup**

After the transition version is out we can finally remove all Python 2 only requirements and associated code, as well as the custom middleware and common logic to support requests to Pylons and Flask, etc

## Tasks

The following tasks have all been defined as GitHub issues in the [main CKAN repo](https://github.com/ckan/ckan/issues?q=is%3Aissue+is%3Aopen+label%3A%22Python+3%22). They are laid out and prioritized in the [Python 3 Support](https://github.com/ckan/ckan/projects/3) project in the same repo.


The issues and project in the repo are the reference place for them, below there is a summary provided for convenience:

### Enabling

* Migrate controllers from core extensions to blueprints ([#4791](https://github.com/ckan/ckan/issues/4791))
* Remove VDM ([#4783](https://github.com/ckan/ckan/issues/4783))
* Remove or replace formencode support ([#4800](https://github.com/ckan/ckan/issues/4800))
* Replace webhelpers with a modern alternative ([#4794](https://github.com/ckan/ckan/issues/4794))
* Replace pyutilib.component.core with a modern alternative ([#4795](https://github.com/ckan/ckan/issues/4795))
* Replace repoze.who-friendlyform with a modern alternative ([#4796](https://github.com/ckan/ckan/issues/4796))
* Write a Flask and Python 3 migration guide for extensions ([#4792](https://github.com/ckan/ckan/issues/4792))

### Transition

* Implement changes to allow running CKAN in both Python 2 and 3 ([#4801](https://github.com/ckan/ckan/issues/4801))
* Replace paste.script with Click ([#4639](https://github.com/ckan/ckan/issues/4639))
* Flask and Python 3 support on selected extensions ([#4793](https://github.com/ckan/ckan/issues/4793))
* Replace paste.deploy converters ([#4797](https://github.com/ckan/ckan/issues/4797))
* Don't use paste.deploy appconfig to parse the ini file ([#4798](https://github.com/ckan/ckan/issues/4798))
* Don't use paste.deploy to create the extension template ([#4799](https://github.com/ckan/ckan/issues/4799))
* Update the WSGI entrypoint to not use paste.deploy ([#4802](https://github.com/ckan/ckan/issues/4802))
* Replace paster make-config ([#4803](https://github.com/ckan/ckan/issues/4803))

### Cleanup

* Remove Fanstatic ([#4804](https://github.com/ckan/ckan/issues/4804))
* Remove AppDispatcher and Pylons middleware ([#4805](https://github.com/ckan/ckan/issues/4805))
* Remove vars in templates context set in views ([#4806](https://github.com/ckan/ckan/issues/4806))
* Remove all remaining Pylons code in core ([#4807](https://github.com/ckan/ckan/issues/4807))