{%hackmd theme-dark %}

## Rationale and benefits

From thread + others

Basic rationale
* Be more clear and explict about deprecation removal status
* Ensure deprecations are fully documented
* Deprecation info now programmatically usable in linters, APIs & docs

Benefits of doing so in the docs
* Consistant and complete rendered appearance
* Outputs a static JSON data file readible and vendorable by anything
* Can add automatic table in What's New and centralized page
* No need to keep multiple seperate data sources in sync
* Takes advantage of existing structured data we already have
* Easily backportable to bugfix-supported versions

Features of this implementation
* Fully backwards-compatible with existing directive usage
* New metadata can be added incrementally (with clear flag to usage)
* Can already generate some basic metadata about existing deprecations
* Highly customizable by authors with fixed, semi-structured and free text
* Only requires `reason`; everything else optional
* Can generate deprecation notices that are far richer but just as concise
* Can deprecate multiple APIs at once, generating deprecation notices for each

## Open items

* Maybe put per-target items (qualifiers, replace) per-target in confg + schema?
* How to handle removed names:
* Move up and use child mechanism
* Put deprecations for entirely removed modules on the new deprecations page(s)

## Changes to implement programmatic deprecation parsing & reStructured Data

### Phase One

* [x] Move deprecated-removed to its own module
* [x] Clean up, refactor, document and update existing implementation
* [x] Use modern Sphinx constructs & sync with upstream improvements
* [x] Refactor logic for easy readibility and extension
* [x] Split monolithic run() into atomic methods & document them
* [x] Fix minor correctness issues with implementation
* [x] Add type annotations for documentation, RSD integration & per upstream
* [ ] Modify directive into two classes, one for generic versionmodified & one for deprecated-removed
* [ ] Output existing data as minimal JSON
* [x] Create and set up basic `rsd` domain
* [x] Add minimal data structure
* [x] Add domain method to populate data structure
* [x] Add class decorator to note deprecation
* [x] Add HTML output step to `build-finished`
* [x] Add `doctree-resolved` transform to remove wrapping
* [ ] Add events for `before-directive-run` and `after-directive-run` + use them
* [ ] Add method to register converter, and do so for removal version
* [ ] Add builder specifically for structured data
* [ ] Ensure directives that output messages are being handled
* [ ] Generate inbound target on deprecated-removed
* [ ] Output in table form via directive (and test on deprecations/removals page)
* [ ] Grab transformed, translated content on `process_doc`
* [ ] Add `:targets:` and `:display:`, and logic/transforms handling them
* [ ] Get source information for referenced objects
* [ ] Generate top-level metadata for output
* [ ] Add list of deprecated and removed names to top-level
* [ ] Allow second arg to take `notplanned`, `notscheduled` and `tbd`
* [ ] Add new options to directive, JSON & rendered output
* [ ] Add remaining output schema elements
* [ ] Add per-file glob default domain config setting
* [ ] Test some basic examples

### Phase Two

* [ ] Hook all directives and filter to supported ones, with config options
* [ ] Add huristics for common cases (e.g. version, "rsd:attribute", etc.)
* [ ] Add support for the other versionmodified directives to deprecated-removed
* [ ] And hueristics for parameters & behavior based on content
* [ ] Add custom handlers for Docutils & Sphinx builtin directives
* [ ] Add for Docutils/Sphinx extension directives too
* [ ] Convert other existing custom directives
* [ ] Validation (deprecation-removed):
* [ ] First and second args are versions
* [ ] Require `reason` if any options passed
* [ ] Require `:reason-desc:` if reason `other`
* [ ] Require `:type:` if `:value:`
* [ ] Require `:replace:` or `:replace-desc:` if `:replace-version:`
* [ ] Require `:replace:` if `dropin`
* [ ] Write some tests of the output (validate schema, check a few semipermanet examples)
* [ ] Add and test more examples

### Further follow-ons

* [ ] Add config option to customize deprecation reasons
* [ ] Add HTML output for messages to output data
* [ ] Support section-level annotations that apply to all objects in section
* [ ] Add `currentobject` directive to declare current object context for section
* [ ] Allow sorting versionmodified directives by version, ascending or descending, and automoving to end
* [ ] Extract, process, integrate and write out object data
* [ ] Allow registering handlers for custom object types

## Restructured Data details

### Directive options

* ``:targets: :role:`<child1>`,:role:`<child2>` ``: Apply the annotation to specific children of this object rather than the whole object (for applying the same message to multiple children, or for when the actual target has been removed)
* `:domain: <domain>` Override the default domain this annotation is listed under
* `:param: <old_param>`: Mark annotation as applying to a specific parameter
* `:type: <type1>`: Type of object that this annotation applies to, as a cross reference or raw value (directive-dependent)
* `:value: <val1>`: specific value this annotation applies to, as a literal
* `:behavior: <description>`: Flag this as annotation as applying to specific _behavior_ of the item rather than the whole item
* `:moreinfo: <role>, <URL>` Role or URL pointing to where more information can be found about this annotation
* `:display: <auto, parent, target, both, none>` If and where to show the annotation in the rendered output

### Config options

* Explictly white/blacklist annotation (and object) types to output to JSON
* Default domain (global and per directory)
* Deployment URL (base and subpath w/placeholders, + default lang, stable, latest)
* Custom deprecation reasons
* Enable/disable gathering reStructuredData for directive and set directives to/to not autopatch (at top level or all)

### Transforms/Processors

* Get translated, transformed content (>≈800)
* Add annotations to child nodes (>850) if `:display:` & replace parent names
* Get source information for referenced objects (env-updated)
* Remove annotations based on `:display:` criterion (<= doctree-resolved)
* Build tables of deprecation/removed annotations (<= doctree-resolved)
* Remove RSD wrapper nodes (= doctree-resolved)

## ``deprecated-removed`` directive

### New options

* `:no-automessage:` Do not include the autogenerated message in the output
* `:reason: <reason>` The reason for the deprecation, pre-registered
* `:reason-desc: <desc>`: Prose description of reason(s)
* `` :replace: :role:`<replace1>` ``: Sphinx role, URL, or (depending on granularity) param name, type or value
* `` :replace-version: `<X.Y>` ``: version in which the replacement API was introduced
* ``:replace-desc: <desc>` ``: Prose description of replacement
* `replace-param`, `replace-type`, `replace-value`: The parameter, param type and param value to replace this with (if these values don't match the current)
* `:dropin:` Flag to indicate the `:replace:` is a drop-in replacement. `:replace:` must be a single role or URL.
* `:warning: <WarningClass>` Deprecation warning class raised if not `DeprecationWarning`, or `False` if it doesn't warn.
* `:warn-version: <X.Y>` Version that started issueing a warning, if not the deprecation version.
* `:error: ExceptionClass` Error raised once the `remove` version is reached, if not a removal, or `False` if behavior changes without an error

### Reason values

- `correctness`: Does not actually produce the specified behavior
- `security`: Has an unfixable security issue
- `safety`: Not safe to use in general
- `usability`: Not easy to use correctly, or easy to use incorrectly
- `obsolete`: No longer generally relevant for modern usage
- `efficiency`: Use results in poor performance
- `alias`: Is a name alias of another identical API
- `defunct`: No longer actively maintained
- `superseded`: Maintained but replaced by another better alternative
- `other`: Reason not listed; see `reason-desc`

### Action plan for deprecated-removed and versionmodified

* [ ] Fix and add to existing `deprecated-removed`
* -> Have validation require third arg to be reason?
* -> Only allow :type: and :value: on specific types?
* Backport to 3.12 and 3.11
* [ ] Add additional validation checks
* `:replace:` matches the type of what its replacing (strictness depending on `:dropin:`)
* Children and ref targets exist/not if not >=removed version
* id required if no parent and no child
* Parameters exist or not in signature depending on removal version
* [ ] Then add to `deprecated` as well
* Backport as well
* [ ] Add deprecated items tabls
* Polish new deprecations/removals page
* Output per version, action and domain in What's New
* Cross-check what's new for things that should be deprecated
* Backport
* [ ] Add to removed names
* Forward-port backports
* Backfill since Python 3.0
* [ ] Add additional formal deprecation notices for things that are informally discouraged/deprecated, but lack them

### Examples

[Sequence instead of dict for *parameters* in `sqlite3.Cursor.execute*` when using named placeholders](https://docs.python.org/3.12/library/sqlite3.html#sqlite3.Cursor.execute)

.. deprecated-removed:: 3.12 3.14
   :child: :meth:`execute`, :meth:`executemany`
   :param: parameters
   :paramtype: ~collections.abc.Sequence
   :behavior: when using named placeholders
   :reason: usability
   :replace: Pass a :class:`dict` instead

Which renders as (under each method if present):

> **Deprecated in version 3.12 (usability), to be removed in version 3.14**: Passing an argument of type [`Sequence`](https://docs.python.org/3.12/library/collections.abc.html#collections.abc.Sequence) to the *parameters* parameter of the [`execute()`](https://docs.python.org/3.12/library/sqlite3.html#sqlite3.Cursor.execute) method when using named placeholders. Pass a :class:`dict` instead.

[`logging.warn` alias](https://docs.python.org/3.12/library/logging.html#logging.Logger.warning)

.. deprecated-removed:: 3.4 notscheduled
   :child: :meth:`!warn`
   :reason: alias
   :replace: :meth:`warning`
   :replace-version: 2.3
   :dropin:

Which renders as

> **Deprecated in version 3.4 (alias), removal not yet scheduled**: The ``warn()`` alias method. Use the [`warning()`](https://docs.python.org/3.12/library/logging.html#logging.Logger.warning) method instead (added in version 2.3), which is a drop-in replacement.

## Base schema

### Annotation item

id: str (e.g. "deprecated_removed:py:sqlite3.Cursor:execute-executemany:parameters:collections.abc.Sequence:behavior-1")
kind: str (e.g. "deprecated_removed")
location: <location>
targets: dict[str, TargetObject]
    <target_id>: <target>
text: <text>
message: None | <text>
description: None | <text>
domain: None | str (e.g. "py")
parent: None |
    physical: None | <object>
    logical: None | <object>
children: None |
    raw: list[str] (e.g. [":meth:`execute`", ":meth:`executemany`"])
    full: list[str] (e.g. ["sqlite3.Cursor.execute", "sqlite3.Cursor.executemany"])
    physical: list[str] (e.g. ["Cursor.execute", "Cursor.executemany"])
    logical: list[str] (e.g. ["execute", "executemany"])
qualifiers: None | <qualifiers>
granularity: None | Literal["object", "param", "type", "value"]
version: None | <version>
display: None | list[Literal["parent", "targets", "both", "none"]]
moreinfo: None | <location>
attributes: None | dataclasses.dataclass

### Annotation target

id: str (e.g. "deprecated_removed:py:sqlite3.Cursor:execute:parameters:collections.abc.Sequence:behavior-1")
object: <object>
location: <location>
message: <text>
text: <text>

### Object

name: str (e.g. "sqlite3.Cursor.execute")
shortname: str (e.g. "execute")
domain: str (e.g. "py")
kind: str (e.g. "py:method")
location: <location>
exists: bool | None

## Python object

module: str (e.g. "sqlite3")
class: str (e.g. "sqlite3.Cursor")
...

### Location reference

raw: str (e.g. ":meth:`execute`")
uri: str (e.g. "https://docs.python.org/3.12/library/sqlite3.html#sqlite3.Cursor.execute")
xref: <xref>
source: <source>

### Cross reference

id: str (e.g. "sqlite3.Cursor.execute")
role: str (e.g. ":py:meth:")
ref: str (e.g. ":py:meth:`sqlite3.Cursor.execute`)
title: str (e.g. "Cursor.execute")

### Item source

docpath: str (e.g. "library/sqlite3.rst")
repopath: str (e.g. "Doc/library/sqlite3.rst")
uri: str (e.g. "https://github.com/python/cpython/blob/3.12/Doc/library/sqlite3.rst")
section:
    title: str (e.g. "Cursor Objects")
    level: int
    location: <location>
line: int
permalink: str (e.g. "https://github.com/python/cpython/blob/1e501122876f7e6abf979fdb756d7edd0d924ba1/Doc/library/sqlite3.rst?plain=1#L1518"

### Annotation qualifiers

param: None | str (e.g. "old_param")
type: None | <object>
value: None |
    equals: None | str (e.g. "old_value")
behavior: None | False | True | <text>

### Text item

raw: str (e.g. "Passing an argument of type :class:`collections.abc.Sequence` to the *parameters* parameter of the :meth:`sqlite3.cursor.execute` method when using named placeholders. Pass a :class:`dict` instead.")
plain: str (e.g. "Passing an argument of type collections.abc.Sequence to the parameters parameter of the sqlite3.cursor.execute method when using named placeholders. Pass a dict instead.")
html: str (?)

### Version item

string: str (e.g. "3.7")
parsed: list[int] (e.g. (3, 7)

### Deprecation attributes

removal: None |
    raw: str (e.g. "3.12", "notplanned", etc.)
    state: Literal["tbd", "notplanned", "notscheduled", "scheduled"]
    removed: bool
    scheduled: bool
    planned: None | bool
    version: None | <version>
reason: None |
    name: list[str]
    message: <text>
    description: None | <text>
replace: None | False |
    raw: str (e.g. ":meth:`logging.Logger.warning`")
    text: str (e.g. "logging.Logger.warning")
    granularity: Literal["object", "param", "type", "value"]
    message: <text>
    description: None | <text>
    object: None | <object>
    qualifiers: None | <qualifiers>
    uri: str | None (e.g. "https://docs.python.org/3.12/library/logging.html#logging.Logger.warning")
    dropin: None | bool
    external: None | bool
    version: None | False | <version>
warning: None | False | <object>
warnversion: None | <version>
error: None | False | <object>

## Deprecation schema & examples

### Top-level schema

schema_version: str
metadata:
    project:
        name: str
        language: str | None
        version: <version>
        release: <version>
    sphinx:
        version: <version>
    rsd:
        version: <version>
        config: <RSD config>
    updated:
        unix: int (e.g. 1672425296)
        iso: str (e.g. "2022-12-30 12:34:56 UTC")
    source:
        uri: https://docs.python.org/3.12/deprecations.json
        domain: https://docs.python.org
        prefix: https://docs.python.org/3.12
        file: deprecations.json
    branch:
        value: "3.12"
        uri: "https://github.com/python/cpython/tree/3.12"
    commit:
        hash: "1e501122876f7e6abf979fdb756d7edd0d924ba1"
        uri: "https://github.com/python/cpython/tree/1e501122876f7e6abf979fdb756d7edd0d924ba1"
        dirty: bool
    custom:
        <dataclass>
enums:
    reason:
        correctness: "Does not actually produce the specified behavior"
        ...
    removal:
        notplanned: "Removal not planned in the immediate future"
        ...
deprecations: <deprecation schema>
bytarget:
    deprecation-domain-parent-child-param_name-param_type-param_value-behavior-id (e.g. "deprecation-py-sqlite3-cursor-execute-parameters-collections-abc-sequence-behavior-1"):
        <deprecation_key>
names: {<Deprecation schema>}
    <py, c, envvar, program, none>:
        deprecated:
            <top-level deprecated name>: str <deprecation_key>
        removed:
            <top-level removed name>: str <deprecation_key>
        mapping:
            domain:parent.child[:param_name[:param_type[:param_value]]][:behavior-id]
                deprecations: list[str] [<deprecation_keys>]
                removals: list[str] [<deprecation_keys>]
                params:
                    deprecated:
                        <param>: str <deprecation_key>
                    removed:
                        <param>: str <deprecation_key>

### Full deprecation item schema

deprecated-removed:<domain>:<logical_parent>[:<children>][:<params>[:<paramtypes>[:<param_values>]]][:<behavior>-<behavior_id>]] (e.g. "py:sqlite3.Cursor:execute+executemany:parameters:collections.abc.Sequence:behavior-1"):
    id: str
    domain: str (e.g. "py")
    source: <item_source>
        doc_path: str (e.g. "library/sqlite3.rst")
        full_path: str (e.g. "Doc/library/sqlite3.rst")
        line: int
        url: str (e.g. "https://github.com/python/cpython/blob/3.12/Doc/library/sqlite3.rst")
        permalink: str (e.g. "https://github.com/python/cpython/blob/1e501122876f7e6abf979fdb756d7edd0d924ba1/Doc/library/sqlite3.rst?plain=1#L1518"
    location: <internal_location>
    physical_parent: None | <object>
        exists: bool
        name: str (e.g. "sqlite3.Cursor")
        shortname: str (e.g. "Cursor")
        domain: str (e.g. "py")
        kind: str (e.g. "py:class")
        children: list[str] (e.g. ["execute", "executemany"])
        context:
            module: str (e.g. "sqlite3")
            class: str (e.g. "sqlite3.Cursor")
            ...
        source: <item_source>
        location: <object_location>
            id: str (e.g. "sqlite3.Cursor")
            role: str (e.g. ":py:class:")
            ref: str (e.g. ":py:class:`sqlite3.Cursor`)
            rel: str (e.g. ":py:class:`Cursor`)
            url: str (e.g. https://docs.python.org/3.12/library/sqlite3.html#sqlite3.Cursor)
    logical_parent: None | <object>
    targets:
        <name>: (e.g. "sqlite3.Cursor.execute"):
            object: <object>
                name: str (e.g. "sqlite3.Cursor.execute")
                shortname: str (e.g. "execute")
                domain: str (e.g. "py")
                kind: str (e.g. "py:method")
                location: None | <object_location>
                    ref: str (e.g. ":py:meth:`sqlite3.Cursor.execute`")
                    rel: str (e.g. ":py:meth:`execute`")
                    url: str (e.g. "https://docs.python.org/3.12/library/sqlite3.html#sqlite3.Cursor.execute")
            annotation:
                id: str
                location: <internal_location>
                    id: str (e.g. "<id>")
                    role: str (e.g. ":ref:")
                    ref: str (e.g. ":ref:`deprecation-py-sqlite3-cursor-execute-parameters-collections-abc-sequence-behavior-1`")
                    url: str (e.g. "https://docs.python.org/3.12/library/sqlite3.html#deprecation-py-sqlite3-cursor-execute-parameters-collections-abc-sequence-behavior-1")
                message: <text>
                    rst: str (e.g. "Passing an argument of type :class:`collections.abc.Sequence` to the *parameters* parameter of the :meth:`sqlite3.cursor.execute` method when using named placeholders. Pass a :class:`dict` instead.")
                    plain: str (e.g. "Passing an argument of type collections.abc.Sequence to the parameters parameter of the sqlite3.cursor.execute method when using named placeholders. Pass a dict instead.")
                text: <text>
                    rst: str (e.g. "Deprecated in version 3.12 due to usability, to be removed in version 3.14: Passing an argument of type :class:`collections.abc.Sequence` to the *parameters* parameter of the :meth:`sqlite3.cursor.execute` method when using named placeholders. Pass a :class:`dict` instead.")
                    plain: str (e.g. "Deprecated in version 3.12 due to usability, to be removed in version 3.14: Passing an argument of type collections.abc.Sequence to the parameters parameter of the sqlite3.cursor.execute method when using named placeholders. Pass a dict instead.")
    qualifiers: None | <qualifiers>
        param: None | str (e.g. "old_param")
        type: None | object
        value: None |
            equals: str | None (e.g. ["old_value"])
    granularity: Literal["object", "param", "type", "value"]
    behavior: None | False |
        rst: str (e.g. "when using :ref:`sqlite3-named-placeholders`")
        plain: str (e.g. "when using named placeholders")
    version: <version>
        string: str (e.g. "3.7")
        parsed: tuple[int, ...] (e.g. (3, 7)
    removal: None |
        raw: str (e.g. "3.12", "notplanned", etc.)
        version: None | <version>
            string: str (e.g. "3.12")
            parsed: tuple[int, ...] (e.g. (3, 12))
        state: Literal["tbd", "notplanned", "notscheduled", "scheduled"]
        planned: bool | None
        scheduled: bool
        removed: bool
    reason:
        name: list[Literal["correctness", "security", "safety", "usability", "obsolete", "efficiency", "alias", "defunct", "superseded", "other"]]
        description: None | <text>
            rst: str (e.g. "due to allowing SQL injection attacks")
            plain: str (e.g. "due to allowing SQL injection attacks")
        message: <text>
            rst: str (e.g. "Pass a :class:`dict` instead.", "Use the :meth:`~logging.Logger.warning` method instead (added in version 2.3), which is a drop-in replacement.")
            plain: str (e.g. "Pass a dict instead.")
    replace: None | False |
        raw: str (e.g. :meth:`logging.Logger.warning`)
        text: str (e.g. "logging.Logger.warning")
        granularity: Literal["object", "param", "type", "value"]
        description: None | <text>
            rst: str (e.g. "Pass a :class:`dict` instead.")
            plain: str (e.g. "Pass a dict instead.")
        object: None | <object>
            name: str (e.g. "logging.Logger.warning")
            shortname: str (e.g. "warning")
            kind: str (e.g. "py:method")
            location: <object_location>
                ref: str (e.g. :meth:`logging.Logger.warning`)
                rel: str (e.g. :meth:`warning`)
                url: str (e.g. https://docs.python.org/3.12/library/logging.html#logging.Logger.warning)
        qualifiers: None | <qualifiers>
        url: str | None (e.g. https://docs.python.org/3.12/library/logging.html#logging.Logger.warning)
        dropin: bool
        external: bool
        version: None | False | <version>
            raw: str (e.g. "3.7")
            parsed: tuple[int, ...] (e.g. (3, 7))
        message: <text>
            rst: str (e.g. "Pass a :class:`dict` instead.", "Use the :meth:`~logging.Logger.warning` method instead (added in version 2.3), which is a drop-in replacement.")
            plain: str (e.g. "Pass a dict instead.")
    warning: None | False | <object>
    warn_version: <version>
    error: None | False | <object>
    description: None | <text>
        rst: str
        plain: str
    message: <text>
        rst: str (e.g. "Passing an argument of type :class:`collections.abc.Sequence` to the *parameters* parameter of the :meth:`sqlite3.Cursor.execute` and :meth:`sqlite3.Cursor.executemany` methods when using named placeholders. Pass a :class:`dict` instead.")
        plain: str (e.g. "Passing an argument of type collections.abc.Sequence to the parameters parameter of the sqlite3.Cursor.execute and sqlite3.Cursor.execute methods when using named placeholders. Pass a dict instead.")
    text: <text>
        rst: str (e.g. "Deprecated in version 3.12 due to usability, to be removed in version 3.14: Passing an argument of type :class:`collections.abc.Sequence` to the *parameters* parameter of the :meth:`sqlite3.Cursor.execute` and :meth:`sqlite3.Cursor.executemany` methods when using named placeholders. Pass a :class:`dict` instead.")
        plain: str (e.g. "Deprecated in version 3.12 due to usability, to be removed in version 3.14: Passing an argument of type collections.abc.Sequence to the parameters parameter of the sqlite3.Cursor.execute and sqlite3.Cursor.execute methods when using named placeholders. Pass a dict instead.")
    moreinfo: None |
        raw: str (e.g. ":ref:`whatsnew312-deprecations-sqlite3`")
        location: None | <location>
            ref: str | None (e.g. ":ref:`whatsnew312-deprecations-sqlite3`")
            url: str (e.g. "https://docs.python.org/3.12/whatsnew/3.12.html#whatsnew312-deprecations-sqlite3")

## Notes

Victor's original PR adding similar functionality as a Python module: [python/cpython#106241](https://github.com/python/cpython/pull/106241)

Maybe for params add something like (speculative)?

deprecated_removed_param_names:
    <name with a param deprecation>:
        deprecations: list[str] <list of deprecation keys>
        removals: list[str] <list of deprecation keys>
        deprecated_params:
            <param>: str <deprecation_key>
        removed_params:
            <param>: str <deprecation_key>