Frictionless Data
Lightweight specifications and tooling to make it effortless to get, share, and validate data.
Who we are
Paul Walsh
Chief Product Officer
Open Knowledge International
https://github.com/pwalsh
Adam Kariv
Engineering Lead
Open Knowledge International
https://github.com/akariv
Where we work
"A world where knowledge creates power for the many, not for the few."
Mission
Open up all essential, public interest information and see it used to create insight that drives positive change.
What we do
- technical platforms for open data
- thought leadership
- community facilitation
CKAN ↗
OpenSpending ↗
OpenTrials ↗
Open Data Index ↗
OKI Labs ↗
OK Network ↗
Frictionless Data
Plumbing, not a platform.
- Generic
- Reusable
- Modular
- Needs-driven
Addressing the pain of working with public data.
Problems
{{ govt_dept }} wants to publish data, or is already publishing data.
What can we expect?
Tidiness
Where is the data in the file? (sheet, header rows…)
Cleaning
Spurious values everywhere
Data Format
Dates, Numbers etc.
Validation
constraints, mandatory columns, data types
File format
Which version of excel? Character encoding?
Metadata
When was this file published? Who published it?
Documentation
What does each column mean?
Solutions
-
Frictionless Data is a set of specifications and tooling to address these problems and more.
-
The solutions are all simple in isolation.
-
Together, they build a powerful stack for working with open data, benefiting publishers and consumers.
On disk
/package/ ## files generally co-located
/package/datapackage.json ## descriptor
/package/data1.csv ## data source
/package/data2.csv ## data source
Descriptor
A Data Package descriptor looks as follows:
{
"title": "Procurement Log May 2016",
"name": "procurement-log-may-2016",
"description": "Narrative description",
"resources": [
{... data resource ...},
{... data resource ...}
],
... other properties ...
}
Descriptor
A Data Resource descriptor looks as follows:
{
"name": "may-2016",
"period": "2016-05",
"data": [ "http://example.com/procurement-may-2016.csv" ],
"schema": {... table schema ...},
"format": "csv",
"encoding": "utf-8",
... other properties ...
}
Descriptor
A Table Schema descriptor looks as follows:
{
"fields": [
{ "name": "transaction_id", "type": "string" },
{ "name": "amount", "type": "number" },
{ "name": "year", "type": "integer",
"constraints": {
"required": true,
"minimum": 1948,
"maximum": 2017
}
}
]
}
Conventions
- Conventions are formalised as specifications
- Specifications enable tooling
- Tooling targets publishers and consumers
Modular
Extendable
That's it
No magic, no amazing insights.
- Table Schema
- Data Resource
- Data Package
Simple, extendable, reusable.
Data Package ↗
{
"title": "My Data Package",
"resources": [
{
"data": [ "path/to-data.csv" ]
}
]
}
Tabular Data Package ↗
{
"profile": "tabular-data-package",
"title": "My Data Package",
"resources": [
{
"data": [ "path/to-data.csv" ],
"schema": {
"fields": [
{ "name": "age", "type": "integer" }
]
}
}
]
}
Table Schema ↗
{
"fields":[
{
"name": "first_name",
"type": "string"
},
{
"name": "email",
"type": "string",
"format": "email"
},
{
"name": "date_of_birth",
"type": "date",
"format": "YYYY-MM-DD",
"constraints": { "required": true }
}
]
}
Libraries
Core libraries that implement the specifications.
- Data Package Python JavaScript Ruby R
- Table Schema Python JavaScript Ruby
Aside: Tool Fund ↗
Demo ↗
Core extensions and utilities
Higher-level applications
Higher-level applications leverage the stack towards data quality and validation goals, and complex processing pipelines.
goodtables ↗
- Concerned with data quality (consistency, structure, schema)
- Generates detailed reports for end users (non-technical)
- Based on Data Quality Spec, can easily customise
- We are working on a web service around this
- goodtables.io : "Continous Data Validation"
Data Package Pipelines ↗
- Stream-based ETL, producing Data Packages
- Meant for loading messy tabular data with a powerful standard library
OpenSpending ↗
- Platform for fiscal data
- OLAP API
- Advanced visualisation components
- Interactive data modeling
- All data handling based on Frictionless Data
- Fiscal Data package specification
- Flat file storage as single point of truth
- SQL and Elasticsearch databases derived
Demo ↗
Data Package Pipelines
An data processing framework
based on Data Packages
Philosophy and Origin
- Contribution oriented
- Many similar tasks vs. one big graph
- Data Package native
–
First use case:
scrape and process EU subsidy data from ~140 different data sources across Europe.
Contribution oriented
- Coding vs Declaring:
- Define processing tasks in YAML, not code
- Validate YAMLs using schema
- Processing pipelines are built of small & simple building blocks
- A powerful 'standard library' of processing steps
Contribution oriented
- Performance:
- Processing is done on data streams
- Limits memory, CPU and disk resource use
- Allows more flexibility in running pipelines concurrently (+hw requirements)
Contribution oriented
- Maintenance:
- Easier to understand and maintain
- Limits the complexity of implementation
–> Allows contribution from non-technical users
Many similar tasks
- Most ETL solutions focus on creating a big graph of tasks
Many similar tasks
-
Our use cases cosists of many similar, disconnected, sources
-
Consider:
- Scraping and processing yearly budgets from Israel's 259 different municipalities
- Each one published in a slightly different format
- Each one in their own web portal etc.
Data Package Native
Benefit from the entire toolset - DRY!
- Each step reads a datapackage from stdin
- Modify metadata
- Modify resources
- Emit a datapackage to stdout
Datapackage + Data are always valid
Example
CO2 Emissions from World Development Indicators
http://api.worldbank.org/v2/en/indicator/EN.ATM.CO2E.PC?downloadformat=excel
Pipeline
worldbank-co2-emissions:
pipeline:
-
run: add_metadata
parameters:
name: 'co2-emissions'
title: 'CO2 emissions (metric tons per capita)'
homepage: 'http://worldbank.org/'
-
run: add_resource
parameters:
name: 'global-data'
url: "http://api.worldbank.org/v2/en/indicator/EN.ATM.CO2E.PC?downloadformat=excel"
format: xls
headers: 4
Pipeline (cont.)
-
run: stream_remote_resources
cache: True
-
run: set_types
parameters:
resources: global-data
types:
"[12][0-9]{3}":
type: number
-
run: dump.to_zip
parameters:
out-file: co2-emisonss-wb.zip
Running it
$ pip install datapackage-pipelines
$ dpp
Available Pipelines:
- ./worldbank-co2-emissions (*)
$ dpp run ./worldbank-co2-emissions
INFO :Main:RUNNING ./worldbank-co2-emissions
INFO :Main:- lib/add_metadata.py
INFO :Main:- lib/add_resource.py
INFO :Main:- lib/stream_remote_resources.py
INFO :Main:- lib/dump/to_zip.py
INFO :Main:DONE lib/add_metadata.py
INFO :Main:DONE lib/add_resource.py
INFO :Main:stream_remote_resources: OPENING http://api.worldbank.org/v2/en/indicator/EN.ATM.CO2E.PC?downloadformat=excel
INFO :Main:stream_remote_resources: TOTAL 264 rows
INFO :Main:stream_remote_resources: Processed 264 rows
INFO :Main:DONE lib/stream_remote_resources.py
INFO :Main:dump.to_zip: INFO :Main:Processed 264 rows
INFO :Main:DONE lib/dump/to_zip.py
INFO :Main:RESULTS:
INFO :Main:SUCCESS: ./worldbank-co2-emissions
{'dataset-name': 'co2-emissions', 'total_row_count': 264}
Custom processors
# Add license information
from datapackage_pipelines.wrapper import ingest, spew
_, datapackage, resource_iterator = ingest()
datapackage['license'] = 'CC-BY-SA'
spew(datapackage, resource_iterator)
What else?
- Advanced standard library
- Scheduled execution
- Execution dashboard
- Pipeline dependencies
- Caching results