owned this note   changed 4 years ago  
Linked with GitHub

Turning Pandas DataFrames to Semantic Knowledge Graph - Cheuk Ting Ho

歡迎來到 PyCon TW 2021 共筆

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

共筆入口：https://hackmd.io/@pycontw/2021
手機版請點選上方 按鈕展開議程列表。
Welcome to PyCon TW 2021 Collaborative Writing
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Collaborative Writing Workplace：https://hackmd.io/@pycontw/2021
Using mobile please tap to unfold the agenda.

To start the collaborative writing here

Pandas Dataframe

  • Pandas is prety wonderful (especailly for tabular data form.)
  • Does not handle nested structure well

Nested Data

  • Nested data can be handled in graph format.

Semantic Knowlege Graph

  • A graph-structured data model
Turn Pandas Dataframe into Semantic Knowlege Graph
Preparation
  • Like extract data from a data lake, define a proper schema and data types.
  • Might need to restrict the size while loading data.
  • Data type matters.
  • to_dict() converts pandas to a dictionary.
  • NA value handling: skip record all together or make it optional.
Flattening
  • from_records load json records back into pandas dataframe.
  • json_normalize to flatten nested structures.

Demo

load csv to pandas dataframe and then load to terminus db.

Speaker and the relative information

Cheuk Ting Ho

  1. github
  2. twitter

TerminusDB

  1. tutorial repo
  2. python client
tags: PyConTW2021
Select a repo