---
# System prepended metadata

title: processing pipeline - next up

---

# processing pipeline - next up

## issues:
* [ ] XML reading memhog
  details in branch xml_relator
  -> Stubi?
  - https://stackoverflow.com/questions/7171140/using-python-iterparse-for-large-xml-files/7171543#7171543
  - https://stackoverflow.com/questions/22380990/how-do-i-free-up-the-memory-used-by-an-lxml-etree
  - https://pretagteam.com/question/why-is-lxmletreeiterparse-eating-up-all-my-memory
  
* [ ] design question
  * [ ] "Population.strategy"
    integration of strategy methods into classes
    * [ ] _classmethod:_ handover still function
      - need to id ismethod() on register, naming appropriately
      + can write clean, readable code as long as object itself 
        is returned or whatever is defined
    * [ ] _staticmethod:_ handover is function
      - id method is crufty, unelegant
  * "strategy"
    * _normal function:_
      still possible as user-provided thing
    
## goals
### Gerald
* Levitate scenario preps in pipeline
  * population exemption for homes in tolling zone
    - select population
    - manipulate
  * network
    - select links by frc/osm:highway
    - manipulate 
 
### Stubi
* [ ] the XML reading thing for XMLRelator
* [ ] poetry @Windows -> 1.1.12 self update
* [ ] feedback runde pipeline
    - my drafts
    - IF design

### Jo
* contribute code to load polygons  
  read_polygon/s
  `getModePerArea.py`
  class `ModeArea`
  * [ ] done how ?
    * polygon input: shapefile
    * shapely
    * fiona: read formats (shapefile)
        alternative to geopandas (tough to install [@Johannes])
  * [ ] prep for strategy declaration
  * [ ] in a new file `formats/polygon.py`

## status

### XML handling
* network
  * got old NetworkXML, *working* rather fine
  * [ ] shift to XMLRelator ?
* events
the file size problems

## tasks
* network
* population

### Gerald
* [x] merge to main branch
* [x] finish 1st version of XMLRelator
  * [x] data only
  * [ ] full XML content
* [ ] example
  * [ ] population inhabitants within polygon
    for now: defined by first homes (assumed to be first activities, which is MATSim requirement), of the selected plan of each person
* [ ] polygon reader
  -> Johannes code
  
## examples:
### for Stubi
1. exp. plans:
  * starting before 8:00
  proto_code: (XMLRelator steps, **NOT** the final Plans object/class)  
  ```python
  pick_legs = table['leg']['dep_time'] <= '08:00:00'
  parent_ids = get_parents(pick_legs.index)
  picked_plans = table['plan'].loc[parent_ids]
  ```
  
2. link volumes:
  * all links with daily volumes
  proto_code: (XMLRelator steps, **NOT** the final object/class)  
  ```python
  pick_legs = table['leg']['mode'].isin('car drt1 drt2 ...'.split())
  child_route_ids = get_children(pick_legs.index, type='route')
  all_crossed_linkIds = table['route'].loc[child_route_ids].__text__.str.split()
    all_crossed_linkId_counts =.value_count()
    ```



