owned this note
owned this note
Published
Linked with GitHub
---
type: slide
slideOptions:
allottedMinutes: 45 # Minutes alloted for a slide.
---
# Metaprogramming
---
## What is Metaprogramming?
- --
Metaprogramming is a programming technique in which computer programs have the ability to treat other programs as their data.
It means that a program can be designed to read, generate, analyze or transform other programs, and even modify itself while running.
---
# Runtime
vs
# Compiletime
---
## Run Time
- Reflection
_Using a runtime, **reflection** allows you to examine, inspect and even modify your codes own behaviour at run time._
- Metaclasses
_In OOP, **Metaclasses** allow you to override the default behaviour of classes within the language._
----
### Reflection
```python=
def square(number):
if not isinstance(number, float):
number = float(number)
return number**2
```
```python=dark
x = 5
def z():
return "Spam and Eggs"
if callable(x):
x()
if callable(z):
z()
```
----
### Use Cases of Reflection
- Serialization / Deserialization
- `dynamic` types
- Getting property information
- Modifying assembly context
---
## Compile Time
- Macros
_**Macros** allow for code generation at compile time by analysing the AST and generating new code from it_
- Templates
_**Templates** allow you to generate source code from a template file to do things such as compile time class generation_
----
### Macros
```csharp=
[HttpPost]
public async Task Create(Schedule schedule)
{ ... }
```
```csharp=
public class Schedule
{
[BsonId]
[BsonRepresentation(BsonType.ObjectId)]
public string Id { get; set; }
}
```
```rust=
#[derive(Serialize)]
struct Parameter<T> {
key: String,
#[rename="type"]
kind: String,
value: T
}
```
```rust=
let mut list = vec![0; 12];
let other_list = vec!["hello", "world"];
```
---
## Why use Metaprogramming?
- Code that writes code
- More efficient and less error prone than doing it manually
- Can change the behaviour of code without having to redeploy
- In a "No Code Development platform", code generation is massively important
---
## Where do we use Metaprogramming?
- TDK
- ~~Listeners V2~~ Connectors
- Kafka Consumers
- Running actions
- Any serialization / deserialization
- Browser actions
- Data Processor
- Apps
- etc...
---
## Let's take a look at an example :eyes:
---
## Data Processor
The Data Processor relies heavily on **_reflection_** to make sure that you can write new actions without having to touch any other area of the system
It is split into 3 parts...
---
1. Telling the UI what actions are available
This includes informing the UI of the controls needed for each action
----
This would typically require a JSON module that's stored in the database
```json=
[
{
"id":"cda452fb-a93a-4b21-b043-53b4c0abea0b",
"control":"@core-sdk/file-path",
"options":{
"label":"File Path",
"fileKind":"file"
},
"parameters":{
"value":"path"
},
"validation":{
}
},
{
"id":"24be1754-367e-4f78-bfca-8708d6db89ad",
"control":"@core-sdk/array",
"options":{
"label":"Lines to Add",
"showBlank":true
},
"parameters":{
"value":"matchText"
},
"validation":{
"required":true,
"requiredItem":false
}
},
{
"id":"d6e16cb8-2534-46b8-99f8-7ae78543813f",
"control":"@core-sdk/checkbox",
"options":{
"label":"Fail If File Does Not Exist",
"fallback":true
},
"parameters":{
"value":"failIfNotExists"
},
"validation":{
}
},
{
"id":"9266864a-9cec-4675-ad67-dc9d2628ab5a",
"control":"@core-sdk/checkbox",
"options":{
"label":"Fail on error?",
"fallback":true
},
"parameters":{
"value":"failOnError"
},
"validation":{
}
}
]
```
----
But we can use reflection:
```python=
def module_builder():
module_list = []
for _, mod in handlers.__dict__.items():
if isinstance(mod, ModuleType) and
mod.__package__.startswith('data_processing'):
module_list.append(mod)
return list(map(module_definition, module_list))
def module_definition(module):
return {
'name': readable(module.__name__),
'description': module.__docs__.description
}
```
----
We do something similar for the controls but we can access each functions signature to get the information we need:
```python=
# This gets the parameter info like key name
function.__parameters__.items()
# This gets the type annotations
function.__annotations__
```
----
This will inspect the following function signature:
```python=
def update_column(
table: pd.DataFrame,
column_to_update: Union[int, str],
new_column: List[Any],
overwrite: bool = "True|False"
) -> pd.DataFrame:
```
From the signature we can extract the function name, the parameter names and the expected types for the parameters
----
The UI then intereperets the type to design the action panel:
---
2. Returning documentation
This includes tool tips on each input too
----
Each function in the data processor has a _docstring_ which is used to generate the docs you see on the UI
----
In Python, you can access the docstring of a function using the `__doc__` property
```python=
action = locate(f'data_processing.handlers.{action_name}')
if not action:
raise Exception(
f"Handler with name {action_name} does not exist"
)
docstring = action.__doc__
parsed_docstring = docstring_parser.parse(docstring)
```
----
The update column action has the following _docstring_
```python=
def update_column( ... ) -> pd.DataFrame:
"""
Overwrite an existing column with new values, if Overwrite is True then it will overwrite the entire
column, if Overwrite is False then it will only overwrite blank cells in the specified column.
You may specify the column to update either by the index or by the column label.
:param table: The table which contains the column you are updating
:param column_to_update: The label or index of the column you are updating
:param new_column: The column which will update the existing column as a list
:param overwrite: Overwrite specifies how the update is done, True to update the whole column, False to update
only the blank cells in that column
:returns: The table with the column at the specified position updated with new values
"""
```
----
This documentation is requested by the UI when the user selects an action and is rendered like so:
---
3. Executing the actions
How does it actually execute these steps?
----
We receive a model from the UI which is then interpreted and uses code generation
```json=
{
"handlers":[
{
"key":"remove_column_8549",
"handler":"remove_column",
"kwargs":{
"table":"{Result}",
"column":"Unnamed: 5"
}
},
{
"key":"remove_9fab",
"handler":"remove",
"kwargs":{
"source":"{remove_column_8549[:,4]}",
"to_remove":[
"%",
"\\\\+"
],
"mask":"None",
"regex":"True"
}
},
{
"key":"cleansed_data",
"handler":"update_column",
"kwargs":{
"table":"{remove_column_8549}",
"column_to_update":"Day change.1",
"new_column":"{remove_9fab}",
"overwrite":"True"
}
}
],
"outputs":[
"cleansed_data"
]
}
```
----
It takes this model and generates the code based on it
```python=
remove_column_a7d4 = remove_column(
table=Result, column="Unnamed: 5"
)
remove_74ea = remove(
source=remove_column_a7d4.iloc[:,4],
to_remove=['%', '\\+'],
mask="None",
regex=True
)
cleansed_data = update_column(
table=remove_column_a7d4,
column_to_update="Day change.1",
new_column=remove_74ea,
overwrite=True
)
```
----
Once it has generated the code, it will execute it
```python=
def execution_environment(
code: str,
deps: Dict[str, Any] = None,
output_vars: Optional[List[str]] = None
) -> Dict[str, Any]:
outputs = {}
namespace = globals()
local_vars = deps
exec(code, namespace, local_vars)
if output_vars:
for output_var in output_vars:
outputs[output_var] = local_vars.get(output_var, None)
return outputs
```
---
## Time for a Chris Demo :computer:
Google Drive
Gist
Clipboard
Sign in with Wallet
