# EPF5 Dev Update - Week 11
## Weekly Highlights
Attended EPF5 weekly standup, office hours, Grandine standup, and our in-house meeting which happens three times a week (Monday, Wednesday, and Friday). I also had an insightful chat with Jimmy Chen.

## Performance and Memory Analysis Report
### Objective

The primary objective of this report is to present the findings from recent performance and memory analysis conducted on the Grandine, with a focus on identifying potential bottlenecks and memory management issues, particularly within the context of the Tokio runtime. The analysis was carried out using Flamegraph, Valgrind, and Heaptrack.

**Heaptrack**

**Valgrind**

**Flamegraph**

### Executive Summary:
The client shows significant activity in asynchronous operations, thread management, and database interactions, with some concerns regarding memory management and performance bottlenecks. These are primarily concentrated within the Tokio runtime, particularly in task scheduling, thread management, and context switching. The key findings include:

* **High Overhead in Tokio Runtime:** Tokio's task polling, context setting, and thread management functions consume a large portion of execution time. This indicates inefficient task scheduling and thread handling.
* **Memory Leaks:** The Tokio runtime leaks memory and potentially wastes memory when managing tasks and threads. Both Valgrind and Heaptrack results confirm this issue.
* **Concurrency Overhead:** Functions related to thread and task management show high sample counts, revealing that concurrency creates substantial overhead. This likely leads to inefficient resource utilization.

**Profiling Data Insights**

The data above highlights several critical areas of concern within the Tokio runtime.

## Major Pointers

1. Concurrency and Threading Issues:

* Thread-related allocations (pthread_create, allocate_stack)
* High thread management overhead:
-`std::sys::pal::unix::thread::Thread::new::thread_start: 90.44%`
-`core::ops::function::FnOnce::call_once{{vtable.shim}}: 90.44%`
* Significant time spent on thread creation and management (90.44%)
2. Asynchronous Runtime (Tokio) Performance:
* Heavy reliance on Tokio's async runtime
a. Tokio Runtime Dominance:
`tokio::runtime::scheduler::multi_thread::worker::Context::run_task: 55.04%`
`tokio::runtime::context::scoped::Scoped<T>::set: 81.59%`
`tokio::runtime::blocking::pool::Inner::run: 82.91%`

b. Task Execution overhead:
`tokio::runtime::task::harness::Harness<T,S>::poll: 82.81%`
`tokio::runtime::task::core::Core<T,S>::poll: 82.81%`

c. Runtime Context Management:
`tokio::runtime::context::scoped::Scoped<T>::set: 81.59%`
`tokio::runtime::context::runtime::enter_runtime: 81.89%`

### Areas of Concern:

* Thread-related allocations (pthread_create, allocate_stack)
* MDBX database operations
* Tokio runtime and Rayon parallel computing library

1. **Tokio Runtime Performance:**
Key Functions:
`tokio::runtime::scheduler::multi_thread::worker::Context::run_task, tokio::runtime::context::scoped::Scoped<T>::set, and tokio::runtime::task::harness::Harness<T,S>::poll`
Task execution and context management functions consume over 80% of execution time.
Thread management operations significantly contribute to overall runtime.

2. **Memory Issues:**
Valgrind and Heaptrack detect memory leaks in task and thread management.
These leaks are likely due to tasks and threads not being properly cleaned up after execution, leading to accumulated unused memory over time. However, since the execution was manually stopped, these leaks may be due to incomplete cleanup processes that didn't have the chance to execute. Further investigation is needed under normal execution conditions to determine if these memory leaks persist.

3. **Concurrency Challenges:**
High concurrency overhead indicated by sample counts in scheduling and thread management functions.
Inefficient task and thread managementw will impacts both memory usage and performance.

### Recommendations
* Improve Thread Handling
* Reduce Concurrency Overhead
* Memory Leak Investigation i.e Memory Management in Async Code
* Blocking Operations Review
* Continuous Profiling and Testing