owned this note
owned this note
Published
Linked with GitHub
# GSOC 2018 Summary: Interactive 3D Visualizations of Human Brain Activity in Jupyter Notebook Using Ipyvolume
## Project goals
The aim of the `ipysurfer` project is to improve interactive, 3-D visualization of brain surfaces using MNE and the Jupyter Notebook. It can be regarded as an alternative to `mayavi` and `PySurfer`-based human brain visualization, and is built on top of other open source tools such as `ipyvolume`, `bqplot`, `pythreejs`, and `ipywidgets`.
## Project overview
During the GSOC period, the initial concept of how cortical surfaces can be visualized in Jupyter Notebook was transformed into the `ipysurfer` package. Two related packages, `MNE-Python` and `PySurfer`, were used as the guiding lines during the development process. As a result, I have used similar or the same naming conventions for code styling, classes, methods and parameter names. Most of the original features supported by `MNE-Python` and `PySurfer` are now implemented in `ipysurfer`.
All code is available in a [public GitHub repository](https://github.com/mne-tools/mne-gsoc2018-3d).
### Changelog
Below are the major features that this project has created in `ipysurfer` or added to `ipyvolume`:
- plotting of cortex meshes in Jupyter Notebook using `ipyvolume`, `ipysurfer.Brain` class should be used for this purpose. A user can plot the left or right hemisphere alone, both hemispheres on the same plot or on separate plots. [Check in Binder](https://mybinder.org/v2/gh/mne-tools/mne-gsoc2018-3d/master?filepath=examples%2Fipysurfer.ipynb)
![Both hemis](https://i.imgur.com/2YtyZuG.png)
![Split hemis](https://i.imgur.com/iwnwdYM.png)
- displaying of activity patterns as surface color variations of a mesh, a user should apply `ipysurfer.Brain.add_data` method for this purpose. Visual controls over color map used for data visualization are provided by the same method. [Check in Binder](https://mybinder.org/v2/gh/mne-tools/mne-gsoc2018-3d/master?filepath=examples%2Fipysurfer.ipynb)
![Color bar](https://i.imgur.com/cir0X3o.gif)
- plotting activation data as dependency over time with visual controls, for further information look at `ipysurfer.TimeViewer` class. [Check in Binder](https://mybinder.org/v2/gh/mne-tools/mne-gsoc2018-3d/master?filepath=examples%2Fipysurfer.ipynb)
![Time viewer](https://i.imgur.com/pzxBYBA.gif)
- creation of a function for plotting `MNE` objects data, instead of raw data, see `ipysurfer.plot_source_estimates`. This function can be integrated into `MNE`. It is built with classes mentioned above, so a user can display color map controls or time viewer effortlessly. [Check in Binder](https://mybinder.org/v2/gh/mne-tools/mne-gsoc2018-3d/master?filepath=examples%2Fplot_stc.ipynb)
![Divergent color map](https://i.imgur.com/Z9JBE3j.png)
- creation of a stand-alone package that combines code for the above mentioned features, i. e. `ipysurfer`.
- Examples of the package usage and code documentation were added as well.
**Enhancements to `ipyvolume`**
`ipyvolume` is a package for visualizing 3-D data in the Jupyter Notebook. `ipysurfer` utilized this package heavily, and several improvements were made to `ipyvolume` as a part of this GSOC project.
- addition of alpha blending support to `ipyvolume`, see [#139](https://github.com/maartenbreddels/ipyvolume/pull/139), [#145](https://github.com/maartenbreddels/ipyvolume/pull/145).
- code styling improvements and unit testing configuration changes, see [#146](https://github.com/maartenbreddels/ipyvolume/pull/146), [#142](https://github.com/maartenbreddels/ipyvolume/pull/142).
- a partial fix of problems with camera orientation changes in `ipyvolume`, see the [problem #156](https://github.com/maartenbreddels/ipyvolume/issues/156) and the corresponding [pull request #158](https://github.com/maartenbreddels/ipyvolume/pull/158).
## Whats left to do
Several things that should be tackled after the end of the program:
- integration of `ipysurfer` into `MNE`, [#23](https://github.com/mne-tools/mne-gsoc2018-3d/issues/23).
- improvement of animation performance, corresponding [issue](https://github.com/mne-tools/mne-gsoc2018-3d/issues/33).
- full solution for the problems with displaying different views in `ipysurfer` should be found, see issues [#31](https://github.com/mne-tools/mne-gsoc2018-3d/issues/34) and [#156](https://github.com/maartenbreddels/ipyvolume/issues/156).
- improve support for other visualization parameters used in `MNE`/`PySurfer` packages, see [#14](https://github.com/mne-tools/mne-gsoc2018-3d/issues/14), [#28](https://github.com/mne-tools/mne-gsoc2018-3d/issues/28) and [#30](https://github.com/mne-tools/mne-gsoc2018-3d/issues/30).
## Examples
Examples of possible ipysurfer usage can be found via this [link](https://github.com/mne-tools/mne-gsoc2018-3d/tree/master/examples). You can also see a presentation below that shows major features. This presentation was generated from `ipysurfer` sample notebooks, and can be found [here](https://github.com/mne-tools/mne-gsoc2018-3d/tree/master/doc/slides).
![Presentation](https://github.com/mne-tools/mne-gsoc2018-3d/blob/master/doc/slides/slides_pres.gif?raw=true)
## Thanks and credits
I would like to thank my mentors [Jean-Rémi King](https://github.com/kingjr), [Eric Larson](https://github.com/larsoner) and
[Chris Holdgraf](https://github.com/choldgraf); [Alexandre Gramfort](https://github.com/agramfort), [Maarten Breddels](https://github.com/maartenbreddels) and other MNE-Python, ipyvolume and bqplot developers for helping me during my work on this project.