---
tags: ols
---

# Plenoptic

`This is a template for you to summarize your OLS journey so far and the project you worked on during the entire period of OLS-1.`

Example you can draw from: https://openlifesci.org/posts/2019/10/21/culture-track

## Project background
_Tell us what motivated you to work on this project._

First, should introduce What `plenoptic` is. From the `README`:

In recent years, [adversarial
examples](https://openai.com/blog/adversarial-example-research/) have
demonstrated how difficult it is to understand how models make sense
of the images they view. The space of all possible images is
impossibly vast and difficult to explore, so that even training a
model on millions of images represent just a small fraction of all
that could be shown. `plenoptic` is a python library that provides
tools to help researchers better understand their models by using
optimization to generate novel images. These images allow researchers
to gain a sense for what features the model ignores and what it
considers important, and they can be used in experiments for further
model testing and validation.

To get a little more technical, all models have three components,
inputs `x`, outputs `y`, and parameters `θ`. When working with models,
we typically either simulate, by holding `x` and `θ` constant, and
generating predictions for `y`, or fit, by holding `x` and `y`
constant and using optimization to find the best-fitting `θ`. However,
for optimization purposes, there's nothing special about `x`, so we
can instead hold `y` and `θ` constant and use optimization to
synthesize new `x`. Synthesis methods are those that to do this: they
take a model with set parameters and generate new images in specific
ways. They allow you to better understand the model by determining
what it considers important and, crucially, what it ignores, as well
as generating novel stimuli for testing the model.

My motivation to work on `plenoptic` came from several sources:

1. As the above (hopefully) explains, I believe there is a scientific
utility to this package, a problem it can solve.
2. Several synthesis methods have been developed in the [Laboratory
for Computational Vision](https://www.cns.nyu.edu/~lcv/) over the
years, but while other researchers have expressed interest in them,
the only method that was really adopted by researchers outside the
lab was the one whose code was available (as MATLAB code on Github)
and, even then, people tended to only use it for the limited
use-case that was available. I wanted to make an easier-to-use,
more general implementation of these methods availab.e
3. The development of differentiable programming frameworks, such as
`pytorch` and `TensorFlow`, in recent years has made the core
technical difficulty of the synthesis approach (computing the
model's gradients) trivial to solve, opening up the possibility of
the afore-mentioned general implementations.
4. In our lab, like many labs, a lot of effort is wasted: graduate
students or postdocs leave the lab and no one knows where the code
they wrote is, how to use it, or understands it. Thus, we either
waste time re-implementing earlier ideas or abandoning them.
5. Having this library available would allow researchers in the lab to
easily and quickly use the methods within in order to better
understand their models, instead of committing to a single method
for time reasons.
6. A group of us, myself included, wanted to improve our software
engineering skills.

## Expectations from this program
_What did you expect or hope to learn from participating in OLS?_

When I started the OLS program, I had two major sets of goals: one
technical and one focusing on community. Technically, I wanted to
learn more about how to build an effective open source library: how to
write effective tests, documentations, and tutorials. I also wanted to
make sure there wasn't anything we should be including that we weren't
aware of, since all of our learning about software engineering and
open source projects had been haphazard. For the community, there was
a small group of us working on this project, but we're all in the same
lab, so I wanted to learn more about how to make the package useful
for a broader community and bring other contributors in.

## Goals set at the beginning of the project
_Missions and goals set in the beginning of your project in OLS_

My goal had been to finish the implementation of the core of the
project: the four synthesis methods described above, as well as a
small collection of simple models.

## Key understanding and accomplishments
_List 2-3 things you've learned or accomplished whilst participating._

I learned a lot about how to build a community, in a variety of
different ways. Some specific examples:

1. I got a lot of feedback on writing a clear and accessible vision
statement. The project is pretty niche and targeted at a specific
research community, so getting feedback from people from a variety
of backgrounds about how to make it understandable was very helpful
2. Added a code of conduct and contributing sections, as well as
thinking about best first issues and how to get people involved.
3. Make the roadmap for the project more explicit, thinking
specifically about the components we want to include, what's inside
and outside the scope of the library, and how to prioritize.

## The main goals achieved in this project
_Was this the same as goals above, that you originally set?_

Did not end up accomplishing the goals I set out: made some progress
on them, but the semester being busy (and COVID-19 happening) meant
that everything moved more slowly than expected.

As I mentioned in last section, most of what I accomplished was
related to thinking about how our project will interact with the
community, how to make it accessible, and how to build a community.

One of the most valuable results of this program was the discussions
it sparked. Because of what we were learning throughout the process, I
had several discussions with my lab mates, making our values and goals
explicit. For example, we had different conceptions about what it
meant for this package to be open, which we didn't realize until we
really discussed it in detail. As a result, we made a specific plan
for what we want to include in the package before we open it (it's
currently a private Github repo), how we'll publicize it when that
happens, and how we'll add content moving forward.

### The initial steps
_Starting out - what did you do to explore the problems / project?_

The initial shape of `plenoptic` started over a year ago, so I started
OLS with sense of what the project was and how it would look at
release, once we had our minimum viable project. The first valuable
steps of this program started with our first assignment: writing the
vision statement. This started with me and the other members of the
project drafting something concise and then iterating on it and
expanding it with feedback from other members of the OLS cohort.

### What elements helped you get there?
_Discussions, connections made, activities carried out, mentors consulted, other?_

Basically all of the program was valuable and helped me make progress,
in different components. The discussions brought up general principles
that the assignments helped make relevant to my project and
concrete. My mentor, Rodrigo, gave me feedback on the project and
pointed out many helpful resources.

## Next steps

### My immediate next step is to...
_Whether you're launching something, bringing someone new in, adding governance, or writing up a project plan or case study - what's happening next?_

The immediate next step is to continue working on implementing the
components of our project discussed earlier and working on the
tutorials, documentations, and tests. That will enable us to release a
v1.0, which we're aiming to do by the end of the summer. We'll also
need to expand the contributing guidelines a little more.

### Longer term tasks
_What will be the long term goals and sustainability plans - this can also be about starting your project if you did not do that already_

Longer term, our goal is to publicize this: submit to the Journal of
Open Source Software and present at relevant conferences. As we're
preparing those, we'd like to reach out through our network and get
feedback from other folks in the lab and researchers in similar labs
to see how accessible they find the library without us guiding them
through it.

### Staying connected
_Do you plan to stay in touch with the OLS community or other members? If yes, how do you think you can do that?_

I do plan to stay in touch, remaining on the gitter / Google Group. A
lot of people's projects sound very exciting, so I'm looking forward
to seeing how they develop!

## Special mentions and acknowledgements
_Please mention ideas, lessons and people who helped you achieve whatever you could in this project so far_

I'd like to specifically thank my mentor Rodrigo for the help he's
given me throughout this program, providing lots of feedback on
assignments and pointing out a variety of helpful resources for
developing an open source project. The members of the community have
also provided lots of helpful feedback throughout and, of course, I'd
like to thank Malvika, Yo, and Berenice for putting such a great
program together!