Guoning's Notes for RNN Project

# Guoning's Notes for RNN Project [![hackmd-github-sync-badge](https://hackmd.io/4qbnTwHDSZCnZPeb4WJjRg/badge)](https://hackmd.io/4qbnTwHDSZCnZPeb4WJjRg) This note is for Guoning to keep track of the progress and the things he needs to do for the project. The note will be updated regularly after meetings.   ## Meeting Notes ### 2025-04-18 with Aaron, Badr and Michele **What Guoning has done:** 1. Modify “PC bumped” tests to each rep value in the common PC spaces - done 2. Hit/miss/cr/fa for behavior comparisons - done 3. Recurrent weights changing pattern between E dom and balanced? - conducted replacement comparison, not pattern analysis yet 4. PC change / loading vector correlates to network response to stim/choice, and other network performance factors we have - done 5. What’s the physical meaning of PC1? Why balanced network changes much in PC1 while E dom not? – not done yet, need to make sure the consistency between runs **What we saw from the data:** 1. We see a discrepancy between the results of the new 32 repetitions run and the previous 16 repetitions run. It could be because of the initialization. The previous run could be a "worse" run by looking at the firing rate and the performance. The new run is more consistent with the earlier runs by Aaron. This need to be confirmed by running more initializations. 2. In the new 32 reps run, we see: - Bumping up PC1 can reach balanced performance - Bumping up other PCs does not improve performance - Replacing output weights increases the performance to similar as balanced - Replacing recurrent weights adds a lot more variance, there are both good and bad performance networks, not robust - Replacing all weights gives one best performance ever seen in post-reversal task 3. The hit/miss/cr/fa analysis looks similar to the performance analysis. 4. The PC1 loading vector does not correlate with the units' response to stim/choice. **Next steps:** 1. Check the consistency of the new run with the previous runs. ### 2025-04-04 with Aaron and Badr **What Guoning has done:** 1. Tests for PC space vs performance and other factors that potentially contribute to the reversal task performance of E dominant / balanced networks **What we saw from the data:** 1. Main factors for different reversal performance between E dominant and Balanced networks: - W_out diff in PC1? Yes - W_out diff in PC2/3? Not much - W_out diff in other PCs? No 2. Why not reaching performance as high as balanced networks - Other factors than weights change in PC 1. They could be potentially related to the averaging effect, recurrent weights, or inhibition level. - PC1 bumped E dom network has already reached the performance of the balanced network with the effect of averaging - Compensate the effect of averaging by replacing the output weights of E dom network by the output weights of the balanced network. It still not reaching the original balanced network performance. This shows that the effect of averaging is not all that matters. - Recurrent weights also important: The network is good enough to perform the reversal task with the weights trained from balanced network, regardless of the inhibition level (comparison between bal.recur. & out. E dom and Balanced network performance) **Next steps:** 1. Modify “PC bumped” tests to each rep value in the common PC spaces. (This potentially better eliminates the effect of averaging.) 2. Hit/miss/cr/fa for behavior comparisons 3. What’s the physical meaning of PC1? Why balanced network changes much in PC1 while E dom not? 4. Check PC value change / loading vector correlation to network response to stim/choice, and other network performance factors we have 5. Recurrent weights changing pattern between E dom and balanced? ### 2025-03-21 with Aaron, Badr and Michele **Badr's Cosyne 2025 poster discussion** Figure changes and poster adjustments. ### 2025-03-10 with Aaron and Badr **What Guoning has done:** 1. Tests for swapping phase 2 weights - Swapped the output weights between the two networks after phase 2 training and tested the performance. The network with the same target output weights performed well, while the network with the reversal output weights performed poorly and can not distinguish the reversed target. This indicates that the output weights are critical for the network to adapt to the reversal task. 2. Further analysis on PCA results - Output weights change for the reversal network can be seen in a subspace that the same target network output weights do not change in - Network with better performance for reversal task has more significant output weights change in this subspace. **Next steps:** 1. Test the robustness of the PC spaces and the output weights change in those spaces across different repetitions. Particularly, explore the PC 2, 3 and 4 spaces. ### 2025-02-28 with Aaron, Badr and Michele **What Guoning has done:** 1. Further analysis on PCA results: - Compared phase 2 output weight subspace - Fixed phase 1 and compared only phase 2 - Ran same analysis on network with low inhibition 2. Identified seeding problem – spawning of worker calls RNGs. A fix is to set a seed for each worker. **What we saw from the data:** 1. The changes for output weights in phase 2 training for same target and reversal networks go in orthogonal directions. More specifically, the reversal network output weights changes mainly in PC2 and the same target network output weights changes mainly in PC3. This shows that there is a particular direction of output weights are trained to change to for adapting to the reversal task. 2. PC1 seems to mainly separate the difference of the output weights between two networks after phase 1 training. Working on to eliminate this effect since we have the same phase 1 training for both networks. 3. When fixing phase 1 training for two networks, the first 2 components does explain most of the variance, and the two networks change in not exactly orthogonal but very different directions 4. However, when comparing the output weights change for the network with low inhibition, the output weights change in the same direction for both networks. This indicates that the network with low inhibition is not able to adapt to the reversal task. **Next steps:** 1. Test hypothesis: - The subspaces of output weights that change the most in are different. (Look closer to the components of those subspaces) - The output weights change for the "bad performance network" (low inhibition) is in the same direction as the "good performance network", but not changing enough to adapt to the reversal task. (Add more training trials?) - The inhibition level could be too critical for the network to adapt to the reversal task in to an extent that the output weights change does not matter. (Test phase with different inhibition levels?) 2. Weight swapping analysis: after phase 2 training, swap the recurrent/output weights between the two networks and see how the network performs. ### 2025-02-07 with Aaron, Badr and Michele **What Guoning has done:** 1. Further analysis on PCA results: - Added PCA components eigen value figure (how much each of all components explains the variance) - Added the comparison of decoding performance and responsiveness pair plot for each quadrant of the network at each phase (8 plots for each network) - Looked at the principal components space for recurrent weights to find which specific connections have higher explanatory power for the variance **What we saw from the data:** 1. In the quadrant comparison plots, two pairs are different for the two networks: both high decoding performance CR and NCR units have less decoding performance in the reversal network after phase 2, while there is not much change in the same target network. 2. In PC 1 subspace of recurrent weights, I to E connections has a larger proportion compared to E to E connections. **Not finished yet:** 1. Random seed fixing problem 2. Target/non-target tone decoding performance problem **Next steps:** 1. Come up with a way of comparing the output weights change during training between the two networks. How's output weights changing that is the key to the network reconfiguration so that the network can do the reversal task? 2. Aaron's suggestion: fix recurrent and train only the output weights, see if the network can learn the reversal task. 3. Continue debugging previous problems 4. Work with Aaron to write up the manuscript ### 2025-01-24 with Aaron, Badr **What Guoning has done:** 1. Ran PCA to see weight variance subspace 2. Tried to run more detailed decoding performance analysis **What we saw from the data:** 1. The PCA analysis shows that the first 2 components explain most of the variance in the recurrent weights. A low dimensional space explains the variance of recurrent weights well, and the two network have similar trajectory. 2. The first two or even three components of the output weights does not explain the variance of output weights well, and the two network have very different trajectories. **Next steps:** 1. Look at the principal components space for recurrent weights, find which specific connections have higher explanatory power for the variance 2. Add PCA components eigen value figure (how much each of all components explains the variance) 3. Add the comparison of decoding performance and responsiveness pair plot for each quadrant of the network at each phase (8 pair plots in total) 4. Random seed fixing problem 5. Target/non-target tone decoding performance problem ### 2025-01-10 with Aaron, Badr and Michele **What Guoning has done:** 1. Replotted the decoding performance vs responsiveness figures: - Adjusted the terms and the plot schemes to be consistent with the previous papers and figures - Made a pair plot format combining phases - Added threshold indication for CR and NCR units - Added threshold indication for highly informative and less informative units 2. Calculated the decoding performance for - reversal network phase 2 decoding performance for the previous tone - same target network phase 1 decoding performance for the new tone **Next steps:** 1. Separate CR/NCR units and highly informative/less informative units in the pair plot ### 2024-12-19 with Badr and Michele **What Guoning has done:** - Ran decoding algorithm (single units) for the networks - Compared the informativity (decoding performance) for different phases in both networks - Compared the weight change with decoding performance **What we saw from the data:** - Decoding performance of the output units decreases after PostRev; - Changes in decoding performance for NCR can be noticed: a decrease, but not very clear on the figures shown; will need better representation **Next steps:** 1. Unify the terms and the plot schemes referring to the previous papers and figures 2. Replot the decoding performance vs responsiveness figures: - Combine phase 1 and 2 and make a *pair plot* format - Separate CR and NCR units - Separate highly informative and less informative units 3. Calculate the decoding performance for - reversal network phase 2 decoding performance for the previous tone - same target network phase 1 decoding performance for the *new* tone --- ### 2024-12-13 with Aaron, Badr and Michele **What Guoning has done:** - Replotted some figures shown last time. Went through some Aaron’s previous notebooks with figures not shown in the manuscript - Compared unit-level response (FR mod) in different tasks - Set up Julia env so that small instances can be run efficiently in local, which enables fast testing and debugging **What we saw from the data:** 1. Recurrent weights are increasing after phase 2 training comparing to phase 1 for both tasks. Among recurrent weights, both I to E and E to E are increasing, while I to E increases a bit more. Note that I to I and E to I connections are not changing by the network setup. The two task regimes perform similarly in terms of recurrent weights. 2. Output weights of the reversal network are changes after phase 2, while the same target network’s remain very similar 3. The responsiveness for individual units is not much task-dependent **Our hypothesis about the network reconfiguration:** - Output weights change the most, feedback through recurrent connections, causing recurrent weight changing, which reconfigures the network to do the reversal task. - Even though the output weights seems to play a more important role in the network reconfiguration, the recurrent weights are also changing, and this subtle effect could be as important for the network to adapt to the new task. **Two ways of addressing the hypothesis:** 1. Break down the unit population to analyze the change in weights and responses: - The units with higher information content are more likely to change their weights and responses. The information content could be measured by the stimulus/choice decoding. - Previous results also showed that the CR/NCR separation is a factor in the network reconfiguration. We can combine the CR/NCR separation with the information content and check units response and connection weights for those categories. 2. Dimensionality reduction methods, e. g. PCA, to see if specific type of units or connections are more important in network reconfiguration. **Next Steps:** 1. Try to run single unit decoding algorithm (Aaron said it will take some time). 2. Continue with the PCA analysis as an example of dim reduction. --- ### 2024-12-06 with Aaron and Michele Our main interest: how the network reconfigures to do the reversal task. 1. Recurrent weights for reversal network and same target network are very well correlated (pretty much the same). The I to I and E to I weights are exactly the same as expected. 2. The I to E and E to E weights are mostly increasing after training phase 2, for both reversal and same target networks. And the I to E weights are increasing a bit more than the E to E weights. 3. The output weights are not highly correlated between the reversal and same target networks, $ r = 0.50 $ after phase 1 and $ r = 0.27 $ after phase 2. Especially after phase 2, the output weights have a weak correlation. And when comparing the same network across different phases, the output weights are highly correlated for same target network, $ r = 0.84 $, while the correlation is low for the reversal network, $ r = 0.23 $. This could indicate that the network weights change to adapt to the new task, and the output weights are more task-specific than the recurrent weights. Aaron's going to help with checking whether these align with the previous results. Next Steps: - Try to figure out why the weights depend more on the training process, but not the task. And why is there an increased recurrent weight after training phase 2 while there is a decreasing response as shown previously. - To have a better idea of how weights evolved, could try dimensionality reduction methods, e. g. PCA, for investigating if there is a subspace and along which the weights changes the most during learning. And compare the subspaces for same target and reversal tasks - Take a look at the unit-level response to different targets. - Can these be compared with the data in vivo? --- ### 2024-11-22 with Aaron, Badr and Michele 1. The output weights are not well correlated. - Need to compare the baseline 2. The responsiveness depends on the initial weights and STDP conditions, not depend on the tasks. Next Steps: 1. Implement a random seed to fix the inter-trial interval and the target selection on each individual trial. 2. Look at the recurrent weights - Have each data point be a connection - Plot IE connections on one, EE on another (sanity check II and EI connections, which are going to be the same since they are not updating) 3. Dig into to output weights some more - We need to see correlations between phase 1 and phase 2 for the same target and reversal - We need to see correlations between weights after phase 1 (this is a sanity check, it should be highly correlated since reversal and sameTarget are identical after phase 1) --- ### 2024-11-08 with Aaron, Badr and Michele Discussion about the current manuscript preparation and Guoning's more specific things to do. >Fig 4h > > - need to look at post-rev output dimension change relative to pre-reversal How is network reconfiguring itself to do the task? > >Does the network >1) output weights stay the same but when it learns the new task the recurrent dynamics reconfigures (we have older results on this). >2) How to account for response type specificity (contributions of CR/NCR)? Is there some stability, or are we changing which cells are responsive, etc. Guoning examines both of these with current model > >Target specific subpopulations of CR/NCR, here’s a cell that stays the same, if we perturb it, what happens? Running that for all 3 inhibitory conditions, not just for balanced network. If you see homeostatic changes, then network will reconfigure outputs versus recurrents. #### Finished. ~~To-do~~ 1. Run separate scripts for different tasks - initialize with same random weights, recurrent weights and feedback weights - training phase 1 (500 trials training, and 400 testing trials) and phase 2 (500 training and 400 testing) without changing the targets, to control the training effect comparing the real reversal - real reversal: same training and testings but change the targets, phase 2 post-reversal - compare the weights: correlations were used previously, start from visualization, use multiple ways to sanity check, scatter plot of correlations between the control output weights and the reversal output weights, add line of unity to compare the changes 2. Figuring out responsive non-responsive units performance - existing files for analysis different units - coloring the dots by CR/NCR pre-reversal in the scatter --- ### 2024-11-01 with Aaron, Badr and Michele Discussion about the current manuscript preparation and Guoning's process on getting started with the project. 1. Aaron had updated the EDFs, along with some notes for them. 2. Decided to write up the results parts of the manuscript, Michele and Badr will need Aaron's help at some point. 3. Guoning will fix the environment set up and run analysis we are interested in. #### Finihsed. ~~To start up~~ 1. ~~Set up python env~~ 2. ~~Set up Julia env~~ 3. ~~Test running a script~~ 4. ~~Make a note for setting up the envs~~ 5. ~~Set up all envs on CRC cluster~~ 6. ~~Make a learning summary if possible~~ Make this note a learning summary? ### 2024-10-31 with Aaron Mainly about the Method part in Nat. Comm. 2024 paper 1. Why not scaling up the network being a potential direction? Think about the purpose first. We want to use the model to answer questions that could be hard implementing in experiment in vivo. Scaling up could help in multiple task learning. But for now improving task accuracy is not a first thing that we want to explore. 2. Questions in the Nat. Comm. paper method part: - see annotations on paper 3. Some general questions: - Do we have a rough idea about how many E and I neurons there are in vivo? (Check ref 42, 43) Dale's law - For the three ways to change the level of inhibitory current, why not change those constants that affect the updating of the weights? They are more neuroscientifically explainable and easy to manipulate and interpret. --- ### 2024-10-25 with Aaron, Badr and Michele Discussion about the current manuscript preparation and near future work. #### We probably need more interpretations for balance network structure analysis - How weight changes in pre and post reversal learning - Expand the analysis of weight in addition to dimensionality analysis. Dimensionality analysis used now is a good starting point but we need more interpretations. - Analysis could be: - analysis of individual responsibility/variability of the neurons to different targets - analysis about "which neurons are contributing more to the outcome population" - think about population analysis that is hard to do in vivo - Discussion about EDFs - Putting EDF1 possibly in the main text? - Some fixings for the subtitles and make captions #### Guoning's possible directions 1. How the network performs relates to how we get the balanced network. Try to explore the difference. 2. Try to understand the weight changes in the network. 3. Explore more about the contributions of neurons to different outcomes. #### Guoning's near future work 1. Talk more to Aaron about the model and analysis 2. Have a better understanding of the parameters 3. Rerun the analysis of weight to get more familiar with running the scripts and to make sure it is up to date --- ### 2024-10-24 with Aaron #### Key points from previous work 1. Model achieved excellent performance before and after reversal 2. Model included units with diverse response profiles consisting of both CR and NCR neurons mirroring what we observed in vivo 3. The level of inhibition in the network is critical for reversal learning capacity 4. We have multiple ways to change the level of inhibitory current: - increase the inhibitory postsynaptic decay rate - extend excitatory STDP depression window (LTD window) - increase inhibitory learning rate 5. Different relative level of excitation and inhibition results in three types of networks: - excitation-dominated - balanced - inhibition-dominated 6. These networks have different learning capacities - excitation-dominant: best pre-reversal performance - balanced: lower pre-reversal but much higher post-reversal - Inhibition-dominant: poor in both pre- and post-reversal. 7. To understand functional roles of CR and NCR neurons, selective inactivation was performed - E dominant network: inactivating either CRs or NCRs has the same effect - Balanced network: NCRs were more critical for recurrent processing, CRs were more important for outputs - ~~What about I dominant network?~~ Because the performance is poor in the beginning so removing doesn't make sense. 8. In balanced networks, rapidly switching between being excited and inhibited could be the mechanism allows the network to remain dynamic by rapidly switching between being excited and inhibited #### General questions 1. Where should I start to hand on the project? Went through an example script and the structure of the code. Recommended to read the functions and class that implement the methods. 2. What is the current focus you are working on? What are the future directions? Cross/multi-region? Finished the reversal part. Could try multiple reversal tasks? Teaching signals? 4. What is inhibitory postsynaptic decay rate? How do you manipulate it? Notice in poster pre: `we indirectly manipulate the level of inhibition in the network by manipulating the inhibitory postsynaptic decay rate` 5. What is LTD window? 6. What is inhibitory learning rate? Inhibitory postsynaptic decay rate, LTD window, and inhibitory learning rate are all parameters that can be manipulated to change the level of inhibition in the network. 7. What kind of dimensionality analysis did you do? Same as Toth et al. paper. #### Technical questions - ~~Software environment package versions~~ #### Still needs time to understand - ~~How is the current calculated based on the weights of the network? How does this relates to the parameters like decay rate, learning rate, etc.?~~ ### Important points Aaron mentioned in July - Go into details of the method part of Nat. Comm. 2024 paper - Code base V2.0 or V2.3 is a good starting point ## Glossary | Term | Definition | Notes | |------|------------|-------| | EPSC | Excitatory postsynaptic current | Defines E to E input current. | | IPSC | Inhibitory postsynaptic current | Defines I to E input current. | | STDP | Spike-timing-dependent plasticity | This is how the network is designed to update the weights. All inputs to excitatory cells were plastic. Inputs to inhibitory units remained fixed. Distinct homosynaptic mechanisms for E-to-E synapses and I-to-E synapses are used. | | FORCE | First-order reduced and controlled error | A method for training recurrent neural networks. | | LTP | Long-term potentiation | Occurred when a presynaptic cell fired before a post synaptic cell. | | LTD | Long-term depression | Occurred when a presynaptic cell fired after a post synaptic cell. | ### Some jargons, conventions, and notes in the codebase 1. Unit index: - input units (Excitatory neurons): 1 - 200 - output units (Excitatory neurons): 201 - 800 - Inhibitory neurons: 801 - 1000 2. Some variable names meaning: - s - stimulus - a - action (choice) - np - nose poke (go trial) - w - withdrawal (no-go trial) 3. Teaching signal was used for test purposes. This can be explored after having a good understanding of everything else.