# CHAT: Matlab advanced May 2021
## Asynchronous chat for the course

Course material at https://hackmd.io/@eglerean/MatlabAdvanced2021

# Day 1
:::spoiler
## Icebreaker 1
***What are your learning goals for this course?***
- answer here
- Learn efficient and flexible coding and useful tricks, especially dealing with large datasets
- Know how to solve problems that I encounter during the next few days and learn as many tricks as possible
- Writing efficient code and any speedup tricks, dealing with memory issues, parallelization
- Learn something interesting and useful about parallel computing and other techniques useful for large problems
- I will be using machine learning in my thesis; so it is a subject of interest for me in MATLAB. Other numerical calculations will also be useful for me in matlab.
- I study Machine learning but i am mostly working with python and its time to learn how ML works in MATLAB. :+1:
- I want to learn how others are using matlab these days, explore new things on machine learning with matlab, how to improve this course for the future years
- I want to learn more advanced matlab techniques, for example: visualizations.
- I want to learn advanced MATLAB techniques and how to draw beautiful plots
- I want to learn good tricks with MATLAB (parallel computing and refinibg my skills)

- GPU computing and parallel computing in MATLAB
- I want to generally improve my Matlab skills and general competence, signal processing would be great as I work with audio
----
- I want to improve the performance of my code and speed of computations, I'll work on numerical simulations in physical optics, it requires too much resources, so I'd like to somehow improve it
- Learn how to use MatLab for Reinforcement Learning
- That's great. Need to make it more advance. Thanks.
-

## Icebreaker 2

***What type of data do you work with? How big is an average dataset (size as in disk space, and also size as dimensions of the data themselves)?***

- I work with functional brain images, stored as 4 dimensional volumes of 91x109x91x1000 (last number varies), usually about 3GB per data file
- I work with simulated channel information obtained from simulators, files can be in order of GB, sometimes 10GB or more.
- I work with artificially or random generated matrices to test conjectures in linear algebra, usually related to asymptotic properties, meaning on very large matrices (it would be nice to get to 1Mx1M).
- I think I will be working with data from solid-state systems and quantum simulations. Data-size hopefully large :)
- Probably in the range of a few GB for machine learning.
- I work with EEG data, up to 128x6M and 6GB in size. Usually the data size is smaller though, maybe around 1GB.
- I work with data from experiments and numerical simulations, usually calculate something from several 2D arrays and/or combine it
- I work with EEG-fMRI, TMS-EEG and EEG datasets. Typically EEG datasets are 64xVeryBigNumber / 7Gb with sampling rate of 20kHz.
- Rather large data (90GB in total), mostly structural MRI but also fMRI
- Audio, generally 2 channel stereo so 2 columns and many rows.small file sizes comparatively
- I work with simulation data mostly. Usually the datasets have been few gigabytes at most, but I wouldn't mind using bigger datasets if I had the right tools for it.

# Anonymous(*) questions and comments
*Anything that comes to your mind, write here below*
*(\*) Anonymous IFF you are not logged in hackmd or authorised other services (e.g. github): use private browswing if you are unsure*

- write a question like this
- When I enter my password into triton, I have the message *permission denied*. Does it mean that the password is wrong, or is it something else
- a reply from Enrico or from anybody else
- another one here like this
- and another ....
- Comment to the above
- Comment to the comment
- ...

## Exercise 1.1

- what was the command to connect to triton with graphics
- `ssh -X triton.aalto.fi`
- Where can we see the exercise? I couldn't find the link
- https://hackmd.io/QGR0U7yYTw6P4lxdQPCkyQ?view#Exercise-11-matlab-workflows
- How did we start the interactive session in triton?
- command `sinteractive`
- How to specify the RAM, GPU and etc by interactive?
- `sinteractive -c=4 --mem=4G -p batch
- Where can I found the maximum available resources?
- https://scicomp.aalto.fi/triton/ref/#partitions
- Is there any time limitation?`
- See link before
- write here what you got when running the exercise, for example:
- ROOM 57: we got "/Users/enrico" as path, etc
- Room 4: 0.8147, 9.10.1602886 (R2021a), /m/home/home6/66/USERNAME/unix, same number, 8.3.0.532 (R2014a), home/USERNAME
- Room 3: 0.8147, 9.10.0.1602886(R2021a)
- >> rng(0); rand

ans =

0.8147

>> version

ans =

'9.10.0.1602886 (R2021a)'

>> pwd

ans =

'/m/home/home8/81/rezapop1/unix'
## Questions and comments here below

- What is the command to open matlab in konsole
- just type `matlab` in the terminal window
I get command not recognized
in triton i think

- you need to first run `module load matlab` on triton

## Exercise 1.2 (13 minutes)

- Link to exercise in the usual main webpage: https://hackmd.io/QGR0U7yYTw6P4lxdQPCkyQ?view#Exercise-12-working-with-big-mat-files

Write here if you are blocked

- VDI doesn't have copy-paste, so it's difficult to even get to the drive where bigdata is
- Copy paste should work, we can check later why it doesnt, however if you open firefox or chrome on VDI you can find the email I sent and click the link from there, it is the main course webpage
- Copy-paste doesn't work from one personal pc to the session on vdi.aalto.fi

https://hackmd.io/@eglerean/MatlabAdvanced2021
- How to change my directory to google drive to read big data?
- You should have an option to download it. Put the mouse on top of the file "bigdata.zip" and there is an arrow to download it
- Is there any way to load it from Google drive without downloading it?
- No it is not possible as far as I know
- How to load my data(bigdata.mat after downloading) into triton from my laptop?
- I will make it available on a triton shared folder

- Do our code and data are secure in triton or others with Aalto account can access them (I mean each time we need to transfer them to our personal laptop)?
- if you are familiar with linux permissions, then those who you give access to will have access. Which means that if you make your files readable and writable only by you, then nobody else will have access. It is actually safer to have file stored in a remote system with correct permissions, rather than on your laptop assuming somebody could steal it and get the sensitive data/code.

- I've passed 3 or 4 online courses at mathworks website before. Can I include it in my study plan and get credits?
- I guess :) unless other course participants object since somebody might consider it unfair. Let's discuss the matter in private.

- ROOM 123: we got this result

- ROOM 3: 1007 variables, the only one with the required properties is 'xifkmjsrbiwtjhfy', 708x708, mean = -0.0021

-jj =

'xifkmjsrbiwtjhfy'

- I used "getfield" function
- :+1:

- I used `x = load('bigdata.mat','x*y')`
- :+1:

## Exercise 1.3

Let's get in touch before 14:40 to see if people need more time.

- When and where the recorded video file of today's section is available?

# Feedback for today
*Write one positive thing and one thing that can be improved. Please let us know if the hands-on coding approach is working for you, or if you'd rather have more theory/lecture*
- Enrico: the access to the demo data can be improved + considerations for those who might not have enough internet speed or disk space

- The hands-on method is good in my opinion, but for the last task, there could've been a bit more theory.
- I like the hands-on coding. However, a bit more time would have been good
- Coding myself works well! However, there could be more hints. For example, the use of 'x*y' in ex. 1.2 could have helped :) Nonetheless, now I know.
- The exercises are useful, but if we can have an example first and then do a similar exercise, and it might help us get into the situation faster.
- The exercises are interesting, but I feel like I spend most of the time trying to figure out how to do string manipulation in matlab or some other fairly trivial stuff that I don't normally use. Some hints could make that part faster.
- The hands-on coding approach was good! Maybe more hints would have been useful
- I understand the topics and many of those are familiar for me. Meanwhile, I do not have enough time for writing the code, unfortunately :-)
:::

## Exercise 1.4

- So memmapfile command is returning a pointer to the structured data?

# Day 2
:::spoiler

## Icebreaker: if you know it, what takes longer to run in your usual matlab computations?

- It takes very long to do bootstrap permutation and other permutation based statistical tests
- I do not know what takes longer...
- Load big .mat files and for loops with matrix multiplication
- Large number of for-loops
- Linear algebra with big arrays
- nested for loops..
- I don't know which parts take longest
- Operations of big matrices
- FFT and inverseFFT filterbanks.
- Not sure really
- The size of data

## Exercise, write a better Pearson's correlation

:::warning
**EXERCISE until 13:35ish**. Start from the code at https://hackmd.io/QGR0U7yYTw6P4lxdQPCkyQ?view#Exercise-write-a-better-Pearson%E2%80%99s-corrlation
:::

In this way we can perform faster than MATLAB corr function:
:::spoiler
tic
mx=mean(x);
my=mean(y);
sx=std(x);
sy=std(y);

Mx= repmat(mx,length(x),1);
My= repmat(my,length(y),1);

tempr=(x - Mx).*(y -My)/(sx*sy);

rr=sum(tempr)/(length(tempr)-1)
myy_correlation_time=toc
:::

- Hide your code while others are having fun :)

- Woohoo, 8e-4 seconds

check this below

:::spoiler

zx = x - mean(x);
zy = y - mean(y);

% in std it computes the mean, but we already 
% have that, so we can use it instead of 
% computing it again

sx = norm(zx); 
sy = norm(zy);

 if(sx == 0 || sy == 0)
        error('we cannot divide by zero')
 end

% dividing sx*sy here is more efficient than
% to divide every component of a vector

r=zx'*zy/(length(tempr)-1)/(sx*sy);

:::

:::spoiler
mx=mean(x);
my=mean(y);
sx=std(x);
sy=std(y);
if(sx == 0 || sy == 0)
error('we cannot divide by zero')
end
r = (x-mx)'*(y-my)/(sx*sy)/1e4;
:::

## Questions, comments, etc... here below

- I prefer using python to draw figures, but Matlab is simpler for me to calculate/process the data. Is it a good idea to use python scripts inside Matlab?
- I try to avoid mixing more than one language into the same "processing step"
- I like to work with python especially because of jupyter notebook / jupyterlab (it's easy to document also big project). Is there something similar for Matlab?
- I think matlab livescript is pretty similar to jupyter notebook (maybe I'm wrong-I'm not an expert XD) ; upd: if I choose 'output inline mode' instead 'output on right' in Live Editor, I don't feel the difference.
- I don't agree. Jupyter is much nicer to read. It looks nearly like a word document
- What do you think of Octave?
- I havent really used it much, but I know of people who have struggled to make matlab code compatible with Octabe beyond simple operations.
- How can we access your recording?
- yes I will share what I recorded
- What about home assignments? upd: I've finished all onramp courses, I think it's a good motivation to start an advanced course :-)
- please pick a longer course and be fair and do 6 hours of it
- Do you do a Python course?
- https://aaltoscicomp.github.io/python-for-scicomp/
- what would you recommend for symbolic math computations? I know that Matlab and Python both support it
- No idea :)
- If any of you knows better, please comment
- I have used Mathematica recently for symbolic computations, I liked it better than matlab for this.

## Feedback for the day
Write something positive and someting that can be improved.

I thought it was better today, the breakout rooms tend to be a bit quiet
-I liked the speed today
-I love today's pace.
-I'm a slowpoke, so today's format is more preferrable for me
I agree with this as well^^^
- today is very good.
-tpdayaa
- I quite liked the pacing for the lecture today, as well as the exercises.

# Day 3

## Question: when is it good for you to have the 4th lecture (about graphics/figures)?
Options could be tue or wed 12-15

- This time next week works for me 12-15
- I cannot join but I'll check the MATLAB page for parallel computing etc.
-Tuesday next week same time works.
- Tuesday +1
- Next week, either day would work
WEdnesday would be better for me
- Next week would be nice
- For me, next week works only on Wednesday, not on Tuesday

**Let's do our last session next wednesday at 12:00**

## Questions or comments, write here below

- what is the deadline for sending the certificates?
- Ideally within the 1st week of June (e.g. before Monday 7th June)
- Where we can find the recorded video files?
- I will upload and share with all people registered
- I've almost finished "Programming techniques". It's quite beneficial, but hard!
- :+1:
- can we use the parallelization in Matlab Online?
- I am not sure, if anyone wants to try please be my guest :)
- upd. I checked, "no clusters were found" :(

- The login node seems to be an easy error to do. Is it usual that somebody uses the login node and how big of a problem that is for everyone else?
- Matlab Online is fine when you need to check something for your submitted paper during a vacation :-D
- :+1: :)

'parpool' requires Parallel Computing Toolbox.Installing now

- 1 worker on Triton, 2 on VDI
- I have 2 workers
- 4 workers
- 4 workers :)
- 2 workers
- 2 workers
- 2
- I seem to have destroyed my matlab install after yesterday :)
- 18 workers in my workstation

-8

### hostname on triton

- csl38.int.triton.aalto.fi

- csl43.int.triton.aalto.fi

- csl25
- I got csl48.int.triton.aalto.fi
- csl47.int.triton.aalto.fi
- csl11
- csl25.int.triton.aalto.fi

- How do you login twice to the node, in order to check what's happening?
- just ssh the node from the login, for example "ssh csl3"
- I write ssh cs148.int.triton.aalto.fi, have this: ssh: Could not resolve hostname cs148.int.triton.aalto.fi: Name or service not known
- You first connect to triton with ssh triton.aalto.fi and then ssh to the node ssh csl48 (just csl48)(also it's an l not a 1)
- I can demo-it again after the break
- it works, thanks
- I tried to ssh into my node to run top using ssh csl125 for example, but it denies my password. Is it again an issue with the keyboard like we saw in the VDI login?
- You can change your keyboard layout from the top right corner of the vdi
- I am using putty in my personal machine,not vdi.
- Probably not a keyboard layout issue then. Did you ssh into the node from the triton login?
- Yes. I did. I have terminal open in triton csl node but trying to ssh into that same node from other terminal is the problem.
- Using that other terminal, first ssh into triton and then ssh into the node from the triton login

- is 'matlab_multithread' a shorthand for a longer command?

- How can we set sbatch time to be several days? Sometimes I need to run matlab days.
- "--time=5-00:00:00"

- When I have to run the same command on multiple datasets using Matlab on Triton, do you have any suggestions where to put the data files best? Or does it not matter where they are?
:

- Is the moving sum possible with communicating parallellisation (spmd(?) and pareval)

- can we somehow connect to both Matlab and Triton fully from a web browser?

- What are the limitations of GPU calculations?

### Feedback for the day
Write something good and something that could be improved. Write also your wishes for the last session which will be next Wednsday at 12:00

- I liked this approach rather than coding along, because I feel like it is easier to see the overview of the workflow first and then try yourself.+1 Especially, when there are many terminal windows and Triton nodes involved :)
- I found it really useful to learn how these things can be done on triton. With such limited time, it would be difficult to find time for both hands on coding and listening to the lecture.

-

- today the explanations were really clear! Other topics to cover: more GPUs in MATLAB, and maybe spmd and other parallelization tools.
- today I learned a lot of things that I will probably use many, many times. So it was a really good lesson today!

- I prefer today's way of the lecture, I would probably like the video of the lecture available soon so I can try out some of the things from the lecture.

- Today very comfortable. Next Wednesday, very interested in the distributed computation in Matlab. Also, advanced figure manipulation techniques are very appreciated.

- It is good that you write the code and explain. After that, we can try to copy the code. IMO this works best for the fast-paced learning

-I'd like to see signal processing in the next lecture if possible

Wanna migrate your code to triton to parallelize it? Contact us https://scicomp.aalto.fi/rse/ (you must be doing research, no students sorry)

# Day 4... Graphics!
*as usual write here below about questions, comments, etc*

- is there any difference when we use '' instead "" in plot specifications?
- Isn't one char array?
- https://se.mathworks.com/help/matlab/characters-and-strings.html
- https://se.mathworks.com/help/matlab/matlab_prog/frequently-asked-questions-about-string-arrays.html

- what is the difference between 'tiled layout' and 'subplot'?
- It's a new thing! See at the bottom https://se.mathworks.com/help/releases/R2019b/matlab/ref/subplot.html

- I have to leave the meeting earlier, will you add the lecture to the drive? Thanks in advance!
- :+1:

- Is it possible to actiavte "expand axes to fill figure" option in Export setup option in File menu using code?
-Thanks!

- Does .svg file format play nice with latex?
- Cont. I googled it myself. It really doesn't. If you are using pdflatex you will probably want to export as pdf.

- Is it possible to save the figure with a transparent background in scripts?
- The most recent version of export_fig includes a -transparent option for .png export. I mentioned plot2svg toolbox
-
It is strange. It did not work on my laptop:(( maybe it depends on the machine. Thank you.

- I cannot get a high-quality figure by .svg for the following piece of code:(

close all
rng(0)
T =20;
x = randn(T,100000);

delta=1.1*max(max(x)); %distance between channels for easier visualization
[nChannels,nSamples]=size(x); %number of channels in the recording and duration of the signal in samples

figure;
plot((x+repmat([1:nChannels]'.*delta,1,nSamples))');

saveas(gcf,'Out.svg')

-are there any other lectures on scientific computation planned in this academic year?

https://scicomp.aalto.fi/training/scip/summer-kickstart/
https://scicomp.aalto.fi/help/garage/

Which software do you suggest for creating pictures for the experimental study as an example plot of the experiment with a human performing a task?
- Inkscape or powerpoint (or google slides)

-Do you make announcements of new courses via triton users mailing? Or it is better to check Aalto scientific computing webpage?
- Triton list is fine, but keep an eye on the pages if you are curious.

--
Please always write above this line