Google Drive
Gist
Clipboard
or
or
By clicking below, you agree to our terms of service.
Sign in with Wallet
New to HackMD? Sign up
| Syntax | Example | Reference | |
|---|---|---|---|
| # Header | Header | 基本排版 | |
| - Unordered List |
|
||
| 1. Ordered List |
|
||
| - [ ] Todo List |
|
||
| > Blockquote | Blockquote |
||
| **Bold font** | Bold font | ||
| *Italics font* | Italics font | ||
| ~~Strikethrough~~ | |||
| 19^th^ | 19th | ||
| H~2~O | H2O | ||
| ++Inserted text++ | Inserted text | ||
| ==Marked text== | Marked text | ||
| [link text](https:// "title") | Link | ||
|  | Image | ||
| `Code` | Code |
在筆記中貼入程式碼 | |
| ```javascript var i = 0; ``` |
|
||
| :smile: |  |
Emoji list | |
| {%youtube youtube_id %} | Externals | ||
| $L^aT_eX$ | LaTeX | ||
| :::info This is a alert area. ::: |
This is a alert area. |
On a scale of 0-10, how likely is it that you would recommend HackMD to your friends, family or business associates?
Please give us some advice and help us improve HackMD.
Do you want to remove this version name and description?
Syncing
xxxxxxxxxx-
Any changes
Be notified of any changes
-
Mention me
Be notified of mention me
-
Unsubscribe
SubscribePython for SciComp 2021 (archive)
Icebreaker questions
Where are you from and what is your background?
What's your Python experience:
None: oooo
basics: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
objects: ooooooo
science: ooooooooooooooooNooooo
big projects: ooooo
basic scripting skills and use of scientific libraries
basic scripting skills and use of scientific libraries. Terrible with for loops.
like to learn how do to "for loop" jobs (typically on 2D matrix type data) with faster build-in numpy methods.
Basic computations. I'm in transition from Octave/Matlab to Python.
Do most R than anything else. A little bit of Jupyter oooo
Same here - R is my main scripting language. Every time I try to start
with Python, something has changed a lot, and setting things up seem
complicated. Now I'll try for real! ooo
Basic of python scientific libraries
python main language for analysis, trying to understand OOP libraries better
norway
Basic, some Pandas, Pyomo, Numpy
Basic
Various projects and teaching
Basic (doing self-study), for-while loops, mostly on spyder
Used R mostly, one course work with base python (not numpy, pandas, scipy)
Questions during introduction
Is this being recorded?
video will be available for 14 days on Twitch and my understanding is that we will put recordings on youtube later
the twitch video feed seems a bit out of focus sometimes… Perhaps a larger font could help.
content will be archived
I am in aalto. I have the anaconda 3 installed but I cannot start the navigator. I can prompt only jupyter notebook or spider. is the notebook enough?
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →I am using an Aalto laptop. I do not have anaconda in my laptop. How do I use jupyter in this case?
could you make the view on the twitch landscape?
Jupyter
https://aaltoscicomp.github.io/python-for-scicomp/jupyter/
Don't hesitate to ask questions here and we will answer.
Does Google Colab offer the same as Jupiter?
What is the bash icon? I can't see it in my jupyter-lab.
Is there any advantage of Jupyter-notebook over a well-commented regular code?
%%bash does not work on my Win10. My Jupyter does not color %%bash, so I guess that's a sign that it doesn't understand what I'm trying to do.
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →%%bash worked on my installation in Windows 10 using anaconda3
%%bash worked on WSL2 with Ubuntu and Jupyter running as service
%%bash worked on an Aalto win10 laptop (git for windows installed)
jupyter-lab: access is denied
(my Aalto computer, anaconda prompt, base environment)
run
python -m jupyterlabDoes this work? It worked for someone's Aalto compter last week.
Is there any online platform to run Python, I am using an Aalto laptop which I am not the administrator for, I cannot install jupyter
Exercise session, end at xx:45
https://aaltoscicomp.github.io/python-for-scicomp/jupyter/#exercises-1
https://aaltoscicomp.github.io/python-for-scicomp/jupyter/#exercises-2
Play around with Jupyter some, we are making sure that everyone is on the same page.
A generic question: What is Github and why was it founded?
%%timeit make my code extrmly slow
Can you explain how the comma in variable-defining works for the exercise 2?
a, b = 0, 1part?a, b = b, afor swapping values without a helper variable.a, b = b, acase, a takes the value of b and b takes the value of a, no helper variable used and needed. One could imagine the values on the right being deallocated at the creation time of the variables on the left. In some other languages you may need to do this:c = a; a = b; b = cbut in Python this is not necessary.Richard and Jarno's mics are very different volume level on. Is that possible to adjust abit:)?
Can we timeit without magics?
Where can we see excercises as they are being solved?
Is there a shortcut for running all code in a notebook?
:
Can someone explain meaning of this: 14.3 ms ± 65.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) ?
Is there timeit function for Python interface directly?
I use https://jupyter.cs.aalto.fi/, are the codes always run by aalto server?
How do I edit a markdown cell?
is there a way to see the "workspace", like in matlab?
%who, or for example all integers with%who int. Not quite the same but similar.%who, as was said above, and also%whoswith more info in the output. They are similar to ones in Octave/Matlab.A Windows applicable option for the %%timeit -things would be useful.
magic %% doesn't work for me.
avariable. You try to square a but haven't definedaearlier, probably.Numpy
https://aaltoscicomp.github.io/python-for-scicomp/numpy/
My jupyter says that numpy does not have an attribute "arrange", even after importing.
rAppending new elements to numpy listlike-objects in a loop seems to take much longer than to regular python lists. Is there some fast way to do it, or should one use regular python(multidimensional)-lists and generate a numpy equivalent at the end of the loop?
A general question: are there any rival python libraries for numpy?
numbais kind of a rival. But it's by the same developers :)%%bash worked but %%timeit is giving me a usageerror. ("Cell magic
%%timitnot found.")why numpy calculate x^2 faster?
how to get the information about row and col?
.shapeA note: I've noticed that Numpy is way slower at AMD processors comparing to Intel. Has anybody tested numpy with the recent Apple ARM M1 processors?
A basic question, is there any trick to differentiate when to use () or [] in coding using python?
[]indicates an indexing, while()is a function call. i.e. if you have an object that can be indexed (e.g. lists, dictionaries etc) you can use[]to get an element (either a stored value from a dictionary, or an index from a list/array) e.g.myarray[2], while with()you would call a function (e.g.)myarray.reshape(2,3).Does numpy support linear algebra operations?
does it matter choosing between float64 and float32
If you dont declare a datatype, what does python (numpy?) use by default?
Can you convert an array of floats to an array of integers?
d.astype(int)(seems to be the preferred way these days?)I tried summing numbers from 0 to n first using a for loop and then using np.arange(n) and sum the array. If I check for relatively small n, I get the same sum with the two methods, while if I go to e.g n = 100000 np.sum() seems to give wrong answer… why? is there a max length on np.arrays?
x.dtype. int64 can hold pretty big numbers.x = np.zeros(100_000, dtype=np.int64)Do
np.array([0,1,2])andlist(range(3))give different output?printthem. The numbers should be the same though.Will this course cover code documentation (use of Sphinx for example)?
Do we nee to submit the exercise somewhere? If yes, where?
Is there a good way to reconsile dimensionality issues such as summing size (3,) and (3,1) results in (3,3) instead of treating both as column vectors?
None, like this:x[:, None]. Then you know broadcasting will only happen along dimensions of length 1, which is less surprising.( please ask at bottom)
Numpy exercises until xx:32
Numpy exercises 1 and 2, if you get done there are advanced exercises at the bottom, or check out some of the documentation or read HackMD
Links to exercises (there are also solutions):
Note that if you registered, you can join your Zoom for exercise help/discussion/etc. (Sweden/Norway/Finland). Link by email.
I do not get automatic syntax suggestions in jupyter (e.g., if I type
priand press tab nothing happens). Any possible reason(s) for this behavior?prand hit TAB? Does it show you a list of keywords that start with "pr"?pridoesn't%config Completer.use_jedi = Falseso although we read the whole row and one column, python will give the result as row vector?
Error got when trying
astype('int'):b.dtype('int')instead ofb.astype('int')bnp.float64Exercises until xx:55
Note that if you registered, you can join your Zoom for exercise help/discussion/etc. (Sweden/Norway/Finland). Link by email.
np.add(a,b,c) cannot create the variable c if it doesn't exist already, right?
addtakes two inputs, not three. Or do I misunderstand something?cmust exist and be of the right shape/dtype/etc.c = np.add(a, b)or not?c = np.add(a, b)creates a new array calledc.np.add(a,b,c)uses an existing array calledc.np.add(a, b, out=c)- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →So if we do not copy the float array into a new variable, how do we add the int and float arrays and save into int array? a being float array and b being int array, np.add(a,b,b) does not work
ǹp.add(a,b,a)). casting to int would likely loose information, which numpy tries to avoid.Anything wrong with
a+awhenais an numpy array? Is it better to donp.add(a,a)?a+awill call thenp.addfunction.b = 2.0*ainstead of readingb = np.add(a, a)ornp.add(a, a, b)+1Pandas
(this is included under day 2, some feedback is under day 1)
Feedback day 1
intro, jupyter, numpy, pandas part 1
Where are you from:
Finland: ooooooooooooooo000000000000000000000000000ooo
Norway: ooooo0ooo0oooooooo
Sweden: oooooooo0ooooooooo
Europe: ooo
Other: o
Was today:
too slow: oo
about right: ooooooooooooooo0ooooooooo0ooo0oooooooooooooooooooooooooooooooooo
too fast: oooooooooooooooooooooooooooooooooooooooooooooooo
boring: o
Name one good thing about today:
Name one thing that should be improved next time:
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →Pandas
https://aaltoscicomp.github.io/python-for-scicomp/pandas/
can you show how to groupby more than one variable?
byargument (e.g.by=['Survived','Sex']), but you'll get 4 plots, so in the call you also have to adjust the layout.Can you scroll up (how did you plot the diagram?
Is it possible to have the y-axis range on both histograms be identical?
sharey=Trueset_indexkind of moved the "Name" field to the first column. It sound like that being the index is a sort of property of the column, is it?Why not titanic
["Heikkinen, Miss. Laina",:]? I know it won't work, but why?The loc and iloc are somewhat unintuitive.
[...]interface did some magic kind of stuff: if it was integer, it would sometimes be selecd based on position, sometimes based on label, which had possibility for confusing. So it's a trade-off "simple but confusing" or "verbose and direct"when i try the command:
titanic.at["Heikkinen, Miss. Laina", "Age"]I get the error:ValueError: At based indexing on an integer index can only have integer indexerstitanic = titanic.set_index("Name"). If not, the index is a number by default.What if you want to use multiple filters? e.g. All passengers above the age 50 who are also male? Can you do this in one selector?
&titanic[(selector 1) & (selector 2)]- I often forget the () and that makes a confusing error!(ask below at bottom)
Exercise until xx:47
https://aaltoscicomp.github.io/python-for-scicomp/pandas/#exercises-1
Play with loading data, read a bit more from the docs and see what you can find.
I never remember when I need () or []. is There a way to get Jupyter to suggest which one I should use?
Is there a "preferred" way to specify both rows and columns? For example, titanic[1:10, "Age"] won't do anything, but titanic[1:10].Age does. And then, to use the values in calculations, you actually have to say titanic[1:10].Age.values
titanic.iloc[1:10]["Age"]might be what I use.titanic.iloc[1:10]["Age"]format. loc won't allow using lists of column names, but this formatting does.Why does
titanic.groupby(titanic["Age"] > titanic["Age"].mean())["Survived"].mean()give different results thantitanic[titanic["Age"] > titanic["Age"].mean()]["Survived"].mean()andtitanic[titanic["Age"] <= titanic["Age"].mean()]["Survived"].mean()?>vs<=?– The first was my solution. The second and third were the model solution from the website. (Replacing < with <= doesn't change the result in the third.)
– I found the solution: Apparently groupby doesn't discard NaN's by default, but places them in some group. So here the problem seems to be that the survival value from those individuals whose age is unknown is placed in the false category. By choosing only those passengers whose age is known, the result will be the same as in the model solution:
titanic[~titanic["Age"].isna()].groupby(titanic["Age"] > titanic["Age"].mean())["Survived"].mean()Comment: I found that I had to update both pandas and matplotlib in order to be able to get plotting to work after importing pandas… AND I had to add %matplotlib inline, My system is linux and mint
Day 2 icebreaker
What's your favorite language besides Python?
R : ooooooooooooooooooooo
C : oooooooo
C++ : ooooo
matlab : ooooooooooooooooooooooo
julia : oo
rust : oo
Fortran: oooo
Java : oooo
JavaScript : o
COBOL :
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →bash: : oooo
Perl : oo
Haskell :
Amiga E: o
Go: o
Markdown:o
NCL: o
None of the above, I only know Python: oooo
Advent of code
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →Is the sound skipping for anyone else?
oh… I guess I still have to wake up
restarting solved it thanks!
levels are ok.
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →hello cat
More Pandas
We are somewhere here: https://aaltoscicomp.github.io/python-for-scicomp/pandas/
Can you input a list (or similar) to extract several rows based on the list content?
Does index always start with 0 in Python?
I would really love to have a way to get the column names from pressing the tab key (for example) when writing e.g.,
titanic["S{press_tab}"]or any other way than having to "hard-code" them or remember them. Is there any way to get this?[""]Can pandas make a N/dimensional dataframe https://stackoverflow.com/questions/36760414/how-to-create-pandas-dataframes-with-more-than-2-dimensions
xarraythat provides labeled multi-dimensional structures, which I think would be the current recommendationonLeft speaker is much louder than right - ok not much
titanic.loc['Lam, Mr. Ali',"Age"]-> how does the interpreter know which column the name is in?.loc, you first specify row and then column.df.loc['Row name', 'Column name']. If you only specify a single thing, that will always be interpreted as a row name:df.loc['Row name'].KeyErrortitanic = titanic.set_index('Name')Is there some logic in using single and double quotes? Both seem to work
i tried to take the runners from material (both the origival and then change it) but it gives an KeyError "The following 'value_vars' are not present in the DataFtitanicrame: [400, 800, 1200, 1500]"
Could you please (again) explain the difference between "pd.merge(df1, df2, …) and df.merge()".
can you explain the 'inplace' option for merging? are we creating a copy here:
runners.merge(age, on="Runner")?in R (Tidyverse) we have left_join, right_join, full_join and such. How does one do this in Pandas?
how=, and so on.Can we have copy of the scripts after every lesson?
-thanks so much o
In the line "titanic["Child"] = titanic["Age"] < 12", the excuating order is from right to left? I mean, "<" goes first and "=" goes second.
Can you show how you generated your runners dataframe?
Can we add more information to the groupby table. Now we calculate just the mean. can we add the sd.dev etc? How?
groupbydocs and you can find how to do itgrouped.agg([np.sum, np.mean, np.std])Until 10:47
https://aaltoscicomp.github.io/python-for-scicomp/pandas/#exercises-2
Note that if you registered, you can join your Zoom for exercise help/discussion/etc. (Sweden/Norway/Finland). Link by email.
Is it possible to select a selection of rows, e.g. 10:40 and 60:100 at the same time? If yes how to do it in a single line?
Is there a way to access a function documention in jupyter similar to the way you access R documentation with F1 in R studio?
?on the end. Something like that.Problem:
type(titanic[titanic.SibSp == 8])gives DataFrame, buttitanic[titanic.SibSp == 8].Nameortitanic[titanic.SibSp == 8]["Name"]returns "'DataFrame' object has no attribute 'Name'" even though printingtitanic[titanic.SibSp == 8]does show the column "Name"… what am I doing wrong?titanic[titanic.SibSp == 8].info()shows no column named "Name" so, yes, there is no column "Name". The same applies totitanic.info(). So, yes, the index column is NOT a column at all!df.index.Q: Is Pandas inherently 1D? We have seen examples of only such datasets. Also the "tidy" dataset sort of implies that the things on grids are not a thing in Pandas. Is it true?
Is it possible to use a .txt file with Pandas or should I convert it to .csv somehow?
pd.read_csv('file.txt', sep=' ')In exercise 2, what does the actual array mean when you write titanic.SibSp.unique() ?
unique([1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3]) = [1, 2, 3]Is there a way to undo the "set_index" thing? See above about the effect of having set "Name" as the index of the Titanic DataFrame.
Break until xx:02
Visualization
https://aaltoscicomp.github.io/python-for-scicomp/data-visualization/
what if we remove '*' from matplotlib.inline?
In the example: why do you save fig and not ax?
can we set DPI when saving the figure?
I am always confused between fig and ax. Both things allow labels. When do you use what, please?
matplotlib.pyplot.xlabeland there isAxes.set_xlabel. Is the first one for 1 picture or the overall fig, and the second command for the subplots?Are there industry standard names for what to import the packages as? e.g. should matlab always be imported as plt?
matplotlib.pyplotis imported aspltby convention/habit.pltis a useful search term which will filter outmatplotlib.pyplotrelevant helpExercise until xx:30
https://aaltoscicomp.github.io/python-for-scicomp/data-visualization/#exercise-matplotlib
This may be simple for some people, but it gets us started. Explore and experiment, most people can probably complete it.
Say I feel there is a way to set the labels to each series, but I do not know/remember how to do it. What is the best way to find out how to set the labels (or do anything else)? I tried
ax?andax.scatter?, but it's too long and not specific about labels. I've tried to add, lafter the color spcification of a series, and then pres tab, but it does nto suggest "label". So, how am I supposed to find out that I should write "label='blabla'". This is about how to find documentation…?to an element, to get documentation info. or??to get the source code (if available). e.g.fig?orfig??dir()on a variable/package to show all possible fields/functions in python. ddir()?dir(ax), but it will list a lot of things.Bottom line: is Google the only way to answer the question efficiently?
my 50cents would be to use vscode with wsl if you are after IDE functionalities
Could you give some opinions on best practices? E.g. is it better to do
fig, ax = plt.subplots()orplt.figure()and calling next statements likeplt.(...)?when making the data2_y_scaled, how does it know that the values are y-values? I am referring to the suggested solution which actually multiplies the y with 2, for y in the list.
ax.scatterasy=....why I couldnot use data2_y_scaled = 2*data2_y
Is it possible to define a customized color palette?
is it possible or smart to include all data sets in one scatter call?
why we dont put plt.view()
%matplotlib inlinemagic command is doing that for us. When the last line returns a plot object, Jupyter will display it.Matplotlib's two interfaces
Styling and customization
Exercises until xx:00, then break until xx:10
don't forget to take your break!
Customization-1, Customization-2, and Customization-3
https://aaltoscicomp.github.io/python-for-scicomp/data-visualization/#exercises-styling-and-customization
Take the example, search the web some, and try to do what it says there. Mostly exploration: you probably won't have enough time to do everything, but take your pick of what is interesting.
Coming up: Data Formats at ~xx:20
Is there a difference between changing the scale of the x axis using the suggested solution and "plt.xscale("log")" in terms of these two commands giving different outcomes in a different setting (e.g. plotting a gallery of plots)?
axhas less side effects on other plots which might be in the same notebook later.when I changed the font size, the plot didnt appear automatically, why?
what is the difference between using "size" and "fontsize" here? I tried both and it does the same thing, I think.
Is there a way to have all tick parameters in one dict?
axand a dictionary, and then inside the function you set the key-value pairs one by one.Seaborn have a way of setting color scale to coloblind suitable
import seaborn as snssns.color_palette("colorblind")I'm trying to run the seaborn example. do I have to install seaborn first? It doesn't produce an output when i run it?
%matplotlib inlineon top solve it? (seaborn uses matplotlib underneath)import seaborn; print(seaborn.__version__). In my case it gives: 0.11.1.Data formats
https://aaltoscicomp.github.io/python-for-scicomp/data-formats/
what would be good for collecting data that has timestamps? (e.g. what and when someone did something in a test)
{"ID" : {"Age" : 12, "eye_track_data" : ...}. But if you need to compare in between subjects, you might want to use a pandas dataframe, where you have e.g. 1 Column for each type of data (or, if multiple entries for each type are present, the column can be a list that contains all individual entries, or again a Dataframe with moe info on the individual Data).(Thaks!)
But overall: I would try to adjust my way of storing data to how the data is going to be used in the end, and think about optimizing data access based on that. Potentially (if size of the data is not prohibiting) even storing it in multiple synchronized copies, that are fit for specific tasks, but that only if you are sure that you will work (i.e. load) with that data a lot…
It is possible to store as well arrays in a cell of a pandas dataframe.
Why is the shape of a 1D numpy array
(4,)and not(4)? Why the "extra" comma?(4)is the same as4.(4,)is different from(,4), I guess?(, 4)generates a syntax error(2,3,)ok?(2, 3,)is a tuple of length 2 (not 3) and identical to(2,3)semantics vs structure (discuss at end)
what does the !head mean on the demo? Is it magic showing of the head of the file?
!means "run shell command". Mac and linux at leastSometimes data (outcome of some software let's say COMSOL) has weird sepration. It has tab sepration and space too. How to you handle that kind of data with multiple seprators?
pd.read_csv('fileName', sep='\t '). Maybe have a look at the read_csv (documentation page)[https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html], since there are a lot more options you can set (skip comment lines, define headers etc pp)could you use windows functions
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →open("filename").readline()[0]type file.csv -Head 3, but I can't check that since I'm on linux myself.What is the difference between the % and ! in jupyterlab? Something on the operator side?
Another data format I have used is pickle, wonder if that's any good +1
netcdf is great in that it's kind of well-defined, but typically the libraries are a bit annoying in that you have to build the structure very explicitly. I.e. you can't just say "save this", but always specify names, dimensions etc. I wonder if it would be possible to create a library that would make reasonable assumptions from the data itself?
How to judge if a format is expected to take-off and be alive long in the future, or is going to desapear in few years?
Feedback
Today was:
too fast: ooo
about right: ooooooooooooooooooooooooooooooooo
too slow: oooo
I wish for:
more exercises : oooooo
more time for exercises : oooooo
more discussion : oooooooo
longer workshop duration: ooooooo
slightly longer workshop duration: ooooo
about right length: ooooooooo
shorter workshop duration: oooO
more in depth on topics: oooooooooooooooooo
about right depth of topics: ooo
less in depth on topics: o
recovery points to get back on track:o
If you have a multiple people watching one stream, how many are/were watching:
2:ooo
3:
4-5:
6-10:oo
10-20:
other, specify:
One good thing about today
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →One thing that we could improve
DAY 3 & 4
Please visit this link https://hackmd.io/@coderefinery/python2021archive_pt2