.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } B G B BB BG G GB GG We know that at least one of the children is a boy, so only \"GG\" is not possible. The event where the family has a new boy is then \"BB\". Hence the probability is : $$\\frac{BB}{BB+GB+BG}=\\frac{1}{3}$$ Draw 2 cards from a deck Problem You 2 draw cards from a standard 52-card deck without replacing them. What is the probability that both cards are of the same suit? Mathematical explanation There are 13 cards of each suit. Draw one card. It can be anything with probability of 1. Now there are 51 cards left and 12 of them are the same suit as the first card you drew. So the chance the second card matches the 1st is \\(\\frac{12}{51}\\) . Drawing marbles Problem A bag contains 3 red marbles and 4 blue marbles. Then, 2 marbles are drawn from the bag, at random, without replacement. If the first marble drawn is red, what is the probability that the second marble is blue? Mathematical explanation On the first draw, the probabilities are the following : we call B the event \"a blue ball is drawn\" and R the event \"a red ball is drawn\" * \\(P(B)=\\frac{4}{7}\\) * \\(P(R)=\\frac{3}{7}\\) On the second draw, if a red ball has been drawn at first, the probabilities are : * \\(P(B|R)=\\frac{4}{6}\\) * \\(P(R|R)=\\frac{2}{6}\\) Hence, the probability of drawing a blue ball if the first ball drawn was red is \\(\\frac{1}{3}\\) if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) { var align = \"center\", indent = \"0em\", linebreak = \"false\"; if (false) { align = (screen.width < 768) ? \"left\" : align; indent = (screen.width < 768) ? \"0em\" : indent; linebreak = (screen.width < 768) ? 'true' : linebreak; } var mathjaxscript = document.createElement('script'); mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#'; mathjaxscript.type = 'text/javascript'; mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML'; mathjaxscript[(window.opera ? \"innerHTML\" : \"text\")] = \"MathJax.Hub.Config({\" + \" config: ['MMLorHTML.js'],\" + \" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } },\" + \" jax: ['input/TeX','input/MathML','output/HTML-CSS'],\" + \" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js'],\" + \" displayAlign: '\"+ align +\"',\" + \" displayIndent: '\"+ indent +\"',\" + \" showMathMenu: true,\" + \" messageStyle: 'normal',\" + \" tex2jax: { \" + \" inlineMath: [ ['\\\\\\\\(','\\\\\\\\)'] ], \" + \" displayMath: [ ['$$','$$'] ],\" + \" processEscapes: true,\" + \" preview: 'TeX',\" + \" }, \" + \" 'HTML-CSS': { \" + \" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} },\" + \" linebreaks: { automatic: \"+ linebreak +\", width: '90% container' },\" + \" }, \" + \"}); \" + \"if ('default' !== 'default') {\" + \"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"}\"; (document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript); }","tags":"Blog","url":"redoules.github.io/blog/Statistics_10days-day3.html","loc":"redoules.github.io/blog/Statistics_10days-day3.html"},{"title":"Day 2 - Probability, Compound Event Probability","text":"Basic probability with dices Problem In this challenge, we practice calculating probability. In a single toss of 2 fair (evenly-weighted) six-sided dice, find the probability that their sum will be at most 9. Mathematical explanation A nice way to think about sums-of-two-dice problems is to lay out the sums in a 6-by-6 grid in the obvious manner. .dataframe tbody tr th:only-of-type { vertical-align: middle; }
.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } 1 2 3 4 5 6 1 2 3 4 5 6 7 2 3 4 5 6 7 8 3 4 5 6 7 8 9 4 5 6 7 8 9 10 5 6 7 8 9 10 11 6 7 8 9 10 11 12 We see that the identic values are on the same diagonal. The number of elements on the diagonal varies from 1 to 6 and then back to 1. let's call A < x the event : the sum all the 2 tosses is at most x. $$P(A\\leq9)=\\sum_{i=2}^{9} P(A = i)$$ $$P(A\\leq9)=1-P(A\\gt9)$$ $$P(A\\leq9)=1-\\sum_{i=10}^{12} P(A = i)$$ The value of \\(P(A = i) = \\frac{i-1}{36}\\) if \\(i \\leq 7\\) and \\(P(A = i) = \\frac{13-i}{36}\\) hence $$P(A\\leq9)=1-\\sum_{i=10}^{12} \\frac{13-i}{36}$$ $$P(A\\leq9)= 1-\\frac{6}{36}$$ $$P(A\\leq9)= \\frac{5}{6}$$ Let's program it sum ([ 1 for d1 in range ( 1 , 7 ) for d2 in range ( 1 , 7 ) if d1 + d2 <= 9 ]) / 36 0 . 8333333333333334 More dices Problem In a single toss of 2 fair (evenly-weighted) six-sided dice, find the probability that the values rolled by each die will be different and the two dice have a sum of 6. Mathematical explanation Let's consider 2 events : A and B. A compound event is a combination of 2 or more simple events. If A and B are simple events, then A∪B denotes the occurence of either A or B. A∩B denotes the occurence of A and B together. We denote A the event \"the values of each dice is different\". The opposit event is A' \"the values of each dice is the same\". $$P(A) = 1-P(A')$$ $$P(A)=1-\\frac{6}{36}$$ $$P(A)=\\frac{5}{6}$$ We denote B the event \"the two dice have a sum of 6\", this probability has been computed on the first part of the article : $$P(B)=\\frac{5}{36}$$ The probability of having 2 dice different of sum 6 is : $$P(A|B) = 4/5$$ The probability that both A and B occure is equal to P(A∩B). Since \\(P(A|B)=\\frac{P(A∩B)}{P(B)}\\) $$P(A∩B)=P(B)*P(A|B)$$ $$P(A∩B)=5/36*4/5$$ $$P(A∩B)=1/9$$ Let's program it sum ([ 1 for d1 in range ( 1 , 7 ) for d2 in range ( 1 , 7 ) if ( d1 + d2 == 6 ) and ( d1 != d2 )]) / 36 0 . 1111111111111111 Compound Event Probability Problem There are 3 urns labeled X, Y, and Z. Urn X contains 4 red balls and 3 black balls. Urn Y contains 5 red balls and 4 black balls. Urn Z contains 4 red balls and 4 black balls. One ball is drawn from each of the urns. What is the probability that, of the 3 balls drawn, are 2 red and is 1 black? Mathematical explanation Let's write the different probabilities: .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Red ball Black ball Urne X $$\\frac{4}{7}$$ $$\\frac{3}{7}$$ Urne Y $$\\frac{5}{9}$$ $$\\frac{4}{9}$$ Urne Z $$\\frac{1}{2}$$ $$\\frac{1}{2}$$ Addition rule A and B are said to be mutually exclusive or disjoint if they have no events in common (i.e., and A∩B=∅ and P(A∩B)=0. The probability of any of 2 or more events occurring is the union (∪) of events. Because disjoint probabilities have no common events, the probability of the union of disjoint events is the sum of the events' individual probabilities. A and B are said to be collectively exhaustive if their union covers all events in the sample space (i.e., A∪B=S and P(A∪B)=1). This brings us to our next fundamental rule of probability: if 2 events, A and B, are disjoint, then the probability of either event is the sum of the probabilities of the 2 events (i.e., P(A or B) = P(A)+P(B)) Mutliplication rule If the outcome of the first event (A) has no impact on the second event (B), then they are considered to be independent (e.g., tossing a fair coin). This brings us to the next fundamental rule of probability: the multiplication rule. It states that if two events, A and B, are independent, then the probability of both events is the product of the probabilities for each event (i.e., P(A and B)= P(A)xP(B)). The chance of all events occurring in a sequence of events is called the intersection (∩) of those events. The balls drawn from the urns are independant hence : p = P(2 red (R) and 1 back (B)) $$p = P(RRB) + P(RBR) + P(BRR)$$ Each of those 3 probability if equal to the product of the probability of drawing each ball \\(P(RRB) = P(R|X) * P(R|Y) * P(B|Z) = 4/7*5/9*1/2\\) \\(P(RRB) = 20/126\\) \\(P(RBR) = 16/126\\) \\(P(BRR) = 15/126\\) this leads to \\(p = 51/126\\) and finally $$p = \\frac{17}{42}$$ Let's program it X = 3 * [ \"B\" ] + 4 * [ \"R\" ] Y = 4 * [ \"B\" ] + 5 * [ \"R\" ] Z = 4 * [ \"B\" ] + 4 * [ \"R\" ] target = [ \"BRR\" , \"RRB\" , \"RBR\" ] sum ([ 1 for x in X for y in Y for z in Z if x + y + z in target ]) / sum ([ 1 for x in X for y in Y for z in Z ]) 0 . 40476190476190477 if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) { var align = \"center\", indent = \"0em\", linebreak = \"false\"; if (false) { align = (screen.width < 768) ? \"left\" : align; indent = (screen.width < 768) ? \"0em\" : indent; linebreak = (screen.width < 768) ? 'true' : linebreak; } var mathjaxscript = document.createElement('script'); mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#'; mathjaxscript.type = 'text/javascript'; mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML'; mathjaxscript[(window.opera ? \"innerHTML\" : \"text\")] = \"MathJax.Hub.Config({\" + \" config: ['MMLorHTML.js'],\" + \" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } },\" + \" jax: ['input/TeX','input/MathML','output/HTML-CSS'],\" + \" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js'],\" + \" displayAlign: '\"+ align +\"',\" + \" displayIndent: '\"+ indent +\"',\" + \" showMathMenu: true,\" + \" messageStyle: 'normal',\" + \" tex2jax: { \" + \" inlineMath: [ ['\\\\\\\\(','\\\\\\\\)'] ], \" + \" displayMath: [ ['$$','$$'] ],\" + \" processEscapes: true,\" + \" preview: 'TeX',\" + \" }, \" + \" 'HTML-CSS': { \" + \" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} },\" + \" linebreaks: { automatic: \"+ linebreak +\", width: '90% container' },\" + \" }, \" + \"}); \" + \"if ('default' !== 'default') {\" + \"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"}\"; (document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript); }","tags":"Blog","url":"redoules.github.io/blog/Statistics_10days-day2.html","loc":"redoules.github.io/blog/Statistics_10days-day2.html"},{"title":"Day 1 - Quartiles, Interquartile Range and standard deviation","text":"Quartile Definition A quartile is a type of quantile. The first quartile (Q1) is defined as the middle number between the smallest number and the median of the data set. The second quartile (Q2) is the median of the data. The third quartile (Q3) is the middle value between the median and the highest value of the data set. Implementation in python without using the scientific libraries def median ( l ): l = sorted ( l ) if len ( l ) % 2 == 0 : return ( l [ len ( l ) // 2 ] + l [( len ( l ) // 2 - 1 )]) / 2 else : return l [ len ( l ) // 2 ] def quartiles ( l ): # check the input is not empty if not l : raise StatsError ( 'no data points passed' ) # 1. order the data set l = sorted ( l ) # 2. divide the data set in two halves mid = int ( len ( l ) / 2 ) Q2 = median ( l ) if ( len ( l ) % 2 == 0 ): # even Q1 = median ( l [: mid ]) Q3 = median ( l [ mid :]) else : # odd Q1 = median ( l [: mid ]) # same as even Q3 = median ( l [ mid + 1 :]) return ( Q1 , Q2 , Q3 ) L = [ 3 , 7 , 8 , 5 , 12 , 14 , 21 , 13 , 18 ] Q1 , Q2 , Q3 = quartiles ( L ) print ( f \"Sample : {L} \\n Q1 : {Q1}, Q2 : {Q2}, Q3 : {Q3}\" ) Sample : [ 3 , 7 , 8 , 5 , 12 , 14 , 21 , 13 , 18 ] Q1 : 6 . 0 , Q2 : 12 , Q3 : 16 . 0 Interquartile Range Definition The interquartile range of an array is the difference between its first (Q1) and third (Q3) quartiles. Hence the interquartile range is Q3-Q1 Implementation in python without using the scientific libraries print ( f \"Interquatile range : {Q3-Q1}\" ) Interquatile range : 10 . 0 Standard deviation Definition The standard deviation (σ) is a measure that is used to quantify the amount of variation or dispersion of a set of data values. A low standard deviation indicates that the data points tend to be close to the mean (also called the expected value) of the set, while a high standard deviation indicates that the data points are spread out over a wider range of values. The standard deviation can be computed with the formula: where µ is the mean : Implementation in python without using the scientific libraries import math X = [ 10 , 40 , 30 , 50 , 20 ] mean = sum ( X ) / len ( X ) X = [( x - mean ) ** 2 for x in X ] std = math . sqrt ( sum ( X ) / len ( X ) ) print ( f \"The distribution {X} has a standard deviation of {std}\" ) The distribution [ 400 . 0 , 100 . 0 , 0 . 0 , 400 . 0 , 100 . 0 ] has a standard deviation of 14 . 142135623730951","tags":"Blog","url":"redoules.github.io/blog/Statistics_10days-day1.html","loc":"redoules.github.io/blog/Statistics_10days-day1.html"},{"title":"Counting values in an array","text":"Using lists If you want to count the number of occurences of an element in a list you can use the .count() function of the list object arr = [ 1 , 2 , 3 , 3 , 4 , 5 , 3 , 6 , 7 , 7 ] print ( f 'Array : {arr} \\n ' ) print ( f 'The number 3 appears {arr.count(3)} times in the list' ) print ( f 'The number 7 appears {arr.count(7)} times in the list' ) print ( f 'The number 4 appears {arr.count(4)} times in the list' ) Array : [ 1 , 2 , 3 , 3 , 4 , 5 , 3 , 6 , 7 , 7 ] The number 3 appears 3 times in the list The number 7 appears 2 times in the list The number 4 appears 1 times in the list Using collections you can get a dictonnary of the number of occurences of each elements in a list thanks to the collections object like this import collections collections . Counter ( arr ) Counter ( { 1 : 1 , 2 : 1 , 3 : 3 , 4 : 1 , 5 : 1 , 6 : 1 , 7 : 2 } ) Using numpy You can have a simular result with numpy by hacking the unique function import numpy as np arr = np . array ( arr ) unique , counts = np . unique ( arr , return_counts = True ) dict ( zip ( unique , counts )) { 1 : 1 , 2 : 1 , 3 : 3 , 4 : 1 , 5 : 1 , 6 : 1 , 7 : 2 }","tags":"Python","url":"redoules.github.io/python/counting.html","loc":"redoules.github.io/python/counting.html"},{"title":"Building a dictonnary using comprehension","text":"An easy way to create a dictionnary in python is to use the comprehension syntaxe. It can be more expressive hence easier to read. d = { key : value for ( key , value ) in iterable } In the example bellow we use the dictionnary comprehension to build a dictonnary from a source list. iterable = list ( range ( 10 )) d = { str ( value ): value ** 2 for value in iterable } # create a dictionnary linking the string value of a number with the square value of this number print ( d ) { '0' : 0 , '1' : 1 , '2' : 4 , '3' : 9 , '4' : 16 , '5' : 25 , '6' : 36 , '7' : 49 , '8' : 64 , '9' : 81 } of course, you can use an other iterable an repack it with the comprehension syntaxe. In the following example, we convert a list of tuples in a dictonnary. iterable = [( \"France\" , 67.12e6 ), ( \"UK\" , 66.02e6 ), ( \"USA\" , 325.7e6 ), ( \"China\" , 1386e6 ), ( \"Germany\" , 82.79e6 )] population = { key : value for ( key , value ) in iterable } print ( population ) { 'France' : 67120000 . 0 , 'UK' : 66020000 . 0 , 'USA' : 325700000 . 0 , 'China' : 1386000000 . 0 , 'Germany' : 82790000 . 0 }","tags":"Python","url":"redoules.github.io/python/dict_comprehension.html","loc":"redoules.github.io/python/dict_comprehension.html"},{"title":"Extracting unique values from a list or an array","text":"Using lists An easy way to extract the unique values of a list in python is to convert the list to a set. A set is an unordered collection of items. Every element is unique (no duplicates) and must be immutable. my_list = [ 10 , 20 , 30 , 40 , 20 , 50 , 60 , 40 ] print ( f \"Original List : {my_list}\" ) my_set = set ( my_list ) my_new_list = list ( my_set ) # the set is converted back to a list with the list() function print ( f \"List of unique numbers : {my_new_list}\" ) Original List : [ 10 , 20 , 30 , 40 , 20 , 50 , 60 , 40 ] List of unique numbers : [ 40 , 10 , 50 , 20 , 60 , 30 ] Using numpy If you are using numpy you can extract the unique values of an array with the unique function builtin numpy: import numpy as np arr = np . array ( my_list ) print ( f 'Initial numpy array : {arr} \\n ' ) unique_arr = np . unique ( arr ) print ( f 'Numpy array with unique values : {unique_arr}' ) Initial numpy array : [ 10 20 30 40 20 50 60 40 ] Numpy array with unique values : [ 10 20 30 40 50 60 ]","tags":"Python","url":"redoules.github.io/python/unique.html","loc":"redoules.github.io/python/unique.html"},{"title":"Sorting an array","text":"Using lists Python provides an iterator to sort an array sorted() you can use it this way : import random # Random lists from [0-999] interval arr = [ random . randint ( 0 , 1000 ) for r in range ( 10 )] print ( f 'Initial random list : {arr} \\n ' ) reversed_arr = list ( sorted ( arr )) print ( f 'Sorted list : {reversed_arr}' ) Initial random list : [ 277 , 347 , 976 , 367 , 604 , 878 , 148 , 670 , 229 , 432 ] Sorted list : [ 148 , 229 , 277 , 347 , 367 , 432 , 604 , 670 , 878 , 976 ] it is also possible to use the sort function from the list object # Random lists from [0-999] interval arr = [ random . randint ( 0 , 1000 ) for r in range ( 10 )] print ( f 'Initial random list : {arr} \\n ' ) arr . sort () print ( f 'Sorted list : {arr}' ) Initial random list : [ 727 , 759 , 68 , 103 , 23 , 90 , 258 , 737 , 791 , 567 ] Sorted list : [ 23 , 68 , 90 , 103 , 258 , 567 , 727 , 737 , 759 , 791 ] Using numpy If you are using numpy you can sort an array by creating a view on the array: import numpy as np arr = np . random . random ( 5 ) print ( f 'Initial random array : {arr} \\n ' ) sorted_arr = np . sort ( arr ) print ( f 'Sorted array : {sorted_arr}' ) Initial random array : [ 0 . 40021786 0 . 13876208 0 . 19939047 0 . 46015169 0 . 43734158 ] Sorted array : [ 0 . 13876208 0 . 19939047 0 . 40021786 0 . 43734158 0 . 46015169 ]","tags":"Python","url":"redoules.github.io/python/sorting.html","loc":"redoules.github.io/python/sorting.html"},{"title":"Day 0 - Median, mean, mode and weighted mean","text":"A reminder The median The median is the value separating the higher half from the lower half of a data sample. For a data set, it may be thought of as the middle value. For a continuous probability distribution, the median is the value such that a number is equally likely to fall above or below it. The mean The arithmetic mean (or simply mean) of a sample is the sum of the sampled values divided by the number of items. The mode The mode of a set of data values is the value that appears most often. It is the value x at which its probability mass function takes its maximum value. In other words, it is the value that is most likely to be sampled. Implementation in python without using the scientific libraries def median ( l ): l = sorted ( l ) if len ( l ) % 2 == 0 : return ( l [ len ( l ) // 2 ] + l [( len ( l ) // 2 - 1 )]) / 2 else : return l [ len ( l ) // 2 ] def mean ( l ): return sum ( l ) / len ( l ) def mode ( data ): dico = { x : data . count ( x ) for x in list ( set ( data ))} return sorted ( sorted ( dico . items ()), key = lambda x : x [ 1 ], reverse = True )[ 0 ][ 0 ] L = [ 64630 , 11735 , 14216 , 99233 , 14470 , 4978 , 73429 , 38120 , 51135 , 67060 , 4978 , 73429 ] print ( f \"Sample : {L} \\n Mean : {mean(L)}, Median : {median(L)}, Mode : {mode(L)}\" ) Sample : [ 64630 , 11735 , 14216 , 99233 , 14470 , 4978 , 73429 , 38120 , 51135 , 67060 , 4978 , 73429 ] Mean : 43117 . 75 , Median : 44627 . 5 , Mode : 4978 The weighted average The weighted arithmetic mean is similar to an ordinary arithmetic mean (the most common type of average), except that instead of each of the data points contributing equally to the final average, some data points contribute more than others. data = [ 10 , 40 , 30 , 50 , 20 ] weights = [ 1 , 2 , 3 , 4 , 5 ] sum_X = sum ([ x * w for x , w in zip ( data , weights )]) print ( round (( sum_X / sum ( weights )), 1 )) 32 . 0","tags":"Blog","url":"redoules.github.io/blog/Statistics_10days-day0.html","loc":"redoules.github.io/blog/Statistics_10days-day0.html"},{"title":"Create a simple bash function","text":"A basic function The synthaxe to define a function is : #!/bin/bash # Basic function my_function () { echo Text displayed by my_function } #once defined, you can use it like so : my_function and it should return user@bash : ./my_function.sh Text displayed by my_function Function with arguments When used, the arguments are specified directly after the function name. Whithin the function they are accessible this the $ symbol followed by the number of the arguement. Hence $1 will take the value of the first arguement, $2 will take the value of the second arguement and so on. #!/bin/bash # Passing arguments to a function say_hello () { echo Hello $1 } say_hello Guillaume and it should return user@bash : ./function_arguements.sh Hello Guillaume Overriding Commands Using the previous example, let's override the echo function in order to make it say hello. To do so, you just need to name the function with the same name as the command you want to replace. When you are calling the original function, make sure you are using the builtin keyword #!/bin/bash # Overriding a function echo () { builtin echo Hello $1 } echo Guillaume user@bash : ./function_arguements.sh Hello Guillaume Returning values Use the keyword return to send back a value to the main program. The returned value will be stored in the $? variable #!/bin/bash # Retruning a value secret_number () { return 126 } secret_number echo The secret number is $? This code should return user@bash : ./retrun_value.sh The secret number is 126","tags":"Linux","url":"redoules.github.io/linux/simple_bash_function.html","loc":"redoules.github.io/linux/simple_bash_function.html"},{"title":"Number of edges in a Complete graph","text":"A complete graph contains \\(\\frac{n(n-1)}{2}\\) edges where \\(n\\) is the number of vertices (or nodes). if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) { var align = \"center\", indent = \"0em\", linebreak = \"false\"; if (false) { align = (screen.width < 768) ? \"left\" : align; indent = (screen.width < 768) ? \"0em\" : indent; linebreak = (screen.width < 768) ? 'true' : linebreak; } var mathjaxscript = document.createElement('script'); mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#'; mathjaxscript.type = 'text/javascript'; mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML'; mathjaxscript[(window.opera ? \"innerHTML\" : \"text\")] = \"MathJax.Hub.Config({\" + \" config: ['MMLorHTML.js'],\" + \" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } },\" + \" jax: ['input/TeX','input/MathML','output/HTML-CSS'],\" + \" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js'],\" + \" displayAlign: '\"+ align +\"',\" + \" displayIndent: '\"+ indent +\"',\" + \" showMathMenu: true,\" + \" messageStyle: 'normal',\" + \" tex2jax: { \" + \" inlineMath: [ ['\\\\\\\\(','\\\\\\\\)'] ], \" + \" displayMath: [ ['$$','$$'] ],\" + \" processEscapes: true,\" + \" preview: 'TeX',\" + \" }, \" + \" 'HTML-CSS': { \" + \" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} },\" + \" linebreaks: { automatic: \"+ linebreak +\", width: '90% container' },\" + \" }, \" + \"}); \" + \"if ('default' !== 'default') {\" + \"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"}\"; (document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript); }","tags":"Mathematics","url":"redoules.github.io/mathematics/Number_edges_Complete_graph.html","loc":"redoules.github.io/mathematics/Number_edges_Complete_graph.html"},{"title":"Reverse an array","text":"Using lists Python provides an iterator to reverse an array reversed() you can use it this way : arr = list ( range ( 5 )) print ( f 'Initial array : {arr} \\n ' ) reversed_arr = list ( reversed ( arr )) print ( f 'Reversed array : {reversed_arr}' ) Initial array : [ 0 , 1 , 2 , 3 , 4 ] Reversed array : [ 4 , 3 , 2 , 1 , 0 ] Using numpy If you are using numpy you can reverse an array by creating a view on the array: import numpy as np arr = np . arange ( 5 ) print ( f 'Initial array : {arr} \\n ' ) reversed_arr = arr [:: - 1 ] print ( f 'Reversed array : {reversed_arr}' ) Initial array : [ 0 1 2 3 4 ] Reversed array : [ 4 3 2 1 0 ]","tags":"Python","url":"redoules.github.io/python/reverse.html","loc":"redoules.github.io/python/reverse.html"},{"title":"Advice for designing your own libraries","text":"Advice for designing your own libraries When designing your own library make sure to think of the following things. I will add new paragraphs to this article as I dicover new good practices. Use standard python objects Try to use standard python objects as much as possible. That way, your library becomes compatible with all the other python libaries. For instance, when I created SAMpy : a library for reading and writing SAMCEF results, it returned dictonnaries, lists and pandas dataframes. Hence the results extracted from SAMCEF where compatible with all the scientific stack of python. Limit the number of functionnalities Following the same logic as before, the objects should do only one thing but do it well. Indeed, having a simple interface will reduce the complexity of your code and make it easier to use your library. Again, with SAMpy, I decided to strictly limit the functionnalities to reading and writing SAMCEF files. Define an exception class for your library You should define your own exceptions in order to make it easier for your users to debug their code thanks to clearer messages that convey more meaning. That way, the user will know if the error comes from your library or something else. Bonus if you group similar exceptions in a hierachy of inerited Exception classes. Example : let's create a Exception related to the age of a person : def check_age ( age ): if age < 0 and age > 130 : raise ValueError If the user inputed an invalid age, the ValueError exception would be thrown. That's fine but imagine you wan't to provide more feedback to your users that don't know the internal of your library. Let's now create a selfexplanatory Exception class AgeInvalidError ( ValueError ): pass def check_age ( age ): if age < 0 and age > 130 : raise AgeInvalidError ( age ) You can also add some helpful text to guide your users along the way: class AgeInvalidError ( ValueError ): print ( \"Age invalid, must be between 0 and 130\" ) pass def check_age ( age ): if age < 0 and age > 130 : raise AgeInvalidError ( age ) If you want to group all the logically linked exceptions, you can create a base class and inherit from it : class BaseAgeInvalidError ( ValueError ): pass class TooYoungError ( BaseAgeInvalidError ): pass class TooOldError ( BaseAgeInvalidError ): pass def check_age ( age ): if age < 0 : raise TooYoungError ( age ) elif age > 130 : raise TooOldError ( age ) Structure your repository You should have a file structure in your repository. It will help other contributers especially future contributers. A nice directory structure for your project should look like this: README . md LICENSE setup . py requirements . txt . / MyPackage . / docs . / tests Some prefer to use reStructured Text, I personnaly prefer Markdown choosealicense.com will help you pick the license to use for your project. For package and distribution management, create a setup.py file a the root of the directory The list of dependencies required to test, build and generate the doc are listed in a pip requirement file placed a the root of the directory and named requirements.txt Put the documentation of your library in the docs directory. Put your tests in the tests directory. Since your tests will need to import your library, I recommend modifying the path to resolve your package property. In order to do so, you can create a context.py file located in the tests directory : import os import sys sys . path . insert ( 0 , os . path . abspath ( os . path . join ( os . path . dirname ( __file__ ), '..' ))) import MyPackage Then within your individual test files you can import your package like so : from .context import MyPackage Finally, your code will go into the MyPackage directory Test your code Once your library is in production, you have to guaranty some level of forward compatibility. Once your interface is defined, write some tests. In the future, when your code is modified, having those tests will make sure that the behaviour of your functions and objects won't be altered. Document your code Of course, you should have a documentation to go along with your library. Make sure to add a lot of commun examples as most users tend to learn from examples. I recommend writing your documentation using Sphinx.","tags":"Python","url":"redoules.github.io/python/design_own_libs.html","loc":"redoules.github.io/python/design_own_libs.html"},{"title":"Safely creating a folder if it doesn't exist","text":"Safely creating a folder if it doesn't exist When you are writing to files in python, if the file doesn't exist it will be created. However, if you are trying to write a file in a directory that doesn't exist, an exception will be returned FileNotFoundError : [ Errno 2 ] No such file or directory : \"directory\" This article will teach you how to make sure the target directory exists. If it doesn't, the function will create that directory. First, let's import os and make sure that the \"test_directory\" doesn't exist import os os . path . exists ( \". \\\\ test_directory\" ) False copy the ensure_dir function in your code. This function will handle the creation of the directory. Credit goes to Parand posted on StackOverflow def ensure_dir ( file_path ): directory = os . path . dirname ( file_path ) if not os . path . exists ( directory ): os . makedirs ( directory ) Let's now use the function and create a folder named \"test_directory\" ensure_dir ( \". \\\\ test_directory\" ) If we test for the existence of the directory, the exists function will now return True os . path . exists ( \". \\\\ test_directory\" ) True","tags":"Python","url":"redoules.github.io/python/ensure_dir.html","loc":"redoules.github.io/python/ensure_dir.html"},{"title":"List all files in a directory","text":"Listing all the files in a directory Let's start with the basics, the most staigthforward way to list all the files in a direcoty is to use a combinaison of the listdir function and isfile form os.path. You can use a list comprehension to store all the results in a list. mypath = \"./test_directory/\" from os import listdir from os.path import isfile , join [ f for f in listdir ( mypath ) if isfile ( join ( mypath , f ))] [ 'logfile.log' , 'myfile.txt' , 'super_music.mp3' , 'textfile.txt' ] Listing all the files of a certain type in a directory similarly, if you want to filter only a certain kind of file based on its extension you can use the endswith method. In the following example, we will filter all the \"txt\" files contained in the directory [ f for f in listdir ( mypath ) if f . endswith ( '.' + \"txt\" )] [ 'myfile.txt' , 'textfile.txt' ] Listing all the files matching a pattern in a directory The glob module finds all the pathnames matching a specified pattern according to the rules used by the Unix shell. You can use the *, ?, and character ranges expressed with [] wildcards import glob glob . glob ( \"*.txt\" ) [ 'myfile.txt' ] Listing files recusively If you want to list all files recursively you can select all the sub-directories using the \"**\" wildcard import glob glob . glob ( mypath + '/**/*.txt' , recursive = True ) [ './test_directory\\\\myfile.txt' , './test_directory\\\\textfile.txt' , './test_directory\\\\subdir1\\\\file_hidden_in_a_sub_direcotry.txt' ] Using a regular expression If you'd rather use a regular expression to select the files, the pathlib library provides the rglob function. from pathlib import Path list ( Path ( \"./test_directory/\" ) . rglob ( \"*.[tT][xX][tT]\" )) [ WindowsPath ( 'test_directory/myfile.txt' ), WindowsPath ( 'test_directory/textfile.txt' ), WindowsPath ( 'test_directory/subdir1/file_hidden_in_a_sub_direcotry.txt' )] Using regular expressions you can for example select multiple types of files. In the following example, we list all the files that finish either with \"txt\" or with \"log\". list ( Path ( \"./test_directory/\" ) . rglob ( \"*.[tl][xo][tg]\" )) [ WindowsPath ( 'test_directory/logfile.log' ), WindowsPath ( 'test_directory/myfile.txt' ), WindowsPath ( 'test_directory/textfile.txt' ), WindowsPath ( 'test_directory/subdir1/file_hidden_in_a_sub_direcotry.txt' )]","tags":"Python","url":"redoules.github.io/python/list_files_directory.html","loc":"redoules.github.io/python/list_files_directory.html"},{"title":"Using Dask on infiniband","text":"InfiniBand (abbreviated IB) is a computer-networking communications standard used in high-performance computing that features very high throughput and very low latency. It is used for data interconnect both among and within computers. InfiniBand is also used as either a direct or switched interconnect between servers and storage systems, as well as an interconnect between storage systems. (source Wikipedia). If you want to leverage this high speed network instead of the regular ethernet network, you have to specify to the scheduler that you want to used infiniband as your interface. Assuming that you Infiniband interface is ib0 , you would call the scheduler like this : dask - scheduler --interface ib0 --scheduler-file ./cluster.yaml you would have to call the worker using the same interface : dask - worker --interface ib0 --scheduler-file ./cluster.yaml","tags":"Python","url":"redoules.github.io/python/dask_infiniband.html","loc":"redoules.github.io/python/dask_infiniband.html"},{"title":"Clearing the current cell in the notebook","text":"In python, you can clear the output of a cell by importing the IPython.display module and using the clear_output function from IPython.display import clear_output print ( \"text to be cleared\" ) clear_output () As you can see, the text \"text to be cleared\" is not displayed because the function clear_output has been called afterward","tags":"Jupyter","url":"redoules.github.io/jupyter/clear_cell.html","loc":"redoules.github.io/jupyter/clear_cell.html"},{"title":"What's inside my .bashrc ?","text":"############ # Anaconda # ############ export PATH = \"/station/guillaume/anaconda3/bin: $PATH \" alias python = '/station/guillaume/anaconda3/bin/python' ######### # Alias # ######### alias ding = 'echo -e \"\\a\"' alias calc = 'python -ic \"from __future__ import division; from math import *\"' alias h = \"history|grep \" alias f = \"find . |grep \" alias p = \"ps aux |grep \" alias cdl = \"cd /data/guillaume\" alias cp = \"rsync -avz --progress\" alias grep = \"grep --color=auto\" alias ls = \"ls -hN --color=auto --group-directories-first\" alias ll = \"ls -hal\" alias sv = \"ssh compute_cluster\" alias ms = \"ls\" alias jl = \"jupyter lab\" alias lst = \"jupyter notebook list\" ########################## # bashrc personnalisation# ########################## force_color_prompt = yes export EDITOR = nano export BROWSER = \"firefox '%s' &\" if [ -n \" $force_color_prompt \" ] ; then if [ -x /usr/bin/tput ] && tput setaf 1 > & /dev/null ; then # We have color support; assume it's compliant with Ecma-48 # (ISO/IEC-6429). (Lack of such support is extremely rare, and such # a case would tend to support setf rather than setaf.) color_prompt = yes else color_prompt = fi fi if [ \" $color_prompt \" = yes ] ; then #\\h : hostname #\\u : user #\\w : current working directory #\\d : date #\\t : time yellow = 226 green = 83 pink = 198 blue = 34 PS1 = \"\\[\\033[38;5;22m\\]\\u\\[ $( tput sgr0 ) \\]\\[\\033[38;5;163m\\]@\\[ $( tput sgr0 ) \\]\\[\\033[38;5;22m\\]\\h\\[ $( tput sgr0 ) \\]\\[\\033[38;5;162m\\]:\\[ $( tput sgr0 ) \\]\\[\\033[38;5;172m\\]{\\[ $( tput sgr0 ) \\]\\[\\033[38;5;39m\\]\\w\\[ $( tput sgr0 ) \\]\\[\\033[38;5;172m\\]}\\[ $( tput sgr0 ) \\]\\[\\033[38;5;162m\\]>\\[ $( tput sgr0 ) \\]\"","tags":"Linux","url":"redoules.github.io/linux/bashrc.html","loc":"redoules.github.io/linux/bashrc.html"},{"title":"Efficient extraction of eigenvalues from a list of tensors","text":"When you manipulate FEM results you generally have either a: * scalar field, * vector field, * tensor field. With tensorial results, it is often useful to extract the eigenvalues in order to find the principal values. I have found that it is easier to store the components of the tensors in a 6 column pandas dataframe (because of the symmetric property of stress and strain tensors) import pandas as pd node = [ 1001 , 1002 , 1003 , 1004 ] #when dealing with FEM results you should remember at which element/node the result is computed (in the example, let's assume that we look at node from 1001 to 1004) tensor1 = [ 1 , 1 , 1 , 0 , 0 , 0 ] #eigen : 1 tensor2 = [ 4 , - 1 , 0 , 2 , 2 , 1 ] #eigen : 5.58443, -1.77931, -0.805118 tensor3 = [ 1 , 6 , 5 , 3 , 3 , 1 ] #eigen : 8.85036, 4.46542, -1.31577 tensor4 = [ 1 , 2 , 3 , 0 , 0 , 0 ] #eigen : 1, 2, 3 df = pd . DataFrame ([ tensor1 , tensor2 , tensor3 , tensor4 ], columns = [ \"XX\" , \"YY\" , \"ZZ\" , \"XY\" , \"XZ\" , \"YZ\" ]) df . index = node df .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } XX YY ZZ XY XZ YZ 1001 1 1 1 0 0 0 1002 4 -1 0 2 2 1 1003 1 6 5 3 3 1 1004 1 2 3 0 0 0 If you want to extract the eigenvalues of a tensor with numpy you have to pass a n by n ndarray to the eigenvalue function. In order to avoid having to loop over each node, this oneliner is highly optimized and will help you invert a large number of tensors efficiently. The steps are basically, create a list of n by n values (here n=3) in the right order => reshape it to a list of tensors => pass it to the eigenvals function import numpy as np from numpy import linalg as LA eigenvals = LA . eigvals ( df [[ \"XX\" , \"XY\" , \"XZ\" , \"XY\" , \"YY\" , \"YZ\" , \"XZ\" , \"YZ\" , \"ZZ\" ]] . values . reshape ( len ( df ), 3 , 3 )) eigenvals array ([[ 1 . , 1 . , 1 . ], [ 5 . 58442834 , - 0 . 80511809 , - 1 . 77931025 ], [ - 1 . 31577211 , 8 . 85035616 , 4 . 46541595 ], [ 1 . , 2 . , 3 . ]])","tags":"Python","url":"redoules.github.io/python/Efficient_extraction_of_eigenvalues_from_a_list_of_tensors.html","loc":"redoules.github.io/python/Efficient_extraction_of_eigenvalues_from_a_list_of_tensors.html"},{"title":"Optimized numpy random number generation on Intel CPU","text":"Python Intel distribution Make sure you have a python intel distribution. When you startup python you should see somethine like : Python 3.6.2 |Intel Corporation| (default, Aug 15 2017, 11:34:02) [MSC v.1900 64 bit (AMD64)] Type 'copyright', 'credits' or 'license' for more information IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help. If not, you can force the installation of the intel optimized python with : conda update --all conda config --add channels intel conda install numpy --channel intel --override-channels oh and by the way, make sure you a running an Intel CPU ;) Comparing numpy.random with numpy.random_intel Let's now test both the rand function with and without the Intel optimization import numpy as np from numpy import random , random_intel % timeit np . random . rand ( 10 ** 5 ) 1 . 06 ms ± 91 . 5 µ s per loop ( mean ± std . dev . of 7 runs , 1000 loops each ) % timeit np . random_intel . rand ( 10 ** 5 ) 225 µ s ± 3 . 46 µ s per loop ( mean ± std . dev . of 7 runs , 1000 loops each )","tags":"Python","url":"redoules.github.io/python/Optimized_numpy_random_intel.html","loc":"redoules.github.io/python/Optimized_numpy_random_intel.html"},{"title":"How to check Linux process information?","text":"How to check Linux process information (CPU usage, memory, user information, etc.)? You need to use the ps command combined with the grep command. In the example, we want to check the information on the nginx process : ps aux | grep nginx It would return the output : root 9976 0.0 0.0 12272 108 ? S mtu 1500 group default qlen 1 link/ether 5c:51:4f:41:7a:b1 inet 169.254.33.33/16 brd 169.254.255.255 scope global dynamic valid_lft forever preferred_lft forever inet6 fe80::390a:f69e:1ba2:2121/64 scope global dynamic valid_lft forever preferred_lft forever 3: eth1: mtu 1500 group default qlen 1 link/ether 0a:00:27:00:00:03 inet 192.168.56.1/24 brd 192.168.56.255 scope global dynamic valid_lft forever preferred_lft forever inet6 fe80::84cd:374e:843f:f82f/64 scope global dynamic valid_lft forever preferred_lft forever 15: eth2: <> mtu 1500 group default qlen 1 link/ether 00:ff:d2:8a:19:c3 inet 169.254.40.62/16 brd 169.254.255.255 scope global dynamic valid_lft forever preferred_lft forever inet6 fe80::f199:2ca0:7aff:283e/64 scope global dynamic valid_lft forever preferred_lft forever 1: lo: mtu 1500 group default qlen 1 link/loopback 00:00:00:00:00:00 inet 127.0.0.1/8 brd 127.255.255.255 scope global dynamic valid_lft forever preferred_lft forever inet6 ::1/128 scope global dynamic valid_lft forever preferred_lft forever 5: wifi0: mtu 1500 group default qlen 1 link/ieee802.11 5c:51:4f:41:7a:ad inet 192.168.1.1/24 brd 192.168.1.255 scope global dynamic valid_lft 42720sec preferred_lft 42720sec inet6 fe80::395f:3594:1dc2:57e3/64 scope global dynamic valid_lft forever preferred_lft forever 21: wifi1: <> mtu 1500 group default qlen 1 link/ieee802.11 5c:51:4f:41:7a:ae inet 169.254.12.77/16 brd 169.254.255.255 scope global dynamic valid_lft forever preferred_lft forever inet6 fe80::58d5:630:cbbd:c4d/64 scope global dynamic valid_lft forever preferred_lft forever 12: eth3: <> mtu 1472 group default qlen 1 link/ether 00:00:00:00:00:00:00:e0:00:00:00:00:00:00:00:00 inet6 fe80::100:7f:fffe/64 scope global dynamic valid_lft forever preferred_lft forever 10: eth4: mtu 1500 group default qlen 1 link/ether 22:b7:57:52:5f:ff inet 192.168.42.106/24 brd 192.168.42.255 scope global dynamic valid_lft 6659sec preferred_lft 6659sec inet6 fe80::5110:eb6f:deb0:45c4/64 scope global dynamic valid_lft forever preferred_lft forever You can select only one interface ip addr show eth0 and only the relevant information will be displayed 4: eth0: <> mtu 1500 group default qlen 1 link/ether 5c:51:4f:41:7a:b1 inet 169.254.33.33/16 brd 169.254.255.255 scope global dynamic valid_lft forever preferred_lft forever inet6 fe80::390a:f69e:1ba2:2121/64 scope global dynamic valid_lft forever preferred_lft forever Check your current ip address the old way The ifconfig command will return information regarding your network interfaces. Let's try it: ifconfig eth1 Link encap:Ethernet HWaddr 0a:00:27:00:00:03 inet adr:192.168.56.1 Bcast:192.168.56.255 Masque:255.255.255.0 adr inet6: fe80::84cd:374e:843f:f82f/64 Scope:Global UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Packets reçus:0 erreurs:0 :0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 Octets reçus:0 (0.0 B) Octets transmis:0 (0.0 B) eth4 Link encap:Ethernet HWaddr 22:b7:57:52:5f:ff inet adr:192.168.42.106 Bcast:192.168.42.255 Masque:255.255.255.0 adr inet6: fe80::5110:eb6f:deb0:45c4/64 Scope:Global UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Packets reçus:0 erreurs:0 :0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 Octets reçus:0 (0.0 B) Octets transmis:0 (0.0 B) lo Link encap:Boucle locale inet adr:127.0.0.1 Masque:255.0.0.0 adr inet6: ::1/128 Scope:Global UP LOOPBACK RUNNING MTU:1500 Metric:1 Packets reçus:0 erreurs:0 :0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 Octets reçus:0 (0.0 B) Octets transmis:0 (0.0 B) wifi0 Link encap:UNSPEC HWaddr 5C-51-4F-41-7A-AD-00-00-00-00-00-00-00-00-00-00 inet adr:192.168.1.1 Bcast:192.168.1.255 Masque:255.255.255.0 adr inet6: fe80::395f:3594:1dc2:57e3/64 Scope:Global UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Packets reçus:0 erreurs:0 :0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 Octets reçus:0 (0.0 B) Octets transmis:0 (0.0 B) On the left column you have the list of network adapters. The lo is the local loopback, it is an interface that points to the localhost. Interfaces starting with eth refer to wired connections over ethernet (or sometimes USB in the case of a phone acting as an access point over USB). Interfaces starting with wlan or wifi refer to wireless connections. On the right column you have some information corresponding to the interface such as the IPv4, the IPv6, the mask, some statistics about the interface and so on.","tags":"Linux","url":"redoules.github.io/linux/get_ip_linux.html","loc":"redoules.github.io/linux/get_ip_linux.html"},{"title":"Check the version of the kernel currently running","text":"Check the version of the kernel currently running The uname command will give you the version of the kernel. In order to get a more useful output, type uname -a This will return : * the hostname * os name * kernel release/version * architecture * etc. variations If you only want the kernel version you can type uname -v if you only want the kernel release you can type uname -r","tags":"Linux","url":"redoules.github.io/linux/version_kernel.html","loc":"redoules.github.io/linux/version_kernel.html"},{"title":"Running the notebook on a remote server","text":"Jupyter hub With JupyterHub you can create a multi-user Hub which spawns, manages, and proxies multiple instances of the single-user Jupyter notebook server. Project Jupyter created JupyterHub to support many users. The Hub can offer notebook servers to a class of students, a corporate data science workgroup, a scientific research project, or a high performance computing group. However, if you are the only one using the server and you just want a simple way to run the notebook on your server and access it through the web interface on a light client without having to install and configure the jupyter hub, you can do the following. Problem with jupyter notebook On your server, run the command jupyter-notebook you should get something like : [I 11:18:44.514 NotebookApp] Serving notebooks from local directory: /volume2/homes/admin [I 11:18:44.515 NotebookApp] 0 active kernels [I 11:18:44.516 NotebookApp] The Jupyter Notebook is running at: http://localhost:8888/?token=357587e3269b0f20f2b7e1918492890ae7573ac7ef1d2023 [I 11:18:44.516 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [W 11:18:44.519 NotebookApp] No web browser found: could not locate runnable browser. [C 11:18:44.520 NotebookApp] Copy/paste this URL into your browser when you connect for the first time, to login with a token: http://localhost:8888/?token=357587e3269b0f20f2b7e1918492890ae7573ac7ef1d2023 and if you try to connect to your server ip (in my example : http://192.168.1.2:8888/?token=357587e3269b0f20f2b7e1918492890ae7573ac7ef1d2023) you will get an \"ERR_CONNECTION_REFUSED\" error. This is because, by default, Jupyter Notebook only accepts connections from localhost. Allowing connexions from other sources From any IP The simplest way to avoid the connection error is to allow the notebook to accept connections from any ip jupyter-notebook --ip = * you will get something like [W 11:26:45.285 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended. [I 11:26:45.342 NotebookApp] Serving notebooks from local directory: /volume2/homes/admin [I 11:26:45.342 NotebookApp] 0 active kernels [I 11:26:45.343 NotebookApp] The Jupyter Notebook is running at: http://[all ip addresses on your system]:8888/?token=52af33d628881824968b4031967e8541a27cc28b1720c199 [I 11:26:45.343 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [W 11:26:45.346 NotebookApp] No web browser found: could not locate runnable browser. [C 11:26:45.347 NotebookApp] Copy/paste this URL into your browser when you connect for the first time, to login with a token: http://localhost:8888/?token=52af33d628881824968b4031967e8541a27cc28b1720c199 and if you connect form a remote client (192.168.1.1 in my example), the following line will be added to the output : [I 11:26:54.798 NotebookApp] 302 GET /?token=52af33d628881824968b4031967e8541a27cc28b1720c199 (192.168.1.1) 111.17ms note that you should only do that if you are the only one using the server because the connection is not encypted. From a specific IP You can also, explicitly specify the ip of the client jupyter-notebook --ip = 192 .168.1.1 [I 11:44:58.104 NotebookApp] JupyterLab extension loaded from C:\\Users\\Guillaume\\Miniconda3\\lib\\site-packages\\jupyterlab [I 11:44:58.104 NotebookApp] JupyterLab application directory is C:\\Users\\Guillaume\\Miniconda3\\share\\jupyter\\lab [I 11:44:58.244 NotebookApp] Serving notebooks from local directory: C:\\Users\\Guillaume [I 11:44:58.245 NotebookApp] 0 active kernels [I 11:44:58.245 NotebookApp] The Jupyter Notebook is running at: http://192.168.1.1:8888/?token=503576dd8fa87d1f2c416df307e9b900e520b4942e317b32 [I 11:44:58.245 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [C 11:44:58.258 NotebookApp] Copy/paste this URL into your browser when you connect for the first time, to login with a token: http://192.168.1.1:8888/?token=503576dd8fa87d1f2c416df307e9b900e520b4942e317b32 [I 11:44:59.083 NotebookApp] Accepting one-time-token-authenticated connection from 192.168.1.1","tags":"Jupyter","url":"redoules.github.io/jupyter/remote_run_notebook.html","loc":"redoules.github.io/jupyter/remote_run_notebook.html"},{"title":"Running multiple calls to a function in parallel with Dask","text":"Dask.distributed is a lightweight library for distributed computing in Python. It allows to create a compute graph. Dask distributed is architectured around 3 parts : the dask-scheduler the dask-worker(s) the dask client Dask architecture The Dask scheduler is a centrally managed, distributed, dynamic task scheduler. It recieves tasks from a/multiple client(s) and spread them across one or multiple dask-worker(s). Dask-scheduler is an event based asynchronous dynamic scheduler, meaning that mutliple clients can submit a list of task to be executed on multiple workers. Internally, the task are represented as a directed acyclic graph. Both new clients and new workers can be connected or disconnected during the execution of the task graph. Tasks can be submited with the function client . submit ( function , * args , ** kwargs ) or by using objects from the dask library such as dask.dataframe, dask.bag or dask.array Setup In this example, we will use a distributed scheduler on a single machine with multiple workers and a single client. We will use the client to submit some tasks to the scheduler. The scheduler will then dispatch those tasks to the workers. The process can be monitored in real time through a web application. For this example, all the computations will be run on a local computer. However dask can scale to a large HPC cluster. First we have to launch the dask-scheduler; from the command line, input dask-scheduler Next, you can load the web dashboard. In order to do so, the scheduler returns the number of the port you have to connect to in the line starting with \"bokeh at :\". The default port is 8787. Since we are running all the programs on the same computer, we just have to login to http://127.0.0.1:8787/status Finally, we have to launch the dask-worker(s). If you want to run the worker(s) on the same computer as the scheduler the type : dask-worker 127 .0.0.1:8786 otherwise, make sure you are inputing the ip address of the computer hosting the dask-scheduler. You can launch as many workers as you want. In this example, we will run 3 workers on the local machine. Use the dask workers within your python code We will now see how to submit multiple calls to a fucntion in parallel on the dask-workers. Import the required libraries and define the function to be executed. import numpy as np import pandas as pd from distributed import Client #function used to do parallel computing on def compute_pi_MonteCarlo ( Nb_Data ): \"\"\" computes the value of pi using the monte carlo method \"\"\" Radius = 1 Nb_Data = int ( round ( Nb_Data )) x = np . random . uniform ( - Radius , Radius , Nb_Data ) y = np . random . uniform ( - Radius , Radius , Nb_Data ) pi_mc = 4 * np . sum ( np . power ( x , 2 ) + np . power ( y , 2 ) < Radius ** 2 ) / Nb_Data err = 100 * np . abs ( pi_mc - np . pi ) / np . pi return [ Nb_Data , pi_mc , err ] In order to connect to the scheduler, we create a client. client = Client ( '127.0.0.1:8786' ) client Client Scheduler: tcp://127.0.0.1:8786 Dashboard: http://127.0.0.1:8787/status Cluster Workers: 3 Cores: 12 Memory: 25.48 GB We submit tasks using the submit method data = [ client . submit ( compute_pi_MonteCarlo , Nb_Data ) for Nb_Data in np . logspace ( 3 , 7 , num = 1200 , dtype = int )] If you look at http://127.0.0.1:8787/status you will see the tasks beeing completed. Once competed, gather the data: data = client . gather ( data ) df = pd . DataFrame ( data ) df . columns = [ \"number of points for MonteCarlo\" , \"value of pi\" , \"error (%)\" ] df . tail () .dataframe thead tr:only-child th { text-align: right; } .dataframe thead th { text-align: left; } .dataframe tbody tr th { vertical-align: top; } number of points for MonteCarlo value of pi error (%) 1195 9697405 3.141296 0.009454 1196 9772184 3.141058 0.017008 1197 9847540 3.141616 0.000739 1198 9923477 3.141009 0.018574 1199 10000000 3.141032 0.017833 There, we have completed a simple example on how to use dask to run multiple functions in parallel. Full source code: import numpy as np import pandas as pd from distributed import Client #function used to do parallel computing on def compute_pi_MonteCarlo ( Nb_Data ): \"\"\" computes the value of pi using the monte carlo method \"\"\" Radius = 1 Nb_Data = int ( round ( Nb_Data )) x = np . random . uniform ( - Radius , Radius , Nb_Data ) y = np . random . uniform ( - Radius , Radius , Nb_Data ) pi_mc = 4 * np . sum ( np . power ( x , 2 ) + np . power ( y , 2 ) < Radius ** 2 ) / Nb_Data err = 100 * np . abs ( pi_mc - np . pi ) / np . pi return [ Nb_Data , pi_mc , err ] #connect to the scheduler client = Client ( '127.0.0.1:8786' ) #submit tasks data = [ client . submit ( compute_pi_MonteCarlo , Nb_Data ) for Nb_Data in np . logspace ( 3 , 7 , num = 1200 , dtype = int )] #gather the results data = client . gather ( data ) df = pd . DataFrame ( data ) df . columns = [ \"number of points for MonteCarlo\" , \"value of pi\" , \"error (%)\" ] df . tail () A word on the environement variables On Windows, to make sure that you can run dask-scheduler and dask-worker from the command line, you have to add the location of the executable to your path. On linux, you can append the location of the dask-worker and scheduler to the path variable with the command export PATH = $PATH :/path/to/dask","tags":"Python","url":"redoules.github.io/python/dask_distributed_parallelism.html","loc":"redoules.github.io/python/dask_distributed_parallelism.html"},{"title":"Plotting data using log axis","text":"Plotting in log axis with matplotlib import matplotlib.pyplot as plt % matplotlib inline import numpy as np x = np . linspace ( 0.1 , 20 ) y = 20 * np . exp ( - x / 10.0 ) Plotting using the standard function then specifying the axis scale One of the easiest way to plot in a log plot is to specify the plot normally and then specify which axis is to be plotted with a log scale. This can be specified by the function set_xscale or set_yscale # Normal plot fig = plt . figure () ax = fig . add_subplot ( 1 , 1 , 1 ) ax . plot ( x , y ) ax . grid () plt . show () # Log x axis plot fig = plt . figure () ax = fig . add_subplot ( 1 , 1 , 1 ) ax . plot ( x , y ) ax . set_xscale ( 'log' ) ax . grid () plt . show () # Log x axis plot fig = plt . figure () ax = fig . add_subplot ( 1 , 1 , 1 ) ax . plot ( x , y ) ax . set_yscale ( 'log' ) ax . grid () plt . show () # Log x axis plot fig = plt . figure () ax = fig . add_subplot ( 1 , 1 , 1 ) ax . plot ( x , y ) ax . set_xscale ( 'log' ) ax . set_yscale ( 'log' ) ax . grid () plt . show () Plotting using the matplotlib defined function Matplotlib has the function : semilogx, semilogy and loglog that can help you avoid having to specify the axis scale. # Plot using semilogx fig = plt . figure () ax = fig . add_subplot ( 1 , 1 , 1 ) ax . semilogx ( x , y ) ax . grid () plt . show () # Plot using semilogy fig = plt . figure () ax = fig . add_subplot ( 1 , 1 , 1 ) ax . semilogy ( x , y ) ax . grid () plt . show () # Plot using loglog fig = plt . figure () ax = fig . add_subplot ( 1 , 1 , 1 ) ax . loglog ( x , y ) ax . grid () plt . show ()","tags":"Python","url":"redoules.github.io/python/logplot.html","loc":"redoules.github.io/python/logplot.html"},{"title":"Downloading a static webpage with python","text":"If you are using python legacy (aka python 2) first of all, stop ! Furthermore, this method won't work in python legacy # Import modules from urllib.request import urlopen The webpage source code can be downloaded with the command urlopen url = \"http://example.com/\" #create a HTTP request in order to read the page page = urlopen ( url ) . read () The source code will be stored in the variable page as a string print ( page ) b ' \\n \\n \\n Example Domain \\n\\n \\n \\n \\n \\n \\n\\n \\n \\n
Example Domain
\\n
This domain is established to be used for illustrative examples in documents. You may use this \\n domain in examples without prior coordination or asking for permission.
\\n
More information...
\\n
element we will then get the estimated value contained in the
element The code First, let's import selenium, beautiful soup and some other libraries # Selenium allows to control chrome programmatically from selenium import webdriver from selenium.webdriver.common.keys import Keys from selenium.webdriver.chrome.options import Options #beautifulsoup is used to parse the dom of the html page import bs4 as BeautifulSoup import numpy as np import pandas as pd import os We will also need some extra libraries for plotting the results import matplotlib.pyplot as plt from matplotlib.transforms import offset_copy import cartopy.crs as ccrs import cartopy.io.img_tiles as cimgt Let's define the GPS coordinates of the two destinations we are interested in. The coordinates can be found the the URL of a google maps search longitudeDestination1 = 48.9361537 latitudeDestination1 = 2.2507129 longitudeDestination2 = 48.7783875 latitudeDestination2 = 2.1803534 We will search on an equally spaced grid of point starting from (long_begin, lat_begin) and going to (long_end, lat_end). In order to do so, we will : * construct the URL from the GPS coordinates * load the url in chrome with driver.get * read the resulting html with driver.page_source * parse the html with beautiful soup in order to find the first element with the class section-directions-trip-numbers * in this element, we will get the estimated travel time by reading the text value of the second
element def get_travel_time ( url , driver ): \"\"\" get the estimated travel time of the google maps given as url \"\"\" resultats = None driver . get ( url ) while resultats == None : soupe = BeautifulSoup . BeautifulSoup ( driver . page_source , \"lxml\" ) soupe . select ( \"section-directions-trip-numbers\" ) resultats = soupe . find ( 'div' , attrs = { \"class\" : u \"section-directions-trip-numbers\" }) return resultats . find_all ( \"span\" )[ 2 ] Once the function is defined, we only need to call it in a loop in order to get all the point of the grid chrome_options = Options () #chrome_options.add_argument(\"--disable-extensions\") #chrome_options.add_argument(\"--disable-gpu\") chrome_options . add_argument ( \"--headless\" ) #make chrome headless. If you want to see the autimation, comment this line driver = webdriver . Chrome ( executable_path = '. \\\\ chromedriver.exe' , chrome_options = chrome_options ) nb = 10 ctn = 0 time = [] for coordX in np . linspace ( long_begin , long_end , nb ): for coordY in np . linspace ( lat_begin , lat_end , nb ): url_journey1 = f \"https://www.google.com/maps/dir/{coordX},{coordY}/@{longitudeDestination1},{latitudeDestination1},12z/data=!3m1!4b1!4m14!4m13!1m0!1m5!1m1!1s0x47e67bff078f6575:0x95df2619f9304bd7!2m2!1d2.1825421!2d48.778384!2m4!2b1!6e0!7e2!8j1570521600!3e0\" url_journey2 = f 'https://www.google.com/maps/dir/{coordX},{coordY}/@{longitudeDestination2},{latitudeDestination2},14z/data=!3m1!4b1!4m14!4m13!1m0!1m5!1m1!1s0x47e665df0cb0b919:0x5f513cdf2fe6d39d!2m2!1d2.2572779!2d48.9368666!2m4!2b1!6e0!7e2!8j1570521600!3e0' temps_user1 = get_travel_time ( url_journey1 , driver ) temps_user2 = get_travel_time ( url_journey2 , driver ) ctn += 1 print ( f 'Downloaded : {ctn/(nb*nb)*100}%' ) time . append ([ coordX , coordY , temps_user1 . text , temps_user2 . text , f 'https://www.google.com/maps/place/{coordX},{coordY}' ]) Downloaded : 1.0 % Downloaded : 2.0 % Downloaded : 3.0 % Downloaded : 4.0 % [...] Downloaded : 96.0 % Downloaded : 97.0 % Downloaded : 98.0 % Downloaded : 99.0 % Downloaded : 100.0 % After gathering the results, the values stored in the time list are string and cannot be interpreted as numerical values without a post processing. This is why I've written the function analyse_time in order to split the text and convert it to a numerical format expressed in minutes. def analyse_time ( time ): \"\"\" Analyse the time given by google maps, splits the lower and higher estimate and converts them to minutes \"\"\" tlow = time . split ( \" - \" )[ 0 ] . replace ( \" \\xa0 \" , \" \" ) thigh = time . split ( \" - \" )[ 1 ] . replace ( \" \\xa0 \" , \" \" ) if ( \"min\" not in tlow ) and ( \"h\" not in tlow ): #example : 26 tlow = int ( tlow . replace ( \" \" , \"\" )) elif \"h\" not in tlow : # example 26 min tlow = tlow . replace ( \"min\" , \"\" ) tlow = int ( tlow . replace ( \" \" , \"\" )) else : if \"min\" in tlow : #example 1h 26min tlow = tlow . split ( \"h\" ) tlow = 60 * int ( tlow [ 0 ] . replace ( \" \" , \"\" )) + int ( tlow [ 1 ] . replace ( \"min\" , \"\" ) . replace ( \" \" , \"\" )) else : #example 1h tlow = 60 * int ( tlow . replace ( \"h\" , \"\" )) if \"h\" not in thigh : thigh = thigh . replace ( \"min\" , \"\" ) thigh = int ( thigh . replace ( \" \" , \"\" )) else : if \"min\" in thigh : thigh = thigh . split ( \"h\" ) thigh = 60 * int ( thigh [ 0 ] . replace ( \" \" , \"\" )) + int ( thigh [ 1 ] . replace ( \"min\" , \"\" ) . replace ( \" \" , \"\" )) else : thigh = 60 * int ( thigh . replace ( \"h\" , \"\" )) return ( tlow , thigh ) For every result previously gathered, let's apply the function analyse_time then put it in a pandas dataframe. While we are at it, I also computed the geometric mean of the minimum time estimated for both users as well of the maximum time. A geometric mean is interesting in this interesting here because we want to avoid have one user doing a long journey while the other is doing a short one. df = [] for t in time : lat = t [ 0 ] lon = t [ 1 ] t1 = analyse_time ( t [ 2 ]) t2 = analyse_time ( t [ 3 ]) geomlow = np . sqrt ( t1 [ 0 ] * t2 [ 0 ]) #geometric mean geomhigh = np . sqrt ( t1 [ 1 ] * t2 [ 1 ]) #geometric mean df . append ([ lat , lon , geomlow , geomhigh , t1 [ 0 ], t1 [ 1 ], t2 [ 0 ], t2 [ 1 ]]) traveltime = pd . DataFrame ( df , columns = [ \"latitude\" , \"longitude\" , \"geometric mean low\" , \"geometric mean high\" , \"time low 1\" , \"time high 1\" , \"time low 2\" , \"time high 2\" ]) traveltime = traveltime . sort_values ( \"geometric mean low\" ) traveltime = traveltime . reset_index () traveltime . to_csv ( \"extraction.csv\" , index = False ) #save it to csv traveltime . head ( 10 ) #print the 10 first rows .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } index latitude longitude geometric mean low geometric mean high time low 1 time high 1 time low 2 time high 2 0 38 48.833333 2.231111 16.124515 33.166248 10 20 26 55 1 89 48.888889 2.260000 16.248077 36.742346 22 45 12 30 2 97 48.900000 2.202222 16.733201 40.987803 35 70 8 24 3 6 48.800000 2.173333 16.733201 35.777088 8 16 35 80 4 98 48.900000 2.231111 17.320508 41.109610 30 65 10 26 5 99 48.900000 2.260000 17.663522 37.815341 26 55 12 26 6 88 48.888889 2.231111 17.663522 40.620192 26 55 12 30 7 7 48.800000 2.202222 17.748239 34.641016 9 16 35 75 8 68 48.866667 2.231111 18.000000 37.416574 18 35 18 40 9 87 48.888889 2.202222 18.330303 43.874822 28 55 12 35 Ok now that we finished preparing the data, it's time to draw some maps. We will use caropy in order to download some Google maps tiles. You might need to manually change the extent of the map. % matplotlib inline plt . rcParams [ 'figure.figsize' ] = 20 , 12 # Create a Stamen terrain background instance. stamen_terrain = cimgt . GoogleTiles () fig = plt . figure () # Create a GeoAxes in the tile's projection. ax = fig . add_subplot ( 1 , 1 , 1 , projection = stamen_terrain . crs ) # Limit the extent of the map to a small longitude/latitude range. ax . set_extent ([ lat_begin * 0.975 , lat_end * 1.02 , long_begin * 0.999 , long_end * 1.001 ], crs = ccrs . Geodetic ()) # Add the Stamen data at zoom level 10. ax . add_image ( stamen_terrain , 10 ) Now, we draw the 10 points that minimize time for user 1, color them is red and make the size of the pot proportionnal to the travel time of the second user. And we do the same for he 10 points that minimize time for user 2, color them is blue and make the size of the pot proportionnal to the travel time of the first user. for i , point in traveltime . sort_values ( \"time low 1\" ) . iterrows (): if i < 10 : ax . plot ( point . longitude , point . latitude , marker = 'o' , c = 'red' , markersize = point [ \"time low 2\" ], alpha = 0.5 , transform = ccrs . Geodetic ()) for i , point in traveltime . sort_values ( \"time low 2\" ) . iterrows (): if i < 10 : ax . plot ( point . longitude , point . latitude , marker = 'o' , c = 'blue' , markersize = point [ \"time low 1\" ], alpha = 0.5 , transform = ccrs . Geodetic ()) To help with the vizualisation, we add two stars on the maps in order to mark the location of the 2 destinations. # Add a marker for destination 1 ax . plot ( latitudeDestination1 , longitudeDestination1 , marker = '*' , c = 'green' , markersize = 25 , alpha = 1 , transform = ccrs . Geodetic ()) # Add a marker for destination 2 ax . plot ( latitudeDestination2 , longitudeDestination2 , marker = '*' , c = 'orange' , markersize = 25 , alpha = 1 , transform = ccrs . Geodetic ()) geodetic_transform = ccrs . Geodetic () . _as_mpl_transform ( ax ) text_transform = offset_copy ( geodetic_transform , units = 'dots' , x =- 25 ) Finally, we draw the maps. The optimal point is where both the dots in blue and in red are smaller. plt . show ()","tags":"Projects","url":"redoules.github.io/projects/get_traffic_data.html","loc":"redoules.github.io/projects/get_traffic_data.html"},{"title":"Get items in one dictionnary but not the other one","text":"Say we have two similar dictonnaries, we want to find all the items that are in the second dictonnary but not in the first one. dict1 = { \"Banana\" : \"yellow\" , \"Orange\" : \"orange\" } dict2 = { \"Banana\" : \"yellow\" , \"Orange\" : \"orange\" , \"Lemon\" : \"yellow\" } In the following code we find the difference of the keys and then rebuild a dict taking the corresponding values. difference = { k : dict2 [ k ] for k in set ( dict2 ) - set ( dict1 ) } difference { 'Lemon' : 'yellow' } Be careful, this operation is not symetric. This means that if we want to find a value present in dict1 but not in dict2 this code won't work dict1 = { \"Banana\" : \"yellow\" , \"Orange\" : \"orange\" , \"Lemon\" : \"yellow\" } dict2 = { \"Banana\" : \"yellow\" , \"Orange\" : \"orange\" } difference = { k : dict2 [ k ] for k in set ( dict2 ) - set ( dict1 ) } difference {}","tags":"Python","url":"redoules.github.io/python/compare_dict.html","loc":"redoules.github.io/python/compare_dict.html"},{"title":"Maximize a window in Windows","text":"A window can be maximized by using the win32 API. We will need to import win32gui and win32con import win32gui import win32con In order to maximize a windows, we need its handle number. In our example we will get the handle of the foreground window by using the GetForegroundWindow method. handle = win32gui . GetForegroundWindow () The window can then be maximized with the command win32gui . ShowWindow ( handle , win32con . SW_MAXIMIZE ) 24 minimized with the command win32gui . ShowWindow ( handle , win32con . SW_MINIMIZE ) 24 or restored to its original size with the command win32gui . ShowWindow ( handle , win32con . SW_NORMAL ) 24 The win32 API allow to hide or show a window. import time win32gui . ShowWindow ( handle , win32con . SW_HIDE ) #hide the window time . sleep ( 1 ) #keep it hidden for a second win32gui . ShowWindow ( handle , win32con . SW_SHOW ) #show the window 0","tags":"Python","url":"redoules.github.io/python/maximize_window.html","loc":"redoules.github.io/python/maximize_window.html"},{"title":"Iterate over a dictionnary","text":"To iterate over a dictionnary in a for loop, you have to use the following syntaxe for key , value in d . items (): Here is a examples printing the keys and values of a dictionnary d = { \"key1\" : \"value 1\" , \"key2\" : \"value 2\" , \"key3\" : \"value 3\" } for key , value in d . items (): print ( f '{key} : {value}' ) key1 : value 1 key2 : value 2 key3 : value 3","tags":"Python","url":"redoules.github.io/python/iterate_dict.html","loc":"redoules.github.io/python/iterate_dict.html"},{"title":"Case sensitive ConfigParser","text":"In order to have a case sensite ConfigParser, simply replace the ConfigParser with the following class : from configparser import ConfigParser class CaseConfigParser ( ConfigParser ): def optionxform ( self , optionstr ): return optionstr","tags":"Python","url":"redoules.github.io/python/case_config.html","loc":"redoules.github.io/python/case_config.html"},{"title":"Write a value to a config file","text":"You can store variables in an ini file for later executions of the script instead of hardcoding the values in the script. ConfigParser can take a pointer to a file an write values to that file using the ini syntaxe config = ConfigParser () config . read ( \"file.ini\" ) config . set ( \"my_section\" , \"my_param\" , \"my_value\" ) with open ( \"file.ini\" , \"w\" ) as f : config . write ( f ) Here is a function designed to help you update ini files def saveParam ( pathToFile , section , param , value ): \"\"\" Save/add value to an ini file pathToFile : string, path to the ini file section : string, name of the section to write to param : string, name of the parameter to write the value to value : value to be written to the parameter If the file doesn't exist, it will be created If the section doesn't exist, it will be created \"\"\" config = ConfigParser () if not os . path . isfile ( pathToFile ): #create the file if it does not exist with open ( pathToFile , \"w\" ) as f : pass config . read ( pathToFile ) if not config . has_section ( section ): #create the section if it does not exist config . add_section ( section ) config . set ( section , param , value ) with open ( pathToFile , \"w\" ) as f : #write value to the file config . write ( f )","tags":"Python","url":"redoules.github.io/python/write_config_file.html","loc":"redoules.github.io/python/write_config_file.html"},{"title":"Parula colormap for matplotlib","text":"from matplotlib.colors import LinearSegmentedColormap cm_data = [[ 0.2081 , 0.1663 , 0.5292 ], [ 0.2116238095 , 0.1897809524 , 0.5776761905 ], [ 0.212252381 , 0.2137714286 , 0.6269714286 ], [ 0.2081 , 0.2386 , 0.6770857143 ], [ 0.1959047619 , 0.2644571429 , 0.7279 ], [ 0.1707285714 , 0.2919380952 , 0.779247619 ], [ 0.1252714286 , 0.3242428571 , 0.8302714286 ], [ 0.0591333333 , 0.3598333333 , 0.8683333333 ], [ 0.0116952381 , 0.3875095238 , 0.8819571429 ], [ 0.0059571429 , 0.4086142857 , 0.8828428571 ], [ 0.0165142857 , 0.4266 , 0.8786333333 ], [ 0.032852381 , 0.4430428571 , 0.8719571429 ], [ 0.0498142857 , 0.4585714286 , 0.8640571429 ], [ 0.0629333333 , 0.4736904762 , 0.8554380952 ], [ 0.0722666667 , 0.4886666667 , 0.8467 ], [ 0.0779428571 , 0.5039857143 , 0.8383714286 ], [ 0.079347619 , 0.5200238095 , 0.8311809524 ], [ 0.0749428571 , 0.5375428571 , 0.8262714286 ], [ 0.0640571429 , 0.5569857143 , 0.8239571429 ], [ 0.0487714286 , 0.5772238095 , 0.8228285714 ], [ 0.0343428571 , 0.5965809524 , 0.819852381 ], [ 0.0265 , 0.6137 , 0.8135 ], [ 0.0238904762 , 0.6286619048 , 0.8037619048 ], [ 0.0230904762 , 0.6417857143 , 0.7912666667 ], [ 0.0227714286 , 0.6534857143 , 0.7767571429 ], [ 0.0266619048 , 0.6641952381 , 0.7607190476 ], [ 0.0383714286 , 0.6742714286 , 0.743552381 ], [ 0.0589714286 , 0.6837571429 , 0.7253857143 ], [ 0.0843 , 0.6928333333 , 0.7061666667 ], [ 0.1132952381 , 0.7015 , 0.6858571429 ], [ 0.1452714286 , 0.7097571429 , 0.6646285714 ], [ 0.1801333333 , 0.7176571429 , 0.6424333333 ], [ 0.2178285714 , 0.7250428571 , 0.6192619048 ], [ 0.2586428571 , 0.7317142857 , 0.5954285714 ], [ 0.3021714286 , 0.7376047619 , 0.5711857143 ], [ 0.3481666667 , 0.7424333333 , 0.5472666667 ], [ 0.3952571429 , 0.7459 , 0.5244428571 ], [ 0.4420095238 , 0.7480809524 , 0.5033142857 ], [ 0.4871238095 , 0.7490619048 , 0.4839761905 ], [ 0.5300285714 , 0.7491142857 , 0.4661142857 ], [ 0.5708571429 , 0.7485190476 , 0.4493904762 ], [ 0.609852381 , 0.7473142857 , 0.4336857143 ], [ 0.6473 , 0.7456 , 0.4188 ], [ 0.6834190476 , 0.7434761905 , 0.4044333333 ], [ 0.7184095238 , 0.7411333333 , 0.3904761905 ], [ 0.7524857143 , 0.7384 , 0.3768142857 ], [ 0.7858428571 , 0.7355666667 , 0.3632714286 ], [ 0.8185047619 , 0.7327333333 , 0.3497904762 ], [ 0.8506571429 , 0.7299 , 0.3360285714 ], [ 0.8824333333 , 0.7274333333 , 0.3217 ], [ 0.9139333333 , 0.7257857143 , 0.3062761905 ], [ 0.9449571429 , 0.7261142857 , 0.2886428571 ], [ 0.9738952381 , 0.7313952381 , 0.266647619 ], [ 0.9937714286 , 0.7454571429 , 0.240347619 ], [ 0.9990428571 , 0.7653142857 , 0.2164142857 ], [ 0.9955333333 , 0.7860571429 , 0.196652381 ], [ 0.988 , 0.8066 , 0.1793666667 ], [ 0.9788571429 , 0.8271428571 , 0.1633142857 ], [ 0.9697 , 0.8481380952 , 0.147452381 ], [ 0.9625857143 , 0.8705142857 , 0.1309 ], [ 0.9588714286 , 0.8949 , 0.1132428571 ], [ 0.9598238095 , 0.9218333333 , 0.0948380952 ], [ 0.9661 , 0.9514428571 , 0.0755333333 ], [ 0.9763 , 0.9831 , 0.0538 ]] parula_map = LinearSegmentedColormap . from_list ( 'parula' , cm_data ) # For use of \"viscm view\" test_cm = parula_map % matplotlib inline import matplotlib.pyplot as plt import numpy as np plt . rcParams [ 'figure.figsize' ] = 20 , 12 try : from viscm import viscm viscm ( parula_map ) except ImportError : print ( \"viscm not found, falling back on simple display\" ) plt . imshow ( np . linspace ( 0 , 100 , 256 )[ None , :], aspect = 'auto' , cmap = parula_map ) plt . show ()","tags":"Python","url":"redoules.github.io/python/Parula.html","loc":"redoules.github.io/python/Parula.html"},{"title":"List all opened windows on Windows","text":"You can use the function get_all_windows to get a dictonnary containing the titles of the opened windows as keys and the handles of those windows as values import win32gui def get_all_windows (): \"\"\" Returns dict with window desc and hwnd, \"\"\" def _MyCallback ( hwnd , extra ): hwnds , classes = extra hwnds . append ( hwnd ) classes [ win32gui . GetWindowText ( hwnd )] = hwnd windows = [] classes = {} win32gui . EnumWindows ( _MyCallback , ( windows , classes )) return classes get_all_windows () { '' : 3802422 , 'Forcepad driver tray window' : 65676 , 'Jauge de batterie' : 131542 , 'Network Flyout' : 131650 , 'Dashlane' : 5570658 , 'Wox' : 131770 , 'JupyterLab - Brave' : 66990 , 'python' : 4261478 , 'Visual Studio Code - Insiders' : 329780 , 'Code - Insiders' : 526478 , 'Documents' : 526010 , 'Windows PowerShell' : 198580 , 'Progression' : 394934 , 'Microsoft Edge' : 131586 , 'Microsoft Store' : 197328 , 'QTrayIconMessageWindow' : 327816 , 'Hidden Window' : 459506 , '.NET-BroadcastEventWindow.4.0.0.0.3e2c690.0' : 131824 , 'SystemResourceNotifyWindow' : 197346 , 'MediaContextNotificationWindow' : 197344 , 'Resilio Sync 2.6.3' : 262934 , } print ( \"List of all opened windows : \" ) for key , value in get_all_windows () . items (): if key != \"\" : print ( \" \\t * \" + key . split ( \" \\n \" )[ 0 ]) List of all opened windows : * Forcepad driver tray window * Jauge de batterie * Network Flyout * Dashlane * Wox * JupyterLab - Brave * python * Visual Studio Code - Insiders * Code - Insiders * Documents * Windows PowerShell * Progression * Microsoft Edge * Microsoft Store * QTrayIconMessageWindow * Hidden Window * . NET - BroadcastEventWindow . 4 . 0 . 0 . 0 . 3 e2c690 . 0 * SystemResourceNotifyWindow * MediaContextNotificationWindow * Resilio Sync 2 . 6 . 3","tags":"Python","url":"redoules.github.io/python/list_windows.html","loc":"redoules.github.io/python/list_windows.html"},{"title":"Reverse column order in pandas","text":"# Import modules import pandas as pd # Example dataframe raw_data = { 'fruit' : [ 'Banana' , 'Orange' , 'Apple' , 'lemon' , \"lime\" , \"plum\" ], 'color' : [ 'yellow' , 'orange' , 'red' , 'yellow' , \"green\" , \"purple\" ], 'kcal' : [ 89 , 47 , 52 , 15 , 30 , 28 ] } df = pd . DataFrame ( raw_data , columns = [ 'fruit' , 'color' , 'kcal' ]) df .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } fruit color kcal 0 Banana yellow 89 1 Orange orange 47 2 Apple red 52 3 lemon yellow 15 4 lime green 30 5 plum purple 28 The columns of a dataframe can be reversed by using the loc accessor and passing :,::-1 . The : before the , means select all rows and the ::-1 after the , means reverse the column order df . loc [:,:: - 1 ] .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } kcal color fruit 0 89 yellow Banana 1 47 orange Orange 2 52 red Apple 3 15 yellow lemon 4 30 green lime 5 28 purple plum","tags":"Python","url":"redoules.github.io/python/reverse_column_order.html","loc":"redoules.github.io/python/reverse_column_order.html"},{"title":"Reverse row order in pandas","text":"# Import modules import pandas as pd # Example dataframe raw_data = { 'fruit' : [ 'Banana' , 'Orange' , 'Apple' , 'lemon' , \"lime\" , \"plum\" ], 'color' : [ 'yellow' , 'orange' , 'red' , 'yellow' , \"green\" , \"purple\" ], 'kcal' : [ 89 , 47 , 52 , 15 , 30 , 28 ] } df = pd . DataFrame ( raw_data , columns = [ 'fruit' , 'color' , 'kcal' ]) df .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } fruit color kcal 0 Banana yellow 89 1 Orange orange 47 2 Apple red 52 3 lemon yellow 15 4 lime green 30 5 plum purple 28 The rows of a dataframe can be reversed by using the loc accessor and passing ::-1 . This notation is the same as the one used to reverse a list in python df . loc [:: - 1 ] .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } fruit color kcal 5 plum purple 28 4 lime green 30 3 lemon yellow 15 2 Apple red 52 1 Orange orange 47 0 Banana yellow 89 If you want to reset the index as well so that the dataframe starts with a 0, you can combine what we just learned with the reset_index method df . loc [:: - 1 ] . reset_index ( drop = True ) .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } fruit color kcal 0 plum purple 28 1 lime green 30 2 lemon yellow 15 3 Apple red 52 4 Orange orange 47 5 Banana yellow 89 that way, the rows are in reverse order but the index column has been re-initialized so it starts with 0","tags":"Python","url":"redoules.github.io/python/reverse_row_order.html","loc":"redoules.github.io/python/reverse_row_order.html"},{"title":"Rename columns in pandas","text":"In this article, we will be renaming columns in a pandas dataframe. First, let's import pandas and create an example dataframe # Import modules import pandas as pd # Example dataframe raw_data = { 'fruit' : [ 'Banana' , 'Orange' , 'Apple' , 'lemon' , \"lime\" , \"plum\" ], 'color' : [ 'yellow' , 'orange' , 'red' , 'yellow' , \"green\" , \"purple\" ], 'kcal' : [ 89 , 47 , 52 , 15 , 30 , 28 ] } df = pd . DataFrame ( raw_data , columns = [ 'fruit' , 'color' , 'kcal' ]) df .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } fruit color kcal 0 Banana yellow 89 1 Orange orange 47 2 Apple red 52 3 lemon yellow 15 4 lime green 30 5 plum purple 28 The most flexible method for renaming columns in pandas is the rename method. It takes a dictornnary as an argument where : * the keys are the old names * the values are the new names you also need to specify the axis. This method can be used to rename either one or multiple columns df = df . rename ({ \"fruit\" : \"produce\" , \"kcal\" : \"energy\" }, axis = \"columns\" ) df .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } produce color energy 0 Banana yellow 89 1 Orange orange 47 2 Apple red 52 3 lemon yellow 15 4 lime green 30 5 plum purple 28 If you want to rename all the columns at once, a common method is to rewrite the columns attribute of the dataframe df . columns = [ \"nice fruit\" , \"bright color\" , \"light kcal\" ] df .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } nice fruit bright color light kcal 0 Banana yellow 89 1 Orange orange 47 2 Apple red 52 3 lemon yellow 15 4 lime green 30 5 plum purple 28 If the only thing you are doing is replacing a space with an underscore, an even better method is to use the str.replace method since you don't have to type all the column names df . columns = df . columns . str . replace ( \" \" , \"_\" ) df .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } nice_fruit bright_color light_kcal 0 Banana yellow 89 1 Orange orange 47 2 Apple red 52 3 lemon yellow 15 4 lime green 30 5 plum purple 28 Similarly, you can use other str methods such as : * capitalize : in order to converts first character to capital letter * lower : in order to have lowercase column names * upper : in order to have uppercase column names * etc. df . columns = df . columns . str . capitalize () df .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Nice_fruit Bright_color Light_kcal 0 Banana yellow 89 1 Orange orange 47 2 Apple red 52 3 lemon yellow 15 4 lime green 30 5 plum purple 28 Finaly, if you only need to add a prefix or a suffix to the columns, you can use the add_prefix method df . add_prefix ( \"pre_\" ) .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } pre_Nice_fruit pre_Bright_color pre_Light_kcal 0 Banana yellow 89 1 Orange orange 47 2 Apple red 52 3 lemon yellow 15 4 lime green 30 5 plum purple 28 or the add_suffix method df . add_suffix ( \"_post\" ) .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Nice_fruit_post Bright_color_post Light_kcal_post 0 Banana yellow 89 1 Orange orange 47 2 Apple red 52 3 lemon yellow 15 4 lime green 30 5 plum purple 28","tags":"Python","url":"redoules.github.io/python/rename_columns_pandas.html","loc":"redoules.github.io/python/rename_columns_pandas.html"},{"title":"Update a value in a TABLE","text":"In this very simple example we will see how to update a row in a sql database Since we are working in the notebook, we will load the sql extension in order to manipulate the database. The database mydatabase.db is a SQLite database already created before the example. #load the extension % load_ext sql #connect to the database % sql sqlite : /// mydatabase . db 'Connected: @mydatabase.db' The content of the TABLE is the following % sql SELECT * FROM tutyfrutty * sqlite : /// mydatabase . db Done . index fruit color kcal 0 Banana yellow 89 1 Orange orange 47 2 Apple red 52 3 lemon yellow 15 4 lime green 30 5 plum purple 28 7 Cranberry red 308 We want to update the kcal value for \"Orange\". We will use the UPDATE keyword in order to do so. We can use the index value in order to access the row and set a new value. % sql UPDATE \"tutyfrutty\" SET \"kcal\" = 48 WHERE \"index\" = 1 ; % sql SELECT * FROM tutyfrutty WHERE \"index\" = 1 ; * sqlite : /// mydatabase . db 1 rows affected . * sqlite : /// mydatabase . db Done . index fruit color kcal 1 Orange orange 48 The WHERE keyword can be used to specify mutilple rows. For instance, if we want to change the kcal value of all yellow fruits : % sql UPDATE \"tutyfrutty\" SET \"kcal\" = 126 WHERE \"color\" = \"yellow\" ; % sql SELECT * FROM tutyfrutty * sqlite : /// mydatabase . db 2 rows affected . * sqlite : /// mydatabase . db Done . index fruit color kcal 0 Banana yellow 126 1 Orange orange 48 2 Apple red 52 3 lemon yellow 126 4 lime green 30 5 plum purple 28 7 Cranberry red 308","tags":"SQL","url":"redoules.github.io/sql/update_table.html","loc":"redoules.github.io/sql/update_table.html"},{"title":"Parse variable from config file","text":"You can store variables in an ini file for later executions of the script instead of hardcoding the values in the script. Here is the content of config.ini : [section1] var_a:hello var_b:world [section2] myvariable: 42 Use the configparser library for an easy access and parsing of the file. import configparser config = configparser . ConfigParser () config . read ( \"config.ini\" ) var_a = config . get ( \"section1\" , \"var_a\" ) var_b = config . get ( \"section1\" , \"var_b\" ) myvariable = config . get ( \"section2\" , \"myvariable\" ) print ( var_a , var_b ) print ( myvariable ) hello world 42","tags":"Python","url":"redoules.github.io/python/config_parse.html","loc":"redoules.github.io/python/config_parse.html"},{"title":"Granting access to a database to a remote client by IP","text":"The MariaDB users are granted access based on where the connection is coming from. By default the access is restricted to: * 127.0.0.1 * localhost The program you are using to connect is not indentifying itself as 127.0.0.1 or localhost. You will have to verify the IP access it's being identified as, then add that to your grant table. In order to add grant an IP address you can use the following commands : GRANT ALL ON * . * TO 'user' @ 'computer.host.com' ; GRANT ALL ON * . * TO 'user' @ '192.168.1.6' ; GRANT ALL ON * . * TO 'user' @ '%' ; The % is a wildcard that means any IP. Be careful when using it especially on the root user. If you want to remove the rights you granted, you can use the command REVOKE ALL PRIVILEGES ON * . * TO 'user' @ 'computer.host.com' ; REVOKE ALL PRIVILEGES ON * . * TO 'user' @ '192.168.1.6' ; REVOKE ALL PRIVILEGES ON * . * FROM 'user' @ '%'","tags":"SQL","url":"redoules.github.io/sql/grant_ip.html","loc":"redoules.github.io/sql/grant_ip.html"},{"title":"Using pandas extended types","text":"Historically, pandas is bound to numpy arrays and its limitations : * integer and bool types cannot store missing data, indeed, np.nan is of type np.float * non-numeric types such as categorical, datetime are not natively supported. Indeed, internally, pandas relies on numpy arrays to efficiently store and perform operations on the data. With the recent versions of pandas, it is possible to define custom types. In order to avoid having to extensively update the internal pandas code with each new extensions, it is now possible to define : * an extension type which describes the data type and can be registered for creation via a string (i.e. ...astype(\"example\") ) * an extension array which is a class that handles the datatype. There is no real restriction on its construction though it must be convertible to a numpy array in order to make it work with the functions implemented in pandas. It is limited to one dimension. Some implementations are already implemented in pandas. For example, IntegerNA is a pandas extension that can handle missing integer data without casting to float (and loosing the precision). In order to use IntergerNA, you need to specify the type as \"Int64\" with a capital \"i\" import pandas as pd import numpy as np s = pd . Series ([ 1 , np . nan , 2 ]) print ( s ) s_ext = pd . Series ([ 1 , np . nan , 2 ], dtype = \"Int64\" ) print ( s_ext ) 0 1 . 0 1 NaN 2 2 . 0 dtype : float64 0 1 . 0 1 NaN 2 2 . 0 dtype : Int64 It is possible to leverage this extension to avoid running out of memory. For instance, we can store data using the \"UInt16\" type, this will avoid having to cast to float64. Other uses might be handling special data types such as : * ip adresses (see cyberpandas) * gps locations * etc... sources : * https://pandas.pydata.org/pandas-docs/stable/extending.html * PyData LA 2018 Extending Pandas with Custom Types - Will Ayd","tags":"Python","url":"redoules.github.io/python/pandas_extended_types.html","loc":"redoules.github.io/python/pandas_extended_types.html"},{"title":"Basic operations with SED","text":"In this article we will learn about some of the main uses we can use sed for : * replacing * deleting * printing For this example we will learn how to remove the comments starting with the '#' sign and the blank lines for the following file : ## Header of input . csv # this file contains information I want to parse with a simple program . # The header , the footer or any comment starting with a \" # \" will be removed # The blank lines will also be removed # img , processed , defaut # bloc 1 0 , a0000 . tif ,, 1 , a0001 . tif , True , \" (139, 63)(145, 91) \" 2 , a0002 . tif , True , \" (93, 72)(24, 162)(31, 64) \" 3 , a0003 . tif ,, 4 , a0004 . tif ,, 5 , a0005 . tif ,, 6 , a0006 . tif ,, 7 , a0007 . tif ,, 8 , a0008 . tif ,, 9 , a0009 . tif , True , \" (127, 80)(104, 60)(87, 63)(53, 78)(17, 126) \" 10 , a0010 . tif ,, 11 , a0011 . tif , True , \" (39, 78)(84, 110) \" # a random comment passing by # end of bloc 1 # bloc 2 12 , a0012 . tif ,, 13 , a0013 . tif ,, 14 , a0014 . tif ,, 15 , a0015 . tif , True , \" (146, 65)(146, 89)(139, 146)(16, 68) \" 16 , a0016 . tif , True , \" (51, 59)(77, 69)(145, 78)(139, 112)(97, 123)(17, 148) \" 17 , a0017 . tif ,, # end of bloc 2 # bloc 3 18 , a0018 . tif ,, 19 , a0019 . tif ,, 20 , a0020 . tif , True , \" (57, 99)(12, 113)(27, 139)(16, 158) \" 21 , a0021 . tif ,, 22 , a0022 . tif ,, 23 , a0023 . tif ,, 24 , a0024 . tif ,, 25 , a0025 . tif ,, 26 , a0026 . tif ,, # end of bloc 3 27 , a0027 . tif , True , \" (11, 86)(29, 74)(92, 68)(109, 129)(132, 104) \" 28 , a0028 . tif ,, 29 , a0029 . tif , True , \" (128, 58) \" 30 , a0030 . tif , True , \" (133, 59)(99, 77)(111, 100)(115, 153) \" 31 , a0031 . tif , True , \" (43, 154)(27, 177) \" ## footer : end of file Anatomy of a SED command If we run the command : sed \"\" input.csv Everything inside the brackets will be interpreted as a sed command. In our case, there is nothing hence the file will be printed to the consol without any modification. You can put in the quotation mark one of sed's many commands for instance s that stands for substitude and is one of the most commonly used. In our csv we use the comma separator, let's say that we want to change it to a semicolon. We would do : sed \"s/,/;/\" input.csv so we have s meaning that we want to use the replace command followed by a / and the caracter(s) we want to replace followed by a / and the caracter(s) we want to replace it with and finally a / . The result is the following : ## Header # this file contains information I want to parse with a simple program . # The header ; the footer or any comment starting with a \"#\" will be removed # The blank lines will also be removed # img ;processed,defaut # bloc 1 0 ;a0000.tif,, 1 ;a0001.tif,True,\"(139, 63)(145, 91)\" 2 ;a0002.tif,True,\"(93, 72)(24, 162)(31, 64)\" 3 ;a0003.tif,, 4 ;a0004.tif,, 5 ;a0005.tif,, 6 ;a0006.tif,, 7 ;a0007.tif,, 8 ;a0008.tif,, 9 ;a0009.tif,True,\"(127, 80)(104, 60)(87, 63)(53, 78)(17, 126)\" 10 ;a0010.tif,, 11 ;a0011.tif,True,\"(39, 78)(84, 110)\" # a random comment passing by # end of bloc 1 # bloc 2 12 ;a0012.tif,, 13 ;a0013.tif,, 14 ;a0014.tif,, 15 ;a0015.tif,True,\"(146, 65)(146, 89)(139, 146)(16, 68)\" 16 ;a0016.tif,True,\"(51, 59)(77, 69)(145, 78)(139, 112)(97, 123)(17, 148)\" 17 ;a0017.tif,, # end of bloc 2 # bloc 3 18 ;a0018.tif,, 19 ;a0019.tif,, 20 ;a0020.tif,True,\"(57, 99)(12, 113)(27, 139)(16, 158)\" 21 ;a0021.tif,, 22 ;a0022.tif,, 23 ;a0023.tif,, 24 ;a0024.tif,, 25 ;a0025.tif,, 26 ;a0026.tif,, # end of bloc 3 27 ;a0027.tif,True,\"(11, 86)(29, 74)(92, 68)(109, 129)(132, 104)\" 28 ;a0028.tif,, 29 ;a0029.tif,True,\"(128, 58)\" 30 ;a0030.tif,True,\"(133, 59)(99, 77)(111, 100)(115, 153)\" 31 ;a0031.tif,True,\"(43, 154)(27, 177)\" ## footer : end of file As you can see, only the first comma has been replaced, in order to repeat the command multiple times per line we need to specify the g option sed \"s/,/;/g\" input.csv ## Header # this file contains information I want to parse with a simple program . # The header ; the footer or any comment starting with a \"#\" will be removed # The blank lines will also be removed # img ;processed;defaut # bloc 1 0 ;a0000.tif;; 1 ;a0001.tif;True;\"(139; 63)(145; 91)\" 2 ;a0002.tif;True;\"(93; 72)(24; 162)(31; 64)\" 3 ;a0003.tif;; 4 ;a0004.tif;; 5 ;a0005.tif;; 6 ;a0006.tif;; 7 ;a0007.tif;; 8 ;a0008.tif;; 9 ;a0009.tif;True;\"(127; 80)(104; 60)(87; 63)(53; 78)(17; 126)\" 10 ;a0010.tif;; 11 ;a0011.tif;True;\"(39; 78)(84; 110)\" # a random comment passing by # end of bloc 1 # bloc 2 12 ;a0012.tif;; 13 ;a0013.tif;; 14 ;a0014.tif;; 15 ;a0015.tif;True;\"(146; 65)(146; 89)(139; 146)(16; 68)\" 16 ;a0016.tif;True;\"(51; 59)(77; 69)(145; 78)(139; 112)(97; 123)(17; 148)\" 17 ;a0017.tif;; # end of bloc 2 # bloc 3 18 ;a0018.tif;; 19 ;a0019.tif;; 20 ;a0020.tif;True;\"(57; 99)(12; 113)(27; 139)(16; 158)\" 21 ;a0021.tif;; 22 ;a0022.tif;; 23 ;a0023.tif;; 24 ;a0024.tif;; 25 ;a0025.tif;; 26 ;a0026.tif;; # end of bloc 3 27 ;a0027.tif;True;\"(11; 86)(29; 74)(92; 68)(109; 129)(132; 104)\" 28 ;a0028.tif;; 29 ;a0029.tif;True;\"(128; 58)\" 30 ;a0030.tif;True;\"(133; 59)(99; 77)(111; 100)(115; 153)\" 31 ;a0031.tif;True;\"(43; 154)(27; 177)\" ## footer : end of file Removing comments In order to remove comments, we can replace the pattern of a comment by nothing. A commant starts with a # sign and is followed by an arbitrarly long chain of characters. In order to match this pattern, we will use a regular expression. sed \"s/#.*//g\" input.csv #.* means : find strings that start with a # , the . stands for any character, finally, the * means that the . can be repeated any number of times. That means that sed will look for a string starting with a # followed by any characters that come after in the line. If we want to do something a bit cleaner, we can try to remove any whitespace before the comments as well. In order to do so, we will use the \\s regular expression that represents a whitespace. If we want to make sure that we removed any whitespace before the comments, we will do \\s* The final regular expression is then \\s*#.* and it will then be replaced by nothing. Finally, the command becomes sed \"s/\\s*#.*//g\" input.csv If we run that command, all our comments have disappeared 0 , a0000 . tif ,, 1 , a0001 . tif , True , \"(139, 63)(145, 91)\" 2 , a0002 . tif , True , \"(93, 72)(24, 162)(31, 64)\" 3 , a0003 . tif ,, 4 , a0004 . tif ,, 5 , a0005 . tif ,, 6 , a0006 . tif ,, 7 , a0007 . tif ,, 8 , a0008 . tif ,, 9 , a0009 . tif , True , \"(127, 80)(104, 60)(87, 63)(53, 78)(17, 126)\" 10 , a0010 . tif ,, 11 , a0011 . tif , True , \"(39, 78)(84, 110)\" 12 , a0012 . tif ,, 13 , a0013 . tif ,, 14 , a0014 . tif ,, 15 , a0015 . tif , True , \"(146, 65)(146, 89)(139, 146)(16, 68)\" 16 , a0016 . tif , True , \"(51, 59)(77, 69)(145, 78)(139, 112)(97, 123)(17, 148)\" 17 , a0017 . tif ,, 18 , a0018 . tif ,, 19 , a0019 . tif ,, 20 , a0020 . tif , True , \"(57, 99)(12, 113)(27, 139)(16, 158)\" 21 , a0021 . tif ,, 22 , a0022 . tif ,, 23 , a0023 . tif ,, 24 , a0024 . tif ,, 25 , a0025 . tif ,, 26 , a0026 . tif ,, 27 , a0027 . tif , True , \"(11, 86)(29, 74)(92, 68)(109, 129)(132, 104)\" 28 , a0028 . tif ,, 29 , a0029 . tif , True , \"(128, 58)\" 30 , a0030 . tif , True , \"(133, 59)(99, 77)(111, 100)(115, 153)\" 31 , a0031 . tif , True , \"(43, 154)(27, 177)\" Removing blank lines In order to remove the blank lines, we need to specify to sed a pattern corresponding to a blank line and use the d command where the d stands for delete. The delete command expects the pattern to the between / Now let's define what pattern a blank line corresponds to. Since there is no symbol for blankness, we can do the following ^$ . ^ means the begining of a line and $ corresponds to the end of a line. So whenever we find a blank line we delete it. sed \"/^ $ / d\" input.csv ## Header # this file contains information I want to parse with a simple program . # The header , the footer or any comment starting with a \" # \" will be removed # The blank lines will also be removed # img , processed , defaut # bloc 1 0 , a0000 . tif ,, 1 , a0001 . tif , True , \" (139, 63)(145, 91) \" 2 , a0002 . tif , True , \" (93, 72)(24, 162)(31, 64) \" 3 , a0003 . tif ,, 4 , a0004 . tif ,, 5 , a0005 . tif ,, 6 , a0006 . tif ,, 7 , a0007 . tif ,, 8 , a0008 . tif ,, 9 , a0009 . tif , True , \" (127, 80)(104, 60)(87, 63)(53, 78)(17, 126) \" 10 , a0010 . tif ,, 11 , a0011 . tif , True , \" (39, 78)(84, 110) \" # a random comment passing by # end of bloc 1 # bloc 2 12 , a0012 . tif ,, 13 , a0013 . tif ,, 14 , a0014 . tif ,, 15 , a0015 . tif , True , \" (146, 65)(146, 89)(139, 146)(16, 68) \" 16 , a0016 . tif , True , \" (51, 59)(77, 69)(145, 78)(139, 112)(97, 123)(17, 148) \" 17 , a0017 . tif ,, # end of bloc 2 # bloc 3 18 , a0018 . tif ,, 19 , a0019 . tif ,, 20 , a0020 . tif , True , \" (57, 99)(12, 113)(27, 139)(16, 158) \" 21 , a0021 . tif ,, 22 , a0022 . tif ,, 23 , a0023 . tif ,, 24 , a0024 . tif ,, 25 , a0025 . tif ,, 26 , a0026 . tif ,, # end of bloc 3 27 , a0027 . tif , True , \" (11, 86)(29, 74)(92, 68)(109, 129)(132, 104) \" 28 , a0028 . tif ,, 29 , a0029 . tif , True , \" (128, 58) \" 30 , a0030 . tif , True , \" (133, 59)(99, 77)(111, 100)(115, 153) \" 31 , a0031 . tif , True , \" (43, 154)(27, 177) \" ## footer : end of file combining the sed commands We can concatenate sed commands by separating them with a semicolon. Hence the final sed command will be : sed \"s/\\s*#.*//g;/^ $ / d\" input.csv and the final output is 0 , a0000 . tif ,, 1 , a0001 . tif , True , \"(139, 63)(145, 91)\" 2 , a0002 . tif , True , \"(93, 72)(24, 162)(31, 64)\" 3 , a0003 . tif ,, 4 , a0004 . tif ,, 5 , a0005 . tif ,, 6 , a0006 . tif ,, 7 , a0007 . tif ,, 8 , a0008 . tif ,, 9 , a0009 . tif , True , \"(127, 80)(104, 60)(87, 63)(53, 78)(17, 126)\" 10 , a0010 . tif ,, 11 , a0011 . tif , True , \"(39, 78)(84, 110)\" 12 , a0012 . tif ,, 13 , a0013 . tif ,, 14 , a0014 . tif ,, 15 , a0015 . tif , True , \"(146, 65)(146, 89)(139, 146)(16, 68)\" 16 , a0016 . tif , True , \"(51, 59)(77, 69)(145, 78)(139, 112)(97, 123)(17, 148)\" 17 , a0017 . tif ,, 18 , a0018 . tif ,, 19 , a0019 . tif ,, 20 , a0020 . tif , True , \"(57, 99)(12, 113)(27, 139)(16, 158)\" 21 , a0021 . tif ,, 22 , a0022 . tif ,, 23 , a0023 . tif ,, 24 , a0024 . tif ,, 25 , a0025 . tif ,, 26 , a0026 . tif ,, 27 , a0027 . tif , True , \"(11, 86)(29, 74)(92, 68)(109, 129)(132, 104)\" 28 , a0028 . tif ,, 29 , a0029 . tif , True , \"(128, 58)\" 30 , a0030 . tif , True , \"(133, 59)(99, 77)(111, 100)(115, 153)\" 31 , a0031 . tif , True , \"(43, 154)(27, 177)\" Overwritting the file If we want to overwrite the result of sed on the input file, we need to add the option -i . So if you are running sed without -i , it is safe and won't alter your files. sed -i \"s/\\s*#.*//g;/^ $ / d\" input.csv","tags":"Linux","url":"redoules.github.io/linux/sed.html","loc":"redoules.github.io/linux/sed.html"},{"title":"How to create a ramdisk?","text":"Writing to a SSD Let's first write 500Mo to a ssd with the command : dd if = /dev/zero of = test.iso bs = 1M count = 500 On my machine with a ssd I get the following results 500 + 0 records in 500 + 0 records out 524288000 bytes ( 524 MB , 500 MiB ) copied , 10 , 153 s , 51 , 6 MB / s Writing to a ramdisk Let's now create a temporary ram disk and see how faster it is mkdir /mnt/ram mount -t tmpfs tmpfs /mnt/ram -o size = 600M cd /mnt/ram dd if = /dev/zero of = test.iso bs = 1M count = 500 The results yield a faster write time : 500 + 0 records in 500 + 0 records out 524288000 bytes ( 524 MB , 500 MiB ) copied , 6 , 39849 s , 81 , 9 MB / s on a machine with linux installed natively the results shoud be even faster.","tags":"Linux","url":"redoules.github.io/linux/ramdisk.html","loc":"redoules.github.io/linux/ramdisk.html"},{"title":"Kullback Leibler Divergence","text":"The Kullback Leibler Divergence also famously called KL Divergence. The KL divergence actually measures the difference between any two distributions. The KL divergence is defined as : $$D_{KL}(p||q)=\\int_{X} p(x)log\\frac{p(x)}{q(x)} dx$$ And it is always non-negative and zero only when P is equal to Q. Indeed, when P is equal to Q, the log of 1 is zero, and that's why the distance is zero. Otherwise, it is always some non-negative quantity. It is can be view as a distance measure but in reality it is not because it isn't a symertic operator and because it doesn't follow the triangle law. $$D_{KL}(p||q)\\neq D_{KL}(q||p)$$ Where to use the KL-Divergence In supervised learning you are always trying to model our data to a particular distribution. So in that case our \\(P\\) can be the unknown distribution. We usually want to build an estimated probability distribution \\(Q\\) based on the sample samples \\(X\\) . When the estimator is perfect, \\(P\\) and \\(Q\\) are the same hence \\(D_{KL}(p||q) = 0\\) . This means that the KL-Divergence can be used as a mesure of the error. Jonathon Shlens explains that the KL-Divergence can be interpreted as measuring the likelihood that samples form an empirical distribution \\(p\\) were generated by a fixed distribution \\(q\\) . $$D_{KL}(p||q)=\\int_{X} p(x)log\\frac{p(x)}{q(x)}dx$$ $$D_{KL}(p||q)=\\int_{X}\\left( -p(x)log q(x) +p(x) log p(x) \\right) dx$$ The entropy of p is defined as \\(H(p)=-\\int_{X}p(x) log(p(x)) dx\\) The cross entropy between p and q is defined as $H(p,q)=-\\int_{X} p(x)log q(x) $ Hence: $$D_{KL}(p||q)=H(p,q) - H(p)$$ In many machine learning algorithm, in particular deep learning, the optimization problem revolves around minimizing the cross-entropy. Taking into account what we learnt above : $$H(p,q) = H(p) + D_{KL}(p||q) $$ The cross entropy between two probability distributions \\(p\\) and \\(q\\) over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set, if a coding scheme is used that is optimized for an \"artificial\" probability distribution \\(q\\) , rather than the \"true\" distribution \\(p\\) . This means that, the cross entropy corresponds to the number of bits needed to encode the distribution \\(p\\) given by \\(H(p)\\) the entropy of \\(p\\) plus the number of bits needed to encode the directed divergence between the two distributions \\(p\\) and \\(q\\) given by \\(D_{KL}(p||q)\\) , the Kullback Leibler Divergence if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) { var align = \"center\", indent = \"0em\", linebreak = \"false\"; if (false) { align = (screen.width < 768) ? \"left\" : align; indent = (screen.width < 768) ? \"0em\" : indent; linebreak = (screen.width < 768) ? 'true' : linebreak; } var mathjaxscript = document.createElement('script'); mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#'; mathjaxscript.type = 'text/javascript'; mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML'; mathjaxscript[(window.opera ? \"innerHTML\" : \"text\")] = \"MathJax.Hub.Config({\" + \" config: ['MMLorHTML.js'],\" + \" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } },\" + \" jax: ['input/TeX','input/MathML','output/HTML-CSS'],\" + \" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js'],\" + \" displayAlign: '\"+ align +\"',\" + \" displayIndent: '\"+ indent +\"',\" + \" showMathMenu: true,\" + \" messageStyle: 'normal',\" + \" tex2jax: { \" + \" inlineMath: [ ['\\\\\\\\(','\\\\\\\\)'] ], \" + \" displayMath: [ ['$$','$$'] ],\" + \" processEscapes: true,\" + \" preview: 'TeX',\" + \" }, \" + \" 'HTML-CSS': { \" + \" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} },\" + \" linebreaks: { automatic: \"+ linebreak +\", width: '90% container' },\" + \" }, \" + \"}); \" + \"if ('default' !== 'default') {\" + \"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"}\"; (document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript); }","tags":"Mathematics","url":"redoules.github.io/mathematics/KLD.html","loc":"redoules.github.io/mathematics/KLD.html"},{"title":"Downloading from quandl","text":"In this example we will download the price of bitcoin from quandl import quandl btc_price_data = quandl . get ( \"BCHARTS/COINBASEEUR\" ) btc_price_data . head () .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Open High Low Close Volume (BTC) Volume (Currency) Weighted Price Date 2015-05-11 215.85 219.15 214.67 217.83 145.863137 31656.416712 217.028218 2015-05-12 218.04 218.50 214.16 215.50 127.225520 27467.730961 215.897966 2015-05-13 216.43 217.45 208.53 208.88 111.808285 24014.796862 214.785486 2015-05-14 209.23 209.83 204.94 207.93 148.228400 30825.088327 207.956696 2015-05-15 207.95 209.51 207.62 208.55 127.718800 26586.847126 208.167060","tags":"Python","url":"redoules.github.io/python/quandl.html","loc":"redoules.github.io/python/quandl.html"},{"title":"Computing the Mayer multiple","text":"I've learnt about the Mayer mutliple from The Inverstor Podcast . The Mayer multiple is the ratio of the bitcoin price divided by the 200-day moving average. It is designed to understand the price of bitcoin without taking in account the short term volatility. It helps investors filter out their emotions during a bull run. Let's see how to compute the Mayer mutliple in python. First, we need to import the data, we will use Quandl to download data from coinbase import quandl btc_price_data = quandl . get ( \"BCHARTS/COINBASEEUR\" ) btc_price_data . tail () .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Open High Low Close Volume (BTC) Volume (Currency) Weighted Price Date 2018-12-12 2966.00 3076.71 2952.05 3026.00 1447.627465 4.372890e+06 3020.728514 2018-12-13 3025.19 3028.06 2861.15 2886.91 2125.242928 6.261750e+06 2946.369017 2018-12-14 2886.91 2919.00 2800.32 2835.50 2527.558347 7.256959e+06 2871.134083 2018-12-15 2835.49 2865.00 2781.47 2830.45 1267.004758 3.568614e+06 2816.575409 2018-12-16 2830.45 2830.45 2830.44 2830.45 0.144249 4.082886e+02 2830.447385 Next we need to compute the 200 days moving average of the price of bitcoin moving_averages = btc_price_data [[ \"Open\" , \"High\" , \"Low\" , \"Close\" ]] . rolling ( window = 200 ) . mean () moving_averages . tail () .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Open High Low Close Date 2018-12-12 5507.88295 5611.72570 5380.60150 5491.43610 2018-12-13 5491.44155 5595.14440 5363.77840 5474.38560 2018-12-14 5474.39910 5577.89585 5347.27500 5457.99125 2018-12-15 5457.94930 5559.49145 5330.78235 5439.75570 2018-12-16 5439.71660 5540.89425 5313.62955 5422.17700 Finally, we can compute the ratio and plot it. % matplotlib inline import matplotlib.pyplot as plt plt . rcParams [ 'savefig.dpi' ] = 300 plt . rcParams [ 'figure.dpi' ] = 163 plt . rcParams [ 'figure.autolayout' ] = False plt . rcParams [ 'figure.figsize' ] = 20 , 12 plt . rcParams [ 'font.size' ] = 26 mayer_multiple = btc_price_data / moving_averages mayer_multiple [ \"High\" ] . plot () plt . title ( \"Mayer Mutliple over time\" ) plt . ylabel ( \"Mayer Mutliple\" ) plt . xlabel ( \"Time\" ) print ( f \"Mayer multiple {mayer_multiple.iloc[-1]['High']}\" ) print ( f \"Mayer multiple average {mayer_multiple.mean()['High']}\" ) Mayer multiple 0 . 5108290958630005 Mayer multiple average 1 . 3789102045356179 Lastly, I wanted to plot the distribution of the Mayer multiple import numpy as np x = mayer_multiple [ \"High\" ] . values x = x [ ~ np . isnan ( x )] n , a , patches = plt . hist ( x , 100 , facecolor = 'green' , alpha = 0.75 , density = True ) plt . axvline ( x = 2.4 , color = \"red\" ) plt . annotate ( 'We are here today' , xy = ( mayer_multiple . iloc [ - 1 ][ \"High\" ], n [( np . abs ( bins - mayer_multiple . iloc [ - 1 ][ \"High\" ])) . argmin ()]), xytext = ( mayer_multiple . iloc [ - 1 ][ \"High\" ] * 3 , n . max () / 2 ), arrowprops = dict ( facecolor = 'black' , shrink = 0.05 ), ) plt . title ( \"Distribution of the Mayer mutliple\" ) plt . plot () []","tags":"Cryptocurrencies","url":"redoules.github.io/cryptocurrencies/mayer_multiple.html","loc":"redoules.github.io/cryptocurrencies/mayer_multiple.html"},{"title":"Load a saved model","text":"A keras model can be loaded from a hdf5 file. Becareful, a keras generated hdf5 can contain either : the model weights (obtained by the .save_weights() method) the model weights and the model architecture (obtained by the .save() method) In our case, we will load both the model and its weights with the load_model function from the keras.models module from keras.models import load_model model = load_model ( \"my_model.h5\" ) type ( model ) keras . engine . sequential . Sequential","tags":"DL","url":"redoules.github.io/dl/keras_load.html","loc":"redoules.github.io/dl/keras_load.html"},{"title":"Get model input shape","text":"Information on the model can be accessed by the .input_shape parameter of the model object from keras.models import load_model model = load_model ( \"my_model.h5\" ) model . input_shape ( None , 28 , 28 )","tags":"DL","url":"redoules.github.io/dl/keras_input.html","loc":"redoules.github.io/dl/keras_input.html"},{"title":"Get model info and number of parameters","text":"Information on the model can be accessed by the .summary() method from keras.models import load_model model = load_model ( \"my_model.h5\" ) model . summary () _________________________________________________________________ Layer ( type ) Output Shape Param # ================================================================= flatten_2 ( Flatten ) ( None , 784 ) 0 _________________________________________________________________ dense_3 ( Dense ) ( None , 128 ) 100480 _________________________________________________________________ dense_4 ( Dense ) ( None , 10 ) 1290 ================================================================= Total params : 101 , 770 Trainable params : 101 , 770 Non - trainable params : 0 _________________________________________________________________","tags":"DL","url":"redoules.github.io/dl/keras_info.html","loc":"redoules.github.io/dl/keras_info.html"},{"title":"Saving the models weights after each epoch","text":"Let's see how we can save the model weights after every epoch. Let's first import some libraries import keras import numpy as np In this example, we will be using the fashion MNIST dataset to do some basic computer vision, where we will train a Keras neural network to classify items of clothing. In order to import the data we will be using the built in function in Keras : keras . datasets . fashion_mnist . load_data () The model is a very simple neural network consisting in 2 fully connected layers. The model loss function is chosen in order to have a multiclass classifier : \"sparse_categorical_crossentropy\" Let's define a simple feedforward network. ##get and preprocess the data fashion_mnist = keras . datasets . fashion_mnist ( train_images , train_labels ), ( test_images , test_labels ) = fashion_mnist . load_data () train_images = train_images / 255.0 test_images = test_images / 255.0 ## define the model model = keras . Sequential ([ keras . layers . Flatten ( input_shape = ( 28 , 28 )), keras . layers . Dense ( 128 , activation = \"relu\" ), keras . layers . Dense ( 10 , activation = \"softmax\" ) ]) model . compile ( optimizer = \"adam\" , loss = \"sparse_categorical_crossentropy\" , metrics = [ \"acc\" ]) In order to automatically save the model weights to a hdf5 format after every epoch we need to import the ModelCheckpoint callback located in keras . callbacks from keras.callbacks import ModelCheckpoint We now need to define the ModelCheckPoint callback, it takes 7 arguments : filepath: string, path to save the model file. monitor: quantity to monitor verbose: verbosity mode, 0 or 1 save_best_only: if save_best_only=True , the latest best model according to the quantity monitored will not be overwritten. mode: one of {auto, min, max} If save_best_only=True , the decision to overwrite the current save file is made based on either the maximization or the minimization of the monitored quantity. save_weights_only: if True, then only the model's weights will be saved ( model.save_weights(filepath) ), else the full model is saved ( model.save(filepath) ). period: Interval (number of epochs) between checkpoints. The callback has to be added to the callbacks list in the fit method. save_to_hdf5 = ModelCheckpoint ( filepath = \"my_model.h5\" , monitor = 'acc' , verbose = 0 , save_best_only = True , save_weights_only = False , mode = 'auto' , period = 1 ) model . fit ( train_images , train_labels , epochs = 5 , callbacks = [ save_to_hdf5 ]) Epoch 1 / 5 60000 / 60000 [ ============================== ] - 29 s 478 us / step - loss : 0 . 3975 - acc : 0 . 8575 Epoch 2 / 5 60000 / 60000 [ ============================== ] - 98 s 2 ms / step - loss : 0 . 3498 - acc : 0 . 8721 Epoch 3 / 5 60000 / 60000 [ ============================== ] - 95 s 2 ms / step - loss : 0 . 3213 - acc : 0 . 8825 Epoch 4 / 5 60000 / 60000 [ ============================== ] - 64 s 1 ms / step - loss : 0 . 3021 - acc : 0 . 8887 : 6 s - loss : Epoch 5 / 5 60000 / 60000 [ ============================== ] - 61 s 1 ms / step - loss : 0 . 2855 - acc : 0 . 8953 : 4 s - loss : 0 . 284 - ETA : 3 s - loss : - ETA : < keras . callbacks . History at 0 x1d70aa62f28 > import os if \"my_model.h5\" in os . listdir () : print ( \"The model is saved to my_model.h5\" ) else : print ( 'no model saved to disk' ) The model is saved to my_model . h5","tags":"DL","url":"redoules.github.io/dl/model_checkpoint_keras.html","loc":"redoules.github.io/dl/model_checkpoint_keras.html"},{"title":"Logging the training progress in a CSV","text":"Let's see how we can log the progress and various metrics during the training process to a csv file. Let's first import some libraries import keras import numpy as np In this example, we will be using the fashion MNIST dataset to do some basic computer vision, where we will train a Keras neural network to classify items of clothing. In order to import the data we will be using the built in function in Keras : keras . datasets . fashion_mnist . load_data () The model is a very simple neural network consisting in 2 fully connected layers. The model loss function is chosen in order to have a multiclass classifier : \"sparse_categorical_crossentropy\" Let's define a simple feedforward network. ##get and preprocess the data fashion_mnist = keras . datasets . fashion_mnist ( train_images , train_labels ), ( test_images , test_labels ) = fashion_mnist . load_data () train_images = train_images / 255.0 test_images = test_images / 255.0 ## define the model model = keras . Sequential ([ keras . layers . Flatten ( input_shape = ( 28 , 28 )), keras . layers . Dense ( 128 , activation = \"relu\" ), keras . layers . Dense ( 10 , activation = \"softmax\" ) ]) model . compile ( optimizer = \"adam\" , loss = \"sparse_categorical_crossentropy\" , metrics = [ \"accuracy\" , 'mae' ]) In order to stream to a csv file the epoch results and metrics, we define a CSV logger. It is a callback located in keras . callbacks Let's first import it from keras.callbacks import CSVLogger We now need to define the callback by specifiying a file to be written to, the separator and whether to append to the file or erase it every time. The callback has to be added to the callbacks list in the fit method. csv_logger = CSVLogger ( filename = \"my_csv.csv\" , separator = ';' , append = False ) model . fit ( train_images , train_labels , epochs = 5 , callbacks = [ csv_logger ]) Epoch 1 / 5 60000 / 60000 [ ============================== ] - 9 s 148 us / step - loss : 0 . 5020 - acc : 0 . 8234 - mean_absolute_error : 4 . 4200 Epoch 2 / 5 60000 / 60000 [ ============================== ] - 8 s 138 us / step - loss : 0 . 3765 - acc : 0 . 8630 - mean_absolute_error : 4 . 4200 Epoch 3 / 5 60000 / 60000 [ ============================== ] - 8 s 129 us / step - loss : 0 . 3371 - acc : 0 . 8789 - mean_absolute_error : 4 . 4200 Epoch 4 / 5 60000 / 60000 [ ============================== ] - 8 s 133 us / step - loss : 0 . 3129 - acc : 0 . 8843 - mean_absolute_error : 4 . 4200 Epoch 5 / 5 60000 / 60000 [ ============================== ] - 9 s 151 us / step - loss : 0 . 2952 - acc : 0 . 8916 - mean_absolute_error : 4 . 4200 < keras . callbacks . History at 0 x1582adc6780 > The results are stored in the my_csv.csv file and contain the epoch results import pandas as pd pd . read_csv ( \"my_csv.csv\" , sep = \";\" ) .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } epoch acc loss mean_absolute_error 0 0 0.823400 0.502013 4.42 1 1 0.863050 0.376516 4.42 2 2 0.878867 0.337097 4.42 3 3 0.884317 0.312893 4.42 4 4 0.891583 0.295157 4.42","tags":"DL","url":"redoules.github.io/dl/csv_logger_keras.html","loc":"redoules.github.io/dl/csv_logger_keras.html"},{"title":"Random integer","text":"The random module comes with the randint function that returns a pseudo random number between 2 values : # Return a random integer N such that a <= N <= b. random . randint ( a , b ) #### import random min_int = 126 max_int = 211 print ( f \"My pseudo random number between {min_int} and {max_int} : {random.randint(min_int, max_int)}\" ) My pseudo random number between 126 and 211 : 206","tags":"Python","url":"redoules.github.io/python/randint.html","loc":"redoules.github.io/python/randint.html"},{"title":"Using tensorboard with Keras","text":"general workflow Let's see how we can get tensorboard to work with a Keras-based Tensorflow code. import tensorflow as tf import keras import numpy as np In this example, we will be using the fashion MNIST dataset to do some basic computer vision, where we will train a Keras neural network to classify items of clothing. In order to import the data we will be using the built in function in Keras : keras . datasets . fashion_mnist . load_data () The model is a very simple neural network consisting in 2 fully connected layers. The model loss function is chosen in order to have a multiclass classifier : \"sparse_categorical_crossentropy\" Finally, let's train the model for 5 epochs ##get and preprocess the data fashion_mnist = keras . datasets . fashion_mnist ( train_images , train_labels ), ( test_images , test_labels ) = fashion_mnist . load_data () train_images = train_images / 255.0 test_images = test_images / 255.0 ## define the model model = keras . Sequential ([ keras . layers . Flatten ( input_shape = ( 28 , 28 )), keras . layers . Dense ( 128 , activation = \"relu\" ), keras . layers . Dense ( 10 , activation = \"softmax\" ) ]) model . compile ( optimizer = \"adam\" , loss = \"sparse_categorical_crossentropy\" , metrics = [ \"accuracy\" ]) model . fit ( train_images , train_labels , epochs = 5 ) Epoch 1 / 5 60000 / 60000 [ ============================== ] - 9 s 143 us / step - loss : 0 . 4939 - acc : 0 . 8254 Epoch 2 / 5 60000 / 60000 [ ============================== ] - 11 s 182 us / step - loss : 0 . 3688 - acc : 0 . 8661 Epoch 3 / 5 60000 / 60000 [ ============================== ] - 10 s 169 us / step - loss : 0 . 3305 - acc : 0 . 8798 Epoch 4 / 5 60000 / 60000 [ ============================== ] - 21 s 350 us / step - loss : 0 . 3079 - acc : 0 . 8874 Epoch 5 / 5 60000 / 60000 [ ============================== ] - 18 s 302 us / step - loss : 0 . 2889 - acc : 0 . 8927 < keras . callbacks . History at 0 x235c1bc1be0 > During the training we can see the process, including the loss and the accuracy in the output. test_loss , test_acc = model . evaluate ( test_images , test_labels ) print ( f \"Test accuracy : {test_acc}\" ) 10000 / 10000 [ ============================== ] - 1 s 67 us / step Test accuracy : 0 . 8763 When the model finishes training, we get an accuracy of about 87%, and we output some sample predictions predictions = model . predict ( test_images ) print ( predictions [ 0 ]) [ 1 . 8075149 e - 05 3 . 6810281 e - 08 6 . 3094416 e - 07 5 . 1111499 e - 07 1 . 6264809 e - 06 3 . 5973577 e - 04 1 . 0840570 e - 06 3 . 1453002 e - 02 1 . 7062060 e - 06 9 . 6816361 e - 01 ] This kind of process only gives us minimal information during the training process. Setting up tensorboard To make it easier to understand, debug, and optimize TensorFlow programs, a suite of visualization tools called TensorBoard is included. You can use TensorBoard to visualize your TensorFlow graph, plot quantitative metrics about the execution of your graph, and show additional data like images that pass through it. When TensorBoard is fully configured, it looks like this: Let's start by importing the time library and tensorboard itself. It can be found in tensorflow.python.keras.callbacks. from time import time from tensorflow.python.keras.callbacks import TensorBoard After having imported our data and defined our model, we specify a log directory where the training information will get written to. #keep in mind that we already imported the data and defined the model. tensorboard = TensorBoard ( log_dir = f \"logs/{time()}\" ) Finally, to tell Keras to call back to TensorBoard we refer to the instant of TensorBoard we created. model . compile ( optimizer = \"adam\" , loss = \"sparse_categorical_crossentropy\" , metrics = [ \"accuracy\" ]) Now, we need to execute the TensorBoard command pointing at the log directory previously specified. tensorboard --logdir = logs/ TensorBoard will return a http address TensorBoard 1 . 12 . 0 at http : // localhost : 6006 ( Press CTRL + C to quit ) Now, if we retrain again, we can take a look in TensorBoard and start investigating the loss and accuracy model . fit ( train_images , train_labels , epochs = 5 , callbacks = [ tensorboard ]) Epoch 1 / 5 60000 / 60000 [ ============================== ] - 41 s 684 us / step - loss : 0 . 4990 - acc : 0 . 8241 Epoch 2 / 5 60000 / 60000 [ ============================== ] - 49 s 812 us / step - loss : 0 . 3765 - acc : 0 . 8648 Epoch 3 / 5 60000 / 60000 [ ============================== ] - 46 s 765 us / step - loss : 0 . 3392 - acc : 0 . 8766 Epoch 4 / 5 60000 / 60000 [ ============================== ] - 48 s 794 us / step - loss : 0 . 3135 - acc : 0 . 8836 Epoch 5 / 5 60000 / 60000 [ ============================== ] - 49 s 813 us / step - loss : 0 . 2971 - acc : 0 . 8897 < keras . callbacks . History at 0 x235be1c76d8 > TensorBoard also give access to a dynamic visualization of the graph","tags":"DL","url":"redoules.github.io/dl/tensorboard_keras.html","loc":"redoules.github.io/dl/tensorboard_keras.html"},{"title":"Install keras using conda","text":"Keras with the Tensorflow backend can be installed by running the following conda command conda install - c conda - forge keras tensorflow If you want a Intel CPU optimized version, install tensorflow-mkl conda install - c conda - forge keras tensorflow - mkl A GPU compatible version is also available conda install - c conda - forge keras tensorflow - gpu","tags":"DL","url":"redoules.github.io/dl/keras_install.html","loc":"redoules.github.io/dl/keras_install.html"},{"title":"Day 9 - Multiple Linear Regression","text":"Problem Here is a simple equation: $$Y=a+b_1\\cdot f_1++b_2\\cdot f_2+...++b_m\\cdot f_m$$ $$Y=a+\\sum_{i=1}^m b_i\\cdot f_i$$ for \\((m+1)\\) read constants \\((a,f_1, f_2, ..., f_m)\\) . We can say that the value of \\(Y\\) depends on \\(m\\) features. We study this equation for \\(n\\) different feature sets \\((f_1, f_2, ..., f_m)\\) and records each respective value of \\(Y\\) . If we have \\(q\\) new feature sets, and without accounting for bias and variance trade-offs,what is the value of \\(Y\\) for each of the sets? Python implementation import numpy as np m = 2 n = 7 x_1 = [ 0.18 , 0.89 ] y_1 = 109.85 x_2 = [ 1.0 , 0.26 ] y_2 = 155.72 x_3 = [ 0.92 , 0.11 ] y_3 = 137.66 x_4 = [ 0.07 , 0.37 ] y_4 = 76.17 x_5 = [ 0.85 , 0.16 ] y_5 = 139.75 x_6 = [ 0.99 , 0.41 ] y_6 = 162.6 x_7 = [ 0.87 , 0.47 ] y_7 = 151.77 q_1 = [ 0.49 , 0.18 ] q_2 = [ 0.57 , 0.83 ] q_3 = [ 0.56 , 0.64 ] q_4 = [ 0.76 , 0.18 ] With scikit learn X = np . array ([ x_1 , x_2 , x_3 , x_4 , x_5 , x_6 , x_7 ]) Y = np . array ([ y_1 , y_2 , y_3 , y_4 , y_5 , y_6 , y_7 ]) X_q = np . array ([ q_1 , q_2 , q_3 , q_4 ]) from sklearn import linear_model lm = linear_model . LinearRegression () lm . fit ( X , Y ) lm . predict ( X_q ) array ([ 105 . 21455835 , 142 . 67095131 , 132 . 93605469 , 129 . 70175405 ]) without scikit learn (but with numpy) from numpy.linalg import inv #center X_R = X - np . mean ( X , axis = 0 ) a = np . mean ( Y ) Y_R = Y - a #calculate b B = inv ( X_R . T @X_R ) @X_R.T@Y_R #predict X_new_R = X_q - np . mean ( X , axis = 0 ) Y_new_R = X_new_R @B Y_new = Y_new_R + a Y_new array ([ 105 . 21455835 , 142 . 67095131 , 132 . 93605469 , 129 . 70175405 ]) if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) { var align = \"center\", indent = \"0em\", linebreak = \"false\"; if (false) { align = (screen.width < 768) ? \"left\" : align; indent = (screen.width < 768) ? \"0em\" : indent; linebreak = (screen.width < 768) ? 'true' : linebreak; } var mathjaxscript = document.createElement('script'); mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#'; mathjaxscript.type = 'text/javascript'; mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML'; mathjaxscript[(window.opera ? \"innerHTML\" : \"text\")] = \"MathJax.Hub.Config({\" + \" config: ['MMLorHTML.js'],\" + \" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } },\" + \" jax: ['input/TeX','input/MathML','output/HTML-CSS'],\" + \" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js'],\" + \" displayAlign: '\"+ align +\"',\" + \" displayIndent: '\"+ indent +\"',\" + \" showMathMenu: true,\" + \" messageStyle: 'normal',\" + \" tex2jax: { \" + \" inlineMath: [ ['\\\\\\\\(','\\\\\\\\)'] ], \" + \" displayMath: [ ['$$','$$'] ],\" + \" processEscapes: true,\" + \" preview: 'TeX',\" + \" }, \" + \" 'HTML-CSS': { \" + \" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} },\" + \" linebreaks: { automatic: \"+ linebreak +\", width: '90% container' },\" + \" }, \" + \"}); \" + \"if ('default' !== 'default') {\" + \"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"}\"; (document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript); }","tags":"Blog","url":"redoules.github.io/blog/Statistics_10days-day9.html","loc":"redoules.github.io/blog/Statistics_10days-day9.html"},{"title":"Multiple Linear Regression","text":"If \\(Y\\) is linearly dependent only on \\(X\\) , then we can use the ordinary least square regression line, \\(\\hat{Y}=a+bX\\) . However, if \\(Y\\) shows linear dependency on \\(m\\) variables \\(X_1\\) , \\(X_2\\) , ..., \\(X_m\\) , then we need to find the values of \\(a\\) and \\(m\\) other constants ( \\(b_1\\) , \\(b_2\\) , ..., \\(b_m\\) ). We can then write the regression equation as: $$\\hat{Y}=a+\\sum_{i=1}^{m}b_iX_i$$ Matrix Form of the Regression Equation Let's consider that \\(Y\\) depends on two variables, \\(X_1\\) and \\(X_2\\) . We write the regression relation as \\(\\hat{Y}=a+b_1X_1+b_2X_2\\) . Consider the following matrix operation: $$\\begin{bmatrix} 1 & X_1 & X_2\\\\ \\end{bmatrix}\\cdot\\begin{bmatrix} a \\\\ b_1\\\\ b_2\\\\ \\end{bmatrix}=a+b_1X_1+b_2X_2$$ We define two matrices, \\(X\\) and \\(B\\) as: $$X=\\begin{bmatrix}1 & X_1 & X_2\\\\\\end{bmatrix}$$ $$B=\\begin{bmatrix}a \\\\b_1\\\\b_2\\\\\\end{bmatrix}$$ Now, we rewrite the regression relation as \\(\\hat{Y}=X\\cdot B\\) . This transforms the regression relation into matrix form. Generalized Matrix Form We will consider that \\(Y\\) shows a linear relationship with \\(m\\) variables, \\(X_1\\) , \\(X_2\\) , ..., \\(X_m\\) . Let's say that we made \\(n\\) observations on different tuples \\((x_1, x_2, ..., x_m)\\) : \\(y_1=a+b_1\\cdot x_{1,1} + b_2\\cdot x_{2,1} + ... + b_m\\cdot x_{m,1}\\) \\(y_2=a+b_2\\cdot x_{1,2} + b_2\\cdot x_{2,2} + ... + b_m\\cdot x_{m,2}\\) \\(...\\) \\(y_n=a+b_n\\cdot x_{1,n} + b_2\\cdot x_{2,n} + ... + b_m\\cdot x_{m,n}\\) Now, we can find the matrices: $$X=\\begin{bmatrix}1 & x_{1,1} & x_{2,1} & x_{3,1} & ... & x_{m,1} \\\\1 & x_{1,2} & x_{2,2} & x_{3,2} & ... & x_{m,2} \\\\1 & x_{1,3} & x_{2,3} & x_{3,3} & ... & x_{m,3} \\\\... & ... & ... & ... & ... & ... \\\\1 & x_{1,n} & x_{2,n} & x_{3,n} & ... & x_{m,n} \\\\\\end{bmatrix}$$ $$Y=\\begin{bmatrix}y_1 \\\\y_2\\\\y_3\\\\...\\\\y_n\\\\\\end{bmatrix}$$ Finding the Matrix B We know that \\(Y=X\\cdot B\\) $$\\Rightarrow X^T\\cdot Y=X^T\\cdot X \\cdot B$$ $$\\Rightarrow (X^T\\cdot X)^{-1}\\cdot X^T \\cdot Y=I\\cdot B$$ $$\\Rightarrow B= (X^T\\cdot X)^{-1}\\cdot X^T \\cdot Y$$ Finding the Value of Y Suppose we want to find the value of for some tuple \\(Y\\) , then \\((x_1, x_2, ..., x_m)\\) , $$Y=\\begin{bmatrix} 1 & x_1 & x_2 & ... & x_m\\\\ \\end{bmatrix}\\cdot B$$ Multiple Regression in Python We can use the fit function in the sklearn.linear_model.LinearRegression class. from sklearn import linear_model x = [[ 5 , 7 ], [ 6 , 6 ], [ 7 , 4 ], [ 8 , 5 ], [ 9 , 6 ]] y = [ 10 , 20 , 60 , 40 , 50 ] lm = linear_model . LinearRegression () lm . fit ( x , y ) a = lm . intercept_ b = lm . coef_ print ( f \"Linear regression coefficients between Y and X : a={a}, b_0={b[0]}, b_1={b[1]}\" ) Linear regression coefficients between Y and X : a = 51 . 953488372092984 , b_0 = 6 . 65116279069768 , b_1 =- 11 . 162790697674419 if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) { var align = \"center\", indent = \"0em\", linebreak = \"false\"; if (false) { align = (screen.width < 768) ? \"left\" : align; indent = (screen.width < 768) ? \"0em\" : indent; linebreak = (screen.width < 768) ? 'true' : linebreak; } var mathjaxscript = document.createElement('script'); mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#'; mathjaxscript.type = 'text/javascript'; mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML'; mathjaxscript[(window.opera ? \"innerHTML\" : \"text\")] = \"MathJax.Hub.Config({\" + \" config: ['MMLorHTML.js'],\" + \" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } },\" + \" jax: ['input/TeX','input/MathML','output/HTML-CSS'],\" + \" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js'],\" + \" displayAlign: '\"+ align +\"',\" + \" displayIndent: '\"+ indent +\"',\" + \" showMathMenu: true,\" + \" messageStyle: 'normal',\" + \" tex2jax: { \" + \" inlineMath: [ ['\\\\\\\\(','\\\\\\\\)'] ], \" + \" displayMath: [ ['$$','$$'] ],\" + \" processEscapes: true,\" + \" preview: 'TeX',\" + \" }, \" + \" 'HTML-CSS': { \" + \" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} },\" + \" linebreaks: { automatic: \"+ linebreak +\", width: '90% container' },\" + \" }, \" + \"}); \" + \"if ('default' !== 'default') {\" + \"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"}\"; (document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript); }","tags":"Machine Learning","url":"redoules.github.io/machine-learning/Multiple_Linear_Regression.html","loc":"redoules.github.io/machine-learning/Multiple_Linear_Regression.html"},{"title":"Least Square Regression Line","text":"Linear Regression If our data shows a linear relationship between \\(X\\) and \\(Y\\) , then the straight line which best describes the relationship is the regression line. The regression line is given by \\(\\hat{Y}\\) =a+bX$. Finding the value of b The value of \\(b\\) can be calculated using either of the following formulae: \\(b=\\frac{n\\sum(x_iy_i)-(\\sum x_i)(\\sum y_i)}{n\\sum(x_i^2)-(\\sum x_i)^2}\\) \\(b=\\rho\\frac{\\sigma_Y}{\\sigma_X}\\) , where \\(\\rho\\) is the Pearson correlation coefficient, \\(\\sigma_X\\) Finding the value of a \\(a=\\bar{y}-b\\cdot\\bar{x}\\) , where \\(\\bar{x}\\) is the mean of \\(X\\) and \\(\\bar{y}\\) is the mean of \\(Y\\) . Coefficient of determination ( \\(R^2\\) ) The coefficient of determination can be computer with : \\(R^2 = \\frac{SSR}{SST}=1-\\frac{SSE}{SST}\\) Where : \\(SST\\) is the total Sum of Squares : \\(SST=\\sum (y_i-\\bar{y})^2\\) \\(SSR\\) is the regression Sum of Squares : \\(SSR=\\sum (\\hat{y_i}-\\bar{y})^2\\) \\(SSE\\) is the error Sum of Squares : \\(SSE=\\sum (\\hat{y_i}-y)^2\\) If \\(SSE\\) is small, we can assume that our fit is good. Linear Regression in Python We can use the fit function in the sklearn.linear_model.LinearRegression class. from sklearn import linear_model import numpy as np xl = [ 1 , 2 , 3 , 4 , 5 ] x = np . asarray ( xl ) . reshape ( - 1 , 1 ) y = [ 2 , 1 , 4 , 3 , 5 ] lm = linear_model . LinearRegression () lm . fit ( x , y ) print ( f 'a = {lm.intercept_}' ) print ( f 'b = {lm.coef_[0]}' ) print ( \"Where Y=a+b*X\" ) a = 0 . 5999999999999996 b = 0 . 8000000000000002 Where Y = a + b * X if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) { var align = \"center\", indent = \"0em\", linebreak = \"false\"; if (false) { align = (screen.width < 768) ? \"left\" : align; indent = (screen.width < 768) ? \"0em\" : indent; linebreak = (screen.width < 768) ? 'true' : linebreak; } var mathjaxscript = document.createElement('script'); mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#'; mathjaxscript.type = 'text/javascript'; mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML'; mathjaxscript[(window.opera ? \"innerHTML\" : \"text\")] = \"MathJax.Hub.Config({\" + \" config: ['MMLorHTML.js'],\" + \" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } },\" + \" jax: ['input/TeX','input/MathML','output/HTML-CSS'],\" + \" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js'],\" + \" displayAlign: '\"+ align +\"',\" + \" displayIndent: '\"+ indent +\"',\" + \" showMathMenu: true,\" + \" messageStyle: 'normal',\" + \" tex2jax: { \" + \" inlineMath: [ ['\\\\\\\\(','\\\\\\\\)'] ], \" + \" displayMath: [ ['$$','$$'] ],\" + \" processEscapes: true,\" + \" preview: 'TeX',\" + \" }, \" + \" 'HTML-CSS': { \" + \" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} },\" + \" linebreaks: { automatic: \"+ linebreak +\", width: '90% container' },\" + \" }, \" + \"}); \" + \"if ('default' !== 'default') {\" + \"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"}\"; (document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript); }","tags":"Machine Learning","url":"redoules.github.io/machine-learning/LeastSquareRegressionLine.html","loc":"redoules.github.io/machine-learning/LeastSquareRegressionLine.html"},{"title":"Day 8 - Least Square Regression Line","text":"Least Square Regression Line Problem A group of five students enrolls in Statistics immediately after taking a Math aptitude test. Each student's Math aptitude test score, \\(x\\) , and Statistics course grade, \\(y\\) , can be expressed as the following list \\((x,y)\\) of points: \\((95, 85)\\) \\((85, 95)\\) \\((80, 70)\\) \\((70, 65)\\) \\((60, 70)\\) If a student scored an 80 on the Math aptitude test, what grade would we expect them to achieve in Statistics? Determine the equation of the best-fit line using the least squares method, then compute and print the value of \\(y\\) when \\(x=80\\) . X = [ 95 , 85 , 80 , 70 , 60 ] Y = [ 85 , 95 , 70 , 65 , 70 ] n = len ( X ) def cov ( X , Y , n ): x_mean = 1 / n * sum ( X ) y_mean = 1 / n * sum ( Y ) return 1 / n * sum ([( X [ i ] - x_mean ) * ( Y [ i ] - y_mean ) for i in range ( n )]) def stdv ( X , mu_x , n ): return ( sum ([( x - mu_x ) ** 2 for x in X ]) / n ) ** 0.5 def pearson_1 ( X , Y , n ): std_x = stdv ( X , 1 / n * sum ( X ), n ) std_y = stdv ( Y , 1 / n * sum ( Y ), n ) return cov ( X , Y , n ) / ( std_x * std_y ) b = pearson_1 ( X , Y , n ) * stdv ( Y , sum ( Y ) / n , n ) / stdv ( X , sum ( X ) / n , n ) a = sum ( Y ) / n - b * sum ( X ) / n print ( f \"If a student scored 80 on the math test, he would most likely score a {round(a+80*b,3)} in statistics\" ) If a student scored 80 on the math test , he would most likely score a 78 . 288 in statistics Pearson correlation coefficient Problem The regression line of \\(y\\) on \\(x\\) is \\(3x+4y+8=0\\) , and the regression line of \\(x\\) on \\(y\\) is \\(4x+3y+7=0\\) . What is the value of the Pearson correlation coefficient? Mathematical explanation The initial equation system is : $$ \\left\\{\\begin{array}{ r @{{}={}} r >{{}}c<{{}} r >{{}}c<{{}} r } 3x+4y+8=0 & (1)\\\\ 4x+3y+7=0 & (2)\\\\ \\end{array} \\right. $$ So we can rewrite the 2 lines this way : $$ \\left\\{\\begin{array}{ r @{{}={}} r >{{}}c<{{}} r >{{}}c<{{}} r } y=-2+(\\frac{-3}{4})x & (1)\\\\ x=-\\frac{7}{4}+(-\\frac{3}{4})y & (2)\\\\ \\end{array} \\right. $$ so \\(b_1=-\\frac{3}{4}\\) and \\(b_2=-\\frac{3}{4}\\) When we apply the Pearson's coefficient formula : let \\(p\\) be the pearson coefficient let \\(\\sigma_X\\) be the standard deviation of \\(x\\) let \\(\\sigma_Y\\) be the standard deviation of \\(y\\) We hence have $$ \\left\\{\\begin{array}{ r @{{}={}} r >{{}}c<{{}} r >{{}}c<{{}} r } p=b_1\\left(\\frac{\\sigma_X}{\\sigma_Y}\\right) & (1)\\\\ p=b_2\\left(\\frac{\\sigma_Y}{\\sigma_X}\\right) & (2)\\\\ \\end{array} \\right. $$ by multiplying theses 2 equations together we get $$p^2=b_1\\cdot b_2$$ $$p^2=\\left(-\\frac{3}{4}\\right)\\left(-\\frac{3}{4}\\right)$$ $$p^2=\\left(-\\frac{9}{16}\\right)$$ finally we get \\(p=\\left(-\\frac{3}{4}\\right)\\) or \\(p=\\left(\\frac{3}{4}\\right)\\) Since \\(X\\) and \\(Y\\) are negatively correlated we have \\(p=\\left(-\\frac{3}{4}\\right)\\) if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) { var align = \"center\", indent = \"0em\", linebreak = \"false\"; if (false) { align = (screen.width < 768) ? \"left\" : align; indent = (screen.width < 768) ? \"0em\" : indent; linebreak = (screen.width < 768) ? 'true' : linebreak; } var mathjaxscript = document.createElement('script'); mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#'; mathjaxscript.type = 'text/javascript'; mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML'; mathjaxscript[(window.opera ? \"innerHTML\" : \"text\")] = \"MathJax.Hub.Config({\" + \" config: ['MMLorHTML.js'],\" + \" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } },\" + \" jax: ['input/TeX','input/MathML','output/HTML-CSS'],\" + \" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js'],\" + \" displayAlign: '\"+ align +\"',\" + \" displayIndent: '\"+ indent +\"',\" + \" showMathMenu: true,\" + \" messageStyle: 'normal',\" + \" tex2jax: { \" + \" inlineMath: [ ['\\\\\\\\(','\\\\\\\\)'] ], \" + \" displayMath: [ ['$$','$$'] ],\" + \" processEscapes: true,\" + \" preview: 'TeX',\" + \" }, \" + \" 'HTML-CSS': { \" + \" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} },\" + \" linebreaks: { automatic: \"+ linebreak +\", width: '90% container' },\" + \" }, \" + \"}); \" + \"if ('default' !== 'default') {\" + \"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"}\"; (document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript); }","tags":"Blog","url":"redoules.github.io/blog/Statistics_10days-day8.html","loc":"redoules.github.io/blog/Statistics_10days-day8.html"},{"title":"Spearman's Rank Correlation Coefficient","text":"A rank correlation is any of several statistics that measure an ordinal association—the relationship between rankings of different ordinal variables or different rankings of the same variable, where a \"ranking\" is the assignment of the ordering labels \"first\", \"second\", \"third\", etc. to different observations of a particular variable. A rank correlation coefficient measures the degree of similarity between two rankings, and can be used to assess the significance of the relation between them. We have two random variables \\(X\\) and \\(Y\\) : * \\(X=\\{x_i, x_2, x_3, ..., x_n\\}\\) * \\(Y=\\{y_i, y_2, y_3, ..., y_n\\}\\) if \\(Rank_X\\) and \\(Rank_Y\\) denote the respective ranks of each data point, then the Spearman's rank correlation coefficient, \\(r_s\\) , is the Pearson correlation coefficient of \\(Rank_X\\) and \\(Rank_Y\\) . What does it means? The Spearman's rank correlation coefficientis is a nonparametric measure of rank correlation (statistical dependence between the rankings of two variables). It assesses how well the relationship between two variables can be described using a monotonic function. The Spearman correlation between two variables is equal to the Pearson correlation between the rank values of those two variables; while Pearson's correlation assesses linear relationships, Spearman's correlation assesses monotonic relationships (whether linear or not). If there are no repeated data values, a perfect Spearman correlation of +1 or −1 occurs when each of the variables is a perfect monotone function of the other. A Spearman correlation of 1 results when the two variables being compared are monotonically related, even if their relationship is not linear. This means that all data-points with greater x-values than that of a given data-point will have greater y-values as well. In contrast, this does not give a perfect Pearson correlation. Example \\(X=\\{0.2, 1.3, 0.2, 1.1, 1.4, 1.5\\}\\) \\(Y=\\{1.9, 2.2, 3.1, 1.2, 2.2, 2.2\\}\\) $$ Rank_X \\quad \\begin{bmatrix} X: & 0.2 & 1.3 & 0.2 & 1.1 & 1.4 & 1.5 \\\\ Rank: & 1 & 3 & 1 & 2 & 4 & 5 \\end{bmatrix} \\quad $$ so, \\(Rank_X = \\{1, 3, 1, 2, 4, 5\\}\\) similarly, \\(Rank_Y=\\{2,3,4,1,3,3\\}\\) \\(r_s\\) equals the Pearson correlation coefficient of \\(Rank_X\\) and \\(Rank_Y\\) , meaning that \\(r=0.158114\\) Special case : \\(X\\) and \\(Y\\) don't contain duplicates $$r_s=1-\\frac{6\\sum d_i^2}{n(n^2-1)}$$ Where, \\(d_i\\) is the difference between the respective values of \\(Rank_X\\) and \\(Rank_Y\\) . if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) { var align = \"center\", indent = \"0em\", linebreak = \"false\"; if (false) { align = (screen.width < 768) ? \"left\" : align; indent = (screen.width < 768) ? \"0em\" : indent; linebreak = (screen.width < 768) ? 'true' : linebreak; } var mathjaxscript = document.createElement('script'); mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#'; mathjaxscript.type = 'text/javascript'; mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML'; mathjaxscript[(window.opera ? \"innerHTML\" : \"text\")] = \"MathJax.Hub.Config({\" + \" config: ['MMLorHTML.js'],\" + \" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } },\" + \" jax: ['input/TeX','input/MathML','output/HTML-CSS'],\" + \" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js'],\" + \" displayAlign: '\"+ align +\"',\" + \" displayIndent: '\"+ indent +\"',\" + \" showMathMenu: true,\" + \" messageStyle: 'normal',\" + \" tex2jax: { \" + \" inlineMath: [ ['\\\\\\\\(','\\\\\\\\)'] ], \" + \" displayMath: [ ['$$','$$'] ],\" + \" processEscapes: true,\" + \" preview: 'TeX',\" + \" }, \" + \" 'HTML-CSS': { \" + \" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} },\" + \" linebreaks: { automatic: \"+ linebreak +\", width: '90% container' },\" + \" }, \" + \"}); \" + \"if ('default' !== 'default') {\" + \"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"}\"; (document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript); }","tags":"Mathematics","url":"redoules.github.io/mathematics/spearman.html","loc":"redoules.github.io/mathematics/spearman.html"},{"title":"Pearson correlation coefficient","text":"Covariance This is a measure of how two random variables change together, or the strength of their correlation. Consider two random variables, \\(X\\) and \\(Y\\) , each with \\(n\\) values (i.e., \\(x_1\\) , \\(x_2\\) , \\(...\\) , \\(x_n\\) and \\(y_1\\) , \\(y_2\\) , \\(...\\) , \\(y_n\\) ). The covariance of \\(X\\) and \\(Y\\) can be found using either of the following equivalent formulas: $$cov(X,Y)=\\frac{1}{n}\\sum_{i=1}^{n}(x_i-\\bar{x})\\cdot(y_i-\\bar{y})$$ or $$cov(X,Y)=\\frac{1}{n^2}\\sum_{i=1}^{n}\\sum_{j=1}^{n}\\frac{1}{2}(x_i-x_j)\\cdot(y_i-y_j))$$ $$cov(X,Y)=\\frac{1}{n^2}\\sum_{i}\\sum_{j\\gt i}^{n}(x_i-x_j)\\cdot(y_i-y_j)$$ where, \\(\\bar{x}\\) is the mean of \\(X\\) (or \\(\\mu_X\\) ) and \\(\\bar{y}\\) is the mean of \\(Y\\) (or \\(\\mu_Y\\) ) Pearson correlation coefficient The pearson correlation coefficient, \\(\\rho_{X,Y}\\) , is given by : $$\\rho_{X,Y}=\\frac{cov(X,Y)}{\\sigma_X\\sigma_Y}=\\frac{\\sum_{i}(x_i-\\bar{x})(y_i-\\bar{y})}{n\\sigma_X\\sigma_Y}$$ Here, \\(\\sigma_X\\) is the standard deviation of \\(X\\) and \\(\\sigma_Y\\) is the standard deviation of \\(Y\\) . You may also see \\(\\rho_{X,Y}\\) written as \\(r_{X,Y}\\) . The pearson correlation coefficient is a measure of the linear correlation between two variables X and Y. if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) { var align = \"center\", indent = \"0em\", linebreak = \"false\"; if (false) { align = (screen.width < 768) ? \"left\" : align; indent = (screen.width < 768) ? \"0em\" : indent; linebreak = (screen.width < 768) ? 'true' : linebreak; } var mathjaxscript = document.createElement('script'); mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#'; mathjaxscript.type = 'text/javascript'; mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML'; mathjaxscript[(window.opera ? \"innerHTML\" : \"text\")] = \"MathJax.Hub.Config({\" + \" config: ['MMLorHTML.js'],\" + \" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } },\" + \" jax: ['input/TeX','input/MathML','output/HTML-CSS'],\" + \" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js'],\" + \" displayAlign: '\"+ align +\"',\" + \" displayIndent: '\"+ indent +\"',\" + \" showMathMenu: true,\" + \" messageStyle: 'normal',\" + \" tex2jax: { \" + \" inlineMath: [ ['\\\\\\\\(','\\\\\\\\)'] ], \" + \" displayMath: [ ['$$','$$'] ],\" + \" processEscapes: true,\" + \" preview: 'TeX',\" + \" }, \" + \" 'HTML-CSS': { \" + \" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} },\" + \" linebreaks: { automatic: \"+ linebreak +\", width: '90% container' },\" + \" }, \" + \"}); \" + \"if ('default' !== 'default') {\" + \"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"}\"; (document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript); }","tags":"Mathematics","url":"redoules.github.io/mathematics/pearson.html","loc":"redoules.github.io/mathematics/pearson.html"},{"title":"Day 7 - Pearson and spearman correlations","text":"Pearson correlation coefficient Problem Given two n-element data sets, \\(X\\) and \\(Y\\) , calculate the value of the Pearson correlation coefficient. Python implementation Using the formula $$\\rho_{X,Y}=\\frac{cov(X,Y)}{\\sigma_X\\sigma_Y}$$ where $$cov(X,Y)=\\frac{1}{n}\\sum_{i=1}^{n}(x_i-\\bar{x})\\cdot(y_i-\\bar{y})$$ n = 10 X = [ 10 , 9.8 , 8 , 7.8 , 7.7 , 7 , 6 , 5 , 4 , 2 ] Y = [ 200 , 44 , 32 , 24 , 22 , 17 , 15 , 12 , 8 , 4 ] def cov ( X , Y , n ): x_mean = 1 / n * sum ( X ) y_mean = 1 / n * sum ( Y ) return 1 / n * sum ([( X [ i ] - x_mean ) * ( Y [ i ] - y_mean ) for i in range ( n )]) def stdv ( X , mu_x , n ): return ( sum ([( x - mu_x ) ** 2 for x in X ]) / n ) ** 0.5 def pearson_1 ( X , Y , n ): std_x = stdv ( X , 1 / n * sum ( X ), n ) std_y = stdv ( Y , 1 / n * sum ( Y ), n ) return cov ( X , Y , n ) / ( std_x * std_y ) pearson_1 ( X , Y , n ) 0 . 6124721937208479 Python implementation Using the formula $$\\rho_{X,Y}=\\frac{\\sum_{i}(x_i-\\bar{x})(y_i-\\bar{y})}{n\\sigma_X\\sigma_Y}$$ def pearson_2 ( X , Y , n ): std_x = stdv ( X , 1 / n * sum ( X ), n ) std_y = stdv ( Y , 1 / n * sum ( Y ), n ) x_mean = 1 / n * sum ( X ) y_mean = 1 / n * sum ( Y ) return sum ([( X [ i ] - x_mean ) * ( Y [ i ] - y_mean ) for i in range ( n )]) / ( n * std_x * std_y ) pearson_2 ( X , Y , n ) 0 . 6124721937208479 Spearman's rank correlation coefficient Problem Given two \\(n\\) -element data sets, \\(X\\) and \\(Y\\) , calculate the value of Spearman's rank correlation coefficient. Python implementation We knwo that in this case, the values in each dataset are unique. Hence we can use the formula : $$r_s=1-\\frac{6\\sum d_i^2}{n(n^2-1)}$$ n = 10 X = [ 10 , 9.8 , 8 , 7.8 , 7.7 , 1.7 , 6 , 5 , 1.4 , 2 ] Y = [ 200 , 44 , 32 , 24 , 22 , 17 , 15 , 12 , 8 , 4 ] def spearman_rank ( X , Y , n ): rank_X = [ sorted ( X ) . index ( v ) + 1 for v in X ] rank_Y = [ sorted ( Y ) . index ( v ) + 1 for v in Y ] d = [( rank_X [ i ] - rank_Y [ i ]) ** 2 for i in range ( n )] return 1 - ( 6 * sum ( d )) / ( n * ( n * n - 1 )) spearman_rank ( X , Y , n ) 0 . 9030303030303031 if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) { var align = \"center\", indent = \"0em\", linebreak = \"false\"; if (false) { align = (screen.width < 768) ? \"left\" : align; indent = (screen.width < 768) ? \"0em\" : indent; linebreak = (screen.width < 768) ? 'true' : linebreak; } var mathjaxscript = document.createElement('script'); mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#'; mathjaxscript.type = 'text/javascript'; mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML'; mathjaxscript[(window.opera ? \"innerHTML\" : \"text\")] = \"MathJax.Hub.Config({\" + \" config: ['MMLorHTML.js'],\" + \" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } },\" + \" jax: ['input/TeX','input/MathML','output/HTML-CSS'],\" + \" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js'],\" + \" displayAlign: '\"+ align +\"',\" + \" displayIndent: '\"+ indent +\"',\" + \" showMathMenu: true,\" + \" messageStyle: 'normal',\" + \" tex2jax: { \" + \" inlineMath: [ ['\\\\\\\\(','\\\\\\\\)'] ], \" + \" displayMath: [ ['$$','$$'] ],\" + \" processEscapes: true,\" + \" preview: 'TeX',\" + \" }, \" + \" 'HTML-CSS': { \" + \" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} },\" + \" linebreaks: { automatic: \"+ linebreak +\", width: '90% container' },\" + \" }, \" + \"}); \" + \"if ('default' !== 'default') {\" + \"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"}\"; (document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript); }","tags":"Blog","url":"redoules.github.io/blog/Statistics_10days-day7.html","loc":"redoules.github.io/blog/Statistics_10days-day7.html"},{"title":"Day 6 - The Central Limit Theorem","text":"Problem 1 A large elevator can transport a maximum of \\(9800\\) kg. Suppose a load of cargo containing \\(49\\) boxes must be transported via the elevator. The box weight of this type of cargo follows a distribution with a mean of \\(\\mu=205\\) kg and a standard deviation of \\(\\sigma=15\\) kg. Based on this information, what is the probability that all boxes can be safely loaded into the freight elevator and transported? Mathematical explanation This problem can be tackled with the central limit theorem. Since the number of boxes is \"large\", the sum of the weight approaches normal distribution with : * \\(\\mu' = n\\mu\\) * \\(\\sigma'=\\sigma\\sqrt{n}\\) If we want to know the probability of the sum of the mass of all boxes to be under a certain weight, we can compute the cumulative density function : $$P(x<9800) = F_X(9800)$$ max_load = 9800 n = 49 mu = 205 st_dev = 15 import math def cumulative ( x , mean , sd ): return 0.5 * ( 1 + math . erf (( x - mean ) / ( sd * math . sqrt ( 2 )))) mu_group = n * mu st_dev_group = st_dev * math . sqrt ( n ) print ( f \"Probability that all the boxes can be lifted by the elevator : {cumulative(max_load, mu_group, st_dev_group)}\" ) Probability that all the boxes can be lifted by the elevator : 0 . 009815328628645315 Problem 2 The number of tickets purchased by each student for the University X vs. University Y football game follows a distribution that has a mean of \\(\\mu=2.4\\) and a standard deviation of \\(\\sigma=2.0\\) . A few hours before the game starts, \\(100\\) eager students line up to purchase last-minute tickets. If there are only \\(250\\) tickets left, what is the probability that all students will be able to purchase tickets? Mathematical explanation We want to know if the sum of all the purchases will exceed the total supply of tickets. Since each student buy follows a Normal distribution and that the number of students is relatively high, the probability that all the students will be able to buy a ticket can be computed by applying the central limit theorem. The total number of tickets bought follows a normal distribution of mean \\(\\mu'=n*\\mu\\) and of standard deviation \\(\\sigma'=\\sigma\\sqrt{n}\\) ticket_supply = 250 n_students = 100 mu = 2.4 st_dev = 2 mu_group = n_students * mu st_dev_group = st_dev * math . sqrt ( n_students ) print ( f \"Probability that all the students can purchase tickets : {cumulative(ticket_supply, mu_group, st_dev_group)}\" ) Probability that all the students can purchase tickets : 0 . 691462461274013 Problem 3 You have a sample of \\(100\\) values from a population with mean \\(\\mu=500\\) and with standard deviation \\(\\sigma=80\\) . Compute the interval that covers the middle \\(95%\\) of the distribution of the sample mean; in other words, compute \\(A\\) and \\(B\\) such that \\(P(A .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } B G B BB BG G GB GG We know that at least one of the children is a boy, so only \"GG\" is not possible. The event where the family has a new boy is then \"BB\". Hence the probability is : $$\\frac{BB}{BB+GB+BG}=\\frac{1}{3}$$ Draw 2 cards from a deck Problem You 2 draw cards from a standard 52-card deck without replacing them. What is the probability that both cards are of the same suit? Mathematical explanation There are 13 cards of each suit. Draw one card. It can be anything with probability of 1. Now there are 51 cards left and 12 of them are the same suit as the first card you drew. So the chance the second card matches the 1st is \\(\\frac{12}{51}\\) . Drawing marbles Problem A bag contains 3 red marbles and 4 blue marbles. Then, 2 marbles are drawn from the bag, at random, without replacement. If the first marble drawn is red, what is the probability that the second marble is blue? Mathematical explanation On the first draw, the probabilities are the following : we call B the event \"a blue ball is drawn\" and R the event \"a red ball is drawn\" * \\(P(B)=\\frac{4}{7}\\) * \\(P(R)=\\frac{3}{7}\\) On the second draw, if a red ball has been drawn at first, the probabilities are : * \\(P(B|R)=\\frac{4}{6}\\) * \\(P(R|R)=\\frac{2}{6}\\) Hence, the probability of drawing a blue ball if the first ball drawn was red is \\(\\frac{1}{3}\\) if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) { var align = \"center\", indent = \"0em\", linebreak = \"false\"; if (false) { align = (screen.width < 768) ? \"left\" : align; indent = (screen.width < 768) ? \"0em\" : indent; linebreak = (screen.width < 768) ? 'true' : linebreak; } var mathjaxscript = document.createElement('script'); mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#'; mathjaxscript.type = 'text/javascript'; mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML'; mathjaxscript[(window.opera ? \"innerHTML\" : \"text\")] = \"MathJax.Hub.Config({\" + \" config: ['MMLorHTML.js'],\" + \" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } },\" + \" jax: ['input/TeX','input/MathML','output/HTML-CSS'],\" + \" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js'],\" + \" displayAlign: '\"+ align +\"',\" + \" displayIndent: '\"+ indent +\"',\" + \" showMathMenu: true,\" + \" messageStyle: 'normal',\" + \" tex2jax: { \" + \" inlineMath: [ ['\\\\\\\\(','\\\\\\\\)'] ], \" + \" displayMath: [ ['$$','$$'] ],\" + \" processEscapes: true,\" + \" preview: 'TeX',\" + \" }, \" + \" 'HTML-CSS': { \" + \" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} },\" + \" linebreaks: { automatic: \"+ linebreak +\", width: '90% container' },\" + \" }, \" + \"}); \" + \"if ('default' !== 'default') {\" + \"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"}\"; (document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript); }","tags":"Blog","url":"redoules.github.io/blog/Statistics_10days-day3.html","loc":"redoules.github.io/blog/Statistics_10days-day3.html"},{"title":"Day 2 - Probability, Compound Event Probability","text":"Basic probability with dices Problem In this challenge, we practice calculating probability. In a single toss of 2 fair (evenly-weighted) six-sided dice, find the probability that their sum will be at most 9. Mathematical explanation A nice way to think about sums-of-two-dice problems is to lay out the sums in a 6-by-6 grid in the obvious manner. .dataframe tbody tr th:only-of-type { vertical-align: middle; }
.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } 1 2 3 4 5 6 1 2 3 4 5 6 7 2 3 4 5 6 7 8 3 4 5 6 7 8 9 4 5 6 7 8 9 10 5 6 7 8 9 10 11 6 7 8 9 10 11 12 We see that the identic values are on the same diagonal. The number of elements on the diagonal varies from 1 to 6 and then back to 1. let's call A < x the event : the sum all the 2 tosses is at most x. $$P(A\\leq9)=\\sum_{i=2}^{9} P(A = i)$$ $$P(A\\leq9)=1-P(A\\gt9)$$ $$P(A\\leq9)=1-\\sum_{i=10}^{12} P(A = i)$$ The value of \\(P(A = i) = \\frac{i-1}{36}\\) if \\(i \\leq 7\\) and \\(P(A = i) = \\frac{13-i}{36}\\) hence $$P(A\\leq9)=1-\\sum_{i=10}^{12} \\frac{13-i}{36}$$ $$P(A\\leq9)= 1-\\frac{6}{36}$$ $$P(A\\leq9)= \\frac{5}{6}$$ Let's program it sum ([ 1 for d1 in range ( 1 , 7 ) for d2 in range ( 1 , 7 ) if d1 + d2 <= 9 ]) / 36 0 . 8333333333333334 More dices Problem In a single toss of 2 fair (evenly-weighted) six-sided dice, find the probability that the values rolled by each die will be different and the two dice have a sum of 6. Mathematical explanation Let's consider 2 events : A and B. A compound event is a combination of 2 or more simple events. If A and B are simple events, then A∪B denotes the occurence of either A or B. A∩B denotes the occurence of A and B together. We denote A the event \"the values of each dice is different\". The opposit event is A' \"the values of each dice is the same\". $$P(A) = 1-P(A')$$ $$P(A)=1-\\frac{6}{36}$$ $$P(A)=\\frac{5}{6}$$ We denote B the event \"the two dice have a sum of 6\", this probability has been computed on the first part of the article : $$P(B)=\\frac{5}{36}$$ The probability of having 2 dice different of sum 6 is : $$P(A|B) = 4/5$$ The probability that both A and B occure is equal to P(A∩B). Since \\(P(A|B)=\\frac{P(A∩B)}{P(B)}\\) $$P(A∩B)=P(B)*P(A|B)$$ $$P(A∩B)=5/36*4/5$$ $$P(A∩B)=1/9$$ Let's program it sum ([ 1 for d1 in range ( 1 , 7 ) for d2 in range ( 1 , 7 ) if ( d1 + d2 == 6 ) and ( d1 != d2 )]) / 36 0 . 1111111111111111 Compound Event Probability Problem There are 3 urns labeled X, Y, and Z. Urn X contains 4 red balls and 3 black balls. Urn Y contains 5 red balls and 4 black balls. Urn Z contains 4 red balls and 4 black balls. One ball is drawn from each of the urns. What is the probability that, of the 3 balls drawn, are 2 red and is 1 black? Mathematical explanation Let's write the different probabilities: .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Red ball Black ball Urne X $$\\frac{4}{7}$$ $$\\frac{3}{7}$$ Urne Y $$\\frac{5}{9}$$ $$\\frac{4}{9}$$ Urne Z $$\\frac{1}{2}$$ $$\\frac{1}{2}$$ Addition rule A and B are said to be mutually exclusive or disjoint if they have no events in common (i.e., and A∩B=∅ and P(A∩B)=0. The probability of any of 2 or more events occurring is the union (∪) of events. Because disjoint probabilities have no common events, the probability of the union of disjoint events is the sum of the events' individual probabilities. A and B are said to be collectively exhaustive if their union covers all events in the sample space (i.e., A∪B=S and P(A∪B)=1). This brings us to our next fundamental rule of probability: if 2 events, A and B, are disjoint, then the probability of either event is the sum of the probabilities of the 2 events (i.e., P(A or B) = P(A)+P(B)) Mutliplication rule If the outcome of the first event (A) has no impact on the second event (B), then they are considered to be independent (e.g., tossing a fair coin). This brings us to the next fundamental rule of probability: the multiplication rule. It states that if two events, A and B, are independent, then the probability of both events is the product of the probabilities for each event (i.e., P(A and B)= P(A)xP(B)). The chance of all events occurring in a sequence of events is called the intersection (∩) of those events. The balls drawn from the urns are independant hence : p = P(2 red (R) and 1 back (B)) $$p = P(RRB) + P(RBR) + P(BRR)$$ Each of those 3 probability if equal to the product of the probability of drawing each ball \\(P(RRB) = P(R|X) * P(R|Y) * P(B|Z) = 4/7*5/9*1/2\\) \\(P(RRB) = 20/126\\) \\(P(RBR) = 16/126\\) \\(P(BRR) = 15/126\\) this leads to \\(p = 51/126\\) and finally $$p = \\frac{17}{42}$$ Let's program it X = 3 * [ \"B\" ] + 4 * [ \"R\" ] Y = 4 * [ \"B\" ] + 5 * [ \"R\" ] Z = 4 * [ \"B\" ] + 4 * [ \"R\" ] target = [ \"BRR\" , \"RRB\" , \"RBR\" ] sum ([ 1 for x in X for y in Y for z in Z if x + y + z in target ]) / sum ([ 1 for x in X for y in Y for z in Z ]) 0 . 40476190476190477 if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) { var align = \"center\", indent = \"0em\", linebreak = \"false\"; if (false) { align = (screen.width < 768) ? \"left\" : align; indent = (screen.width < 768) ? \"0em\" : indent; linebreak = (screen.width < 768) ? 'true' : linebreak; } var mathjaxscript = document.createElement('script'); mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#'; mathjaxscript.type = 'text/javascript'; mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML'; mathjaxscript[(window.opera ? \"innerHTML\" : \"text\")] = \"MathJax.Hub.Config({\" + \" config: ['MMLorHTML.js'],\" + \" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } },\" + \" jax: ['input/TeX','input/MathML','output/HTML-CSS'],\" + \" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js'],\" + \" displayAlign: '\"+ align +\"',\" + \" displayIndent: '\"+ indent +\"',\" + \" showMathMenu: true,\" + \" messageStyle: 'normal',\" + \" tex2jax: { \" + \" inlineMath: [ ['\\\\\\\\(','\\\\\\\\)'] ], \" + \" displayMath: [ ['$$','$$'] ],\" + \" processEscapes: true,\" + \" preview: 'TeX',\" + \" }, \" + \" 'HTML-CSS': { \" + \" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} },\" + \" linebreaks: { automatic: \"+ linebreak +\", width: '90% container' },\" + \" }, \" + \"}); \" + \"if ('default' !== 'default') {\" + \"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"}\"; (document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript); }","tags":"Blog","url":"redoules.github.io/blog/Statistics_10days-day2.html","loc":"redoules.github.io/blog/Statistics_10days-day2.html"},{"title":"Day 1 - Quartiles, Interquartile Range and standard deviation","text":"Quartile Definition A quartile is a type of quantile. The first quartile (Q1) is defined as the middle number between the smallest number and the median of the data set. The second quartile (Q2) is the median of the data. The third quartile (Q3) is the middle value between the median and the highest value of the data set. Implementation in python without using the scientific libraries def median ( l ): l = sorted ( l ) if len ( l ) % 2 == 0 : return ( l [ len ( l ) // 2 ] + l [( len ( l ) // 2 - 1 )]) / 2 else : return l [ len ( l ) // 2 ] def quartiles ( l ): # check the input is not empty if not l : raise StatsError ( 'no data points passed' ) # 1. order the data set l = sorted ( l ) # 2. divide the data set in two halves mid = int ( len ( l ) / 2 ) Q2 = median ( l ) if ( len ( l ) % 2 == 0 ): # even Q1 = median ( l [: mid ]) Q3 = median ( l [ mid :]) else : # odd Q1 = median ( l [: mid ]) # same as even Q3 = median ( l [ mid + 1 :]) return ( Q1 , Q2 , Q3 ) L = [ 3 , 7 , 8 , 5 , 12 , 14 , 21 , 13 , 18 ] Q1 , Q2 , Q3 = quartiles ( L ) print ( f \"Sample : {L} \\n Q1 : {Q1}, Q2 : {Q2}, Q3 : {Q3}\" ) Sample : [ 3 , 7 , 8 , 5 , 12 , 14 , 21 , 13 , 18 ] Q1 : 6 . 0 , Q2 : 12 , Q3 : 16 . 0 Interquartile Range Definition The interquartile range of an array is the difference between its first (Q1) and third (Q3) quartiles. Hence the interquartile range is Q3-Q1 Implementation in python without using the scientific libraries print ( f \"Interquatile range : {Q3-Q1}\" ) Interquatile range : 10 . 0 Standard deviation Definition The standard deviation (σ) is a measure that is used to quantify the amount of variation or dispersion of a set of data values. A low standard deviation indicates that the data points tend to be close to the mean (also called the expected value) of the set, while a high standard deviation indicates that the data points are spread out over a wider range of values. The standard deviation can be computed with the formula: where µ is the mean : Implementation in python without using the scientific libraries import math X = [ 10 , 40 , 30 , 50 , 20 ] mean = sum ( X ) / len ( X ) X = [( x - mean ) ** 2 for x in X ] std = math . sqrt ( sum ( X ) / len ( X ) ) print ( f \"The distribution {X} has a standard deviation of {std}\" ) The distribution [ 400 . 0 , 100 . 0 , 0 . 0 , 400 . 0 , 100 . 0 ] has a standard deviation of 14 . 142135623730951","tags":"Blog","url":"redoules.github.io/blog/Statistics_10days-day1.html","loc":"redoules.github.io/blog/Statistics_10days-day1.html"},{"title":"Counting values in an array","text":"Using lists If you want to count the number of occurences of an element in a list you can use the .count() function of the list object arr = [ 1 , 2 , 3 , 3 , 4 , 5 , 3 , 6 , 7 , 7 ] print ( f 'Array : {arr} \\n ' ) print ( f 'The number 3 appears {arr.count(3)} times in the list' ) print ( f 'The number 7 appears {arr.count(7)} times in the list' ) print ( f 'The number 4 appears {arr.count(4)} times in the list' ) Array : [ 1 , 2 , 3 , 3 , 4 , 5 , 3 , 6 , 7 , 7 ] The number 3 appears 3 times in the list The number 7 appears 2 times in the list The number 4 appears 1 times in the list Using collections you can get a dictonnary of the number of occurences of each elements in a list thanks to the collections object like this import collections collections . Counter ( arr ) Counter ( { 1 : 1 , 2 : 1 , 3 : 3 , 4 : 1 , 5 : 1 , 6 : 1 , 7 : 2 } ) Using numpy You can have a simular result with numpy by hacking the unique function import numpy as np arr = np . array ( arr ) unique , counts = np . unique ( arr , return_counts = True ) dict ( zip ( unique , counts )) { 1 : 1 , 2 : 1 , 3 : 3 , 4 : 1 , 5 : 1 , 6 : 1 , 7 : 2 }","tags":"Python","url":"redoules.github.io/python/counting.html","loc":"redoules.github.io/python/counting.html"},{"title":"Building a dictonnary using comprehension","text":"An easy way to create a dictionnary in python is to use the comprehension syntaxe. It can be more expressive hence easier to read. d = { key : value for ( key , value ) in iterable } In the example bellow we use the dictionnary comprehension to build a dictonnary from a source list. iterable = list ( range ( 10 )) d = { str ( value ): value ** 2 for value in iterable } # create a dictionnary linking the string value of a number with the square value of this number print ( d ) { '0' : 0 , '1' : 1 , '2' : 4 , '3' : 9 , '4' : 16 , '5' : 25 , '6' : 36 , '7' : 49 , '8' : 64 , '9' : 81 } of course, you can use an other iterable an repack it with the comprehension syntaxe. In the following example, we convert a list of tuples in a dictonnary. iterable = [( \"France\" , 67.12e6 ), ( \"UK\" , 66.02e6 ), ( \"USA\" , 325.7e6 ), ( \"China\" , 1386e6 ), ( \"Germany\" , 82.79e6 )] population = { key : value for ( key , value ) in iterable } print ( population ) { 'France' : 67120000 . 0 , 'UK' : 66020000 . 0 , 'USA' : 325700000 . 0 , 'China' : 1386000000 . 0 , 'Germany' : 82790000 . 0 }","tags":"Python","url":"redoules.github.io/python/dict_comprehension.html","loc":"redoules.github.io/python/dict_comprehension.html"},{"title":"Extracting unique values from a list or an array","text":"Using lists An easy way to extract the unique values of a list in python is to convert the list to a set. A set is an unordered collection of items. Every element is unique (no duplicates) and must be immutable. my_list = [ 10 , 20 , 30 , 40 , 20 , 50 , 60 , 40 ] print ( f \"Original List : {my_list}\" ) my_set = set ( my_list ) my_new_list = list ( my_set ) # the set is converted back to a list with the list() function print ( f \"List of unique numbers : {my_new_list}\" ) Original List : [ 10 , 20 , 30 , 40 , 20 , 50 , 60 , 40 ] List of unique numbers : [ 40 , 10 , 50 , 20 , 60 , 30 ] Using numpy If you are using numpy you can extract the unique values of an array with the unique function builtin numpy: import numpy as np arr = np . array ( my_list ) print ( f 'Initial numpy array : {arr} \\n ' ) unique_arr = np . unique ( arr ) print ( f 'Numpy array with unique values : {unique_arr}' ) Initial numpy array : [ 10 20 30 40 20 50 60 40 ] Numpy array with unique values : [ 10 20 30 40 50 60 ]","tags":"Python","url":"redoules.github.io/python/unique.html","loc":"redoules.github.io/python/unique.html"},{"title":"Sorting an array","text":"Using lists Python provides an iterator to sort an array sorted() you can use it this way : import random # Random lists from [0-999] interval arr = [ random . randint ( 0 , 1000 ) for r in range ( 10 )] print ( f 'Initial random list : {arr} \\n ' ) reversed_arr = list ( sorted ( arr )) print ( f 'Sorted list : {reversed_arr}' ) Initial random list : [ 277 , 347 , 976 , 367 , 604 , 878 , 148 , 670 , 229 , 432 ] Sorted list : [ 148 , 229 , 277 , 347 , 367 , 432 , 604 , 670 , 878 , 976 ] it is also possible to use the sort function from the list object # Random lists from [0-999] interval arr = [ random . randint ( 0 , 1000 ) for r in range ( 10 )] print ( f 'Initial random list : {arr} \\n ' ) arr . sort () print ( f 'Sorted list : {arr}' ) Initial random list : [ 727 , 759 , 68 , 103 , 23 , 90 , 258 , 737 , 791 , 567 ] Sorted list : [ 23 , 68 , 90 , 103 , 258 , 567 , 727 , 737 , 759 , 791 ] Using numpy If you are using numpy you can sort an array by creating a view on the array: import numpy as np arr = np . random . random ( 5 ) print ( f 'Initial random array : {arr} \\n ' ) sorted_arr = np . sort ( arr ) print ( f 'Sorted array : {sorted_arr}' ) Initial random array : [ 0 . 40021786 0 . 13876208 0 . 19939047 0 . 46015169 0 . 43734158 ] Sorted array : [ 0 . 13876208 0 . 19939047 0 . 40021786 0 . 43734158 0 . 46015169 ]","tags":"Python","url":"redoules.github.io/python/sorting.html","loc":"redoules.github.io/python/sorting.html"},{"title":"Day 0 - Median, mean, mode and weighted mean","text":"A reminder The median The median is the value separating the higher half from the lower half of a data sample. For a data set, it may be thought of as the middle value. For a continuous probability distribution, the median is the value such that a number is equally likely to fall above or below it. The mean The arithmetic mean (or simply mean) of a sample is the sum of the sampled values divided by the number of items. The mode The mode of a set of data values is the value that appears most often. It is the value x at which its probability mass function takes its maximum value. In other words, it is the value that is most likely to be sampled. Implementation in python without using the scientific libraries def median ( l ): l = sorted ( l ) if len ( l ) % 2 == 0 : return ( l [ len ( l ) // 2 ] + l [( len ( l ) // 2 - 1 )]) / 2 else : return l [ len ( l ) // 2 ] def mean ( l ): return sum ( l ) / len ( l ) def mode ( data ): dico = { x : data . count ( x ) for x in list ( set ( data ))} return sorted ( sorted ( dico . items ()), key = lambda x : x [ 1 ], reverse = True )[ 0 ][ 0 ] L = [ 64630 , 11735 , 14216 , 99233 , 14470 , 4978 , 73429 , 38120 , 51135 , 67060 , 4978 , 73429 ] print ( f \"Sample : {L} \\n Mean : {mean(L)}, Median : {median(L)}, Mode : {mode(L)}\" ) Sample : [ 64630 , 11735 , 14216 , 99233 , 14470 , 4978 , 73429 , 38120 , 51135 , 67060 , 4978 , 73429 ] Mean : 43117 . 75 , Median : 44627 . 5 , Mode : 4978 The weighted average The weighted arithmetic mean is similar to an ordinary arithmetic mean (the most common type of average), except that instead of each of the data points contributing equally to the final average, some data points contribute more than others. data = [ 10 , 40 , 30 , 50 , 20 ] weights = [ 1 , 2 , 3 , 4 , 5 ] sum_X = sum ([ x * w for x , w in zip ( data , weights )]) print ( round (( sum_X / sum ( weights )), 1 )) 32 . 0","tags":"Blog","url":"redoules.github.io/blog/Statistics_10days-day0.html","loc":"redoules.github.io/blog/Statistics_10days-day0.html"},{"title":"Create a simple bash function","text":"A basic function The synthaxe to define a function is : #!/bin/bash # Basic function my_function () { echo Text displayed by my_function } #once defined, you can use it like so : my_function and it should return user@bash : ./my_function.sh Text displayed by my_function Function with arguments When used, the arguments are specified directly after the function name. Whithin the function they are accessible this the $ symbol followed by the number of the arguement. Hence $1 will take the value of the first arguement, $2 will take the value of the second arguement and so on. #!/bin/bash # Passing arguments to a function say_hello () { echo Hello $1 } say_hello Guillaume and it should return user@bash : ./function_arguements.sh Hello Guillaume Overriding Commands Using the previous example, let's override the echo function in order to make it say hello. To do so, you just need to name the function with the same name as the command you want to replace. When you are calling the original function, make sure you are using the builtin keyword #!/bin/bash # Overriding a function echo () { builtin echo Hello $1 } echo Guillaume user@bash : ./function_arguements.sh Hello Guillaume Returning values Use the keyword return to send back a value to the main program. The returned value will be stored in the $? variable #!/bin/bash # Retruning a value secret_number () { return 126 } secret_number echo The secret number is $? This code should return user@bash : ./retrun_value.sh The secret number is 126","tags":"Linux","url":"redoules.github.io/linux/simple_bash_function.html","loc":"redoules.github.io/linux/simple_bash_function.html"},{"title":"Number of edges in a Complete graph","text":"A complete graph contains \\(\\frac{n(n-1)}{2}\\) edges where \\(n\\) is the number of vertices (or nodes). if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) { var align = \"center\", indent = \"0em\", linebreak = \"false\"; if (false) { align = (screen.width < 768) ? \"left\" : align; indent = (screen.width < 768) ? \"0em\" : indent; linebreak = (screen.width < 768) ? 'true' : linebreak; } var mathjaxscript = document.createElement('script'); mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#'; mathjaxscript.type = 'text/javascript'; mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML'; mathjaxscript[(window.opera ? \"innerHTML\" : \"text\")] = \"MathJax.Hub.Config({\" + \" config: ['MMLorHTML.js'],\" + \" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } },\" + \" jax: ['input/TeX','input/MathML','output/HTML-CSS'],\" + \" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js'],\" + \" displayAlign: '\"+ align +\"',\" + \" displayIndent: '\"+ indent +\"',\" + \" showMathMenu: true,\" + \" messageStyle: 'normal',\" + \" tex2jax: { \" + \" inlineMath: [ ['\\\\\\\\(','\\\\\\\\)'] ], \" + \" displayMath: [ ['$$','$$'] ],\" + \" processEscapes: true,\" + \" preview: 'TeX',\" + \" }, \" + \" 'HTML-CSS': { \" + \" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} },\" + \" linebreaks: { automatic: \"+ linebreak +\", width: '90% container' },\" + \" }, \" + \"}); \" + \"if ('default' !== 'default') {\" + \"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {\" + \"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;\" + \"VARIANT['normal'].fonts.unshift('MathJax_default');\" + \"VARIANT['bold'].fonts.unshift('MathJax_default-bold');\" + \"VARIANT['italic'].fonts.unshift('MathJax_default-italic');\" + \"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');\" + \"});\" + \"}\"; (document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript); }","tags":"Mathematics","url":"redoules.github.io/mathematics/Number_edges_Complete_graph.html","loc":"redoules.github.io/mathematics/Number_edges_Complete_graph.html"},{"title":"Reverse an array","text":"Using lists Python provides an iterator to reverse an array reversed() you can use it this way : arr = list ( range ( 5 )) print ( f 'Initial array : {arr} \\n ' ) reversed_arr = list ( reversed ( arr )) print ( f 'Reversed array : {reversed_arr}' ) Initial array : [ 0 , 1 , 2 , 3 , 4 ] Reversed array : [ 4 , 3 , 2 , 1 , 0 ] Using numpy If you are using numpy you can reverse an array by creating a view on the array: import numpy as np arr = np . arange ( 5 ) print ( f 'Initial array : {arr} \\n ' ) reversed_arr = arr [:: - 1 ] print ( f 'Reversed array : {reversed_arr}' ) Initial array : [ 0 1 2 3 4 ] Reversed array : [ 4 3 2 1 0 ]","tags":"Python","url":"redoules.github.io/python/reverse.html","loc":"redoules.github.io/python/reverse.html"},{"title":"Advice for designing your own libraries","text":"Advice for designing your own libraries When designing your own library make sure to think of the following things. I will add new paragraphs to this article as I dicover new good practices. Use standard python objects Try to use standard python objects as much as possible. That way, your library becomes compatible with all the other python libaries. For instance, when I created SAMpy : a library for reading and writing SAMCEF results, it returned dictonnaries, lists and pandas dataframes. Hence the results extracted from SAMCEF where compatible with all the scientific stack of python. Limit the number of functionnalities Following the same logic as before, the objects should do only one thing but do it well. Indeed, having a simple interface will reduce the complexity of your code and make it easier to use your library. Again, with SAMpy, I decided to strictly limit the functionnalities to reading and writing SAMCEF files. Define an exception class for your library You should define your own exceptions in order to make it easier for your users to debug their code thanks to clearer messages that convey more meaning. That way, the user will know if the error comes from your library or something else. Bonus if you group similar exceptions in a hierachy of inerited Exception classes. Example : let's create a Exception related to the age of a person : def check_age ( age ): if age < 0 and age > 130 : raise ValueError If the user inputed an invalid age, the ValueError exception would be thrown. That's fine but imagine you wan't to provide more feedback to your users that don't know the internal of your library. Let's now create a selfexplanatory Exception class AgeInvalidError ( ValueError ): pass def check_age ( age ): if age < 0 and age > 130 : raise AgeInvalidError ( age ) You can also add some helpful text to guide your users along the way: class AgeInvalidError ( ValueError ): print ( \"Age invalid, must be between 0 and 130\" ) pass def check_age ( age ): if age < 0 and age > 130 : raise AgeInvalidError ( age ) If you want to group all the logically linked exceptions, you can create a base class and inherit from it : class BaseAgeInvalidError ( ValueError ): pass class TooYoungError ( BaseAgeInvalidError ): pass class TooOldError ( BaseAgeInvalidError ): pass def check_age ( age ): if age < 0 : raise TooYoungError ( age ) elif age > 130 : raise TooOldError ( age ) Structure your repository You should have a file structure in your repository. It will help other contributers especially future contributers. A nice directory structure for your project should look like this: README . md LICENSE setup . py requirements . txt . / MyPackage . / docs . / tests Some prefer to use reStructured Text, I personnaly prefer Markdown choosealicense.com will help you pick the license to use for your project. For package and distribution management, create a setup.py file a the root of the directory The list of dependencies required to test, build and generate the doc are listed in a pip requirement file placed a the root of the directory and named requirements.txt Put the documentation of your library in the docs directory. Put your tests in the tests directory. Since your tests will need to import your library, I recommend modifying the path to resolve your package property. In order to do so, you can create a context.py file located in the tests directory : import os import sys sys . path . insert ( 0 , os . path . abspath ( os . path . join ( os . path . dirname ( __file__ ), '..' ))) import MyPackage Then within your individual test files you can import your package like so : from .context import MyPackage Finally, your code will go into the MyPackage directory Test your code Once your library is in production, you have to guaranty some level of forward compatibility. Once your interface is defined, write some tests. In the future, when your code is modified, having those tests will make sure that the behaviour of your functions and objects won't be altered. Document your code Of course, you should have a documentation to go along with your library. Make sure to add a lot of commun examples as most users tend to learn from examples. I recommend writing your documentation using Sphinx.","tags":"Python","url":"redoules.github.io/python/design_own_libs.html","loc":"redoules.github.io/python/design_own_libs.html"},{"title":"Safely creating a folder if it doesn't exist","text":"Safely creating a folder if it doesn't exist When you are writing to files in python, if the file doesn't exist it will be created. However, if you are trying to write a file in a directory that doesn't exist, an exception will be returned FileNotFoundError : [ Errno 2 ] No such file or directory : \"directory\" This article will teach you how to make sure the target directory exists. If it doesn't, the function will create that directory. First, let's import os and make sure that the \"test_directory\" doesn't exist import os os . path . exists ( \". \\\\ test_directory\" ) False copy the ensure_dir function in your code. This function will handle the creation of the directory. Credit goes to Parand posted on StackOverflow def ensure_dir ( file_path ): directory = os . path . dirname ( file_path ) if not os . path . exists ( directory ): os . makedirs ( directory ) Let's now use the function and create a folder named \"test_directory\" ensure_dir ( \". \\\\ test_directory\" ) If we test for the existence of the directory, the exists function will now return True os . path . exists ( \". \\\\ test_directory\" ) True","tags":"Python","url":"redoules.github.io/python/ensure_dir.html","loc":"redoules.github.io/python/ensure_dir.html"},{"title":"List all files in a directory","text":"Listing all the files in a directory Let's start with the basics, the most staigthforward way to list all the files in a direcoty is to use a combinaison of the listdir function and isfile form os.path. You can use a list comprehension to store all the results in a list. mypath = \"./test_directory/\" from os import listdir from os.path import isfile , join [ f for f in listdir ( mypath ) if isfile ( join ( mypath , f ))] [ 'logfile.log' , 'myfile.txt' , 'super_music.mp3' , 'textfile.txt' ] Listing all the files of a certain type in a directory similarly, if you want to filter only a certain kind of file based on its extension you can use the endswith method. In the following example, we will filter all the \"txt\" files contained in the directory [ f for f in listdir ( mypath ) if f . endswith ( '.' + \"txt\" )] [ 'myfile.txt' , 'textfile.txt' ] Listing all the files matching a pattern in a directory The glob module finds all the pathnames matching a specified pattern according to the rules used by the Unix shell. You can use the *, ?, and character ranges expressed with [] wildcards import glob glob . glob ( \"*.txt\" ) [ 'myfile.txt' ] Listing files recusively If you want to list all files recursively you can select all the sub-directories using the \"**\" wildcard import glob glob . glob ( mypath + '/**/*.txt' , recursive = True ) [ './test_directory\\\\myfile.txt' , './test_directory\\\\textfile.txt' , './test_directory\\\\subdir1\\\\file_hidden_in_a_sub_direcotry.txt' ] Using a regular expression If you'd rather use a regular expression to select the files, the pathlib library provides the rglob function. from pathlib import Path list ( Path ( \"./test_directory/\" ) . rglob ( \"*.[tT][xX][tT]\" )) [ WindowsPath ( 'test_directory/myfile.txt' ), WindowsPath ( 'test_directory/textfile.txt' ), WindowsPath ( 'test_directory/subdir1/file_hidden_in_a_sub_direcotry.txt' )] Using regular expressions you can for example select multiple types of files. In the following example, we list all the files that finish either with \"txt\" or with \"log\". list ( Path ( \"./test_directory/\" ) . rglob ( \"*.[tl][xo][tg]\" )) [ WindowsPath ( 'test_directory/logfile.log' ), WindowsPath ( 'test_directory/myfile.txt' ), WindowsPath ( 'test_directory/textfile.txt' ), WindowsPath ( 'test_directory/subdir1/file_hidden_in_a_sub_direcotry.txt' )]","tags":"Python","url":"redoules.github.io/python/list_files_directory.html","loc":"redoules.github.io/python/list_files_directory.html"},{"title":"Using Dask on infiniband","text":"InfiniBand (abbreviated IB) is a computer-networking communications standard used in high-performance computing that features very high throughput and very low latency. It is used for data interconnect both among and within computers. InfiniBand is also used as either a direct or switched interconnect between servers and storage systems, as well as an interconnect between storage systems. (source Wikipedia). If you want to leverage this high speed network instead of the regular ethernet network, you have to specify to the scheduler that you want to used infiniband as your interface. Assuming that you Infiniband interface is ib0 , you would call the scheduler like this : dask - scheduler --interface ib0 --scheduler-file ./cluster.yaml you would have to call the worker using the same interface : dask - worker --interface ib0 --scheduler-file ./cluster.yaml","tags":"Python","url":"redoules.github.io/python/dask_infiniband.html","loc":"redoules.github.io/python/dask_infiniband.html"},{"title":"Clearing the current cell in the notebook","text":"In python, you can clear the output of a cell by importing the IPython.display module and using the clear_output function from IPython.display import clear_output print ( \"text to be cleared\" ) clear_output () As you can see, the text \"text to be cleared\" is not displayed because the function clear_output has been called afterward","tags":"Jupyter","url":"redoules.github.io/jupyter/clear_cell.html","loc":"redoules.github.io/jupyter/clear_cell.html"},{"title":"What's inside my .bashrc ?","text":"############ # Anaconda # ############ export PATH = \"/station/guillaume/anaconda3/bin: $PATH \" alias python = '/station/guillaume/anaconda3/bin/python' ######### # Alias # ######### alias ding = 'echo -e \"\\a\"' alias calc = 'python -ic \"from __future__ import division; from math import *\"' alias h = \"history|grep \" alias f = \"find . |grep \" alias p = \"ps aux |grep \" alias cdl = \"cd /data/guillaume\" alias cp = \"rsync -avz --progress\" alias grep = \"grep --color=auto\" alias ls = \"ls -hN --color=auto --group-directories-first\" alias ll = \"ls -hal\" alias sv = \"ssh compute_cluster\" alias ms = \"ls\" alias jl = \"jupyter lab\" alias lst = \"jupyter notebook list\" ########################## # bashrc personnalisation# ########################## force_color_prompt = yes export EDITOR = nano export BROWSER = \"firefox '%s' &\" if [ -n \" $force_color_prompt \" ] ; then if [ -x /usr/bin/tput ] && tput setaf 1 > & /dev/null ; then # We have color support; assume it's compliant with Ecma-48 # (ISO/IEC-6429). (Lack of such support is extremely rare, and such # a case would tend to support setf rather than setaf.) color_prompt = yes else color_prompt = fi fi if [ \" $color_prompt \" = yes ] ; then #\\h : hostname #\\u : user #\\w : current working directory #\\d : date #\\t : time yellow = 226 green = 83 pink = 198 blue = 34 PS1 = \"\\[\\033[38;5;22m\\]\\u\\[ $( tput sgr0 ) \\]\\[\\033[38;5;163m\\]@\\[ $( tput sgr0 ) \\]\\[\\033[38;5;22m\\]\\h\\[ $( tput sgr0 ) \\]\\[\\033[38;5;162m\\]:\\[ $( tput sgr0 ) \\]\\[\\033[38;5;172m\\]{\\[ $( tput sgr0 ) \\]\\[\\033[38;5;39m\\]\\w\\[ $( tput sgr0 ) \\]\\[\\033[38;5;172m\\]}\\[ $( tput sgr0 ) \\]\\[\\033[38;5;162m\\]>\\[ $( tput sgr0 ) \\]\"","tags":"Linux","url":"redoules.github.io/linux/bashrc.html","loc":"redoules.github.io/linux/bashrc.html"},{"title":"Efficient extraction of eigenvalues from a list of tensors","text":"When you manipulate FEM results you generally have either a: * scalar field, * vector field, * tensor field. With tensorial results, it is often useful to extract the eigenvalues in order to find the principal values. I have found that it is easier to store the components of the tensors in a 6 column pandas dataframe (because of the symmetric property of stress and strain tensors) import pandas as pd node = [ 1001 , 1002 , 1003 , 1004 ] #when dealing with FEM results you should remember at which element/node the result is computed (in the example, let's assume that we look at node from 1001 to 1004) tensor1 = [ 1 , 1 , 1 , 0 , 0 , 0 ] #eigen : 1 tensor2 = [ 4 , - 1 , 0 , 2 , 2 , 1 ] #eigen : 5.58443, -1.77931, -0.805118 tensor3 = [ 1 , 6 , 5 , 3 , 3 , 1 ] #eigen : 8.85036, 4.46542, -1.31577 tensor4 = [ 1 , 2 , 3 , 0 , 0 , 0 ] #eigen : 1, 2, 3 df = pd . DataFrame ([ tensor1 , tensor2 , tensor3 , tensor4 ], columns = [ \"XX\" , \"YY\" , \"ZZ\" , \"XY\" , \"XZ\" , \"YZ\" ]) df . index = node df .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } XX YY ZZ XY XZ YZ 1001 1 1 1 0 0 0 1002 4 -1 0 2 2 1 1003 1 6 5 3 3 1 1004 1 2 3 0 0 0 If you want to extract the eigenvalues of a tensor with numpy you have to pass a n by n ndarray to the eigenvalue function. In order to avoid having to loop over each node, this oneliner is highly optimized and will help you invert a large number of tensors efficiently. The steps are basically, create a list of n by n values (here n=3) in the right order => reshape it to a list of tensors => pass it to the eigenvals function import numpy as np from numpy import linalg as LA eigenvals = LA . eigvals ( df [[ \"XX\" , \"XY\" , \"XZ\" , \"XY\" , \"YY\" , \"YZ\" , \"XZ\" , \"YZ\" , \"ZZ\" ]] . values . reshape ( len ( df ), 3 , 3 )) eigenvals array ([[ 1 . , 1 . , 1 . ], [ 5 . 58442834 , - 0 . 80511809 , - 1 . 77931025 ], [ - 1 . 31577211 , 8 . 85035616 , 4 . 46541595 ], [ 1 . , 2 . , 3 . ]])","tags":"Python","url":"redoules.github.io/python/Efficient_extraction_of_eigenvalues_from_a_list_of_tensors.html","loc":"redoules.github.io/python/Efficient_extraction_of_eigenvalues_from_a_list_of_tensors.html"},{"title":"Optimized numpy random number generation on Intel CPU","text":"Python Intel distribution Make sure you have a python intel distribution. When you startup python you should see somethine like : Python 3.6.2 |Intel Corporation| (default, Aug 15 2017, 11:34:02) [MSC v.1900 64 bit (AMD64)] Type 'copyright', 'credits' or 'license' for more information IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help. If not, you can force the installation of the intel optimized python with : conda update --all conda config --add channels intel conda install numpy --channel intel --override-channels oh and by the way, make sure you a running an Intel CPU ;) Comparing numpy.random with numpy.random_intel Let's now test both the rand function with and without the Intel optimization import numpy as np from numpy import random , random_intel % timeit np . random . rand ( 10 ** 5 ) 1 . 06 ms ± 91 . 5 µ s per loop ( mean ± std . dev . of 7 runs , 1000 loops each ) % timeit np . random_intel . rand ( 10 ** 5 ) 225 µ s ± 3 . 46 µ s per loop ( mean ± std . dev . of 7 runs , 1000 loops each )","tags":"Python","url":"redoules.github.io/python/Optimized_numpy_random_intel.html","loc":"redoules.github.io/python/Optimized_numpy_random_intel.html"},{"title":"How to check Linux process information?","text":"How to check Linux process information (CPU usage, memory, user information, etc.)? You need to use the ps command combined with the grep command. In the example, we want to check the information on the nginx process : ps aux | grep nginx It would return the output : root 9976 0.0 0.0 12272 108 ? S mtu 1500 group default qlen 1 link/ether 5c:51:4f:41:7a:b1 inet 169.254.33.33/16 brd 169.254.255.255 scope global dynamic valid_lft forever preferred_lft forever inet6 fe80::390a:f69e:1ba2:2121/64 scope global dynamic valid_lft forever preferred_lft forever 3: eth1: mtu 1500 group default qlen 1 link/ether 0a:00:27:00:00:03 inet 192.168.56.1/24 brd 192.168.56.255 scope global dynamic valid_lft forever preferred_lft forever inet6 fe80::84cd:374e:843f:f82f/64 scope global dynamic valid_lft forever preferred_lft forever 15: eth2: <> mtu 1500 group default qlen 1 link/ether 00:ff:d2:8a:19:c3 inet 169.254.40.62/16 brd 169.254.255.255 scope global dynamic valid_lft forever preferred_lft forever inet6 fe80::f199:2ca0:7aff:283e/64 scope global dynamic valid_lft forever preferred_lft forever 1: lo: mtu 1500 group default qlen 1 link/loopback 00:00:00:00:00:00 inet 127.0.0.1/8 brd 127.255.255.255 scope global dynamic valid_lft forever preferred_lft forever inet6 ::1/128 scope global dynamic valid_lft forever preferred_lft forever 5: wifi0: mtu 1500 group default qlen 1 link/ieee802.11 5c:51:4f:41:7a:ad inet 192.168.1.1/24 brd 192.168.1.255 scope global dynamic valid_lft 42720sec preferred_lft 42720sec inet6 fe80::395f:3594:1dc2:57e3/64 scope global dynamic valid_lft forever preferred_lft forever 21: wifi1: <> mtu 1500 group default qlen 1 link/ieee802.11 5c:51:4f:41:7a:ae inet 169.254.12.77/16 brd 169.254.255.255 scope global dynamic valid_lft forever preferred_lft forever inet6 fe80::58d5:630:cbbd:c4d/64 scope global dynamic valid_lft forever preferred_lft forever 12: eth3: <> mtu 1472 group default qlen 1 link/ether 00:00:00:00:00:00:00:e0:00:00:00:00:00:00:00:00 inet6 fe80::100:7f:fffe/64 scope global dynamic valid_lft forever preferred_lft forever 10: eth4: mtu 1500 group default qlen 1 link/ether 22:b7:57:52:5f:ff inet 192.168.42.106/24 brd 192.168.42.255 scope global dynamic valid_lft 6659sec preferred_lft 6659sec inet6 fe80::5110:eb6f:deb0:45c4/64 scope global dynamic valid_lft forever preferred_lft forever You can select only one interface ip addr show eth0 and only the relevant information will be displayed 4: eth0: <> mtu 1500 group default qlen 1 link/ether 5c:51:4f:41:7a:b1 inet 169.254.33.33/16 brd 169.254.255.255 scope global dynamic valid_lft forever preferred_lft forever inet6 fe80::390a:f69e:1ba2:2121/64 scope global dynamic valid_lft forever preferred_lft forever Check your current ip address the old way The ifconfig command will return information regarding your network interfaces. Let's try it: ifconfig eth1 Link encap:Ethernet HWaddr 0a:00:27:00:00:03 inet adr:192.168.56.1 Bcast:192.168.56.255 Masque:255.255.255.0 adr inet6: fe80::84cd:374e:843f:f82f/64 Scope:Global UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Packets reçus:0 erreurs:0 :0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 Octets reçus:0 (0.0 B) Octets transmis:0 (0.0 B) eth4 Link encap:Ethernet HWaddr 22:b7:57:52:5f:ff inet adr:192.168.42.106 Bcast:192.168.42.255 Masque:255.255.255.0 adr inet6: fe80::5110:eb6f:deb0:45c4/64 Scope:Global UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Packets reçus:0 erreurs:0 :0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 Octets reçus:0 (0.0 B) Octets transmis:0 (0.0 B) lo Link encap:Boucle locale inet adr:127.0.0.1 Masque:255.0.0.0 adr inet6: ::1/128 Scope:Global UP LOOPBACK RUNNING MTU:1500 Metric:1 Packets reçus:0 erreurs:0 :0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 Octets reçus:0 (0.0 B) Octets transmis:0 (0.0 B) wifi0 Link encap:UNSPEC HWaddr 5C-51-4F-41-7A-AD-00-00-00-00-00-00-00-00-00-00 inet adr:192.168.1.1 Bcast:192.168.1.255 Masque:255.255.255.0 adr inet6: fe80::395f:3594:1dc2:57e3/64 Scope:Global UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Packets reçus:0 erreurs:0 :0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 Octets reçus:0 (0.0 B) Octets transmis:0 (0.0 B) On the left column you have the list of network adapters. The lo is the local loopback, it is an interface that points to the localhost. Interfaces starting with eth refer to wired connections over ethernet (or sometimes USB in the case of a phone acting as an access point over USB). Interfaces starting with wlan or wifi refer to wireless connections. On the right column you have some information corresponding to the interface such as the IPv4, the IPv6, the mask, some statistics about the interface and so on.","tags":"Linux","url":"redoules.github.io/linux/get_ip_linux.html","loc":"redoules.github.io/linux/get_ip_linux.html"},{"title":"Check the version of the kernel currently running","text":"Check the version of the kernel currently running The uname command will give you the version of the kernel. In order to get a more useful output, type uname -a This will return : * the hostname * os name * kernel release/version * architecture * etc. variations If you only want the kernel version you can type uname -v if you only want the kernel release you can type uname -r","tags":"Linux","url":"redoules.github.io/linux/version_kernel.html","loc":"redoules.github.io/linux/version_kernel.html"},{"title":"Running the notebook on a remote server","text":"Jupyter hub With JupyterHub you can create a multi-user Hub which spawns, manages, and proxies multiple instances of the single-user Jupyter notebook server. Project Jupyter created JupyterHub to support many users. The Hub can offer notebook servers to a class of students, a corporate data science workgroup, a scientific research project, or a high performance computing group. However, if you are the only one using the server and you just want a simple way to run the notebook on your server and access it through the web interface on a light client without having to install and configure the jupyter hub, you can do the following. Problem with jupyter notebook On your server, run the command jupyter-notebook you should get something like : [I 11:18:44.514 NotebookApp] Serving notebooks from local directory: /volume2/homes/admin [I 11:18:44.515 NotebookApp] 0 active kernels [I 11:18:44.516 NotebookApp] The Jupyter Notebook is running at: http://localhost:8888/?token=357587e3269b0f20f2b7e1918492890ae7573ac7ef1d2023 [I 11:18:44.516 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [W 11:18:44.519 NotebookApp] No web browser found: could not locate runnable browser. [C 11:18:44.520 NotebookApp] Copy/paste this URL into your browser when you connect for the first time, to login with a token: http://localhost:8888/?token=357587e3269b0f20f2b7e1918492890ae7573ac7ef1d2023 and if you try to connect to your server ip (in my example : http://192.168.1.2:8888/?token=357587e3269b0f20f2b7e1918492890ae7573ac7ef1d2023) you will get an \"ERR_CONNECTION_REFUSED\" error. This is because, by default, Jupyter Notebook only accepts connections from localhost. Allowing connexions from other sources From any IP The simplest way to avoid the connection error is to allow the notebook to accept connections from any ip jupyter-notebook --ip = * you will get something like [W 11:26:45.285 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended. [I 11:26:45.342 NotebookApp] Serving notebooks from local directory: /volume2/homes/admin [I 11:26:45.342 NotebookApp] 0 active kernels [I 11:26:45.343 NotebookApp] The Jupyter Notebook is running at: http://[all ip addresses on your system]:8888/?token=52af33d628881824968b4031967e8541a27cc28b1720c199 [I 11:26:45.343 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [W 11:26:45.346 NotebookApp] No web browser found: could not locate runnable browser. [C 11:26:45.347 NotebookApp] Copy/paste this URL into your browser when you connect for the first time, to login with a token: http://localhost:8888/?token=52af33d628881824968b4031967e8541a27cc28b1720c199 and if you connect form a remote client (192.168.1.1 in my example), the following line will be added to the output : [I 11:26:54.798 NotebookApp] 302 GET /?token=52af33d628881824968b4031967e8541a27cc28b1720c199 (192.168.1.1) 111.17ms note that you should only do that if you are the only one using the server because the connection is not encypted. From a specific IP You can also, explicitly specify the ip of the client jupyter-notebook --ip = 192 .168.1.1 [I 11:44:58.104 NotebookApp] JupyterLab extension loaded from C:\\Users\\Guillaume\\Miniconda3\\lib\\site-packages\\jupyterlab [I 11:44:58.104 NotebookApp] JupyterLab application directory is C:\\Users\\Guillaume\\Miniconda3\\share\\jupyter\\lab [I 11:44:58.244 NotebookApp] Serving notebooks from local directory: C:\\Users\\Guillaume [I 11:44:58.245 NotebookApp] 0 active kernels [I 11:44:58.245 NotebookApp] The Jupyter Notebook is running at: http://192.168.1.1:8888/?token=503576dd8fa87d1f2c416df307e9b900e520b4942e317b32 [I 11:44:58.245 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [C 11:44:58.258 NotebookApp] Copy/paste this URL into your browser when you connect for the first time, to login with a token: http://192.168.1.1:8888/?token=503576dd8fa87d1f2c416df307e9b900e520b4942e317b32 [I 11:44:59.083 NotebookApp] Accepting one-time-token-authenticated connection from 192.168.1.1","tags":"Jupyter","url":"redoules.github.io/jupyter/remote_run_notebook.html","loc":"redoules.github.io/jupyter/remote_run_notebook.html"},{"title":"Running multiple calls to a function in parallel with Dask","text":"Dask.distributed is a lightweight library for distributed computing in Python. It allows to create a compute graph. Dask distributed is architectured around 3 parts : the dask-scheduler the dask-worker(s) the dask client Dask architecture The Dask scheduler is a centrally managed, distributed, dynamic task scheduler. It recieves tasks from a/multiple client(s) and spread them across one or multiple dask-worker(s). Dask-scheduler is an event based asynchronous dynamic scheduler, meaning that mutliple clients can submit a list of task to be executed on multiple workers. Internally, the task are represented as a directed acyclic graph. Both new clients and new workers can be connected or disconnected during the execution of the task graph. Tasks can be submited with the function client . submit ( function , * args , ** kwargs ) or by using objects from the dask library such as dask.dataframe, dask.bag or dask.array Setup In this example, we will use a distributed scheduler on a single machine with multiple workers and a single client. We will use the client to submit some tasks to the scheduler. The scheduler will then dispatch those tasks to the workers. The process can be monitored in real time through a web application. For this example, all the computations will be run on a local computer. However dask can scale to a large HPC cluster. First we have to launch the dask-scheduler; from the command line, input dask-scheduler Next, you can load the web dashboard. In order to do so, the scheduler returns the number of the port you have to connect to in the line starting with \"bokeh at :\". The default port is 8787. Since we are running all the programs on the same computer, we just have to login to http://127.0.0.1:8787/status Finally, we have to launch the dask-worker(s). If you want to run the worker(s) on the same computer as the scheduler the type : dask-worker 127 .0.0.1:8786 otherwise, make sure you are inputing the ip address of the computer hosting the dask-scheduler. You can launch as many workers as you want. In this example, we will run 3 workers on the local machine. Use the dask workers within your python code We will now see how to submit multiple calls to a fucntion in parallel on the dask-workers. Import the required libraries and define the function to be executed. import numpy as np import pandas as pd from distributed import Client #function used to do parallel computing on def compute_pi_MonteCarlo ( Nb_Data ): \"\"\" computes the value of pi using the monte carlo method \"\"\" Radius = 1 Nb_Data = int ( round ( Nb_Data )) x = np . random . uniform ( - Radius , Radius , Nb_Data ) y = np . random . uniform ( - Radius , Radius , Nb_Data ) pi_mc = 4 * np . sum ( np . power ( x , 2 ) + np . power ( y , 2 ) < Radius ** 2 ) / Nb_Data err = 100 * np . abs ( pi_mc - np . pi ) / np . pi return [ Nb_Data , pi_mc , err ] In order to connect to the scheduler, we create a client. client = Client ( '127.0.0.1:8786' ) client Client Scheduler: tcp://127.0.0.1:8786 Dashboard: http://127.0.0.1:8787/status Cluster Workers: 3 Cores: 12 Memory: 25.48 GB We submit tasks using the submit method data = [ client . submit ( compute_pi_MonteCarlo , Nb_Data ) for Nb_Data in np . logspace ( 3 , 7 , num = 1200 , dtype = int )] If you look at http://127.0.0.1:8787/status you will see the tasks beeing completed. Once competed, gather the data: data = client . gather ( data ) df = pd . DataFrame ( data ) df . columns = [ \"number of points for MonteCarlo\" , \"value of pi\" , \"error (%)\" ] df . tail () .dataframe thead tr:only-child th { text-align: right; } .dataframe thead th { text-align: left; } .dataframe tbody tr th { vertical-align: top; } number of points for MonteCarlo value of pi error (%) 1195 9697405 3.141296 0.009454 1196 9772184 3.141058 0.017008 1197 9847540 3.141616 0.000739 1198 9923477 3.141009 0.018574 1199 10000000 3.141032 0.017833 There, we have completed a simple example on how to use dask to run multiple functions in parallel. Full source code: import numpy as np import pandas as pd from distributed import Client #function used to do parallel computing on def compute_pi_MonteCarlo ( Nb_Data ): \"\"\" computes the value of pi using the monte carlo method \"\"\" Radius = 1 Nb_Data = int ( round ( Nb_Data )) x = np . random . uniform ( - Radius , Radius , Nb_Data ) y = np . random . uniform ( - Radius , Radius , Nb_Data ) pi_mc = 4 * np . sum ( np . power ( x , 2 ) + np . power ( y , 2 ) < Radius ** 2 ) / Nb_Data err = 100 * np . abs ( pi_mc - np . pi ) / np . pi return [ Nb_Data , pi_mc , err ] #connect to the scheduler client = Client ( '127.0.0.1:8786' ) #submit tasks data = [ client . submit ( compute_pi_MonteCarlo , Nb_Data ) for Nb_Data in np . logspace ( 3 , 7 , num = 1200 , dtype = int )] #gather the results data = client . gather ( data ) df = pd . DataFrame ( data ) df . columns = [ \"number of points for MonteCarlo\" , \"value of pi\" , \"error (%)\" ] df . tail () A word on the environement variables On Windows, to make sure that you can run dask-scheduler and dask-worker from the command line, you have to add the location of the executable to your path. On linux, you can append the location of the dask-worker and scheduler to the path variable with the command export PATH = $PATH :/path/to/dask","tags":"Python","url":"redoules.github.io/python/dask_distributed_parallelism.html","loc":"redoules.github.io/python/dask_distributed_parallelism.html"},{"title":"Plotting data using log axis","text":"Plotting in log axis with matplotlib import matplotlib.pyplot as plt % matplotlib inline import numpy as np x = np . linspace ( 0.1 , 20 ) y = 20 * np . exp ( - x / 10.0 ) Plotting using the standard function then specifying the axis scale One of the easiest way to plot in a log plot is to specify the plot normally and then specify which axis is to be plotted with a log scale. This can be specified by the function set_xscale or set_yscale # Normal plot fig = plt . figure () ax = fig . add_subplot ( 1 , 1 , 1 ) ax . plot ( x , y ) ax . grid () plt . show () # Log x axis plot fig = plt . figure () ax = fig . add_subplot ( 1 , 1 , 1 ) ax . plot ( x , y ) ax . set_xscale ( 'log' ) ax . grid () plt . show () # Log x axis plot fig = plt . figure () ax = fig . add_subplot ( 1 , 1 , 1 ) ax . plot ( x , y ) ax . set_yscale ( 'log' ) ax . grid () plt . show () # Log x axis plot fig = plt . figure () ax = fig . add_subplot ( 1 , 1 , 1 ) ax . plot ( x , y ) ax . set_xscale ( 'log' ) ax . set_yscale ( 'log' ) ax . grid () plt . show () Plotting using the matplotlib defined function Matplotlib has the function : semilogx, semilogy and loglog that can help you avoid having to specify the axis scale. # Plot using semilogx fig = plt . figure () ax = fig . add_subplot ( 1 , 1 , 1 ) ax . semilogx ( x , y ) ax . grid () plt . show () # Plot using semilogy fig = plt . figure () ax = fig . add_subplot ( 1 , 1 , 1 ) ax . semilogy ( x , y ) ax . grid () plt . show () # Plot using loglog fig = plt . figure () ax = fig . add_subplot ( 1 , 1 , 1 ) ax . loglog ( x , y ) ax . grid () plt . show ()","tags":"Python","url":"redoules.github.io/python/logplot.html","loc":"redoules.github.io/python/logplot.html"},{"title":"Downloading a static webpage with python","text":"If you are using python legacy (aka python 2) first of all, stop ! Furthermore, this method won't work in python legacy # Import modules from urllib.request import urlopen The webpage source code can be downloaded with the command urlopen url = \"http://example.com/\" #create a HTTP request in order to read the page page = urlopen ( url ) . read () The source code will be stored in the variable page as a string print ( page ) b ' \\n \\n \\n Example Domain \\n\\n \\n \\n \\n \\n \\n\\n \\n \\n
Example Domain
\\n
This domain is established to be used for illustrative examples in documents. You may use this \\n domain in examples without prior coordination or asking for permission.
\\n
More information...
\\n