guillaumeredoules/redoules.github.io
1
0
Fork 0
mirror of https://github.com/redoules/redoules.github.io.git synced 2025-12-12 15:59:34 +00:00
Code Issues 3 Releases Wiki Activity
44f740504b
redoules.github.io/linux/sed.html
Guillaume 44f740504b added an article 
about uploading data to a sharepoint site
2020-07-20 20:20:09 +02:00

543 lines
48 KiB
HTML
Raw Blame History

<!DOCTYPE html>
<html lang="fr">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
<meta name="description" content="Data Science for Political and Social Phenomena">
<meta name="author" content="Guillaume Redoulès">
<link rel="icon" href="../favicon.ico">
<title>Basic operations with SED - Linux</title>
<!-- JQuery -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
<script>
window.jQuery || document.write('<script src="../theme/js/jquery.min.js"><\/script>')
</script>
<!-- Bootstrap core CSS -->
<link rel="stylesheet" href="../theme/css/bootstrap.css" />
<!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
<link rel="stylesheet" type="text/css" href="../theme/css/ie10-viewport-bug-workaround.css" />
<!-- Custom styles for this template -->
<link rel="stylesheet" type="text/css" href="../theme/css/style.css" />
<link rel="stylesheet" type="text/css" href="../theme/css/notebooks.css" />
<link href='https://fonts.googleapis.com/css?family=PT+Serif:400,700|Roboto:400,500,700' rel='stylesheet' type='text/css'>
<!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
<![endif]-->
<meta name="tags" content="Bash" />
</head>
<body>
<div class="navbar navbar-fixed-top">
<div class="container">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="..">Guillaume Redoulès</a>
</div>
<div class="navbar-collapse collapse" id="searchbar">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">About<span class="caret"></span></a>
<ul class="dropdown-menu">
<li><a href="../pages/about.html">About Guillaume</a></li>
<li><a href="https://github.com/redoules">GitHub</a></li>
<li><a href="https://www.linkedin.com/in/guillaume-redoul%C3%A8s-33923860/">LinkedIn</a></li>
</ul>
</li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Data Science<span class="caret"></span></a>
<ul class="dropdown-menu">
<li><a href="..#Blog">Blog</a></li>
<li><a href="..#Python">Python</a></li>
<li><a href="..#Bash">Bash</a></li>
<li><a href="..#SQL">SQL</a></li>
<li><a href="..#Mathematics">Mathematics</a></li>
<li><a href="..#Machine_Learning">Machine Learning</a></li>
<li><a href="..#Projects">Projects</a></li>
</ul>
</li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Projects<span class="caret"></span></a>
<ul class="dropdown-menu">
<li><a href="https://github.com/redoules/redoules.github.io">Notes (Github)</a></li>
</ul>
</li>
<!--<li class="dropdown">
<a href="../feeds/blog.rss.xml">Blog RSS</a>
</li>-->
</ul>
<form class="navbar-form" action="../search.html" onsubmit="return validateForm(this.elements['q'].value);">
<div class="form-group" style="display:inline;">
<div class="input-group" style="display:table;">
<span class="input-group-addon" style="width:1%;"><span class="glyphicon glyphicon-search"></span></span>
<input class="form-control search-query" name="q" id="tipue_search_input" placeholder="e.g. scikit KNN, pandas merge" required autocomplete="off" type="text">
</div>
</div>
</form>
</div>
<!--/.nav-collapse -->
</div>
</div>
<!-- end of header section -->
<div class="container">
<!-- <div class="alert alert-warning" role="alert">
Did you find this page useful? Please do me a quick favor and <a href="#" class="alert-link">endorse me for data science on LinkedIn</a>.
</div> -->
<section id="content" class="body">
<header>
<h1>
Basic operations with SED
</h1>
<ol class="breadcrumb">
<li>
<time class="published" datetime="2018-12-24T15:03:00+01:00">
24 décembre 2018
</time>
</li>
<li>Linux</li>
<li>Bash</li>
</ol>
</header>
<div class='article_content'>
<p>In this article we will learn about some of the main uses we can use <code>sed</code> for :
* replacing
* deleting
* printing </p>
<p>For this example we will learn how to remove the comments starting with the '#' sign and the blank lines for the following file :</p>
<div class="highlight"><pre><span></span><span class="o">##</span> <span class="n">Header</span> <span class="k">of</span> <span class="k">input</span><span class="p">.</span><span class="n">csv</span>
<span class="o">#</span><span class="n">this</span> <span class="n">file</span> <span class="k">contains</span> <span class="n">information</span> <span class="n">I</span> <span class="n">want</span> <span class="k">to</span> <span class="n">parse</span> <span class="k">with</span> <span class="n">a</span> <span class="k">simple</span> <span class="n">program</span><span class="p">.</span>
<span class="o">#</span><span class="n">The</span> <span class="n">header</span><span class="p">,</span> <span class="n">the</span> <span class="n">footer</span> <span class="k">or</span> <span class="k">any</span> <span class="k">comment</span> <span class="n">starting</span> <span class="k">with</span> <span class="n">a</span> <span class="ss">&quot;#&quot;</span> <span class="n">will</span> <span class="n">be</span> <span class="n">removed</span>
<span class="o">#</span><span class="n">The</span> <span class="n">blank</span> <span class="n">lines</span> <span class="n">will</span> <span class="n">also</span> <span class="n">be</span> <span class="n">removed</span>
<span class="o">#</span><span class="n">img</span><span class="p">,</span><span class="n">processed</span><span class="p">,</span><span class="n">defaut</span>
<span class="o">#</span><span class="n">bloc</span> <span class="mi">1</span>
<span class="mi">0</span><span class="p">,</span><span class="n">a0000</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">1</span><span class="p">,</span><span class="n">a0001</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(139, 63)(145, 91)&quot;</span>
<span class="mi">2</span><span class="p">,</span><span class="n">a0002</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(93, 72)(24, 162)(31, 64)&quot;</span>
<span class="mi">3</span><span class="p">,</span><span class="n">a0003</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">4</span><span class="p">,</span><span class="n">a0004</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">5</span><span class="p">,</span><span class="n">a0005</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">6</span><span class="p">,</span><span class="n">a0006</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">7</span><span class="p">,</span><span class="n">a0007</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">8</span><span class="p">,</span><span class="n">a0008</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">9</span><span class="p">,</span><span class="n">a0009</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(127, 80)(104, 60)(87, 63)(53, 78)(17, 126)&quot;</span>
<span class="mi">10</span><span class="p">,</span><span class="n">a0010</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">11</span><span class="p">,</span><span class="n">a0011</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(39, 78)(84, 110)&quot;</span> <span class="o">#</span> <span class="n">a</span> <span class="n">random</span> <span class="k">comment</span> <span class="n">passing</span> <span class="k">by</span>
<span class="o">#</span><span class="k">end</span> <span class="k">of</span> <span class="n">bloc</span> <span class="mi">1</span>
<span class="o">#</span><span class="n">bloc</span> <span class="mi">2</span>
<span class="mi">12</span><span class="p">,</span><span class="n">a0012</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">13</span><span class="p">,</span><span class="n">a0013</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">14</span><span class="p">,</span><span class="n">a0014</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">15</span><span class="p">,</span><span class="n">a0015</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(146, 65)(146, 89)(139, 146)(16, 68)&quot;</span>
<span class="mi">16</span><span class="p">,</span><span class="n">a0016</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(51, 59)(77, 69)(145, 78)(139, 112)(97, 123)(17, 148)&quot;</span>
<span class="mi">17</span><span class="p">,</span><span class="n">a0017</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="o">#</span><span class="k">end</span> <span class="k">of</span> <span class="n">bloc</span> <span class="mi">2</span>
<span class="o">#</span><span class="n">bloc</span> <span class="mi">3</span>
<span class="mi">18</span><span class="p">,</span><span class="n">a0018</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">19</span><span class="p">,</span><span class="n">a0019</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">20</span><span class="p">,</span><span class="n">a0020</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(57, 99)(12, 113)(27, 139)(16, 158)&quot;</span>
<span class="mi">21</span><span class="p">,</span><span class="n">a0021</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">22</span><span class="p">,</span><span class="n">a0022</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">23</span><span class="p">,</span><span class="n">a0023</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">24</span><span class="p">,</span><span class="n">a0024</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">25</span><span class="p">,</span><span class="n">a0025</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">26</span><span class="p">,</span><span class="n">a0026</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="o">#</span><span class="k">end</span> <span class="k">of</span> <span class="n">bloc</span> <span class="mi">3</span>
<span class="mi">27</span><span class="p">,</span><span class="n">a0027</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(11, 86)(29, 74)(92, 68)(109, 129)(132, 104)&quot;</span>
<span class="mi">28</span><span class="p">,</span><span class="n">a0028</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">29</span><span class="p">,</span><span class="n">a0029</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(128, 58)&quot;</span>
<span class="mi">30</span><span class="p">,</span><span class="n">a0030</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(133, 59)(99, 77)(111, 100)(115, 153)&quot;</span>
<span class="mi">31</span><span class="p">,</span><span class="n">a0031</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(43, 154)(27, 177)&quot;</span>
<span class="o">##</span> <span class="n">footer</span> <span class="p">:</span> <span class="k">end</span> <span class="k">of</span> <span class="n">file</span>
</pre></div>
<h2>Anatomy of a SED command</h2>
<p>If we run the command : </p>
<div class="highlight"><pre><span></span>sed <span class="s2">&quot;&quot;</span> input.csv
</pre></div>
<p>Everything inside the brackets will be interpreted as a sed command. In our case, there is nothing hence the file will be printed to the consol without any modification.</p>
<p>You can put in the quotation mark one of sed's many commands for instance <code>s</code> that stands for substitude and is one of the most commonly used.</p>
<p>In our csv we use the comma separator, let's say that we want to change it to a semicolon. We would do :</p>
<div class="highlight"><pre><span></span>sed <span class="s2">&quot;s/,/;/&quot;</span> input.csv
</pre></div>
<p>so we have <code>s</code> meaning that we want to use the replace command followed by a <code>/</code> and the caracter(s) we want to replace followed by a <code>/</code> and the caracter(s) we want to replace it with and finally a <code>/</code>.</p>
<p>The result is the following :</p>
<div class="highlight"><pre><span></span><span class="o">##</span> <span class="n">Header</span>
<span class="o">#</span><span class="n">this</span> <span class="n">file</span> <span class="k">contains</span> <span class="n">information</span> <span class="n">I</span> <span class="n">want</span> <span class="k">to</span> <span class="n">parse</span> <span class="k">with</span> <span class="n">a</span> <span class="k">simple</span> <span class="n">program</span><span class="p">.</span>
<span class="o">#</span><span class="n">The</span> <span class="n">header</span><span class="p">;</span> <span class="n">the</span> <span class="n">footer</span> <span class="k">or</span> <span class="k">any</span> <span class="k">comment</span> <span class="n">starting</span> <span class="k">with</span> <span class="n">a</span> <span class="ss">&quot;#&quot;</span> <span class="n">will</span> <span class="n">be</span> <span class="n">removed</span>
<span class="o">#</span><span class="n">The</span> <span class="n">blank</span> <span class="n">lines</span> <span class="n">will</span> <span class="n">also</span> <span class="n">be</span> <span class="n">removed</span>
<span class="o">#</span><span class="n">img</span><span class="p">;</span><span class="n">processed</span><span class="p">,</span><span class="n">defaut</span>
<span class="o">#</span><span class="n">bloc</span> <span class="mi">1</span>
<span class="mi">0</span><span class="p">;</span><span class="n">a0000</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">1</span><span class="p">;</span><span class="n">a0001</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(139, 63)(145, 91)&quot;</span>
<span class="mi">2</span><span class="p">;</span><span class="n">a0002</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(93, 72)(24, 162)(31, 64)&quot;</span>
<span class="mi">3</span><span class="p">;</span><span class="n">a0003</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">4</span><span class="p">;</span><span class="n">a0004</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">5</span><span class="p">;</span><span class="n">a0005</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">6</span><span class="p">;</span><span class="n">a0006</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">7</span><span class="p">;</span><span class="n">a0007</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">8</span><span class="p">;</span><span class="n">a0008</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">9</span><span class="p">;</span><span class="n">a0009</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(127, 80)(104, 60)(87, 63)(53, 78)(17, 126)&quot;</span>
<span class="mi">10</span><span class="p">;</span><span class="n">a0010</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">11</span><span class="p">;</span><span class="n">a0011</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(39, 78)(84, 110)&quot;</span> <span class="o">#</span> <span class="n">a</span> <span class="n">random</span> <span class="k">comment</span> <span class="n">passing</span> <span class="k">by</span>
<span class="o">#</span><span class="k">end</span> <span class="k">of</span> <span class="n">bloc</span> <span class="mi">1</span>
<span class="o">#</span><span class="n">bloc</span> <span class="mi">2</span>
<span class="mi">12</span><span class="p">;</span><span class="n">a0012</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">13</span><span class="p">;</span><span class="n">a0013</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">14</span><span class="p">;</span><span class="n">a0014</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">15</span><span class="p">;</span><span class="n">a0015</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(146, 65)(146, 89)(139, 146)(16, 68)&quot;</span>
<span class="mi">16</span><span class="p">;</span><span class="n">a0016</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(51, 59)(77, 69)(145, 78)(139, 112)(97, 123)(17, 148)&quot;</span>
<span class="mi">17</span><span class="p">;</span><span class="n">a0017</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="o">#</span><span class="k">end</span> <span class="k">of</span> <span class="n">bloc</span> <span class="mi">2</span>
<span class="o">#</span><span class="n">bloc</span> <span class="mi">3</span>
<span class="mi">18</span><span class="p">;</span><span class="n">a0018</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">19</span><span class="p">;</span><span class="n">a0019</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">20</span><span class="p">;</span><span class="n">a0020</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(57, 99)(12, 113)(27, 139)(16, 158)&quot;</span>
<span class="mi">21</span><span class="p">;</span><span class="n">a0021</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">22</span><span class="p">;</span><span class="n">a0022</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">23</span><span class="p">;</span><span class="n">a0023</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">24</span><span class="p">;</span><span class="n">a0024</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">25</span><span class="p">;</span><span class="n">a0025</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">26</span><span class="p">;</span><span class="n">a0026</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="o">#</span><span class="k">end</span> <span class="k">of</span> <span class="n">bloc</span> <span class="mi">3</span>
<span class="mi">27</span><span class="p">;</span><span class="n">a0027</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(11, 86)(29, 74)(92, 68)(109, 129)(132, 104)&quot;</span>
<span class="mi">28</span><span class="p">;</span><span class="n">a0028</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">29</span><span class="p">;</span><span class="n">a0029</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(128, 58)&quot;</span>
<span class="mi">30</span><span class="p">;</span><span class="n">a0030</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(133, 59)(99, 77)(111, 100)(115, 153)&quot;</span>
<span class="mi">31</span><span class="p">;</span><span class="n">a0031</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(43, 154)(27, 177)&quot;</span>
<span class="o">##</span> <span class="n">footer</span> <span class="p">:</span> <span class="k">end</span> <span class="k">of</span> <span class="n">file</span>
</pre></div>
<p>As you can see, only the first comma has been replaced, in order to repeat the command multiple times per line we need to specify the <code>g</code> option</p>
<div class="highlight"><pre><span></span>sed <span class="s2">&quot;s/,/;/g&quot;</span> input.csv
</pre></div>
<div class="highlight"><pre><span></span><span class="o">##</span> <span class="n">Header</span>
<span class="o">#</span><span class="n">this</span> <span class="n">file</span> <span class="k">contains</span> <span class="n">information</span> <span class="n">I</span> <span class="n">want</span> <span class="k">to</span> <span class="n">parse</span> <span class="k">with</span> <span class="n">a</span> <span class="k">simple</span> <span class="n">program</span><span class="p">.</span>
<span class="o">#</span><span class="n">The</span> <span class="n">header</span><span class="p">;</span> <span class="n">the</span> <span class="n">footer</span> <span class="k">or</span> <span class="k">any</span> <span class="k">comment</span> <span class="n">starting</span> <span class="k">with</span> <span class="n">a</span> <span class="ss">&quot;#&quot;</span> <span class="n">will</span> <span class="n">be</span> <span class="n">removed</span>
<span class="o">#</span><span class="n">The</span> <span class="n">blank</span> <span class="n">lines</span> <span class="n">will</span> <span class="n">also</span> <span class="n">be</span> <span class="n">removed</span>
<span class="o">#</span><span class="n">img</span><span class="p">;</span><span class="n">processed</span><span class="p">;</span><span class="n">defaut</span>
<span class="o">#</span><span class="n">bloc</span> <span class="mi">1</span>
<span class="mi">0</span><span class="p">;</span><span class="n">a0000</span><span class="p">.</span><span class="n">tif</span><span class="p">;;</span>
<span class="mi">1</span><span class="p">;</span><span class="n">a0001</span><span class="p">.</span><span class="n">tif</span><span class="p">;</span><span class="k">True</span><span class="p">;</span><span class="ss">&quot;(139; 63)(145; 91)&quot;</span>
<span class="mi">2</span><span class="p">;</span><span class="n">a0002</span><span class="p">.</span><span class="n">tif</span><span class="p">;</span><span class="k">True</span><span class="p">;</span><span class="ss">&quot;(93; 72)(24; 162)(31; 64)&quot;</span>
<span class="mi">3</span><span class="p">;</span><span class="n">a0003</span><span class="p">.</span><span class="n">tif</span><span class="p">;;</span>
<span class="mi">4</span><span class="p">;</span><span class="n">a0004</span><span class="p">.</span><span class="n">tif</span><span class="p">;;</span>
<span class="mi">5</span><span class="p">;</span><span class="n">a0005</span><span class="p">.</span><span class="n">tif</span><span class="p">;;</span>
<span class="mi">6</span><span class="p">;</span><span class="n">a0006</span><span class="p">.</span><span class="n">tif</span><span class="p">;;</span>
<span class="mi">7</span><span class="p">;</span><span class="n">a0007</span><span class="p">.</span><span class="n">tif</span><span class="p">;;</span>
<span class="mi">8</span><span class="p">;</span><span class="n">a0008</span><span class="p">.</span><span class="n">tif</span><span class="p">;;</span>
<span class="mi">9</span><span class="p">;</span><span class="n">a0009</span><span class="p">.</span><span class="n">tif</span><span class="p">;</span><span class="k">True</span><span class="p">;</span><span class="ss">&quot;(127; 80)(104; 60)(87; 63)(53; 78)(17; 126)&quot;</span>
<span class="mi">10</span><span class="p">;</span><span class="n">a0010</span><span class="p">.</span><span class="n">tif</span><span class="p">;;</span>
<span class="mi">11</span><span class="p">;</span><span class="n">a0011</span><span class="p">.</span><span class="n">tif</span><span class="p">;</span><span class="k">True</span><span class="p">;</span><span class="ss">&quot;(39; 78)(84; 110)&quot;</span> <span class="o">#</span> <span class="n">a</span> <span class="n">random</span> <span class="k">comment</span> <span class="n">passing</span> <span class="k">by</span>
<span class="o">#</span><span class="k">end</span> <span class="k">of</span> <span class="n">bloc</span> <span class="mi">1</span>
<span class="o">#</span><span class="n">bloc</span> <span class="mi">2</span>
<span class="mi">12</span><span class="p">;</span><span class="n">a0012</span><span class="p">.</span><span class="n">tif</span><span class="p">;;</span>
<span class="mi">13</span><span class="p">;</span><span class="n">a0013</span><span class="p">.</span><span class="n">tif</span><span class="p">;;</span>
<span class="mi">14</span><span class="p">;</span><span class="n">a0014</span><span class="p">.</span><span class="n">tif</span><span class="p">;;</span>
<span class="mi">15</span><span class="p">;</span><span class="n">a0015</span><span class="p">.</span><span class="n">tif</span><span class="p">;</span><span class="k">True</span><span class="p">;</span><span class="ss">&quot;(146; 65)(146; 89)(139; 146)(16; 68)&quot;</span>
<span class="mi">16</span><span class="p">;</span><span class="n">a0016</span><span class="p">.</span><span class="n">tif</span><span class="p">;</span><span class="k">True</span><span class="p">;</span><span class="ss">&quot;(51; 59)(77; 69)(145; 78)(139; 112)(97; 123)(17; 148)&quot;</span>
<span class="mi">17</span><span class="p">;</span><span class="n">a0017</span><span class="p">.</span><span class="n">tif</span><span class="p">;;</span>
<span class="o">#</span><span class="k">end</span> <span class="k">of</span> <span class="n">bloc</span> <span class="mi">2</span>
<span class="o">#</span><span class="n">bloc</span> <span class="mi">3</span>
<span class="mi">18</span><span class="p">;</span><span class="n">a0018</span><span class="p">.</span><span class="n">tif</span><span class="p">;;</span>
<span class="mi">19</span><span class="p">;</span><span class="n">a0019</span><span class="p">.</span><span class="n">tif</span><span class="p">;;</span>
<span class="mi">20</span><span class="p">;</span><span class="n">a0020</span><span class="p">.</span><span class="n">tif</span><span class="p">;</span><span class="k">True</span><span class="p">;</span><span class="ss">&quot;(57; 99)(12; 113)(27; 139)(16; 158)&quot;</span>
<span class="mi">21</span><span class="p">;</span><span class="n">a0021</span><span class="p">.</span><span class="n">tif</span><span class="p">;;</span>
<span class="mi">22</span><span class="p">;</span><span class="n">a0022</span><span class="p">.</span><span class="n">tif</span><span class="p">;;</span>
<span class="mi">23</span><span class="p">;</span><span class="n">a0023</span><span class="p">.</span><span class="n">tif</span><span class="p">;;</span>
<span class="mi">24</span><span class="p">;</span><span class="n">a0024</span><span class="p">.</span><span class="n">tif</span><span class="p">;;</span>
<span class="mi">25</span><span class="p">;</span><span class="n">a0025</span><span class="p">.</span><span class="n">tif</span><span class="p">;;</span>
<span class="mi">26</span><span class="p">;</span><span class="n">a0026</span><span class="p">.</span><span class="n">tif</span><span class="p">;;</span>
<span class="o">#</span><span class="k">end</span> <span class="k">of</span> <span class="n">bloc</span> <span class="mi">3</span>
<span class="mi">27</span><span class="p">;</span><span class="n">a0027</span><span class="p">.</span><span class="n">tif</span><span class="p">;</span><span class="k">True</span><span class="p">;</span><span class="ss">&quot;(11; 86)(29; 74)(92; 68)(109; 129)(132; 104)&quot;</span>
<span class="mi">28</span><span class="p">;</span><span class="n">a0028</span><span class="p">.</span><span class="n">tif</span><span class="p">;;</span>
<span class="mi">29</span><span class="p">;</span><span class="n">a0029</span><span class="p">.</span><span class="n">tif</span><span class="p">;</span><span class="k">True</span><span class="p">;</span><span class="ss">&quot;(128; 58)&quot;</span>
<span class="mi">30</span><span class="p">;</span><span class="n">a0030</span><span class="p">.</span><span class="n">tif</span><span class="p">;</span><span class="k">True</span><span class="p">;</span><span class="ss">&quot;(133; 59)(99; 77)(111; 100)(115; 153)&quot;</span>
<span class="mi">31</span><span class="p">;</span><span class="n">a0031</span><span class="p">.</span><span class="n">tif</span><span class="p">;</span><span class="k">True</span><span class="p">;</span><span class="ss">&quot;(43; 154)(27; 177)&quot;</span>
<span class="o">##</span> <span class="n">footer</span> <span class="p">:</span> <span class="k">end</span> <span class="k">of</span> <span class="n">file</span>
</pre></div>
<h2>Removing comments</h2>
<p>In order to remove comments, we can replace the pattern of a comment by nothing. A commant starts with a # sign and is followed by an arbitrarly long chain of characters. In order to match this pattern, we will use a regular expression.</p>
<div class="highlight"><pre><span></span>sed <span class="s2">&quot;s/#.*//g&quot;</span> input.csv
</pre></div>
<p><code>#.*</code> means : find strings that start with a <code>#</code>, the <code>.</code> stands for any character, finally, the <code>*</code> means that the <code>.</code> can be repeated any number of times. That means that <code>sed</code> will look for a string starting with a # followed by any characters that come after in the line.</p>
<p>If we want to do something a bit cleaner, we can try to remove any whitespace before the comments as well. In order to do so, we will use the <code>\s</code> regular expression that represents a whitespace. If we want to make sure that we removed any whitespace before the comments, we will do <code>\s*</code></p>
<p>The final regular expression is then <code>\s*#.*</code> and it will then be replaced by nothing. </p>
<p>Finally, the command becomes </p>
<div class="highlight"><pre><span></span>sed <span class="s2">&quot;s/\s*#.*//g&quot;</span> input.csv
</pre></div>
<p>If we run that command, all our comments have disappeared </p>
<div class="highlight"><pre><span></span><span class="mi">0</span><span class="p">,</span><span class="n">a0000</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">1</span><span class="p">,</span><span class="n">a0001</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(139, 63)(145, 91)&quot;</span>
<span class="mi">2</span><span class="p">,</span><span class="n">a0002</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(93, 72)(24, 162)(31, 64)&quot;</span>
<span class="mi">3</span><span class="p">,</span><span class="n">a0003</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">4</span><span class="p">,</span><span class="n">a0004</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">5</span><span class="p">,</span><span class="n">a0005</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">6</span><span class="p">,</span><span class="n">a0006</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">7</span><span class="p">,</span><span class="n">a0007</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">8</span><span class="p">,</span><span class="n">a0008</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">9</span><span class="p">,</span><span class="n">a0009</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(127, 80)(104, 60)(87, 63)(53, 78)(17, 126)&quot;</span>
<span class="mi">10</span><span class="p">,</span><span class="n">a0010</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">11</span><span class="p">,</span><span class="n">a0011</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(39, 78)(84, 110)&quot;</span>
<span class="mi">12</span><span class="p">,</span><span class="n">a0012</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">13</span><span class="p">,</span><span class="n">a0013</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">14</span><span class="p">,</span><span class="n">a0014</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">15</span><span class="p">,</span><span class="n">a0015</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(146, 65)(146, 89)(139, 146)(16, 68)&quot;</span>
<span class="mi">16</span><span class="p">,</span><span class="n">a0016</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(51, 59)(77, 69)(145, 78)(139, 112)(97, 123)(17, 148)&quot;</span>
<span class="mi">17</span><span class="p">,</span><span class="n">a0017</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">18</span><span class="p">,</span><span class="n">a0018</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">19</span><span class="p">,</span><span class="n">a0019</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">20</span><span class="p">,</span><span class="n">a0020</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(57, 99)(12, 113)(27, 139)(16, 158)&quot;</span>
<span class="mi">21</span><span class="p">,</span><span class="n">a0021</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">22</span><span class="p">,</span><span class="n">a0022</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">23</span><span class="p">,</span><span class="n">a0023</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">24</span><span class="p">,</span><span class="n">a0024</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">25</span><span class="p">,</span><span class="n">a0025</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">26</span><span class="p">,</span><span class="n">a0026</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">27</span><span class="p">,</span><span class="n">a0027</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(11, 86)(29, 74)(92, 68)(109, 129)(132, 104)&quot;</span>
<span class="mi">28</span><span class="p">,</span><span class="n">a0028</span><span class="p">.</span><span class="n">tif</span><span class="p">,,</span>
<span class="mi">29</span><span class="p">,</span><span class="n">a0029</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(128, 58)&quot;</span>
<span class="mi">30</span><span class="p">,</span><span class="n">a0030</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(133, 59)(99, 77)(111, 100)(115, 153)&quot;</span>
<span class="mi">31</span><span class="p">,</span><span class="n">a0031</span><span class="p">.</span><span class="n">tif</span><span class="p">,</span><span class="k">True</span><span class="p">,</span><span class="ss">&quot;(43, 154)(27, 177)&quot;</span>
</pre></div>
<h2>Removing blank lines</h2>
<p>In order to remove the blank lines, we need to specify to <code>sed</code> a pattern corresponding to a blank line and use the <code>d</code> command where the <code>d</code> stands for delete. The delete command expects the pattern to the between /</p>
<p>Now let's define what pattern a blank line corresponds to. Since there is no symbol for blankness, we can do the following <code>^$</code>. <code>^</code> means the begining of a line and <code>$</code> corresponds to the end of a line.</p>
<p>So whenever we find a blank line we delete it. </p>
<div class="highlight"><pre><span></span>sed <span class="s2">&quot;/^</span>$<span class="s2">/ d&quot;</span> input.csv
</pre></div>
<div class="highlight"><pre><span></span><span class="err">## Header</span>
<span class="err">#this file contains information I want to parse with a simple program.</span>
<span class="err">#The header, the footer or any comment starting with a &quot;#&quot; will be removed</span>
<span class="err">#The blank lines will also be removed</span>
<span class="err">#img,processed,defaut</span>
<span class="err">#bloc 1</span>
<span class="err">0,a0000.tif,,</span>
<span class="err">1,a0001.tif,True,&quot;(139, 63)(145, 91)&quot;</span>
<span class="err">2,a0002.tif,True,&quot;(93, 72)(24, 162)(31, 64)&quot;</span>
<span class="err">3,a0003.tif,,</span>
<span class="err">4,a0004.tif,,</span>
<span class="err">5,a0005.tif,,</span>
<span class="err">6,a0006.tif,,</span>
<span class="err">7,a0007.tif,,</span>
<span class="err">8,a0008.tif,,</span>
<span class="err">9,a0009.tif,True,&quot;(127, 80)(104, 60)(87, 63)(53, 78)(17, 126)&quot;</span>
<span class="err">10,a0010.tif,,</span>
<span class="err">11,a0011.tif,True,&quot;(39, 78)(84, 110)&quot; # a random comment passing by</span>
<span class="err">#end of bloc 1</span>
<span class="err">#bloc 2</span>
<span class="err">12,a0012.tif,,</span>
<span class="err">13,a0013.tif,,</span>
<span class="err">14,a0014.tif,,</span>
<span class="err">15,a0015.tif,True,&quot;(146, 65)(146, 89)(139, 146)(16, 68)&quot;</span>
<span class="err">16,a0016.tif,True,&quot;(51, 59)(77, 69)(145, 78)(139, 112)(97, 123)(17, 148)&quot;</span>
<span class="err">17,a0017.tif,,</span>
<span class="err">#end of bloc 2</span>
<span class="err">#bloc 3</span>
<span class="err">18,a0018.tif,,</span>
<span class="err">19,a0019.tif,,</span>
<span class="err">20,a0020.tif,True,&quot;(57, 99)(12, 113)(27, 139)(16, 158)&quot;</span>
<span class="err">21,a0021.tif,,</span>
<span class="err">22,a0022.tif,,</span>
<span class="err">23,a0023.tif,,</span>
<span class="err">24,a0024.tif,,</span>
<span class="err">25,a0025.tif,,</span>
<span class="err">26,a0026.tif,,</span>
<span class="err">#end of bloc 3</span>
<span class="err">27,a0027.tif,True,&quot;(11, 86)(29, 74)(92, 68)(109, 129)(132, 104)&quot;</span>
<span class="err">28,a0028.tif,,</span>
<span class="err">29,a0029.tif,True,&quot;(128, 58)&quot;</span>
<span class="err">30,a0030.tif,True,&quot;(133, 59)(99, 77)(111, 100)(115, 153)&quot;</span>
<span class="err">31,a0031.tif,True,&quot;(43, 154)(27, 177)&quot;</span>
<span class="err">## footer : end of file</span>
</pre></div>
<h3>combining the sed commands</h3>
<p>We can concatenate sed commands by separating them with a semicolon. Hence the final <code>sed</code> command will be :</p>
<div class="highlight"><pre><span></span>sed <span class="s2">&quot;s/\s*#.*//g;/^</span>$<span class="s2">/ d&quot;</span> input.csv
</pre></div>
<p>and the final output is </p>
<div class="highlight"><pre><span></span><span class="err">0,a0000.tif,,</span>
<span class="err">1,a0001.tif,True,&quot;(139, 63)(145, 91)&quot;</span>
<span class="err">2,a0002.tif,True,&quot;(93, 72)(24, 162)(31, 64)&quot;</span>
<span class="err">3,a0003.tif,,</span>
<span class="err">4,a0004.tif,,</span>
<span class="err">5,a0005.tif,,</span>
<span class="err">6,a0006.tif,,</span>
<span class="err">7,a0007.tif,,</span>
<span class="err">8,a0008.tif,,</span>
<span class="err">9,a0009.tif,True,&quot;(127, 80)(104, 60)(87, 63)(53, 78)(17, 126)&quot;</span>
<span class="err">10,a0010.tif,,</span>
<span class="err">11,a0011.tif,True,&quot;(39, 78)(84, 110)&quot;</span>
<span class="err">12,a0012.tif,,</span>
<span class="err">13,a0013.tif,,</span>
<span class="err">14,a0014.tif,,</span>
<span class="err">15,a0015.tif,True,&quot;(146, 65)(146, 89)(139, 146)(16, 68)&quot;</span>
<span class="err">16,a0016.tif,True,&quot;(51, 59)(77, 69)(145, 78)(139, 112)(97, 123)(17, 148)&quot;</span>
<span class="err">17,a0017.tif,,</span>
<span class="err">18,a0018.tif,,</span>
<span class="err">19,a0019.tif,,</span>
<span class="err">20,a0020.tif,True,&quot;(57, 99)(12, 113)(27, 139)(16, 158)&quot;</span>
<span class="err">21,a0021.tif,,</span>
<span class="err">22,a0022.tif,,</span>
<span class="err">23,a0023.tif,,</span>
<span class="err">24,a0024.tif,,</span>
<span class="err">25,a0025.tif,,</span>
<span class="err">26,a0026.tif,,</span>
<span class="err">27,a0027.tif,True,&quot;(11, 86)(29, 74)(92, 68)(109, 129)(132, 104)&quot;</span>
<span class="err">28,a0028.tif,,</span>
<span class="err">29,a0029.tif,True,&quot;(128, 58)&quot;</span>
<span class="err">30,a0030.tif,True,&quot;(133, 59)(99, 77)(111, 100)(115, 153)&quot;</span>
<span class="err">31,a0031.tif,True,&quot;(43, 154)(27, 177)&quot;</span>
</pre></div>
<h3>Overwritting the file</h3>
<p>If we want to overwrite the result of <code>sed</code> on the input file, we need to add the option <code>-i</code>. So if you are running <code>sed</code> without <code>-i</code>, it is safe and won't alter your files.</p>
<div class="highlight"><pre><span></span>sed -i <span class="s2">&quot;s/\s*#.*//g;/^</span>$<span class="s2">/ d&quot;</span> input.csv
</pre></div>
</div>
<aside>
<div class="bug-reporting__panel">
<h3>Find an error or bug? Have a suggestion?</h3>
<p>Everything on this site is avaliable on GitHub. Head on over and <a href='https://github.com/redoules/redoules.github.io/issues/new'>submit an issue.</a> You can also message me directly by <a href='mailto:guillaume.redoules@gadz.org'>email</a>.</p>
</div>
</aside>
</section>
</div>
<!-- start of footer section -->
<footer class="footer">
<div class="container">
<p class="text-muted">
<center>This project contains 119 pages and is available on <a href="https://github.com/redoules/redoules.github.io">GitHub</a>.
<br/>
Copyright &copy; Guillaume Redoulès,
<time datetime="2018">2018</time>.
</center>
</p>
</div>
</footer>
<!-- This jQuery line finds any span that contains code highlighting classes and then selects the parent <pre> tag and adds a border. This is done as a workaround to visually distinguish the code inputs and outputs -->
<script>
$( ".hll, .n, .c, .err, .k, .o, .cm, .cp, .c1, .cs, .gd, .ge, .gr, .gh, .gi, .go, .gp, .gs, .gu, .gt, .kc, .kd, .kn, .kp, .kr, .kt, .m, .s, .na, .nb, .nc, .no, .nd, .ni, .ne, .nf, .nl, .nn, .nt, .nv, .ow, .w, .mf, .mh, .mi, .mo, .sb, .sc, .sd, .s2, .se, .sh, .si, .sx, .sr, .s1, .ss, .bp, .vc, .vg, .vi, .il" ).parent( "pre" ).css( "border", "1px solid #DEDEDE" );
</script>
<!-- Load Google Analytics -->
<script>
/*
(function(i, s, o, g, r, a, m) {
i['GoogleAnalyticsObject'] = r;
i[r] = i[r] || function() {
(i[r].q = i[r].q || []).push(arguments)
}, i[r].l = 1 * new Date();
a = s.createElement(o),
m = s.getElementsByTagName(o)[0];
a.async = 1;
a.src = g;
m.parentNode.insertBefore(a, m)
})(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga');
ga('create', 'UA-66582-32', 'auto');
ga('send', 'pageview');
*/
</script>
<!-- End of Google Analytics -->
<!-- Bootstrap core JavaScript
================================================== -->
<!-- Placed at the end of the document so the pages load faster -->
<script src="../theme/js/bootstrap.min.js"></script>
<!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
<script src="../theme/js/ie10-viewport-bug-workaround.js"></script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink