Jekyll2020-02-25T10:49:02+00:00http://dotkomblog.com/feed.xmlDotkomAdventures in data, tech and investingHow I solved my own problem with code2020-02-24T00:00:00+00:002020-02-24T00:00:00+00:00http://dotkomblog.com/coding/2020/02/24/trello-power-up<p><em>It’s been a while since I wrote anything here. Two years. I guess it’s not a coincidence that my son is turning two in a few weeks…</em></p>
<p>In 2020 I completely surprised myself by actually finishing one of my numerous pet projects: I wrote <a href="https://trello.com/power-ups/5e14aa34ce4580518945bf9b/text-to-cards">a Trello Power-Up</a> called Text to Cards. This is my first publicly available “software product” ever. And the greatest thing: I solved a problem for myself. Exciting times.</p>
<p>(<em>“What’s a Power-Up?” - you might ask. Simply put these are sort of “plugins” on Trello, that enhance the platform by adding some new functionality or introducing integrations with other services. Read more <a href="https://help.trello.com/article/1094-what-are-power-ups">here</a>.</em></p>
<p><em>“What’s Trello?” - you might also ask. In that case please checkout <a href="https://trello.com">Trello.com</a></em>)</p>
<h2 id="what-is-text-to-cards">What is Text to Cards?</h2>
<blockquote>
<p><a href="https://trello.com/power-ups/5e14aa34ce4580518945bf9b/text-to-cards">Text to Cards</a> is a simple Power-Up for Trello that helps transforming your longform texts into neat little Trello cards. It will extract card names and descriptions from your text and add the proper labels, members and due date to the card. <br /><br /> You can create dozens of cards with just one click. (<a href="https://somiandras.gitbook.io/text-to-cards/">Text to Cards docs</a>)</p>
</blockquote>
<p>It’s not the most revolutionary software ever written in the history of mankind, but probably a neat tool for someone having the same issue as me: trying to create multiple cards on Trello from a single, long(ish) text without a hundred rounds of copy-pasting and clicking around.</p>
<p>The idea came back in the days when I was managing my team at KBC - I even started to code up a prototype 4 years ago -, where during our regular weekly discussions we spent disproportionately long time with trying to enter the right thing to the right part of the right cards. This can completely derail an otherwise productive discussion.</p>
<p>Nowadays I tend to spend quite some time on calls with people I work with, and I usually just type up a few notes in my text editor to stay in the flow. You guessed it: most of the time these notes end up on my Trello boards as cards.</p>
<h2 id="how-does-it-work">How does it work?</h2>
<p>Instead of adding these cards one by one now I can simply copy my notes in the Power-Up, spice it up with a few “hints” to mark the cards (less than 30 seconds of extra work), and <em>boom</em>, my cards are ready.</p>
<p>And there is even a live preview of the soon-to-be created cards:</p>
<p><img src="https://text-to-cards.netlify.com/preview.png" alt="screenshot" /></p>
<p>Text to Cards uses the following hints to extract cards:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">::card title</code> to find cards and separate their title from description</li>
<li><code class="language-plaintext highlighter-rouge">@username</code> or <code class="language-plaintext highlighter-rouge">@initials</code> for adding members</li>
<li><code class="language-plaintext highlighter-rouge">#label</code> for adding labels</li>
<li><code class="language-plaintext highlighter-rouge">$due: YYYY-MM-DD</code> for adding a due date</li>
</ul>
<p>So this - completely made up - meeting memo:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>::Create docs for the power-up
@andrassomi will set up a Gitbook site with a few simple pages
to document the power-up. $due: 2020-02-10
#Docs #Product
::Add board members and available labels to main window
@andrassomi - Board members and labels should appear somewhere on the #UI to make text editing easier.
</code></pre></div></div>
<p>will become these cards:</p>
<p><img src="https://blobs.gitbook.com/assets%2F-M--b8JLQtIxcG9fkDVz%2F-M-6YoNcXC09W8cXUXcp%2F-M-1V0TK1lWz_kgJnuiu%2Fscreenshot1.png?generation=1580674797272740&alt=media" alt="screenshot of cards" /></p>
<p>I even created a small documentation page, <a href="https://somiandras.gitbook.io/text-to-cards/">check it out for the detailed functionality</a>.</p>
<h2 id="how-did-i-create-the-power-up">How did I create the Power-Up?</h2>
<p>While I use Python all the time in my work and I even tend to consider myself a more experienced Pythonista, this Power-Up project is fully client-side Javascript (Power-Ups are just websites that Trello loads in an iframe).</p>
<p>So I had to make myself somewhat familiar (again) with the extremely confusing modern JS ecosystem and some other interesting tools:</p>
<ul>
<li><strong>webpack</strong>: much nicer than my last memories from a few years earlier, but I still wasn’t always sure I knew what I was doing (but hey, it just worked!).</li>
<li><strong>Vue.js</strong>: still comfortably easy, but for me (the outsider) the ecosystem is starting to feel almost as intimidating as other giant frameworks which I deliberately avoid now.</li>
<li><strong>jest</strong>: I think this is my first project where tests became really relevant and increased my productivity (not sure it has anything to do with the test runner, or I just happened to test the right things).</li>
<li><strong>Netlify</strong>: The greatest and most pleasant surprise of all. This service - even in the free tier - is amazing, it just builds and deploys my project straight from the Github repo, with (almost) zero configuration.</li>
<li><strong>ngrok</strong>: Another magical tool, which exposes my local development server to a proper external url. It was important as I did not have any other means to try out Trello functionality, but to actually look at the Power-Up on Trello. Without ngrok it means committing and deploying the code you want to try out (not nice).</li>
<li>(And let’s not forget <strong>VS Code</strong>, my go-to code editor which served me extremely well during this experiment as well.)</li>
</ul>
<p>I’m not telling you there wasn’t any frustration along the way, but I still somehow managed to create this little piece software (and plan to further work on it, especially if someone actually starts using it).</p>
<p>Check out the <a href="https://trello.com/power-ups/5e14aa34ce4580518945bf9b/text-to-cards">Power-Up on Trello</a>.</p>
<h2 id="more-to-come">More to come?</h2>
<p>I enjoyed this project so much. Maybe 2020 will be the year of actually putting some other stuff out there. We’ll see.</p>
<hr />
<p><em><strong>Disclaimer:</strong> Just to make sure it’s clear for everyone: I am not affiliated with Trello in any form.</em></p>Somi AndrásIt’s been a while since I wrote anything here. Two years. I guess it’s not a coincidence that my son is turning two in a few weeks…The holidays bitcoin doom in perspective2018-01-02T00:00:00+00:002018-01-02T00:00:00+00:00http://dotkomblog.com/2018/01/02/the-bitcoin-doom-in-perspective<p>Only a few days after <a href="https://dotkomblog.com/investing/2017/12/17/jumping-off-the-bitcoin-rocket/">my previous post</a> I started getting congratulations for pinpointing the top of the crypto bubble. I was surprised partially because I rarely get direct feedback but mostly because some of this came from people whom I never thought would be interested in cryptocurrencies.</p>
<p>Yet, they reacted almost immediately to the drop that came after, which is not just a coincidence with the financial and mainstream media going nuts on the <em>’freefall’</em> of bitcoin. So I got curious: was this drop really something noteworthy?</p>
<p><strong>Spoiler alert:</strong> it was not, or at least not in the context of the insane crypto universe.</p>
<p>On the charts below you can see the drawdowns of bitcoin, ethereum and litecoin from the past three years. The main point: even though bitcoin dropped few percentage point deeper in December than the previous throughs, it is still in the range of <em>’business as usual’</em>. It can easily get worse but it does not qualify as a collapse yet.</p>
<p><em>(For the uninitiated: drawdown is the relative price decrease from the previous known peak in any given moment, so it’s mostly negative and equals zero only when the price itself hits new highs.)</em></p>
<p><img src="/img/posts/btc_drawdown.png" alt="" /></p>
<p>I obtained (free) Bitfinex price data via the <a href="https://www.quandl.com/data/BITFINEX-Bitfinex">Quandl API</a>. It only starts from late 2014, except for ethereum, which starts from early 2016 (hence the flat line on that chart). I adjusted the starting point to Jan 13, 2015, when bitcoin hit the low after <em>falling almost 90 percent</em> from 2013 peaks, to only include drawdowns of the extreme, 60-fold rise in the past years.</p>
<p>Few things that pop up in this story:</p>
<h3 id="1-never-listen-to-the-media">1. Never listen to the media</h3>
<p>It’s just an amplifier of your own beliefs and fears, run by people who are just as clueless as anyone else but play it out on <em>your</em> psychological blindspots.</p>
<h3 id="2-always-think-in-relative-terms">2. Always think in relative terms</h3>
<p>And compare current things to events that already happened. In this case I looked at 3 years of history, which is very short, but given the intensity of the bitcoin story, it can already provide some sobering context.</p>
<h3 id="3-never-predict-markets">3. Never predict markets</h3>
<p>And especially don’t trust anyone else’s predictions, no matter how much your psyche craves for being told what’s going to happen. Most <em>’experts’</em> try to make the <em>Big Prediction That Becomes True and Makes Me Famous</em> (I was guilty in this, too!), and all the financial and psychological incentives are set up to support this. (I wrote about <a href="https://dotkomblog.com/life/2017/10/08/why-data-will-replace-experts/">the pitfalls of experts here</a>.)</p>
<p>It’s amusing that I published my post exactly the day when bitcoin took a turn, but it’s still just a funny accident, nothing more. I have no idea what’s going to happen. I wouldn’t be surprised neither if we saw even higher peaks soon nor if the situation escalated into a real collapse. I’m still leaving the party anyways.</p>Somi AndrásOnly a few days after my previous post I started getting congratulations for pinpointing the top of the crypto bubble. I was surprised partially because I rarely get direct feedback but mostly because some of this came from people whom I never thought would be interested in cryptocurrencies.Jumping off the bitcoin rocket2017-12-17T00:00:00+00:002017-12-17T00:00:00+00:00http://dotkomblog.com/investing/2017/12/17/jumping-off-the-bitcoin-rocket<p>Whatever you try to explain around bitcoin and crypto currencies, from the price and the total market value (over <a href="https://blockchain.info/charts/market-cap">USD 300 billion</a> at this moment) through the energy consumption to the number of new <em>’investors’</em>, you will quickly end up with an exponential <em>’hockey stick’</em> chart.</p>
<p><img src="/img/posts/btcusd.png" alt="" /></p>
<p>These trends have two peculiar features: they are not likely to last, and our mind completely fails us when trying to predict them. I don’t know what and when will happen, but for me time has come to gradually pull out most of what I have in cryptos, and retain only a fraction that I can be comfortable with losing completely.</p>
<p>This is where I stand now:</p>
<p><img src="/img/posts/Screenshot 2017-12-17 17.03.10.png" alt="" /></p>
<p>(Mind you, I never considered myself as a crypto investor, and I advise you against looking at it as an investment at all, no matter what the price is. I was just <a href="https://dotkomblog.com/investing/2017/08/08/how-i-tricked-myself-into-cryptocurrencies/">lucky to find myself in this market</a> out of sheer curiosity and a healthy dose of laziness.)</p>
<h2 id="energy-is-the-key">Energy is the key</h2>
<p>The price itself is the least scary of the hockey sticks. You can add a few zeros to the price of an imaginary asset, but scaling energy supply is a different matter. And the energy consumption of the bitcoin blockchain network <em>roughly doubled</em> since August, when <a href="https://dotkomblog.com/blockchain/2017/08/30/real-price-of-bitcoin/">I previously wrote</a> about the environmental danger of the cryptocurrencies in their current form.</p>
<p><img src="/img/posts/Screenshot 2017-12-17 16.07.50.png" alt="" /></p>
<p>(According to Digiconomist by now the network already exceeded Denmark in terms of energy consumption, and half of the Czech Republic or quarter of the Netherlands could be served by the energy spent on bitcoin.)</p>
<p>Let’s assume that the four month growth was only one fifth of the Digiconomist estimate. That would still mean over 70 percent annual growth in energy consumption, or fivefold growth in 3 years time and 14-fold in 5 years. If the consumption really did double in 4 months, there’s no point in calculating for years ahead.</p>
<h2 id="the-feedback-loop">The feedback loop</h2>
<p>Mining is profitable, but only if energy costs are relatively low and the price of the reward goes higher and higher (as the amount of reward decreases over time). You can have a hefty margin if you’re mining on subsidized energy in China, the price increases in an exponential fashion, and you invest in new equipment in extreme pace.</p>
<p>But none of these seem to be scalable on rational terms in the long run, and energy will be an issue sooner than later. On top of that, the price is not a function of mining costs, instead it depends on the demand from the crowds with serious condition of <a href="https://en.wikipedia.org/wiki/Fear_of_missing_out"><em>FOMO</em></a> and crave for investment returns after years of zero yields.</p>
<p>Just an anecdotal evidence: the app of Coinbase, the widely known platform for buying and selling bitcoin, hit the top spot as the <a href="https://techcrunch.com/2017/12/07/coinbase-hits-top-spot-on-apples-us-app-store/">most downloaded free app in Apple’s App Store</a>. Coinbase also warned this week that their servers could not keep up with the number of new account openers trying to identify themselves.</p>
<p>So I think we have a self-amplifying feedback loop, and these rarely end well from an investment perspective, even though they can operate unexpectedly long, and bring stellar short-term gains before collapsing.</p>Somi AndrásWhatever you try to explain around bitcoin and crypto currencies, from the price and the total market value (over USD 300 billion at this moment) through the energy consumption to the number of new ’investors’, you will quickly end up with an exponential ’hockey stick’ chart.Is Amazon really as trustworthy as your bank?2017-11-14T00:00:00+00:002017-11-14T00:00:00+00:00http://dotkomblog.com/data/2017/11/14/is-amazon-more-trustworthy<p>The Verge published an <a href="https://www.theverge.com/2017/10/27/16550640/verge-tech-survey-amazon-facebook-google-twitter-popularity">interesting survey</a> a few weeks ago about the popularity of the big tech companies and the level of trust towards them. The topic is really exciting, especially when one of the top charts states no less than <em>’People trust Amazon almost as much as their bank’</em>.</p>
<p><img src="https://cdn.vox-cdn.com/thumbor/5AwKVJmieBV5qP38fMSQ3_CwNAo=/1200x0/filters:no_upscale()/cdn.vox-cdn.com/uploads/chorus_asset/file/9547237/Amazon_01_01.jpg" alt="" /></p>
<p>I know from personal experience this is the type of question that keeps banking executives awake during the night. Unfortunatelly the conclusion here is just as useless as it is bold, and suffers heavily from a handful of biases.</p>
<h2 id="issue-1-distorted-visualisation">Issue 1: Distorted visualisation</h2>
<p>First, this visualisation shows nothing of the message in the title. The only information this chart can clearly convey is that banks had the highest rate of people who <em>did not contribute</em> an answer (the rightmost yellow bars). All the other bars are disaligned and distorted by these meaningless answers and there are no labels to the rescue either.</p>
<p>I tried to decode the percentages by taking the pixel width of the bars (for the sake of simplicity I counted <em>’Greatly’/‘Somewhat distrust’</em> categories together). It turns out that actually Microsoft had the highest rate of <em>’Greatly trust’</em> answers in the survey, and Google seem to be ahead of Amazon in this aspect, too.</p>
<p><img src="/img/posts/Screenshot 2017-11-14 10.32.52.png" alt="" /></p>
<p>If we adjust the percentages for the people who did not have opinion, the actual ratios of (usable) opinions would look something like this:</p>
<p><img src="/img/posts/Dashboard 1.png" alt="" /></p>
<p>Amazon, Microsoft and Google has almost the same rate of trust (53%, 52%, 52%, respectively) amongst people who can decide whether they trust them or not, and they are visibly behind banks (58%) in this respect. Mind you, this adjusment has a tricky bias too, as it might selectively exclude some (but not all) answers of certain individuals, and would also decrease sample size (which in turn increases uncertainty).</p>
<h2 id="issue-2-too-much-noise-too-weak-signal">Issue 2: Too much noise, too weak signal</h2>
<p>Uncertainty is a nasty beast, that eats survey results for breakfast. No matter how we try to fix the visualisaton, these percentages really mean nothing if we take into account the uncertainty of the survey (emphasis is mine):</p>
<blockquote>
<p>This survey, conducted from September 28th to October 10th, included 1,520 people nationally representative of the US, based on 2016 US Census estimates. <strong>The margin of error is ±3 percent</strong>, with a confidence level of 95 percent.</p>
</blockquote>
<p>So the differences between the original percentages for Microsoft, Google and Amazon all fall in the margin of error, and things get even blurrier with my rudimentary adjustments, as I could easily miscalculated by as much as half or even one percentage point.</p>
<p>This all means one thing: based on this survey we actually <em>don’t know</em> which of Amazon, Google, Microsoft or banks in general has the highest level of trust. The numbers for these companies are <em>virtually indistinguishable</em> from each other.</p>
<h2 id="issue-3-confirmation-bias">Issue 3: Confirmation bias</h2>
<p>But for some reason the creator of the chart still wanted to single out Amazon as a contender for banks. Not Microsoft and not Google, even though they are likely in the same <em>’league of trust’</em>. Why?</p>
<p>We will never know for sure, but maybe because in the breakout report concerning <a href="https://www.theverge.com/2017/10/27/16552614/amazon-popularity-user-survey-prime-echo-trust">Amazon results</a> also seem to deliver some positive messages on the company, and this chart had to fit in, even by misrepresenting the facts (not necessarily on purpose, this is just how our mind works).</p>
<h2 id="do-not-trust-surveys">Do not trust surveys</h2>
<p>The conclusion here is not that trust towards Amazon is not comparable to banks and not the opposite either. I don’t know anything about that.</p>
<p>The takeaway is that we should not trust surveys packing blockbuster conclusions into distorted charts, based on questioning a handful of people on some mostly unconscious feelings.</p>Somi AndrásThe Verge published an interesting survey a few weeks ago about the popularity of the big tech companies and the level of trust towards them. The topic is really exciting, especially when one of the top charts states no less than ’People trust Amazon almost as much as their bank’.Looking for a good read?2017-10-30T00:00:00+00:002017-10-30T00:00:00+00:00http://dotkomblog.com/reading/2017/10/30/reading-list-so-far<p>I had <a href="https://dotkomblog.com/life/2017/07/18/changes/">a pledge in July</a> to read as much as I can besides my studies and other activities. More precisely, I aimed for one book per week, <em>’serious’</em> and entertaining ones inlcuded. It’s been almost 4 months now, so I assessed where I stand with this plan.</p>
<p>It turns out I scored around 70% on my goal, as I fully read 12 books in about 17 weeks. That translates to roughly one book in 10 days on average, not counting the ones I haven’t finished yet or completely abandoned. It’s not that far from my initial target, so I’ll just try to keep the pace.</p>
<p>It’s much more important that I found some true gems that I would happily recommend to anyone. I was lucky to pick great books in investments and data, and much less lucky with sci-fi and fantasy books for leisure time. The outlier is <strong>Never Split the Difference</strong>, which ended up as one of my favourites this summer. It’s a practical eye-opener on key elements of negotiations in our everyday life.</p>
<h2 id="the-full-list">The full list</h2>
<p>So here’s my full list from the past 4 months. I won’t review the books, but feel free to ask about any of them, or let me know, what do you think about the ones you already read.</p>
<p><em>(The scoring is completely subjective on a scale from 1 to 10. It reflects how satisfied I was with the book in general, taking into account the relevance, the novelty of the ideas, readability, plot, style, etc. 8 or above means I enjoyed and recommend it, less than 5 means it is not worth the time. With the trilogies I followed this logic: if I ended up reading the sequel, the book scores at least 5, even if I lost interest or was utterly disappointed later.)</em></p>
<h3 id="finished">Finished</h3>
<table>
<thead>
<tr>
<th style="text-align: left">Title</th>
<th style="text-align: center">Author</th>
<th style="text-align: center">Topic</th>
<th style="text-align: center">Score</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left">What Works on Wall Street: The Classic Guide to the Best-Performing Investment Strategies of All Time</td>
<td style="text-align: center">James O’Shaughnessy</td>
<td style="text-align: center">investments</td>
<td style="text-align: center">9</td>
</tr>
<tr>
<td style="text-align: left">Your Complete Guide to Factor-Based Investing: The Way Smart Money Invests Today</td>
<td style="text-align: center">Andrew L. Berkin, Larry E. Swedroe</td>
<td style="text-align: center">investments</td>
<td style="text-align: center">10</td>
</tr>
<tr>
<td style="text-align: left">Fortune’s Formula: The Untold Story of the Scientific Betting System That Beat the Casinos and Wall Street investments</td>
<td style="text-align: center">William Poundstone</td>
<td style="text-align: center">investments</td>
<td style="text-align: center"> 8</td>
</tr>
<tr>
<td style="text-align: left">Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are</td>
<td style="text-align: center">Seth Stephens-Davidowitz</td>
<td style="text-align: center">data science</td>
<td style="text-align: center">9</td>
</tr>
<tr>
<td style="text-align: left">The Signal and the Noise: The Art and Science of Prediction</td>
<td style="text-align: center">Nate Silver</td>
<td style="text-align: center">data science</td>
<td style="text-align: center">8 </td>
</tr>
<tr>
<td style="text-align: left">Never Split the Difference: Negotiating as if Your Life Depended on It</td>
<td style="text-align: center">Christopher Voss</td>
<td style="text-align: center">negotiation skills</td>
<td style="text-align: center">10</td>
</tr>
<tr>
<td style="text-align: left">Dark Tower I: The Gunslinger</td>
<td style="text-align: center">Stephen King</td>
<td style="text-align: center">fantasy</td>
<td style="text-align: center">6</td>
</tr>
<tr>
<td style="text-align: left">The Fifth Season: The Broken Earth, Book 1</td>
<td style="text-align: center">N. K. Jemisin</td>
<td style="text-align: center">fantasy</td>
<td style="text-align: center">6</td>
</tr>
<tr>
<td style="text-align: left">The Obelisk Gate: The Broken Earth, Book 2</td>
<td style="text-align: center">N. K. Jemisin</td>
<td style="text-align: center">fantasy</td>
<td style="text-align: center">4</td>
</tr>
<tr>
<td style="text-align: left">Annihilation: The Southern Reach Trilogy, Book 1</td>
<td style="text-align: center">Jeff VanderMeer</td>
<td style="text-align: center">sci-fi</td>
<td style="text-align: center">6</td>
</tr>
<tr>
<td style="text-align: left">Authority: The Southern Reach Trilogy, Book 2</td>
<td style="text-align: center">Jeff VanderMeer</td>
<td style="text-align: center">sci-fi</td>
<td style="text-align: center">5</td>
</tr>
<tr>
<td style="text-align: left">Acceptance: The Southern Reach Trilogy, Book 3</td>
<td style="text-align: center">Jeff VanderMeer</td>
<td style="text-align: center">sci-fi</td>
<td style="text-align: center">3</td>
</tr>
</tbody>
</table>
<h3 id="paused-but-to-be-continued">Paused, but to be continued</h3>
<table>
<thead>
<tr>
<th style="text-align: left">Title</th>
<th style="text-align: center">Author</th>
<th style="text-align: center">Topic</th>
<th style="text-align: center">Score</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left">Thinking Fast and Slow</td>
<td style="text-align: center">Daniel Kahnemann</td>
<td style="text-align: center">psychology </td>
<td style="text-align: center"> 7</td>
</tr>
<tr>
<td style="text-align: left">The Data Science Handbook: Advice and Insights from 25 Amazing Data Scientists</td>
<td style="text-align: center">Carl Shan, William Chen, Henry Wang, Max Song </td>
<td style="text-align: center">data science</td>
<td style="text-align: center"> 7</td>
</tr>
</tbody>
</table>
<h3 id="completely-abandoned">Completely abandoned</h3>
<table>
<thead>
<tr>
<th style="text-align: left">Title</th>
<th style="text-align: center">Author</th>
<th style="text-align: center">Topic</th>
<th style="text-align: center">Score</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left">Antifragile: Things that Gain from Disorder</td>
<td style="text-align: center">Nassim Nicholas Taleb</td>
<td style="text-align: center"><em>(Life, The Universe and Everything)</em></td>
<td style="text-align: center">3</td>
</tr>
</tbody>
</table>Somi AndrásI had a pledge in July to read as much as I can besides my studies and other activities. More precisely, I aimed for one book per week, ’serious’ and entertaining ones inlcuded. It’s been almost 4 months now, so I assessed where I stand with this plan.Beware of iterable default params in Python2017-10-29T00:00:00+00:002017-10-29T00:00:00+00:00http://dotkomblog.com/coding/2017/10/29/python-iterable-params<p>I got a question in this <a href="https://dotkomblog.com/learning/2017/10/27/the-biggest-challenge-in-self-learning/">Python assessment quiz</a> the other day, that surprised me:</p>
<p><img src="/img/posts/python_assess2.png" alt="" /></p>
<p>I picked this as an answer:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="p">[</span><span class="mi">2</span><span class="p">]</span>
<span class="p">[</span><span class="mi">3</span><span class="p">]</span>
</code></pre></div></div>
<p>but the right one was:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">]</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span>
</code></pre></div></div>
<p>So, the keyword param <code class="language-plaintext highlighter-rouge">items</code> retains its previous value, even though the function <code class="language-plaintext highlighter-rouge">f</code> is called repeatedly. This is not what I would intuitively expect from it.</p>
<p>I don’t know about every other languages, but I knew that in JavaScript a similar code would work as I expected. So this:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">f</span> <span class="p">(</span><span class="nx">a</span><span class="p">,</span> <span class="nx">items</span><span class="o">=</span><span class="p">[])</span> <span class="p">{</span>
<span class="nx">items</span><span class="p">.</span><span class="nx">push</span><span class="p">(</span><span class="nx">a</span><span class="p">)</span>
<span class="k">return</span> <span class="nx">items</span>
<span class="p">}</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">f</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">f</span><span class="p">(</span><span class="mi">2</span><span class="p">))</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">f</span><span class="p">(</span><span class="mi">3</span><span class="p">))</span>
</code></pre></div></div>
<p>would gracefully output this:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="p">[</span><span class="mi">2</span><span class="p">]</span>
<span class="p">[</span><span class="mi">3</span><span class="p">]</span>
</code></pre></div></div>
<p>It turns out, MDN explicitly calls out Python when explainig JS default params.</p>
<blockquote>
<p>The default argument gets evaluated at call time, so unlike e.g. in Python, a new object is created each time the function is called. (<a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Default_parameters">Default parameters - MDN</a>)</p>
</blockquote>
<p>I found the relevant section in the Python docs too, which warns about this behaviour. Strangely enough it also admits that this is not what was intended, and proposes a ‘workaround’ (which in turn appears all over the place in Python libraries):</p>
<blockquote>
<p>Default parameter values are evaluated from left to right when the function definition is executed. This means that the expression is evaluated once, when the function is defined, and that the same “pre-computed” value is used for each call. This is especially important to understand when a default parameter is a mutable object, such as a list or a dictionary: <strong>if the function modifies the object (e.g. by appending an item to a list), the default value is in effect modified. This is generally not what was intended.</strong> A way around this is to use None as the default, and explicitly test for it in the body of the function (<a href="https://docs.python.org/3/reference/compound_stmts.html#function-definitions">The Python Language Reference</a> - emphasis is mine)</p>
</blockquote>Somi AndrásI got a question in this Python assessment quiz the other day, that surprised me:The biggest challenge of self-learning2017-10-27T00:00:00+00:002017-10-27T00:00:00+00:00http://dotkomblog.com/learning/2017/10/27/the-biggest-challenge-in-self-learning<p>I was a bit surprised (in a positive way) to get this result in this <a href="https://www.pluralsight.com/product/pluralsight-iq">Pluralsight assessment</a>:</p>
<p><img src="/img/posts/python_assess.png" alt="" /></p>
<p>This is not an unquestionable source of truth, just a nice quiz, even though Pluralsight claims to have a sophisticated methodology to build the questionaire on the fly, based on your previous answers (which it <em>seems</em> to be doing). I don’t see a reason why they would want to systematically overestimate my skills, nevertheless it’s still just an uncertain result from mere 18 questions.</p>
<h2 id="its-been-only-a-year">It’s been only a year</h2>
<p>I checked my learning log (I have a rather detailed account of my courses from the past few years) and it turned out that I first ventured into <em>Python Land</em> just a little more than a year ago, during summer of 2016. Feels lot more.</p>
<p>It all started with a two-day long <a href="https://www.facebook.com/events/2044086375817540/">Data Science Bootcamp by Streambright</a> (which I sort of enjoyed then, but I value the experience a lot more now) for which I quickly skimmed some introductory courses to at least have some idea about the basic syntax. Then a lot more followed, packed into a very busy year.</p>
<p>It was a sort of relief realising that I have only been playing around with Python for a year or so, as I tend to beat up myself for not knowing something or being slow when trying to accomplish things in Python (or in anything else, actually). I stopped for a while to appreciate the journey taken so far.</p>
<h2 id="how-good-do-you-think-you-are">“How good do you think you are?”</h2>
<p>After a few years of self-learning I feel that I know lot more than a complete newbie, but I have no way to confidently label myself as <em>junior</em>, <em>experienced</em> or whatever labels people use. Not that I am that eager to categorize myself, but in some cases it’s somewhat necessary for efficient communication.</p>
<p>So this question I got recently is a difficult one for me:</p>
<blockquote>
<p>‘How proficient do you think you are in coding?’</p>
</blockquote>
<p>My honest answer is that I can only guess but I don’t know.</p>Somi AndrásI was a bit surprised (in a positive way) to get this result in this Pluralsight assessment:The hardest simple questions in life2017-10-23T00:00:00+00:002017-10-23T00:00:00+00:00http://dotkomblog.com/life/2017/10/23/whats-your-ikigai<p>You can stumble upon the chart below rather frequently around the web (this time on <a href="http://ritholtz.com/2017/10/ikigai-japanese-secret-long-meaningful-life/">The Big Picture</a>), but for some reason the message hit me now stronger than usual. Maybe it’s because we are on holiday in Southern Portugal, no stress, no crowd, no bad weather, so I have time and mental ease to contemplate things like this.</p>
<p><img src="http://ritholtz.com/wp-content/uploads/2017/10/ikigai.jpg" alt="" /></p>
<blockquote>
<p>The word ikigai is usually used to indicate the source of value in one’s life or the things that make one’s life worthwhile. Secondly, the word is used to refer to mental and spiritual circumstances under which individuals feel that their lives are valuable. It’s not necessarily linked to one’s economic status or the present state of society. Even if a person feels that the present is dark, but they have a goal in mind, they may feel ikigai. Behaviours that make us feel ikigai are not actions we are forced to take—these are natural and spontaneous actions. (<a href="https://en.wikipedia.org/wiki/Ikigai">Wikipedia</a>)</p>
</blockquote>
<p>It might be also interesting to observe which circle you instinctively start the process with. Do you ask yourself what do the world need, or do you focus on what you can earn money with?</p>
<p>As I looked at the chart I immediately thought about the top section: what do I love to do? Maybe it’s just pure selfishness, and not some noble awakening of self-consciousness, but here’s my train of thought from there, anyway.</p>
<h3 id="1-what-i-love-to-do">1. What I love to do?</h3>
<p>It’s the hardest one, and I do remember the times when I was struggling with it. Nowadays I tend to think I have an answer: I love to understand suff. Read, study, analyse, reason and find logical and practical proofs about connections, trends, currents, possible outcomes and impacts in complex systems.</p>
<p>Luckily, everything is a system, from the human mind and behaviour through companies, societies and economies (stock markets!) to all aspects of Mother Nature, so there are plenty of things to think and read about. But that’s too general to test with the other filters, so let’s make it more specific: I love to <em>understand</em> data, the <em>process</em> of understanding data and to <em>explain</em> what I understand.</p>
<h3 id="2-am-i-good-at-it">2. Am I good at it?</h3>
<p>As it’s my natural habit I definitely have <em>comparative advantage</em> in carrying out (data) analysis, but even if this sounds promising, it just means I suck (even more) in many other things.</p>
<p>In the meantime I am definitely not amongs the <em>best</em>, neither by theoretical or academic background nor for practical skills or competence. I just have a little of both, but luckily the lack of crucial knowledge (or at least assuming it to be crucial) creates enough frustration to fuel the drive to learn more.</p>
<p>But the most important realization is that the criterium for <em>ikigai</em> is not being the <em>best</em>, just simply being <em>good</em>. So with all my humbleness I settle with this: in general I am good at analysing data.</p>
<h3 id="3-does-the-world-need-it">3. Does the world need it?</h3>
<p>For me it’s a crystal clear yes. The world needs people to make sense of data and help utilise it in a productive way. And luckily this field is getting broader every day, so all sorts of people can make themselves useful in data related areas. Of course, it’s a much more nuanced question whether the world needs the specific data analysis that I do… This brings me back to the previous point: I’m willing to learn the ways the world needs.</p>
<h3 id="4-can-i-be-paid-for-it">4. Can I be paid for it?</h3>
<p>It’s not by chance that this is the last one. Money is the least important of the four aspects, which does not mean at all that it isn’t important (it is very important, but there are even more important ones). And let’s not forget that being paid for what you do is also part of <em>ikigai</em>.</p>
<p>Anyway, I dare to say yes to this one, too. I do see the possibility to build a sustainable freelance career in digital data and analytics, and also see job openings (not many though) that I would be willing to take, not only from passion but for financial reasons as well. (Also, I <em>was</em> payed for doing analysis before, though not strictly in the digital data field.)</p>
<h3 id="bottom-line">Bottom line</h3>
<p>The bottom line here is not how incredibly enlightened I am to oversimplify the greatest questions of life while casually vandalizing thousand years old oriental wisdom (I’m not even sure mine would qualify as real answer in a philosophical sense). The questions are not simple at all, and on top of that, your perspective might change over time, making certain parts clearer or more opaque. So the only real point here is that it’s worth meditating on <em>ikigai</em> every now and then.</p>Somi AndrásYou can stumble upon the chart below rather frequently around the web (this time on The Big Picture), but for some reason the message hit me now stronger than usual. Maybe it’s because we are on holiday in Southern Portugal, no stress, no crowd, no bad weather, so I have time and mental ease to contemplate things like this.Why data should replace expert advice2017-10-08T00:00:00+00:002017-10-08T00:00:00+00:00http://dotkomblog.com/life/2017/10/08/why-data-will-replace-experts<p>This summer we cancelled our holiday trip to Italy, just two days before departure, because the doctor advised my then few-weeks pregnant wife to avoid flying. The reason was a blurry little something he saw on the ultrasound (I mean, besides our future child, who was also a blurry little something at that point). According to the same doctor, this was quite common and would most probably disappear without any noticable effect (it did, indeed).</p>
<p>Obviously we felt that a short (and not quite exotic) holiday in the Mediterranian wasn’t worth the risk of any issues with the pregnancy. But what if we could have known the <em>exact</em> risk? Like there were some kind of problems in exactly 1 in a million similar cases? Or 1 in a billion? Could this change our minds? Is it really how it should work?</p>
<p>As quite a few common fallacies are packed into this little story it keeps bugging my analytical mind.</p>
<h2 id="inability-with-probability">Inability with probability</h2>
<p>One thing that came into play here is that we are genetically not prepared to grasp probabilistic outcomes. Maybe our ancestors did not want to assess the chances of being slaughtered by a sabertooth tiger by repeating the same experiment thousand times, just to have a fair understanding of the probability distribution. Most of them simply ran away, and the remaining prehistoric statisticians fell victim to natural selection.</p>
<p>So the human brain is wired for loss-aversion, as a clear evidence of being lousy with probabilities. It forces us to avoid scenarios where the perceived loss is high, even when the expected value (the sum of outcomes, each weighted by its probability) is better than in other alternatives. Like when we act to avoid losing our future child, even if the chances are equal to being hit by an asteroid with a rainbow-colored unicorn named Jeff riding on it.</p>
<p>It works the other way around, too. People buy lottery tickets or invest in lottery-like securities for the promise of extreme payout with extreme low probability (with a clearly negative expected value, even with apparently low costs).</p>
<p>So we are in general very bad at assessing tail-risks and expected values, especially when it includes lot of factors. Our case was not a banal dilemma, but people react similarly to smaller common problems. And huge businesses are built upon these fallacies, usually with some kind of <em>‘experts’</em> and their advice in the middle.</p>
<h2 id="experts-dont-know-better">Experts don’t know better</h2>
<p>Most likely our doctor had no idea of the exact probabilities in a case like ours. Things that might happen in the body of a pregnant woman under special circumstances is too complex to predict in details just by relying on <em>medical folklore</em> and the subjective memories of a single person (except when the symptoms are obvious, and it’s already too late).</p>
<p>But admitting this is not an option. Should something bad occur on the trip he approved, the doctor would not be able to explain that it was not related to travelling (he wouldn’t even know himself). So he tossed the risk around, playing it out on our loss-aversion, to make it our call to make the trip, <em>against</em> his advice, even if it was unnecessary precaution (maybe it was, maybe not, I don’t know.)</p>
<p>Or think of predictions of stock market pundits. In that circus you have to say something specific and certain <em>all the time</em>. It’s like buying lottery tickets: maybe you get lucky and it will make you <em>The Expert</em>. The pay-off can be huge while being wrong is quite okay if you wrapped your message in proper amount of bullshit. That’s why stock market talking heads make as many long shots as they can and use clichés and professional language, that might sound knowledgable but in fact is pure gibberish.</p>
<h2 id="we-need-data-and-experts-to-explain-it">We need data and experts to explain it</h2>
<p>So experts are not free from all the psychological pitfalls every human have to deal with, even burdened by some others. Still, it does not necessarily mean we should get rid of them. On the contrary.</p>
<p>Besides many important things the doctors have always provided great service by <em>explaining the current condition</em> of a patient, a good stock market analyst could give you comprehensive summary of the <em>current state of a company</em>, real-estate agent can <em>guide you through the process</em> of buying a new home, and so on. You just shouldn’t expect these people to make predictions or see through complex problems in their entirety.</p>
<p>The ultimate solution is using data in place of anecdotal evidence. Our advice should have come in a form like</p>
<blockquote>
<p><em>’in XX thousand cases flying caused minor issues in 0.00X% and major issues in 0.000X% of the cases for pregnant women with similar condition, but in certain circumstances with some specific actions these rates fell by XX%.’</em></p>
</blockquote>
<p>so we could make a more educated decision (but still have to digest probabilities ourselves, which is hard).</p>
<p>Better, imagine that a clear guidance was based on a <em>real-time</em> model that <em>continuously</em> evaluates hundreds of different external and internal factors about the body of the mother (obtained through wearable or even implanted devices) and compares it to a vast database of pregnant women under different circumstances all around the world. It’s not even sci-fi, it’s all existing technology.</p>
<p>These things will mean much less ‘expert’ guesses in the future, and maybe a little less irrationality in everyday decision-making. But for a long-long time we will still need human experts to build, finetune, operate and explain the results of such artificial intelligence. So the best experts are not threatened by the looming AI revoluion, but will benefit the most from transferring their expertise to this new platform.</p>
<p>And in the meantime we will have the coolest kid in March. That’s a fact.</p>Somi AndrásThis summer we cancelled our holiday trip to Italy, just two days before departure, because the doctor advised my then few-weeks pregnant wife to avoid flying. The reason was a blurry little something he saw on the ultrasound (I mean, besides our future child, who was also a blurry little something at that point). According to the same doctor, this was quite common and would most probably disappear without any noticable effect (it did, indeed).Visualising air traffic2017-09-15T00:00:00+00:002017-09-15T00:00:00+00:00http://dotkomblog.com/data/2017/09/15/first-tableau-viz<p>My last course in the Data Analyst nanodegree was about data visualisation in Tableau. I didn’t have a lot of experience with Tableau before, but now I fell in love with the software quite fast. (I tried it many years ago, but I left, partially because the hefty license fees back then).</p>
<p>So this is what I submitted for the assignment. I downloaded the data from the <a href="https://www.transtats.bts.gov/OT_Delay/OT_DelayCause1.asp?pn=1">Bureau of Transportation Statistics</a>, and used it without any major modifications, as the real goal of the project was to create an engaging visualisation.</p>
<p>Unfortunately I did not optimize the layout for embedding, so for the best experience click ‘Full Screen’ at the bottom, or check out the original on <a href="https://public.tableau.com/views/Flights_final/USFlights?:embed=y&:display_count=yes&publish=yes">Tableau Public</a>). You can switch between slides at the top with the arrows or by clicking the dots.</p>
<div class="tableauPlaceholder" id="viz1505504085887" style="position: relative"><noscript><a href="#"><img alt="US Flights " src="https://public.tableau.com/static/images/Fl/Flights_final/USFlights/1_rss.png" style="border: none" /></a></noscript><object class="tableauViz" style="display:none;"><param name="host_url" value="https%3A%2F%2Fpublic.tableau.com%2F" /> <param name="site_root" value="" /><param name="name" value="Flights_final/USFlights" /><param name="tabs" value="no" /><param name="toolbar" value="yes" /><param name="static_image" value="https://public.tableau.com/static/images/Fl/Flights_final/USFlights/1.png" /> <param name="animate_transition" value="yes" /><param name="display_static_image" value="yes" /><param name="display_spinner" value="yes" /><param name="display_overlay" value="yes" /><param name="display_count" value="yes" /><param name="filter" value="publish=yes" /></object></div>
<script type="text/javascript"> var divElement = document.getElementById('viz1505504085887'); var vizElement = divElement.getElementsByTagName('object')[0]; vizElement.style.width='100%';vizElement.style.height=(divElement.offsetWidth*0.75)+'px'; var scriptElement = document.createElement('script'); scriptElement.src = 'https://public.tableau.com/javascripts/api/viz_v1.js'; vizElement.parentNode.insertBefore(scriptElement, vizElement); </script>
<p>In general I’m satisfied of what came out of my first excursion, even though I obviously only explored a tiny little part of Tableau. My heart aches when I think about all those boring Excel charts I created all these years…</p>Somi AndrásMy last course in the Data Analyst nanodegree was about data visualisation in Tableau. I didn’t have a lot of experience with Tableau before, but now I fell in love with the software quite fast. (I tried it many years ago, but I left, partially because the hefty license fees back then).