Jon Calder

Naming things is hard

Mon, 04 Dec 2017 00:00:00 +0000

‘There are only two hard things in Computer Science: cache invalidation and naming things.’

The above quip by Phil Karlton is fairly well known and often quoted, sometimes with amusing extensions:

There are two hard things in computer science: cache invalidation, naming things, and off-by-one errors.
— Jeff Atwood (@codinghorror) August 31, 2014

There are only 2 hard things in computer science:
0. Cache invalidation
1. Naming things
7. Asynchronous callbacks
2. Off-by-one errors
— Paweł Zajączkowski (@gvaireth) September 18, 2017

These are funny, but they do also convey some truth: in the midst of all the technicalities and abstractions we can find ourselves caught up with in the world of programming, it’s surprising how often seemingly ‘simple’ things like naming things trip us up.

I was recently reminded of this when a difference of opinions about function names sparked some healthy debate in a pull request I was reviewing for one of my personal projects.

So the question I want to raise is this:

“When (if ever) is it a good idea to adopt a prefixing convention for the names of exported functions in an R package?”

Disclaimers

Before I dive into the details I feel it is important to state a few things upfront.

Firstly, I want to thank my friends and collaborators Katrin and Lorenz who are strong proponents of open source software and for whom I have a lot of respect. On this occasion they both seem to disagree with me, but that is not a bad thing - discussion and debate is valuable, and that’s not gonna happen when everyone agrees with each other all the time. I did also ask for their permission before publishing this post.

Secondly, my purpose in writing about this is less about trying to determine who is right, and more about attempting to convert this experience into insight. Often we learn more from our failures than from our successes, but it’s harder to share our mistakes than it is to share our triumphs. So this post is my way of being vulnerable about something that is a work in progress and the process of trying to improve on it. As an aside, if you haven’t encountered survivorship bias before, I highly recommend you read this.

Thirdly, I was wrong. This is especially important in light of the previous point. I was wrong to raise the issue of function names (for the package as a whole) in a pull request which was focused on something else. This is a valuable lesson. One should always aim to keep pull requests (and issues) narrow in scope because attempting to tackle multiple problems in one place (unless they are inherently linked or dependent on each other) is messy and complicates matters unnecessarily.

Lastly, what I share will be my own opinion, but it is just an opinion and I’m always open to learning from others with different views. My hope is that collectively we can share some worthwhile perspectives from both sides and possibly encourage more thinking and conversation around this or related issues.

Background Context

Ok, so having made those upfront disclaimers, I’ll begin by summarizing the back-story and context in which the discussion arose. If you’d like to refer to the pull request itself - it can be found here.

excercism.io is a learning platform that aims to provide code practice and mentorship for everyone. I got involved in developing the R track on Exercism and wrote about it earlier this year. Unlike most online learning platforms, with Exercism, all the coding happens on your machine in an environment you’re familiar with. So Exercism provides a command line tool which leverages an API in order to facilitate the process of fetching and submitting exercises.

A few months ago, I had an idea to write an R package which wraps this API. The thinking was that the user experience (for R users) might be improved upon by facilitating interaction with exercism.io directly from R itself. This removes the need for switching repeatedly between R and a shell when fetching, iterating on and submitting exercises - although now the addition of terminal tabs in RStudio 1.1 has already reduced this friction to a degree. In any case, there are additional opportunities for Exercism helper functions in the package which can be context aware and integrate with the RStudio if it is being used. An example this could be functions (or addins) which make use of the rstudioapi to detect which problem was last worked on when submitting so that it doesn’t need to be specified manually.

Katrin, who is a co-maintainer for the R track on exercism.io, has also been collaborating on this R package with me and has had some great ideas like leveraging testthat’s auto_test() to facilitate and encourage test driven development, as this is one of the implicit goals of Exercism. In the PR introducing this feature, the potential for function name confusion was soon evident when this new Exercism specific version of testthat::auto_test() was (initially) given the name autotest(). This reminded me that I’d in fact been thinking for a while about renaming all the exported functions to adopt the prefixing convention ex_* (for a few different reasons which I’ll get to later). So I figured this “name clash” was as good a catalyst as any, and made the suggestion to start adopting the new naming convention in the PR. Once again it’s worth noting that this was a mistake - I should have instead opened a separate issue to discuss my proposed change in naming conventions.

Discussion & follow-up

The suggestion was met with some resistance, and after some further discussion it became clear to me that it was a thoughtfully considered resistance. So I asked my friend Lorenz to weigh in on the discussion too, given that he knows Katrin and I but is not involved in the project and thus has the benefit of a more neutral perspective. To my surprise, he did not agree with me either!

But I did still seem to have Jenny Bryan on my side (thanks Jenny!), and I figured Hadley had to have thought it was a good idea at some point given the str_ prefix for stringr and fct_ prefix for forcats among others. So after thinking on the problem for a while, out of curiousity I eventually tweeted out a poll to see if I could get any sense of where the #rstats community falls on this issue.

What is your take on prefixing conventions for #rstats function names? (e.g. stringr/stringi, forcats, googlesheets)
— Jon Calder (@jonmcalder) October 20, 2017

At a glance it looks like a reasonable proportion are actually in favour of prefixing conventions for function names (or at least not against the idea), but of course there are a number of disclaimers to make here:

Character limits (at the time of the poll) made it hard to communicate the question clearly or to include any additional context for the question, so that probably leaves a lot of room for interpretation
I don’t have much reach on Twitter, so there weren’t many responses (81 votes is not much to go on)
Even if there had been a good number of responses, Twitter polls need to be looked at skeptically given the potential for sampling bias
Speaking of sampling bias, most of the votes came in after Hadley tweeted a reply to the poll so it makes sense that the results would be skewed towards his legions of followers (I’m one of them and the degree of influence is clear because his packages are what got me considering prefixing conventions in the first place, among others like googlesheets)

Maëlle had two helpful follow-ups for me. Firstly, she encouraged me to blog about this (and I don’t think I would have done so otherwise so thanks Maëlle!). Secondly, she directed me to the ROpenSci review process for her package ropenaq, which provides access to air quality data via the OpenAQ API. In his review of the package, Andrew MacDonald suggested the following:

“I was thinking that the names of functions might be a bit similar to functions in other packages that also use geography. What do you think of prefixing every function with a package-specific string? Perhaps something like aq_ before all the user-facing functions (i.e. countries() becomes aqcountries()). This is similar to another rOpenSci package, geonames, which uses GN (as in GNcities()). This has the added benefit of playing very nicely with Rstudio, which will show all the package functions as completions when users type aq and hit tab.”

Interestingly, this suggestion (although the original inspiration may have come from elsewhere) was later incorporated into ROpenSci’s packaging guide:

Consider an object_verb() naming scheme for functions in your package that take a common data type or interact with a common API. object refers to the data/API and verb the primary action. This scheme helps avoid namespace conflicts with packages that may have similar verbs, and makes code readable and easy to auto-complete. For instance, in stringi, functions starting with stri_ manipulate strings (stri_join(), stri_sort(), and in googlesheets functions starting with gs_ are calls to the Google Sheets API (gs_auth(), gs_user(), gs_download()).

Though I hadn’t seen this recommendation from ROpenSci at the time, it aligns very strongly with my initial reasoning for wanting to change the function names in the exercism package. It is primarily an API package, and all functions either interact with the exercism.io API or act on some (local) Exercism data/code (exercises). A potential objection could be that in some cases the ex_* prefix may be interpreted either as exercism_* or as exercise_*, but I don’t think that’s a problem since either way the context is common and shared implicitly.

Having said that, I’m also aware that a prefixing convention is not suitable in the majority of cases and there are reasons to avoid it, otherwise it would already be far more common. I’ve not tried to summarize the arguments for and against it here since this post is already quite lengthy, but I believe Katrin and Lorenz both raised a number of good points over in the original PR thread, so I would encourage you to read through that to get some more insight into the potential pros and cons.

Below is an overview of the currently exported functions for exercism, along with a brief description of what they do and potential new names for each should we adopt a prefixing convention:

Current Function	Description	New Name?
`set_api_key()`	Set an environment variable for the provided exercism.io API key, and store in .Renviron so that it can persist for future sessions.	`ex_set_key()`
`set_exercism_path()`	Set an environment variable for the provided exercism path, and store in .Renviron so that it can persist for future sessions.	`ex_set_path()`
`track_status()`	Fetches current track status from exercism.io	`ex_status()`
`check_next_problem()`	Returns the next problem for a language track	`ex_check()`
`fetch_problem()`	Fetches the files for a problem via the Exercism API and writes them into a new problem folder in the Exercism directory	`ex_fetch()`
`fetch_next()`	Checks for the next problem via the Exercism API, and writes the files into the folder in the Exercism directory	*special case of `ex_fetch()`
`open_exercise()`	Open files for an exercism.io problem	`ex_open()`
`start_testing()`	Exercism- and R-specific wrapper for `testthat::auto_test()` that starts testing your solution against the problem’s test cases.	`ex_auto_test()`
`submit()`	Submits the specified solution to exercism.io	`ex_submit()`
`skip_problem()`	Marks a problem as ‘skipped’ via the Exercism API	`ex_skip()`
`browse_exercise()`	Navigate to an exercise description on exercism.io	`ex_browse()`
`browse_solution()`	Navigate to an exercise solution on exercism.io	*special case of `ex_browse()`

So looking at the above, do you think this a good use case for an object_verb() naming convention? How should one determine this? Please feel free to comment with your thoughts and suggestions either in the comments section below, on Twitter, or in this RStudio Community discussion thread.

The spark of statistics

Wed, 26 Jul 2017 00:00:00 +0000

I’ve enjoyed maths for pretty much as long as I can remember. I’m aware that it is rare to make such a statement. Probably straight up weird to some. But that just shows how much I have to be grateful for. I am indebted to my parents, extended family and friends and a number of great teachers over the years who made sacrifices in order to give me solid learning opportunities and a supportive environment.

I mention my affinity for maths because it is pretty hard to enjoy statistics without first enjoying maths. A few weeks ago I watched a cool Ted Talk entitled “Why you should love statistics”. It got me thinking about how I came to love statistics, and for me it most definitely began with a love for maths.

I tutored a lot of undergraduate statistics during my university days and most of the students who struggled had little appreciation for statistics because they were constantly caught up in the underlying mathematical mechanics. My recollection is that often their maths deficiency was not due to a lack of ability, but could instead be attributed to a foundational gap of some sort. Whether it was moving schools, a change of teacher, prolonged sickness or some other catalyst for falling behind, I was left with the impression that somewhere along the way seeds of doubt had been sown for these students, and proved a force to be reckoned with further on down the road.

Alan Smith begins his Ted Talk by highlighting the low levels of numeracy in a number of OECD countries but is quick to warn us against what he considers a false dichotomy: the idea that some of us can “do numbers” and some of us can’t. I’m inclined to agree with him. Sure, some aspects of numeracy and mathematical ability are probably innate, but I think that as with many other skillsets, the driving factors for success are (1) a conducive learning environment; (2) a good support structure; (3) a positive mindset; and (4) practice. The problem is, boxes (1) and (2) aren’t ticked for many young students (especially in less developed countries like mine which are plagued by poverty and related hardships), and even if they are, it’s hard to have a positive mindset (3) and to invest in practice (4) if you’re given the impression that you simply can’t do numbers after getting off to a rocky start, or after hitting a few speedbumps along the way.

Two recent cases came to mind which re-inforce this idea for me. The first was a powerful personal story shared as part of a message at church. Given that I’m referencing the story outside of the context in which it was originally shared, I’ve paraphrased the story below and avoided any direct reference to the speaker himself.

In Standard 5 I discovered that I was stupid. In the process of switching schools, I had moved from Standard 3 straight to Standard 5. So in addition to the language barrier I now had to overcome at the new school, I also had to catch up on Standard 4. Catching up on the maths content I had missed proved particularly difficult. I recall fractions were a problem. My teacher said to me: “As soon as you have an opportunity to choose subjects, you should choose history, because you will never be able to do maths.” At the end of that year I got 32% for maths. I remember thinking, where did I get all these marks?

After getting to Standard 6 the following year, I found out that I could only change subjects in Standard 7. So that meant I had to endure another year of maths. I had a new teacher and after our first maths test, she called me in (no surprise there). She said: “I’m not gonna mark your paper because this is not who you are. Not only do you have the potential to get good grades, but you are actually the best maths student we have in this school.” What a strange thing to say… …she didn’t know me at all. At the time I thought she was really, really crazy (and I think she was a little bit). In the next test, I got 62%. I hadn’t studied harder. I didn’t change anything. And then she called me in again and said: “This is not who you are. You are the best maths student in this school.”

The following test, I became that.

The second example came in the form of a local news article about Tim Schlesinger, who was recently selected by the South African Mathematics Foundation to represent the country at the International Mathematics Olympiad and Pan African Mathematics Olympiad competitions overseas. The extract below is taken directly from the article:

He also gave words of advice to other learners who would like to do better in maths, saying they should focus on enjoying maths.

“From there, everything else follows easily. Many think that you are either born able to do maths or not, but you really can learn both theory and problem solving. It has been encouraging to see how much time and effort others are willing to invest in young mathematicians.”

So having (hopefully) convinced you that many people who think they can’t do maths probably could if they would just stick it out long enough to overcome their doubts - let’s now get back to my story!

One of my earliest recollections of enjoying maths was during grade 3, when we were first introduced to “word problems”. The irony of that statement is not lost on me, because by matric I had discovered that English would actually be my most challenging subject at school. Spelling and grammar were fine. Sure, there often seemed to be more exceptions than there were rules (especially in the case of English), but at least there were SOME rules to follow. However when it came to the required ‘creative’ interpretations of poetry and literature I often found myself at a loss.

Anyway, word problems introduced me to the elegance of thinking about and communicating a concept or problem mathematically, by contrast to the wordy version which often seemed (to me at least) a lot more complex. Happily, this enjoyment of maths continued for me right through both primary and secondary school, and my family still laughs at the recollection that sometimes when I got home from school in the afternoons my mom had to ask me to go and play outside with my younger sister before diving into my maths homework.

Admittedly by my 2nd year of university I thought some of the ‘pure math’ started getting a little weird though. Eventually I grew to appreciate the more abstract notions of mathematical rigor that were introduced to us in our Real Analysis module as we looked at the construction of the real number system, limits and convergence, continuity etc but it did require a more concerted effort. It was around this time that statistics started to take precedence for me.

I recall our first encounter with double integrals surfacing early on in a 2nd year statistics course, long before it was actually introduced in the concurrent and pre-requisite maths (calculus) course. We were given a quick run through and just had to roll with it until our maths syllabus caught up. Once these theory gaps had been filled in we had a much better appreciation of the applied nature of what we were learning in the statistical domain. This was a picture of things to come, since as I moved into the latter part of my degree and transitioned into post-grad, maths became subservient to my pursuit of statistics.

The interesting thing though is that I barely even knew statistics existed during most of my time at school. So how did I come to be interested in statistics at university? I believe a key catalyst was something which took place back in my grade 11 biology class. While covering the human reproductive system, we began a brief foray into the field of genetics, looking at chromosomes, DNA and RNA, proteins and all that life-shaping magic. Somewhere in the midst of all that, one day my teacher entered into a discussion about red-green colour blindness, highlighting that it is much more common in men. And so we were introduced to the concept of X-linked recessive inheritance.

I’m pretty sure he simplified things significantly in order to shield us from some of the complexities involved, so I will do the same here (with apologies to any geneticists who might be reading this). Here are the key pieces of information:

the family of mutations that are commonly responsible for certain types of (red-green) color blindness can only occur on the X chromosome
men have one X and one Y chromosome whereas women have two X chromosomes
in women, the mutation needs to be present on both X chromosomes to cause color blindness, so some women can be ‘carriers’ while remaining unaffected
since men have a single X chromosome, they will be color blind if this chromosome has the mutation

The resulting inheritance pattern, known as X-linked recessive inheritance, is nicely illustrated in the below diagram.

(diagram sourced from Wikipedia)

The diagram shows that a woman who carries an X-linked recessive disorder has a 50 percent chance (with each pregnancy) of having a son who is affected and a 50 percent chance of having a daughter who carries one copy of the mutated gene. In the interest of being more thorough/clear: there is an equal 25 percent chance of having an unaffected boy, affected boy, non-carrier girl, or carrier girl.

Of course there are other scenarios and combinations to consider too; for example the sons of a man with an X-linked recessive disorder will not be affected (since boys always receive the Y chromosome of their father), and his daughters will carry one copy of the mutated gene (since girls always receive the X chromosome of their father). Similarly, one can enumerate the possibilities for offspring of an affected mother and unaffected father, affected father and carrier mother, or affected father and affected mother. Things then get a little more interesting when starting to look at the propogation of such a disorder across a few generations with different combinations of these scenarios in effect in each generation.

And it was some combination of the above mentioned scenarios that my teacher threw out as a suggestion for further investigation (homework) that day which sparked my first meaningful exploration into the world of independent and conditional events, and the law of total probability. I’m pretty sure I didn’t know anything about conditional or marginal probabilities at the time, but as I sat down that evening to work through a few scenarios I recall that it came quite naturally to me to work out the underlying probabilities by stringing together the basic principles.

The next day I returned to school armed with a half-page of scribbled workings and the moment the question was asked my arm went up. As it turned out, no-one else had really taken the teachers suggestion from the previous day seriously and my raised hand was not met with any competition. So I proudly described the results of my probabilistic investigation to the teacher and the rest of class. He was suitably impressed and immediately affirmed my aptitude and efforts, whilst also encouraging me to pursue statistics further after leaving school. In hindsight it’s clear to me that this was good advice.

So Mr Moore, wherever you are now, thank you for all the things you taught me, and thank you for your words of affirmation which have stayed with me ever since that day in biology class.

Another thing Alan Smith highlights in his talk which I was not previously aware of is the etymology of the word statistics. It is ‘the science of dealing with data about the condition of a state or community’. He describes it simply as ‘the science of us’. So to me there is something quite poetic in the way my introduction to statistics came via biology and genetics - a different variant of the science of us.

I love statistics, so obviously I’m biased, but I think the Ted Talk (shown below) is well worth a watch, whatever your perspective on the matter. I hope that it encourages you to become more fascinated by numbers.

Data Science Podcasts

Wed, 31 May 2017 00:00:00 +0000

Podcasts are awesome. Especially when you’re stuck in traffic on the way to work.

Below are some podcasts I listen to that relate to data science and statistics. Each of them has something slightly different to offer, so if this is an area of interest to you then I recommend you give these a try!

Not So Standard Deviations

Roger Peng and Hilary Parker talk about the latest in data science and data analysis in academia and industry.

Data Skeptic

Data Skeptic is your source for a perspective of scientific skepticism on topics in statistics, machine learning, big data, artificial intelligence, and data science.

More or Less: Behind the Stats

Tim Harford and the More or Less team from BBC Radio 4 try to make sense of the statistics that surround us.

The R-Podcast

Giving practical advice on how to use R for powerful and innovative data analyses. The host of the R-Podcast is Eric Nantz, a statistician working in the life sciences industry who has been using R since 2004.

Partially Derivative

Hosted by Jonathon, Vidya, and Chris, Partially Derivative is a podcast about data science in the world around us. Episodes are a mix of explorations into the techniques used in data science and discussions with the field’s leading experts.

Linear Digressions

Hosts Katie Malone and Ben Jaffe explore machine learning and data science through interesting (and often very unusual) applications.

Are there other data science podcasts missing from this list that you can recommend? Feel free to comment below and let me know!

R track on exercism.io

Wed, 05 Apr 2017 00:00:00 +0000

As I’ve said before, when it comes to programming I’m a firm believer in the “learn by doing” approach. exercism.io is a project which exemplifies this.

I came across exercism.io earlier this year while exploring GitHub looking for an open source project to contribute to. The premise is fairly simple:

pick a language that you’d like to learn or improve on
fetch an exercise via the Command Line Interface (CLI)
code up and submit a solution for the exercise
return to the site to get feedback on your solution and/or to compare it to the solutions others have come up with

The problems are very simple to begin with, (usually starting out with the traditional “Hello, World!”), progressing to more difficult problems further on. It’s aimed at both newbies and experienced developers, with the philosophy being that for newer programmers the exercises are achievable but “with enough complexity to uncover bite-sized knowledge gaps”, whereas for more experienced developers, the problems “provide a balance of constraints and freedom and encourage you to explore trade-offs and best-practices”.

More experienced developers are also encouraged to get involved reviewing the solutions of others and/or to contribute to the project which is all completely open source, whether it is the website itself, API’s, the CLI, documentation or track content. The site is also well suited to those with experience in one or more languages who are wanting to ramp up in a new language, or to get a sense of the how the idiomatic approach to a problem might differ from language to language.

There are currently 35 active language tracks, with another 20+ language tracks planned or upcoming (essentially in an incubation stage). One of those which is hopefully soon-to-be-launched is the R language track, which I’ve been contributing to over the past two months and am pretty excited about.

If you’re interested, there are a number of ways you can get involved:

Firstly, give the R language track a try! (you’ll need to login on exercism.io using your GitHub account)
Once you’ve submitted a solution to a problem, you’ll be able to see other peoples submissions and are encouraged to comment on these and get discussions going around style and best practice, pros and cons of different approaches etc.
If you encounter any problems along the way (e.g. setup/install instructions, fetching or submitting exercises, running tests etc) then raise it via the online chat support in order to get help and/or bring it to someone’s attention if something needs to be fixed
With regard to the above, if you encounter an issue which applies specifically to the R track, then please open an issue on the R track repo
Consider becoming a mentor for the R track (if you’re interested, please reach out to me on Twitter or via my contact page)
Checkout the list of unimplemented exercises for the R track and follow the instructions there to submit a pull request
Lastly, look through exercism.io's contributing doc, which outlines a number of other ways you can get involved (either with a specific language track, or across other areas of the project as well)

Based on my involvement so far the exercism.io community seems very friendly and open, and I think its a great open source initiative, so there’s no reason why the #rstats community shouldn’t be well represented there.

On that note, a big thank you to exercism’s contributors, and especially to Jonathan Boiser and Katrina Owen for being friendly, helpful and supportive of my involvement in the project thus far. You guys are excellent role models for the greater open source community.

satRday Cape Town

Mon, 13 Mar 2017 00:00:00 +0000

Almost a full month has passed since satRday Cape Town. Time races on (as always) bringing plenty of challenges and opportunities with each new day, but part of me wants to go back and re-live the experience, just to have some more time to take it all in again.

It’s an event which I hope will remain firmly ensconced in my memory for many years to come. I met many amazing individuals and made a whole lot of new friends. It was a fantastic time. As a result, I figured it would be a good idea to thank some of the people who made it possible.

Right at the outset, I think it’s only right to give a big shoutout to Andrew: without whom, satRday would not have come to South Africa. To give you some context, I believe he first got the process started around May 2016 with a proposal to host satRday in Cape Town. Over the following nine months as things started to come together, he worked tirelessly in order to nail down all the conference details ahead of time, and also planned and executed brilliantly during the week of the conference itself. If asked, he’ll probably say the execution was mostly down to his ability to delegate but the truth is he put in most of the hours. We all owe him a debt of gratitude.

R-ladies at Boulders Beach to see some penguins @JennyBryan @SteffLocke pic.twitter.com/fe8StAIPJh
— Julia Silge (@juliasilge) February 19, 2017

To Ryan and Etienne, who helped out with a myriad of things including web design, planning and chairing sessions, manning the registration table, recording talks, and sorting out catering - thank you for all your hard work.

Also a special thanks to Katie Lennard who helped out as a teaching assistant for both days of workshops prior to speaking on Saturday.

To our wonderful keynote speakers - Julia Silge, Jenny Bryan and Steph Locke - who all travelled a long way in order to be with us: we really appreciate you and are so grateful that you were able to be there to play your part in making the conference such a success. We really value all the effort you put into your workshops and talks, and also your willingness to invest time in conversation with many of us in the midst of it all. I hope you enjoyed your time in South Africa!

To all of the speakers and to those who led tutorials - especially those of you who travelled from elsewhere in SA or even further afield in order to be here - thank you for the time and effort spent in preparation to present at satRday Cape Town. Thank you also for being flexible in dealing with the time constraints due to the packed programme!

I hope that you all got a lot out of the experience as we did. It was a treat to have such a diverse group spanning many disciplines within both industry and academia, and I think the breadth and depth of topics covered reflect a vibrant and rapidly growing R community in South Africa.

Then to the management and staff of Workshop17 for making your magnificent facility available for this conference and for looking after us while we were there - thank you!

My final thanks go to all of our sponsors for Cape Town satRday, and especially to Vantage Data for sponsoring the Data Visualisation Challenge along with post-conference drinks. This extended the opportunity for networking and was a great way to wrap things up after a jam-packed day.

There were over 200 people in attendance, with tickets sold out for both the workshops and the conference day itself. Thank you to all of you for being part of the event. I hope that like me you were rewarded by the experience, in the form of both new knowledge and fresh inspiration.

For those who weren’t able to make it, note that the workshop resources, as well as slides and videos from the talks are available via the conference website.

Here's To The Ones Who Dream

Wed, 08 Feb 2017 00:00:00 +0000

Last night I finally went to go and see La La Land. I’m happy to say it lived up to my expectations.

Over the past few years I’ve made a habit of listening to podcasts while driving to work in the morning. I find it’s a great way to redeem that time, since it allows me to connect with a wide range of different of things I’m interested in: news, tech, software, design, psychology, science, and the arts.

SongExploder falls pretty squarely into that last (arts) category. Produced and edited by host and creator Hrishikesh Hirway, it’s “a podcast where musicians take apart their songs, and piece by piece, tell the story of how they were made.” And it really is quite masterfully done. Don’t believe me? Vulture described it as “probably the best use of the podcast format ever”, and over the last two years it’s won awards for best podcast from Quartz, Vulture and The Academy of Podcasters among others. Seriously, you should check it out.

So it came to be that on the Thursday of my first week back at work for 2017, I fired up Episode 92 of Song Exploder during my morning commute. A tranquil piano theme can be heard in the background as Hirway narrates the introduction:

“The film La La Land tells the story of Mia, an aspiring actress played by Emma Stone, and Sebastian, a jazz pianist played by Ryan Gosling, both of them struggling artists in Los Angeles. The musical was written and directed by Damien Chazelle in collaboration with composer Justin Hurwitz. It’s the third film they’ve made together, and the follow-up to the Oscar-winning film Whiplash. In this episode, Justin Hurwitz breaks down a song from the film sung by Emma Stone; it’s called Audition (The Fools Who Dream).”

Less than 2 minutes in and I was already hooked. I absolutely loved Whiplash! It’s one of very few film dvds I actually own, so that probably says quite a lot. I like to think my appreciation for movies is pretty broad (probably like many people) and I love to listen to different soundtracks, but I suppose as a musician myself there is just something particularly magical to me about movies that make music their substance, as opposed to a flavourant. The conventional production order of picture and then music often has to be reversed (at least in parts), which makes for something unique. Anyway, Whiplash made a big impression. So a new film following in it’s wake was sure to be something I’d be interested in. And the fact that it also involved a jazz pianist was a serious bonus for someone like me.

Listening to Justin Hurwitz talk about the movie and the work that went into this song in particular gave me a good sense of the central theme of the film. It’s all about the struggle of chasing your dream(s). For Chazelle and Hurwitz, getting this movie made was a dream they had to chase. At times they had to put it aside, but it really was a passion project, and I think that shows in the final product. Justin speaks of producing “a little over 1900 piano demos for this movie”. That sounds like a lot of hard work and dedication to me. And Chazelle has spoken in interviews of how there “wasn’t a lot of excitement in the room when we initially pitched La La Land around town”. LionsGate were the only ones (eventually) brave enough to take it on.

There’s plenty of praise and plenty of criticism floating around online regarding the movie and it’s ending in particular, but in the interest of avoiding any direct spoilers I’m not going to dig into those details. I get that it’s probably not everyones cup of tea but I really thought it was a great movie and thoroughly enjoyed it. The contrast of traditional vs modern ideas and the push towards compromise and trade-offs is thoughtfully interwoven into many of the movie’s themes: the development of romance vs relationship, career growth vs dreams & ambitions, music then and now, and especially into the style of the movie-making itself. When you take a few steps back I think it’s all really quite brilliant.

The movie left me with much to think about and reflect on (and that alone is usually a sign of a good movie), but overall it resonated with and inspired me. A few years ago I remember chatting with my brother after he had done the Myers Briggs personality test, and having got a fair bit out of it in terms of understanding how he was wired to see things and work with different people he encouraged me to do it too. It turns out we balance each other out quite well in some respects, one example being that by comparison to him I’m quite the realist. But movies like La La Land remind me of the dreamer that lives inside of me. At the end of the day we all have dreams and ambitions. Though we may need to compromise at times, holding onto and pursuing our dreams is what will drive us to do something great. As composer Justin Hurwitz says:

“If I stop feeling like a dreamer then I don’t think my music is going to be very good”.

Halite

Thu, 02 Feb 2017 00:00:00 +0000

Halite is an artificial intelligence programming challenge. Players control a bot using the programming language of their choice. Bots fight for control of a 2D grid and the bot with the most territory at the end wins.

The company behind Halite is Two Sigma, a New York City-based hedge fund that uses a variety of technological methods, including artificial intelligence, machine learning, and distributed computing, for its trading strategies. According to the company’s website:

“Each year we like to challenge ourselves with a hard, but fun, engineering problem. It always involves programming artificial intelligence systems to battle in a head-to-head competition. In previous years we’ve built AIs to play computer games, card games, and shuffleboard against our employees and each other. 2016’s epic battle saw humans and AIs squaring off across an air hockey table.”

Isn’t that cool? Check out the video below for a recap of the “Human vs Machine” event from last year.

Anyway, back to Halite. The game was originally conceived and developed by Benjamin Spector and Michael Truell, during a summmer internship at Two Sigma in 2016. They then assisted with further development of the game leading up to the companies annual programming competition. After a very successful 3-week tournament in August, comprising about thirty-five players, the company decided to launch the competition publically for a limited time.

The public version of the competition was launched in November 2016, and is being run with the help of Cornell Tech. It was initially scheduled to finish today (2nd February), but has been extended to the 12th February. There are currently just over 1500 players on the leaderboard, with plenty of competition for the top few spots and new versions of these bots being submitted on a daily basis.

I stumbled across the competition sometime in mid-December, and was immediately hooked after looking into it and submitting my first bot. The friends and colleagues I have shared it with seem to have had the same experience, so kudos to the team at Two Sigma/Cornell Tech for making it so easy to get up and running.

Halite is suitable for a wide range of programming enthusiasts, regardless of background or level of experience. The simplest Halite bot requires just 10 lines of code, but obviously you can expect your bot to evolve into something much more complex as you iterate on your ideas.

If you think you might be interested, simply head over to the Getting Started page and follow the instructions. The basics of the game are very simple to grasp, but as they say, the devil is in the details, and there is surprising scope for depth of strategy and innovation due to the mechanics involved in expansion and combat.

For this reason, after you have worked your way through the rest of the “Basics” section on the website (Introducing Halite and Improving the Random Bot), I’d strongly advise you to read through the Game Rules page very carefully and make sure you understand what’s communicated on it. From there, the best way to fast-track your learnings and start getting competitive on the leaderboard is to read through Nick Malaguti’s excellent forum post - So you’ve Improved the Random Bot. Now what?

Getting through all of that should give you a good idea of most of the important things to consider. From there it’s up to you - iterate from that point and/or experiment with different ideas from scratch. Browse the forums and you’ll find some great discussions and further ideas to explore. As you begin to do this, you’ll want to compare the strategies you come up with by testing different versions of your bots against each other locally. To do this, I’d recommend using this Halite Match Manager.

Good luck! May the best bot win!

Welcome to R-bloggers!

Mon, 23 Jan 2017 00:00:00 +0000

Hooray! I am now part of the R-bloggers community!

R-bloggers is a blog aggregator for content created by the Rstats community. It was founded by Tal Galili towards the end of 2009, in order to help connect R bloggers with R users (and each other).

I came across R-bloggers relatively soon after it got going and have benefitted so greatly from it over the years. It has been quite something to see how it has grown and flourished in this time. Just take a look at Tal’s annual retrospectives between 2010 and 2012 which highlight the early growth of R-bloggers and you’ll see what I mean:

2009: R-bloggers gets up and running, 17 contributing blogs
2010: 143 contributing blogs, 3000 posts, 600 000 visits from over 200 countries
2011: almost 300 contributing blogs, over 1.4 million visits
2012: over 400 contributing blogs, 2.7 million visits by over 1.1 million people

These days R-bloggers continues to serve a large global audience with over 600 contributing bloggers. It is the go-to place for R news and tutorials and one of the core spokes of the R community.

Tal, I’m sure I speak for many in the R community when I say thank you for continuing to invest in and develop this resource which we all benefit from on a daily basis. Having appreciated content from R-bloggers for so many years I now look forward to being able to contribute some of my own and hope that I will be able to add value for others in the process.

Eduardo from Data Science LA did a really nice interview with Tal at the useR conference back in 2014. They discussed how he came to start R-bloggers, his thoughts on and involvement with R and the community, as well as another of his projects, R-Users: “A job board for people and companies looking to hire R users”.

Listening to it again now, two years later, I was struck by how much of their discussion is still so relevant, and Tal has some great insight and advice to share (e.g. don’t get too caught up with newer tech developments before they’re ready, unless they really solve a problem that you have).

Two of the things he mentions in particular I would like to highlight:

learn to create R packages as soon as possible
supplement your R and stats knowledge with some C++ (i.e. learn to use Rcpp)

These are two areas which have been really pivotal for me in terms of improving my knowledge and understanding of R, and empowering myself to dig deeper and do more with it. If you’re looking for good learning resources in either of these areas, I highly recommend Hadley Wickham’s books R packages and Advanced R (which has a great chapter on Rcpp).

When asked what his advice would be to an R user back in 2007, I had to chuckle at Tal’s remark not to invest in Tinn-R because it won’t be around much longer :-)

But jokes aside, whether you are a new or long time R user, it’s a great interview that is worth a listen.

Why swirl?

Mon, 09 Jan 2017 00:00:00 +0000

swirl is a software package for the R programming language that turns the R console into an interactive learning environment. Users receive immediate feedback as they are guided through self-paced lessons in data science and R programming.

I think I first came across swirl in mid-2014, while working through the early stages of the Data Science Specialization on Coursera, put together by Johns Hopkins University professors Roger Peng, Brian Caffo and Jeff Leek.

The courses in this specialization make extensive use of swirl to introduce and explore various topics in the form of interactive programming assignments in R (e.g. R programming). This is probably largely due to the fact that these three professors were involved in the early development of swirl with it’s creator Nick Carchedi who is a graduate of Johns Hopkins Biostatistics. For those who may be new to R or data science - these Coursera courses are an excellent resource which I can highly recommend.

Of course there are many, many great resources for getting started (and later improving) with R programming: try the online learning suggestions from RStudio, or take a look through some of these Quora answers and you will find a wealth of R resources: online courses, books, tutorials and more. If that’s not enough for you - Google is your friend. There is so much out there!

I’d like to focus on swirl in this post however, since I feel it doesn’t really get as much attention as it should. There are three key ways in which I think swirl has got it right as a learning platform:

it directly facilitates learning by doing
it runs from within R
it is free and open source

Learn by Doing

First is the “learn by doing” component, which is almost as crucial to learning to program as it is to learning to ride a bike, or to drive a car.

Sure, programming is mostly cognitive, but in my experience you only begin to internalize and understand concepts, syntax etc properly when you start to apply them. The primary learning you do when learning to ride a bike or drive a car happens when getting onto a bike or into a car for the first time and (usually) having someone walk you through the interface, controls etc as you experience them for yourself.

The need to learn by doing might seem obvious, but if you stop and think about it for a moment, you might find that given the wealth of great R programming books, videos, courses etc its entirely plausible that you could begin your learning path there and after being pulled in different directions end up spending very little (or no) time in R. A few days ago I was chatting to one of my cousins who has been learning R on and off for a while now and when I asked him how it was going he said he thinks his learning process so far has lacked sufficient “hands on” practice with R.

swirl is great because it forces you to learn by doing, whether that means copying code/syntax from an instruction, interrogating R’s help documentation in order to work out how to make use of a particular function, or experimenting with different options by trial and error.

Learn R, in R

Second is the “learn R, in R” component, which is what makes swirl pretty unique. There are a few other learning platforms which embed or reproduce the R console within their own web environments to produce a similar outcome, but at the end of the day, in some or other (possibly quite small) respects none can be equivalent to the reality of working in a local R environment on your own computer, whether that be RStudio (recommended!), some other IDE of your choosing, or even just running R from the terminal.

Since swirl runs within R itself, when you learn with swirl, you can work with the exact same local R environment you will later use to program in R. By way of a poor analogy, this is somewhat like the difference between taking your driving lessons in someone elses car and taking them in the car that you plan to use once you have obtained your license. If you have the option of the latter, there are certainly benefits to doing so.

At the same time I do acknowledge that there can also be some downsides to this when considered from certain perspectives (e.g. the swirl approach of embedding the learning environment within R is not as flexible as the reverse and does constrain the learning environment to some extent). However as mentioned above I think what swirl has to offer is unique in this respect and it is worth highlighting that the advantages are significant.

I am certainly not averse to other R learning platforms and realize that some have other advantages over swirl. For example, it is worth noting that the creator of swirl is now working at DataCamp, which is an excellent learning platform that offers a host of courses in both Python and R, some of which in fact still make use of swirl.

Free and Open Source

Third is the fact that swirl is completely free and open source. This means that students, instructors and other R developers can dig into both swirl itself and swirl lesson content, learn from it, and then suggest and/or contribute towards corrections and improvements over time. Being able to look behind the curtain opens up a world of learning opportunities.

For example, my first ever pull request on GitHub, was in the swirl courses repository. It was only a simple spelling correction for lesson content from one of the aforementioned Coursera swirl courses, but you’ve got to start somewhere. It’s open source projects like swirl that offer these opportunities for “getting started”.

In a little over two years since that first pull request, and I have been able to contribute in a number of different ways to various swirl projects and courses, learning as I go. A few months ago, I also decided to get started on two of my own swirl courses, which I’m continuing to work on iteratively as and when I can make time. I’ve still got a long way to go, but I’ve already learnt so much through the process, and will definitely discuss these further in later posts.

My recommendation is that you checkout swirl if you haven’t already done so, because it is worth using. The easiest starting point is the students page on the swirl website, which provides all the steps you’ll need (especially if you are completely new to R).

And if you’re a more advanced R user, why not consider contributing a course? The documentation for swirl is excellent, and it is very easy to get started creating a course. All the information you’ll need to do so is provided on the instructors page on the swirl website. Also note that the developers of swirl have created a swirl course network (SCN) in an effort to “list and organize all publicly available swirl courses”. If you have created a swirl course or are considering doing so, please take a look at this to find out how to share your course.

If you need help or would like to collaborate on a swirl course please get in touch with me. I’ll be happy to assist where and when I am able to do so.

satRday in Cape Town

Sat, 17 Dec 2016 00:00:00 +0000

In just two months time, the second satRday will be taking place right on my doorstep, in Cape Town, South Africa. I keep telling people I am looking forward to it, but to be honest that’s really quite an understatement. I am super excited!

satRdays are SQLSaturday-inspired, community-led, one-day, regional and affordable conferences around the world to support collaboration, networking and innovation within the R community.

If you don’t know what satRdays are about, here is the back story…

In early-mid 2015, Steph Locke got conversation going around the possibility of a SQLSaturday-inspired R conference after raising the idea at EARL 2015 Boston. With fantastic support from the #rstats community, a proposal was developed and later submitted to the R Consortium in January 2016.

In March 2016, the R Consortium announced their support for satRdays, with three events to be hosted initially, and plans for additional meet-ups as the R user base grows. Multiple locations were suggested for the first three satRdays and the agreed upon strategy was to host one in Eastern Europe, one in North America, and one on another continent elsewhere in the world.

Having been involved with the project early on, Gergely Daróczi agreed to organise the first satRday in Budapest. More than 2000 people from the online community voted on where the next two satRday events should take place, and Puerto Rico and Cape Town claimed the most votes for North America and the rest of the world.

The first satRday took place in Budapest on the 3rd September 2016 and was a big success, with almost 200 people from 19 countries registering for the event. The conference delegates comprised of a good mix of R enthusiasts from both industry and academia, who were treated to high quality workshops as well as invited and contributed talks from 25 local and international speakers.

So that brings us to the present, with preparations for Cape Town satRday already well under way. The conference will take place on the 18th February 2017, and the venue is Workshop 17, located in the heart of Cape Town’s beautiful V&A Waterfront.

Keynote speakers Steph Locke, Jenny Bryan and Julia Silge will be conducting workshops on the 16th & 17th February, covering the building and validation of logistic regression models, the intersection of R, Git & Github, and the development of reactive web apps with Shiny (you can find further workshop details here).

A number of other speakers have already been confirmed, with sessions planned around topics such as R in the Sciences, R and Big Data, and Applications of R in Industry. Presentations may take the form of a workshop (90min), standard talk (20min), lighting talk (5min) or poster, and proposals can be submitted here.

Note that early-bird registration closes less than a week from now, and presentation proposals will be accepted up until 20th January 2017. Below is a summary of all the key dates relating to the conference:

Event	Date
Early-Bird Registration Deadline	2016-12-23
Workshop Submission Deadline	2017-01-20
Talk Submission Deadline	2017-01-20
Notification of Acceptance	2017-01-25
Registration Deadline	2017-02-15
Workshops	2017-02-16 / 2017-02-17
Conference	2017-02-18

All this information - and more - can be found on the website. I look forward to seeing many of you at the conference!

Introducing hugo!

Mon, 12 Dec 2016 00:00:00 +0000

Roughly two months ago, I embarked on a short mission: to setup an instance of hubot for integration with Slack. Despite being armed with only a very limited working knowledge of Javascript (hubot runs on node.js and scripts are written in CoffeeScript), the tutorials seemed pretty thorough and made the setup process look do-able, so I decided to have a crack at it.

I’ll begin with my inspiration for doing this, and then provide an overview of the setup process (which I suggest you skip over if you’re not interested in the details), followed by some examples of the few things I implemented with my instance of hubot at the outset. In case you haven’t yet deduced it from the title of this post, his name is Hugo.

Motivation

As I mentioned in my previous post, Lorenz and I have been collaborating on some #rstats projects over the past few months and very early on we started making use of Slack, also using it’s Github & Trello integrations which both involve bots.

Sidenote: If you are involved in any form of regular collaboration and haven’t already had a look at Slack I highly recommend you do. We’ve found it to be hugely valuable for facilitating communication, and also a lot of fun to use, especially due to the broad possible range of integrations with other tools.

Bots are a hot topic at the moment, and the Slack team seem to be heavily invested in their support for a wide range of different bot integrations. Based on their list - which is ranked according to popularity by default - Hubot is the most popular. It was originally developed at Github to automate their company chat room and was later rewritten and made open source.

My decision to explore Hubot further was motivated by a combination of curiosity, a desire to learn, and the hope of adding further to the enjoyment I was already getting from using Slack. Hubot was immediately appealing to me since it is completely scriptable, and it’s popularity and solid documentation provided further affirmation that it was worth giving a try.

Setup process

As I alluded to above, the team at Github have provided a detailed set of instructions on the docs page for hubot, so that’s where you want to begin if you’re looking to give this a try yourself. Here are the essential steps:

Install `node.js` and `npm` (if you don’t already have these)

Installation instructions for OS X/Windows/Linux

Install the hubot generator using `npm`

npm install -g yo generator-hubot

Create a new instance of hubot

From a new directory e.g. myhubot, run:
yo hubot --adapter=slack

This script will prompt you to describe the bot you are going to build, and create a file that NPM can use to help manage your project.

Setup a Custom Bot on your Slack team

Navigate to the Custom Bot creation page and click on Add Configuration. Then follow the prompts in order to register your bot and create a token. This token can then be used to allow your instance of hubot to log into your Slack team as a bot.

Run hubot

You can then run your instance of hubot locally using the run script below (note that you’ll need to copy-and-paste your token in from the previous step):

HUBOT_SLACK_TOKEN=xoxb-YOUR-TOKEN-HERE ./bin/hubot --adapter slack

By default, hubot should then join your #general channel within Slack, where you can test it out e.g. with:

<bot_name> pug me

Deploy your bot

Once you’ve verified that hubot is working locally, it is a good idea to deploy your bot somewhere e.g. Heroku (otherwise you’ll need to have hubot running permanently on your local machine which probably isn’t practical).

Again, the Github docs are great and provide setup instructions for deploying hubot on Heroku here. They also have instructions for deploying on Azure, along with more general instructions for deploying on Unix or Windows (e.g. via another cloud computing service such as Digital Ocean).

If you go with Heroku and are making use of their free usage tier, note that free dynos sleep after 30 minutes of inactivity. Since interaction is done through chat, hubot has to be online and in the room to respond to messages. To get around this, you can use the hubot-heroku-keepalive script, which can be configured to keep your free dyno alive (given the limitations on the free plan, more than 18 hours/day will require an upgrade).

Add your own custom scripts

So hopefully by this stage you have got your instance of hubot configured and deployed, so all that’s left to do is to make the bot your own! For a start, have a look through the scripting overview provided here, and then get going adding your own custom scripts to your bot to give it new functionality, personality etc.

Hugo in action

So below are some examples of Hugo in action based on a few simple scripts I added after going through the above setup process.

Hugo taunts us whenever we mention Hadley Wickham (which is fairly often given that we’re mostly working on #Rstats related stuff - such is the man’s influence on the R community)

He makes use of lolcats for many of his responses (because I really like lolcats)

He celebrates when we get successful build notifications from Travis

And occasionally the celebrations don’t involve lolcats…

Strangely enough I couldn’t spot any failure notifications within our Slack channel but when there is one, Hugo will chime in with an appropriately themed image - which again usually has a high chance of being a lolcat like one of those shown below.

Further Thoughts

All of the above examples were straightforward to script as they just rely on rudimentary regular expressions, but I was somewhat surprised at the entertainment value we got out of these simple eavesdrop and respond behaviours from a chat bot. Hopefully I’ll be able to revisit this over time and explore some more advanced interactions. There are also loads of hubot scripts all over Github which I am yet to explore (e.g. here).

If you have any ideas or suggestions for Hugo please feel free to comment below or take a look at the repo and submit a pull request - given the limited amount of time I’ve invested so far and the fact that I’m just winging it I’d love to learn from someone who actually knows what they’re doing!

Breaking the Silence

Thu, 08 Dec 2016 00:00:00 +0000

In April of 2016, I made a new friend. His name is Lorenz and he is from Switzerland. We met one evening when I arrived to pick him up for a small church group meeting, and he ambled up to the driver side door only to be reminded - by my presence in the drivers seat - that South Africa is not part of the two thirds of the world’s population that drive on the right hand side of the road. Anyway, I believe it was while chatting on our way home later that evening that he first mentioned R and that immediately took our budding friendship to the next level :-)

He had made his way over here to South Africa to undergo a 3 month internship, partnering with the Community Organisation Resource Centre (CORC) and Shack / Slum Dwellers International (SDI) to collect and analyse data on informal settlements in the Cape Flats. During the course of his stay in Cape Town, we squeezed in occassional chats about #rstats in addition to the odd email back and forth. We also ventured out to the local R User Group meetup on one occassion to hear from the legendary Dirk Eddelbuettel.

Mutually encouraged by our passion for R and statistics, our friendship blossomed quickly and we exchanged thoughts on various topics, but never really progressed beyond just talking about things, and the time soon came when Lorenz was due to move on to Kenya to continue with another internship there. So we said our goodbyes with the hope of being able to reconnect (somewhere) one day, and that was that.

And then one day towards the end of August, completely out of the blue, Lorenz shared this meme with me via e-mail:

Ironically, at that point in time my 3 year old GitHub account was effectively lying dormant, with a history of 1, 9, and 4 contributions in 2013, 2014 and 2015 respectively, and a majestic 0 contributions for 2016. Anyway, we quickly struck up a conversation over email to catch up, and before long, he directed my attention towards a Github repo he had created, which amongst other things outlined some early ideas for an R package. He invited me to collaborate, and somehow just 3 days later I found myself active on Github.

Over the 3 month period since then, I have notched up 275 Github contributions, ranging from early progress on collaborative projects with Lorenz, to a few personal projects, branching out to a number of issues and pull requests on R packages and books from other authors, and ultimately to this blog itself. It has been really rewarding to start getting a little more active in the R community and of course to learn and grow in doing so. I was first introduced to R at university in 2005 - so in hindsight it really shouldn’t have taken me so long to get to this stage. Clearly Lorenz was the catalyst I needed to get me going, so I am really grateful to him for that.

So when did Lorenz start using R? Almost 2 years ago now, he tells me. Take a bow son!

About Me

Mon, 01 Jan 0001 00:00:00 +0000

Hi, I’m Jon.

I’m a data scientist, R enthusiast, hobbyist musician, hopeful runner and lifelong learner. I work as an automation engineer at Libryo, where I am engaged in the use of computation to assist with content ingestion, analysis, and anything else it might be able to help us with. I live in the beautiful city of Cape Town and I am proudly South African.

Please feel free to connect with me! Either send me a direct message using the form available on the Contact page, or have a look at the footer for my social media, Github and other links.

For a more lengthy bio, keep on reading below!

Bio

I grew up in the heart of Kwa-Zulu Natal, South Africa, in a small city called Pietermaritzburg - aka ‘the sleepy hollow’. Located about 90km away from the heart of Durban, PMB is probably best known for being the (joint) home of the world’s largest and oldest ultramarathon - the Comrades Marathon - which is run between these two cities.

My dad and I after Cape Town 12 (2016)

Running is probably the simplest sport there is, but certainly not the easiest, and a race of this distance presents a substantial challenge - both physically and mentally. I grew up watching the race on tv, out alongside the road and sometimes even looking over the wall from the garden (when the race went past our house). To this day, I remain in awe of the thousands of people who complete this journey each year. One of those people is my father, who has 13 Comrades medals to his name. I remember him once saying of the race, “you will learn things about yourself you wouldn’t discover another way”. I think I’d like to complete the Comrades Marathon one day.

But don’t let that give you the wrong impression - most of the stuff that I’m drawn to generally falls much more towards the mental end of the physically vs mentally demanding spectrum…

I’ve always been something of an academic. I thrived in the structured learning environment that school, and later university provided me. Learning is something that I really enjoy and I hope to be someone who never stops learning, both intellectually and otherwise.

I left PMB in 2005 to study at the University of Cape Town - tackling three years of actuarial science before switching streams to graduate with a BSc in Mathematical Statistics. During my honours the following year I discovered the world of Operations Research and Decision Science in particular which really piqued my interest. So after completing my BSc Honours in Statistics, I ended up undertaking an MSc in Operations Research, sparked by stakeholders from the Stormers rugby franchise in the Western Cape.

This resulted in a research project which combined aspects of sports science, decision theory, and a whole lot of modelling and simulation in R. Although I was first introduced to R at the outset of my undergraduate years at UCT, it was here where I began to develop my enthusiasm for R as a programming language.

During the latter part of my MSc, in 2011, I started working in the software industry as a data analyst for RightShift - a tech startup developing online games. The company experienced rapid growth in it’s player base, which gave me good exposure to some of the challenges of managing and manipulating large amounts of data. Our core tooling included MySQL, SQL Server, and much of the Microsoft Business Intelligence stack - SSAS, SSIS and SSRS - along with a good deal of Excel. Over and above that I also did a few projects which leveraged R to embed dynamic reports and visualizations on the company wiki, and for some Shiny apps.

I completed my MSc in 2013, and produced two publications (see here) shortly thereafter with the help of my supervisor Dr Ian Durbach. RightShift was acquired by Derivco in ²⁰¹⁵⁄₂₀₁₆, which saw me transition into more of a product design role, but with a particular focus on modeling all math related aspects of game design. Then in 2017, I joined one of the data science teams at Derivco, which allowed me to be more involved facilitating product analysis on an ongoing basis.

At the beginning of 2018 I joined Libryo - a legal-tech start-up founded in London by three South African entrepeneurs. Libryo is an organisation on a mission to make it easier for companies to know the law, and keep it. We’re on an exciting journey, and my part in it is only just beginning!

In my spare time, I enjoy running, a myriad of different forms of moving images (sport, e-sport, movies, tv series etc), listening to music, and - when we can find time - making music with my brother and other friends (for more info see ryancalder.co.za). I am also a keen sound engineer with a casual interest in various aspects of both studio and live production. Along with all of this I am also trying to be more involved in R and other open-source development, and hope to continue growing my technical skillset in pursuit of becoming a more rounded data scientist.

Contact Me

Mon, 01 Jan 0001 00:00:00 +0000

Curriculum Vitae

Mon, 01 Jan 0001 00:00:00 +0000

Jon Calder

“No man should escape our universities without knowing how little he knows.” – J. Robert Oppenheimer

Experience

February 2018 - Present

Automation Engineer; Libryo

Research and development on how to best automate & augment processes on the Libryo Platform
Database management, maintenance and support
Data analysis, dashboarding and reporting
Software development and maintenance of development, staging and production environments
Strategic support and consultation around technology decisions
Assist with the management and implementation of strategic objectives

September 2017 - Present

Guest Lecturer; University of Cape Town

Data Science for Industry is a 24-lecture, 12 credit module in the MSc in Data Science program at the University of Cape Town
Teaching material for the development of (1) R packages and (2) Shiny apps which comprise the communication section of the course

September 2016 - January 2018

Data Scientist; Derivco - Cape Town

work with product and design teams to inform and assist with product and feature design
assist with both functional and technical product documentation
conduct ongoing industry and competitor research to inform the design of product features
determine product data requirements and support delivery of data, monitoring and analysis to operators
expand on underling game maths and service maths requirements for product features as they are designed, developed & iterated upon
develop tools to model and simulate the impact of changes to product configuration

October 2011 - August 2016

Data Analyst; RightShift

design, build and maintain ETL process to update database environments as well as deploy and manage SSAS cubes, and optimizing these processes in order to allow for rapid scaling}
create, manage and maintain reports to service the daily, weekly and monthly requirements of the business in various respects; as well as reporting directly to high level management
measure application data in order to identify trends in player behaviour, virality, revenue and associated key metrics
track and monitor general application performance across multiple platforms and identify potential bugs and problems
assist marketing and customer support teams with additional tailored reporting and analysis to provide specific insight into various interventions and offers
perform ongoing analysis in order to inform decision making on game and meta-game development and configuration utilizing A-B testing

2008 - 2011

Student Tutor; University of Cape Town, Department of Statistics

Tutored undergraduate statistics courses from 1st year through to 3rd year
Responsibilities involved:
- facilitating weekly class tutorials
- invigilating & marking of tests and exams
- ad-hoc one-on-one hot-seat tutoring

2008 - 2011

Senior Tutor; Axiom Tutoring

Tutoring of mathematics and statistics from high school through to undergraduate university level
e-mail administration & allocation of tutors to students
one-on-one & small group tutoring
preparation & typesetting of notes and worksheets

2007 Summer

Student Internship; Momentum Collective Benefits

worked as part of a group recording, capturing and compiling historical data for later use by the actuarial team to re-evaluate exposure to risk
gained insight into the insurance/group life industry, and experience in the commercial use of Microsoft Excel and VBA macros to automate data compilation and processing tasks

Publications

Modelling uncertainty in stochastic multicriteria acceptability analysis

Ian N. Durbach, Jon M. Calder
Omega (October 2015)
Available at: http://dx.doi.org/10.1016/j.omega.2015.10.015

Decision Support for Evaluating Player Performance in Rugby Union

Jon M Calder, Ian N. Durbach
International Journal of Sports Science & Coaching (February 2015)
Available at: http://dx.doi.org/10.¹²⁶⁰⁄₁₇₄₇-9541.10.1.21

Technical Experience

Software & Programming Languages

R, C++, Java & Python

Proficient with R, some experience with C++, Java, Scala & Python

Databases & Scripting

MySQL, T-SQL, Excel, VBA, SSAS, SSMS, SSRS + limited experience with MDX

Document Preparation

Word, Powerpoint, Markdown, LaTeX

Statistical Packages

Some experience with Statistica, SPSS, EViews & Genstat

Web Development

Some experience with HTML & CSS, PHP, Javascript, Wordpress

Education

2009 - 2013

MSc in Operations Research (with distinction); University of Cape Town, Department of Statistics

Decision modeling, decision support, logical fallacies & reasoning, performance analysis in sport.

Player performance evaluation in rugby using Stochastic Multi-Criteria Acceptability Analysis with simplified uncertainty formats

Masters dissertation

Supervisor: Dr Ian Durbach

Description: This study investigates the use of SMAA as a decision support tool for rugby coaches by making use of game data to inform possible rankings of players within positional profiles based on preference criteria.

2008

BSc (Honours) in Statistics; University of Cape Town, Department of Statistics

Operations Research, Decision Modelling, Theory of Statistics, Multivariate Analysis, Econometrics, Portfolio Management.

Application of Rasch Analysis to the Assessment of Grade 6 Literacy and Numeracy In The Western Cape

Honours dissertation

Supervisors: Prof Tim Dunne, Kutlwano Ramaboa

Description: Rasch analysis is utilized to assess the validity of a numeracy and literacy test administered to grade six learners from over 1000 schools in the Western Cape in 2007. Rasch analysis is introduced in the context of a background to modern test theory, and the data are then examined with respect to invariance across the latent trait, targeting and reliability, item characteristics and differential item function.

2005 - 2007

BSc in Statistics; University of Cape Town, Department of Statistics

Undergraduate courses included Statistics, Mathematics, Computer Science, Economics, Accounting, Finance and Actuarial Science

2000 - 2004

Carter High School; Pietermaritzburg, South Africa

Finished fourth in the Vulindlela/Pietermaritzburg district and third in the Midlands Area (awarded by the Kwa-Zulu Natal Department of Education)
Awarded the dux prize for top scholar in Grade 12, with distinctions for English, Afrikaans, Mathematics, Biology, Physical Science and Geography
Awarded Academic Colours in Grade 10 and Academic Honours in Grades 11 and 12.
Awarded Effort Certificates from Grade 8-12

Human Languages

English (home language)
Afrikaans (2nd language)
French (beginner)

Achievements and Extra Murals

Music

Studied classical piano - theory and practical - at the Music School in Pietermaritzburg (1998-2003)
Was graded at various Eistedfords during this period and performed at additional concerts
Achieved a grade 6 performance level

Sport

1st Team Outdoor hockey (Captain) and 1st Team Indoor hockey at Carter High School
Awarded half colours
Other sports played (mostly club/social) while at school and university include: soccer (indoor and outdoor), action cricket, cricket, squash, touch rugby and ultimate frisbee

Community Service

Since 2017: Member of RWeekly
Since 2017: Co-organiser of satRdays in South Africa (e.g. see satRday CT 2018)
Since 2017: Mentor and maintainer of the R track on exercism.io
2007: 1st Year mentor, Smuts Hall (UCT Residence)
2004: Grade 8 mentor, Carter High School
2003: Member of Matric Dance Committee, Carter High School

Interests & Hobbies

Piano/Keyboard
Sound engineering
Running

Projects

Mon, 01 Jan 0001 00:00:00 +0000

Open Source Projects

rweekly.org

Weekly Updates from the Entire R Community

I am an active contributor and part of the core team

exercism.io

Download and solve practice problems in over 30 different languages

I am currently a maintainer of the R track

R packages I’ve collaborated on

styler: Non-invasive pretty printing of R code
pRojects: A tool for setting up R projects and related infrastructure
refactor: Better factor handling for R
exercism: An R package for interacting with the exercism.io API
precommit: pre-commit hooks for R projects
strcode: Structure your code better
gitsum: Summarise the history of any git repository

Other personal projects

exercism track config: Shiny app for visualizing track config on exercism.io
regular expressions: Swirl course on regular expressions in R
diagrammer: Swirl course for the DiagrammeR package
hugobot: A chat bot built on the Hubot framework

Jon Calder

Naming things is hard

Disclaimers

Background Context

Discussion & follow-up

The spark of statistics

Data Science Podcasts

R track on exercism.io

satRday Cape Town

Here's To The Ones Who Dream

Halite

Welcome to R-bloggers!

Why swirl?

Learn by Doing

Learn R, in R

Free and Open Source

satRday in Cape Town

Introducing hugo!

Motivation

Setup process

Install node.js and npm (if you don’t already have these)

Install the hubot generator using npm

Create a new instance of hubot

Setup a Custom Bot on your Slack team

Run hubot

Deploy your bot

Add your own custom scripts

Hugo in action

Further Thoughts

Breaking the Silence

About Me

Bio

Contact Me

Curriculum Vitae

Jon Calder

Experience

Publications

Modelling uncertainty in stochastic multicriteria acceptability analysis

Decision Support for Evaluating Player Performance in Rugby Union

Technical Experience

Software & Programming Languages

R, C++, Java & Python

Databases & Scripting

Document Preparation

Statistical Packages

Web Development

Education

Player performance evaluation in rugby using Stochastic Multi-Criteria Acceptability Analysis with simplified uncertainty formats

Masters dissertation

Supervisor: Dr Ian Durbach

Application of Rasch Analysis to the Assessment of Grade 6 Literacy and Numeracy In The Western Cape

Honours dissertation

Supervisors: Prof Tim Dunne, Kutlwano Ramaboa

Human Languages

Achievements and Extra Murals

Music

Sport

Community Service

Interests & Hobbies

Projects

Open Source Projects

R packages I’ve collaborated on

Other personal projects

Install `node.js` and `npm` (if you don’t already have these)

Install the hubot generator using `npm`