Last night, I gave a talk at the very first Reproducibility and Productivity in Data Science (“RAPIDS”) meetup on the relative unimportance of code, the state of data management, and what we at Prodo.AI are trying to do about it.

As (almost) always, I wrote the majority of the content ahead of time, so if you weren’t there (and I know you weren’t), please take a look. I think you’ll enjoy it.

It focuses on an open-source tool we’ve been building to help us manage our own data problems, Plz, which I also encourage you to check out for all your data science and machine learning needs.

Here’s how it starts:

There’s a misconception that software developers have.

We think software is important.

Because of this, we often focus the majority of our attention towards the code that makes up the software. You know, the Python, or Java, or Ruby that the programmers write. We spend a lot of time grooming it, making it more maintainable, more readable, and generally prettier.

Unfortunately, we spend almost no time at all regarding the data that flows through our programs. Typically we’re content with a few examples for our test cases, a couple of edge cases so that we can verify our systems work in some sort of staging environment… and that’s about it. We tend to look at “production” data when there’s a bug.

This is kind of funny, when you think about it, because the software isn’t important. The data is.

Let me know your thoughts, and if you do try Plz, please file as many issues as you like. We want to help bring data science into the 21st century.

Check it out.

Originally posted as a Twitter thread.

I’ve been thinking a lot about serverless architecture and I’ve realised what bugs me about it.

This is gonna be a long one. Not gonna lie, I wrote a draft first. If I hadn’t, this thread would end right before it got interesting because I’d get distracted by a butterfly.

At its heart, serverless/FaaS/lambda/whatever-you-call-it works for me because it ties action to cost. The more I do (or the more inefficiently I do it), the more I pay. More users, more money. Gojko Adzic wrote about this years ago. It’s not new.

The problem, I find, is that in doing so, it breaks your application apart. Now we have many applications. They all work with HTTP, or message queues, or some other data interchange layer I don’t care about when I’m designing functionality for a user.

I was listening to Ron Jeffries and Chet Hendrickson on the Cucumber podcast, and I heard something that helped me reframe agility and craft in a way that was very meaningful to me.

He was prompted by Sal Freudenberg talking about asking for permission to use techniques such as test-driven development, rather than treating it as part of the job.

Chet Hendrickson: I think you’re right; it seems odd that we ask permission. We believe we have to have permission to do good work. I believe that on the first day of a new greenfield project, the codebase is perfect. It does exactly nothing, and it does it in zero lines of code.