Technical posts

The following is a modest summary of a single chapter of David Deutsch’s book on reason and knowledge, The Beginning of Infinity, and several adjacent concepts. The specific chapter, entitled “Artificial Creativity”, examines a source of possible confusion surrounding artificial intelligence. If this simple summary piques your interest, do yourself a favor and read Deutsch’s full work.

In this blog post, I’ll first summarize David Deutsch’s analysis of the generation of knowledge and all its inherent complexities. After that and with it as a foundation, I’ll pivot into Deutsch’s commentary of artificial intelligence.

Part 1: The Infinite Reach of Knowledge

One of David Deutsch’s main theses in The Beginning of Infinity is that knowledge is the currency of the universe. With sufficient knowledge, he argues, a person can accomplish any task that does not violate the laws of physics (such as traveling faster than the speed of light). This is true because, as Deutsch writes, "There can be no regularity in nature without an explanation, for to believe in an explanation-less regularity is to invoke the supernatural."

Upon first reading, I had some difficulty parsing this claim. However, with a bit of additional thought I was able to work through it. First, imagine being an early human and stumbling upon some new "regularity" in the world — say, that water seems to always run downhill, or that the four seasons follow a predictable schedule. Such observations may seem hopelessly mysterious while brand new, but with hindsight we know that those two particular regularities each have concrete explanations (gravity¹, and the orbit and tilt of Earth, respectively). Deutsch's point, which almost seems too simple to state, is that all regularities will have such explanations (whether we immediately know them or not!). This extrapolation is what unlocks the full power of explanation, and thus knowledge.

Everyday, I wake up feeling snowed in by emails, and I work at a company (Zapier), that makes every effort to reduce email congestion by favoring other forms of remote communication. Still, the emails are too many, which is why I set out to both filter the signal from the noise, and then send that signal straight to Slack.

Signal, over here. Noise, over there.

I use the following Gmail filters to delineate emails I care about from... the rest. I know that 9 out of 10 relevant emails are from GitHub, so I focus there.

First, I created two labels, GitHub, and GitHub (Mentions).

Then, I added these three filters to route all email from GitHub:

Gmail Labels

Correctly mocking external entities in your unit tests is a bit tricky syntactically, and I have to look up the exact incantations to type every single time they're needed. Thus, I'll make a quick sales pitch about mocking, quickly explain it to the best of my current understanding, and end with a sequence of tricks to be updated as I learn more. In essence, it's my own personal cheat sheet, stored in an accessible location I'm unlikely to misplace.

Everyone knows about Django's Q objects and how to chain them together for complex lookups. The syntax is straight forward.

That generates this unexciting SQL:

But what happens when you have an unknown amount of filters?

At Stik.com, we're using Django to supply a RESTful interface to our database. Our support staff is also using its admin for simple ticket items. Fortunately for the business, but unfortunately for the Django admin, we have lots of tables with hundreds of millions of rows, and a few in the billions. We really didn't want to spend the time writing our own admin section for support staff, so the Django admin was a godsend... until we rolled it out to production and saw admin pages never respond ever. Read on to see what our problem was and how we fixed it.

The problem(s):

Large tables in InnoDB may not be able to even render their change_list view due to (possibly 2) expensive COUNT(*) queries. With tables of 100M+ records, this can take 15+ minutes to complete, even on fast servers.

Before we get started, I'm assuming you've mastered or completed the first part of the tutorial, your site code itself is complete, your virtual environment is functional in local development, and you have your domain name pointed the IP of the EC2 instance we just made. If any of these assumptions are false then you are probably getting ahead of yourself.

This is super fast and easy. Here we go:

Step 1: Literally the first step

Create and log into your http://aws.amazon.com account. Yadda yadda yadda.

I've had the same debate at every company for which I've ever worked, and the setup is always the same. Legacy tools are in place that technically get the job done but the annoying, but newer programmers are calling for new methods and tools. Some of the older dogs and management don't want to change, citing the overhead and cost of switching gears -- while the voices of change insist that a smoother, faster, easier, and more stable iteration process awaits at the end of that overhead. On healthy teams this leads to a lot of passionate, respectful debate. On unhealthy teams it spawns gnashing of teeth, finger pointing, and name-calling.

First, allow me to make a few assumptions:

1. You're running Mac OS X 10.8.
2. You have MAMP 2.x installed (and this is preventing MySQL-python from installing).
3. You're going to run your Django site within a virtualenv (If you weren't planning on doing this, now's a great time to change your mind!).
4. You're tired of PHP and you want to use Django, but because of other projects are unable to remove MAMP and start fresh.

Why it's failing:

MySQL-python won't compile because it needs the MySQL headers which are not in the expected location under a MAMP install. Luckily, this isn't hard to remedy.