Theoretical Fun

The company I work for, like many other tech companies, has a thing where you get to spend a certain amount of your time working on projects of your own choosing. The most well known-example of this is Google’s now-defunct 80/20 rule but there are plenty of others.

This past week I had an opportunity to do some of this, and ended up brushing off my very theoretical computer science roots and digging into some graph theory in order to solve a very practical problem. Of all the things I get to do in software, this is far and away my favourite, and I had a blast working out the possible solutions.

In my career so far I’ve been lucky enough to have several opportunities to solve problems of this nature and then build out the solutions, and even publish some of them to talk about. It is extremely rewarding work. There are of course a few I can’t talk about as they are still internal, but here’s a sort of “hall of fame”: two other problems that I had a ton of fun solving at a computer-science-theoretical level and also resulted in really awesome (and sometimes popular!) practical solutions.

Wheel-of-Fortune Memory Allocator

https://github.com/eapache/wof_alloc

Arguably the overarching project here was Wireshark‘s wmem framework which I also designed and built, but while wmem was a lot of fun to work on it wasn’t particularly novel; it was just another memory pool library with a few minor tweaks to suit Wireshark’s use case and migration path. However, Wireshark’s unique memory allocation pattern demanded a custom allocator algorithm which I eventually extracted into this library.

As far as I can remember this was the first practical problem of this nature that I got to take the entire way: from identifying the problem, designing the solution, to building and shipping it, and I am forever grateful to the Wireshark core team for trusting a (at the time) university student to go and rewrite the deepest foundational layer of the project. At just under 1000 lines of pure C this allocator has been shipping in released versions of the project for 5 years now effectively unchanged, and the standalone version gets a fair number of hits on GitHub.

Sarama Async Producer

https://github.com/Shopify/sarama/wiki/Producer-implementation

A little while later, my first major project at my current job was to write a golang client for the Apache Kafka platform. Like wmem, a lot of it was pretty standard work (in this case implementing message formats and managing TCP connections). However, the design of the producer implementation was another one of these instances of theoretical fun. The requirements on a performant Kafka producer are complex and varied, balancing messages and batching across topics and partitions and brokers, while also maintaining message order in the face of partial failures and retries. I can’t tell you how many evenings I spent wandering around in a fog, stopping occasionally only to make tweaks to weird hypothetical flow diagrams on my whiteboard.

To a certain extent this project was less practical than the allocator; the reference java implementation was open-source and would have been fairly straight-forward to copy, but I did it myself for two main reasons:

  • I was working in Go, which provided me with very different concurrency primitives and indicated a very different design.
  • I was young, and not as aggressively pragmatic at the time.

To a certain extent it was also less successful; it has continued to evolve over the last four years and there are still parts of the design that I’m not entirely happy with. That said, it’s also far and away the most complex single algorithm I’ve ever built, and judging by Sarama’s popularity, it’s doing its job just fine in practice. I even managed to get a conference talk out of it.

GraphQL Warden

https://github.com/rmosolgo/graphql-ruby/issues/1333

If you recall my recent post on what I’ve been working on recently for my “real” job, it won’t surprise you that my most recent project is tangentially related to that. This one hasn’t been built yet (and my own pragmatism means it may never get built) but even from a design perspective I had a lot of fun with the process here. I definitely got way too excited when I realized that I could represent the two halves of the process as symmetric reachability problems in a graph-theoretical isomorphism of the schema.

And just maybe, the next time I get a chance to spend some “me time” at work, I’ll write an actual implementation of it!

Advertisements

Other Opinions #65 – Trying to make sense of Bourdain

https://www.popehat.com/2018/06/10/randazza-trying-to-make-sense-of-bourdain/

There is something to this one that seems to strike home for me. The idea of “success” as ultimately unsatisfying is something that doesn’t get talked about enough. Give me a family and community any day of the week.

Disclaimer: I don’t necessarily agree with or endorse everything that I link to. I link to things that are interesting and/or thought-provoking. Caveat lector.

This Means War – Is Trudeau Ready?

(Betteridge’s law says “no”.)

Last week, American president Donald Trump announced two major tariffs on Canadian aluminum and steel. This was a watershed moment in the relationship between the two countries, who have long shared a highly cooperative diplomacy and a tightly integrated economy. We are now in the midst of a trade war, and that requires a major shift in perspective. In the upper echelons of business, much is made of the difference between a peacetime CEO and a wartime CEO. The analogy is drawn from politics of course, and now we have an opportunity to see the original essence of that analogy in practice, as Canada shifts diplomatically from peace to war. The question of the moment then, is whether Justin Trudeau is ready for it?

Trudeau’s first two years as Prime Minister have been characterized by positivity, just like his campaign. He continues to talk about opportunity, growth, and inclusion every chance he gets. He is, in other words, the very picture of a peacetime leader. But peacetime leaders tend to get crushed in times of war; just ask Neville Chamberlain, the British Prime Minister leading up to World War Two. Like Trudeau he talked a good early game, pushing back against Germany when possible but also accommodating them in the name of a broader peace. Like Trudeau, Chamberlain combined calculated displays of strength and resolve with a general flavour of good will. His policies were widely popular among the electorate. And like Trudeau, Chamberlain was not ready for war.

Barely nine months in to the second world war, after a string of disasters culminating in a wholesale retreat from Norway, Chamberlain resigned. His replacement was none other than Sir Winston Churchill, a war-time leader if there ever was one. Churchill was everything that Chamberlain was not: one of the world’s greatest orators, direct, focused, and completely unwilling to back down. Where Chamberlain was diplomatic, refined, and heavily invested in keeping the peace, Churchill was a leader with only one goal: to win the war. He stuck to his guns even when his choices were massively unpopular, which in fact they were at the time. He was appointed Prime Minister on Chamberlain’s resignation, not elected, and lost the post at the very next public election.

So what does this mean for Canada today? I suppose it is possible that Trudeau will be able to pivot, transitioning from a peacetime role to a wartime one. If he can pull that off then he will likely go down in history as one of Canada’s greatest leaders. However, it seems unlikely. The required shift in perspective would be very much out of character, and his initial response to the tariffs has been… tepid. Retaliatory tariffs, yes, but dollar-for-dollar; literally a call, not a raise, and one that (per Coyne) will harm Canada far more than it has any persuasive power over the United States.

If Trudeau cannot pivot, then we are in for a rough couple of years. Barring a true catastrophe, Trudeau is unlikely to resign before next year’s election, but there is no-one currently on the ballot with the necessary capabilities. The NDP have always been a peacetime party, and Jagmeet Singh is no exception. Andrew Scheer was elected leader of the conservatives as a direct response to Trudeau, in an effort to win back some of the voters turned off by Harper’s determination and negativity. Ironically for them, (and for me, as somebody who voted against him in the 2015 election) Stephen Harper is now the Prime Minister we need.

Up until this moment, I have been generally happy with Trudeau as Prime Minister; he was a welcome breath of fresh air after so many years of Harper, and is generally closer to my positions on policy. Today, I wish we’d given Mr. Harper one more term.

Other Opinions #63 – The case for swagger: Canadians have a lot to be proud of

https://www.theglobeandmail.com/business/commentary/article-the-case-for-swagger-canadians-have-a-lot-to-be-proud-of/

Disclaimer: I don’t necessarily agree with or endorse everything that I link to. I link to things that are interesting and/or thought-provoking. Caveat lector.

Capital is the New American Aristocracy

Do you know that feeling, when some person or article says something with which you’ve agreed for years but hadn’t ever been able to properly articulate?

It’s one of the delusions of our meritocratic class, however, to assume that if our actions are individually blameless, then the sum of our actions will be good for society.

The Atlantic think-piece that this is from (The 9.9 Percent is the New American Aristocracy) is of course excessively long, but worth reading if you’re interested in that kind of thing, especially if you enjoyed my essay on Brexit, Trump, and Capital in the Twenty-First Century. While the Atlantic piece takes a rather different tack to get there, it ends up at roughly the same set of possible recommendations, and I think the underlying thesis is the same: there are a lot of not-directly-monetary ways in which the future prospects of the working class have suffered over the last few decades.

Money may be the measure of wealth, but it is far from the only form of it. Family, friends, social networks, personal health, culture, education, and even location are all ways of being rich, too. These nonfinancial forms of wealth, as it turns out, aren’t simply perks of membership in our aristocracy. They define us.

This was basically one of the theses of my essay, so maybe the Atlantic article doesn’t tack that far from it after all.

Originally, I focused on the value of unskilled labour as the primary change in recent generations. I still think this is generally true, but the Atlantic piece focuses elsewhere: on the walls that the upper classes are building around the other forms of capital. Education is a form of capital, but with the increasing class-segregation of top university admission processes it becomes less accessible to those at the bottom of the heap. Location is also a form of capital, but as the cost of living skyrockets in prime locations (most notable Silicon Valley) that too becomes inaccessible. It seems inconceivable today that a poor person from the slums of Detroit could fight their way into a good university, then afford to move to San Francisco in order to be able to work a decent job. But if we really want social mobility to be a feature of our economy then that’s the story we have to enable.

Finding A Solution

In 2016, I mentioned a basic income as one possible partial solution, with the caveat that it didn’t seem politically or economically feasible. Things have changed a lot in the last two years. From Finland, to California, to Ontario, a number of organizations and governments have started pilot projects of the idea. Perhaps more interestingly, in the current Ontario election, all three major parties have pledged to continue the experiment, effectively guaranteeing a path forward regardless the winner.

In addition to the political will that has sprung up recently, the economics suddenly seem more favourable as well. The cost of expanding Ontario’s experiment to the entirety of Canada has been pegged at only $43 billion, which is eminently affordable in the context of the many hundreds of billions of dollars that Canada already spends on various programs. And proponents are quick to point out that that number doesn’t even include the expected savings in health care, incarceration, and other government services which typically result from lifting people out of poverty.

The purpose of this essay wasn’t originally to sell people on the idea of a basic income, so I’ll leave it at that, but it does seem to me like an extremely promising approach. I’m looking forward to the results of some of the pilot projects, and in the mean time I’m going to do a bit more research and try to raise the profile of this idea.

I hope you’ll join me.