Grand visions of holiday productivity, including wrapping up all outstanding book notes, were predictably optimistic. Continuing on:
Boomerang: Travels in the New Third World
Six Easy Pieces
CODE
One of my favorite reads of 2011 was James Gleick’s The Information
Aaron Swartz killed himself yesterday, 2 years after being arrested for illegally downloading journal articles from JSTOR on the MIT campus, and 4 months after prosecutors filed a superseding indictment that expanded the charges from four to thirteen felony counts. I didn’t know Aaron personally except for a small email exchange, but I always admired his work and spark from a distance. His loss hurts, and this post is surely in part borne out of that hurt, but the thoughts are not new.
The prosecutors in this case, Carmen M. Ortiz, Scott L. Garland, and Stephen P. Heymann, deserve to be shamed. I do not blame them for Aaron’s death; that was his own tragic doing. But I emphatically blame them for pressing these charges in the manner they did, threatening Aaron with decades in jail and millions of dollars in fines for a non-violent offense against two private parties under a law that another federal circuit, the 9th, has said should be adjudicated in civil court. JSTOR has publicly stated that their legal interest ended when they got their documents back, and the damage to MIT was a few days inconvenience. Where is the compelling public interest that spurred the government to be at all involved? What possible reading of the crime demands impoverishment and a life in federal prison for justice? The pursuit of this case by these prosecutors was, plainly and simply, disgusting.
It’s hard to decide which larger issue to focus on. Society’s witch-hunt reaction to crimes involving computers? The wrong-headed incentives that drive prosecutors? The dying industries that keep throwing money at politicians to keep these laws on the books? Surely all of these and more are a part of this conversation, but it’s all been said and will continue to be said. Right now we just sit back, exhausted, and mourn the wholly unnecessary loss of a brilliant and inspiring individual.
Rest in peace, Aaron.
I’ve fallen hopelessly behind in my book reports, so I’m going to dribble out the backlog in short rundowns by the end of the year:
The Selfish Gene
The Social Animal
A Mathematician’s Lament
The behavior and requirements of distributed data storage systems are frequently counterintuitive (eventual consistency, anyone?). One of these counterintuitive behaviors is something I call “the extra node paradox”: if you add an extra node to your cluster, you actually increase your likelihood of data loss. This follows from two simple observations:
Say you have a cluster of n nodes, with each data block replicated to r of those. Then
There are only (n choose r) possible ways to replicate a data block in a cluster. This number is probably far smaller than the number of data blocks that you’re storing, which implies that if your data is evenly distributed across the available replications, any r-node loss will hit some blocks replicated to only those r nodes. To make it concrete: say you have a 20 node cluster with 3TB storage per node, a block size of 512kB, and a replication factor of 3, so that you can store 1TB (~2m blocks) per node, or about 40m blocks total. The number of unique replications available is only (20 choose 3)=1140, implying a loss of roughly 35k blocks for any 3 node failure. Holding everything else constant, if you bump the cluster size to 200 nodes, you jump to 1,313,400 unique replications, guaranteeing a loss of 30 blocks. To have any chance of not losing any of your 40m blocks, your cluster would need to have 623 nodes, each storing only ~64k blocks (~32GB). For the same data set but with larger blocks (say 64MB, the hdfs default), you would have a total of 3.125k blocks. With 20 nodes you would lose 274 blocks, and you’d need 125 nodes to have any chance of not losing data, each storing only ~2.5k blocks (~162GB).
By analogy, if you flip 2 coins and fewer than 2 land on heads, you’re not very surprised, but if you flip 50 coins and fewer than 2 land on heads, you are. If you have more nodes, there are more nodes that can fail, and so more will (read up on the binomial distribution if you need to convince yourself).
The ‘paradox’ is that if you’re considering the impact of adding an extra node on your chance of losing any given block, that probability goes down – you’re getting an increasing overall likelihood of data loss from a group of blocks that are individually growing less likely to be lost.
Finding the actual probability of a simultaneous r-node failure in a real-world system is complicated by the fact that ‘simultaneous failure’ actually means ‘within the window of time when the data from the first failed node is being redistributed to good nodes’, but it doesn’t fundamentally change the conclusion. The main takeaways from this are that you can’t stop worrying about single-node reliability just because you’re distributed, and that the actual level of safety you get from a given replication factor is sensitive to the size of the cluster.
I’ve been hunting for a good little history of the world, a quick read that I can pick up every few years to get a reminder of the sequence of it all. To that end, E.H. Gombrich’s A Little History of the World
Unfortunately, though, the book falls short of what I want in many ways. Despite its title, it isn’t really “of the world”, it’s “of Europe from an Austrian’s perspective”. While Gombrich dedicates a few chapters to India and China in early history, the East is shelved from the birth of Christ until European colonialism brings it back into the picture. North America makes nothing more than a small cameo appearance, and South America and Africa outside of Egypt are both ignored almost entirely.
Worse, Gombrich focuses rather narrowly on the sequence of religions, rulers, wars, and empires that shaped the Continent. Science is wholly represented by nameless cavemen, Leonardo da Vinci, an account of Galileo in which math is equated with magic, one mention of Niels Bohr in the afterword, and vague references to technology and progress. None of Pythagoras, Archimedes, Euclid, Copernicus, Kepler, Newton, Descartes, Euler, Darwin, Maxwell, Einstein, or others even merit a namecheck, while the story of the Reformation goes deep enough to include such household names as Jan Hus and Huldrych Zwingli. I can’t deny the impact of war and religion, but any book that uses more ink to romanticize those who worked so hard at tearing the world down than to even just acknowledge those who have built it up is not one that I want to use as the foundation of my kids’ understanding of history.
That said, the book does have a lot of good things going for it. While you never forget that it’s aimed at children, it’s well written and not afraid to touch on challenging ideas. There are genuinely nice overviews of the founding stories of some of the major world religions, thanks at least in part to Gombrich’s revisions with the help of his son, a preeminent scholar of early Buddhism. And I don’t mean to imply that a strong European focus is inherently bad – as long as you realize that’s what you’re getting, it’s perfectly fine, and if you don’t know your Visigoths from your Ostrogoths, your Guelphs from your Ghibellines, or your Babenbergs from your Habsburgs, this book will do a nice job of catching you up.
In Daniel Kahneman’s Thinking, Fast and Slow, the mind is divided into two systems. System 2 is the deliberate serial thinking process which we commonly regard as our self, and System 1 is the automatic parallel reacting process which makes up much more of us than we generally realize. System 1 is responsible for the first assessment of any information that comes in, most of which will be filtered out or handled without System 2 ever getting involved. When System 2 does come into the picture, it’s slow and lazy, frequently only giving a cursory assessment to System 1′s preference before confirming it. Most of the time this is fine – in fact, a key trait of expertise is that System 1 becomes adept at making these first decisions – but System 1 also has characteristic biases that this book explores in fascinating detail.
A principal bias that Kahneman describes is ‘WYSIATI’: what you see is all there is. WYSIATI is the “remarkable asymmetry between the ways our mind treats information that is currently available and information that we do not have”. System 1 is “a machine for jumping to conclusions”, forming opinions from information that’s at hand without stopping to consider what other information might be needed to actually justify those opinions. Some of the effects of WYSIATI include being overconfident in cases where we may be missing critical information, regarding the same information differently depending on how it is presented, and neglecting base rates when examples are readily available.
This idea appears repeatedly throughout the book, and where it isn’t explicitly summoned it’s often easy to see lurking in the shadows. The ‘anchoring effect’ describes how we latch onto numbers that are in front of us when trying to estimate quantities, even if we know that those numbers are completely random. The ‘availability heuristic’ answers questions about an event’s frequency or significance by simply gauging how easily instances come to mind (an example of ‘substitution’, our tendency to avoid hard questions by answering easier ones in their place). And ‘hindsight bias’ leads us to “assess the quality of a decision not by whether the process was sound but by whether its outcome was good or bad”. When our response ought to be “I don’t know”, System 1 comes up with an answer anyways, and System 2 often just simply goes with it.
Kahneman won the Nobel Prize for his development (with Amos Tversky) of prospect theory, and he dedicates several chapters to explaining the subject and its implications. Prospect theory is a critique of expected utility theory, which was developed to explain why rational decision makers might select a sure thing over a bet that has a larger expected value. For instance, we might prefer $100 over a 30% chance at $1000, despite the latter having a higher expected value of $300. If we choose the natural logarithm as the basis of our expected utility function (other functions can work, too), the expected utility of $100 is ln(100)*1 = 4.61, while the expected utility of a 30% chance at $1000 is ln(1000)*0.3 = 2.07. If we’re attempting to maximize utility, the choice of the sure thing over the gamble is clearly justified in this scenario.
Prospect theory observes that this model of decision making is incomplete in that it neglects the reference point of the decision maker. From different reference points, the same payout looks like either a loss or a gain, and we’re more averse to loss than we are to risk: when faced with a potential loss, we frequently become risk-seeking. Going back to our example of the choice between $100 and a 30% chance at $1000, prospect theory observes that a person starting with nothing may actually see the bet as a potential loss of the $100 they were otherwise guaranteed. In contrast, a person who already has the $1000 in hand perceives the certain option as a loss of $900, and is more likely to take the gamble despite its lower utility.
The implications of loss aversion are significant. The ‘endowment effect’ describes our decreased willingness to trade something we have for something we don’t, even if when we lacked both items we would have considered the second greater in value than the first. We work harder to avoid failure than we do to exceed expectations. Credit card companies seek legislation that requires differential pricing on cash and credit transactions to be called “cash discounts” rather than “credit surcharges”, because we’re more willing to accept forgone discounts than pay a surcharge – we make inconsistent decisions when equivalent scenarios are framed as a loss versus as a gain.
The last section of the book focuses on the distinction between the ‘experiencing’ and the ‘remembering’ selves. In one experiment, subjects who had to go through a painful medical procedure were asked at regular intervals during the procedure how much pain they felt at that moment. After the procedure was complete, they were asked to rate the total amount of pain they had experienced. The value that people assigned to this final question did not correspond to the sum of values they had assigned to the questions during the procedure; instead, they followed what Kahneman calls the ‘peak-end rule’ – their rating corresponded to an average of the peak level of pain during the procedure and the level of pain right at the end. In addition, the length of the procedure had no impact on the rating of total pain, a phenomenon called ‘duration neglect’. Our remembering selves punish our experiencing selves by causing us to make choices that lead us to suffer more in the moment than we’ll remember afterwards.
Very little is said about the actual mechanisms for all of this. Kahneman stresses that System 1 and System 2 do not correspond neatly to specific parts of the brain or body, but are instead rhetorical shorthand for complex interactions that manifest themselves as behaviors that can be observed via experiment. This decision to focus on observable behaviors is much appreciated, as it sidesteps the need for the Anatomical Latin laundry lists that are annoyingly common in psychology books aimed at the general public. Avoiding impenetrable anatomy recitations isn’t the only way that this book is better than its shelfmates. Where other books try to punch things up with jokes or hackneyed narrative, Kahneman keeps us engaged with concise and insightful discussion. Compare this to Dan Ariely’s Predictably Irrational, which covers much of the same ground but includes so much ‘entertaining’ filler that it feels thin and condescending. Kahneman trusts his audience enough to avoid doing this, and the result is a rich and rewarding read that I highly recommend.
So from here on out I’m going to be writing a little something about every book I read, in an effort to make more stick. These won’t be formal, just a brief synopsis and a collection of takeaways and other random thoughts. The only rule is that the post has to go up before the next book is finished.
First up: An Engine, Not a Camera: How Financial Models Shape Markets
One of the book’s main illustrations of this idea is the empirical history of option pricing. MacKenzie divides this history into three periods: pre-1976, when prices did not adhere well to Black-Scholes values; from 1976 to 1987, when they did; and post-1987, which has seen a persistent volatility skew that pulls prices out of line with theory. The mechanism MacKenzie cites for both shifts was the application of the theory by traders.
Following its publication in 1973, Black-Scholes allowed analysts to identify options that were mispriced according to the model. As traders rushed in to take advantage of those opportunities, their activity whittled away discrepancies until real prices more or less matched theory, beginning around 1976. One might be tempted to believe that this simply implies that the theory was correct, and the shift to match theoretical prices was the result of the market waking up to reality and fixing itself.
In 1987, however, things changed again. Option pricing theory was crucial to the development of portfolio insurance, which allowed investors to limit their exposure to losses from declining stock portfolios by selling corresponding index futures as prices went down. Two key assumptions underlying portfolio insurance (inherited from Black-Scholes) were that there would be no large gaps between successive prices (allowing time to make adjustments to the portfolio), and that trades (made to realize those adjustments) did not affect broader market prices. While it was accepted that the first assumption could be violated as a result of the market reacting to a dramatic world event, it was less appreciated that it could also come under pressure if the second assumption were violated, which grew more likely as the popularity of portfolio insurance increased:
By June of 1987, the portfolios “insured” by LOR and its licensees were sufficiently large that Leland was pointing out that “if the market goes down 3 percent, which, in those days, would have been a very large one-day move, we could double the volume [of trading] in the New York Stock Exchange”
On October 19th, it fell 22.6 percent. MacKenzie shares from an interview with Nassim Taleb:
… the crowd detected a pattern of a guy who had to sell [as] the market went lower. So what do you do? You push lower … and you see him getting even more nervous. … It’s chemistry between participants. And here’s what happened. You understand, these guys are looking at each other for ten years…. They go to each other’s houses and they’re each other’s best friends and everything. Now one of them is a broker. He has an order to sell. They can read on his face if he’s nervous or not. They can read it. They’re animals. They detect things. So this is how it happened in the stock-market crash. They kept selling. They see the guys sell more….
This is a great example of one of my favorite aspects of the book: its use of a large number of personal interviews to bring focus to the human aspect of all this, the individuals and groups behind the development and application of these theories. The pages are filled with little behind-the-scenes anecdotes: the pay-to-play deal behind Milton Friedman’s 1971 paper “The Need for Futures Markets in Currencies”, which provided a ‘stamp of authenticity’ ahead of the opening of the Chicago Mercantile Exchange’s International Money Market; the personal cajoling required to get traders to actually participate in the early days of the new exchanges; the exploitation of temporary put-call parity discrepancies on the Amex by literally standing between the two different specialists responsible for puts and calls; the phone calls throughout the night and early morning that, with 3 minutes to spare, allowed the Merc to re-open on October 20th, 1987. The list goes on.
A particularly interesting section is the account of how index futures were brought to market. Futures trading was legally distinguished from gambling by the fact that a future implies the possibility of delivery of the underlying commodity. If a future could only be settled in cash (as is the case for an index future), it was legally a gamble, and therefore illegal according to state-level gambling laws. To get around this, Chicago Mercantile Exchange chairman Leo Melamed and other futures exchanges lobbied for the creation of a new federal agency, the Commodity Futures Trading Commission, whose 1974 charter allowed it to preempt state laws. The legislation creating the CFTC was carefully drafted to expand the definition of ‘commodity’ to cover securities without actually using the word ‘securities’, which would have drawn the attention of the SEC (who would have tried to block the legislation to protect their authority). Once established, though, the CFTC was in a position to negotiate with the SEC, and, in 1982, reached an agreement with them that explicitly allowed index futures to be traded under the CFTC’s jurisdiction, thus circumventing those state gambling restrictions. The key to the SEC acquiescing? The fact that what was being traded was “a figment of Melamed’s imagination” – that is, not a security or a derivative, but a future that could only be settled in cash, the very trait that had made them illegal in the first place. If you don’t like the regulatory climate, just convince Leo Melamed to get you a new one.
The one disappointing aspect of the book is something MacKenzie couldn’t have possibly avoided: it was published in 2006, predating the 2008 financial meltdown. The other two arguably most significant market events of the last 30 years, the 1987 crash and the collapse of Long Term Capital Management, are covered in great detail, but the fact that such a large event related directly to the subject matter of the book is missing makes it feel incomplete. Luckily enough, MacKenzie has almost a dozen papers written since, available on his website.
One last minor note: in setting up discussion of Mandelbrot’s work to use the Lévy distribution in place of the normal distribution in modeling the random walks of stock prices, MacKenzie discusses how mathemeticians “redefin[ed] ‘polyhedron’ so that an anomalous polyhedron isn’t a polyhedron at all” to work around apparent counterexamples to a prized theorem. This struck me because I recognized that theorem as Euler’s Polyhedron Formula, V – E + F = 2, which is the subject of a book I read last year called Euler’s Gem
Matt Gattis tweeted a quiz earlier tonight: 10 girls and 10 guys in a group. Sally dated 5 of the guys, Bob dated 2 of the girls. What’s the probability that Bob dated Sally? Think about it for a bit, then read on.
Kui Tang has a nice write up of the solution over on his blog, but I thought I’d bang out a quick alternate explanation for those of us who like to visualize our probabilities: imagine a 10×10 grid of cells, the x axis corresponding to the men and the y axis to the women, with each cell either on or off depending on whether the x,y pair had been on a date. Take and count up all unique grid configurations that have Sally going on 5 dates and Bob going on 2. That’s your denominator. Your numerator is then the number of these unique grids that have Sally matched with Bob. These are huge numbers, but then recognize that all possible non-Bob/non-Sally cell state configurations repeat for every unique Bob/Sally configuration, and so neatly cancel out.
The math given in Kui’s post is the same thing expressed with counting formulas, but I think picturing the problem as a set of unique grid layouts helps give a better intuitive understanding of what’s going on. It’s hard to accidentally overcount, for instance, because its clear that the visual equivalent to (10 2) * (10 5) counts the Bob+Sally cell too many times, and it erases the questions about the cases where only one other woman has dated at all or 2 other women have dated 10 guys, because it’s clear they’ve been taken into account as part of the massive number of states that cancel out when you do the tally.
Color is a new photo sharing app that builds social networks based on proximity. You take a picture with the app, and it turns around and starts grouping you with and sharing photos from other people nearby who have done the same. Sounds kind of dumb, right? Why would I want to see photos from nearby strangers?
Well, Sequoia thinks there’s something there, and has put $41 million into the company before it’s really even launched (thanks to a killer pitch deck). “Not since Google” have they seen this. Given that “this” currently refers to an app that I can’t even get to work on my phone, I’m left hoping that there’s a lot more going on here.
So what could that be? I’m going to put on my magic hat of credulity now, and describe what I (yes, I, random internet wantrepreneur) would be willing to bet $41 million on in this space.
Color is being run by Bill Nguyen, who sold Onebox for $850M in 2000, Lala to Apple for over $80M in 2009, and (at least until 11:41am today) spent time at AdGent. I’m not going to say that his presence means Color will be successful, but I do take it as a pretty good sign that there’s no possible way their actual business story is “Color shows you photos taken by people in the same room and then money pours out”.
From the TechCrunch writeup:
Color is also making use of every phone sensor it can access. The application was demoed to me in the basement of Color’s office — where there was no cell signal or GPS reception. But the app still managed to work normally, automatically placing the people who were sitting around me in the same group. It does this using a variety of tricks: it uses the camera to check for lighting conditions, and even uses the phone’s microphone to ‘listen’ to the ambient surroundings. If two phones are capturing similar audio, then they’re probably close to each other.
Remember The Dark Knight, when Batman hacked into everyone’s cellphone and streamed back sonar data to build a cohesive picture of what was happening everywhere in the city? That sounds awfully similar to what’s going on here – photo, GPS, and audio streams feeding back to Color in such a way that they can build a real-time model of where all their users are, who they’re with, and what’s happening around them.
With that kind of technology, who cares what their frontend does? Based on the quality of the first release of the phone apps, they’re clearly not sweating it too much. Whatever hook they try to snag users with is just a way to get that datastream, so they should ride whatever wave is currently popular. This week it’s Instagram and Path, so, sure, do that. Next week it’s going to be something else, so next week they’ll shift their apps towards that, or if they really can’t figure out how to get traction, they’ll release an api and let others do it for them. It doesn’t matter how that data comes in, as long as it comes in.
The web is training advertisers how to most effectively work with real time data (tracking cookies, ad auctions, sentiment analysis, twitter monitoring, all of that). How many companies work on this? How much money is being spent on these efforts, and how much is being made? There’s already one $190B company in this space on the web; the startup that can bring the same sort of tools into the real world might actually have a shot at becoming another.
Facebook, Foursquare, Yelp, Gowalla, Brightkite, Loopt, and everyone else with check-in functionality are already going for this. The biggest differences with Color seem to be that they want check-ins to be implicit byproducts of actions users have other motivations for (you’re not trying to get a free soda, you’re taking a picture to, uh, show to strangers in the same room), and that they’re handling far more inputs than just location.
These differences are both potentially huge. Other services risk crossing a mental line where explicitly checking in feels like work done for compensation (which is bad, which is why Foursquare is set up as a game), whereas this is an attempt to keep motivation purely social. And using multimedia opens up the door for all kinds of data points – facial recognition to keep track of people who aren’t actively using their product, brand recognition to note logos on clothes or labels on bottles, song recognition to track what music people are actually listening to – that advertisers would pay through the nose for.
Taking the credulity hat back off, even though I really do think the potential business models could make a ton of money, I’m equally convinced that this initial attempt at getting users isn’t going to make it very far. With $41M in the bank, though, they’ve got plenty of room to fail.
Update: Bill Nguyen confirms all of this almost point-by-point in an interview with Business Insider:
Photo sharing is not our mission. We think it’s cool and we think it’s fun, but we’re a data mining company.
It’s amazing how long you can go without basic life skills. Pretty much all the food that went into my body from college until last year came from a can, a box, or a restaurant, and either tasted bad, actively tried to kill me, or drained my bank account (or all three). The first step is acknowledging that you have a problem. The second step is learning to cook. The third step, apparently, is writing a post about what you learn over the course of a year of step two.
From learning to cook, I’ve saved hundreds of dollars, eaten much better, and picked up a new skill that I might actually be able to use in a post-apocalyptic setting (my first!). This is all basic stuff, but if you’re starting from zero like I was, it may be helpful.
- Get a decent chef’s knife. I picked up an OXO for under $20, and I love it. I’ve used other peoples’ nicer knives since getting this one, and there is a difference, but starting with a decent knife for cheap means you get to practice knife skills and maintenance without caring too much when you drop it on the floor a half-inch from your toe. Related: picking up hot things without putting sharp objects down first is not advised.
- Epicurious is awesome. Probably 80% of the recipes I tried this year came from this site. The recipes are usually well written and you can find ones from all over the difficulty spectrum (from quick-and-easy to spend-your-Saturday-hating-yourself-for-sucking-before-finally-just-ordering-a-pizza).
- When I first started, I thought salt was something that primarily goes on at the table so everyone gets as much or as little as they want. This is dead wrong, especially for meat. If you’re cooking steak, chicken, or pork, get some kosher salt (big flakes), sprinkle it on generously, and let it sit for a bit at room temperature before throwing it into the pan or oven. This locks in the moisture, and if you do it right, you shouldn’t really notice a salty flavor, it should just taste better. In college, I had a tradition of cooking myself a steak whenever I finished a big project, and I always wondered why it never tasted as good as what you get at a restaurant. It was the salt thing.
- When you’re chopping things up, make the results the same size so they cook at the same rate.
- If you’re frying, sauteing, or grilling chicken or pork, make sure you use cuts that are thin enough or that you can finish cooking in the oven. I’ve started butterflying chicken breasts before throwing them on the pan, and the difference is stark. A full breast takes too long to cook through and will either burn on the outside or dry out before it’s done, where the half breast stays moist and picks up a nice brown color while cooking in a much shorter time.
- I’ve been amazed at how many recipes want an onion. Learn how to chop one and save time and tears.
- Keep the pan hot. Every time you add something cool to the pan, you cool the pan off (stupid physics), so right when doing so, ramp up the burner and then taper it back to where it was as the food heats up to where you want it. I’ve only internalized this one in the last couple of months, after repeatedly banging my head on needing twice as much time as a recipe suggests. Keep things hot, and they cook faster. Similarly, it takes a lot less time to boil water if you put a lid on the pot. Weird how that works, right?
- Get to know by heart how long different basics take to cook, and how long things can sit when they’re done, so you only panic appropriately when everything else isn’t finished yet. Rice takes 20 minutes and can sit for a while, while steamed veggies take 10 and can’t. Chicken dries out quickly, while steak can rest for a spell. If you turn the heat down a notch, you can keep onions sauteing for a good while, but not so much for garlic, and not at all for peppers. That sort of thing.