November 18, 2011
The Algorithms of Wall Street
On May 6, 2010, the Dow Jones Industrial Average fell nearly 1000 points - the largest single-day drop in history - only to regain nearly everything that was lost within minutes.
So what happened? Was there some sort of dramatic news that was reported, and then recanted? A war that almost happened? An election reversed? These are the sorts of dramatic events that seem necessary for such an extreme trading pattern.
In fact, there was no such root cause. While the details are still sketchy, one theory for the cause of the "Flash Crash" is that a single large sell order triggered high-frequency trading algorithms (i.e., computer-directed trades) to start selling. The algorithms were optimized to deal with small changes in price, and as prices started plummeting, the computers continued to trade with each other, driving prices down even further. Where a professional trader could tell that prices were too low, there were no people in the loop - it was just computers buying and selling to each other.
While this is a dramatic example of how eliminating people from a process can prove disastrous, it highlights just how important computer-to-computer interactions are. In fact, I believe that computer-to-computer communication will turn out to be one of a (very) long string of "Connection Revolutions", which I would like to briefly outline. As usual, I will blur the line between biological and technological advances, because I see them as being very similar.
I want to quickly outline each of these revolutions, and possibly follow up with a blog post about each one.
Very early in the history of life, organisms gained the ability to sense their environment, and respond to it. These first senses were incredibly rudimentary, and chemical in nature. An organism that recognizes the chemicals that occur near its food source, for example, will be more successful in finding food, and will pass that trait on to its offspring.
Eventually, organisms not only sensed the environment, but intentionally altered the environment by emitting chemicals, thus communicating with other organisms. These communication channels can be surprisingly complex, allowing even one-celled bacteria to determine important information about their environment, and to coordinate their actions.
This coordination allowed some organisms to become so closely linked that they could specialize, subdivide their tasks, and become multicellular. At first, their larger size allowed them to become bigger, and thus move up the food chain. In time, specialization and intimate intercellular communication made specialized organs possible, allowing bodies to grow much larger, and much, much more complex.
One of those organs was the brain, composed of cells whose only job is communication. Brains take in stimuli collected from all over the body, process it, and send instructions for how to react. By centralizing this information processing, relations between stimuli could be discovered and responded to.
Some animals learned to signal directly, chemically, visually (think peacocks feathers or flowers), and most importantly for our timeline, aurally. Some animals learned to produce and sense sounds. In the case of humans, our supersized brains allowed us to progress beyond simple grunts and communicate whatever we wanted.
This allowed people to communicate about things other than the immediate present. Concepts and ideas could be discussed and examined. Instructions about how to hunt, plant crops, etc. could be relayed unambiguously. It is possible that language is what enabled our ancestors great migration out of Africa.
Writing allowed language to be conveyed not just to those immediately present, but across time and space. Vastly more knowledge could be preserved and distributed, making complex societies and technologies possible.
The printing press seems like an incremental improvement to hand-written writing, but its effects were much more important than the technological improvement would predict. By making books financially accessible to everyone, I believe the printing press was a necessary condition for universal literacy, democracy, the Renaissance, and the Industrial Revolution. By making it possible to (nearly) perfectly duplicate even seemingly unimportant books, knowledge about disparate subjects could be gathered, assimilated, and serve as inputs for further knowledge and technology creation.
The rise of mass media (i.e., TV, radio) was another important change. For the first time, media could be produced and distributed simultaneously to entire nations. This had important impact in creating a shared culture, and a shared way of seeing the world. Its masters held great power, and some, like Joseph Goebbels, understood just how revolutionary and powerful it could be.
The Internet is the next great communication revolution. I definitely want to write another post about this, but I want to outline a few of the important trends that I see from the Internet.
First, the ability to communicate one-to-one with anyone else, immediately. Every person is connected to every other person, and theoretically could communicate with any of them, without intermediaries.
Second, we are storing even unimportant things, digitally, where they can be preserved and searched forever. By aggregating seemingly unimportant things (e.g., which web pages link to which other web pages), amazing things can be built and discovered (e.g., Google).
Third, the Internet allows our things to talk to each other. Never before have our tools been able to tell us, or each other, what they need. We are more and more able to craft an environment that understands us, and can interact meaningfully with us. This has some dangers, like the "Flash Crash", but I believe that the benefits will become more and more apparent.
These changes are incredibly important, and I believe that we are like the readers of the first printed books, seeing a cool new technology, but blind to the societal changes (e.g., the Renaissance, democracy, and the Industrial Revolution) waiting in the wings.