Managing IT for Value

The Vodka was Great but the Meat was Rotten

2011-04-03T12:14:00.011+02:00

Viareggio, 2 April 2011

Introduction

When I announced last year that I would give a lecture entitled “The Crowd: Wisdom or Madness?” the only thing on most people’s minds was “What is that title supposed to mean?” Although it was a pretty cryptic title, it wasn’t a totally hopeless task to figure out what the lecture would be about; after all, the book The Wisdom of Crowds was available for purchase, so there was a chance that some of you might have heard of it.

But I really do have to concede that this year’s title must seem truly cryptic to you. And yet, believe it or not, there was a small chance of figuring out what it was about. It is something that is buried in the script (online) of my very first lecture, way back in 1992, nearly twenty years ago:

In fact, the best things to come out of machine translation programs were the jokes. As you can imagine, the hardest things to translate are idiomatic phrases and slogans. They gave the program the following phrase to translate into Russian: “The spirit is willing, but the flesh is weak.” The program came up with: “The vodka was great, but the meat was rotten.”

Artificial Intelligence Revisited

For many years I had wanted to give a talk here on automatic computer translation of language. It seemed like the perfect topic, because after all that’s what this association is all about: language. So many of us are professionally involved in languages – several are language teachers, some are translators. But the timing never seemed to be quite right. The progress I saw in the automatic language translation programs was not very convincing. They were slow, clumsy, and expensive – and in the end, not very good. But over the last few years, something has changed. Suddenly automatic translation is not just better than before – it’s a lot better. So what happened to make it click? That’s what I wanted to talk about today.

But when I went back to my old lecture of nearly twenty years ago to get the quote for my title, I realized that the story of why language translation has gotten so much better has a lot to do with what has happened in many of the areas I talked about in that lecture, and I decided it would be a good idea to take up the story again where I left it off, way back in 1992. And as we’ll see, as I always do in these lectures when I start predicting, I got a few things right – and a lot of things wrong.

Computer Chess

One of the things I talked about in my lecture of twenty years ago was “computer chess.” Back then, there was a lot of interest in building a computer that could match the best human chess players. Even much earlier, I pointed out, people had been fascinated by chess, and I briefly recounted the story of the so-called Mechanical Turk, a contraption that was exhibited for the first time in 1770:

A couple of hundred years ago there was actually a man touring around Europe with a beautiful robot-like machine with arms that hung over a chess board, that would play an excellent game of chess against all comers. However, it was discovered after some time that there was a man hidden inside of the robot who just happened to play a very good game of chess.

(Interestingly, Amazon.com picked up on the term “Mechanical Turk” again to describe a service they introduced in 2005 – because it was about people helping out machines to finish jobs they couldn’t do well yet, like recognizing images. Equally interesting is that the service is based on the idea of crowds – the topic of last year’s lecture.)

People thought that if a computer could beat a human in chess then, well, we were on our way to building intelligent machines. IBM had set itself a “Grand Challenge” to do just that, and had come up with a computer that it called Deep Thought (named after a computer in the series The Hitchhiker’s Guide to the Galaxy by Douglas Adams – and I think you can imagine where Adams might have gotten the name). A few years before my lecture in 1992, Deep Thought had managed to beat a grand master and generated a lot of excitement. But when matched against world champion Garry Kasparov, it was roughed up pretty badly. I reported in my talk that Kasparov had declared afterward:

I can't visualize living with the knowledge that a computer is stronger than the human mind. I had to challenge Deep Thought for this match, to protect the human race.

But I also reported that many felt that the days of the human race’s protection by Kasparov and others like him were numbered:

Now just about everybody accepts that within 5 or ten years a computer will be built that can beat any human being.

That was in 1992, and now we are in a position to know whether that prediction was accurate. Let’s take up that story and see how it turned out.

Deep Blue

After that humiliating defeat, IBM rolled up its sleeves and got back to work. Deep Thought was consigned to the dustbin and a larger, more powerful successor was designed, named Deep Blue. (They got more patriotic – IBM’s nickname is “Big Blue.”). It took a while to get everything worked out, but on May 11, 1997 – just under five years from the date I made that prediction in my talk – Deep Blue beat Garry Kasparov in a six-game match. A computer had finally beaten the world’s best human player.

Kasparov was furious. He said that he had “sometimes seen deep intelligence and creativity” in the computer’s moves. And what did he conclude from this observation? In a bizarre throwback to the Mechanical Turk chess machine of two centuries earlier, Kasparov alleged that Deep Blue had cheated, by getting help from … humans. Now, if you think you see a paradox in the best human being in the world thinking that the machine that beat him got help from other human beings, you’re not alone. After all, if he was the best human player on Earth, then what other human player could have possibly helped Deep Blue to beat him? The best player on Mars?

The whole episode ended with Kasparov challenging Deep Blue to a rematch. IBM refused and then dismantled Deep Blue, presumably just to make sure that such a rematch wouldn’t happen. (I didn’t read any reports about whether they found a little man inside when they dismantled it.) They also made a pretty good movie about it, called Game Over: Kasparov and the Machine. Like Kasparov, the film also implied that there may have been a conspiracy behind it all. But the conspiracy implied by the film was a different one – a plot to boost the stock price of IBM. Actually, that was a much more reasonable conjecture. IBM never claimed it was doing this solely for the good of mankind. Why not advance the state of technology and get some publicity in the process with these Grand Challenges? In fact, IBM was so happy with all the publicity that it set about finding itself another Grand Challenge to work on.

In the first Grand Challenge, IBM had set out to build a computer that could outperform humans in a game. In the next Grand Challenge, IBM decided to set out to build a computer that could outperform humans in … another game. And this brings us back to the topic of my lecture by way of a television game show that I used to love to watch as a kid.

Television Game Shows

The 1950s and 1960s were a time of fabulous creativity and growth in TV game shows in America, and they were being churned out at an amazing rate during those years. Many of them were eventually syndicated around the world, including Italy. One show from 1956 called The Price is Right had a great run in Italy from 1983 to 2001 as Ok, il prezzo è giusto, hosted much of that time by Iva Zanicchi (who called the show “the triumph of consumerism”). Another show called What’s My Line was invented in 1950, at the very dawn of the television era – and is alive and well today here in Italy, over 60 years later, under the name of I Soliti Ignoti, hosted by Fabrizio Frizzi.

The king of daytime television during much of that golden era of game shows was the multi-talented Merv Griffin. Born in 1925 in San Mateo, California (not far from my own birthplace), he had been a child prodigy pianist and soon ended up in show business in Hollywood. He appeared in several films after he was discovered by Doris Day, but eventually got tired of movies and decided to move into the rapidly expanding medium of television. He turned out to have a knack for understanding the new formats that would work on television, and became a successful host of both talk shows and game shows. He once showed up in the audience at the musical at my high school (High Button Shoes, if I remember correctly) – apparently he had a niece in the cast.

Before long, he was not only a host of TV game shows; he was also a creator of them. One day, as he told the story, he was flying to New York from Duluth, Minnesota with his wife. He was trying to come up with ideas for a new game show, and his wife made a suggestion. In the 1950s there had been a series of scandals around the TV quiz shows. The most prominent such scandal involved the show Twenty-One, where it turned out that the outcomes were fixed beforehand. After those scandals, people had stopped creating quiz shows. Griffin’s wife suggested that this might be a good time to propose a new one, perhaps with some kind of new twist to distinguish it. She thought she had an idea.

Jeopardy!

Her idea was to turn things around: instead of providing a question and expecting an answer from the contestant, you provide the answer and the contestant has to provide the question. For example, if you provided the answer “1861”, the contestant would have to provide the question “In what year was Italy unified?” To an answer “10 Downing Street,” the correct question might be “Where does the Prime Minister of Great Britain live?” The contestant would always have to provide the response in the form of a question.

The idea was so good that it was accepted sight unseen by NBC, and the show was named Jeopardy! (the exclamation point is part of the name). It was launched in 1964 and became a hit immediately. The game was a straightforward quiz, with several categories of questions (or, I should say: answers) for the contestant to choose from, according to ascending monetary rewards. At the end of each show was a section called “Final Jeopardy,” where each contestant had one last chance to influence the outcome by wagering up to his entire winnings on a single question.

That final question was accompanied by a catchy little tune that was timed to last exactly 30 seconds, the time that the contestants had to contemplate their final response. The title of that tune is “Think!” and it has an interesting story all its own. It was composed by none other than the versatile musician Merv Griffin himself – and was in fact originally a lullaby for his son. It has so insinuated itself into American popular culture now that it’s used for just about any situation in which there is a countdown while waiting for something – for example, during discussions in baseball games or at the horse racing track, or even during Perry Mason or similar legal TV shows when awaiting a verdict. Merv Griffin liked to boast that for the 10 minutes he spent writing that little melody, he had received over 70 million dollars in royalties over his lifetime.

I once attended a Jeopardy show in the 1970s when I was a student at Yale. With a fellow student, I went down to New York and was part of the studio audience for a series of three shows and got to see the host Art Fleming in action firsthand. Back in those days, there weren’t the fancy electronic displays there are today. Instead, it was literally a board with cardboard signs in the categories. When a selection was made by a contestant, a burly stage hand standing behind the board would simply pull off the piece of cardboard covering the question. For me, it was a bit like exposing the Wizard of Oz.

Later, when I was a student at Berkeley, I participated in an audition when the show’s producers swung through San Francisco looking for new contestants. I didn’t do so badly in the audition, but in the end they chose a sharp, good-looking young woman with a vivacious personality, and I can’t really say I disagreed with their choice.

Jeopardy was an enormous international success, with adaptations in over 25 countries – including Italy in the early 1970s, with Rischiatutto hosted by Mike Bongiorno. Jeopardy wasn’t the only successful TV game show that Griffin invented, by the way. The hugely successful Wheel of Fortune show was also his invention, and as we all know, it lived on with great success in Italy as the Ruota della Fortuna. He was eventually named by Forbes as the richest Hollywood performer in history, in large part due to his ventures in game shows.

So what was it that made Jeopardy so successful? It turned out that the clever inverse “answer/question” format encouraged extremely sophisticated uses of the English language, with many puns, shades of meaning, ambiguities, and subtle references that challenged the very best minds. The categories ranged over the entire spectrum of human knowledge, and so contestants had to be very well-read indeed. This was not a game for dummies – and it soon attracted a huge cult following. If that seems strange to you, consider the game show L’Eredità (“The Heredity”) that is running today on Italian television. Its final section, called “the guillotine” – in which the challenge is to guess a hidden word from among five words that are related in a way that the contestant must deduce – now numbers among its many fans no less a personality than the great Italian intellectual, professor of semiotics, and author (The Name of the Rose) Umberto Eco, who says he never misses it on TV (the show’s producers once personally dedicated a “guillotine” puzzle to Eco).

In summary, a successful contestant on the Jeopardy show had to exhibit an extraordinary command of the English language in all its grammatical nuances as well as a mastery of a broad range of subjects ranging from history to geography to literature to current events and popular culture. It turned out that Jeopardy embodied exactly the type of challenges that were being confronted by IBM and others in the area known as natural language processing.

Natural Language Processing

At this point, I’d like to go back once again to my old lecture and take up another topic and see how it turned out. Back then when I gave my lecture, nearly twenty years ago, there were essentially two opposing approaches being tried to making “intelligent machines”:

· Teaching things to them;

· Letting them learn for themselves.

The first approach involved the general idea of what were called expert systems. Suppose you wanted to make a computer that knew a lot about making medical diagnoses. The strategy was to find an expert in making medical diagnoses, and get all the knowledge out of his brain and into the computer – perhaps by interviewing him to coax the expertise out of him. This approach made a lot of sense to everybody. After all, human beings are the source of the knowledge we’re after, so why not go straight to the source?

Later on, this approach acquired even more credibility in the popular mind as the idea of the so-called Semantic Web emerged. Have you ever searched for a word or phrase on the Web and received the wrong answer because the meaning was ambiguous? For example, take the “Paris Hilton” problem: are you looking for a hotel in France or browsing for movie star gossip? Those who promoted the Semantic Web idea proposed that experts create various kinds of “dictionaries” inside the Web that can help sort out the different meanings a word or phrase can have. The more accomplished and capable the experts, the better the dictionaries will be, and the “smarter” the Web will become. This approach was adopted enthusiastically because it also seemed to lead directly and eventually (if not immediately) to the Holy Grail of computers that understand human language. And that made perfect sense, too: who was better qualified to teach computers to understand human language than humans themselves?

The second approach involved the general idea of “learning by example.” Officially, this approach became known as Machine Learning, whereby an initially ignorant computer would essentially learn from experience. It was quite successful in some areas, especially controlling sophisticated machinery like robots or helicopters. Take the case of the helicopter: it’s not that easy to get the control and balancing of a helicopter exactly right. So instead of trying to calculate exactly the right program to do the job, you just let the helicopter fly and see what happens. When the helicopter crashes, the control program considers that a bad outcome (thank goodness for that), and learns from the experience by avoiding the next time whatever it did to make it crash. (By the way, if you’re getting worried: of course we’re talking about unmanned model helicopters, not the real thing.) In time, after a certain number of crashes, the control program will have adjusted itself to do a far better job of keeping the helicopter in the air than a man-made program might have done. And in the control of robotic machinery the Machine Learning approach became almost universally the method of choice: you just let the machinery adjust itself after each “bad” outcome and it would eventually converge to the desired “good” outcome.

But a bunch of robots and helicopters learning how to move around and fly is a lot different from a computer learning to understand language. One is merely “motor learning,” whereas the other is “cognitive learning.” Why, they’re not even on the same intellectual plane! Everybody knew that there was no way that Machine Learning, as effective as it might have been for controlling our manufacturing plants and vehicles, would ever lead to a way for computers to work with language, the highest expression of human intellect.

Indeed, in my lecture nearly twenty years ago, I dared to make the bold claim that in my opinion, finding a way to deal effectively with human language would be equivalent to solving the problem of Artificial Intelligence itself; because natural language, with all its complexities and subtleties, was the very embodiment of human intelligence. A computer could never “process” natural language if it did not truly understand it, I patiently explained to my audience that day in 1992.

And I was dead wrong.

The Rise of Machine Learning

As the years passed, expert systems and the Semantic Web began to run into problems. Progress seemed frustratingly slow. It seemed to take forever to create those dictionaries of meanings by experts. One well-known dictionary of medical terminology (called SNOMED) ended up with over 370 thousand names at last count, and they’re still not sure whether they have separate terms in there that really mean the same thing. Over time, many began to voice the opinion that the dream of the Semantic Web has not been realized and may never be realizable. And with that looming failure, the dream of being able to process human language seemed to fade.

But then something strange happened. Machine Learning began to have a string of successes in a most unexpected area: processing human language.

How could that be? What could have possibly happened to turn Machine Learning – that simple, primitive technique used mostly for teaching motor control to mechanical contraptions that walked or flew about – into a successful approach to dealing with language, that most sophisticated and subtle of all human characteristics that defines our very humanity? What had changed to make it all possible when it hadn’t been possible before? What new and deep algorithms were discovered to capture the essence of language acquisition? Well, it turned out that it wasn’t really the algorithms that enabled the breakthrough.

It was the data.

The explosion of online data

One reason that Machine Learning had been so successful with teaching motor control to robots, helicopters, and the like was that they could have as many examples as they needed to learn from. Did the helicopter crash? Send it up again. Did the robot make the wrong move? Have it try again … and again … and again … thousands of times if necessary, until it finally gets it right.

But human language is a different story. While it’s true that humans acquire much of their language skill from examples, it’s also true that they also have seemingly infinite resources for obtaining their examples – reading, talking, listening, studying, every day of their lives. Even if we thought Machine Learning of language through examples might be a feasible approach, where would all those examples come from?

In 1992, when I gave that lecture, the question was reasonable. The total amount of written language in electronic form was infinitesimal. Hardly anybody except software people like me used computers to write things. The Internet was small. Few people had e-mail. The number of websites was, literally, almost zero. The biggest stores of information in electronic form were databases full of numbers, not words – and even they weren’t very large, all things considered.

But just a few years later, in the mid-nineties, things began to change quickly. With the arrival of the first web browsers like Mosaic, the number of websites began to shoot straight up. Today it is estimated that there may be somewhere around 150 million websites. And the size of the Internet itself? A recent study by IDC estimated that the Internet consists of around 1.2 zettabytes. That’s 1.2 with twenty zeroes after it. It would take the entire population of the world sending SMS messages continuously for the next century to produce that much data.

Today we can find nearly anything in electronic form. The Wikipedia is by far the largest encyclopedia ever written, with over 3.5 million articles in the English language edition alone – and it’s all right there online, available to anyone (and any computer) who wants it. There are books, journals, newspapers, dictionaries … the list is seemingly endless. Suddenly humans don’t have the monopoly on resources from which to draw examples for learning language. Computers have as many or even more – thousands, millions, even billions of examples drawn from the huge pool of information that is now in electronic form. But of course the real question is: does it make a difference?

It turns out that it does make a difference. Peter Norvig, the Chief Technical Officer at Google (and therefore someone who should know what he’s talking about) has put it this way:

In the modern day, [with Machine Learning, we] observe how words and combinations of words are used, and from that build computer models of what the phrases mean. This approach is hopeless with a small amount of data, but somewhere in the range of millions or billions of examples, we pass a threshold, and the hopeless suddenly becomes effective, and computer models sometimes meet or exceed human performance.

It turned out that, in a way, Machine Learning of human language through examples was a bit like Einstein’s Theory of Relativity. Relativity seemed impossible when it was introduced partly because nobody had ever had the experience of traveling at speeds near the velocity of light (except in the car of a certain Welsh friend). Similarly, the idea of a computer acquiring any kind of useful language proficiency through examples originally seemed impossible partly because nobody had ever had the experience of having millions or billions of examples to work with.

The irony is that in the end, Machine Learning turned out to work much more like the way in which we ourselves learn language than the expert-system approach does: in our everyday lives, we also observe how words and combinations of words are used, and from that deduce what the phrases mean. When I have just learned a new word in Italian, for instance, it doesn’t mean much to me – it seems bereft of “semantics”. It’s just another arrangement of letters. But every time I hear it used by somebody or read it in a newspaper or a book, its meaning deepens for me, until eventually I can nearly feel its meaning viscerally as I speak it. I am literally learning through examples.

With the explosion in the amount of information in electronic form in the 1990s, Machine Learning started to replace the expert system approach to processing natural language. The learning process, or “training,” was accomplished through large corpora. A corpus (that’s the single of corpora) is a set of electronic documents full of examples with the correct answers marked so that the machine can learn from them. A huge number of such corpora have become available in recent years. An example is the Open American National Corpus, which contains 15 million words of American English. It is all nicely annotated for parts of speech, verb stems, and the like, and is available to any person (or computer) for study. Another example is the British National Corpus, which contains 100 million words with samples of both written and spoken English from many sources. Yet another example is the Corpus of Contemporary American English (COCA), which contains over 410 million words. Using this ever-increasing pool of corpora for training examples, Machine Learning programs just got better and better and better.

As one success after another was obtained with this approach, companies began to realize the vast commercial potential behind having a computer that could process human language. One of those companies was IBM, which had created a so-called Semantic Analysis and Integration department to track developments in this area.

IBM Watson and the Jeopardy Challenge

The destinies of the Jeopardy game show and IBM came together one evening in 2004 at a restaurant in which an IBM research manager, seeing an especially exciting round of the show being played on television, decided that this should become the next Grand Challenge. The next several years were spent in building a powerful computer for natural language processing based on Machine Learning, which they christened “Watson” – after IBM’s founder.

They fed Watson with every type of resource imaginable: literary works, dictionaries, news articles, databases, anything they could get their hands on – and they could get their hands on a lot, as I noted earlier. Naturally, they fed it the entire Wikipedia. By the time they were done, Watson had ingested 200 million pages of information.

It took a while to get off the ground. In its first tests in 2006, its performance was lousy. But just two years later it was holding its own against human Jeopardy contestants, and by 2010 Watson was ready for prime time. Arrangements had been made for Watson to compete on the Jeopardy show against the two most successful contestants of all time. Both IBM and Jeopardy executives realized the huge marketing value of this occasion and played it up with lots of publicity.

Then came the big day, less than two months ago [February 14, 2011]. As a computer filling an entire room, Watson couldn’t exactly stand up there at a podium next to the human contestants, so they gave him a snappy looking avatar that lit up as he “thought”. Since the game involved pressing a button when you thought you could respond correctly, he was also outfitted with a mechanical finger.

During the match, we were given an insight into how Watson works: during each answer/question session, the viewer would see the three topmost candidate answers that Watson was considering. For each answer, a meter showed the probability Watson was calculating that the answer might be the correct one. The one with the highest probability and also passed a threshold was selected. Otherwise, Watson just shut up and let the others have their chance.

The point here is that Watson wasn’t “thinking” in the same sense that we humans think: he was just calculating, like other computers do. That’s how this kind of Machine Learning works: it’s all based on estimating probabilities of things being right. In fact, it’s generally called statistical Machine Learning for that reason. I’ll get back to that later.

I’m sure you’re all dying to find out what happened, so I won’t keep you waiting: Watson won handily, winning over three times as much as each of the others.

The Aftermath

So what happened after Watson’s successful debut on Jeopardy? Certainly nobody claimed he had cheated, like Kasparov had done 14 years earlier with his predecessor. There were a few gripes here and there, like complaints about his speedy mechanical finger beating the others to the buzzer, but overall there was general acknowledgement that he had won, fair and square.

But what everybody did seem to wonder was, “What does this all mean?” There was article after article in nationally syndicated newspapers, and interviews with analysts on TV talk shows, discussing whether this all meant that intelligent machines were about to take over the world. A month ago [28 February 2011] the New York Times carried an invited article on this subject by none other than Hubert Dreyfus, the philosopher from the University of California at Berkeley whom I had discussed at length in my lecture nearly twenty years earlier on the same subject. Here is what he had to say back then:

Great artists have always sensed the truth, stubbornly denied by both philosophers and technologists, that the basis of human intelligence cannot be isolated and explicitly understood.

His essential argument involved the fact that humans have bodies and therefore a context in the world around us, whereas computers don’t. Here is an excerpt from what he had to say in the New York Times article, where he was analyzing the reason that Watson failed to understand the relevance of a subtle clue during one of the sessions:

… Watson doesn’t understand relevance at all. It only measures statistical frequencies. The fact is, things are relevant for human beings because at root we are beings for whom things matter. Relevance and mattering are two sides of the same coin. As [the philosopher] Haugeland [has] said, “The problem with computers is that they just don’t give a damn.”

That didn’t impress one of the readers of the article, who commented that

… many of our politicians, criminals and military people [also] lack understanding of human beings and seemingly have no compassion for humans.

In summary, nearly all of the philosophers, analysts, and commentators agreed that Watson didn’t represent a revolution in Artificial Intelligence. Yet they also universally agreed that it had accomplished something very impressive. I can think of no better way to put this into the proper perspective than to go back once again to something I discussed in my lecture of nearly twenty years ago.

Artificial intelligence has sometimes been defined as the use of computers to solve problems that previously could only be solved by applying human intelligence. Now, the problem is that this definition has a sliding meaning, as the computer scientist David Parnas has noted. In the Middle Ages, it was thought that arithmetic required intelligence. Now we realize that it is simply a mechanical process, and we've built pocket calculators to do it for us.

In the 1990s chess was only the most recent of those tasks we thought were in the exclusive domain of the human mind. Likewise, we had never thought that a computer could do anything useful with language by mere mechanical computation – surely that was an exclusively human activity. And yet, Watson (and many other systems like it) had shown that it was possible. Watson had literally and figuratively beat humans at their own game.

What’s Next?

So just what are the possibilities opened up by the conquest of natural language by computers? We can start the discussion with Watson himself, which is a so-called question-answering system: you ask it a question and it gives you an answer (except in Jeopardy, of course, where you ask it an answer and it gives you a question).

I presented one of the very first question-answering programs during my lecture back in 1992. That program was called Eliza (after the character in My Fair Lady), and imitated a psychoanalyst. Lucia Ghiselli and I together read the script of one of Eliza’s most famous conversations (Eliza eventually decided that Lucia had a problem with her boyfriend; she probably got into trouble at home that evening).

But there are many, many much more serious applications for question-answering systems, and that is one reason IBM created a Grand Challenge for itself in that area. After the Jeopardy match, IBM began a campaign to publicize the great future Watson was going to have. And the first application they put up in lights was the very application we talked about earlier: an expert medical diagnosis system (those poor expert systems people must be gnashing their teeth at seeing those primitive Machine Learning people succeeding in just the area they were working on much earlier).

Here’s how Jennifer Chu-Carroll, an IBM researcher on the Watson project, put it in an interview with Computerworld magazine:

Think of some version of Watson being a physician’s assistant. In its spare time, Watson can read all the latest medical journals and get updated. Then it can go with the doctor into exam rooms and listen in as patients tell doctors about their symptoms. It can start coming up with hypotheses about what ails the patient.

(I wonder what else Watson does “in its spare time” ... dream of electric sheep?)

A well-read Watson-like system could answer questions and make recommendations in a number of fields. You may not be surprised to hear that the financial industry is looking into the idea of using Watson-like programs to get good stock tips. And of course the military is always interested for lots of applications they prefer not to talk about (and maybe we’d prefer not to know about). The company Comprendo, right here in Pisa, has created a question-answer system that is being used by the Italian telephone companies to answer questions by customers about their services, saving a lot of time for harried call center operators.

Coping with the Vastness

Question-answering isn’t the only opportunity for computer language processing. The possibilities are as vast as the Internet itself. In fact, in large measure they are vast because of the vastness of the Internet.

I mentioned earlier that the size of the Internet today is estimated at about 1.2 zettabytes. But incredible as that number seems, it’s just the beginning: by 2020 the size of the Internet is predicted to be over 35 zettabyes. But most importantly (and relevant to this talk), over 70% of that data is being generated by individuals like you and me, with our social networks, blogs, tweets, and e-mails. And most of that is plain old text written in English, Chinese, Italian, and all of the other human languages around the world.

To put it simply, the Internet has gotten so large that it is beginning to be beyond our capabilities to manage it. Much of this is our fault, of course: we regularly receive messages (especially jokes) that we forward to all our friends, duplicating them many times over; we chat endlessly on the social networks and forums; one of my brothers once proudly told me that he had never deleted a single e-mail message. The result of all this is that nobody can read – much less find – everything that he needs to any more. We need help. And computers that can process language can provide that help.

With the advent of natural language processing, a new discipline known as text analytics has emerged. Text analytics help us sort through all the masses of documents, messages, words, phrases – everything that’s written down out there in the Internet – to help us do whatever we’re trying to do. I’d like to present a couple of examples now of what we might be trying to do.

What are they saying about me?

Those of you who have heard my last four or five lectures will remember that they have always somehow, some way, involved the social networks. That is the measure of how central they have become in our lives. I know people (and I’ll bet you do, too) who track their entire day on Facebook or Twitter or some forum, discussing everything you could possibly imagine: what they had for breakfast; the beautiful new Gucci shoes they just bought; how their sleeping pills made them groggy; what a great job the President is doing; what a terrible job the President is doing. You name it, and you can be sure that it’s being discussed out there on the social networks.

And that makes the social networks a powerful force in every aspect of life now – including business life. Consider the case of Canadian musician David Carroll and his band, the Sons of Maxwell. At a stopover in 2008 at the Chicago airport on a United Airlines flight from Canada down to Nebraska, he looked out the window to see the baggage handlers tossing guitars – one of which was his – onto the tarmac like they were sacks of potatoes. Sure enough, his 3500 dollar Taylor guitar turned out to have a broken neck when he arrived in Nebraska. (Now, there is at least one professional guitarist in our association, and several of us are amateur guitarists, so we can all appreciate this. In fact, many years ago on a flight between New York and San Francisco, my own guitar arrived with a hole in the back, punctured by something that went right through the case. I played it again just a few weeks ago [early March 2011] – that hole is still there.)

You can imagine the wall of indifference Carroll faced when he tried to complain to United about it. So he decided to take matters into his own hands. He made a music video about his experience, composing a clever little tune and lyrics just for the occasion, entitled “United Breaks Guitars.” He uploaded the video to YouTube and sat back to see what happened.

What happened was that it was viewed by 150 thousand people in one single day. Three days later that had risen to half a million. Just over a month later it had been seen by 5 million people. In short: the video had gone viral.

As you can imagine, it was a public relations disaster for United Airlines, and they rushed to make amends as quickly as possible. As for Carroll, he got a great boost from the incident, too, and became a sought-after speaker on customer service. (On one of those speaking trips in 2009, United managed to lose his luggage. One does wonder …)

The moral of the story is that, for businesses, much of what is known as “customer relations” has migrated onto the social networks and you had better be aware of what people are saying about you out there. But there’s just too much of it. Nobody can sit down and read everything they’re saying about you, much less even find it. That is where something called sentiment analysis comes in.

Nowadays it seems that everybody has an opinion and is happy to express it in a very public way. As soon as people do anything, they get out there on their favorite social network and write a review. You can see reader reviews of books on Amazon.com; traveler reviews of hotels and restaurants on TripAdvisor.com; reviews of products of all kinds on specialized forums all over the Internet. Sentiment analysis systems read texts like these reviews, or complaints, or whatever might be fed to them, and they try to determine whether something positive or negative is being said. As you can imagine, they won’t always get it right, because language can sometimes be convoluted and idiomatic (when Michael Jackson called something “bad” he was probably telling you that it was good). But these systems can still read a lot more than you can, and get a sense of whether things are going well or not. If you’re anybody whose reputation is a key part of his business (such as the fashion industry), then you have to get out there and protect your brand name in the blogosphere.

e-Discovery

Protecting one’s reputation for making nice clothes and serving good food isn’t the only motivation people might have for sifting through lots of electronic information; another pretty good motivation is staying out of jail. In lawsuits, there is a task known as “discovery,” where lawyers look for documents that are considered relevant to the case. Now, this was already a rather boring and frustrating job back in the old days when most documents were on paper. But now, when most documents are in electronic format, in every conceivable form from text files to spreadsheets to e-mails, it’s often downright impossible. There’s just too much out there. So-called “e-discovery” systems are out there now, reading the documents and deciding whether they’re relevant to a case. And those programs don’t get tired and bored like we do.

It’s not just lawyers who have to worry. Pharmaceutical companies have a legal obligation to be aware of complaints of people who have used their products and had problems (like, say, a sleeping pill making you unusually groggy). If somebody has made that complaint publicly on a social network, then it is definitely in your best interests to be aware of it or you will find yourself with a lot of explaining to do after an incident occurs. But here, too, the job has become simply too big for people to do by themselves.

Text analytics systems are able not only to sift through masses of electronic documentation, but they’re also able now to do some pretty clever sleuthing with what they find. They can read through piles of seemingly unrelated newspaper articles and piece together indirect or even intentionally hidden relationships among persons of interest (for example, two people might be trying to hush up the fact that one of them has a financial interest in the other’s company.) They can look for suspicious patterns that might indicate there is some funny business going on (for example, if somebody closes a bank account in one country and opens another one in a different country on the same day, it may indicate some financial hocus-pocus that deserves a closer look by the authorities). That’s another area in which Comprendo, the company from Pisa I mentioned earlier, has been involved.

This is all very impressive, but it’s also rather worrisome from one particular point of view: human employment. The tasks I just mentioned have been performed up to now by highly qualified – and well paid – humans. As much as we had gotten used to the idea of computers automating menial tasks like adding numbers and assembling automobiles, nobody ever expected them to encroach into this area. Here’s what Tom Mitchell, chairman of the Machine Learning department at Carnegie-Mellon, had to say about it in the New York Times a few weeks ago [4 March 2011]:

The economic impact will be huge. We’re at the beginning of a 10-year period where we’re going to transition from computers that can’t understand language to a point where computers can understand quite a bit about language.

Who would have ever thought it …

Conclusion

I certainly wouldn’t have thought it back then in 1992 when I boldly stated that mastering computer understanding of human language would be equivalent to solving the entire problem of artificial intelligence. But my mistake wasn’t in making that bold statement – in fact, most researchers still think that statement is true. (The official way to express it is to say that computer language understanding is “AI-Complete”). No, my mistake was in thinking that the problem of computer language understanding had to be completely solved before anything useful could be done. It turns out that a lot of useful things can be done by computers with language without having to fully understand its meaning.

Sure, that means that the Holy Grail of perfect translation of idioms, puns, and poetry is still beyond the reach of computers – that task requires full understanding and may never be fully realized. But that’s not the right perspective on what is happening here. Rather, it reflects the longstanding tension between two different approaches to the relationship between human beings and computers: “AI versus IA.” The first, Artificial Intelligence, was championed by pioneers such as John McCarthy, and focused on building machines that think. The second, Intelligence Augmentation, was championed by other pioneers such as Douglas Engelbart (who, among much else, gave us the computer mouse), who focused on building machines that help people think. In the “IA” approach, it doesn’t matter whether Watson really understands what he’s doing; all that matters is whether he’s doing something that is useful to us. In this approach, computers will always be our assistants, not our masters. Of course, it’s unnerving to see how much our assistants can do now – but they remain our assistants nonetheless.

Epilogue

After writing those last words, I decided to reassure myself by going back to the phrase that provided the title for this talk, and seeing how well a computer translation program would do today. Nowadays that’s easy to arrange, because some of the very best programs are right there online and freely available for all to try out. Google Translate is an example of a modern translation system that uses the same “statistical machine learning” techniques that I have presented and discussed during this talk. So I fired it up, set the “input language” to English and the “output language” to Italian, and I fed it the phrase:

The spirit is willing but the flesh is weak.

In return I received:

Lo spirito è pronto ma la carne è debole.

Oh my … that’s not bad at all …

Resources

YouTube: you can find all the television sessions of the IBM Watson Jeopardy Challenge online at YouTube. You can also find an entire course on Machine Learning courtesy of Stanford University, although it is not for the faint of heart.

Stephen Baker, Final Jeopardy. The author followed the IBM Watson team around as it prepared for its Jeopardy Challenge, then wrote a book about it.

IDC, 2010 Digital Universe Study. Estimates and ruminations on the size of the Internet, now and future.

Comprendo (www.comprendo.it) is a local company just down the road in Pisa that does advanced applications in many areas of text analytics, including some of those mentioned in this talk. Their website has examples you can try out.

Stephen Marsland, Machine Learning: An Algorithmic Perspective. A textbook for those who want to know more about machine learning.

A Rational Emergent Architecture: Fake it!

2010-04-23T15:56:00.002+02:00

One aspect of the agile approach to software development that puzzles a lot of people is the idea of "emergent architecture." The idea that the architecture of a system can just "emerge" out of the development process seems strange enough to them; but if they are working in mission-critical sectors that include, say, safety-critical embedded systems, then the idea of emergent architecture seems like a showstopper.

The word "emergent" is anathema in the mission-critical domain, because emergent behavior can wreak havoc on all kinds of things – in particular, it makes it difficult to construct systems out of reusable components, because you keep having to do the validation effort over and over again at higher levels (there's a lot of work in that area going on right now).

Part of the problem, however, is that people are associating emergent architecture with emergent behavior. But "emergent architecture" is talking about the way the architecture is developed, not the way it behaves once it's developed.

It would be a good idea to re-read a paper written several years ago by Parnas and Clements called A Rational Design Process: How and Why to Fake It. In this paper, they argue that we never really follow the ideal, rational design process, for all the reasons we know about. But that doesn't mean that the finished product has to be irrational. On the contrary, it continues to be important to document it as though it had been the product of a rational design process. Consider this:

Mathematicians diligently polish their proofs, usually presenting a proof very different from the first one that they discovered. A first proof is often the result of a tortured discovery process. As mathematicians work on proofs, understanding grows and simplifications are found. Eventually, some mathematician finds a simpler proof that makes the truth of the theorem more apparent. The simpler proofs are published because the readers are interested in the truth of the theorem, not the process of discovering it. Analogous reasoning applies to software. Those who read the software documentation want to understand the programs, not to relive their discovery.

The principle of "emergent architecture" gives developers a chance to find the best possible architecture – like in the mathematical proofs described above, the well-understood and simplest architecture for the system at hand. During this process, they may well also make use of the same principles that others use to combat emergent behavior (e.g. "correctness by construction").

Once the architecture is there, people don't want to relive the process of discovery, they just want to know what's there; then it can be described in a rational way using the best practices for architecture description today, such as those of the IEEE 1471 Standard.

Architects of mission-critical systems need efficient, well-understood architectures as much as anybody else, and it would be a pity if they avoided the ideas about emergent architectures coming out of the agile community because of a misunderstanding.

Does Agile help you to know when to stop?

2010-04-15T17:27:00.002+02:00

This week I had to teach two seminars - one on UML and one on agile methods. In the first seminar I was talking about use cases and requirements analysis, and I repeated the oft-heard remark that one of the biggest problems in software development is that "you never know when to stop." I think I mentioned it when talking about system context.

Then, during the agile seminar, we did a session of the Extreme Hour - where everybody builds his favorite coffee machine. An amazing number of inventive features came out for the coffee machines imagined by the group, ranging from battery power to background music to being mobile to being able to take outside. Many of the features came up during iterative sessions - that is, they weren't there in the beginning, but as people gained experience with the "system" they thought of more features.

At the end, one participant said, "Considering that problem of 'never knowing when to stop', agile development would make it worse - because it keeps eliciting more features as you go along." I must admit that it gave me pause to hear that, and I had to reflect a bit. This is what I think now:

The problem of "never knowing when to stop" will always be with us. It doesn't matter whether you are doing BDUF development or agile development or any other kind. But agile development frames the conversation about adding features in a better way. It ensures that the discussion between developer and client is immediate and continuous, including the discussion about the costs of adding features. This includes the techniques of optional scope contracts and the like, which give developers and customers a framework for discussion about features that doesn't become quickly antagonistic, but rather a real discussion with decisions that can be made together and based on facts. In a fixed-price contract, for example, you will quickly end up with an antagonistic discussion regarding "when to stop" where the developer is pushing to "stop earlier", and the customer is pushing to "stop later."

In other words, an agile approach to the problem of "never knowing when to stop" doesn't get rid of the problem or even make dealing with it any easier -- it just enables you to do a better job of dealing with it.

Agile Model Driven Development?

2010-04-14T16:45:00.003+02:00

Yesterday I attended a webinar presented by Stephen Mellor, who is the father of Executable UML. He had a funny story to tell. He had been invited to the meeting that produced the Agile Manifesto, and where he ended up being one of the signatories. Being a proponent of model driven development, he wanted to know whether it was considered agile enough for him to be "qualified" to sign the manifesto. It came out during the discussions that agilists didn't like models because they don't execute, like code, and they can't be tested automatically, like code. In the end, he realized they were basically saying that models are essentially just "documentation", which doesn't correspond to the code, is high-maintenance, etc. Then he described a conversation that went something like this:

"So you don't like models because they can't execute?"

"That's right"

"And you would like models if they could execute?"

"Yes"

"But they can execute."

"I still don't like them."

In spite of this, he decided they all had enough in common so that he could sign the manifesto in good faith.

Then he went on to basically say that with executable models, you can be just as agile as a coder can, with the same feedback, test, etc.

It all seemed convincing to me. My concern with model driven development is reflected in Martin Fowler's remarks in the third edition of UML Distilled. Once we have a fully executing modeling system, we basically have another programming language/environment. Then it will go into competition with other programming environments. Certainly it will be higher level, better in many ways, etc., but in practical terms we have seen that it's pretty hard to get a different programming environment/language accepted. Fowler mentioned Smalltalk as a "better programming environment" that never really got widespread acceptance.

Plus, in a way that's much more vague, I admit, I find myself thinking back to Fred Brooks's classic "No Silver Bullet" article, where he basically said that programming environments weren't where the real problem lies -- even higher level programming environments could only go so far in attacking the essential complexity of software. So although I'm very much in favor of the model driven approach, I'd still be a bit wary of expecting too much improvement too soon.

The Crowd: Wisdom or Madness?

2010-04-10T12:09:00.008+02:00

Viareggio, 10 April 2010

Introduction

A few years ago a brother gave me a book on musical temperament. Reading the book inspired the topic of my lecture in 2003. This year, instead of a book leading me to the topic of my lecture, it’s the other way around: the topic of my lecture led me to a book.

With the rise of the Internet, Wikipedia, and social networking, the idea of talking about “collective intelligence” had been percolating in my mind for quite a while. Finally I decided to talk about it this year, and that led me to read a book called The Wisdom of Crowds by James Surowiecki – so now you know where part of the title of this lecture came from. It turned out that the title of that book was partly a homage to another book, called The Madness of Crowds. And there you have it, the title of this year’s lecture –a homage to a homage. In any case, it does capture the essence of a question I want to talk about today: are we collectively smart or stupid?

I. The Wisdom

The Wisdom of Crowds starts out with a story about a country fair in England where a crowd correctly guesses the weight of a cow. But before I get to that story, I want to dwell a moment on the story that really had me spellbound: the story of this guy’s life. I’m not talking about the author of the book. I’m talking about the man who reported on the cow-weight-guessing at the fair. His name was Francis Galton.

Francis Galton and Vox Populi

Peter Bernstein has written, “Francis Galton was one of those men of a Victorian age who roamed the earth as if he owned it.” The Victorian Age – what a time that was! If you were lucky enough to be born into the right circumstances and had your wits about you and a solid dose of intelligence, the world really was your oyster. Francis Galton certainly fit the bill. He was a half-cousin of Charles Darwin, no slouch himself. Darwin’s trip to the Galapagos Islands inspired Galton to travel to Africa, where he had his own share of adventures. One of his hobbies was admiring beautiful women and running up check marks on cards grading each woman he saw (he determined that the most beautiful women in Britain were in London, the ugliest in Aberdeen). While in Africa, he encountered the Hottentot tribe. The curvaceous figures of the women astonished him so much that he was dying to take the measurements of one of them. But he was understandably worried that a tribesman would grant him his wish (of dying, that is) if he caught him wrapping his arms around one of the women, measuring tape in hand. So he surveyed her – literally. He casually paced off several meters from where she was standing, and then used his surveying instruments to triangulate his measurements from a safe distance, thereby managing to obtain her dimensions without ever touching her (thereby saving himself a lot of trouble).

In other words, he was a genuine English eccentric. I have a friend and colleague who loves to collect stories about English eccentrics (another fan of English eccentrics is the author Bill Bryson). One thing they seem to have in common is an extraordinary intellect. Galton could read any book by the time he was four. He didn’t just dabble in several scientific disciplines, he produced important results in them. In mathematics he invented the concept of “correlation,” a keystone of modern statistics, and the concept of “reversion to the mean” (ditto). You may not immediately recognize the term, but “reversion to the mean” is actually ingrained in our everyday lives. Every time you say “this rain has to stop sometime” you’re invoking reversion to the mean. Every time you say, “what goes up must come down,” you’re invoking reversion to the mean.

Galton liked to study late into the night, and to keep himself awake he invented a machine that would douse him with water if he dozed off. He also invented a device that would allow him to read underwater, and almost drowned in his bathtub one day. He was also the proud inventor of fingerprinting. It was he who got the idea that all fingerprints were unique and did not change throughout a person’s life, and therefore would be useful for forensic purposes. On the somewhat darker side, he was also the inventor of eugenics, the study of heredity in human intelligence that gave Adolf Hitler some really nasty ideas several decades later. (To be fair, it doesn’t seem that he shared Hitler’s point of view, but things are admittedly somewhat fuzzy in the historical record.)

I said earlier that the reason Galton got into Surowiecki’s book was that story about guessing the weight of a cow at a country fair. As you can see from the episode with the Hottentot women, Galton loved to measure things. It turned out by a stroke of luck that the various guesses made by the people had been written down on cards and saved. So suddenly Galton had a mass of data on his hands – irresistible for one of the founding fathers of modern statistics. (In statistics they have a saying: “Without the data, yo’ chatta’ don’t matta’”). He took the guesses (just under 800 of them) and calculated the average. Then he noted in an article he wrote in Nature magazine in 1907:

According to the democratic principle of “one vote one value,” the middlemost estimate expresses the vox populi ...

(The title of the article was “Vox Populi,” too.) In other words, as Surowiecki put it, “That number represented, you could say, the collective wisdom of the … crowd. If the crowd were a single person, that was how much it would have guessed the ox weighed.”

The crowd had guessed 1,197 pounds. The right answer turned out to be 1,198 pounds. With typical English understatement, Galton summed it up with:

This result is, I think, more creditable to the trustworthiness of a democratic judgment than might have been expected.

Certainly more than he might have expected. As the inventor of eugenics, whose goal was essentially to weed out inferior intellects from the population, he must have been shocked that those inferior intellects got it spot on.

Finding the USS Scorpion

Surowiecki recounts another impressive story regarding a lost submarine. On 22 May 1968 the USS Scorpion (SSN-589), a Skipjack-class submarine of the United States Navy, disappeared in the Atlantic, for reasons that remain unclear today. Now, the Atlantic Ocean is a big place; and when ships sink, often the simplest thing to do is just leave them there rather than go looking for them. After all, the Titanic sank in 1912 but they didn’t go looking for it until 1985. Same with the German Battleship Bismarck: it went down in 1941, and they got around to looking for it in 1989. But the Scorpion was different: it was a nuclear submarine – and you can’t just leave a nuclear submarine lying around. So they had to find it, period.

But as I said, the Atlantic Ocean is a big place. The Navy knew the last position of the submarine before radio contact was lost, but not much else – especially since they didn’t know why it went down. Now, normally you would expect that the Navy would gather up the best experts they could find and pick the most plausible solution offered by one of them. Dr. John Craven, who was in charge of the search operation, decided on a different tack. For starters, he decided to use something called Bayesian Search Theory. This was not a coincidence: two years earlier, another nuclear device (this time a hydrogen bomb) had been lost when a B-52 plane had crashed in Palomares, Spain, and Bayesian Search Theory had been developed to help find it. (For my part, I find it rather unsettling that the military loses so many nuclear devices.)

Then, instead having specialists work together, Dr. Craven assembled the broadest multi-disciplinary team he could find – people who knew something about all kinds of different fields – and had them, independently of each other, guess what happened to the submarine. He even used bottles of whiskey as prizes in the guessing game. Using the Bayesian Search Theory, he collected all these guesses together and pieced them into a composite picture that was, “roughly speaking, the group’s collective estimate of where the submarine was.” As Surowiecki stresses, the estimated location didn’t correspond to any of the individual guesses. None of these people individually had enough wisdom to guess right. But putting them all together: they found the submarine less than 200 meters away from where their collective estimate said it would be.

Collective Intelligence

How the heck did they do it? That is what has everybody so excited. Somehow, some way, it appears that a crowd of diverse, independently acting people possesses something called collective intelligence – an intelligence that is even more intelligent than the most intelligent individual in that crowd; and there are attempts being made everywhere to harness that collective intelligence.

A prestigious example is the Center for Collective Intelligence at the Massachusetts Institute of Technology. Here they have launched a number of projects to try to exploit collective intelligence in some very surprising applications. For example, one project has the title The Climate Collaboratorium: Harnessing Collective Intelligence to Address Climate Change Issues. Another is called Collective Intelligence In Healthcare, and “ … focuses on harnessing the collective intelligence of medical professionals, researchers, and others to provide better healthcare for individual patients.” Yet another is entitled Collective Prediction, whose purpose is “ … to make accurate predictions about future events such as product sales, political events, and outcomes of medical treatments.” Let’s talk about that last one now.

Prediction Markets

The lesson from Galton’s cow-weight-guessing story is that a crowd can make a better guess than an individual. An obvious way to harness this phenomenon is to put the crowd to work in an area that, by its very nature, always involves guessing: predicting the future. It turns out that this is a huge business. There are so-called prediction markets today for just about anything. They are also called event derivatives. (You have just heard the word derivative for the first, but not the last time, in this talk.)

Some are familiar, such as horse-racing handicapping and election polls. The most famous operation is the Iowa Electronic Markets run by the University of Iowa. Others are less familiar, but no less successful. The Hollywood Stock Exchange is a virtual, Web-based game where the players effectively make predictions about things like box office success of films and this year’s Oscar winners. They always seem to do better than the “experts”. In 2007, they managed to predict 32 of the 39 major-category Oscar nominees and 7 out of 8 top-category winners. Corporations are very interested in harnessing the power of prediction markets to find out whether their products will sell.

But the oddest prediction market of all involved the military. On September 11, 2001, over and above the obvious human tragedy, the Pentagon also got a black eye for not seeing it coming, in spite of all its intelligence-gathering mechanisms. So they decided to try a more innovative approach and proposed the creation of what they called the Policy Analysis Market. It would be a prediction market like others, in the sense that it would be freely open to the general public; anybody could participate. But the topics didn’t involve Hollywood actors or horses. Rather, they involved questions like “Will the leader of Syria be assassinated?” “When will the next terrorist attack in Baghdad occur?” Officially, the Policy Analysis Market was called “a market in the future of the Middle East,” and any kind of possible political development in the Middle East was fair game for a bet.

One thing they didn’t expect was the enormous backlash from all sides. Senator Ron Wyden stated in a press conference that “the idea of a federal betting parlor on atrocities and terrorism is ridiculous and it’s grotesque.” Others objected that betting on things like assassinations could actually create “assassination markets” that would encourage some to try and make the predictions come true. The Pentagon finally gave up on the idea – much to the chagrin of Surowiecki himself, a big supporter.

The interesting thing is that policy analysis markets as such didn’t disappear as a consequence, they just migrated into the private sphere. Today, commercial prediction markets like Intrade allow bets on events like the capture of Osama Bin Laden or whether Iran will be bombed – very much the kind of thing envisioned for the original Policy Analysis Market.

Democracy

It’s all very nice that the aggregating mechanisms of collective intelligence can be harnessed for horse racing, acting award ceremonies, and the like. But why not apply them to the noblest purpose of them all? Let’s go back to that sentence of Francis Galton where he talks about the “…the democratic principle of ‘one vote one value’ …” Because in the end, this is what democracy is all about, isn’t it? The people’s choice, vox populi. And what could be more important than making democracy work as effectively as possible, so that the best collective choice is made by the people? And that brings us to the subject of the aggregating mechanism for political choices of the crowd: voting.

Galton reminds us that at the heart of democracy is the simple principle “One Person, One Vote.” But that simple principle obscures a question with a surprisingly complicated answer: “What is the best way to implement the voting process?”

Many of you will have voted in the regional elections here in Tuscany two weeks ago [28 March 2010]. As usual in Italy, there were many parties on the ballot, and I’m sure that at least some of you didn’t vote for the person you really wanted, because of a classic voting dilemma that might be called the “outsider” dilemma. It goes something like this: “I know that my candidate doesn’t really have a chance, so if I vote for my candidate, I’m throwing away my vote – or worse, I’m playing into the hand of the candidate I really don’t want to win – so I’m going to vote for this other candidate instead, who isn’t the one I really want, but is better than the one I really don’t want.” The classic outsider presidential candidate in the United States for many years was Ralph Nader, the consumer rights crusader, and many people who would have liked to vote for him did not, for exactly the reasons I outlined above.

This problem of “not wanting to waste your vote” manifests itself in many ways. Last month [March 2010] people were pretty shocked when the historic U.S. health reform bill passed in the House without one single Republican vote. Aside from the obvious partisan issues, there was a feeling that the voting system itself is broken, so there’s a lot of interest right now in finding better ways to vote. A few weeks ago [24 March 2010] New York Times columnist Thomas Friedman suggested introducing the Alternative Vote – which is actually known under a variety of names: Instant Runoff Voting, the preferential ballot, and ranked choice voting. The basic idea is that you can list more than one candidate, in order of preference. It has been in use in Australia, for example, for many years (although several readers reminded Friedman that another essential characteristic of voting in Australia is that it is compulsory).

Alternative Voting is only one of several different voting systems – exhaustive ballot, contingent votes, two round systems, and others. The point is that it’s not just the collective intelligence of the crowd that matters, it’s how you aggregate their choices in the best way to produce the best result. And as I said, where could that be more important than in our political systems?

Ants

Prediction markets and elections provide an example of harnessing the collective intelligence of human beings. But the classic examples of collective intelligence that everybody is more familiar with are found in the animal world. Insects like ants and termites build these amazing structures by just following simple rules that they instinctively know. A well-known phenomenon is called the “Ant Spiral of Death”. This happens when ants get lost for some reason. A lost ant obeys a simple rule like “follow the ant in front of you.” But when they’re all lost, they sometimes just end up forming a circle by following that rule, and keep walking and walking, sometimes for days, until they literally drop dead from exhaustion.

But normally their innate rules work very well, and allow them to create structures of jaw-dropping complexity and sophistication. There are several videos on YouTube that show excavations of large ant colonies by researchers. In one such video the narrator remarks,

[After weeks of excavations], at last they begin to see the structure of the city-state. There are subterranean highways connecting the main chambers, and off the main routes are side roads. The paths branch and lead to many fungus gardens and rubbish pits. The tunnels are designed to ensure good ventilation and provide the shortest transport routes. Everything looks like it has been designed by a single architect, a single mind, but of course that isn’t true. This colossal and complex city was created by the collective will of the ant colony – the super-organism. The structure covers 50 square meters and goes 8 meters into the earth. In its construction, the colony moved 40 tons of soil. Billions of antloads of soil were brought to the surface. Each load weighed four times as much as the worker, and in human terms, was carried a kilometer to the surface. It is the equivalent of building the Great Wall of China. It is truly a wonder of the world.

Emergence

The giant ant colony is an example of what is called emergent behavior. Through the individual behavior of numerous “agents” (e.g. ants), a kind of collective behavior emerges that can only be associated with the whole – the super-organism, as the narrator above called it. Here, too, the military has poked its head through the window to see if there is anything it can make use of. Sure enough, there is active research going on in the military on what is known as swarming.

A swarm of insects is an object of envy to military aviation planners. We’ve all seen how they move around in formations that compose and recompose themselves without any seeming instructions from anywhere. How does their “command and control” system work? The Air Force would like to know, and it would like to harness it for its own use. They have this idea of creating swarms of Unmanned Aerial Vehicles (UAVs) that they send out into the field against the enemy. Like insect swarms, there would be lots of them (thousands); they would be self-sufficient; their behavior would not be entirely predictable; and they would be resistant to individual losses.

If you’re nervous about the idea of a bunch of deadly air vehicles swarming around, with nobody sure about just what they’re going to do next, you’re in good company. The military’s biggest problem is convincing people this won’t get out of control.

Indeed, this is the flip side of emergent behavior: it’s not always a good thing. Sometimes you don’t want emergent behavior. Sometimes, when you combine things, you just want them to do what they know how to do and nothing else. In my work in safety-critical systems (e.g. automobiles, trains, space systems), emergent behavior is a huge problem. Engineers are always worried that, even after putting a bunch of well-tested components together in, say, a car, they’ll interact in such a way that some totally unexpected (and ruinous) behavior emerges out of the whole. You want your car to do what you tell it to do. You don’t want it to acquire a mind of its own.

Mitchel Resnick of MIT has created a programming system called StarLogo for studying environments with lots of simple agents (ants, fish, termites, etc.) obeying simple rules. StarLogo is based on the original Logo system, which was developed by Seymour Papert of MIT for teaching children to program. It can be freely downloaded from the Internet and has been successfully used in high school educational programs. Based upon his work studying the “massively parallel microworlds” of ants and the like, Mitchel Resnick has written a book entitled Turtles, Termites, and Traffic Jams – which brings me to the next subject.

Traffic

Three things have always puzzled me; after all these years, decades, sometimes even centuries of research:

· We still can’t predict the weather. Actually, we do finally have a plausible explanation from Chaos theory regarding why this is the case (and why it will always be the case). You can find a discussion of that in my previous lecture on Chaos theory.

· We still don’t know why we sleep. Incredibly, scientists still don’t really know why we need sleep. We sleep for a third of our lives and we still don’t know why! To a great extent, scientists have given up for now on the why, and are instead concentrating on finding ways to make sure that we get enough of it.

· We still don’t know what causes traffic jams.

I would have thought that, by now, people would have figured out what causes traffic jams and done something about it. It turns out it’s not that simple, and incredibly, there are enormous academic disputes about just where the problem lies. There are a number of perfectly respectable scientists who even say that traffic jams have no cause at all – they just appear “spontaneously”.

If ants can (mostly) manage their traffic smoothly, why can’t humans? A little reflection yields an intuitive idea of the reason: the ants are all following some kind of common set of rules that yields a high degree of coordination. But human drivers all have minds of their own. Certainly they all have to follow the basic rules of traffic, but aside from that, they pretty much do what they want to do. So one idea is to have drivers behave more like ants – that is, get them to coordinate better.

The so-called automated highway system (also called a Smart Road) is an example of this. They actually built one of these about 20 years ago, in Southern California. The road had sensors built into it that basically took control away from the driver and gave it to the cars. The result was that the cars all lined up beautifully and traffic flow was smooth as silk.

The problem is that drivers don’t like to give up their autonomy. They don’t want the road – smart or not – telling them what to do. They want to be in the driver’s seat, so to speak. So people are looking for ways to provide coordination without taking control away from the driver. I have actually come into contact with projects in which they have done this. The HIDENETS project, sponsored by the European Commission, studied the concept of car platooning. This consists of cars connected with each other with wireless communication so that each knows what the other is doing and they can try to coordinate, kind of like a flock of birds. Time will tell how well that idea works out – in the meantime we’ll have to get used to sitting in those traffic jams.

We’ve spent a lot of time so far talking about the different ways in which people are harnessing the wisdom of crowds, whether in finding lost things, or predicting the future, or getting traffic untangled. I think it’s time to take a look at the other side now.

II. The Madness

We had that great British eccentric Francis Galton to introduce us to the wisdom of crowds. And now we are fortunate to have his Scottish contemporary, Charles Mackay, to introduce us to the madness of crowds. Was Mackay also an eccentric? Well, he certainly shared with Galton the characteristic of being brilliant in a number of different fields. In fact, in his life he was most famous for his songwriting (like the popular Cheer, Boys, Cheer). He also wrote a dictionary of Lowland Scotch.

But he is known today for a book entitled Extraordinary Popular Delusions and the Madness of Crowds, published when he was only 27 years old. The book is still in print– in no less than six different editions. (And by the way, it is in the public domain and you can download it free on the Internet; I did just that, and put it on my Kindle to read.) The book gets off to a roaring start like this:

Men, it has been well said, think in herds; it will be seen that they go mad in herds, while they only recover their senses slowly, and one by one.

So why is the book so famous? It actually covered a broad swath of topics that ranged from witch hunts to alchemists to the crusades. But the part of the book that earned Mackay immortality concerned economic bubbles. It’s probably not surprising to most of you that there is so much interest in financial craziness – after all, it’s been in the news a lot lately with the worldwide economic crisis. What may be surprising to you (and it certainly was to me) is how long this craziness has been going on. Even without the advantages of modern technology, transport, and communication, people have been managing to create enormous financial messes for centuries.

One bubble described by Mackay involved the South Sea Company, which was founded in 1711. The company had a monopoly on trading in Spain’s South American colonies and speculation in the company’s stock eventually led to a huge crash. Then there was the Mississippi Company bubble around the same time. This one also had to do with speculating on the riches of New World colonies of European countries – but this time the European country was France and the New World colony was Louisiana. In any case, the result was the same: a huge crash and a lot of ruined lives.

But the most famous bubble described by Mackay occurred nearly a century earlier than the other two, and this time a European country was able to make a perfectly good financial mess all on its own, without any help from New World colonies.

The Dutch Tulip Mania

The economic bubble and crash that has the dubious honor of being the first on record occurred in 17^th Century Holland. Some things are so closely associated with a country that we forget they weren’t always there. It’s hard to imagine Italy without the tomato (in the region of Puglia where my wife is from, they call it oro rosso – red gold), but of course it has only been there since the discovery of the New World. Likewise, think “Holland” and the first thing that comes to mind is the tulip. But the tulip was only introduced into Holland in the mid-16^th century from Turkey. Tulips were soon a big hit in Holland and everybody wanted them. Prices got higher and higher. In order to satisfy demand, an exchange was created in which people could buy “tulip futures” – contracts on future deliveries of tulips. These first-of-a-kind financial “instruments” were then further refined as options – the opportunity but not the obligation to buy tulips. Options are a form of – and here’s that word again – derivatives.

By early 1637, a tulip contract was selling for more than ten times the annual income of a skilled craftsman. But as we all know now from Francis Galton and the principle of reversion to the mean, “what goes up must come down.” In February 1637 the market crashed, and a legend was born: the Dutch Tulip Mania.

To this day, financial bubbles – for example, the dotcom bubble in the late 1990s that burst in the year 2000 – are commonly referred to as Tulip Mania.

To be fair, modern-day research into the Dutch Tulip Mania has led to other opinions about just how maniacal it all was. For one thing, some think that the reporting wasn’t always unbiased; for example, it is thought that some reporters like Mackay might have had a religious and moral agenda, and overhyped their stories to warn people about the evils of financial speculation (we could probably use some of that today). Furthermore, there are those such as Earl Thompson of the University of California at Los Angeles who argue that not only was the Dutch Tulip Mania not maniacal, but on the contrary could be explained in perfectly rational terms, as an excellent illustration of what is known as the efficient market hypothesis. So what’s that? It has a lot to do with the (surprisingly) fascinating story of modern finance, which I’d like to talk about now.

Efficient Markets

Economics was rocked to the core in the twentieth century, starting in its very first year. In 1900, a young French mathematician named Louis Bachelier published his doctoral thesis, called The Theory of Speculation. I have to admit that this time the person we are dealing with is neither British nor eccentric (unless having a wine merchant for a father and a poet for a mother counts), but young Bachelier certainly did have extraordinary talent. His thesis involved something called Brownian Motion (named after the Scottish botanist Robert Brown), which describes the motion of particles in a fluid. The interesting thing about this is that in doing so, he anticipated the Nobel Prize winning work of none other than Albert Einstein by five years – which is a pretty cool thing to be able to say about your work. Bachelier used Brownian motion to study the movements of stock markets; Einstein merely used it to prove the existence of atoms. In addition, Bachelier’s thesis involved the study of stock options, which we now know from the Dutch Tulip Mania are a form of derivatives (that word again). He invented a way of describing the value of options that still bears his name: Bachelier diagrams.

With his sophisticated use of Brownian motion and the like in his work, Bachelier could arguably be called the father of “mathematical finance,” which eventually came to dominate most of the twentieth century, as we’ll see. But the part of his work I want to talk about now is a statement he made as a conclusion of his research.

Bachelier claimed that the movements of stock markets are random.

When you read that claim, your first thought is, “Maybe Bachelier really was an eccentric; no, maybe he was downright crazy.” Because it doesn’t make any sense. It’s obvious to anybody that stock markets don’t just wander around randomly like a bunch of drunks; they move in response to real, significant events, like company reports, mergers, acquisitions, all those things. They move in patterns. Only a fool would make such an absurd claim as Bachelier’s. And so he was ignored by pretty much everyone.

The years passed. A Great War devastated Europe (Bachelier himself was drafted into the French Army as a private). We lived through the greatest financial madness of all time and the subsequent stock market crash of ‘29, followed by the Great Depression. Then the Second World War, which gave us, among much else, the electronic computer (invented for military purposes, of course).

Then, in 1953, more than half a century after Bachelier’s original work on the stock market (Bachelier was dead by that time), a British fellow named Kendall, armed with piles of stock market data (without which, his chatta’ didn’t matta’) and all the modern tools of computing, decided to go about finding and categorizing all the patterns in that data. Who knows, maybe he even secretly harbored the hope of making some easy money – there is an active branch of finance called technical analysis, which tries to identify patterns in the stock market and exploit them for financial gain, and if Kendall were able to identify and categorize all the patterns, he could be a step ahead of the pack. Whether or not he had that in mind (unlikely, actually), Kendall took his piles of data and “crunched the numbers,” and then tallied up the grand total of patterns he had identified.

Zero.

Not one single pattern. Nothing. After a lot of head-scratching, people began to remember Bachelier’s work. But this time they couldn’t ignore it like before – they had hard data that confirmed it. Stock market movements really are random.

But knowing the fact didn’t help people understand why it was true. Economists had to wait another decade before the first plausible explanation arrived, which is still with us today. It was provided by Professor Eugene Fama of the University of Chicago, and has become known as the Efficient Market Hypothesis. Here’s the basic idea. Think of the stock market as a pond full of voracious, greedy piranha (come to think of it, the analogy is probably pretty good from several points of view). When any new information arrives – like a company reporting great sales, or acquiring another company – that sort of thing, the piranha jump on the information, analyzing it, looking at it from every perspective. They “strip it to the bones,” leaving no fact undigested, and then they act on it. If it’s good news, they buy; if it’s bad news, they sell. And here’s the important part: they not only act on the immediate information, but on all of its implications for the future: after all, if you think a company’s stock price is going to go up in a month, why wait a month to buy it? You buy it now. The result is that the market always reflects all the information that is currently available. Its next reaction will be to the next new information, and of course, by definition, nobody knows when that will arrive – in other words, the arrival of new information is random and thus so are the market’s movements.

So now you know why market movements are random. But the characteristic of efficient markets that is relevant to this lecture is that this “piranha pond” consists of investors of all types, from all backgrounds, all shapes and sizes, all levels of intelligence, all interpreting financial information from their own point of view and drawing their own conclusions. Like a kind of large, sprawling financial democracy, each investor, through the act of buying or selling a stock, is effectively casting a vote that reflects his opinion of that stock’s value. Sound familiar? Yes indeed, we are once again back to the wisdom of the crowd, and a corollary of the efficient market hypothesis is that the crowd of piranha, in its collective wisdom, gives us the best possible estimate of the value of a stock – better than even the smartest individual can do. Or, in layman’s terms: “you can’t beat the market.”

And that’s why people are always so angry at professional money managers. Only the smallest percentage of them ever manages to do better than you would by just spreading your money around in the market. (The popular mathematician and author J.A. Paolos calls that situation “slightly scandalous” and it’s hard to disagree with him.)

Behavioral Finance and the Human Factor

The efficient market hypothesis dominated finance for the second half of the twentieth century, and reinforced the impression of stock markets as rational, well-functioning investment mechanisms – the finest possible illustration of the Wisdom of the Crowd. But a nagging question remained: if the stock market is so rational and wise, what about all those things Charles Mackay was writing about? The Dutch Tulip Mania? The Mississippi Company? For that matter, what about the Great Crash of 1929?

In the last decade of the twentieth century, some people began to question whether the wisdom of the stock market hadn’t maybe been a bit exaggerated, and its madness a bit downplayed. One of those people was Robert Shiller, an economics professor from my own alma mater, Yale University. Like Francis Galton, Shiller also preferred to work with real data (instead of just chatta’) and basically said, “Okay, if stock market prices are supposed to be the wisest estimate of the real value of companies, let’s go verify whether that has turned out to be true in the past.” After all, data was lying around from over a hundred years of stock market activity and company financial reports. All he had to do was compare them with each other to see whether the crowd had guessed right – in retrospect, it’s surprising that nobody had thought of it before. Sure enough: Shiller found that, time and again, those “wise” investors were way off the right estimate of a company’s real value. His explanation? Simple: human foibles – the madness of the crowd.

Shiller became one of the leading proponents of a new discipline known as behavioral finance, which explicitly acknowledges the influence in financial decision-making of overconfidence, overreaction, and all those other quirks of humanity we know and love. His comment: “When we started doing behavioral finance we were total outcasts … nobody appreciated us. I had tenure so I could do it …”

In the year 2000, right at the very height of the dotcom boom, Shiller published a book called Irrational Exuberance, warning about the current madness of dotcom investors. (This madness was going on in Italy, too, by the way – one company was stampeded by crazed investors desperately trying to buy its shares because it had a product with “Net” in its name. It turned out that the product, WC-Net, was a detergent for cleaning toilets). When the market crashed shortly thereafter, a lot of believers in behavioral finance were born.

In 2005, Shiller published a revised version of Irrational Exuberance. This time he included a discussion of the growing housing bubble. He pointed out, for example, that median home prices in the United States had grown to be as much as nine times greater than median income (remember that discussion about Dutch tulips costing ten times the annual income of a skilled worker?). And as we now know, he was right – the madness of the crowd had struck again and we are still now enduring its consequences. This time the culprit wasn’t the Internet. It was something far more sinister – that word again: derivatives. The legendary American investor Warren Buffett has called derivatives “financial weapons of mass destruction.” You may have also seen the news last month [March 2010] that four banks were being put on trial for their handling of derivative transactions with municipalities in Italy.

As a matter of fact, though, Shiller isn’t against derivatives per se. He shares the view of James Morgan, a columnist from the Financial Times, who said, “A derivative is like a razor. You can use it to shave yourself … Or you can use it to commit suicide.” Shiller is in favor of derivatives when they are used responsibly to diminish risk. But derivatives can also be used for very wild and aggressive speculation, and seem to bring out the worst madness in the crowd. Witness the mess the world is in right now.

The current financial disaster has thrown the field of economics into a serious identity crisis. The century that began with brilliant economists like Louis Bachelier laying the foundation for a rational, crystalline mathematical model of financial behavior ended with brilliant economists like Robert Shiller tearing down that very foundation. In an article last week [26 March 2010] in the New York Times, David Brooks discussed the causes and nature of all this soul-searching among economists. The principal cause is that so few economists saw the crash coming.

“Where were the intellectual agenda-setters when this crisis was building?” asked Barry Eichengreen of the University of California, Berkeley, in The National Interest. “Why did they fail to see the train wreck coming?”

I mean, it’s pretty embarrassing when the worst economic collapse since the Great Depression happens and nobody (except one or two like Shiller) foresees it. It kind of makes you reflect. Right now, in the tug-of-war between the Mathematical Economists and the Behavioral Economists, the behavioralists seem to have the upper hand. As Brooks concludes,

Economics achieved coherence as a science by amputating most of human nature. Now economists are starting with those parts of emotional life that they can count and model (the activities that make them economists). But once they’re in this terrain, they’ll surely find that the processes that make up the inner life are not amenable to the methodologies of social science. The moral and social yearnings of fully realized human beings are not reducible to universal laws and cannot be studied like physics.

In other words: human nature being what it is, there will always be a little madness mixed in with the wisdom. And that has implications for the limits on how far we can go with these ideas, as we’ll see now.

The Wisdom of Corporations?

As I said earlier in this talk, people have been trying to harness the wisdom of the crowds in many different fields, and the field of business and finance is no exception. In fact, Surowiecki’s book The Wisdom of Crowds was a New York Times bestseller not in the general category, but in the business category; Surowiecki’s hope and intention was to inspire businesspeople to find ways to harness the ideas in the book. For example, he outlined ways in which the ideas of collective intelligence could be used to produce superior corporate decision-making processes.

I had the occasion in Rome exactly one month ago [10 March 2010] to meet a person who is working in this area right now. Professor Michele Missikoff is a director of the Laboratory for Enterprise Knowledge and Systems at the Italian National Research Centers in Rome. He has been trying to raise awareness in Europe of the possibilities for harnessing the power of the crowd in the enterprise. He is promoting a view that each person in the enterprise can be considered as a valuable carrier of knowledge – however partial and incomplete – that contributes to a collective, enterprise-wide “wisdom” that is more powerful than anything any individuals could put together on their own. This could lead to whole new ways of managing knowledge in the enterprise.

“Considering the importance of the enterprise in our economy, it’s amazing that more hasn’t been done to try to make it function better,” he told me. “The current economic crisis has made authorities in the European Community more aware of the need to look at enterprise improvement from an economic point of view rather than just a technical point of view.”

“So what’s holding back progress?” I asked.

“The enterprise is the single most complex structure ever made by humans,” he explained. “The problem is that it’s not just made by humans, it’s made of humans. And human nature being what it is … for example, it’s not always easy to convince a manager that he should relinquish his hard-earned right to make decisions to the collective intelligence of some kind of group.” He shrugged his shoulders and smiled, “That’s just the way we humans are.”

III. The Future

I mentioned at the beginning of this talk that I was originally led to this topic not by reading a book, but rather by what I was seeing happen all around me. Examples of trying to harness collective intelligence are everywhere now, especially on the Internet. You don’t have to look any farther than your home page on the Web for a great example of this – that is, if your home page is Google, the famous search engine.

Did you ever wonder how Google actually finds the things you look for on the web? Among all the web pages out there, how does it find the ones that are relevant to your search, sort them and rank them so that you get the best ones? Simple: it uses a technique for ranking pages imaginatively named … PageRank.

Now, you could be forgiven if you thought that the name stood for “ranking pages.” But you’d be wrong. The “Page” actually refers to Larry Page, one of the co-founders of Google. He developed this technique while he was a student at Stanford University (by the way, the work of an Italian, Massimo Marchiori at the University of Padua, is cited as being one of the sources of inspiration for this technique). Another interesting fact is that, although the name PageRank is a trademark of Google, the patent on the technique is actually owned by Stanford University, and they made a bundle of money licensing the exclusive rights to Google.

So how does this page-ranking technique work? Let’s hear it from Google themselves:

PageRank relies on the uniquely democratic nature of the web by using its vast link structure as an indicator of an individual page’s value. In essence, Google interprets a link from page A to page B as a vote, by page A, for page B. … Votes cast by pages that are themselves “important” weigh more heavily and help to make other pages “important”.

Democratic. Vote. Those words again! Yes, that’s right; Google essentially harnesses the wisdom of crowds to let the Internet itself tell you which pages are relevant to your search.

Perhaps that’s not so surprising, because the Internet is the most fertile ground you could possibly imagine for harnessing the wisdom of crowds. And even less surprising is that where it’s all happening right now is in the social networks – the ultimate aggregators of crowds.

There probably aren’t three people in this room who aren’t registered on Facebook (Italy has one of the world’s most enthusiastic Facebook communities). Last year I introduced you to Twitter, who taught us that the burning question on everybody’s mind was “What are you doing right now?”

This year, we have gone one step further. Writing a couple of weeks ago [21 March 2010] in the New York Times, David Carr described the unnerving experience of sitting in a bar in Austin, Texas, where suddenly about seventy people, all sitting in different parts of the bar, got up out of their seats and rushed over to another bar. What made them all suddenly move together as a group, like a swarm of bees?

At large events, people have always moved in groups to the next big thing. But … the ubiquity of so-called ubiquitous presence — location-based services like Foursquare and Gowalla — meant the hive suddenly knew what it was collectively doing. … It was striking to see the digital location effect in the wild, with people reacting to an unseen dog whistle and moving en masse, on command.

Ubiquitous presence? Location-based services? What on earth is he talking about? This is the new frontier in social networking. The burning question isn’t just “What are you doing right now?” any more, but also “Where are you right now?” Social networking is now morphing into geosocial networking. In geosocial networking, friends not only tell each other what they are doing but where they are doing it.

But how do they do this? It has a lot to do with the concept of ubiquitous presence mentioned above. This is a hot topic in the research programs of the European Union right now, as a matter of fact. In its so-called Seventh Framework Programme of Research, the term “ubiquitous presence” is cited often, and refers to the idea of everything being interconnected, from your refrigerator to your clothes to you yourself. (Another name commonly used in association with the concept is the Internet of Things.) But how do you connect you to others? Simple: your mobile phone.

Foursquare, one of the “location-based services” mentioned above, works like this: when a person enters a place (like a popular bar) he can check-in using his smartphone. In that way, his friends all know where he is. Of course, Foursquare also finds out where he is, and privacy issues immediately come to mind. Carr writes:

To someone not in their 20s whose location generally isn’t that interesting to others … the idea of handing over your privacy with both hands to strap on a digital ankle bracelet sounds profoundly unattractive. … But to a younger cohort that lives on the grid, the location of people you know and care about is vital information, the coin of the realm.

Like in a beehive, we are beginning to see new forms of emergent behavior in social networks, like groups suddenly standing up in a bar and moving to a different place. Like those cars coordinating with each other on the highways, think of this as a kind of “human platooning,” where everybody knows where everybody else is. This kind of collective, emergent behavior is also encouraged by services like Foursquare, who have introduced game-like facilities and rewards into the service. For example, you can earn badges by checking into places. And if you check in the most times to a popular place (like a bar), you can be named “mayor” of that place. These kinds of awards are reminiscent of the whiskey prizes using in finding the submarine Scorpion, and in the same way, they encourage the crowd to interact in all kinds of unexpected ways.

Furthermore, location-awareness is not destined to remain in the purely physical world. As Carr writes:

… What if location became not just a physical place, but a digital one? The possibilities for old and new media could be significant. “The check-in is bigger than location,” said Yancey Strickler of Kickstarter, a Web site that helps with fund-raising for media products. “Think of media: Checking into watching ‘Lost,’ being declared the mayor of ‘The Brothers Karamazov’ or earning a badge for braving free jazz.”

Recently, much larger social networking companies like Facebook and Twitter have also announced that they will soon provide location-aware facilities. This means that crowds will arise more and more often, with their members getting together and acting in collective, emergent ways that we can’t begin to predict. Where will all this end? Once again I find myself at the end of a talk having to use the same old, tired line: Only time will tell.

So I guess I didn’t do a very good job of answering the question I posed at the beginning of this talk: are we collectively smart or stupid? As usual, I ended up equivocating: we’ll always be a little wise, and we’ll always be a little mad. But most of all, more and more we’ll be we. Unlike the individual, royal “We” of Francis Galton’s beloved Queen Victoria, the “we” of the future will be ever more connected, interacting, self-aware, the ultimate universal swarm of humanity.

Resources

James Surowiecki, The Wisdom of Crowds, Anchor (August 16, 2005)

Charles Mackay, Extraordinary Popular Delusions and the Madness of Crowds, with a foreword by Andrew Tobias (1841; New York: Harmony Books, 1980). ISBN 0-517-53919-5. Also available free on the Internet, e.g. www.manybooks.net.

Toby Segaran, Programming Collective Intelligence: Building Smart Web 2.0 Applications, O'Reilly Media; illustrated edition (August 16, 2007)

Don Tapscott, Anthony D. Williams, Wikinomics: How Mass Collaboration Changes Everything, Portfolio Hardcover; Expanded edition (April 17, 2008)

Jeff Howe, Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business, Three Rivers Press; unedited edition (September 15, 2009)

Richard Ogle, Smart World: Breakthrough Creativity And the New Science of Ideas, Harvard Business School Press; 1 edition (June 5, 2007)

Keith Sawyer, Group Genius: The Creative Power of Collaboration, Basic Books (June 4, 2007)

Steven Johnson, Emergence: the connected lives of ants, brains, cities and software (2002) Scribner, ISBN 0-684-86876-8.

Mitchel Resnick, Turtles, Termites, and Traffic Jams: Explorations in Massively Parallel Microworlds (1997), MIT Press, ISBN 0262680939.

Peter Bernstein, Against the Gods: The Remarkable Story of Risk (1998), Wiley & Sons.

Peter Bernstein, Capital Ideas: The Improbable Origins of Modern Wall Street (1992), Wiley & Sons.

Thirty Years of Domain Engineering

2009-05-07T12:43:00.002+02:00

“Thirty years?” At least some readers will be asking themselves that question about the title of this post right now. “But domain engineering isn’t more than about ten or fifteen years old! The conferences on product family engineering only started in the mid-nineties!”

In his wonderful book A Short History of Nearly Everything, Bill Bryson cited a quotation that went something like this: The history of a new idea generally passes through three phases. First, the idea is not understood by anybody. Second, it is finally understood and accepted after years and years. Third, it is then attributed to the wrong person.

James Neighbors introduced the concept of domain engineering in his PhD thesis in 1980. (By the way, aside from coining the expression “domain analysis” in his thesis, he also wrote about “domain specific languages”, nearly thirty years before a special issue in IEEE Software on DSLs was enthusiastically received by a large practicing community.) He expanded on that work over the next four years with his Draco system, which essentially introduced the domain engineering process as it is known today in the product line community. Draco was so innovative that colleagues who were reimplementing it in other places were still grappling with its subtleties years later.

But when I have spoken with many in the software product lines community, they admit that they have never even heard of Neighbors and his work (even though it was published in the standard, widely available channels). When they are told about it, they offer an explanation of why the product lines work is somehow different (it usually has something to do with “commercial focus” or the like). But the explanations are generally not very convincing, at least to me.

It is ironic indeed that in the field of software reuse, of all fields, history also is being forgotten. In precisely the field that preaches not reinventing the wheel, too many of us are doing just that, by not knowing what has been done before. And it is too bad not just for reasons of correct attribution, but also because we are depriving ourselves of some great work. Some of the best work ever done in computer science was done early on. Alan Perlis once said that the programming language Algol60 was “a great improvement over most of its successors.” A lot is still with us today – Lisp was invented in the 1950s and is still going strong. Many very deep concepts were invented, although not all panned out – such as “call by name” in Algol60. But even many of those concepts that didn’t pan out were simply ahead of their time, and bound to come back when the world was ready (either through better technology, or mindset, or whatever).

A lot of things that were explored in the early days of reuse are coming back now, such as introducing systematic reuse into organizations. That’s proof of their viability. The mindset is there now, the technology is more powerful than it was twenty years ago when it was first tried. But that doesn’t mean those earlier efforts were without merit. Why give up the chance to benefit from the insights of those who went before? Aside from the issue of giving credit where credit is due, we’re doing a disservice to ourselves by ignoring our past.

Agile + Reuse = Efficient Projects

2009-04-22T14:05:00.006+02:00

A couple of years ago I wrote a paper with some reflections on how the dynamics of the capital markets could help illuminate the dynamics of agile software development projects. Briefly: the Efficient Market Hypothesis explains how the market ideally will reflect all information available to investors. One point that is often overlooked about this is that an efficient market also reflects all implications for the future based upon the current information available. (In other words, if you think a stock price will rise in two weeks you buy it now, you don’t wait two weeks to do it.)

I then elaborated the idea of an Efficient Project: agile developers try to construct software that way. Just look at the so-called XP Customer Bill of Rights: “You can cancel at any time and be left with a useful working system reflecting investment to date.” The system reflects all the information available to date.

Furthermore, the YAGNI (You Aren’t Going to Need It) principle of XP says that the system should reflect only the information available to date. Don’t implement anything that isn’t implied by the requirements you have now. Don’t try to second-guess the future.

There’s even a parallel in there to the markets: momentum investing is one of the causes of market inefficiency; it causes bubbles, with people investing beyond what the current information (e.g. company revenues) implies. “Momentum implementation” occurs when projects implement features beyond what is called for by the requirements.

This is great advice. It’s a way of keeping systems from adding useless functionality. However: I think there has been a traditional tendency in the agile community to go too far with this. Probably because the agile movement began with projects with truly unstable requirements, a tendency has grown up around this to consider the future to be entirely unpredictable.

But the future is in fact rarely entirely unpredictable. The only project with entirely unpredictable requirements would be one in which you sit down in front of a white sheet of paper and say, “I’m going to do … something – anything.”

The fact is that there is a continuum of predictability of the future, of requirements. But the agilists have made it a bit too easy on themselves in that respect, and tend not to look hard enough for those aspects of the system which are predictable.

And that’s where reuse comes in. Reuse is all about being able to predict the future. In some ways it is the mirror image of agile. It says, “I can predict the future in these important ways, and I can implement a system that reflects the implications of this future.” It's the "You Are Going to Need It" of software engineering.

And this isn’t just dreaming. One of the most successful examples of this is product line development. The people in Nokia lay out an entire vision for future software in their phone families, and implement the corresponding system now. The product line developers also study where the future is unpredictable, say, in terms of features, and even then try to constrain the unpredictability to squeeze out any amount of partial predictability they can. For example, they introduce variation points in feature models that say, “okay, we’re not sure exactly what a system’s features will be, but we can get it down to variations on this theme and implement based on that.”

It’s tough analysis to do, there’s no silver bullet, but it’s rewarding when it works. And agilists tend to exclude this type of analysis due to a mindset that focuses on unpredictability.

This is not to say that agile and reuse are opposed. On the contrary, they are probably the two most important software engineering techniques we have, and can work together to balance the unpredictability and the predictability of systems and their features.

Surprised to hear this? You shouldn’t be: speaking of silver bullets, probably the most important software engineering paper ever written was No Silver Bullet by Fred Brooks (Martin Fowler once told me that no other paper had had so much impact with so few words.) Brooks said that the software development problem was essentially intractable, and that there were only a few truly powerful tools to combat the problem. Two in particular he mentioned: software reuse was one of them. Another was the idea of “growing a system” – essentially the idea of incremental, agile software development.

Why did he single out reuse and agile? Because he said that essentially the only solution to the software problem was to write less software. Reuse is the way to write less software when the future is predictable. Agile is the way to write less software when the future is not predictable. Leave either one out and you end up writing more software than you should. Use them together and you have an efficient project - a project in which the amount of software written is not too much and not too little.

Reuse and agile – if they were good enough for Brooks, they should be good enough for us.

Agile and Reuse

2009-04-14T09:16:00.004+02:00

There was an interesting discussion over on the Yahoo XP Discussion List over the last few days on the topic of "reuse across projects." One thing that strikes me is that nearly all of what was said during that discussion has been said many times before. This in itself is not necessarily a problem, but it does leave me with the impression that many remain unaware of the reuse community and, in a kind of ironic twist, "reinvent the wheel" of discussion around reuse.

The agile community has a particularly uncomfortable relationship with reuse. I can testify to this on the basis of discussions all the way to the top -- yes, the top -- of the community where skepticism was expressed. In the discussions over the past few days, the idea of "emergent reuse" was cited with approval. But what is that, if not the notion that reuse only makes sense after several exemplars have been made? Once again, the wheel of thinking about reuse reinvented.

Borrowing an anecdote from another time: I once saw Jean Samet speaking at the History of Programming Languages conference. Defending COBOL from its detractors, she noted that only COBOL had a truly complete facility for I/O. The others punt (and it's true, just look at C and Ada, which farm it out to libraries). She said, "And you know why? Because it's hard, that's why." Simple as that.

I defend the agilists all the time with that anecdote. I tell people that agile may, in its essence, "only" be iterative software development, but that detracts nothing from the fact that they were the ones to finally make iterative software development happen. Why? Because it's hard, and that's why people didn't do it before. It's hard to plan iterations, to time-box them, to re-plan, etc. But the agilists simply rolled up their sleeves and did the hard work of figuring it out and putting it into practice.

The agilists should put this attitude to work and realize that reuse isn't practiced often enough for the simple reason that it's hard. It's hard to distill that perfect interface that makes software easily reusable. It's hard to provide the robustness and elegance needed to make reuse work.

Agilists are always inviting other communities to become familiar with what they're doing before judging them. I think the agilists should become familiar with the reuse community ... even better, participate in it. Come to the Eleventh International Conference on Software Reuse in Washington this September. We can talk about it.

The Contractual Process

2009-04-09T12:10:00.001+02:00

A few days ago during an agile workshop I was explaining the concept of optional scope contracts in agile processes, and listening to myself talk, the word “waterfall” suddenly came to mind. I had never thought of it in those terms before, even though this is clearly what it is.

What I hadn’t been thinking clearly about was the fact that contracting also has a process, which usually tracks or mirrors the software development process, but doesn’t have to. But we don’t see that until we radically change the software process away from the classic waterfall process.

The waterfall process for software development has a number of known problems. The agile process (basically an iterative process) arose in response to the need to get risk and scope under control, allowing to developer to reassess the state of development continuously and intervene to take decisions.

But when agile processes are adopted, it’s actually the exception more than the rule that the contracting process is changed, too. The contracting process stays waterfall (requirements up front, etc.). We end up with a mismatch between the two processes. If people were to think this way, in terms of processes, maybe they would start “getting it” about the possibility of doing contracting in different ways – effectively, an agile contracting process.

Reconciling IT Governance and Quality

2009-04-08T16:21:00.000+02:00

After the Total Quality Management (TQM) wave that swept over the industry at large during the 1980s, and the success of ISO 9000 for the software industry in the 1990s, the quality imperative has continued its march in the new millennium with a spike in popularity of Six Sigma. Today it is not unusual to see a company’s commercial brochure highlight its special commitment to quality, as a way of identifying itself as a “quality company.”

Is there anything wrong with being a “quality company”?

Six Sigma dates from 1986, but was popularized along with the ISO 9000 movement in the software quality sector in the 1990s and early 2000s, and so it is a bit early to assess the long-term financial performance of companies that have embraced these movements as a governing objective to date. But a useful comparison can be made to the performance of companies that embraced Total Quality Management, a phenomenon that has been with us for some time now.The figure above shows the financial performance of several TQM companies relative to their peers on the Standard & Poors 500 during a full decade in which TQM adoption was at a peak. Among them were several technology leaders, such as Xerox, IBM, and General Motors—all of whom pioneered software systems respected for their high quality (yes, even GM).

Although it would clearly be an exaggeration to say that the financial performance of these companies was disastrous during that period, it is equally clear that the results were not what might have been hoped for, given the undeniable technical superiority of the systems that were produced under their rigorous quality programs. What explanation might exist for the inability of companies who adopted a quality-oriented governing objective to generate financial results as impressive as their technical results?

Some reflection reveals that quality is well-suited as an operational framework, but does not offer an economic framework for strategic decision-making. Some of the most critical decisions that a company is faced with have little to do with its quality program. At the same time that General Motors was embracing TQM, it embarked on a multi-year program of investment in factory automation and robotics, spearheading many software innovations such as the Manufacturing Automation Protocol (MAP). In retrospect, though, this was an ill-advised allocation of precious company resources, which certainly contributed to GM’s under-performance of the market by a full ten percent during that period. It is now generally recognized that at the same time IBM was pursuing TQM in those days, it was paying a heavy premium for its acquisition of Lotus Development Corporation.

In other words: both GM and IBM had great quality programs, but terrible strategic programs, and the bad strategy won out. Now look at how each of those two companies is faring today: IBM is thriving, while GM fights for its very survival. It certainly isn't their quality programs that are making the difference: it is their competitive strategies - one very successful, the other not.

Another, more subtle problem with a “quality strategy” is related to the very fact that programs like ISO 9000 have become so well-accepted: in many markets (for example, aerospace and defense), quality certification has become mandatory for participation – a “union card” for market entry, thereby levelling the field for all players and reducing dramatically the possibilities for building competitive advantage based on quality. Indeed, in these markets, any benefits from quality tend to accrue to customers.

Yet another problem is the “corporate culture” that sometimes arises around quality. A colleague of mine relates that he first began to suspect problems with TQM as a corporate culture when his company worked with Eastman Kodak and observed their operations. G. Newman has described the problem in the following way: “...the fadmongers [of TQM] have converted a pragmatic, economic issue into an ideological, fanatical crusade. The language is revealing. The terms of quality as an economic issue are analysis, cost, benefit, and tradeoff. The terms of quality as a crusade are total, 100 percent quality, and zero defects; they are the absolutes of zealots. This language may have its place in pep talks ... but once it is taken seriously and literally, we are in trouble.” When quality becomes a company-level obsession elaborate (and expensive) bureaucratic infrastructures too often arise, with the inevitable adverse financial consequences.

Quality only adds business value if customers are willing to pay more for higher quality. But in some sectors of the software industry, technical innovation is valued over quality per se. In fact, a December 2005 article in the Wall Street Journal noted observations that a quality management process is thought to actually hinder innovation in many cases. “For stuff you’re already good at, you get better and better,” Michael Tushman, a management professor at Harvard Business School was quoted as saying. “But it can actually get in the way of things that are more exploratory.”

Given these considerations, the practical consequences become evident. Quality is not suitable as a top-down governing strategy, to be followed in any and all cases and contexts. It is up to the business strategist to determine—in his own particular context—whether quality should become a competitive weapon in pursuit of business value.

The Public Value of Monetary Integration

2009-03-18T09:53:00.000+01:00

Recently there have been rumblings again about whether the introduction of the Euro was a good thing (Paul Krugman mentioned it in his column last Monday in the New York Times).

Several years ago, when I was living in Germany, I attended a conference where I found myself talking to a professor and his wife about the recent fall of the Berlin Wall. The recently united Germany was in the middle of a crisis of second thoughts. The former West Germany, in particular, was greatly irritated at the "Ossies" wanting a free ride all the time ("Do you know what they do? They come up to the cashier in a supermarket and refuse to pay, saying 'I suffered all these years, now I deserve this!'"). But the professor and his wife were adamant: reunification was absolutely the right thing to do.

"It will do something priceless: it will keep them from ever going to war. That's worth the entire cost of reunification."

They have a saying in Italian, Conosco i miei polli - I know my chickens. That's the way many people feel about European history: again and again, European countries have gone to war with each other. They could do it again - I know my chickens. Monetary integration is one more step toward making that harder to do in the future.

Put another way: the Public Value of monetary union is huge. Sure, the financial cost may turn out to be enormous, maybe even a net loss. But the political value is enormous, and makes it all worth it.

Public Value

2009-03-15T19:28:00.000+01:00

There’s a lot of help out there for those of us in private enterprise who are wondering about how to measure the value of our IT operations. But what about those in public or non-profit organizations?

You might want to consider the concept of Public Value first articulated by Moore at Harvard in 1995, and then further elaborated by international entities such as the World Bank; European entities such as the IDABC, and private organisations, the Gartner Group in particular. A number of frameworks for measuring the Public Value of IT have been proposed, all of which tend to share three conceptual elements:

Financial and organizational value – this element is closest to the classic techniques of value determination already known in the private sector, such as measures of financial Return on Investment, as well as more qualitatively valued improvements in architecture and organisation.
Political value – this element assesses the value of achieving policy-related goals, such as the degree of implementation of laws and directives related to IT-readiness;
Constituent value – this element captures the value of the improved end user experience, in terms of decreased administrative burden, more inclusive public services, and so forth. This of the Standard Cost Model for reduction of administrative burden and similar initiatives.

From my point of view, the best thing about the concept of Public Value is that it makes a correct separation of the different dimensions of value into something you can work with. I’ll get back to this later.