The Continuing Saga of Thomas Jefferson and the Internet
Talk Delivered at The David Library Lecture Series
on “The Unfinished Constitution” Washington Crossing PA

I want to cover a lot of ground tonight, and I want to make some connections that might be new to you. I want to focus on two parts of our Constitution, one familiar, one not-so-familiar. The familiar one is the provision prohibiting Congress from making any law “abridging the freedom of speech or of the press” – the First Amendment. The not-so-familiar one is the provision granting to Congress the power “to promote the Progress of Science” by “securing to Authors the exclusive Right to their Writings” – the so-called “Copyright Clause” of Article I Sec. 8.

The interplay between these two provisions – one a grant of power to the government, the other a prohibition on government’s use of power – is complicated, fascinating, and even profound.

To begin with, there is, and always must be, tension between them. It’s built in, as it were. Copyright law restricts free expression – indeed, that is the very point of copyright law. That’s its job. Copyright works by giving Authors certain exclusive rights – monopoly rights – to their expression, and it allows them to restrict the speech of others where that speech conflicts with those exclusive rights. I cannot reproduce today’s New York Times and distribute it to my friends or put it on my Facebook page – copyright restricts my freedom to speak. I cannot walk into a bar at Washington Crossing and sing my version of Bob Dylan’s Like a Rolling Stone (though I have a terrific cover version of the song . . .) – copyright restricts my freedom to speak. I cannot take the final scene from the Harry Potter motion picture and insert it into the video I’m making on the occasion of my parents’ 50th anniversary – copyright restricts my freedom to speak. I cannot translate Jonathan Franzen’s Freedom into Italian – copyright restricts my freedom to speak. If I do any of these things (without the permission of the copyright holder), I’m subject to legal sanction. That’s how copyright works – by restricting expression.

In fact, the Copyright Act specifically authorizes the seizure and destruction of books, DVDs, and the like – one of the very few places in our law that does so. US marshals can (and do) take books and throw them into the incinerator. Now, they only do so, mind you, on court order, after due process; I’m not suggesting that we live in some sort of barbaric, book-burning society. Not at all. But the fact remains that our copyright law permits the destruction, in certain circumstances, of books and newspapers and CDs and DVDs and . . . , and there is an obvious tension between such law and the freedom of speech protected by the 1st Amendment.

At the same time, of course, copyright law also encourages speech, and the production and dissemination of expressive communications – music and sculpture and news reporting and movies and all the rest. It is and was intended to be, as the Supreme Court put it recently, one of “the engines of free expression . . . by establishing a marketable right to the use of one’s expression, copyright supplies the economic incentive to create and disseminate ideas.”

Like I said – it’s complicated.
I want to look tonight a little more closely at that tension between copyright and free expression, As I said, it is a tension inherent in the very notion of copyright law; so it’s always been there, ever since we’ve had copyright law – and because one of the very first bills enacted by the very first Congress in 1791 was a Copyright Act, we’ve had copyright for a long time.

It’s only recently, however – I’d say the last 20 years or so – that we’ve begun to look carefully at this tension, and to consider and to worry about its broader implications. This is due largely to the rise of the Internet and related digital technologies, which has moved copyright law from the outer periphery of the legal universe (and the outer periphery of our culture) to the very center of both. We all now have the ability at our fingertips to make millions of copies, at virtually no cost, of pretty much anything we can get onto our computers – songs, movies, software, articles, photographs, etc. – and to distribute those copies to millions of people around the globe – again, at virtually no cost. Pretty much all of that (whether you know it or not) is copyright-protected information; that’s just the way copyright law works these days. Somebody owns the copyright in just about everything you have on your computer, and just about everything you find on the Internet.

The scope and shape of copyright law thus has a very significant and substantial impact on the shape of the Net – on what you can find there, how you can get it, etc. And as the Internet has become a more important feature of our world, so too has copyright law become a more important feature of our legal world. For those of you in my generation, I would venture to bet that copyright law never came up during your breakfast table conversations with your family when you were growing up (unless your parents happened to be in the publishing or the entertainment business); but I bet your kids talk about copyright law – about file-sharing, re-mixing videos, and all the rest – maybe a lot. Maybe they’ve even heard about the Pirate Party – a political party in Sweden that has just gotten enough votes to be represented in the Swedish Parliament, and whose platform, basically, is: No More Copyright.

In pre-Internet analog days, this tension and conflict between copyright and free speech was an interesting but fairly insignificant question; today it has taken center stage.

As it happens, no one had more interesting or influential things to say about both of these subjects – copyright and the freedom of speech – than Thomas Jefferson. [Having just written a book about Jefferson and spent 12 years or so immersed in his work, I find that’s often true; there’s an astonishing range of things – from meteorology to linguistics to gardening to cryptology to paleontology and many others – which turn out, when you start to look closely at them, to have Jefferson’s fingerprints all over them.]
On the one hand, Jefferson was our first great free expression and First Amendment absolutist. Freedom of expression was a central tenet – really, the central tenet – of Jefferson’s creed.
To preserve the freedom of the human mind & freedom of expression and the press, he wrote, every spirit should be ready to devote itself to martyrdom; for as long as we may think as we will, and speak as we think, the condition of mankind will proceed in improvement. . . .
Diffusion of knowledge among the people is the only sure foundation that can be devised for the preservation of freedom and happiness. And then, his now-famous words: Were I faced with a choice between a government without newspapers, and newspapers without government, I would not hesitate for a moment to take the latter.”

The first object of government is to leave open to all the avenues to truth, and freedom of expression and freedom of the press are the most effectual means for doing that. The United States, he wrote, will demonstrate to the world the falsehood that freedom of [speech, and] freedom of the press are incompatible with orderly government.

And it was Jefferson’s election in 1800 that enshrined these principles into our government and our law – at a time when that was by no means foreordained. Adams and the Federalists, you may recall, during John Adams’s first (and, thankfully, only) term as President, enacted the most extraordinary restriction on the freedom of speech the United States had ever seen or ever was to see. The Sedition Act of 1798 made it a federal crime, punishable by 2 years in prison, to criticize the government — to “write or utter or publish,” any “malicious writings against the government of the United States, or either House of Congress, or the President,” or anything that would “bring them into disrepute.” Dozens of U.S. newspaper editors and pamphleteers had been rounded up and tossed in jail under its terms.

The Sedition Act would have destroyed the United States before the United States had even had the chance to really become the United States. The election of 1800 was very much a national referendum on the Sedition Act, and Jefferson prevailed; and his very first act, upon being sworn in as President, was to sign the bill repealing it.

Protecting the freedom of expression was a task of the very highest order, in Jefferson’s view, because freedom of expression was a natural right, belonging to all. It is not given to us by law, nor is it derived from law. It is just in the “nature” of things, part of the way the world is constructed, derived not from the laws of Man but “the laws of Nature and of Nature’s God”: if you bring two human beings together, they will think, and they will attempt to communicate with one another about what they are thinking. They’ll do that without any law to help them. Humans communicate with one another not because the law enables them to do so; they communicate with one another because—well, because that’s the kind of beings we are, and that is what is in our nature. Law’s job is not to enable that communication, but to protect it when it does occur.

Copyright, though, is different. Copyright is what Jefferson called – and I believe he was the first person to point this out and make this distinction — “social law.” Copyright does not derive from the nature of things, from the way the world is, or is constructed, because it is in the nature of things that ideas move freely from one person to another. As he memorably put it in an 1813 letter that has become one of the foundational documents for intellectual property law in the US:
If nature has made any one thing less susceptible than all others of exclusive property, it is the “idea.” That ideas should freely spread from one to another over the globe seems to have been particularly and benevolently designed by nature. Ideas are like the air we breathe – incapable of confinement or of exclusive appropriation, and expansible over all space.

The only way to keep an idea to yourself is to . . . well, to keep it to yourself. The moment it is divulged, it forces itself into the possession of every one, and the receiver cannot dispossess himself of it. Once it gets loose, it is like the air we breathe, expansible over all space, incapable of confinement.

And, like fire, ideas don’t get “used up” as more people use them: The peculiar character of ideas is that no one possesses an idea the less because others possess more; he who receives an idea from me receives instruction himself without lessening mine, just as he who lights his candle at mine receives light without darkening me.

Copyright, in other words, doesn’t come from the laws of nature, it comes from the laws of man. It is not, like freedom of expression, antecedent to the law, but entirely dependent on it.

What difference does all this make? A great deal. It does not mean that we should get rid of copyright law –it’s not an anti-copyright (or pro-copyright) notion. But it does mean that copyright law should always serve free expression, and not vice versa. It means that when these two great forces come into conflict with one another – and as I said, they do with some frequency nowadays – we know where we stand. We have our thumb on the scales on the side of free speech; we need to be vigilant and alert to the circumstances where copyright law is not serving the cause of free expression, where it is interfering with our right to speak and communicate with one another, and we need to adjust it accordingly.

To put it bluntly: When we’re destroying books under the authority of copyright law, we not only need to be sure that they’re the right books; we need to be damned sure.

To get a better sense, perhaps, of what this might mean, and a better sense of the complicated ways in which copyright and free expression are intertwined on the Net, I want to tell a copyright and free expression story.

In the mid-90s, when the Internet was just becoming “the Internet,” this fixture in our lives, many people began to realize that copyright law could strangle the medium before it even got going. As I mentioned, just about everything on the Net is protected by copyright – I’m not talking about bootleg songs or pirated DVDs, I’m talking about everything – every email you send, every blog posting, every picture of your children and grandchildren you post on Facebook, every product review you leave at Amazon.com, every video of stupid pet tricks posted to Youtube. All of it.

Hundreds of millions, probably billions, of such copyright-protected works have been making their way across the Internet since I began this talk.

Copyright means that you need permission from the copyright holder when you “copy” a work that is protected by copyright. Yet the very act of transmitting every one of those files from one place to another on the Internet involves making dozens of “copies,” as the message makes its way from server to server across the Net. If the Internet Service Providers who are moving these files across the Net have to obtain the permission of the copyright holders before they do all this (as it appeared, in the mid-1990s, to be the case) if they’re liable for all those copies they’re making, . . . many people, by the mid 90s, started to realize that we won’t have much of an Internet if that’s the case, because nobody in their right mind would go into that business.

“We understand,” the ISPs said, “that some of the stuff – maybe lots of the stuff – that we’re passing along from one user to another is infringing someone’s copyright. But we’re just standing in the middle, passing things along from user to user. Don’t hold us responsible for that – not if you want to have a robust Internet.”

So in 1998, Congress did a very smart thing. [We criticize them when they screw things up (and we can do so freely, thanks in large part to Jefferson) – we should dispense praise when they get it right]. In 1998 Congress passed a law – the DMCA – giving “providers of online services” an immunity from the claims of copyright infringement based on the actions of their users. We would suspend ordinary copyright law, in other words, weaken it, in order to allow these entities to do their indispensable work in creating this robust free-speech-enhancing place.

This immunity from copyright claims has been an astonishing success. It is in large part responsible for the explosion, over the last 10 years or so, in “social media” and “user-generated content” (or “Web 2.0”) services and applications. Facebook, Myspace, Twitter, Youtube, Craigslist, Tumblr, Blogger, Flickr . . . hundreds of thousands of sites, some of which are household names around the globe, all of which share one common characteristic: they provide no “content” of their own, but rely entirely on their users, who are charged with making the site valuable and engaging and attractive for other users.

Without the DMCA immunity from copyright liability, you wouldn’t get any of them. They couldn’t exist. Why not? Because without an immunity, their potential liability for their users’ infringements, at the scale at which they operate, would be astronomically large. The amount of stuff posted to YouTube every month is greater than the combined output of all US TV networks since their inception – if YouTube (or Facebook, or Craigslist, or . . .) were liable for even a tiny fraction of that, their copyright liability for a single day’s worth of uploaded content would be measured in the hundreds of millions or billions of dollars. Without the immunity from copyright infringement claims, allowing users to communicate and to exchange content freely with one another would be unthinkable; no rational investor would have provided financing for, say, the Facebook business plan without assurances on that score, some protection against the outsized risk.

It was a brilliant stroke – a truly Jeffersonian stroke – adjusting copyright law in the interest of free expression.

[Among other things, it explains why virtually all of UGC/Web 2.0 sites with global reach started in the United States – because United States law gave them this immunity from copyright liability]

And the events of the past year show us just how significant a development this was. Without the DMCA immunity, there’s no Facebook, YouTube, Blogger, or Twitter; and you can make a very strong argument that without Facebook, Youtube, Blogger, or Twitter, Hosni Mubarak is still the President of Egypt. For the first time in a history stretching back over 5000 years, ordinary Egyptians were able to freely communicate with one another, thanks to Facebook, and Twitter, and YouTube, and that has, in fact, changed the world we live in.

A direct line, in other words, connecting US copyright law – and a little, hidden-away provision of US copyright law, at that – and the Arab Spring uprisings. Jefferson – the Jefferson whose motto was
Malo periculosam libertatem quam quietam servitutem.
[I prefer the tumult of liberty to the quiet of servitude.]
would be thrilled, and proud.

Google Engineer Accidently Shares His Internal Memo About Google + Platform

Steve: Who’s Going to Protect Us From Cheap and Mediocre Now?

by Jean-Louis Gassée

Not so fast.
Until the last sinew, the last synapse gives up, Steve will continue to influence the company he co-founded and later recreated. Seeing he could no longer ‘‘meet [his] duties and expectations as Apple’s CEO’’, Jobs kicks himself upstairs and becomes Chairman, director, and “mere” Apple employee. In a distant future, I see him haunting the circular hallways of Apple’s Cupertino spaceship, theCommendatore hunting the clock punchers and damning the linear thinkers straight to Hell.

Let’s review. In 1983, Apple’s Board of Directors felt that Steve required “adult supervision’’. John Sculley, the designated grownup, replaced Jobs as CEO and eventually pushed him out of the company.

Fast forward a decade and a half. In 1997, Steve returns to run his company unchallenged…but not unassisted. The Apple 2.0 management team, hand-picked, well-groomed, isn’t so much a stroke of genius as it is an emblem of the enfant terrible all grown up. As the Fortune chart below shows, Apple has no lack of ‘‘bench strength’’– and who’s providing the adult supervision now?

With Steve as Chairman, Tim Cook, Apple’s long-time COO, moves to the center of the chart.He joined the company 13 years ago, has always reported directly to Steve and saw his responsibilities increase over time. He now drives the team that made Apple the most valued and valuable high-tech company in the world.
As for ourselves: No whining. It’s our job, as consumers, to protect ourselves, to vote with our wallets against the bean counters, the Paint by Numbers product planners. It’s our place to provide ‘‘constructive feedback’’ when Apple products fail to meet the combined aesthetic and functional standards Dear Leader drilled into the marketplace. From MobileMe to “skeuomorphic” calendars, address books and bookshelves — to say nothing of fresh Lion bugs. Steve’s Apple may not be perfect, but…

A portentous example: The 1998 Bondi Blue iMac, the first visible re-assertion of Steve’s style — and ofJony Ive’s portfolio in the making:

Immediately iconic, users adored their iMacs. The unexpected shape and color set a new standard for high-tech products, so much so Apple competitors tried to rub the amulet for luck — and showed us what they really stood for: Cheap, imitative mediocrity. I recall going to Palo Alto’s Fry’s store and seeing beige PC clone boxes with candy-colored plastic inserts that approximated the iMac palette.

As a Forbes article put it, speaking of Dell’s similar fig-leaf attempt:

“Dell, ever concerned with keeping its inventory low, seems to be approaching colored notebooks in a much less risky way, using cheaper plastic inserts. Of course, the appearance of the Inspiron doesn’t inspire the way the first iMacs and iBooks did.”

The aesthetic knockoffs weren’t just cheap, they were ugly. The inserts looked even worse than the faux-wood ‘‘accents’’ on Chrysler dashboards. No cojones, no imagination, no taste.

Fast forward a bit more: Steve introduces the Apple Store. We’ll pass over the record-beating numbers and address the two messages the store imparts.
First, the architecture, an expression of the Apple ethos, says: ‘This is what we think of ourselves’.
Second, once inside the store, the experience states: ‘Here’s what we think of our relationship with you, our customer’.
In comparison, I see carriers trying to spruce up their store fronts with shiny metal appliqués — but go inside and you find cheap trade-show modular furniture.

Taste matters. Let’s turn to this YouTube video of the opening of an Apple Store clone. Not a Chinese counterfeit but a Microsoft Store in Scottsdale, Arizona. It starts much like the “real” thing: Happy customer, rows of high-fiving employees, a decor that looks familiar.  But 40 seconds into the one minute video, we get the “tell”, the killer detail that gives the imitation away.  Here we get the men in suits and ties:

Still more evidence of Steve’s influence: Just as HP decides to spin off its PC business (or perhaps not), PC clone makers demand an additional $100 subsidy per ‘‘ultra-portable’’ laptop from Intel. Why? They want to compete with Apple’s increasingly popular MacBook Air. It seems that the “Apple tax”, the premium we’re willing to pay for quality, isn’t enough to dissuade us.

PC clone makers can’t match Apple’s cost or its Bill Of Materials (BOM). The way Apple procures parts and subsystems, the way it runs contract manufacturing and stays on top of complicated but delicate distribution logistics is evidence of the company’s aggressive Supply-Chain Management (SCM). Steve – and thus Apple – understands that the channels need to be fed Just So, neither starved nor stuffed.

I found the BOM story interesting and looked up current ultra-portable prices. Who better than Sony in that product category? I went to their site and got this:

A nice MacBook Air competitor starting at $1969. The real thing starts at $1299.
Quite a reversal of the old world order and, I hope, a source of satisfaction for Jobs.

Spanning an amazing arc of thirty years, the company with the anti-establishment image has become the most disciplined, best-managed high-tech giant — and arbiter of taste.

When I first met Steve, in February 1981, he was sitting cross-legged on a credenza in the Apple board room, picking his toes. Since then I’ve watched with glee as he went against received wisdom, causing pundits to have fits at every turn. I picture them as a gaggle of eunuchs standing around the caliph’s bed, braying in high-pitched voice: ‘Steve, you’re doing it wrong!’

For a long time, I’ve seen him as having an animal inside him, the one with the desires, the instinct, the drive. In 1985, that animal threw Steve to the ground. He picked himself up at Pixar — you’d be a captain of industry for doing no more — and NeXT. Then, in 1997, armed with Pixar’s success and Next’s technical prowess, he came back to run Apple and make it really his.

He had learned to ride the animal.

Steve and Tim both speak, rightly, of Apple being at the crossroads of technology and humanities, liberal arts. In tribute to Jobs’ aesthetic sense, and why it deeply matters, I’ll conclude with a quote from Herman Hesse’s Steppenwolf:

‘’Before all else, I learned all these playthings were not mere idle trifles invented by manufacturers and dealers for the purposes of gain.  They were, on the contrary, a little or, rather, a big world, authoritative and beautiful, many sided, containing a multiplicity of things all of which had the one and only aim of serving love, refining the senses, giving life to the dead world around us, endowing it in a magical way with new instruments of love, from powder and scent to the dancing show, from ring to cigarette case, from waist buckle to handbag.  This bag was no bag, this purse no purse, flowers no flowers, the fan no fan.  All were the plastic material of love, of magic and delight.  Each was a messenger, a smuggler, a weapon, a battle cry.’’

— JLG@mondaynote.com

Is Facebook unethical, clueless or unlucky?

Location: CalaCompound, Brentwood, CA
Date/Time: December, 13th 2009 11:20AM

===============================

Facebook proved again this week that they are either the most
unethical or clueless internet company in the world. An amazing
accomplishment since Facebook is also one of the most promising, and
certainly fastest growing, internet companies of all time.  Perhaps
I’m being hyperbolic (who me?), or maybe they are a little of both,
but the fact remains they screw up on important issues almost as if
it’s a “best practice” to do so.

In case you missed it, when you logged into Facebook this week you
were road blocked with a popup explaining that they “we’re making some
changes to give you more control.” Sounds good, and like most users
looking to quickly get into a website or application, I simply clicked
through the message. How important could it be?

When faced with a TOS (Terms of Service) or license the world has been
trained to hit the word “agree,” and click, click, click until they
get to the actual website or software they were trying to get to in
the first place.

Everyone in the industry knows this, and certainly a company built off
of studying social behavior like Facebook would. Since the ToS is
considered a formality, it is up to technology companies–in fact our
industry–to behave. If we don’t behave well then we are going to get
regulated by clueless politicians and policy makers. That would suck
for everyone.

So What Happens When you Clickthrough?
===================
In this case, if you simply click through the windows you’ve exposed
all of your private Facebook information, including comments, friends,
pictures and status updates, to “everyone.” In other words clicking
through changes everything in Facebook terms–unlike every other
license or update screen you’ve experienced in your life.

I’m sorry, what the frack just happened? I turned over my friend list,
photos and status updates to everyone in the world? Why on earth would
anyone do that with their Facebook page?

The entire purpose of Facebook since inception has been to share your
information with a small group of people in your private network.
Everyone knows that and everyone expects that. In fact, Facebook’s
success is largely based on the face that people feel save putting
their private information on Facebook.

When you do get to the second page a series of confusing radio buttons
default–yes defaults–to giving everyone access to your social graph.
Wow. I’ve been using the internet since before images were supported.
I’ve been a member of every social network since Six Degrees and Ryze,
almost a decade before Facebook became available to the public, and I
was confused by their settings page. An average user, certainly, has
no idea what is going on by these changes.

So why is Facebook trying to trick their users?

Simple: search results.

Facebook is trying to dupe hundreds of millions of users they’ve spent
years attracting into exposing their data for Facebook’s personal
gain: pageviews. Yes, Facebook is tricking us into exposing all our
items so that those personal items get indexed in search
engines–including Facebook’s–in order to drive more traffic to
Facebook.

So why is this wrong?
==================
While there is nothing wrong with having a service that is “public by
default,” it is highly unethical to flip your users over to public in
a such a deceitful way

Twitter is, of course, public by default, and we all know that
Facebook is obsessed with Twitter innovations including their short
status updates, their API and most of all, their “open by default”
strategy.

Facebook has had a couple of innovations in their history, like their
application layer and news feed (which is now gone), but for the past
couple of years they’ve given up on innovation and focused on stealing
ideas from Twitter and out-executing them, while not caring about user
rights. This is challenging for Twitter, which is run by the highly
ethical Evan Williams and Biz Stone. In fact, those two guys are
massively conservative when it comes to their user base.

Facebook continues their non-stop copying of Twitter, and even after
the absurdly stupid “Facebook Beacon” debacle, they continue to try
and sneak unethical behavior past the masses–and the industry.

The result? They’re winning and winning big!

It is so depressing when one of our leading companies bases their
ethics on “will we get caught?” and perhaps more precisely: “if we do
get caught will it cost us anything in relation to the money we’ll
make when we go public?”

The Issue Facebook is creating for all Internet companies
===============================
Another problem Facebook is creating with their reckless behavior is
that they are simultaneously making users distrust the internet and
bringing the attention of regulators.

As an industry we should police ourselves and do everything we can to
create trust with users.

It would be great if the “adults” sitting around Zuckerberg’s cube
would explain to the Golden Child that just because he’s on the Forbes
billionaires list and he generates a mob of sycophants around him at
the TED conference, that doesn’t mean he gets a free pass to bring the
heat down on all of us.

Behave yourself dude!

How would you do it better?
====================
If Facebook was more concerned with ethics than world domination, they
would simply post a popup that said something like:

“Dear Facebook Members,

Good news, we’ve now added the option to share your content with
everyone! Be sure to check out this new feature here and be sure to
consider if you want to expose your content to the world before
changing your settings!”

Of course, that would result in 1% of users turning their service to
“everyone” (i.e. public) a month. It would take years to convert a
meaningful amount of users and their personal data into revenue
generating public objects. With Facebook’s IPO–the one that will save
Silicon Valley–around the corner, there is simply nothing we can do.

Facebook’s IPO and revenue growth trumps user’s rights, right?

Growth at all costs!

Long live the Golden Child!

Ticker: FCBK FTW!

Can I still get a friends and family allotment?

#fail

====================

Questions (hit reply, or post to your blog):

1. Is Facebook clueless, unethical or just unlucky? Why?
2. Will Facebook’s latest behavior result in more lawsuits and/or
industry regulation?
3. Do you trust Facebook with your information?

all the best,

Jason

Why Craigslist Is Such a Mess

Craigslist founder Craig Newmark. 
Photo: PLATON

The Internet’s great promise is to make the world’s information universally accessible and useful. So how come when you arrive at the most popular dating site in the US you find a stream of anonymous come-ons intermixed with insults, ads for prostitutes, naked pictures, and obvious scams? In a design straight from the earliest days of the Web, miscellaneous posts compete for attention on page after page of blue links, undifferentiated by tags or ratings or even usernames. Millions of people apparently believe that love awaits here, but it is well hidden. Is this really the best we can do?

Odd perhaps, but no odder than what you see at the most popular job-search site: another wasteland of hypertext links, one line after another, without recommendations or networking features or even protection against duplicate postings. Subject to a highly unpredictable filtering system that produces daily outrage among people whose help-wanted ads have been removed without explanation, this site not only beats its competitors—Monster, CareerBuilder, Yahoo’s HotJobs—but garners more traffic than all of them combined. Are our standards really so low?

But if you really want to see a mess, go visit the nation’s greatest apartment-hunting site, the first likely choice of anybody searching for a rental or a roommate. On this site, contrary to every principle of usability and common sense, you can’t easily browse pictures of the apartments for rent. Customer support? Visit the help desk if you enjoy being insulted. How much market share does this housing site have? In many cities, a huge percentage. It isn’t worth trying to compare its traffic to competitors’, because at this scale there are no competitors.

Each of these sites, of course, is merely one of the many sections of craigslist, which dominates the market in facilitating face-to-face transactions, whether people are connecting to buy and sell, give something away, rent an apartment, or have some sex. With more than 47 million unique users every month in the US alone—nearly a fifth of the nation’s adult population—it is the most important community site going and yet the most underdeveloped. Think of any Web feature that has become popular in the past 10 years: Chances are craigslist has considered it and rejected it. If you try to build a third-party application designed to make craigslist work better, the management will almost certainly throw up technical roadblocks to shut you down.

Craigslist is not only gigantic in scale and totally resistant to business cooperation, it is also mostly free. The only things that cost money to post on the site are job ads in some cities ($25 to $75), apartment listings by brokers in New York ($10), and—in a special case born of recent legal trouble—advertisements in categories commonly used by prostitutes, because authorities encourage vendors to maintain a record that would aid investigators. There is no banner advertising. They won’t let you join them, and at this price you can’t beat them either.

At times it has occurred to people that the problems with craigslist could be solved by appealing to its eponym, Craig Newmark. Newmark is under lots of pressure these days. His company is being sued by eBay, a competitor and minority shareholder angry at being excluded from the company’s deliberations. The attorney general of South Carolina has blustered about prosecuting his CEO for facilitating prostitution, and there have been strong challenges from law enforcement agencies in other states, too. The tabloids have relentlessly played up stories about two so-called craigslist killers, one who allegedly used the site’s erotic-services section to lure victims and another who used the help-wanted ads. Newmark responds to such criticism with extreme serenity. Inquire about his finances and he talks about his hummingbird feeder. When his Twitter page asks him, “What are you doing?” he retweets in the voice of a squirrel.

“Run, run, run,” he says. “Dig, dig.”

Though the company is privately held and does not respond to questions about its finances, it is evident that craigslist earns stupendous amounts of cash. One recent report, from a consulting firm that counted the paid ads, estimates that revenue could top $100 million in 2009. Should craigslist ever be sold, the price likely would run into the billions. Newmark, by these lights, is a very rich man. When anybody reminds him of this, the craigslist founder says there is nothing he would care to do with that much money, should it ever come into his hands. He already has a parking space, a hummingbird feeder, a small home with a view, and a shower with strong water pressure. What else is he supposed to want? Frustration over these sorts of replies sometimes becomes comical. In a July 2007 television interview, Charlie Rose spent half the program attempting to get Newmark to admit his good fortune, and failing. “I don’t have anywhere near as much control as you think,” Newmark said.

“I’m not talking how much control; I’m talking percentage of ownership,” Rose said. Rose is usually kind to his guests, but the scent of unacknowledged wealth brought out his ferocity.

“Oh, same thing from my point of view,” Newmark said, trying to move the topic along.

“Do you own more than 50 percent of craigslist or not?” Rose asked.

“No.”

“You don’t?”

“Correct.”

“In other words, other people own that, or you’ve given it away or whatever.”

“Could be, Charlie.”

“OK, but I’m—why are you so …?”

“Coy?”

“Yeah.”

“It doesn’t matter,” Newmark said. “I mean …”

“I know it doesn’t matter,” Rose repeated, his face a mask of pain.

Newmark’s claim of almost total disinterest in wealth dovetails with the way craigslist does business. Besides offering nearly all of its features for free, it scorns advertising, refuses investment, ignores design, and does not innovate. Ordinarily, a company that showed such complete disdain for the normal rules of business would be vulnerable to competition, but craigslist has no serious rivals. The glory of the site is its size and its price. But seen from another angle, craigslist is one of the strangest monopolies in history, where customers are locked in by fees set at zero and where the ambiance of neglect is not a way to extract more profit but the expression of a worldview.

The axioms of this worldview are easy to state. “People are good and trustworthy and generally just concerned with getting through the day,” Newmark says. If most people are good and their needs are simple, all you have to do to serve them well is build a minimal infrastructure allowing them to get together and work things out for themselves. Any additional features are almost certainly superfluous and could even be damaging.

Newmark has been working hard to extend the influence of his worldview. His public pronouncements have the delighted yet apologetic tone of a man who has stumbled on a secret hiding in plain sight and who finds it embarrassingly necessary to point out something that should long have been obvious. He seems to have discovered a new way to run a business. He suspects that it may be the right way to run the world.

Public spirited and mild-mannered, politically liberal and socially awkward, Newmark has one trait that mattered a lot in craigslist’s success: He is willing to perform the same task again and again. During the company’s first years, Newmark approved nearly every message on the list, and in the decade since he has spent much of his time eliminating offensive ones. Even by the most conservative accounting, he has passed judgment on tens of thousands of classified ads. Very few people could do this and thrive.

Newmark knows that he is not typical. He tends to interpret things literally, and when he was younger other people often confused him. In 1972, while still a college student, he read Language in Thought and Action, the classic book on communication by S. I. Hayakawa, and it helped him understand himself better. “All of a sudden I’m thinking, ‘It can’t be that everyone else has a problem. It has to be me,’” he says.

We are sitting in a San Francisco coffee shop called Reverie Café Bar, where Newmark spends long hours and has given countless interviews. Many things in his life are a matter of routine. When he talks, he calls upon a repertoire of conversational gambits he has been collecting forever, and he has a selection of sound effects on his mobile phone, such as a cymbal crash, that he can trigger to make it clear he is joking. When people misunderstand him, he doesn’t get upset. “I’m the Forrest Gump of the Internet,” he says. He loves customer service. “I’ll only be doing this as long as I live,” he says. He taps his phone, triggering a ghostly whaaahahaha. “And after that, who knows?”

Email has always been an ideal outlet for Newmark’s genial nature. Craigslist began in 1995 as a mailing list with announcements of events of interest to technical people, and as more of them began to subscribe, he encouraged readers to post their own news, archived the messages on a Web page, and tried to make sure all the content was legitimate. After Netscape’s IPO in August of that year, craigslist became a portal into the dotcom scene. Within two years, he had thousands of readers, most of whom he didn’t know. This was a big responsibility for somebody who is not an extrovert. “I used to email him every day,” says Christina Murphy, one of the first tech recruiters to use craigslist regularly. “If I made a mistake in a job posting, I would have to call him and ask for a change. It drove him insane.” Murphy, along with an Internet consultant named Nancy Melone, began meeting with Newmark, trying to map out a more professional future for craigslist that didn’t require its founder to take phone calls. Job postings were an obvious source of revenue, and in 1998 they launched a nonprofit called List Foundation. Recruiters would pay $30 for ads, everything else would be free, and any money left after paying the cost of upkeep and administration would be given away. Melone was CEO. Newmark’s willingness to cede so much control worried Murphy, who soon quit the venture. “It was a beautiful, perfect little world,” she says. “And it was being taken over by other forces.”

For nearly a year, the site was available at two URLs, craigslist.org and the less embarrassingly personal listfoundation.org. But Melone and Newmark were pulling in different directions, or rather, Melone was pulling and Newmark was digging in his heels. By the end of the decade, the Internet frenzy was at its peak and the smartest minds of the new industry all agreed that the right strategy was to get big fast in anticipation of a sale or an IPO. Melone wanted to raise prices. Newmark worried about charging for listings at all. Melone wanted to become a dotcom; Newmark was wedded to the idea that craigslist was a community service. Melone was gregarious, a talker. Newmark had vast powers of passive resistance. A split was inevitable, and one day in late September 1999, craigslist users who came in through thelistfoundation.org address found themselves automatically bounced to a new, for-profit Web site, called MetroVox. Run by Melone, it offered similar sorts of community listings and had a far more aggressive plan to grow. Melone said that Newmark had authorized the switch; Newmark announced that he’d been blindsided.

This was craigslist’s first serious competitive challenge, and perhaps its last. Newmark had some personal qualities that ought to have been fatal in an entrepreneur. Aside from his communication problems and an aversion to exerting authority, he cared nothing for entrepreneurship. But in the battle with MetroVox he had an asset that more than compensated for these shortcomings: For years he had worked on his site with an uncanny, machine-like constancy, doing all the painstaking and repetitive things that would make most people desperate with frustration and boredom, and he had done them happily. And now his users paid him back in the most obvious possible way: They stopped using the List Foundation address, resumed posting their free ads at craigslist.org, and emailed Newmark when problems occurred. Less than a year later, the bubble burst and MetroVox faded away.

Newmark abandoned the idea of running craigslist as a nonprofit, which would have required him to learn and follow too many rules. He realized that nobody could stop him from giving away his money if he made too much of it, and in the meantime he handed out a significant portion of his ownership to others as a way to avoid acquiring too much authority. “I was worried about going middle-aged crazy,” he says. He also put great distance between himself and any executive responsibility. The current CEO, Jim Buckmaster, joined the site in 2000 as a programmer and handles every business and strategic issue. It was Buckmaster who crafted the current strategy for growth—a slow, bloblike, seemingly unstoppable accretion of new craigslist cities, each an exact clone of the others, launched with no marketing or publicity. Sometimes a new site grows very slowly for a long time. But eventually listings hit a certain volume, after which the site becomes so familiar and essential that it is more or less taken for granted by everybody except the distressed publishers of local newspapers. Revenue from newspaper classified ads is off nearly 50 percent in the past decade, a drop that comes to almost $10 billion. Only a fraction of this loss is because of Newmark’s company, but as the largest online classified site, craigslist is easy to blame.

Because he is the founder of a remarkable Internet company that also happens to be helping the nation’s dailies go out of business, Newmark’s opinion is eagerly sought, and he spends an increasing amount of time at conferences and international meetings, where he attempts to answer questions about how to best defend the public interest in an era of cheap and ubiquitous media. As we watch the birds on the patio of Reverie, Newmark tries out some of the phrases he is hoping to use in the coming months as he unfolds the lessons of craigslist. “My big mission is to help make grassroots democracy as much a part of our government as representative democracy,” he says.

Jim Buckmaster was hired as a programmer in 2000. A year later he became CEO.
Platon, grooming by Tamara Brown/Artist Untied

Many people who have heard Newmark’s public remarks find the ideals admirable but difficult to apply. What would such an approach mean in practice? His cause is not helped by the fact that if the craigslist management style resembles any political system, it is not democracy but rather a low-key popular dictatorship. Its inner workings are obscure, it publishes no account of its income or expenses, it has no obligation to respond to criticism, and all authority rests in the hands of a single man. Ask Newmark about any feature you would like to see on craigslist and you will always get the same response.

“Ask Jim,” he says.

“How do you get your feedback? Have you ever done a poll or anything like that?”

“The thought makes me tired. But you can suggest that to Jim if you wish.”

“What if Jim says no?”

“If you want to ask him again, you can,” he says.

At this point in our conversation I begin to feel the spirit of Charlie Rose upon me. After all, Newmark is the founder, a major shareholder, and the public face of the company.

“What would it take to get you to fire Jim?” I ask.

Newmark matches me mischief for mischief.

“Ask Jim.”

It is easy to find hypocrisy in the idealism of a business owner who prescribes democracy for others while relieving himself of the tiresome burden of democratic consultation in the domain where he has the most power. But of course, craigslist is not a polity; it is just an online classified advertising site, one that manages to serve some basic human needs with startling efficiency. It is difficult to overstate the scale of this accomplishment. Craigslist gets more traffic than either eBay or Amazon .com. eBay has more than 16,000 employees. Amazon has more than 20,000. Craigslist has 30. Craigslist may have little to teach us about how to make decisions, but that’s not the aspect of democracy that concerns Newmark most. He cares about the details, about executing all the little obvious things we’d like government to do. “I’m not interested in politics, I’m interested in governance,” he says. “Customer service is public service.”

Last year Newmark got about 195,000 email messages. He estimates that roughly 60 percent were spam. He read all the rest and replied to many. He has a boss now, a customer service manager named Clint Powell, who was hired about six years ago. But he maintains his habits for reasons that have little to do with the normal logic of work. They are part of his identity, an unconventional mode of self-realization through which he took hold of a barrier that always separated him from the world and made it into a kind of performance. Athletes compete. Artists create. Newmark answers email. He knows that this will seem absurd from the outside, but he is blessed not to care. In fact, he likes to treat people to a laugh when he can. It’s sometimes impossible to discern his intention exactly, and this is essential to the effect. On our way out of the cafè, I step aside to let Newmark go ahead, and he walks face-first into the plate glass door.

Jim Buckmaster is tall and thin, Newmark is short and round, and when they stand together they look like a binary number. In 2004, I saw them give a talk in which Newmark, who is 5’7″, stood on a milk crate and was still barely eye-to-eye with his CEO, who is 6’7″. It was a memorable performance, but they don’t have much opportunity for the gag these days because their joint appearances are rare. At the craigslist office, the two men work in the same room, but their desks are set up so they sit back-to-back. They are not social friends, and in fact they almost never talk. Newmark does not excel at chitchat, and Buckmaster is a quiet type, too.

Buckmaster dropped out of medical school at the University of Michigan in 1986. He hung around the university for 10 years, studying the classics, doing data entry work, and teaching himself programming. By 1999, he was working as a webmaster in San Francisco for a dotcom called Creditland, where he was not happy. “The marketing side had attained ascendancy,” he says. He posted his résumé on craigslist, and Newmark found it.

Craigslist was very unlike Creditland. “It wasn’t even really clear who decided to hire me,” Buckmaster says. He looked around and began finding things to do. He wrote forum software to give users a chance to interact. When he realized that every post had to be reviewed and published by hand, he created the automated process that allowed craigslist to grow. He coded a search engine. A year after he arrived he was CEO. There was no competition for the job, no ritual transfer of power, and no instructions. “In the entire time I’ve been here, I don’t think Craig has ever said to me, ‘This is the way it has to be,’” Buckmaster says. The only topic he can remember their disagreeing about is the peace sign that adorns the craigslist Web address. “Craig thought it was associated with the hippies and that hippies were discredited,” Buckmaster says. “Whereas I think peace is among the most desirable things you can have.”

The long-running tech-industry war between engineers and marketers has been ended at craigslist by the simple expedient of having no marketers. Only programmers, customer service reps, and accounting staff work at craigslist. There is no business development, no human resources, no sales. As a result, there are no meetings. The staff communicates by email and IM. This is a nice environment for employees of a certain temperament. “Not that we’re a Shangri-La or anything,” Buckmaster says, “but no technical people have ever left the company of their own accord.”

The purity of this culture is its most tenaciously guarded asset. A few years ago, Phillip Knowlton, a Bay Area psychologist who was on the craigslist staff in the site’s early years, sold his 28 percent stake in the company to eBay. Buckmaster and Newmark approved eBay founder Pierre Omidyar, himself a programmer, as the representative of eBay on the craigslist board. But at that point, Omidyar no longer ran eBay, and he was replaced by an eBay vice president who had overseen the acquisition of a craigslist competitor in Europe. When eBay launched a competing service in the US, Buckmaster responded by reorganizing craigslist and weakening eBay’s influence. The companies have since sued each other. While the dueling complaints hinge on questions of stock dilution and conflict of interest, it is hard to imagine any conventional business executive being satisfied with the way craigslist operates. What kind of company declares itself uninterested in maximizing profit? “Companies looking to maximize revenue need to throw as many revenue-generating opportunities at users as they will tolerate,” Buckmaster says. “We have absolutely no interest in doing that, which I think has been instrumental to the success of craigslist.”

Buckmaster and I talk in the San Francisco penthouse condo of Susan MacTavish Best, who owns a small PR company. Best and Buckmaster lived together as a couple for five years. Though they are now separated, they remain friends, and she continues to serve as a kind of translation mechanism by which the hints and silences of craigslist management are converted into responses suitable for the press. Queries, in recent months, have concerned mostly sex and violence. That the world would expect craigslist to take responsibility for the rare violent criminal who lures victims through an ad strikes Buckmaster as absurd. He points to the thousands of people who die every year in auto accidents. “Does anybody call up the head of GM and say, ‘Somebody just got killed using your product? How can you sleep at night? Don’t you realize that a person is dead?’”

Buckmaster’s dispassionate protest reflects his cast of mind. Emotional appeals are more likely to provoke his skepticism than his sympathy, and when the complaints come from aspiring Internet entrepreneurs he is especially prone to sarcasm. He hears many such complaints, because one of the most curious things about craigslist is that a company designed and run entirely by programmers is so hostile to outsiders who want to pull neat technical tricks to improve the site. A few years ago, independent programmer Jeff Atwood created a service that would allow people to search multiple cities at once or even search craigslist globally. Buckmaster arranged some technical interference to kill it off. Another programmer named Ryan Sit created a service called Listpic, which scraped images from craigslist and dumped them into an interface for browsing: You could scan through all the photos from the apartment listings or see pictures of all the dogs up for adoption. Buckmaster banished Listpic, too.

He had specific objections to both. Listpic ran ads, it put a high burden on craigslist servers, and when he looked at traffic records he noticed that Listpic was being used mainly to enhance enjoyment of the sexy images people posted in their erotic-services ads. Universal search subverts craigslist’s mission to enable local, face-to-face transactions; it increases the risk of scams and can be exploited to snatch up bargains, giving technically sophisticated users an advantage over casual browsers. But the very surfeit of these practical objections—many of which probably have technical solutions—hints that the real explanation lies elsewhere, and with a minimum of pressure Buckmaster will state it plainly. It is the same reason that craigslist has never done any of the things that would win approval among Web entrepreneurs, the same reason he has never updated its 1999-era Web design. The reason is that craigslist’s users are not asking for such changes.

“I hear this all the time,” Buckmaster says. “You guys are so primitive, you are like cavemen. Don’t you have any sense of aesthetic? But the people I hear it from are invariably working for firms that want the job of redoing the site. In all the complaints and requests we get from users, this is never one of them. Time spent on the site, the number of people who post—we’re the leader. It could be we’re doing one or two things right.”

This ends the debate for him, but his tone is oddly non-triumphal; in fact, Buckmaster’s statement of fealty to users has a weary sound that I don’t understand until weeks later. Only after I have spent every spare hour on craigslist—browsing the ads, tracking the spam, reading the help forums, contacting users—do I finally begin to grasp something of his situation. The truth is that a lot of people complain about craigslist. Buckmaster is correct that few of them complain about the design. They complain about spam, they complain about fraud, they complain about the posting rules, they complain about the search, they complain about uploading images. They complain about every way a classified transaction can go wrong. They seldom complain about amazing new features they imagine they might possibly want to use, because they are too busy complaining about the simple features they depend on that don’t work as well as they’d like. By eliminating marketing, sales, and business development, craigslist’s programmers have cut out all the cushioning layers that separate them from the users they serve, and any right they have to teach lessons in public service comes from the odd situation of running a company that is directly subservient only to the public. Here’s the lesson: The public is a motherfucker.

Craig Newmark says that craigslist works because people are good, and he has stuck to this point of view without wavering. Whether you accept it as true will depend on your standard of goodness.

Sometimes entire categories of craigslist are rendered nearly unusable by spam. Con artists prowl the listings, paying sellers with fake cashier’s checks and luring buyers to share their credit card numbers. Other evils are more subtle. Business owners whose judgment is distorted by self-interest fail to understand the rules and put the same item in multiple categories or repost it many times a day to insure it stays prominent, crowding out other sellers. A woman listing a car forgets to tell buyers about problems with the title until they’ve made a long trip out to see it. In all transactions there is a possibility of misunderstanding as well as abuse, and at 99.99 percent perfection there would still be thousands of angry people every month.

Newmark says that craigslist works because people are good.

The battle flows back and forth. Captchas—distorted words that can be interpreted by humans more easily than by machines—tamed spam on craigslist for a while. Then it came back full force, not because the spammers had solved the difficult problem in artificial intelligence but because they had hacked an easier problem in global economics. I recently established a friendly email dialog with a young man in Dhaka, Bangladesh, who works on a 13-person team that creates craigslist spam. He fills in Captchas, creates new accounts with masked IP addresses, and posts ads all day long using text from a database provided by his employer, an anonymous spam king. The going price for a spam post on craigslist is about 50 cents, with large discounts for volume. When I told Buckmaster about my new friend, he took the news calmly. “These are technically sophisticated people who take pride in their work, and when we knock them down they don’t just decide to go find something else to do. You could say we are breeding the perfect spammer.”

Without a computer science research department to work on evil-fighting algorithms, or a call center to take complaints, Buckmaster has settled on a different approach, one that involves haiku. The little poems he has written appear on the screen at times when users might expect a helpful message from the staff. They function as a gnomic clue that what you are seeing is intentional, while discouraging further conversation or inquiry. For instance, start too many conversations in the forums and your new threads may fail to show up. Instead, you will see this:

frogs croak and gulls cry
silently a river floods
a red leaf floats by

Attempt to post a message that is similar to one you’ve already entered, and this may appear:

a wafer thin mint
that’s been sent before it seems
one is enough, thanks

The slight delays in cognitive processing that these haiku cause are valuable. They open a space for reflection, during which you can rethink your need for service. But haiku can’t solve everything. Supporting the poems are tens of thousands of users who are willing to devote two or three seconds of time to flag inappropriate ads or forum posts. Too many flags on an advertisement and it will vanish. The staff can lower the number of flags required to vaporize an ad if they want to clean out an especially polluted category, and they can raise the threshold if people grow flag-happy. Users whose listings are flagged off the site get no hint as to what they may have done to attract ire. Instead, they are directed to the “flag help” forum, where pseudonymous volunteers will offer an educated guess while having some fun at their expense. Last spring a baffled user posted a query about why her ferrets-for-sale ad disappeared. Within 60 seconds there was this reply: “Train the ferrets to read the terms of use. Maybe they can help you out next time. Pet sales are prohibited on this site.”

An ad can be flagged off the site for any reason. Reject too many people for a job opening and they may flag your ad in spite every time they see it—and every new ad you post, too. Describe yourself as incredibly handsome and cynical date-seekers may flag you as a favor to the innocent. The claim that craigslist, used by millions of strangers, is somehow a democracy begins to be believable exactly here, in the crotchets, irritations, prejudices, and minor forms of harassment that characterize life in a small town where any proposal you make is subject to the judgment of everybody.

Flag something as inappropriate in the discussion forums, where craigslist employees have the final word about what goes, and these lines appear.

thanks for flagging this
staff will look at it shortly
hey, a dragonfly!

Buckmaster’s sly haiku evokes an entire scene. Somewhere, at this moment, an innocent party is staring at a computer screen, furious at an offensive remark. Somebody else is fruitlessly trading insults with volunteers on the help desk. A third person is checking the site again and again, looking for a listing that was submitted but never appeared. All craigslist can offer at these moments is a shrug and a joke, in the style of a Dilbert cartoon.

This is old-fashioned. But craigslist is old-fashioned in any number of ways. It relies on email and the telephone in an era of SMS and social networks. It sticks to traceless transactions in an industry that makes its living collecting data from every touch. And just as people who run technical companies are reaching an apex of confidence in their ability to invent new forms of community based on sharing everything, craigslist still treats social life as dangerously complex, deserving the most jaded caution. Corporate isolation, user anonymity, refusal of excessive profit, glacial adoption of new features: These all signal Newmark and Buckmaster’s wariness about what humans, including themselves, might do if given the chance. There may be a peace sign on every page, but the implicit political philosophy of craigslist has a deeply conservative, even a tragic cast. Every day the choristers of the social web chirp their advice about openness and trust; craigslist follows none of it, and every day it grows.

Contributing editor Gary Wolf (gary@aether.com) wrote about tracking personal data in issue 17.07

Google’s chief economist says executives in wired organizations need a sharper understanding of how technology empowers innovation.

More than ten years into the widespread business adoption of the Web, some managers still fail to grasp the economic implications of cheap and ubiquitous information on and about their business. Hal Varian, professor of information sciences, business, and economics at the University of California at Berkeley, says it’s imperative for managers to gain a keener understanding of the potential for technology to reconfigure their industries. Varian, currently serving as Google’s chief economist, compares the current period to previous times of industrialization when new technologies combined to create ever more complex and valuable systems—and thus reshaped the economy.

Varian spoke with McKinsey’s James Manyika, a director in the San Francisco office, in Napa, California, in October 2008. Watch the video or read the transcript of his comments below.

Hal Varian on how the Web challenges managers

Google’s chief economist on how technology empowers innovation.

On flexible innovation

We’re in the middle of a period that I refer to as a period of “combinatorial innovation.” So if you look historically, you’ll find periods in history where there would be the availability of a different component parts that innovators could combine or recombine to create new inventions. In the 1800s, it was interchangeable parts. In 1920, it was electronics. In the 1970s, it was integrated circuits.

Now what we see is a period where you have Internet components, where you have software, protocols, languages, and capabilities to combine these component parts in ways that create totally new innovations. The great thing about the current period is that component parts are all bits. That means you never run out of them. You can reproduce them, you can duplicate them, you can spread them around the world, and you can have thousands and tens of thousands of innovators combining or recombining the same component parts to create new innovation. So there’s no shortage. There are no inventory delays. It’s a situation where the components are available for everyone, and so we get this tremendous burst of innovation that we’re seeing.

On corporations and work

The question is, “What are other periods where we saw technology influence the way organizations work?” One nice example comes from the works of Alfred Chandler, where he describes how the telegraph and the railroad had a big impact on the development of the modern corporation. And this was a synergistic operation: one, you had to have a large organization to manage these technologies, and two, you had to have the communications and transportation infrastructure to enable the management at a distance.

So I think now, with what we’re seeing with mobility, we’re going to have a totally different concept of what it means to go to work. The work goes to you, and you’re able to deal with your work at any time and any place, using the infrastructure that’s now become available.

At the base, there’s the innovation infrastructure making better, faster, cheaper networks. There’s the improvement in the human–computer interface because the big challenge in mobile communication has always been dealing with this—quite limited—interface. But then, the kinds of innovations I think will arise on top of that will be innovations in how work is done. And that’s going to be one of the most exciting aspects, in my opinion.

If you look at the beginning of the 20th century, we saw the rise of mass production. Henry Ford and the entire team were down on the factory floor raising this, lowering that, speeding up the assembly line, changing the way things were built, and were able to extract far more efficiencies than were available before. I think the same thing is happening now with digital technology. When we’re all networked, we all have access to the same documents, to the same capabilities, to this common infrastructure, and we can improve the way work—intellectual work, knowledge work—flows through the organization. And again, in my opinion, that will lead to a substantial advantage in terms of productivity.

On free goods and value

Back in the early days of the Web, every document had at the bottom, “Copyright 1997. Do not redistribute.” Now every document has at the bottom, “Copyright 2008. Click here to send to your friends.” So there’s already been a big revolution in how we view intellectual property. So it’s not so much the question of what’s owned or what’s not owned. It’s a question of how can you leverage the assets you have to realize the most value.

I think that the availability of these very inexpensive platforms you’re creating, in disseminating content, means that it’s become intensely competitive. The content is as valuable as it ever was, it’s just the competition that’s pushed the prices down to something that approximates zero. So it’s not something that the content producers necessarily embrace, but it’s something they’re forced into by the nature of the technological change.

In these models, there is typically a revenue-generating component somewhere in the value chain. And most commonly today we’re seeing it on the advertising side. To look at this from a historical perspective, it’s really not so new. If you look at the 1920s, the technological question in the ’20s was, “How can we build a business model around broadcast radio?” And nobody really had a good idea. And back in the mid-1990s we asked, “How can we build a business model around the Internet?” And the preferred model at the time was a micropayments system. That never happened, for some reasons, but what did happen instead is we moved into the advertising model, and the advertising’s model been phenomenally successful.

We have to look at today’s economy and say, “What is it that’s really scarce in the Internet economy?” And the answer is attention. [Psychologist] Herb Simon recognized this many years ago. He said, “A wealth of information creates a poverty of attention.” So being able to capture someone’s attention at the right time is a very valuable asset. And Google really has built an entire business around this, because we’re capturing your attention when you’re doing a search for something you’re interested in. That’s the ideal time to show you an advertisement for a product that may be related or complimentary to what your search is all about.

On workers and managers

I keep saying the sexy job in the next ten years will be statisticians. People think I’m joking, but who would’ve guessed that computer engineers would’ve been the sexy job of the 1990s? The ability to take data—to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it—that’s going to be a hugely important skill in the next decades, not only at the professional level but even at the educational level for elementary school kids, for high school kids, for college kids. Because now we really do have essentially free and ubiquitous data. So the complimentary scarce factor is the ability to understand that data and extract value from it.

I think statisticians are part of it, but it’s just a part. You also want to be able to visualize the data, communicate the data, and utilize it effectively. But I do think those skills—of being able to access, understand, and communicate the insights you get from data analysis—are going to be extremely important. Managers need to be able to access and understand the data themselves.

You always have this problem of being surrounded by “yes men” and people who want to predigest everything for you. In the old organization, you had to have this whole army of people digesting information to be able to feed it to the decision maker at the top. But that’s not the way it works anymore: the information can be available across the ranks, to everyone in the organization. And what you need to ensure is that people have access to the data they need to make their day-to-day decisions. And this can be done much more easily than it could be done in the past. And it really empowers the knowledge workers to work more effectively.

On computer monitoring and risks

One of the really interesting phenomena that’s been going on in the last 20 years is what I call “computer-mediated transactions.” So now, in the middle of almost every transaction from person to person or organization to organization, there’s a computer. And the computer can monitor that transaction, record the information, collect the data, and assure that the transaction is carried out the way it was intended to be carried out. So one of the subtle implications of this is you can now write contracts and make contracts enforceable that simply weren’t enforceable before.

Let me give you an example. Suppose you go rent a car and they say, “Hey, we’ll give you $10 off if you don’t go over the speed limit.” Well, that might sound like a good deal, but what’s to keep you from going over the speed limit? Well, the answer is now they’ve got a transponder in the trunk and it will monitor your behavior and charge you accordingly. And the same thing happens with semitrucks: virtually every semi on the road today has a computer in it. And that computer improves the logistics. It monitors the performance of the driver and it helps things get to the consumer more quickly. So there are a lot of capabilities of that sort that allow you to contract on terms that were just not available to you before.

[At the same time,] you get a new technology in and people are excited about the positive sides of it. Then you see there are also some negative aspects. And you’ll have a regulatory infrastructure that arises to deal with those. I think everybody is very excited about the intended aspects of this technology—the fact that you can personalize, the fact that you can monitor, the fact that you can provide products that are more closely suited to a consumer’s interests and needs. What people are worried about are the unintended consequences, the downsides, the negative sides, the security, the identity theft, the possibility of extortion or embarrassment. These are the problems: not what people want to do but what could happen if these technologies weren’t appropriately managed.

On reshaping industries

We’re obviously going to see enormous change in the traditional marketing industry. You look at TV, you look at print, you look at radio and other media of that sort. On the Internet, we’ve learned to measure advertising effectiveness, and the challenge now is to move those same effectiveness measures over to the offline media.

That can be done. I think we’re going to see vast improvements in how those industries function in the future. And in general, if we look at service industries—well, everybody I think is in agreement that we’re going to see lots of efficiency improvements in services, because we do have this network capability. We have the technological infrastructure. We can improve communication flows. The second beneficiaries of that will be with service industries who’ve already seen a lot of advances in manufacturing productivity. And the tough nut is the one we’re working on cracking now.

What I actually work on to a large extent is a current feeding of the auction model that we have at Google. As you know, all of our ads are sold by auction. That’s a relatively novel pricing mechanism in the ad world. And there’re a lot of intricacies that involve how you manage that. We’d like to extend that model to the offline world: to radio, TV, print, and other media. It’s a model that was enabled by the Internet. It’s not something you could’ve done without that information technology there. And it’s a great model for all sorts of resource allocation issues.

I think the people who originally designed the model way back in 2001 had a very, very useful insight. They recognized that the content provider has impressions to sell. So you’ve got some space in your TV show. You’ve got some space on your page. You’ve got some space that’s available to put an ad. But what the advertiser wants to pay for is clicks or conversions or visits. So they don’t really care how many impressions they show. Normally, what they care about is getting people into their store and, ultimately, getting people to purchase. So you have to build a system that allows the publisher to sell impressions but the advertiser to buy clicks. And I think we’ve managed to accomplish that in a nice, elegant way. 

By Rob Iati, Partner, The TABB GroupJULY 10, 2009

I want to talk about a big topic here today: the quest for computable knowledge. It’s a topic that spans a lot of history, and that I’ve personally spent a long time working on. I want to talk about the history. I want to talk about my own efforts in this direction. And I want to talk about what I think the future holds.

So what do I mean by “computable knowledge”? There’s pure knowledge—in a sense just facts we know. And then there’s computable knowledge. Things we can work out—compute—somehow. Somehow we have to organize—systematize—knowledge to the point that we can build on it—compute from it. And we have to know methods and models for the world that let us do that computation.

Well, I think in history the first really big step in this direction was taken a really long time ago—with the invention of counting and arithmetic. The big idea that we know pretty much existed by 20,000 BC was that you could just abstractly count objects, independent of what the objects were. And then that there were definite unchanging rules of arithmetic that could let one abstractly compute things.

But of course just counting things is a very coarse form of systematic knowledge. Human language lets us describe much more, but it isn’t systematic—it doesn’t allow us to go directly from our knowledge to computing new things. But it was still a crucial step in perhaps 4000 BC when written language first emerged—and it became possible to systematically record and transmit knowledge about things.

It didn’t take long before numbers and writing led to kings in Babylon making pretty broad censuses of people and commodities. From which at least it was possible to compute taxes. But when it came to working out more about what would happen in the world—well, probably most people just assumed it was all just fate, and that nothing much could be predicted.

Thousands of years went by. But then something happened. People had known that there were regularities to be seen if not on earth, at least in the heavens. And then it was realized that one could use arithmetic—the same arithmetic that worked for commerce and for land surveying—to predict things about the heavens. To work out the behavior of the planets, and even to say things about spectacular events like eclipses. It was the beginning of the tradition of exact science as we know it.

Of course, it wasn’t at all clear where the boundaries were. Things worked in predicting the heavens, so why not predict the weather, the rise and fall of kingdoms, and everything? Of course, that didn’t work so well. But still, with people like Pythagoras around 500 BC, it seemed that nature, and music, and much more—even if not human affairs—could perhaps be described, and computed, using numbers. There were other possibilities too, though, even at that time.

And indeed on the other side of the world, probably around 400 BC, Panini was coming up with rules—not numbers, just rules—that described the grammar of Sanskrit. Going from human language—and finding a formal way to describe its structure—and in effect to use that to compute the poetic forms that could be produced. That idea did reappear a few times in scientific history; for example, Lucretius talked about how atoms might make up the universe as letters make up words and sentences. But for practical purposes, the notion of creating formal systems from the structure of languages was lost for more than 2000 years.

And instead, what emerged in Greek times—probably around 350 BC—was the idea of logic. The notion—found in the works of Aristotle—that the structure of human arguments, of human reasoning, can be represented in a stylized form—using logic. The idea that just as numbers let one abstractly count things, so logic could let one abstractly see the structure of certain forms of deduction and reasoning—and rhetoric. So that one could find ways to derive conclusions about the world by structured, formal, reasoning.

But what really took off from that idea was not really general reasoning—but instead specific forms of reasoning about arithmetic and geometry—about mathematics. And quite quickly—with Euclid and so on—there began to be a whole tradition of finding the truths of mathematics by derivation and by proof.

At the same time, though, there was another tradition. The tradition of trying to organize knowledge—of making lists and categories of things—like in Aristotle. And of just outright collecting knowledge, like at the library of Alexandria, where, as it happens, Euclid worked.

Well, in a sense these were academic, philosophical pursuits. But by 200 BC people like Archimedes were putting mathematics—and computation—firmly into practice in creating technology. And it’s looking as if Archimedes probably did another important thing too: he really started mechanizing the doing of computation. You know, I’ve spent much of my life working in that kind of direction, and it’s fun to realize that I’m in a very old business. Because it’s looking as if Archimedes may well have started building gear-like devices that could do computation—say astronomical predictions—22 hundred years beforeMathematica and so on.

Well, after all the successes of Greek philosophy and mathematics and so on, one gets the impression that lots of people thought that everything that could be figured had been figured out. And nobody pushed hard for a long time to do more. Still, just like in Babylonian times, there were people trying to compute—to predict—more about the world. A lot of it was hocus pocus. A notable example was Ramon Lull, from around 1300, who invented a whole combinatorial scheme for generating possible ideas, and explaining what could happen in the world. That didn’t work so well. But still, there was a general feeling that the kind of systematic derivations that existed in mathematics should somehow be applicable to at least some of the goings-on in the natural world.

And by the end of the 1500s—with Galileo and so on—there was a notion that physical processes could be understood in the “language of mathematics”. The big breakthrough, though, was Isaac Newton in 1687 introducing what he called “mathematical principles of natural philosophy”. Really pointing out that things in “natural philosophy” could be worked out not by some kind of humanlike reasoning, but rather by representing the world in terms of mathematical constructs—and then using the abstract methods of mathematics (and calculus and so on) to work out their behavior.

Why it worked wasn’t clear. But the big fact was that it seemed to be possible to work out all sorts of unexpected things—and get the right answers—in mechanics, both celestial and terrestrial. And it was that surprising success that has propelled mathematics as the foundation for exact science for the past 300 years.

But back to the main story. A lot of relevant stuff was going on at the end of the 1600s. In the 1660s, people like John Graunt were in effect inventing statistics. Going beyond the Babylonian census to have more abstract mathematical representations of features of states—and things like life tables. And around 1700 there was something else: Gottfried Leibniz talking about his “characteristica universalis”.

You see, Leibniz had started taking seriously the idea that it might be possible to really make knowledge computable. He wanted to invent a universal symbolic language that could represent everything. And then effectively to apply methods of logic and mathematics to this symbolic representation—to resolve all human arguments. He started building clockwork computers. He tried to persuade the leaders of his time to start collecting knowledge in big libraries. But even though I think he had very much the right idea—one just couldn’t get there with the technology of 1700. It was a few hundred years too early.

But even though Leibniz’s big idea didn’t work out, the notion of systematizing knowledge was becoming more and more popular. The British Museum was founded in 1700, as a kind of universal collecting place—not of knowledge, but of actual things, natural and artificial. In 1750, Carl Linneus went beyond Aristotle, and came up with the modern scheme for systematically classifying living organisms. In the mid-1700s encyclopedias—like Encyclopedia Britannica—were getting founded and there was an effort to collect everything that was known into systematic books. Meanwhile, on the computation side, mathematics was doing pretty well, both in terms of abstract theorem development, and practical use in physical science and engineering. There were efforts to make it more systematic.

In the 1830s Charles Babbage got serious about automating the computation and printing of mathematical tables, and started imagining a kind of universal “analytical engine”, which, as Ada Lovelace described it, could “weave algebraical patterns just as the Jacquard loom weaves flowers and leaves”. Then in the 1850s George Boole pointed out that logic was really just like mathematics—and that one could use mathematical methods to work out questions in logic. And somehow it was beginning to seem that mathematics had a great deal of universality.

Indeed, between non-Euclidean geometry in the 1820s, abstract algebra in the mid-1800s, and transfinite numbers in the 1880s, it had begun to seem like mathematics was a kind of universal framework for abstraction. In 1879 Gottlob Frege came up with predicate logic, in effect trying to find a way to represent general truths, whether in mathematics or elsewhere. Mathematics was doing so well in physics and engineering. So many new theoretical areas were springing up from algebra, calculus, geometry and so on. It must have seemed in the late 1800s as if the whole world would soon be described and worked out in terms of mathematics.

There were efforts—by Peano and so on—to come up with definitive axiomatization of mathematics. And there was an increasing conviction that the methods of Euclid—starting from axioms and then systematically deriving theorems—would unravel all of mathematics, and perhaps all of science. By 1910, there were efforts like the Whitehead-Russell Principia Mathematica, where the notion was to formalize mathematics in terms of logic, and then, in effect, just build up everything from a modest set of axioms. And David Hilbert had the idea that really mathematics should be almost mechanical: that one could in effect just churn out all truths automatically.

Well, along with all this theoretical and foundational activity, there were also practical things going on. Starting in the mid-1600s—possibly with precursors going back to Archimedes—mechanical calculators were increasingly developed, and by the end of the 1800s they were commonplace. The idea of the Jacquard loom—and of things like player pianos—had introduced a notion of programming: having punched cards that could specify what operations the mechanical device should perform. And very gradually that idea began to be generalized.

There had emerged from calculus and so on the notion of an abstract mathematical function: f(x), where somehow the function itself was a bit like the variable. But just what could a function be? Perhaps it was something built from logic-like constructs, as in the work on Moses Schönfinkel on combinators. Perhaps something built with rewrite rules, as in the work of Emil Post. Perhaps something built with the operation of “primitive recursion”—in effect a kind of arithmetic recurrence.

All of these seemed like possible representations, but none seemed fundamental. And with the discovery of the Ackermann function in 1920, for example, it became clear that at least primitive recursion wasn’t the complete story of reasonable functions. But from all this, there was at least emerging a notion of being able to treat functions like numbers and other data. Actually, Leibniz had already suggested numbering possible logic expressions way back in 1679.

What did this have to do with the foundations of mathematics? Things like set theory had developed, and there had started to be all sorts of brewing paradoxes and things as set theory started to try to talk about itself. But still it seemed that with fancy enough footwork Hilbert’s idea of systematically finding all truths in mathematics could be saved. But that all changed in 1931 with the arrival of Gödel’s Theorem.

Gödel started by taking one of those paradox-like statements: “This statement is unprovable.” But then he did something very interesting. He showed that that statement could be expressed as a statement in arithmetic. He found a way to set up equations about integers and other constructs in arithmetic, and have them represent his at-first-not-mathematical-seeming statement. That had a big consequence: it showed that with mathematics itself, one could set up statements that one could then show couldn’t be proved or disproved within mathematics. Mathematical statements about which mathematics just didn’t have anything to say.

Well, that was pretty interesting for the philosophy of mathematics. But in some sense it was a technicality in Gödel’s proof that really changed the world. You see, to show what he showed, Gödel in effect ended up inventing programming. Because to show that his funny statement could be represented in terms of arithmetic, he actually showed that any of a huge class of statements could also be represented that way.

At first it wasn’t incredibly clear what the significance of this was. Whether it was just a technical detail of the particular mathematical systems—things like “general recursive functions”—that Gödel had considered. But what happened next was that in 1935 Alonzo Church came up with lambda calculus—and then in 1936 Alan Turing came up with Turing machines.

Both of them were in effect trying to come up with ways to describe everything that could reasonably be computed. Turing’s scheme was the clearest. He in effect had a way of describing a procedure for computing things—potentially by machine. One might have thought that to compute different things, one would always have to have a different machine. One square-roots machine. Another exponentials machine. And a quite different machine to do logic puzzles. But the crucial thing that Turing showed was that in fact there are universal machines—which can just be programmed to do any of these operations, or in fact to emulate any other Turing machine.

At first it wasn’t clear quite what the significance of this was. After all, perhaps there were different ways to construct machines that would have different properties. But meanwhile, people were constructing more and more machines that did practical computations. Beyond calculators, there had been Hermann Hollerith’s census-counting machine. There were starting to be machines for doing logic, and for combinatorially breaking codes. There were machines for doing more and more elaborate equation solving. And by the 1940s, electronics was becoming established as the underlying technology of choice.

There was also increasing knowledge of the physiology of the brain, and it was beginning to look as if somehow it might all just work with electronics. And by the 1940s, people like McCullough and Pitts were using Turing’s universal machine idea as evidence that somehow neural circuits could successfully reproduce all the things brains do. Particularly through John von Neumann that idea then merged with practical work going on to try to make programmable electronic computers. And the result was that the theoretical idea of universal computation got turned into the idea of producing universally programming electronic computers—and the idea of software.

In the early days of electronic computers, it was pretty much assumed that computers were somehow like electronic brains, and that eventually computers would be able to take over brain-like work, just like mechanical machines had been able to take over most mechanical work. Movies began to portray computers that could be asked questions, and compute answers. There didn’t seem to be any doubt that all knowledge would soon be computable—even if perhaps computers would show signs of the “simple logic” on which they were based—somehow being more “robotic” in their answers than humans would be.

There were some practical efforts to actually start working out how computable knowledge would be set up. Following up on crowdsourced projects like the assembly of the Oxford English Dictionary at the end of the 1800s, there were projects like the Mundaneum—which in particular tried to collect all the world’s knowledge on 12 million index cards, and be able to answer questions sent in by telegraph. By 1945, Vannevar Bush was talking about the “memex” that would provide computerized access to all the world’s knowledge. And by the mid-1950s, the idea of artificial intelligence was becoming all the rage. In a sense artificial intelligence was thought of a bit like mathematics: the idea was to make a general-purpose thinking machine, that could start just from nothing, and learn and figure out anything, just like humans—rather than to make something that started from a large corpus of existing knowledge or methods.

One particular direction that was pursued a lot was handling human language. In the mid-1950s, at almost exactly the same time, two important things had happened. Noam Chomsky had suggested the idea—in a sense finally a followup to Panini from 400 BC—that the grammars of human languages could be represented in algorithmic form. And at almost exactly the same time, the idea had been introduced of constructing languages for computers in algorithmic form.

But despite the idea of an algorithmic structure to human language, it proved a lot more difficult than expected to do actual computations with human language—whether for language transformation, information retrieval or whatever. Computer languages were a much better idea, though. They provided precise formal specifications for what computers should do, given in a form that was close enough to human language that people could learn them a bit like the way they learn human languages.

Meanwhile, through the 1960s, 1970s, and 1980s, computers were gradually becoming more and more powerful, and more and more common. There were ideas like relational databases. Then there were applications like graphics and word processing. And the concept that had begun as a kind of footnote to technical work on the foundations of mathematics had become a central part of our world.

But still, the early idea that computers would be like brains—and that all the world’s knowledge would be computable—just hadn’t happened. There were many places computers were extremely useful. But there was more to do. Just how far can the notion of computation go? How much can one compute? About the world? About human knowledge?

Well, I’ve thought about these things a lot over the years. In fact, the three large projects of my life have all in a sense been concerned with aspects of this very question.

In Mathematica, for example, my goal has been to create a framework for doing every possible form of formal computation. Mathematica is in a sense a generalization of the usual idea of a computer language. In a sense, whatMathematica tries to do is to imagine all possible computations that people might want to do. And then to try to identify repeated structures—repeated lumps of computational work—that exist across all those computations. And then the role of the Mathematica language is to give names to those structures—those lumps of computational work. And to implement them as the built-in functions of the system.

I wanted Mathematica to be a very general system. Not a system that could just handle things like numbers, or strings, or even formulas. But a system that could handle any structure that one might want to build. So to do that I in effect went back to thinking about the foundations of computation. And ended up defining what one can call unified symbolic programming. One starts by representing absolutely everything in a single unified way: as a symbolic expression. And then one introduces primitives that represent in a unified way what can be done with those expressions.

In building Mathematica over the past 23 years one of the big ideas has been to include in it as much—in a sense formal—knowledge as possible. The methods, the algorithms, the structures that have emerged throughout the fields of mathematics and computation. It’s really been an interesting thing: we have this very unified environment, and as we add more and more to it, it’s a kind of recursive process. Because it’s unified, and because in a sense so much of what it does is automated, the new things we add get to build on everything that’s there before. In a sense we get to see the generality of the idea of computation; we get to use it to create ways to handle even more kinds of things.

You know, it’s funny how long it takes for paradigms to sink in. The basic ideas for the unified symbolic programming that exists in Mathematica I came up with nearly 30 years ago. But every few years I realize more that one can build with those ideas. Mathematica started with technical and mathematical computation. But it’s turned out that its foundations are general enough to go far beyond that. To make a lot more things computable in systematic ways.

Well, one of the reasons I wanted to build Mathematica in the first place was that I wanted to use it myself. To explore just what the broad implications are of the fundamental idea of computation. You see, while computation has been of great practical importance—even in science—there’s a lot more to explore about its implications for the foundations of science and other things. If we’re going to be able to do science—or in general to make knowledge systematic—we kind of have to imagine that there are ultimately theories for how things work. But the question is: what are the primitives, what’s the raw material, for those theories?

Well, in the exact sciences there’s been a lot of success over the past 300 years with theories based on mathematics. With taking those ideas of numbers, and algebra, and calculus, and so on—and building theories of the world out of them. That’s certainly the way the big successes of areas like physics have worked. But of course there’s a lot in the world that we don’t yet know how to explain. There’s a lot in nature, for example, that just seems somehow complex, and beyond our theories.

But the big thing that I started asking myself nearly 30 years ago now is whether the reason it seems that way is just that we’ve been thinking too narrowly about our theories.

Maybe it’s not enough to use the primitives that we happen to have developed in the course of mathematics. Maybe the world doesn’t happen to work that way. Well, what other primitives could we use? In the past, we would have had no idea. But now that we understand the notion of computation, we do have an idea. What about just using simple programs? What about basing our theories and models not just on the constructs of math, but on the general kinds of rules we find in programs?

Well, OK. But what kind of programs?

Normally when we use programs we do it in a very engineering kind of way. We set up particular programs—usually very complicated ones—that perform particular tasks we want. Well, my idea of nearly 30 years ago was to ask, what if we just look at the world of programs as we look at an area of natural science? If we just see what’s out there in the computational universe of possible programs? Let’s say we just start with the simplest possible programs, and see what they do.

I happened to study a lot some systems called cellular automata, that just consist of rows of black and white cells, with little rules for how the colors of the cells should be updated. Well, at first I’d assumed that if the rule for the cellular automaton was simple, its behavior would somehow have to be correspondingly simple. I mean, that’s the intuition we tend to have from everyday life, and from today’s engineering and so on. If you want to make something complicated, you have to go to a lot of effort. And you have to follow complicated rules and plans. But I decided that I should just do the experiment and see what was true. Just trying running every possible simple cellular automaton program, and see what it did.

And the result was really, really surprising. And it kind of shattered my intuition about how things work. Because what I found was that even very simple programs—started off in the simplest way—could produce incredibly complex behavior. Patterns that if you saw them you’d say, “That must have been produced by something really complicated.”

Well, so what I found, by doing in effect empirical computational science, was that in the computational universe there’s incredible richness in a sense very near at hand. I think this is a pretty fundamental thing. And actually I think it explains a pretty fundamental observation about our world.

You see, even though when we build things it always seems to take a lot of effort to build something that is complicated, nature doesn’t seem to work that way. Instead, it seems as if nature has some kind of secret that effortlessly produces all sorts of complexity. And I think we now know what that secret is. It’s that nature is sampling all sorts of programs in the computational universe. But even though the programs are simple, they just don’t always happen to be ones whose behavior is simple. They don’t happen to be the programs that correspond to things we’ve built with our mathematics and our traditional mathematical science.

Ultimately it’s not that you can’t build complexity from mathematical primitives and so on. But what’s happened is that the exact sciences have tended to just define themselves to be about cases where that doesn’t happen. We haven’t studied the full computational universe of possibilities, only a thin set that we’ve historically found to be tractable.

Well, this has many implications. It gives us a “new kind of science”—as I pointed out in the title of the big book I wrote about all this. A kind of science that in a sense generalizes what we’ve had before. That uses a much broader set of primitives to describe the world.

Already that science has had lots and lots and lots of applications. All sorts of new models of natural and man-made phenomena. Where the foundation is not a mathematical equation, but a computational rule. The science has shown us new ways to think about all sorts of things. Not just traditional science. Also technology—creating things that go beyond what one can foresee from traditional engineering. In art—capturing the essence of the richness that seems to give nature its aesthetic. Even in philosophy, thinking about old questions like free will. Maybe the science will even get us to the ultimate point in a question for knowing about the world: being able to give us an ultimate fundamental theory of physics.

You know, the way physics has gone in the last few hundred years, it seems kind of hopeless to even imagine that one might be able to find a truly fundamental theory of physics. It seems like at every stage, models in physics have just been getting more and more complicated. And one might assume, given the obvious complexity that we see in our universe, that there’s no possibility for there to be an ultimate simple theory of it all.

But here’s the critical thing one learns from studying the computational universe: if one just samples possible programs, even very simple ones can have great richness in their behavior. So then the question is: in the computational universe of possible programs, just where might the program for our universe lie? Is it in effect a giant program? Or something tiny? That we might be able to find just by searching the space of programs. Well, I don’t know for sure. Though a crucial fact about our universe, long noted by theologians, is that it certainly isn’t as complicated as it could be. There’s at least some order in it. And maybe that means we can find a program for it that’s really small.

So what’s actually involved in universe hunting? I think one has to concentrate on in a sense very abstract models, where there aren’t built-in notions of space, or time, or matter, or really anything that’s too familiar from our existing physics. But if one just starts enumerating very simple sets of rules, the remarkable thing that happens is that one starts finding candidate artificial universes that at least aren’t obviously wrong—obviously not our universe.

There’s one big problem with all this. A fundamental pheonomenon I call computational irreducibility.

You see, once we start thinking in computational terms, we start to be able to ask some fundamental questions. Traditional theoretical science has been very big on the idea of predictability: of somehow working out what systems will do. Well, in the past it always seemed reasonable to assume that if the science was done properly, that kind of prediction would be possible. That somehow the scientist would be able to be smarter than systems in nature, and work out what they would do more efficiently than they do it themselves. But one of the big discoveries from what I’ve done is what I call the Principle of Computational Equivalence. Which says that as soon as a system isn’t just obviously simple in its behavior, it will be as sophisticated computationally as any other system.

So this means that the scientist will just be equivalent to the system being studied—not smarter than it. And that leads to what I call computational irreducibility: that a great many systems in effect perform computations that we can’t reduce, or predict. That we just have to simulate to see what they will do. Well, that’s obviously a problem for universe hunting. Because we can’t expect to simulate every step of the evolution of our universe.

And indeed what happens is that you get candidate universes that flap around in all sorts of complicated ways. They could actually be our universe. But computational irreducibility makes it really hard to tell. It can even be like Gödel’s theorem: it becomes undecidable whether a particular candidate universe has a particular property like our real universe. Still, I think it’s quite possible that we’ll be lucky—and be able to find our universe out in the computational universe. And it’ll be an exciting moment—being able to sort of hold in our hand a little program that is our universe. Of course, then we start wondering why it’s this program, and not another one. And getting concerned about Copernican kinds of issues.

Actually, I have a sneaking suspicion that the final story will be more bizarre than all of that. That there is some generalization of the Principle of Computational Equivalence that will somehow actually mean that with appropriate interpretation, sort of all conceivable universes are in precise detail, our actual universe, and its complete history.

But those are issues for another day. For now the main point is that from NKS—this “new kind of science”—one learns just how much of the world can really be thought of in computational terms. There are limits from computational irreducibility to how easy it is to work out consequences. But there’s much more than just traditional science that can be represented in computational terms.

Well, back to the main thread of computable knowledge. In the last decade or so a lot of practical things have happened with it. The web has arisen, putting huge amounts of material in computer-readable form. Wikipedia has organized a lot of material—encyclopedia style—and has codified a lot of “folk knowledge” about the world. And search engines have arisen, giving one efficient ways to find anything that’s been explicitly written down on the web—fact, fiction, or otherwise. Meanwhile, all sorts of things that at one time or another were considered tests for artificial intelligence—playing chess, doing integrals, doing autonomous control—have been cracked in algorithmic ways.

But the general problem of making the world’s knowledge computable—of making computers really act like the science-fiction computers of the 1950s—has always seemed too difficult. Or at least that’s the way it seemed to me. And I guess every decade or so I would wonder again: are we ready to try to make the world’s knowledge computable yet? And I would say, “No, not yet.” We’re closer than Leibniz was. But we’re not there yet.

Well, a few years ago I was thinking about this again. And I realized: no, it’s not so crazy any more. I mean, I have Mathematica, which gives a foundation—a language for representing knowledge and what can be computed about it. And from NKS I understood a lot more about what can be represented computationally, and just how simple the underlying rules to produce richness and complexity might be. And thinking that way got me started on the third big project of my life—which has turned into Wolfram|Alpha. And in fact, today [June 15, 2009] it’s exactly one month since Wolfram|Alpha was first launched out into the world.

The idea of Wolfram|Alpha was to see just how far we can get today with the goal of making the world’s knowledge computable. How much of the world’s data can we curate? How many of the methods and models from science and other areas can we encode? Can we let people access all this using their own free-form human language? And can we show them the results in a way that they can readily understand? Well, I wasn’t sure how difficult it would be. Or whether in the first decade of the 21st century it’d be possible at all. But I’m happy to say that it worked out much better than I’d expected.

Of course, it’s a very long-term project. But we’ve already managed to capture a decent amount of all the systematic information that you’d find in a standard reference library. We’ve managed to encode—in about 6 million lines ofMathematica code—a decent slice of the various methods and models that are known today.

And by using ideas from NKS—and a lot of hard work—we’ve been able to get seriously started on the problem of understanding the free-form language that we humans walk up to a computer and type in. It’s a different problem than the usual natural-language processing problem. Where one has to understand large chunks of complete text, say on the web. Here we have to take small utterances—sloppily written questions—and see whether one can map them onto the precise symbolic forms that represent the computable knowledge we know.

Before we released Wolfram|Alpha out into the wild, we had to try to learn that pidgin language people use—by looking at corpora of questions and answers and statements. But in the past month we have seen something wonderful: we have hundreds of millions of actual examples of humans communicating with the system. So—like a child learning language or something—we can now start to learn just how to understand what we’re given.

Well, so what will happen with Wolfram|Alpha, and this whole quest to make the world’s knowledge computable? Wolfram|Alpha will get better—hopefully quite quickly. So that one will really be able to ask it more and more of those questions one might have asked a science-fiction computer of the 1950s.

It’s interesting to see how it compares to the early ideas of artificial intelligence. In a sense, those concentrated on trying to make a computer operate like a person—or a brain—but bigger, faster, and stronger. To be able to work things out by reasoning. A bit like pre-Newtonian natural philosophy. But what we’re trying to do with Wolfram|Alpha is to leverage on all the achievements of science and engineering and so on. We’re not trying to fly by emulating birds; we’re just trying to build an efficient airplane. So when you ask Wolfram|Alpha to work out something in physics or math or whatever, it’s not figuring out the answer by reasoning like a person—it’s just trying to blast through, using the best methods known to our civilization, to get the answer.

It’s not obvious that in 2009 computers would yet be powerful enough to get lots of answers in what amounts to a human reaction time. But they are. And that—combined with the existence of the web, and being able to deliver answers on it—is what makes Wolfram|Alpha possible.

What of the future? Well, Wolfram|Alpha is in a sense taking existing knowledge, and encoding it in computable form, and computing answers from it. And almost everything it’s asked is unique—it’s never been asked the same question before, and nobody’s ever written down the answer on the web before. It’s getting figured out in real time, when it’s asked. And it’s in effect coming to new conclusions that have never been seen before.

But in a sense Wolfram|Alpha is very traditional in its knowledge: it’s using models and methods and structures that already exist—that are already part of the existing canon of science, engineering, and so on. But what about inventing new models and methods on the fly? Well, in a sense NKS gives a clear direction for doing that. We can just look out there in the computational universe and see if we can find things that are useful to us.

At first, that might seem crazy. By sampling a space of simple programs, how could we ever get anything rich enough to be useful for practical purposes? Well, that’s a key lesson from NKS: out in that universe of simple programs are lots with all sorts of elaborate behavior. Maybe even rich enough to be our whole universe. But certainly rich enough to be useful for lots of purposes.

Traditionally in doing engineering—or science—we tend to want to construct things step by step, understanding how we will achieve our goals. But NKS tells us that there’s a much richer supply of things out there in the computational universe—just ready to be mined. It’s a bit like the situation with physical materials: we make technology in effect by going out in the material world and seeing materials with certain properties, and then realizing that we can harness them for our particular technological purposes. And so it is in the computational universe.

In fact, increasingly in Mathematica we are using algorithms that we were not constructed step-by-step by humans, but are instead just found by searching the computational universe. And that’s also been a key methodology in developing Wolfram|Alpha. But going forward, it’s something I think we’ll be able to use on the fly.

We did an experiment a few years ago called WolframTones. That samples the computational universe of simple programs to find musical forms. And it’s remarkable how well it works. Each little program out there defines a sort of artificial world, with its own definite rules that we recognize as “making sense”, but which has enough richness in its behavior to make it seem interesting and aesthetically engaging to us.

Well, we can do the same thing with modeling, with recognizing patterns, with constructing engineering objects. Just going out into the computational universe and mining it for things that are useful to us. Mass customizing not only art, but also scientific theories and engineering inventions. Even mathematics.

You know, if you look at all the mathematics that we do today, its axioms fit comfortably on a page or two. And from that page or two emerge all the millions of theorems that constitute the mathematics of today. But why use the particular axioms that we have used so far? What happens if we just start looking at the whole space of possible axiom systems? In a sense at the space of possible mathematics? Well, it’s pretty interesting out there. All sorts of different kinds of things happen.

If one looks at the history of mathematics, one gets the impression that somehow the mathematics that’s been studied is all there could possibly be. That we’ve reached the edge of the generalization—the abstraction—that can be done. But from studying the space of all possible mathematics, it’s clear that isn’t true. One can find our areas of mathematics in there. Logic, for example, turns out to be about the 50,000th axiom system in the space of all possible axiom systems.

But there doesn’t seem to be anything special about the axiom systems we’ve actually used. And in fact, what I suspect is that really the choice of them is just a reflection of the history of mathematics: that everything we see today is what we get by generalizing specifically from the arithmetic and geometry that were studied in ancient Babylon. That history of mathematics has informed what we’ve been able to do in theoretical science. But there’s so much more out there. That we can see in the computational universe. And that maybe we’ll even be able to find and explore on the fly with a future Wolfram|Alpha.

So what will the future hold for all of this? We’ll be able to compute from the knowledge that our civilization has accumulated. We’ll be able to discover and invent more and more on the fly. And in time I expect that more and more of what we have in the world will end up being “mined” from the computational universe.

What will this mean? Well, right now the things we build in engineering are usually sort of limited to things where we can understand each part of what we do. When we write programs, we do it one piece at a time, always setting it up so that—at least apart from bugs—we can foresee what the program will do. But increasingly, the technology we use will come from mining the computational universe. We’ll know what the technology does—we’ll know its purpose—but we won’t necessarily understand how it does it.

Actually, it’ll be a little like looking at systems in nature. There’ll be the same kinds of complexity. Actually probably more. Because even in an area like biology, where there’s a lot happening even at a molecular scale, the complexity is in many ways limited by the effects of processes like natural selection. But we won’t have those kinds of constraints when we get our technology from the computational universe. We’ll potentially be able to get the most efficient, most optimal, version of everything. And it almost always won’t have any of the simplicity—the identifiable pieces and mechanisms—that exist in our technology today.

So then the big question is: what will the technology do? What purposes will we find for our technology? It’s a very interesting thing, to look at our technology today, and to ask what people at other times in history would have thought about it. Because what we realize is that not only would they not understand how it works. But they also would not understand why anyone would make it. The purposes evolve as the technology evolves.

And as we look at the future of computable knowledge, we may ask what purposes will be found for it. Some we can foresee. But many, I suspect, we cannot. Today our technology is rife with history: we see particular mechanisms over and over again because they were invented at some point, and propagated into the future. But as more of our technology is found—in a sense from scratch—in the computational universe, less history will be visible. But where the thread of history remains—and the arc of our civilization continues to be visible—is in the purposes that are found for things. There is much that is possible; but what we choose to do depends on the thread of our history.

Today is an exciting moment—indeed, I suspect, a defining moment in human history. When we are making a transition from a world in which computation is just one element to a world in which computation is our central concept. In which our thinking, our actions, and our knowledge all revolve around computation.

And what makes this possible is that we are finally now getting into a position where we can take all that knowledge that we as humans have accumulated in our civilization, and encapsulate it in an active computational form. And when we do this, we make it possible to dramatically extend and generalize all our achievements so far.

In a sense our whole history so far has been played out in a tiny corner of the computational universe. But we are now in a position to take all of our knowledge and achievements, and go out and colonize the whole computational universe, extending our human purposes and experience to an unrecognizably broad domain.

I feel extremely fortunate to live at a time in history when all of this is unfolding, and to be in a position myself to contribute to what can happen. I thank you all here for recognizing the journey I have taken so far. And I look forward to everything that will be possible in the years to come.
Note: a summary timeline of the quest for computable knowledge, based on Stephen Wolfram’s earlier notes, is available here.