A Reactionary Theory of Modern History

Thursday, February 28th, 2008

Mencius Moldbug notes that the only history taught and discussed today comes from a decidedly progressive point of view, so he offers up ten books from which to construct a reactionary theory of modern history:

This Psychologist Might Outsmart the Math Brains Competing for the Netflix Prize

Thursday, February 28th, 2008

This Psychologist Might Outsmart the Math Brains Competing for the Netflix Prize:

Many of the contestants begin, like Cinematch does, with something called the k-nearest-neighbor algorithm — or, as the pros call it, kNN. This is what Amazon.com uses to tell you that “customers who purchased Y also purchased Z.” Suppose Netflix wants to know what you’ll think of Not Another Teen Movie. It compiles a list of movies that are “neighbors” — films that received a high score from users who also liked Not Another Teen Movie and films that received a low score from people who didn’t care for that Jaime Pressly yuk-fest. It then predicts your rating based on how you’ve rated those neighbors. The approach has the advantage of being quite intuitive: If you gave Scream five stars, you’ll probably enjoy Not Another Teen Movie.

BellKor uses kNN, but it also employs more abstruse algorithms that identify dimensions along which movies, and movie watchers, vary. One such scale would be “highbrow” to “lowbrow”; you can rank movies this way, and users too, distinguishing between those who reach for Children of Men and those who prefer Children of the Corn.

Of course, this system breaks down when applied to people who like both of those movies. You can address this problem by adding more dimensions — rating movies on a “chick flick” to “jock movie” scale or a “horror” to “romantic comedy” scale. You might imagine that if you kept track of enough of these coordinates, you could use them to profile users’ likes and dislikes pretty well. The problem is, how do you know the attributes you’ve selected are the right ones? Maybe you’re analyzing a lot of data that’s not really helping you make good predictions, and maybe there are variables that do drive people’s ratings that you’ve completely missed.

BellKor (along with lots of other teams) deals with this problem by means of a tool called singular value decomposition, or SVD, that determines the best dimensions along which to rate movies. These dimensions aren’t human-generated scales like “highbrow” versus “lowbrow”; typically they’re baroque mathematical combinations of many ratings that can’t be described in words, only in pages-long lists of numbers. At the end, SVD often finds relationships between movies that no film critic could ever have thought of but that do help predict future ratings.

The danger is that it’s all too easy to find apparent patterns in what’s really random noise. If you use these mathematical hallucinations to predict ratings, you fail. Avoiding that disaster — called overfitting — is a bit of an art; and being very good at it separates masters like BellKor from the rest of the field.

In other words: The computer scientists and statisticians at the top of the leaderboard have developed elaborate and carefully tuned algorithms for representing movie watchers by lists of numbers, from which their tastes in movies can be estimated by a formula. Which is fine, in Gavin Potter’s view — except people aren’t lists of numbers and don’t watch movies as if they were.

Potter likes to use what psychologists know about human behavior. “The fact that these ratings were made by humans seems to me to be an important piece of information that should be and needs to be used,” he says. [...] One such phenomenon is the anchoring effect, a problem endemic to any numerical rating scheme. If a customer watches three movies in a row that merit four stars — say, the Star Wars trilogy — and then sees one that’s a bit better — say, Blade Runner — they’ll likely give the last movie five stars. But if they started the week with one-star stinkers like the Star Wars prequels, Blade Runner might get only a 4 or even a 3. Anchoring suggests that rating systems need to take account of inertia — a user who has recently given a lot of above-average ratings is likely to continue to do so. Potter finds precisely this phenomenon in the Netflix data; and by being aware of it, he’s able to account for its biasing effects and thus more accurately pin down users’ true tastes.

Echocrome Trailer

Thursday, February 28th, 2008

I never thought I’d see an M.C. Escher video game. This Echocrome Trailer is mind-bending:

The case against democracy

Wednesday, February 27th, 2008

I recently admitted that Mencius Moldbug has hijacked my brain with his ten red pills. Earlier in that same piece he asks us to consider the case against democracy:

Have you ever considered the possibility that democracy is bunk?

I grew up believing in democracy. I’ll bet you did too. I spent 20 years of my life in democratic schools. I’ll bet you did too.

Suppose you were a Catholic in 16th-century Spain. Imagine how hard it would be for you to stop believing in Catholicism.

You are a Catholic. Your parents were Catholics. You were educated by Catholics. You are governed by Catholics. All your friends are Catholics. All the books you’ve ever read were written by Catholics.

Sure, you’re aware that not everyone in the world is a Catholic. You’re also aware that this is the cause of all the violence, death and destruction in the world.

Look at what Protestants do when they get into power. They nail genitals to the city gates. They behead their own wives. Crazy stuff! And let’s not even start on the Turks…

Now suppose you’re you. But you have a time machine that lets you talk to this 16th-century Spanish Catholic version of you.

How do you convince this guy or gal that the answer to all the world’s problems is not “more Catholicism”? How do you say, um, dude, this Trinity thing — the virgin birth — transsubstantiation… ya know…

So you see how hard it is to explain that democracy is bunk.

Of course, I could be wrong. Who the heck am I? No one. And everyone who is someone agrees: democracy is wonderful.

So I’m not telling you that democracy is bunk. I’m just suggesting you might want to consider the possibility.

Or even just consider considering the possibility. The way you consider, like, UFOs, or something. Put it down in the “extremely improbable, but not inherently impossible” category.

48% of teenagers bought no CDs at all in 2007

Wednesday, February 27th, 2008

Wow. 48% of teenagers bought no CDs at all in 2007:

The illegal sharing of music online continued to soar in 2007, but there was one sign of hope that legal downloading was picking up steam. In the last year, Apple Inc.’s iTunes store, which sells only digital downloads, jumped ahead of Best Buy Co. to become the No. 2 U.S. music seller, trailing Wal-Mart Stores Inc.

It will be interesting to see if Amazon’s MP3 store can compete with iTunes.

The Truth About Autism: Scientists Reconsider What They Think They Know

Wednesday, February 27th, 2008

Frankly, it’s difficult to watch the first third of Amanda Bagg’s video about her autistic way of thinking. Then the computer-generated voice kicks in, speaking the words she has typed:

David Wolman discusses The Truth About Autism:

The YouTube post, she says, was a political statement, designed to call attention to people’s tendency to underestimate autistics. It wasn’t her first video post, but this one took off. “When the number of viewers began to climb, I got scared out of my mind,” Baggs says. As the hit count neared 100,000, her blog was flooded. At 200,000, scientists were inviting her to visit their labs. By 300,000, the TV people came calling, hearts warmed by the story of a young woman’s fiery spirit and the rare glimpse into what has long been regarded as the solitary imprisonment of the autistic mind. “I’ve said a million times that I’m not trapped in my own world,’” Baggs says. “Yet what do most of these news stories lead with? Saying exactly that.”

My Favorite Liar

Wednesday, February 27th, 2008

In My Favorite Liar, Kai Chang describes a technique used by one of his favorite professors:

One of my favorite professors in college was a self-confessed liar.

I guess that statement requires a bit of explanation.

The topic of Corporate Finance/Capital Markets is, even within the world of the Dismal Science, a exceptionally dry and boring subject matter, encumbered by complex mathematic models and obscure economic theory.

What made Dr. K memorable was a gimmick he employed that began with his introduction at the beginning of his first class:

“Now I know some of you have already heard of me, but for the benefit of those who are unfamiliar, let me explain how I teach. Between today until the class right before finals, it is my intention to work into each of my lectures … one lie. Your job, as students, among other things, is to try and catch me in the Lie of the Day.” And thus began our ten-week course.

This was an insidiously brilliant technique to focus our attention — by offering an open invitation for students to challenge his statements, he transmitted lessons that lasted far beyond the immediate subject matter and taught us to constantly checksum new statements and claims with what we already accept as fact. Early in the quarter, the Lie of the Day was usually obvious — immediately triggering a forest of raised hands to challenge the falsehood. Dr. K would smile, draw a line through that section of the board, and utter his trademark phrase “Very good! In fact, the opposite is true. Moving on … “

As the quarter progressed, the Lie of the Day became more subtle, and many ended up slipping past a majority of the students unnoticed until a particularly alert person stopped the lecture to flag the disinformation. Every once in a while, a lecture would end with nobody catching the lie which created its own unique classroom experience — in any other college lecture, end of the class hour prompts a swift rush of feet and zipping up of bookbags as students make a beeline for the door; on the days when nobody caught the lie, we all sat in silence, looking at each other as Dr. K, looking quite pleased with himself, said with a sly grin: “Ah ha! Each of you has one falsehood in your lecture notes. Discuss amongst yourselves what it might be, and I will tell you next Monday. That is all.” Those lectures forced us to puzzle things out, work out various angles in study groups so we could approach him with our theories the following week.

Brilliant … but what made Dr. K’s technique most insidiously evil and genius was, during the most technically difficult lecture of the entire quarter, there was no lie. At the end of the lecture in which he was not called on any lie, he offered the same challenge to work through the notes; on the following Monday, he fielded our theories for what the falsehood might be (and shooting them down “no, in fact that is true — look at [x]“) for almost ten minutes before he finally revealed: “Do you remember the first lecture — how I said that ‘every lecture has a lie?’”

Exhausted from having our best theories shot down, we nodded.

“Well — THAT was a lie. My previous lecture was completely on the level. But I am glad you reviewed your notes rigorously this weekend — a lot of it will be on the final. Moving on … ” Which prompted an rousing melange of exasperated groans and laughter from the classroom.

And while my knowledge of the Economics of Capital Markets has faded in time, the lessons that stayed with me was his real legacy; I’ve had many instructors before and since, but few that I remember with as much fondness — and why my favorite professor was a chronic liar.

Intellectual Property Conference

Tuesday, February 26th, 2008

I recently attended a conference on intellectual property — you know, patents, copyrights, trademarks, and trade secrets — and I learned two things: (1) I know much more about intellectual property than I realized, and (2) you can’t skim a conference. I’ve been spoiled by the blogosphere.

On a less meta level, I did learn a few things — or at least noted a few interesting quips worth repeating.

It’s interesting to note that the U.S. doesn’t simply have fairly mature IP law; IP made its way into our Constitution. Here’s the factoid I did not know: within 13 years, the U.S., with its low patent fees, had more patents than Great Britain, the home of the Industrial Revolution — and a much older country.

Through most of the 19th century, innovation was the work of independent inventors. Bell, for instance, originally outsourced its R&D. It’s only after a number of legal rulings enforced employment agreements that transferred IP to employers that companies like Bell brought their R&D in house and formed groups like Bell Labs.

Today, there is talk about outsourcing innovation again, via markets like Eli Lily’s Innocentive.

At any rate, a lot of patents, especially software patents, are defensive. On the one hand, if you’re Microsoft, you can afford to spend $100 million per year in legal fees over alleged IP infringement. On the other hand, if you’re Microsoft, you have to spend $100 million per year in legal fees over alleged IP infringement — because you’re a big, fat target. The balance of terror depends on both your resources to bring to the legal battle and the resources other people can extract from you.

One recent issue in IP is the appearance of patent trolls — firms that pop up with a dormant patent they’ve bought off the inventor and use to blackmail a big, successful firm that thinks it has been operating in the clear. This sounds awful and evil until you realize that the original inventor didn’t stop the big, successful firm, because the original inventor wasn’t a lawyer; he was an inventor.

Patent trolls give big, successful firms a much bigger incentive to play nicely with the little guys, because the little guys can sell their IP to someone who knows how to use it — either in the marketplace or in a court of law.

Also, it’s easy for a big company to complain about patent trolls when it has brought all its own trolls in house.

Anyway, these “trolls” seem to favor the Eastern District of Texas — where all the asbestos lawyers had great success a couple legal fads ago.

For most large companies then, the focus on IP is on legalities, and IP’s largely seen as a negative right — the right to sue someone and get an injunction against their using that IP. IP assets are “owned” by lawyers, and no one argues with simply holding IP “just in case” it becomes valuable later. The rewards for successfully licensing IP are dwarfed by the painful consequences for… successfully licensing IP. “Why did we license that technology away?”

So, at this point, IP is far from liquid and far from fungible, but a number of folks would like to see IP licenses move toward some kind of standardized but parameterized form, like the better-known financial options. As one fellow pointed out, oil-derivative transaction volume dwarfs gasoline sales. The same could happen with IP. “Wall Street will set you free.”

Free! Why $0.00 Is the Future of Business

Tuesday, February 26th, 2008

In Free! Why $0.00 Is the Future of Business, Chris Anderson gives a decent run-down of how and why so many more products are going to be nominally free to consumers — but as editor-in-chief of Wired, I don’t think he had his article edited for content:

Milton Friedman himself reminded us time and time again that “there’s no such thing as a free lunch.

“But Friedman was wrong in two ways. First, a free lunch doesn’t necessarily mean the food is being given away or that you’ll pay for it later — it could just mean someone else is picking up the tab. Second, in the digital realm, as we’ve seen, the main feedstocks of the information economy — storage, processing power, and bandwidth — are getting cheaper by the day. Two of the main scarcity functions of traditional economics — the marginal costs of manufacturing and distribution — are rushing headlong to zip. It’s as if the restaurant suddenly didn’t have to pay any food or labor costs for that lunch.

Surely economics has something to say about that?

It does. The word is externalities, a concept that holds that money is not the only scarcity in the world. Chief among the others are your time and respect, two factors that we’ve always known about but have only recently been able to measure properly. The “attention economy” and “reputation economy” are too fuzzy to merit an academic department, but there’s something real at the heart of both. Thanks to Google, we now have a handy way to convert from reputation (PageRank) to attention (traffic) to money (ads). Anything you can consistently convert to cash is a form of currency itself, and Google plays the role of central banker for these new economies.

When Milton Friedman said that there’s no such thing as a free lunch, the whole point was that someone pays; it’s just not obvious who — and how. Obviously you can charge $0.00 for something.

Regarding externalities: “You keep using that word. I do not think it means what you think it means.” Externalities are not non-pecuniary costs or benefits; they are costs or benefits to parties outside a transaction. If your neighbor turns his garage into a nightclub and plays deafening house music all night long, he might be happy, and his paying customers might be happy, but you might be miserable. That’s the externality.

Have Some Phun With Your Kids

Tuesday, February 26th, 2008

Have Some Phun With Your Kids, geek dad Ken Denmead suggests, with this “work of pure genius”:

Download it and get to the phun

The case of the 500-mile email

Tuesday, February 26th, 2008

The case of the 500-mile email makes for a great story:

I was working in a job running the campus email system some years ago when I got a call from the chairman of the statistics department.

“We’re having a problem sending email out of the department.”

“What’s the problem?” I asked.

“We can’t send mail more than 500 miles,” the chairman explained.

I choked on my latte. “Come again?”

“We can’t send mail farther than 500 miles from here,” he repeated. “A little bit more, actually. Call it 520 miles. But no farther.”

“Um… Email really doesn’t work that way, generally,” I said, trying to keep panic out of my voice. One doesn’t display panic when speaking to a department chairman, even of a relatively impoverished department like statistics. “What makes you think you can’t send mail more than 500 miles?”

“It’s not what I think,” the chairman replied testily. “You see, when we first noticed this happening, a few days ago — “

“You waited a few DAYS?” I interrupted, a tremor tinging my voice. “And you couldn’t send email this whole time?”

“We could send email. Just not more than — “

” — 500 miles, yes,” I finished for him, “I got that. But why didn’t you call earlier?”

“Well, we hadn’t collected enough data to be sure of what was going on until just now.” Right. This is the chairman of statistics. “Anyway, I asked one of the geostatisticians to look into it — “

“Geostatisticians…”

” — yes, and she’s produced a map showing the radius within which we can send email to be slightly more than 500 miles. There are a number of destinations within that radius that we can’t reach, either, or reach sporadically, but we can never email farther than this radius.”

“I see,” I said, and put my head in my hands. “When did this start? A few days ago, you said, but did anything change in your systems at that time?”

“Well, the consultant came in and patched our server and rebooted it. But I called him, and he said he didn’t touch the mail system.”

“Okay, let me take a look, and I’ll call you back,” I said, scarcely believing that I was playing along. It wasn’t April Fool’s Day. I tried to remember if someone owed me a practical joke.

I logged into their department’s server, and sent a few test mails. This was in the Research Triangle of North Carolina, and a test mail to my own account was delivered without a hitch. Ditto for one sent to Richmond, and Atlanta, and Washington. Another to Princeton (400 miles) worked.

But then I tried to send an email to Memphis (600 miles). It failed. Boston, failed. Detroit, failed. I got out my address book and started trying to narrow this down. New York (420 miles) worked, but Providence (580 miles) failed.

I was beginning to wonder if I had lost my sanity. I tried emailing a friend who lived in North Carolina, but whose ISP was in Seattle. Thankfully, it failed. If the problem had had to do with the geography of the human recipient and not his mail server, I think I would have broken down in tears.

Having established that — unbelievably — the problem as reported was true, and repeatable, I took a look at the sendmail.cf file. It looked fairly normal. In fact, it looked familiar.

I diffed it against the sendmail.cf in my home directory. It hadn’t been altered — it was a sendmail.cf I had written. And I was fairly certain I hadn’t enabled the “FAIL_MAIL_OVER_500_MILES” option. At a loss, I telnetted into the SMTP port. The server happily responded with a SunOS sendmail banner.

Wait a minute… a SunOS sendmail banner? At the time, Sun was still shipping Sendmail 5 with its operating system, even though Sendmail 8 was fairly mature. Being a good system administrator, I had standardized on Sendmail 8. And also being a good system administrator, I had written a sendmail.cf that used the nice long self-documenting option and variable names available in Sendmail 8 rather than the cryptic punctuation-mark codes that had been used in Sendmail 5.

The pieces fell into place, all at once, and I again choked on the dregs of my now-cold latte. When the consultant had “patched the server,” he had apparently upgraded the version of SunOS, and in so doing downgraded Sendmail. The upgrade helpfully left the sendmail.cf alone, even though it was now the wrong version.

It so happens that Sendmail 5 — at least, the version that Sun shipped, which had some tweaks — could deal with the Sendmail 8 sendmail.cf, as most of the rules had at that point remained unaltered. But the new long configuration options — those it saw as junk, and skipped. And the sendmail binary had no defaults compiled in for most of these, so, finding no suitable settings in the sendmail.cf file, they were set to zero.

One of the settings that was set to zero was the timeout to connect to the remote SMTP server. Some experimentation established that on this particular machine with its typical load, a zero timeout would abort a connect call in slightly over three milliseconds.

An odd feature of our campus network at the time was that it was 100% switched. An outgoing packet wouldn’t incur a router delay until hitting the POP and reaching a router on the far side. So time to connect to a lightly-loaded remote host on a nearby network would actually largely be governed by the speed of light distance to the destination rather than by incidental router delays.

Feeling slightly giddy, I typed into my shell:

$ units
1311 units, 63 prefixes

You have: 3 millilightseconds
You want: miles
* 558.84719
/ 0.0017893979

“500 miles, or a little bit more.”

The Brash Programmers at 37signals Will Tell You: Keep It Simple, Stupid

Tuesday, February 26th, 2008

The Brash Programmers at 37signals Will Tell You: Keep It Simple, Stupid — and reap the rewards:

When he released Basecamp in February 2004, Fried expected the monthly subscription fees, which today range from $12 to $149, to generate sales of $5,000 a month by the end of Basecamp’s first year; they reached that target in six weeks. Five months later, Hansson packaged his Ruby shortcuts and released them as Ruby on Rails, which started winning converts almost immediately.
[...]
Rails has continued its run of popularity; over the years, tens of thousands of programmers have used it to create countless online applications, including podcasting service Odeo and microblogging phenomenon Twitter. And Basecamp, 37signals’ Rails-powered, easy-to-use online collaboration software, boasts more than 2 million account holders. Signal vs. Noise, the 37signals blog, pulls in 75,000 readers a day.

Gun Buyback Misfires

Tuesday, February 26th, 2008

Oakland’s Gun Buyback Misfires in a completely predictable way:

Oakland’s recent gun buyback was especially ridiculous. The police offered up to $250 for a gun “no questions asked, no ID required.” The first people in line? Two gun dealers from Reno with 60 cheap handguns. Fortunately the buyback did manage to get some guns off the street, too bad they were turned in by a bunch of senior citizens from an assisted living facility. Whew, the streets are safe at last.

Alex Tabarrok explains the problem:

Imagine that instead of guns, the Oakland police decided, for whatever strange reason, to buy back sneakers. The idea of a gun buyback is to reduce the supply of guns in Oakland. Do you think that a sneaker buyback program would reduce the number of people wearing sneakers in Oakland? Of course not.

All that would happen is that people would reach into the back of their closet and sell the police a bunch of old, tired, stinky sneakers.

It gets worse:

Imagine that gun dealers offered a guarantee with every gun: Whenever this gun gets old and wears down, the dealer will buy back the gun for $250.

The dealer’s guarantee makes guns more valuable, so people will buy more guns.

But the story is exactly the same when it’s the police offering the guarantee. If buyers know that they can sell their old guns in a buyback, they are more likely to buy new guns.

Abusing Amazon images

Tuesday, February 26th, 2008

Nat Gertler has a fascinating piece on Abusing Amazon images — by reverse-engineering Amazon’s image-generator:

Amazon.com feeds out a lot of product images, putting out the same book cover (say) in a variety of sizes and formats. By experimentation, I found that they don’t actually have all the sizes and formats stored. Instead, they have a system that generates each requested image. The details of size and format are built into the image’s URL. What that means that, if you want, you can create URLs that generate odd and unlikely Amazon images (you can see my gallery of images here). The proper combination of product choice and added elements and effects could create an interesting visual. What you see here is my best understanding of things based on trial and error and messing with various example URLs I’ve found.

Democracy is a commons, not a market

Tuesday, February 26th, 2008

Before Mencius hijacked my brain, I read another, softer anti-democratic piece, Bryan Caplan’s The Myth of the Rational Voter: Why Democracies Choose Bad Policies. I’ve discussed it before, but in an election year, it all bears repeating.

And there’s the fact that I finally got around to reading the book, not just a good summary or two.

Caplan’s ideas are fairly straightforward, if controversial. The key is that he has gone to the data — the Survey of Americans and Economists on the Economy. He isn’t just rhapsodizing about political philosophy from his lofty perch.

Despite all the jokes about economists, they really do agree on a number of issues — and ordinary folks do not “get” this economic point of view. An important, but subtle, point is that ordinary folks don’t simply make random “mistakes” from an economic perspective. Those would cancel each other out in the aggregate. They make systematic mistakes with systematic biases — against markets, against foreigners, toward making work rather than improving efficiency, and toward pessimism rather than optimism.

The other key point is that citizens have very little incentive to educate themselves on the issues and to improve their understanding of the implications of the economic policies they’re voting for or against, because “Democracy is a commons, not a market.”

Your individual vote has a negligible effect on an election. Consider it zero. But your vote does not have a negligible effect on how you feel about yourself. So we all have an incentive to vote for things that feel good, even if they won’t work. We can be rationally irrational.

If a policy is bad for the economy, it’s bad for the economy, and if it’s good for the economy, it’s good for the economy, but your one vote, one way or the other, doesn’t change whether that policy get implemented or not. You don’t individually have the power to choose policy for the nation. But you do have the power to vote your heart.

Is that so bad? Think of it this way: since you don’t actually choose policy, you don’t have to put your money where your mouth is. And people have all kinds of strong opinions that disappear as soon as you ask, “Wanna bet?”