Last Word on the Fires

Although fires are still burning, most of the nearby residential areas have been repopulated, Governor Schwarzenegger and President Bush have made their obligatory appearances, and the weather has returned to near normal. Except for some minor wind damage and a pool full of ash we were unaffected. But we do know some folks who were affected significantly, including my wife's boss who lost her house. Some observations:

• The traditional media are amazingly poor conduits of information. Despite around-the-clock coverage from all the of the local channels, I really had no idea what was going on in my community. A single Google map application was vastly more informative.
• TV and radio coverage are less about information than about commiseration.
• If i hear the words "perfect storm" again, i'll wretch.
• Whether liberal or conservative, people should keep their political views with respect to the fires under wraps until people that matter are out of harm's way.
• I got messages regarding the fires from Germany, Switzerland, and Brazil.
• It's getting harder to get by without the Internet.

Latest on the Fires

We returned to our house today, although our neighborhood is still technically among the evacuated areas of our community. Quite a few houses in our town (Poway, CA) were destroyed. The closest are about a mile northeast of our house, but the fires directly to the east made me more nervous. As of Monday morning, i was reasonably certain that the fires would spread to our neighborhood, but amazingly they did not cross the major street that borders our neighborhood on both the north and east after a southward turn.

The air is still filled with smoke, but the winds have diminished and the skies are partially clear now. Schools are closed for the whole week, and both my office and Emily's are officially closed (my office borders one of the worst-hit areas of fire damage). Fortunately, our electrical power has been restored, so we're able to live more or less normally except that we can't really spend much time outside. It sounds as if the weather will be more conducive to fighting the fires tomorrow, but it's likely that there will be areas burning and people evacuated into the weekend.

California Hates You Too, Glenn Beck

I've always thought Glenn Beck is a smug, whiny idiot whose secret ambition is to be Robin to Rush Limbaugh's neocon Batman. Fortunately for me, he saved me the need to prove it to anyone who might be skeptical.

San Diego County on Fire

This is what the sun looked like at about 4 pm this afternoon from my back yard.


There are currently two fairly large wild fires burning in San Diego's east county. The one responsible for this smoke is east of Ramona along the Highway 78. High Santa Ana winds are making the fires essentially unstoppable, and the fire departments are taking the approach of trying to evacuate the communities most likely in its path. Not sure if it'll get this far west like the big fires back in 2003, but the winds are supposed to continue for the next couple of days.

Reid vs. Google

I was lucky enough to start my career in computer software at a research facility that one of my colleagues described as a "halfway house to the real world". There was a broad cross section of humanity, from recent college grads to world-renowned research scientists. There was one guy who was nearing retirement. Although he was a smart guy, a former nuclear engineer, his skills had atrophied and it was clear that he was just trying to hold on until he could collect a pension.

At the time, i thought it was a little sad but i didn't really imagine myself working as an engineer into my 60s. What i didn't anticipate was that the "old guy" in the IT organization would someday be the 40-something. It's hard to say exactly what the truth is behind this lawsuit, brought against Google by computer scientist Brian Reid, but the implication is clear: experience is not as valuable as the willingness to put the company before your personal interests.

Places like Google, and Microsoft in an earlier era, are unique in the sense that the rewards that can accrue to people with a good idea and no external distractions are very significant. But it does make you wonder if there's room at the elite levels of software development for people with lives. On one hand, i believe that any corporate environment should be a meritocracy. On the other hand, i feel that experience, even if it's only in a non-technology-specific sense should be valued. As the resident old guy in an Internet organization for the last few years, I've experienced the intense frustration of the young and ambitious needlessly repeating the mistakes that we made twenty years ago. It's not simply that experience in software development is undervalued, it's that many software organizations, especially in the Internet space, are culturally conditioned to believe that there situation is novel and not subject to the rules of earlier eras.

You would not believe the insanity i've seen propagated as a result of this. I've seen younger programmers go through the pain of rediscovering what many of us have known about optimization and performance for years. I've seen virtually unbounded arrogance with respect to estimates of how long projects will take, and the inevitable assignment of blame to the naysayers when the death march reaches its logical conclusion. I've seen the rediscovery of ancient technologies given fancy new names. What i have not seen is productivity.

Google's a really cool place with a lot of interesting technology, and i hope this is an isolated case that doesn't really reflect their values. If it does, then they'll eventually end up like Microsoft-- painted into a corner by their inability to learn from their own mistakes.

Back To Work

It was a strange week for a number of reasons, but mostly because i started a new job at Slacker, Inc., a relatively new company in the digital music space that's trying basically to bring personalized music to everywhere. I like the model, which is based on the idea of personalized radio, and also the fact that there's a device component. It's interesting to work next to people with oscilloscopes on their desks.

Gebrselassie Sets Marathon Record

One of the most amazing athletic feats of all time occurred yesterday and you probably didn't even hear about it. Haile Gebrselassie, possibly the greatest distance runner of all time set the world record in the marathon, running 2:04:26. That's nothing short of super-human.

Bayesian Statistics

As i've moved from one domain to another during my computer software career, there are certain ideas that seem to follow along. The calculation of eigenvalues and eigenvectors pops up in many different places, as does singular value decomposition. Markov chains have shown up in everything from statistical mechanics to language analysis. I've used K-means clustering in drug discovery and for music personalization.

Another idea that seems to be universally applied these days is Bayesian statistics. When i first encountered the idea a decade or so ago, it was a fairly obscure concept known primarily to statistics geeks, but Paul Graham gave the Bayesian approach a boost when he suggested using it for spam filtering.

Conceptually, the idea of Bayesian statistics is fairly easy to comprehend: evidence (or belief) should factor into probabilities. An oft-cited example is the probability that the sun will rise tomorrow. Normal statistics doesn't say much about that probability, but the Bayesian approach contends that its highly likely because the sun has risen reliably for quite some time and so there's reason to believe that it will tomorrow. In practice, i don't find this example to be very useful. Another common example that i like better is this: suppose you have to determine the probability that somebody has cancer. With no further evidence of the person's habits, you could guess the probability by simply determining the percentage of people who have cancer among the whole population. But if you know that the person is a smoker, you might expect this probability to be higher.

So Bayesian probabilities are essentially conditional probabilities (the probability of event A given event B, usually written P(A/B). The so-called Bayes formula expresses this conditional probability like this:

P(A/B) = P(B/A) * P(A)/P(B)

This says the probability of event A given that event B has happened is equal to the probability of event B given that A has happened times the probability of event A divided by the marginal probability of event B. Which is completely non-intuitive to me.

Another way of thinking about this is that the probability of event A given event B is different that the probability of event A without the condition of event B by the factor P(B/A)/P(B). So for example, the probability that somebody has cancer given that they're a smoker, P(cancer/smoker), is different than the probability that they have cancer, P(cancer), without the evidence of smoking by the factor P(smoker/cancer)/P(smoker). If the latter factor is >> 1 then the idea that smokers are more likely to have cancer is supported.

This is still not particularly intuitive to me, but let's look at it a bit. Take the extreme case that P(smoker) = 1, that is, everybody smokes. Then this factor can at most be 1.0, so the knowledge that somebody is a smoker is not helpful. Similarly, if you were to poll a population of cancer patients and find that they were all smokers, then the numerator of this factor is 1. If P(smoker) is less than 1, then this factor tells us that being a smoker definitely influences the probability of cancer. In Bayesian terms the probability P(cancer/smoker), which is called the posterior probability, is greater than P(cancer), which is called the prior probability.

At a glance, it doesn't seem like the above equation does much for you, since the thing you're trying to find, P(A/B), has to be calculated in terms of an equally complex term, P(B/A). The trick is that P(B/A) is often easier to determine from known data. Put in terms of scientific method, it's often easier to determine the probability of the evidence given the truth of the hypothesis than to determine the probability of the hypothesis given the evidence.

An example from recommendation technology would be something like this: suppose that you have a large collection of ratings for both albums and artists, with values of 1 to 4. You want to determine the probability that a given user would give an artist S a 4 rating, P(S=4). Without any other information, you could estimate this probability by looking at the percentage of 4 ratings for that artist among the entire set of ratings.

However, suppose that the user has not rated the artist, but has rated one of the artist's albums. The value of that album rating can be considered evidence of the user's opinion about the artist. Let's say the rating for the album is M, then we can look at P(S=4/M) . Bayes formula gives us

P(S=4/M) = P(M/S=4) * P(S=4)/P(M)

Here, the P(M) term is more complicated than it looks. I glossed over the fact above that the denominator of this formula is a marginal probability. To understand the origin and concept of a marginal probability, it helps to know the relationship between conditional and joint probabilities. A joint probability is the probability of two events occurring together, usually written as P(A,B). Joint probability can be written in terms of conditional probability

P(A,B) = P(B/A) * P(A)

where P(A) is a marginal probability, meaning that it's the probability of the event A occurring regardless of event B. The reason it's called marginal is because if you consider a table listing the probabilities of all the possible joint events (assuming A and B take on discrete values), then if you sum up the joint probabilities across one row (meaning the total probability for a given value of A regardless of the value of B), then you'd write the sum in the margin. Get it? The marginal probability for P(M) can be written like this:

P(S,M) = P(1,M) + P(2,M) + P(3,M) + P(4,M)

or

P(M/S=1)*P(S=1) + P(M/S=2)*P(S=2) + P(M/S=3)*P(S=3) + P(M/S=4)*P(S=4)

The formula now is composed of terms that we can derive from the ratings data. Given an actual value for the rating M for a particular user, we could then predict the mostly likely artist rating for that user.

This is a contrived example, and i wouldn't really recommend it as a method for doing music recommendations, but i hope it does illustrate how this concept can be applied in various domains. There are far more interesting applications of Bayesian statistics in machine learning, the physical sciences, and even search and rescue.

Surprise in Recommendations

There's a principle in user interface design called "The Principle of Least Surprise" (or astonishment), which states that any action with a potentially ambiguous interpretation should result in the the least surprising consequence. Often the results of recommender systems are interpreted similarly: people evaluating the recommendations deem a recommendation to be good if it is expected. Once, long ago, i wrote here about how this doesn't seem to provide a good experience with music recommendations, because if a recommendation is not surprising, it is probably already familiar.

I've since decided that a bigger problem with music recommendations is that even great recommendations can't be appreciated unless you listen to the music (again, if you already know the recommended music, then it's not a compelling recommendation unless it's in the context of something like a personalized radio stream). However, i still think about this problem occasionally. It still seems to me that the interesting recommendations are those that you don't expect, but which still match your taste (if you eliminate the latter restriction, it's easy to make unexpected recommendations).

The main reason why unexpected recommendations are rare is (i think) because most recommendation systems are based on some measure of similarity between either items or users (this is the idea behind so-called collaborative filtering systems and content-based systems obviously seek to find item similarity). Often the set of recommended items will be chosen by comparison with similar items or by comparison with the tastes of similar users. So suppose that you like The Shins and the system discovers that other people who like the Shins often also like the The Decemberists. The latter is a good recommendation by most standards (including my personal subjective standards). But it is not a surprising recommendation.

Surprising in this context does not mean obscure like say Neutral Milk Hotel, or outrageous like say Iron Maiden. For me a surprising recommendation would be something that maybe takes a detour into a different, but adjacent genre. For example, i've been listening to a lot of Richard Thompson recently, and i'd be surprised but pleased if there were a connection from The Shins to Thompson via the intersection of indie rock with alt. country and alt. country with folk. It's unlikely that many systems would make that connection because the two artists don't have a significant shared fan base (though i'm sure it's larger than just me).

A mathematical model of surprise was developed a while back, but it treats surprise more in the sense of jumping-out-of-the-bushes rather than one-of-these-things-is-not-like-the-other. The idea of surprise that i have in mind is more like Ted Dunning's use of the word in his paper Accurate Methods for the Statistics of Surprise and Coincidence, in which surprise is more of a rare but significant co-occurence.

The reason why even the latter approach doesn't often produce the sorts of surprises that i want is that the information isn't in the data. For example, the connection that i claim between The Shins and Richard Thompson above is subjective. In the general population of music listeners there simply isn't enough data to establish a connection between the two artists. It seems like what is needed is something that can infer or enhance these unusual connections from a single person's listening habits. Or, perhaps, this connection could be found with content-based recommendation systems or some future refinement thereof.

I suppose that if you imagined a giant network that connects every musical artist with closest neighbors and so on until every pair of artists (A,B) are connected by some path, then you could over time weight the connections based on an individual user so that certain paths become "shorter". It might even be possible to structure this as a Bayesian network, where information about a user's artist preferences is used as evidence to connect previously unrelated artists for that individual. But, honestly, i'm not sure how you'd scale this across millions of users.

Man Sues God

This article about Nebraska State Senator Ernie Chambers looks like an Onion article, but apparently is not.