Daniel L: I could not agree with you more. I think you’re absolutely spot on. As a researcher, the more interesting problem is predicting the unexpected, finding the uncommon, discovering the atypical.

However, I think most of the world disagrees with you (us). Web search engines make lots and lots of money by basically serving people the common, mundane, expected results (and advertisements), but doing it in a way that allows the user to remain lazy.

Giving lazy people the expected appears to be more lucrative than giving active information seekers the unexpected.

My experience is that it is fairly easy to be right with your predictions 50% of the time. Or even 80% of the time. Sometimes using rather crude algorithms.

But the devil is in the remaining 50% or 20%.

Maybe you are a publisher. You are pretty good at predicting booksellers. Some guy comes in with a crazy novel “Dune”. It is nonsensical to you. You reject it. In fact, 40 different publishers reject it. Turns out that the novel in question becomes one of the two most important scifi novel of all times.

My point is that if all you are predicting is the “predictible” stuff, there might not be much value in your work. Basically, you must be able to predict the unexpected.

That’s how Physics made its bread and butter. Setup a crazy experiment to measure the speed of light in different. Nobody has any idea what will come out of it, but some crazy guy called Einstein predicts that the speed of light will always come out to the same value. His prediction made him famous (among other things).

Predicting that I will need milk next week, on the other hand, has very little value.

I mean, we have been there in the late nineties with the “data mining” fad. All companies were doing association rules to find out “hidden” relations between products and services. Most of this work was quickly discarded since it did not bring any actual value to the business.

(Some of it did, when Gred Linden got Amazon to add the item-to-item recommender system, but that’s another unexpected story. Managers back then predicted that it would not be useful to the company!)

If they know that 50% of queries are predictable in a 12 month ahead forecast then doesn’t this mean that they can organize their infrastructure better to satisfy these ’same’ queries?

Doesn’t it demonstrate that people (in large numbers) show similar behavior and interests (health, food and drink, travel) which helps in the targeting of ads?

Increases, decreases
Decreases, Increases
What the hell do we care
What entropy does?

It depends on whether you are more a follower of Parmenides, or Heraclitus. Right?

Isn’t that a tautology? But your point is well taken–there’s little value of investing in forecasting when we know that the world is changing quickly.

The value of recommender systems–that’s a whole other story, but I think we’ve beaten that one to death a few times.

Think about making predictions about behavior in the middle ages (in Europe). Certainly, it was not very difficult! There was the odd, unexpected local war or drought… but mostly, everything was routine.

Anyhow, I have worked a bit on recommender systems before it became such a big deal… and I eventually gave up for several reasons, and one of them was my own inability to predict my own behavior, even based on very reliable data. I would study for hours the reviews of a given movie, and find that I could not predict at all whether I would enjoy it (even obviously bad movies, could generate enjoyment…)

I don’t think I can predict what I’ll be searching for in a week, let alone in a year.

Maybe I am an odd fellow. Maybe.