Sticking to first name only just in case. Do you guys hire people who work remotely but with a willingness to make a pigrimage to Mountain View on a regular basis? Or maybe I should ask this, are all your team based there?

Cheers,

Nic.

Full disclosure: I lead the Product Analytics data science team at LinkedIn. I can’t speak for Twitter, but I can tell you that my team is as badass as they get. You can look at my bio (MIT -> CMU -> founding team at Endeca -> Google -> LinkedIn) to a…

http://www.radiolab.org/2010/sep/20/

This is similar to a rule we know about in politics, about how if you control the discourse (terminology, definitions) then you control the policy/outcome. Think about the abortion debates. Where you stand on those debates has everything to do with how you’ve chosen the objective function: life vs. choice. The left doesn’t define itself as “anti-life”, because life isn’t their objective function — choice is. And the right doesn’t define itself as “anti-choice”, because choice isn’t their objective function — life is. We could deep end into what is or isn’t life.. whose choice is at stake, blah blah blah. I don’t want to do that – that’s not my point. My point is to show how you can control the ultimate workings of a system, the ultimate policies (aka algorithms) that get implemented by how you choose your objective function. It doesn’t matter how much you measure condition A against condition B, if the objective function has been specifically designed to favor condition B.

I think you see what I’m saying, when you write, “in that case you can compare the old and new approaches using the new objective function. Of course that should favor the new approach”. But I see it more strongly than that. I see it as the new objective function not just favoring the latter, but essentially being capable of pre-selecting the latter. As in a rigged game.

So if the HiPPO can still rig the game, by being the person who chooses the objective function, what does it even matter if the HiPPO can’t select algorithm A or algorithm B directly? The choice of objective function is essentially the same thing. Natch?

Seeing how this is not a commercial blog, I’m not interested in affiliate programs. But thanks for the offer. And you’re certainly welcome to promote my blog in your newsletter if you think it would benefit your readers.

I sometimes write about events here myself, but mostly when I am participating in them personally and have something to contribute to the discussion.

I look forward to hearing back.
-Adam

Adam R. Kahn
Chief Operating Officer, Rising Media, Inc.
O: 508.644.0639 | akahn@risingmedia.com
@Adam_Kahn

I was trying to make a slightly deeper point than that, which is to say that two different versions of one’s algorithm/interface/whatever might not be unify-able under any single OEC. It’s not just a matter of not being able to pick the right one. It’s that each of your two conditions A and B require a different “right one”.

Jeremy, so good to have you back! And your point is well-taken — testing assumes an objective function, and I think your main point is that you can’t test whether you have chosen the right objective function.

The typical problem with objective functions is that they are models — attempts to achieve the right trade-off between realism and the simplicity that enables analysis. We can use data for validation, e.g., if we think that CTR or speed is a proxy for user happiness, then we can collect independent data on user happiness (e.g., through surveys) and compute the correlations. So we can evaluate objective functions to some extent.

But there are limits. Science won’t tell us which problems to solve. Indeed, one of my main points is that science is about choosing which problems to solve. That’s why we have scientists!

And, if we get paid enough, maybe we even get to be HiPPOs.

As for emergent semantics, I mean something more along the lines of the discussion over the “unreasonable effectiveness of data“.

So don’t you still need a HiPPO, or at least one voice in the room that shouts louder than all the rest, to come to an OEC agreement?

In the most general sense, I don’t disagree with this post. Broadly, what you’re saying is sage and should be followed.

My main problem with the whole A/B testing approach, where I continue to doubt its efficacy, is when you’re trying to “leap” a system, rather than “evolve” it.

And by “leap”, I don’t just mean large changes to the UI, UX, or algorithm. I understand that A/B testing can handle all *sizes* of changes, both small and large.

What I am talking about is when the change is qualitatively different, not just quantitatively different. That kind of a leap. For these sorts of changes, I remain unconvinced that A/B testing is capable of meeting the challenge. And that is because in order to make some of these leaps, it is not enough just to change the UI, UX, or algorithm. No, what has to be changed is the OEC (or KPI or whatever you call it). What has to be changed is the metric itself.

So what you have is a condition A that is best measured by (let’s call it) OEC_a. And condition B that is best measured by OEC_b. And because the metrics themselves are different, because the system itself has taken a qualitative leap, A/B testing is scientifically impossible to perform. A/B testing, science itself, requires sameness of metric across all conditions.

In those cases, those leaps, it seems to me that you still need a strong “Jobsian” (or Lindenian? Tunkelangian?) voice, to say that the metric itself needs to change. That we’re not just using a new algorithm or new UI with an old metric. That we need to have a new metric, too.

So that’s my main problem with this whole thing. Like said, 98.3% of the time it works just fine. But 98.3% of the time you’re doing “normal science” are you not? (http://en.wikipedia.org/wiki/Normal_science) When you need to step outside of normal science, when you need a Kuhnian “paradigm shift”, then your A/B testing also falls apart. The creation of an OEC is essentially the same thing as the establishment of a paradigm. A/B testing does not allow you to do paradigm shifts (http://en.wikipedia.org/wiki/Paradigm_shift).

For a good example of this, I point you to a blog post by this nice fellow, on the difference between “set retrieval” and “ranked retrieval”:

http://thenoisychannel.com/2008/08/24/set-retrieval-vs-ranked-retrieval/

To me, in order to go from ranked retrieval to set retrieval, you can’t just A/B test your way into it. You need to change the OEC itself. Because the OEC for ranked retrieval is different than the one for set retrieval. A/B testing cannot, therefore, help you make that leap.

A little violence here and that ends in holistic agreement is well worth it. Now on to ruling the world…

As for your desire to get your hands on our data, I will note that two of our data scientists used to work at Juice. No pressure.

Of course, that’s also an argument in favor of open data as a public good. But a lot of data won’t (and shouldn’t!) be open any time soon, whether it’s web search logs, social network activity data, etc. So people like me will continue to follow the data.

Never hurts to listen to the guru of motivation, Dan Pink:

http://www.ted.com/talks/dan_pink_on_motivation.html

And yes, planning to be at RecSys. I hear there’s going to be a great Industry Day.

BTW – I’m hoping to make it to RecSys 2012 in Dublin. Are you going to be there?

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC240752/

http://www.mhs.manchester.ac.uk/research/publications/30093.pdf

http://www.cnbc.cmu.edu/~behrmann/dlpapers/HumphreysAvidanBehrmann2006_EBR.pdf

this if should be added with else as mention below, as to show words fetched before non-dictionary word ::

if (segSuffix != null) {
memoized.put(input, prefix + ” ” + segSuffix);
return prefix + ” ” + segSuffix;
}else{
return prefix ;
}

Also see my more recent post: http://thenoisychannel.com/2010/02/28/you-cant-hurry-relevance/

The literature is scattered with these sorts of things. Though scattered is the correct term, not centralized.

The first thing that pops to mind is this study:

http://irgupf.com/2009/03/19/long-term-versus-evolutionary-thinking-part-2-of-2/

In it, the Google process is compared against Diane’s process. The Google process is “use a small query box, stay simple, and use a few words as possible to describe your need”. Diane’s process is “use a bigger box, and let the users use as many words as needed. Don’t encourage them to use a “stay small” process.

And Diane’s process won.

Can we bring more concrete evidence to bear? Examples or study designs to test the hypothesis that process is as or more essential than data and algorithms?

But honestly, I almost if not never hear folks talk about process. The dichotomy is always between data and algorithm. It’s never about the fluid, oft capricious, infinitely more important of how and when and especially why various pieces of data are wed to various scraps of algorithm.

In fact, I remember having a discussion with you a year or two ago, when we were debating the core of what HCIR meant. I didn’t see HCIR only as faceted search, or only as visual search, or only as just a way of allowing humans more decision-making capability. Rather, I see HCIR as a system that allows humans input in deciding when and how and where various pieces of data should be fed to various algorithms, and when and how and where various algorithms should be wrapped around various pieces of data.

Most information systems, web search systems especially, feed data to algorithms, or wrap algorithms around data. But they embody a completely rigid and inflexible process. The user is given no way of altering the ultimate procedural goals of the system. Those are set by the system designers. And even if those goals are set in a data-driven way, once that historical data has spoken, you the current system user have no way of disagreeing and altering the process.

I want that process to be alterable not by experts such as you or me, but by the users of the system themselves. That’s HCIR. HCIR is about allowing users to alter the data, or to alter the algorithm, but instead to alter the relationship between the data and the algorithm. I.e the process.

we’ll have a separate post to analyze your Manichean perspective on better data vs better interaction design

Pfft.. Manichean. I see your smiley face, but pfft, nevertheless!

But seriously, the duality is usually between more data vs. better algorithms. All I’m saying is that it is a triumvirate: data, algorithms, and process. The last part, the process, is usually ignored. But it is what allows you to connect data to algorithms. It’s the glue that holds it all together. Bad glue = bad outcome.

And yes, Jeremy, we’ll have a separate post to analyze your Manichean perspective on better data vs better interaction design .

I’d like to insert a placeholder here for a big data argument.. algorithms vs. data vs. better data vs. better interaction design.

http://www.flickr.com/photos/10916594@N00/sets/72157629304618287/

I love snacks. But I understand they don’t always make for the healthiest diet.

I’d be interested in hearing arguments as to why short contributions imply lower quality.

The iPad/iPhone with its tiny applications as shown tat there can’t be much value in a small package.

Aditi, relevance feedback is also a possible way to guide expansion — basically allowing the user to help define the similarity metric. But I’m inclined to have that feedback be in the form of facet values rather than people, so that the communication is more transparent. “More like this person” doesn’t necessarily communicate what the user likes about a person.

So when you wrote about the difficulty of guided exploration for LinkedIn, I immediately thought -why not relevance feedback? With LinkedIn data, you have so many informative features for each person. You could easily have the recruiter select examples of the right kind of candidate, and have the system use relevance feedback to suggest more. If you did this within the framework of faceted navigation — continuing to show the distributions of companies, universities, and whatnot, the recruiter might themselves learn, “Oh, CMU and MIT seem to be really similar, maybe I can start there next time”.

It’s really fascinating how different two implementations of the same idea can be when developed in isolation from each other.

We incorporated the same guided expansion concept, but looked at it from the opposite angle. That being a bottom-up from the record, rather than testing alternate filter parameters. The algorithm that I came up with is (at a very high level):

1) Compute nearest-neighbors for every record in the full corpus. The distance (semblance) metrics are project specific. A huge number of weights can be taken into account. Index pointers from every record to the top X records above a fixed threshold on the computed distance value. Let’s call these the semblance indexes. This is an expensive but off-line computation.

2) When a user search is executed, create the result set as usual. Also create the faceted navigation results as usual. Let’s call these the core results.

3) Create a second set of results which is the union of the core result set, and all the records that are in the semblance indexes to the records in the result set.

4) Compute the faceted navigation results for the expanded set.

5) You can now compare the faceted navigation results from the core result set to the expanded results. There is a multitude of ways we’ve come up with for presenting that.

Step 3 is the expansion set. This was probably brought to mind by reading all of those fuzzy logic books in the 90s.

I think the net effect would be similar. Ya?

First, I wanted to place my screenshot on your blog, not post it as a hidden link on my website. I seem to recall in the past your explaining how to insert an image into a comment on TheNoisyChannel.com, but I couldn’t find your instructions in my e-mail, by searching your blog site, or via Google queries targeting your site’s content. Would you mind repeating those steps if this can be done?

Second, I really like the simplicity of checking or un-checking a check box to change or remove a criterion from a query. The older OO Query app displayed in my screenshot does not have this level of simplicity… something to think about if I ever decide to update and adapt that framework to work with distributed graphs, finish it, and make it available on the market.

Also, I initially was thinking that a similar decision tree-oriented tool might be helpful to you, the algorithm designer, in experimenting with your backwards faceted search framework – not end users (recruiters). My tool was designed for people such as FAA airspace monitoring personnel and it offered a lot of power and flexibility, albeit, at the cost of about a one hour learning curve. But this is off topic…

Thanks again, and if you would like some informal help on this project, I would be happy to offer some of my time. My thinking has evolved quite a bit since 2003 when I developed the OO Query tool and I might be able offer some ideas both in the UX and computational speed /storage areas.