Category: General

Loic Le Meur wrote a post today arguing that we need search by authority for Twitter.

His argument:

Comments about your brand or yourself coming from @techcrunch with 36000 followers are not equal than someone with 100 followers. Most people use Twitter with a few friends, but when someone who has thousands, if not tens of thousands of followers starts to speak, you have to pay attention.

I think he’s missing the point of Twitter, or perhaps viewing Twitter narrowly through the lens of a viral marketing evangelist. Twitter is a communication platform, not a marketing platform, and there’s a subtle difference. Much as I wouldn’t want my email or phone prioritizing people based on their number of friends, I wouldn’t want Twitter to apply some global “authority” filter when I’m perfectly capable of deciding whom I want to listen to.

It’s easy to speculate that Le Meur’s argument is self-serving, since he has over 15,000 followers. He also follows over 15,000 people, which shows how little value he actually places on following someone (unless he’s the world’s fastest speed reader).

I don’t dismiss his idea entirely; I can see some value in getting an aggregate view of online punditry. In fact, I’m responding to his argument myself, precisely because his opinion carries weight in the online community and deserves a rebuttal.

But I suspect that Twitter, with its design for immediate, personal communication, isn’t the best vehicle for assembling this view. Note that I’m responding by blogging, not by tweeting.

 
Dan Farber recently shared this observation about the future of journalism:

While the Internet is growing as the place where people go for news, the revenue simply isn’t catching up fast enough. The less obvious part of the Internet overtaking newspapers as the main source for national and international news is that much of the seed content–the original reporting that breaks national and international news and is subsequently refactored by legions of bloggers–comes from the reporters and editors working at the financially strapped newspapers and national and local television outlets.

Matt Asay, wondering whether we’re headed towards a model that looks like “More front page, op-ed, and nothing in between?“, sums it up eloquently:

blogging helps to destroy the business models powering its original source material

I abhor waste, and I’m always amazed that, a decade into the mainstream use of the web, we still have so much inefficiency in the duplication of content.

In retail, there is still a surprisingly high variance in the pricing of the same product among competing sellers, even though price comparison services have been available for years.

In news, much of the content is syndicated from a handful of wire services. Perhaps that commodification of content is part of the malaise in the news industry, but I doubt it; after all, much of the commodification predates the growth in online news. Rather, the problem seems to be that the gains from online advertising revenue aren’t compensating for the offline losses.

I would love to see a world in which original contributions of all sorts are highly valued and rewarded. We see the profit from innovation in physical goods, most notably from Apple’s success in consumer goods. But digital content is different, and I worry about the tension between the high cost of producing it and the low cost of reproducing it.

I spend more time reading blogs than reading news, but I realize that bloggers, myself included, assume an ecosystem in which old-school news organizations do much of the heavy lifting. I play by the rules of fair use and the link economy, giving credit to my sources and linking to them.

But is that enough? Are we slowly nibbling on the hand that feeds us? Is is reasonable to expect journalists, as Jeff Jarvis seems to suggest, to live by links alone? As the title of this post indicates, I don’t think so, but I wish I could offer more constructive suggestions.

Merry Christmas / and Happy Newton Day to all! I hope all of you are spending some time offline for the holidays.

I couldn’t kick my daily blogging habit, especially after I saw an article in the Wall Street Journal about the dreadful controversy of unfriending people on social networks:

Now, people who have accumulated hundreds, or in some cases more than a thousand, friends are cutting loose some of the ones they have lost touch with or who were little more than acquaintances from the start. It’s a shift from the days when users, eager to boast about their online popularity, added new friends with abandon, whether or not they really knew them.

Even Michael Arrington has chimed in with a post about the meaning of friendship. It’s one of his more soberly written pieces; perhaps the holiday spirit is getting to him. His argument in a nutshell:

It’s clear that the more friends you have on any given service, the more noise you have to wade through to find the golden signal. In the real world when you don’t want to be friends with someone, you just find ways not to spend time with them. But online, you click that friend button because it seems so easy, and it’s considered insulting if you don’t. And then you pay.

When I was a child, I remember the importance placed on the notion of a “best friend”. The key, of course, was scarcity. You could only have one best friend, and public declaration of who was your best friend enforced this constraint.

If online social networks are going to claim the same validity as their offline counterparts, they need to reflect the real-world scarcity of attention. Otherwise, the notion of an online social connection becomes a sham.

For example, we know that no one can possibly maintain thousands of meaningful social relationships. Hence, if you are one among the thousands of people that someone is following on Twitter, then you should assume that your relationship with that person isn’t worth the bits its printed on.

Hopefully we’re smart enough as human beings to figure this out. But it would be nice for the online social networks to actually reflect attention scarcity constraints. Then we might be able to leverage them to build far more useful applications.

On that note, I’m going offline to spend the day with my most important connections.

I recently read an article by Matthew Shaer in the Christian Science Monitor entitled “The future of search: Do you ask Google or the gaggle?” and subtitled “To improve results, new search engines rely on users instead of computers.” The article goes on to talk about Google’s SearchWiki, Jimmy Wales’s Wikia Search, and a number of “people-powered” search tools.

I agree strongly with Wales on the value of transparency and that “because search is so secretive, and so propriety, there are fewer checks and balances”. But I agree just as strongly with Shaer that “handing over control to a community could engender a flood of spam, or devolve into a mess of internecine backbiting among users”, both of which he’s observed on the Yahoo Answers site.

Wales ultimately sees the question as not whether humans make the decisions, but rather by what process, i.e., democratically vs. top-down. His Wikia Search effort is an attempt to take repeat Wikipedia’s success for general web search.

But, while I like democracy as a political system (as Churchill said, it’s the worst form of government except all the others that have been tried), I’m not sold on Wikia Search or any of the crop of people-powered search engines.

Perhaps the problem is that, much as in electoral democracy, we need to be vigilant about attempts to game the system. The anonymity of web users is as much a problem as the secrecy of search ranking algorithms. since it allows people to game “people-powered” systems with impunity.

Would a transparent people-powered search system work? Perhaps, assuming it could address the privacy concerns of users. I’m all for transparent social navigation.

But let’s not forget the other part of people power: giving users meaningful control. Crowdsourcing might improve on the current crop of ranking algorithms, but what I really want is a search engine that provides me with transparency, control, and guidance. Let me get under the hood.

Given my role at Endeca, I am hardly objective about the competitive landscape of enterprise search. But, while reading an article about enterprise search in the latest issue of Information Age,  I was pleasantly surprised to find myself agreeing with Autonomy CEO Mike Lynch that the enterprise search industry is beset by “marketing and hype”, and that the technologies available are far from equal.

Not surprisingly, there are a variety of  perspectives among the major enterprise search vendors about how best to address the challenges of enterprise search:

• Autonomy promotes “meaning-based computing”, its branding of its information extraction and text mining techniques.
• Dave Armstrong, a head of products and marketing for Google’s Enterprise division, questions the feasibility of structuring content and emphasizes the importance of search for unstructured data.
• Martyn Christian, IBM’s VP of enterprise content management, asserts that search should not be used to address problems better served by classification and metadata.
• Endeca (not mentioned in the article) emphasizes an interaction-centric “guided summarization” approach that readers here will recognize as human-computer information retrieval.
• Microsoft’s FAST is mentioned, but the only quotation cited is from a disgruntled former customer.

Note that I am trying to convert vendor slogans into vendor-independent terms that have some traction in the information retrieval research community. My hope is that, through neutral forums like the SIGIR Industry Track, we can do a better job as vendors of keeping ourselves honest, as well as engaging academic researchers to help connect their work to the real world.

Above all, let’s strive to compete on technology and ideas, rather than on obfuscation through marketing.

In my previous posts, I discussed applying Mankiw’s Brief Principles of Macroeconomics to the attention economy postulated by Herb Simon and went through the first seven of ten economic principles, which concern how people make decisions and how the economy works as a whole. In this final post of the series, I’ll consider the last three principles, which concern how people interact.

8. A Country’s Standard of Living Depends on Its Ability to Produce Goods and Services.

Over the past two decades, we’ve increasingly heard that we live in an information economy. In the United States, the information economy has been estimated to represent 63% of the total GNP–and that estimate is over a decade old! Even allowing for the inherent challenge in defining the information economy, it’s clear that a large fraction of the goods and services produced in the United States represent information goods, and–present economic concerns notwithstanding–have contributed to the steadily advancing standard of living in this country.

But what if we restrict our attention to the information and attention markets, rather than overall standard of living? Can we still derive insight from Mankiw’s principle?

I think we can best answer that question by looking at asymmetries in the global information market. Information providers, in which I’ll include everything from traditional media companies to web search engines, are heavily concentrated in the United States. As a result, far more attention flows into the United States than out of it. In global economic terms, the United States has a attention trade surplus.

9. Prices Rise When the Government Prints Too Much Money.

Of course, there’s no government printing a liquid currency specific to information or attention. Nonetheless, we can see effects akin to inflation when larger amounts of information become to people without an corresponding increase in the value that information represents. This information glut is the root cause of information overload, and the result is that all information becomes perceived as less valuable.

On the other side, there can be no inflation in the attention market, since people’s attention represents real, rather than nominal, value. If everyone were to have their 15 minutes of fame, then the fame wouldn’t be worth much.

10. Society Faces a Short-Run Tradeoff Between Inflation and Unemployment.

Here I have to admit that it’s a bit of a stretch to apply this principle of macroeconomics to information markets. But this is the last of the ten principles, so I at least owe it a try.

Reducing inflation, in the sense described by the previous principle, requires reducing (or the slowing growth of) the amount of information available for consumption. Naturally, information producers resist such a reduction, as they would like to use this information to gain the attention of information consumers. But it’s a tragedy of the commons: if all of the information producers attempt to optimize for their self-interest independently, the result will be a devaluation of everyone’s information.

This is the hard choice we face as a society when we attempt to remove friction from information and attention markets. It is tempting to reduce the cost of publication to essentially nothing and optimizing the liquidity of attention markets through auction models like those used for search advertising. We can introduce friction to reduce inflation, but only at a cost.

To sum up: information and attention may not be traditional economic goods, but they nonetheless follow general economic principles. And technologists who work with information would do well to learn from those principles.

I’d like to close this series with a story I heard from a colleague at Yahoo Research (either Prabhakar Raghavan or Usama Fayyad) about economics and information. Yahoo runs a online personals site, and encountered a problem common to such sites: women complaining about being inundated by email from men. Yahoo’s engineers saw this as a technical problem and brainstormed technical solutions, such as automatically detecting and filtering out messages that might draw complaints.

But an economist on the staff quickly identified the problem: the lack of scarcity in the system’s attention market. He proposed a simple solution: give men a limited supply of “digital roses” to hand out to women. Then the invisible hand of market economics solves the problem on its own.

I don’t know whether or how Yahoo ultimately implemented this approach, or whether they considered its applicability to other gender pairings. But, as information and attention become increasingly important economic goods, we would do well to learn from their example.

My recent posts about the macroeconomics of information and attention triggered an unexpected controversy about who deserves the credit for the concept of attention economics. The Wikipedia entry for attention economy credits Herbert Simon, and I had always thought he came up with the idea. Perhaps I’m biased because of the five years I spent at CMU.

But Michael Goldhaber posted a comment in which he made a case that he deserved credit for introducing the idea. Unfortunately, I couldn’t ask the late Herb Simon to respond.

But I did find an explanation on thw WorkingCogs blog that I thought might satisfy all parties:

Herbert Simon is often credited with being the first person to describe what attention economics is – that a wealth of information leads to a dearth of attention due to the fact that there is so much information out there and only so much attention that can be given to information, and the idea behind rationalizing how much attention any one information source receives.

Golhaber (1997) seminal paper (on an online peer reviewed journal) is however the crucial turning point for this idea. This article presents the strong hypothesis and its consequences. In what follows we will try to introduce the idea of attention economy, mostly from Goldhaber’s point of view and how some popular pages implemented attention technology. Goldhaber has been preparing a book for the last 10 years, and he blogs prolifically. 

I hope this explanation offers an equitable allocation of credit and resolves the unintended controversy.

In my previous posts, I discussed applying Mankiw’s Brief Principles of Macroeconomics to the attention economy postulated by Herb Simon and went through his first set of ten economic principles, which concern how people make decisions. In this post, I’ll consider the second set of principles, which concern how the economy works as a whole.

5. Trade Can Make Everyone Better Off.

Trade unifies the information and attention markets, since information and attention are often the commodities being traded. While these commodities can be monetized, they are more often traded as is, especially in the online markets.

In the online world, the market for trading information and attention is the link economy. Links are themselves a form of information, but more importantly they serve to promote other information. In the early days of the web, that promotion had a straightforward effect: a link offered the possibility that a browsing user would click on it and thereby consume the linked content.

Today, however, links serve an even more important role: link analysis are the main basis used by web search engines to determine the authority of a web page, which in turn is a major factor in determining whether and how prominently that page appears in search results. Indeed, a key aspect of web search is the arms race between search engines and “black hat” search engine optimization (SEO) experts who try to game the ranking algorithms by link spamming.

Despite the fraud, however, the link economy is a critical mechanism for creating value. As many people have pointed out, the best sites are designed to give users the information they want, even if that means directing users to other sites. This enlightened altruism earns users’ trust and loyalty. Sites also develop trust relationships with one another, formally or informally cooperating to satisfy complementary information needs.

6. Markets Are Usually a Good Way to Organize Economic Activity.

It’s hard to imagine an alternative to a market economy for information and attention (at least in the free world), but central planning is often a matter of degree. Governments may restrict what information is published or who can access it, and governments may themselves act as information producers and consumers.

While the broad freedom to publish and consume information is so taken for granted in modern democracies, one might ask if this free-for-all leads to the efficient allocation of information and attention resources. In fact, the massive duplication of online content and the prevalence of spam might suggest inefficiencies in the present allocation.

Nonetheless, it’s hard to imagine that we’d do better with central planning. In countries where governments attempt to tightly control the flow of information, citizens often manage to work around those controls, and the trend seems to be towards loosening control of information, e.g., in China. Indeed, information and attention markets may be the poster child for the effectiveness of free markets in general.

7. Governments Can Sometimes Improve Market Outcomes.

While central planning is generally a bad idea, governments can and do help information markets in at least two ways: providing infrastructure and regulating against monopolies.

One of the most dramatic examples of government creating value through infrastructure is the Internet itself. It is impossible to imagine any single information producer making the investment in such an ambitious and far-sighted project. Yet the Internet has created enormous value for both information producers and consumers.

Government also serves a key role in regulating against monopolies. Much of the historical concern about Microsoft and the more recent concern about Google reflects the critical role these companies play in routing information consumers to information producers. Government intervention–or, more importantly, the threat of government intervention–helps ensure that no one will exert and abuse monopolistic control over this market.

To sum up: information and attention create a global economy and thus are subject to the market dynamics familiar to such economies. As with other economies, the invisible hand usually knows best, but at times it is necessary for governments to make far-sighted investments or prevent abuse.

In the next and final post in this series, I’ll consider the set of Mankiw’s principles on how people interact.

Continue: The Macroeconomics of Information and Attention: How People Interact

In my previous post, I discussed applying Mankiw’s Brief Principles of Macroeconomics to the attention economy postulated by Herb Simon. Let’s start with the first set of basic economic principles, which concern how people make decisions.

1. People Face Tradeoffs.

• Information Markets
  In the 1990s, the web quickly exploded from an obscure protocol into an ocean of data, some of it even useful. Even before the Internet age, people had to allocate their scarce attention among the available media–something Herb Simon could see happening as early as 1971! Today, there is no question that reading everything–or even everything likely to be useful–isn’t a feasible option for anyone. Moreover, the tradeoff problem for information is recursive: choosing what information to consume is itself a problem that requires consume information. Simon recognized this problem and proposes bounded rationality as a model to describe our process of making decisions despite our inability to process all of the pertinent information. The result is satisficing: a decision-making strategy which aims for adequacy, rather than optimality.
• Attention Markets
  Now let’s consider the consumers of attention–that is, information producers who would like to see as much of the collective attention consumer budget devoted to their particular information product. Such producers include retailers, publishers, politicians, et cetera. These information producers / attention consumers also face tradeoffs. Their main tool to obtain attention consumers is to their finite budgets to invest in marketing. In the online world, that means spending on advertising and search engine optimization (SEO). The trade-offs among marketing vehicles can seem almost as overwhelming as those facing information consumers, thus fostering multi-billion dollar advertising and SEO industries.

2. The Cost of Something is What You Give Up to Get It.

• Information Markets
  Tradeoffs mean that people choose what information we consume based not only on the value of that information, but on opportunity costs. Those opportunity costs typically take two forms: money and attention. We can think of the attention cost as the cost of not being able to consume other information that might also be valuable. In a rational market, there should be a clear exchange rate between these two costs, a currency exchange to quantify what it means to say that “time is money”. But information consumers at best display limited rationality. In particular, the near-total dominance of advertising-supported information products implies a very low value on time: most people aren’t willing to spend even pennies to skip ads. As a result, most information consumers make trade-offs as if all information is free, and the only scarce commodity is their own attention.
• Attention Markets
  Information producers also have tradeoffs to optimize. In general, they don’t consume attention for its own sake, but rather in the hopes of converting that attention into some sort of profitable action, like selling a product or obtaining a vote.Hence, retailers allocate advertising budget among they keywords they purchase as pay-per-click search engine advertisements, and politicians budget campaign funds among regions and demographic segments. The fierce competitiveness of attention markets ensures that, regardless of the total budget involved, information producers are well aware of the opportunity costs of their decisions.

3. Rational People Think at the Margin.

• Information Markets
  There is an enormous redundancy in the information available, particularly online where the cost of distribution is practically zero. The marginal value of information to a consumer is the amount of value the consumer experiences from it, given the information he or she already has. Much of the information overload problem reflects that, even if each piece of information has significant value in absolute terms, the redundancy makes the marginal value much lower–often indistinguishable from zero. While this redundancy is part of what makes web search engines so effective, it is also a major challenge for consumers who would prefer to see the ocean of data de-duplicated.
• Attention Markets
  The consequence of thinking at the margin is interesting for attention markets. It may be the leading factor in web search being a natural monopoly, since information consumers find little benefit from using more than one search engine. Google’s success in part reflect a desire by information consumers focus primarily on a single channel. In a related vein, marginal value also affects network externalities (aka network effects) in attention markets. For example, the pay-per-click model of web search advertising worked around the congestion of the display advertising business. Moreover, because the business model was to auction ad placement, Google experienced a positive network externality as its market share of consumers increased and advertisers raised their bids to compete for this treasure trove of attention budget.

4. People Respond to Incentives.

• Information Markets
  As we’ve noted already, most people aren’t willing to spend even pennies for online information. While we may question the overall rationality of how people weigh money against time, there is no question that they respond to incentives. The New York Times had a painful lesson in lost readership when it tried to get users to pay for a premium TimesSelect service that included previously free (ad-supported) content; it didn’t take long for them to revert to their old model. Other information providers have had more success with “freemium” models that offer free services but try to upsell premium services for a fee. But incentives in information markets aren’t just monetary. Users abhor the attention cost of clutter–which is one of the reasons cited for Google’s triumph over Altavista. For that matter, spammers and phishers have learned to exploit “social engineering” techniques that essentially lure people with the promise of purportedly valuable information.
• Attention Markets
  Indeed, spam is a case study in how attention consumers respond to incentives. Junk mail has been with long before the Internet, but at least the sender needed to achieve a response rate that overcame the cost of postage. The mass adoption of email made the cost of distribution essentially zero to spammers. In fact, there are price wars for spamming services, with offers as low as $80 per million emails sent. A recent study suggests that spammers achieve an average response rate of under 0.00001%., and yet the volume of spam accounts for an estimated 80% to 96.5% of overall email traffic.One of the ideas proposed to combat spam is attention bond mechanisms:

  The Attention Bond Mechanism (ABM) is a means of using sender-posted bonds to eliminate spam and facilitate mutually agreeable communication. The ABM can be applied to email and to other communications media.

  For example, if you want to send me an email message, you might post a dollar as a bond in order for the message to be delivered. As long as I am satisfied that the email is not spam, you keep your dollar. But, if I do flag the email as spam, I take the dollar as compensation for my wasted time.

To sum up: information and attention obey the same principles of supply and demand as more familiar economic goods. In the next post, I’ll consider the second set of Mankiw’s principles, which concern how the economy works as a whole.

Continue: How the Economy Works as A Whole