| | | | | | | | | |

Search Engines: Clogged with Commerce and Begging for an Upgrade

By Cheryl Woodard, Executive Director
July 13, 2004

Search engines are great shopping resources – for video games, cameras, or even summer camps. But for practical information, the search engines have run into serious trouble. Commercial sites elbow out more substantial, community-minded resources, which have become nearly impossible to find. Fortunately, these problems can be fixed if search engine companies work to balance community and commercial needs more effectively.

Picture the parent of a troubled teenager turning to Google or Yahoo in search of credible information sources, effective treatment programs, and perhaps to find places for connecting with other parents. I studied search results at Google and Yahoo to see how well they deliver on a parent’s reasonable expectations in this kind of circumstance.

Our Askquestions.org reporters are currently researching an article about treatment programs for troubled teenagers and as a result, I know that there are many free and effective treatment options available. Family doctors, school counselors and religious communities all offer relevant services to families. Private therapists are also readily available, and most states offer publicly funded counseling or therapy programs aimed at keeping teens off drugs and out of the court system. We also found online message boards and community sites tapping the word-of-mouth network among parents. And we located expert advice from government agencies, academics, and the medical community. The consensus opinion across all these sources is that families should seek local, community-based programs that treat the whole family, not just the teen. And luckily, plenty of community-based programs are available.

Both Search Engines Missed the Good Stuff
I was stunned to discover that none of this information appeared when I searched on the phrase ‘troubled teen’ at Google and Yahoo, even when I waded through 100 search results at each site. Instead, I was confronted with a staggering number of listings all pointing to one commercial option: coercive residential treatment centers (RTCs) that include boot camps, wilderness programs or behavior modification programs. Costing between $1000 and $2500 per week and lasting for many weeks or months, RTCs are a growing and profitable industry. But mental health experts warn against them.

RTCs consume about one-fourth of U.S. spending for children's mental health, according to a report from the US Surgeon General’s office. And yet, recent well-controlled studies find that RTCs not only fail to help troubled youths but they can actually exacerbate their problems. "When you cut off ties with parents, family, and home [as these approaches do]", says Dr. James C. MacIntyre, associate professor of psychiatry at Albany Medical College in Albany, New York, "you make it much less likely that the child will eventually be reintegrated into society.” And many journalists, parents, and experts describe plainly predatory marketing practices used by some of the RTCs to lure desperate families into choosing expensive options based on false promises. Lawsuits against RTCs number in the hundreds. Survivors run web sites and nonprofit groups aimed at helping each other cope with the devastating after-effects. Some of these resources are listed below.

At both Google and Yahoo, searching on the phrase ‘troubled teen’ delivers almost nothing except commissioned referral agencies and the RTCs themselves. Deception abounds. The referral agencies do not disclose the steep commissions they collect from schools and escort services, for example. And interconnections between the various arms of this industry are common. Six different URLs among the top 25 results at Yahoo point to one site, TeenSuccess.org, which is a referral agency. TeenSuccess.org is registered through a proxy DNS registry, and there is no ‘about us’ information provided at the site itself. A consumer has no way to determine who owns TeenSuccess.org, or to appreciate the site’s many commercial ties to RTCs and other businesses.

The following table shows my analysis of the top 100 non-sponsored results for the ‘troubled teen’ search at Google and Yahoo conducted on the same day in June 2004. I spent time looking at every site, noting interconnections between the various sites, the links among them, and the people behind them. Only 15% of the listings at Google and 2% at Yahoo were the kind of nonprofit, noncommercial resources that would be most useful to parents.

Searching on the phrase ‘troubled teen’



Total number of results
Total Listings per Page (sponsored and non-sponsored)
Percent of first page, non-sponsored results that are commercial
Percent of the top 100 results that are non-commercial
Number of valuable news articles in the first 100 results
Number of commercial sites with multiple listings in top 100
Number of redirected ULS on first page

Google and Yahoo programmers probably know exactly how many pages of results most people will read before fatigue sets in. My personal limit was 100 listings, which equals 10 pages at Google and 5 pages at Yahoo. I didn’t include the sponsored links in my analysis, but I noted that the RTCs and referral agencies dominate those listings, too. Including the sponsored links, you begin to appreciate that a searching parent has to scroll past a truckload of commercial listings to find only a handful of non-commercial sites and valuable news articles.

Commercial Sites Are Not Good Citizens
There’s another problem with the commercial sites-- they don’t share, and a parent who wanders into one of them tends to get stuck inside a kind of RTC echo chamber where no other options are considered, and no dissenting views are expressed. By contrast, non-commercial sites always linked to external resources. Indeed, our reporters have found most of their best sources by following links either from journalists, academics or government sources. But the commercial sites link only to one another. None of them cite independent information sources, even though there is plenty available. And none of them refer to media reports or news articles about themselves or their industry.

I found 6 media sites in the top 20 results at Google and 2 at Yahoo, but only one is a news article. The others contained very little content except for ads, links or paid listings from RTCs, and referral agencies like TeenSuccess.org. In all, I found 13 good news articles in the top 100 Google results and only 3 at Yahoo, But none of the excellent and fairly recent reporting on this subject from the New York Times, Fox News, and Salon.com (among others) appeared in the first 100 listings either in the “news” section or the regular search results at either site. I’ve included a list of these news stories below.

Adding everything together, Yahoo was far more clogged with commercial sites than Google and had more examples of outright manipulations. For example, four different companies were able to include themselves repeatedly among the top 20 listings at Yahoo, while only one company got multiple listings among the top 20 Google results. Yahoo also had 10 redirected links on their first results page, while Google had none.

From a user’s viewpoint, this search experience is frustratingly commercial at both Yahoo and Google. All of the most effective and affordable treatment options are missing. Journalists, academics and other impartial information sources are scarce and hard to find in the overwhelming numbers of sites served up (nearly 500,000 at Google and over 800,000 at Yahoo). Welcome advice from other parents cannot be heard. And important government research statistics or information reports do not appear. The results are wildly out of balance in favor of commercial rather than consumer needs.
How to Upgrade the Search Engines
How can the search engine companies improve this experience for consumers? Boosting visibility of noncommercial information sources would be a vast improvement. And establishing some ground rules for commercial sites would also increase the user’s satisfaction. I offer the following recommendations:

  • Create a separate search channel for noncommercial, publicly funded, and nonprofit services and information sources using the same model as Google.com/Uncle Sam, which restricts results to government sites. Searching Google.com/UncleSam for “troubled teen” produced about 5,000 listings, all of them from public institutions like the National Institutes of Mental Health, Congress, and several state agencies. Consumers would appreciate having the same option to bypass the commercial sites and find other community-based information sources, parenting bulletin boards, and non-profit service agencies in a Google.com/Community search channel.

  • Create a task force of suitable people to establish criteria for inclusion in the community search channel. Nonprofits could certainly use the help reaching their constituents. Few nonprofits have a marketing budget to match the commercial companies who spend thousands of dollars for sponsored links at Google and Yahoo. Google’s website claims that they have a program to help nonprofits use their Adwords program at reduced rates, but for the past month or more, we have been unable access the adwords.google.com/grants page, and I noted that nearly all of the sponsored links at Google represent commercial companies who are undoubtedly outbidding the nonprofits. Giving low-cost sponsored links to nonprofits is a good idea, but boosting their ranking in the non-sponsored listings would be an even bigger boon, and a valuable service to consumers.

  • Boost the visibility of credible media sites. We found many valuable articles in media outlets that did not appear among the top 100 listings at either search engine. In every case, reporters have interviewed experts and assessed the evidence to provide exactly the kind of information parents sit down to find when they search the Internet.

  • Set standards for commercial sites by, for example, requiring them to correctly identify themselves. Clearly the commercial sites have used link exchanges, redirected URLs and other tactics to gain unfair advantages with the search engines at a significant cost to users.

I can imagine this troubled teen scenario re-occurring in many other situations where businesses and consumers have conflicting interests. Confronted with health, career, or family issues, consumers want access to the low-cost or free community-based options available to them, but advertisers want to channel consumers towards the goods and services that generate the most profits. This tension is a constant issue across the Web, and in every other public medium. But because of their unique role as gatekeepers on the Internet, the search engines have the greatest opportunity to find a workable middle ground that supports community goals as well as commercial ones.

The magazine industry has a long tradition of balancing the competing needs of readers and advertisers, and so we know that it can be done. I can offer many examples from the twenty plus years I’ve spent as a publisher and consultant to magazines. Take mini-vans, for instance. When the car companies were aggressively marketing the first mini-vans to families in the early1980s – and buying lots of ads in parenting magazines by the way – magazine editors were writing critical reviews of the vans. Their main objection was that the mini-vans were regulated as trucks, not cars, and therefore lacked the safety features people take for granted in a passenger car. Fairly quickly, the auto industry responded by boosting the safety features of the mini-vans. It was a win-win situation for consumers, editors and advertisers alike, and I think we can reasonably demand an equally balanced approach from search engine companies.

Luckily, Google and Yahoo don’t need to hire editors or experts to balance competing commercial interests like magazine publishers do. Instead, the search engines only need to help consumers find the non-commercial resources already available to them.

The Noncommercial Sites for ‘Troubled Teens’ Found by Reporters at AskQuestions.Org (but not found among the top 100 listings at Google or Yahoo)

1. Survivors Groups
International Survivors Action Committee www.isaccorp.com/index.html
A non-profit group of families with first-hand experience offering tips to parents about finding non-abusive residential treatment programs.
TheStraights.com www.thestraights.com A collection of news articles and information to help parents avoid the most abusive programs, the ones with many pending lawsuits and criminal investigations. The site is maintained by Wesley Fager, a father who unwittingly sent his son into an abusive program.
Another site opposed to RTCs is NoSpank www.nospank.net/boot.htm.

2. Parents Forums
Fornits http://fornits.com/wwf/ Includes information submitted by other parents along with many first-hand stories.
Ivillage http://messageboards.ivillage.com/iv-pstroubled Has thousands of postings and replies about handling troubled teens.
Parents Network at UC Berkeley http://parents.berkeley.edu/ Thousands of parents offer advice and recommendations to each other.

3. Academic/Expert Agencies
The Center for Adolescent Studies at Indiana University http://education.indiana.edu/cas/adol/adol.html Offers many links, articles, and resources for parents and teens, both.
Children Youth and Family Consortium at University of Minnesota http://www.cyfc.umn.edu/adolescents/index.html Offers a host of resources, articles and tips for parents.

4. Professional Associations
The Independent Educational Consultants Association (IECA)
http://iecaonline.com/ a non-profit, international professional association representing full-time experienced independent educational advisors.

5. Community-Based Treatment Programs
Because I Love You http://www.becauseiloveyou.org/ is a nonprofit organization that helps parents develop effective strategies for dealing with teens.
The National Self-Help Clearinghouse helps people find support groups or start new ones. http://mentalhelp.net/selfhelp/

6. Federal Government Information Sites:
Mental Health Clearinghouse http://www.mentalhealth.org/child/childhealth.asp
Centers for Disease Control and Prevention www.cdc.gov/mentalhealth/
Food and Drug Administration www.fda.gov
Substance Abuse and Mental Health Services Administration www.samhsa.gov A Family Guide to Keeping Youth Mentally Healthy and Drug Free www.samhsa.gov/centers/clearinghouse/clearinghouses.html
SAMHSA's National Mental Health Information Center http://www.mentalhealth.samhsa.gov
National Institute of Mental Health Child and Adolescent Mental Health www.nimh.nih.gov/publicat/pubListing.cfm?dID=23

Recent News Articles that did not appear on Google and Yahoo searches:

From Fox News: “Drug War Casualties” by Radley Balko, Thursday, May 23, 2002 http://www.webdiva.org/fox/

From the Wikipedia online encyclopedia: “World Wide Association of Specialty Programs and Schools”is a collection of news articles and report on one association of RTCs. http://en.wikipedia.org/wiki/WWASP

From the Observer magazine of the British newspaper the Guardian, “The Last Resort” by Decca Aitkenhead, June 29, 2003. http://observer.guardian.co.uk/magazine/story/0,11913,987172,00.html

From Salon.com, “I Was a Hired Thug for Tough Love” by Sheerly Avni , August 30, 2000, http://dir.salon.com/mwt/feature/2000/08/30/wilderness_camps/index.html

From the magazine Psychology Today, “Family Ties”, By Ken Gordon, August 2, 2002

From Mother Jones Magazine, "Drug Mistreatment" by By Jake Ginsky
February 18, 2000, http://www.motherjones.com/news/feature/2000/02/rehab.html

From Ivanhoe.com health news, “Whats Wrong With My Child? Mental Health in Kids—A White Paper” by Mildred Leinweber Dawson, published March 3, 2003 (requires a small payment to read). www.ivanhoe.com

AskQuestions.org is a service of the Public Interest News Gate, a non-profit organization dedicated to increasing public access to information.