We Need to Talk About Google. Again.

Hello, dear readers. It has been much longer than I intended since my last post. I’ve had a few things going on in my non-newsletter life, and have also realized that one of the challenges of writing about nuance is that there are not a lot of natural stopping points. But my open tab situation is worse than usual (which is saying a lot), so it’s time to write something I can share before my computer gets any angrier at me. If you like what you read, please consider forwarding this newsletter to someone else you think would enjoy it. And if someone forwarded this newsletter to you please consider subscribing!

Many years ago, when I was a relatively new librarian, I did a workshop on Google search for my hometown library. I made an extensive slide deck with dozens of screenshots all labeled with arrows and explanations so folks could refer to them after the workshop. Then a month or so later Google made significant changes to its search interface and the slides were rendered obsolete. I offer this story to say: I am used to Google making changes that make something I’ve created out of date very quickly. Which is why I suppose I’m not that surprised that only a few months after writing about the deteriorating quality of Google search results, Google has gone ahead and managed to make search worse. Disappointed, but not surprised. 

Google has been slowly rolling out AI overviews into search results for the past year, and introduced them more broadly in May. These overviews don’t appear in response to all queries, but only when Google’s technology determines that they’re the most efficient way to provide the information a searcher is looking for. What are the factors that go into that determination? Great question, wish I knew the answer. The overviews are, according to Google, “’supported by info from across the web and Google’s Knowledge Graph, a collection of info about people, places, and things.” The overviews also include links to the sources where the information came from (though those links are not easily viewable unless you expand the results), and the “About This Result” info notes that “Generative AI is experimental and info quality may vary.” While the overviews do include a small footnote that “Generative AI is experimental” the accompanying warning that info quality may vary is only viewable if you click to see more about the result. Which is… a choice.

This “Search Generative Experience” could have a significant impact on website traffic, journalism and publishing, and advertising. There are lots of folks writing about those issues, so I want to spend some time on two questions I’ve been pondering: what problem are these AI overviews trying to solve, and how well are they solving it?

What’s the problem?

My initial reaction to AI-generated answers in search results was that it was a solution in search of a problem, or at least a solution to a problem that had not been clearly defined. After seeing some of the results (and how bad they were), my more cynical reaction was that Google was adding AI-generated results because all of these various models need a lot of training, and you can get consumers to do it for free (much like they did with reCAPTCHAs). 

One problem it could be solving (but definitely isn’t yet) is cognitive overload. Mike Caufield recently wrote about the challenge of synthesizing information from multiple websites, especially when you’re trying to sort through information from news sites, professional organizations, commercial sites, and answers from web forums like Reddit. And, as I wrote about before, search results are so full of SEO garbage that you’re often sifting through sites that don’t have much to do with your query. An AI-generated summary could, in theory, alleviate some of that cognitive overload. But, as Caufield also notes, these summaries are often providing vague or incomplete answers and “with the blender of AI set on the puree setting, we’re never quite sure” which information is coming from where. 

Layering these AI summaries on top of an internet that is already full of bad or outdated information, SEO-focused content, and, well, shitposting reminds me of the nursery rhyme about the old woman who swallowed a fly, and then swallowed a spider to catch the fly, then a bird to catch the spider, and so on. We can’t solve the problem of cognitive overload by adding more and more information to sift through and evaluate.

So these summaries don’t help with cognitive overload. Not only that, they also cut off the kind of thinking and evaluating that eliminating cognitive overload would theoretically make more time for. The benefit of a summary is not the summary itself, it’s the act of summarizing: figuring out what’s important to your information need, weighing various perspectives, finding unexpected information, and making sure you understand the information well enough to put it in your own words. Even if these summaries get better at helping with cognitive overload, the answers they provide will always be less complete than the ones you have synthesized yourself.

According to an interview with Liz Reid, Google’s head of Search, the AI overviews are resulting in people doing more searches “because they suddenly can ask questions that were too hard before.” As she tells it, these overviews provide a springboard to deeper research and more targeted searching.

I am, to put it mildly, skeptical of this analysis. I know many of the AI summaries I’ve seen have led me to conduct additional searches, but it’s mostly so I can try to figure out what the hell is going on.

A fruit by any other name

One of the reasons it took me a while to write this was because I wanted to be able to explain what was happening with these AI summaries, which is not really possible. So instead I’m going to share a couple of searches I’ve done, what I noticed, and some shouting into the void. 

A couple of weeks ago I saw someone on Bluesky posting about the AI-generated overview for [fruits that end with um] and decided to try it myself. Take a quick look at the screenshot and see if you notice any issues.

What’s that? None of these are foods that end with um, and the only real food (coconut) notably does not end in um? Good eye. So if these summaries are “supported by info from across the web and Google’s Knowledge Graph” where did this information come from?

Google does at least link the source, which appears to be this four-year-old post on Quora, which is an obvious joke. Obvious if you’re a human, that is. Many people have noted the need for a sarcasm tag or font when communicating on the internet, but that need seems especially urgent when you are using data from the web to train a program that is, ostensibly, providing at least semi-accurate answers.

I did the same search in another browser and did not get an AI-generated result, but did still get the joke answer from Quora highlighted at the top of the results. This is what I mean by the “old woman who swallowed a fly” problem. If the search results are already bad, a summary of them is not going to be any better. 

I have tried this search and similar ones several times over the past week, and the results have changed but they’ve never actually answered the question accurately.

Please note: Eurofruit is a trade magazine, Moonfruit was a software company, barefruit isn’t real, and none of them end with “um.” I’m proposing an iteration of the Turing test based on the ability to identify a plum.

Planet of bad search results

I was watching Rise of the Planet of the Apes the other night and was curious about how the apes’ signed language was developed, and if there had been any reaction from the deaf community about the portrayal of signed languages. It took me a few searches to get something even close to answering my question, and when I did… yikes. 

The last sentence of that summary is syntactically challenging but seems to imply that American Sign Language (ASL) is a “broken” language. ASL, to be clear, is not a “broken” language, and the implication that it is reinforces the misperception that ASL is somehow a “lesser” language, an idea built on deeply ableist beliefs about Deaf people. So where did that line about a broken language come from? From a flashcard on Quizlet. It’s not clear who made these particular cards, but based on the other ones the user has made it’s reasonable to assume that they are/were a student, but not a student of ASL, linguistics, or Deaf history. So why did Google prioritize this information in the summary? This is a non-rhetorical question, but one I don’t have an answer to.

I did some more searching, both because I was still interested in finding an answer and also because I had some morbid curiosity about how bad the results were. 

The sources used in this summary are all in the top five search results on this page, which has been the case for many of my searches. Summarizing (or sometimes flat-out plagiarizing) the top results of a Google search does not strike me as something worth the tremendous amount of water and energy required to produce them.

You may have noticed that Quora showed up several times in the linked sources for my searches, as does a site called Poe.com. Quora is a question-and-answer website (similar to the now defunct Yahoo! Answers) with a decidedly mixed reputation for accuracy and reliability. Poe.com is a chatbot developed by Quora that people can use to ask questions of a range of different AI platforms built on large language models. Yes, you read that correctly; one of the sources cited in these AI-generated summaries is also an AI chatbot. It’s an AI-generated summary of an AI-generated summary. I am having a hard time figuring out how to convey how very, very bad that is. Whatever is going on, it’s abundantly clear that this product is not ready for use by the general public, no matter how many warning labels you put on it. 

Is there any way to get decent search results?

I mean… I don’t have a solution for the abundance of misinformation and SEO garbage on the internet, but it turns out there is a way to at least clear the clutter from your search results. The linked post goes into more detail about how to make this your default search, but you can use this option for any set of results you get on Google. From the same menu bar where you find Images, News, and other ways to refine your search there is now an option for Web. Clicking on this will give you a results page without AI summaries, without ‘People Also Asked’, and without a ton of Sponsored content. It does not solve the larger problems, but at least it gets you back to a search results page that is primarily focused on search results (what a novel idea).

I want to be able to give some advice or guidance on how to push back on the encroachment of artificial intelligence, but it feels an awful lot like tilting at windmills. The ways AI summaries have made Google even less useful are obvious, but it seems like every site I visit is pushing some new AI integration, and it’s harder and harder (or impossible) to opt-out. The crypto and NFT bubbles were relatively easy to avoid, but generative AI seems to be integrated into everything, even though the hype demonstrably does not match reality (remember stories about ChatGPT-4 “acing” the bar exam? Turns out it scored in the 48th percentile, not the 90th). Microsoft seems to have abandoned their climate pledge in order to go all-in on generative AI. It can be hard not to feel like a Cassandra (which Google’s AI summary helpfully informs me may be either a Greek mythological figure or a NoSQL database). 

I may be tilting at windmills or shouting into the void or pointing out that the emperor is naked or doing some other thing that is a metaphor for futility. But I believe it’s important to keep tilting and shouting and pointing.

Thanks for reading! Want to work with me to help your community develop the skills they need to be savvy consumers and creators of information? Get in touch!