Google is putting tomatoes in our fruit salad

Welcome ! If you like what you read, please consider forwarding this newsletter to someone else you think would enjoy it. And if someone forwarded this newsletter to you please consider subscribing!

Several months ago I read Parker Molloy’s excellent piece about the role understanding context plays in making sense of reality, and I have thought of it many times since then (it’s subscriber-only, but you should subscribe). It was very much on my mind as I saw screenshot after screenshot Google’s extremely bad AI-generated summaries. We know that context (or the lack thereof) is easily manipulated for political and personal gain, and that context is necessary for making sense of the world. I wanted to explore a few kinds of context that are missing from Google’s AI summaries. 

The context of the question

I have spent a significant part of my career teaching people how to make the context of their information need more legible to search engines. We may know what we mean when we search [impact of sun exposure], but a search engine doesn’t know if we’re looking for information on protecting ourselves from sunburn and skin cancer, or for information on vitamin D or Seasonal Affective Disorder. We can add that context to our search in different ways: 

• Adding more specific terms

• Describing the type of information we want (ex. review, survey, timeline)

• Specifying the type of site where we think we’ll find the information we want (ex. limiting to .gov or .edu, or adding [reddit] to a search

I could (and have) talked for hours about this, but you get the idea. We know that our information need has a context, even if we can’t fully articulate it until we get unsatisfactory search results back. We can also evaluate our search results, recognize that they’ve missed the mark, and iterate on those search results. 

Without a detailed, specific prompt AI cannot understand the context of your question and the information you need. Which, in some ways, is a good thing. Having to think about what you need and be able to clearly articulate it are useful skills no matter what kind of search you’re doing. But searchers are not used to giving such detailed prompts when searching online, and neither the format of the search bar nor the AI-generated summary promotes iterating on or refining results. 

Google has never understood the context of our information needs; these AI-generated summaries just make it obvious (ish. More on that later). 

The context of the information

Google has always made assumptions about our information needs, and when those needs are unclear or ambiguous we get a wide range of possible sources and perspectives. Typically, we can scan the results and look for a source that aligns with our needs; when you search [sun exposure] you know if you’re looking for medical advice, sunscreen recommendations, sunburn care, or tips on getting more sun in the winter. An AI-generated summary collapses all of that into one answer. This might be okay, except we know that in addition to useful information the internet also contains a lot of things that are not true: jokes, satire and parodies, mis- and disinformation. Mike Caufield wrote about the “context collapse” that happens when Google’s AI generates these summaries; a search for [cheese sliding off pizza] will likely include recipes and advice, but as we all now know it will also include a Reddit post that recommends adding Elmer’s glue to the sauce (still the top search result, for the record). But Google cannot distinguish between these contexts so they are smooshed (technical term) into one summary. The “About this result” notes that summaries are supported by “info from across the web and Google’s Knowledge Graph, a collection of info about people, places, and things” but I don’t see evidence of that. Part of the weirdness we’re seeing, as Caufiled notes, is because Google is creating these summaries from a specific result set, not a large body of information such as what ChatGPT and other generative AIs use.

Another AI summary I saw confidently declared that 13 U.S. Presidents have attended UW-Madison, earning an impressive 59 degrees in total. I saw comments about Google’s AI just making things up, but it’s actually a little more interesting than that. The summary linked to its source of information, which was a page from Wisconsin’s Alumni Association listing graduates who shared a name with a U.S. President. 

There is a crucial piece of context directly above that list of “presidential graduates” which Google appears to ignore. The word “humor” even appears in that introduction; if Google’s AI can’t identify as humor things that are labeled as humor it does not stand much of a chance against The Onion when summarizing search results.

The context in which the question is being asked

Beyond understanding the context of your specific information need, Google also can’t understand the larger context in which your question exists. 

NBC shared this screenshot of an AI-generated response to the question [how many muslim presidents in us]. If you don’t have the context of the last, say, 20 years of American politics this could seem like a perfectly straightforward question.

A question about a Muslim president has a context. As does any question about a president and religion, but Google’s AI doesn’t know that, it just takes the question at face value. Which would be okay if the internet were not full of conspiracy theories and Islamophobia. Understanding the context of the question means not just providing a factual answer, but explaining that questions about Obama’s religion are rooted in racism and Islamaphobia. Google’s AI failed on both. Also, side note to NBC, which introduced this screenshot by saying that “Some of the false answers verged into politically incorrect territory…”: this information is not “politically incorrect.” It’s just incorrect. If we’re going to deal with AI-generated disinformation we need to be clear in labeling it as such.

If Google’s AI could understand larger context, it might recognize that lots of people are suddenly asking about eating rocks and identify that there may be a larger context for those questions that it’s missing. Maybe it’s a good thing that an AI can listen to someone ask the same question over and over and over again without getting annoyed, but I don’t think it’s a good thing that it can’t noticed when it’s being manipulated.

The context in which the summary was created

These AI-generated summaries are showing up in search results whether or not you want them (though you can avoid them, at least for now). Generative AI tools have been shoved into all kinds of platforms, whether or not people have asked for them, and whether or not they’re ready for a wider user base. I’m sure many folks have seen this post from Scott Jenson, former project manager at Google: 

“AI” is the hot new thing, and the fear of being left out is palpable; combine that with a “move fast and break things” ethos and you have, well, Google telling you to go eat rocks. Even if you take what Google’s motivations for creating these overviews at face value, the problem they’re trying to solve is one they created. Search results have become so bloated with SEO garbage that a summary of results could be genuinely helpful, especially if it prompted a searcher to use that summary to refine their search. Human authors know the goal of a summary they’re writing; Google’s AI is just shoving everything into a blender and seeing what comes out.

The context we need to understand

Companies and individuals who are deeply invested (financially and emotionally) in AI are not always giving us the full context when promoting what various AI applications are capable of. There was a lot of fanfare when it was announced that GPT-4 had scored in the 90th percentile on the Bar exam. The context that was missing, and that has now become public, was that GPT-4 was scored against February test takers, who tend to score lower:

This is not a thing I knew about the Bar exam, and I expect most non-lawyers don’t know that either. Even though humans are often aware of context when sharing information, it doesn’t mean they’re always going to share it if it undermines their argument. If context isn’t offered, we need to ask for it – especially if you’re a journalist reporting on any aspect of AI. 

We also need to put these obviously false AI-generated response in context. A Google spokesperson said in a statement that “The examples we’ve seen are generally very uncommon queries, and aren’t representative of most people’s experience using Search.” Which is possibly true, but also very much misses the point. These results are being shared because they are funny, or contain obvious misinformation, or are dangerously wrong (please never use AI to identify a mushroom). I’m worried about the ones that aren’t being shared because they just look… normal. I wrote about this in my last post; that summary about ASL being a “broken language” may not appear obviously wrong to someone who is unfamiliar with ASL or historical discrimination against Deaf people. 

Without recognizing this context, we’re creating the perfect set-up for Google’s AI to become a confirmation bias machine.

Thanks for reading! Want to work with me to help your community develop the skills they need to be savvy consumers and creators of information? Get in touch!