Uncorrected Transcript: "Search and Ye Might Find" from 99% Invisible

Uncorrected Transcript

This is going to sound like a stand up comedy bit, but I was on my way to the airport recently and I needed the confirmation email from my flight so I could get my boarding pass.

So I pulled out my phone and I typed in some keywords into the search bar of the mail program.

I started with the airline I thought I was taking.

But that search pulled up mailing list, promotions and old flights and I know if I searched long enough and hard enough I could eventually find it. But the thing is by now I'm at the airport and I just have this naive conviction that searching should be better and easier. But I often come up empty.

There are days when I feel like I've control left up my life into oblivion and I'll never easily find the email. I need much less old photos and documents on my hard drive. The good news is I'm not alone. Adam Rogers feels this particular pain too. You know, you think email is a problem. So try to find a slack, you once wrote, you know, try to find a tweet, try to search for a photo on flicker instagram Pinterest.

Like I dare you Adams a senior tech correspondent at Insider.

He's been thinking and writing about what's known in the industry as search for last decade.

People have been grumbling about not being able to find things online, both in our private data and on the public web.

I would say to people, I think search sucks and other people would say, oh my God, it really does.

For Adam, the stakes for this go beyond me not be able to find that one email, I will tell you, I'll tell you a personal story.

Um that was one of the ways that I started to think about this.

My grandmother was a professor of Library science at UC Berkeley.

When Adam's grandmother was alive, she lived in a small apartment, every wall was filled with rows and rows of books. She was a librarian. They were really well organized.

The books held all the knowledge that Adam's grandmother wanted to access.

It was arranged by topic and author, complete with important search tools, notes and tabs stuck into all the various volumes that she could reference when looking to pull up some tidbit of information.

Adam was charmed by the whole thing, but as she got older and was a little bit less coherent, she didn't remember stuff as well.

And the notes became less and less coherent and uh I find that terrifying.

Adam's grandmother had built a working search function and it failed all the same.

Millions of pages of information were still there lining the walls but she couldn't access this archive of knowledge according to Adam, the same thing could be happening when we search for anything on our computers more and more.

We outsource our information, our personal information and are autobiographical memories.

We outsource that more and more to digital media now, not just the sum total of human knowledge, but also our own photos, emails and memories stored in the cloud as we put more and more of that somewhere else.

Besides, our own heads, search becomes more and more of a existential crisis, not just an annoying thing where I can't find the email with my confirmation number in it, we begin to potentially literally forget ourselves.

Our ability to search and retrieve information at our whim feels like one of the most important developments of the digital age.

So how do we get to a point where it feels like search is failing us and how do we fix it before?

It's too late today. The word search sort of fuel synonymous with the word google.

But I think it's actually one of the oldest design problems in the world from the time humans started writing stuff down the struggle has been how to organize it all so that its contents wouldn't be law lost in the stacks.

Search has always been an attempt to fix that problem.

I think for our purposes here, it does make a lot of sense to think of search as a designed experience.

It wasn't just walking into a library and wandering around ever since, you know, the priests who ran the libraries of alexandria knew which scrolls went in which cubbyholes or however they did it And leading up through the dewey decimal system or Library of Congress numbers. These things are search. These things are what makes search possible long before we could search for things online.

Google was essentially a person, a reference librarian.

If you wanted to find something on say growing vegetables, you could go to the gardening or farming sections of the library.

But in the thousands of books in that huge section you'd quickly get overwhelmed.

That's where reference librarians and archivists come in.

They take your topic and help you narrow it down even further, applying their own nuanced knowledge and specialized training to help you search better and find exactly what you're looking for.

That's how search operated for centuries by topic mediated by human to human interaction and it works pretty well.

Good evening Doctor.

But eventually this easy flow of search hit a snag.

In the mid 19 forties, the snag was highlighted by an american engineer named Vannevar Bush.

Sir, I've known you a number of years and last week I managed to mispronounce van ever and I apologize Well you've got plenty of company. They read most everyone mispronounces it.

I think the record was held by one Coast master who managed to pronounce it four different ways at the time, Vancouver was convinced that problems with search were hindering human progress.

He pointed to Mendel's ideas on genetics.

For example, these ideas were lost for an entire generation, he said because they were buried in the avalanche of newer research.

There was no efficient way for people to parse through all that information but never was a big believer in the potential of machines.

The analytical machine which will supplement a man's thinking methods which will think for it.

We'll have as great an effect on his grasp of the world and his access to data and so on.

His manipulation of will have as great an effect in that way as the invention of the machine.

Way back took the load off of men by giving them mechanical power instead of the power of their muscles.

So in 1945, van ever took to the page and dreamed up an imaginary futuristic solution to the problem of search.

A machine called mimics. The mimics would make search easier. It would look like a desk.

There'd be a keyboard, viewing screens and storage space for all of human knowledge as long as it was on microfilm and could fit into a dust drawer on the left side there would be all the information in the universe and it would all have links.

And then on the right side you would follow those links for the information you wanted.

So the search became about connections within the what you were looking for theoretically the user could teach the mimics which words were relevant to each other.

So if the word vulture and one document makes me think of death, I could tell the mimics to connect those two words.

Then when I search for the word vulture, all the documents featuring death that I previously linked would show up, I could scroll through all the results by turning a crank.

In essence, the mimics user could build their own little analog algorithm for search.

The mimics was never built but paul con, someone who wrote a whole book about jennifer's mimics.

Well in the 19 nineties he animated how the mimics might have worked 1st. He runs through an encyclopedia finds an interesting but sketchy article and leaves it projected next. In the history he finds another pertinent item and ties the two together. Van ever called this process of linking two key words together. Trail building.

Thus he goes building a trail of many items.

Occasionally he inserts a comment of his own, either linking it into the main trail or joining it by a side trail to a particular item.

The idea of searching documents not by broad topic but specific word and then linking related documents together turned out to be right on the mark.

About 50 years after we never dreamed up the mimics his ideas about search came to fruition with a little thing called the World Wide Web.

This mimics inspired idea of searching by keyword became the new default.

Whereas searching by topic like you would in a library is similar to looking through the table of contents of a book. Keyword search is like using the index. It is much more precise and searching by keyword worked well for a time when you type in a word and get only a couple of dozen results.

But as the digital information space gets bigger and bigger, things begin to evolve and change in 1993 there were 130 yes, you heard that right? 130 websites on the Internet. Three years later, there are over 100,000.

So that starts to happen very soon when the World Wide Web becomes something that's more than just an academic, more than just the thing that connects all of the physics labs.

Doing an online search by keyword was now posing problems because the internet was starting to feel crowded and clunky.

And the usual ways of searching, we're feeling outdated.

The kind of searches that I might have done on lexisnexis proquest in the eighties or nineties, these carefully constructed searches, we had to pay per search.

So you really wanted that one search to work because you couldn't keep doing it because it cost money.

And that felt like a very constructed and constrained experience which brings us to win search as we know it today was born a couple of kids at stanford come up with another approach to searching called Page Rank and that becomes the basis for google google runs using Page Rank and what Page Rank says is we're going to count how many things linked to something.

The idea behind the page Rank algorithm is not only will your search engine find the sites featuring whatever word you asked for, but it will find the best sites with that word we're going to say that if something has a lot of links to it, that's a better source of information that that we now prioritize that above things that have the same kind of information in them.

But don't have as many links as the internet expanded. More and more spammy or irrelevant.

Web pages would come up in search results.

Page rank help filter those out by emphasizing high quality results A dot E D U. Link or anything from the BBC for example would rank higher in search results than say and in the blog in part because many other sites would be linking to the more well known pages. To me. This is super cool. I love this as an insight because it turns out that that's the same way that like memories are reinforced in the brain.

How many times do we go back to them?

The same way that ants tell each other there's food over there, but not over there.

If an ant walks over the trail and leaves pheromones on it again and again and again, that trail becomes more important to the colony pretty soon with the help of page rank google became a verb.

And since everybody goes to them, they have statistics on what people search for and how they search for them.

So they have this incredible statistical Um you know, they get billions and billions and billions of searches every day and only about 15% of the searches that they've seen a given day.

Our new that they've never seen before.

So 85% of the searches that the world does on Google every day are things they've already seen.

They've seen that search before and they can reproduce it, type in some words and here's the stuff here's access to human knowledge.

Google is the biggest, bestest, most innovative website.

Well, I guess search engine in the galaxy.

Here's a clip from a TV segment in the early 2000's offering its viewers tips on how to search on Google.

The hot new search engine in town.

So we're here to pump you up with a few tricks that will bring that power back to your browser.

There's seven tricks in tow. The first one really easy to do. Open up google here in Mozilla by the way.

They just released a new version of Mozilla Mozilla dot org. They're up to 1.2 now. Very, very good browser time with google at our fingertips and the proliferation of search bars on every digital interface.

It felt like we had finally made it to the very top of human knowledge and it felt like not a constrained experience. It felt like, oh that's done, that's fixed. It works.

In fact the google search bar with all of its millions of data points is so good.

It changed our expectations of what search is and today that's part of the problem.

We were all trained very well to think well now search bars are just like the google search bar everywhere and everywhere.

I see a search bar, it's going to be just as good as a google search bar is and then you try that on amazon.

For many of us when we type a query into an e commerce website, we expect that the results will be ranked for us by relevance to our search but that is not how it works.

So a place that's trying to sell something is trying to sell.

Like if it has more of one thing in its warehouses than another, it'll try to push that onto you.

If it has something that's on sale, it might show you that first.

If it has a product where the people who make it have a pay for play deal with the e commerce site, it'll show you that stuff first.

The result is that the thing you search for that you're trying to buy will be buried by results for stuff that the company wants you to buy.

You can put in the specific name of something in the search bar on amazon and it won't show up until you scroll two or three screens down and that, I mean I'm still shocked by that I'm naive or I'm stupid or something. I'm stunned. Like no, I put in the name of the book. I know what book I want and you know you still gotta scroll to find it because there are all kinds of other competitive commercial pressures.

The same thing happens with private search when we're trying to retrieve something like a flight confirmation number or a vacation photo from the cloud.

On some level we expect every search bar to work like google's even though it isn't google.

So personal information management turns out to still be kind of an unsolved problem.

More so than wide search.

Adam says that the reason searching your own files isn't as streamlined as web search is because your personal email or photo database is private.

Software engineers can't improve on algorithms as easily as google can on the web because with your private data there's less statistical commonalities to draw from a grain of salt here, Google supposedly doesn't release information about this. So we don't know for sure about all that. What we do know is that not being able to find our stuff can have consequences.

And that's a personal and also a cultural amnesia that I think becomes troubling you.

You lose moments from the past.

You end up only existing in the moment like where you where you share the stuff and then it doesn't exist anymore because you can't go back and search it in the years since google rose to prominence. It set the standard for what search could be. But lately that glow seems to be fading and in fact it turns out that almost as soon as it was born, google almost without us noticing began to fail.

I found it incredibly chilling to be talking to somebody who works on search at google and have that person say yeah search really isn't a solved problem and it hasn't been for a long time.

In the early two thousand's, google started getting bogged down by monetization and people trying to gain the system.

One way this played out was with S. C. O. Or search engine optimization.

SeO is basically the process of getting a web page rank higher in web search results.

In the early years of S. C. O. Web Masters would often stuff their web pages with keywords to get them to rank higher regardless of actual relevance and the strategies keep evolving.

I was just reading some folks today talking about this on social media about how like everything you look for on Google has these like 2000 word introductions of meaningless gibberish like before you get to the recipe or before you get to the instructions for changing the memory card or whatever.

And that's because one of the things that google came to prioritize it seems they're not that open about these things is how long somebody spends on the page, add all that up and this is what you get when you type in a web search query it can feel like you're playing hide and seek.

But for information to find the thing you search for first, you'll need to scroll past rows of stuff, labeled ads or maps and horizontal boxes of questions you didn't ask.

Another reason.

Search engines like google so often missed their mark is because unlike say a librarian's approach, which might be more like here are 10 books you could read to try to figure out on your own, Google tries to give you the most popular quote, unquote best search results.

In other words, a direct answer, it can sort of understand what's on a web page.

Find the information that it thinks you're looking for based on its statistical analysis of all of the billions and billions of searches that it sees all the time and feed you an answer and we now have come to think,

oh well that must be the answer then.

But when a search engine prioritizes the visibility of one seemingly popular answer over another, sometimes it can lead to misleading or even harmful outcomes.

Like in 2021, when the result of a single Google search sparked an uproar in India, an angry flashpoint has emerged in Karnataka.

After search engine Google showed Canada language in bad light, calling it the Ugliest language in the country, huge outcry was triggered after someone had typed the words ugliest language in India into Google and the search engine embarrassingly answered, Google named an actual language in India and insulted 43 million people in the process, outrage prompted Google to issue a detailed statement apologizing for the Gaff and said they would improve their algorithm.

But the Karnataka government has made it clear an apology is not enough.

So in light of all this search engines are evolving again to fix the issue of private search not working like the google search bar.

Some search companies are supposedly considering creating search engines that could dig through both your personal information and go out on the web.

You'll pay us a little bit of money and then it'll be private and we'll search your personal information as well as the web

and we'll do it really well.

So you'll be able to find your flight confirmation number and your frequent flyer number and the name of that person who you worked with that one time and all that other stuff that we search for in our own computer that one document and to fix the issue of constrained single answer results.

Google announced last year and a I technology that would attempt to understand not just what the searcher is typing but what the searcher is thinking meaning.

The search would no longer be driven by topic or keyword but by what the Ai thinks the searcher meant by those words you can have a kind of iterative back and forth because you'll have a search engine that actually understands language instead of just looks for keywords.

There's a promise in this and there's also a risk that as search companies start to behave more like humans with their own ideas.

They might continue to algorithmic lee nudge us away from our intended searches and into unexpected directions, which feels creepy and consequential.

If you think about all the life decisions that you make in a given year based on information you process after googling but google's vision of the future points to something already starting to happen in the world of search this time.

On the part of the searcher one move that a lot of people now use is they will use google with the keywords that they have to search for but tell it to only search Reddit increasingly people are turning to Reddit to search for information but they're using google to do it because read its own search function is supposedly not very good.

So you can use google to find the subject and then you have human beings actually answering your question on Reddit.

Someone looking for information on keeping their garden vegetables alive can eavesdrop on a conversation between niche experts in humid or desert climates for a particular plant.

Or they can engage in a real human conversation about say how much water does a thing actually need using google to search Reddit for human conversations. It's a little yankee but it may be the best solution for search that we have right now in a way it comes close to replicating the experience of talking to a good old fashioned librarian.

The thing that the conversation with the reference librarian, that was that sort of the classic model of search would give you is if you didn't really know what you were looking for, the reference librarian would help you figure out what you were looking for.

It turns out you were looking for something different.

Turns out there's a hole related thing, There's a whole other room of the library that has more that you might want between Google's new Ai and the wild world of Reddit.

The future of search is beginning to look a lot like the past.

Ultimately, the questions pertaining to search aren't getting easy answers anytime soon.

Questions like could there be a better way to order the world's information?

How do we organize the stuff that we know so that the next person can know it too? And what time am I flying out of JFK? Anyway, these are some big questions, the answers to which we may never know.

Coming up after the break, I talked with our Digital director, Kirk Cole said about his personal favorite approach to searching. So when we were making this episode last and I relied in part on another 99 p.

I colleague to help us better understand search both its history and, you know, workarounds to use search more effectively.

And so we called on kurt Kolstad Digital Director, there's a reason why he's called Digital Director.

And so he's joined me to go through, you know, both of those in a little bit more in depth.

I've been steeped in this world to search for a very long time now and like everybody else, I have to search for stuff.

But I also ran a number of web publications which relied in part on visitors from search engines for ad revenue. So, so yeah, it's fair to say I have spent some time studying S. C. O. Yeah, and that is search engine optimization, which I know, you know, kind of peripherally from having stuff on the web for a while. Um Yeah, so what is that like as a person who ran websites using sc oh, what is that all about?

So a lot of it is white hat, right?

It's like above board stuff like, hey, I've got a local architecture firm and when people search for an architecture firm in this area, I should show up in the search results.

But a lot of it is ethically much murkier and it's been that way basically from the beginning.

Okay, so what are some examples in the beginning of people using S. C. O. In a in a bad way. Right? So I have one of me using it in a bad way, which is that?

Yeah, I mean, I was a teenager, I am coming from inside the house. So in the in the nineties I built a fan page for a band that I will never tell you the name of um out of embarrassment. And one of the things I did was add in any word, I thought somebody might search in relation to the band like other similar bands and stuff like that. Just tons and tons of these words at the bottom of page in small thought and sort of half hidden, you know with the color that kind of match the page. I mean, I was you know, basically.

Yeah, well it turns out this is a well known tactic called keyword stuffing and it used to kind of work like google used to be a little bit simpler and we just kind of read what's on the page and try to deliver results. But of course these days the ai is much more sophisticated and if you actually try to do that today, google will not only catch you, but they will like potentially delist your site for doing something like that. Okay. So if the algorithm is getting smarter and catching these kinds of things, why isn't search just getting better and better. Yeah. So the search engine optimization industry is huge and they're highly motivated. So every time google comes up with something, they come up with something else and it makes sense if you think about it, like they throw so much money at this because if they can get their business to come up first in a search organically as it's called like without having to pay for ads or anything. It can mean the difference between success and failure, right? So everyone is forced to get more clever about their strategies and then google for its part has become a lot less transparent over time about what they factor into their search results and for good reason right to deter these unethical players who learn the rules and then learn how to cheat from them. Yeah, so it's kind of this arms race but also like things get more secret as part of the arms race as well.

Um and so you know given the current state of things, you know what is your advice for people trying to search and find stuff? Well it depends upon the kind of thing as you'd imagine. But I want to walk you and our listeners through my personal favorite strategy which honestly I used to search for everything from beard trimmer reviews to tips for taking care of succulents.

And so I'll start with the first example and that's called a transactional search right?

Like I'm looking to buy a beard trimmer.

I mean now that you mention it, I seem to need a new beard trimmer all the time.

So I'm apparently not buying the right beard trimmer. So so same here, they always seem to break. Yeah they do. Alright.

So I could start on amazon and type in beard trimmer and then sort those results by average customer reviews because that way at least hopefully you get the best ones first and that can work or I can search in google and clicked the shopping tab and you know, I'll get some results there, but here's what I actually do in google, type in something like best beard trimmer, then site colon, reddit dot com, and that way you'll only get the results from Reddit's website.

And for those who don't know Reddit is this huge website with tons of sub forums called subreddit and there are some of these that are really big like news and some are really niche subjects like men's hair care and I'm pretty much, you know, I follow what the logic of this is, but I guess my question is um why would you trust read it over, say like reviews on amazon or a third party website?

It seems like it's still user reviews.

Yeah, Yeah, I mean, that's a great question because, you know, like you wouldn't think like it just seems kind of random, right? Um but the reality is that review websites, particularly the ones that have gained their way to the top of google are a crapshoot, you might find legitimately good and helpful reviews or you might find somebody who's writing about a product specifically because they'll get a commission if you click a link on their site and buy that product and then on amazon reviews have become sketchier and sketchier over the years for various reasons that some of which we talked about in this episode? You can feel it, you know what I'm saying when you look at it. Yeah, you really can write. But on reddit there are these sub credits for virtually anything and gaming every related community to a certain product or or something. It would actually be pretty hard to. Maybe someday a eyes will figure it out, but for now it's kind of out of reach for most S. C. O. S. And with that beard trimmer search for instance, I put this in and I landed on conversations in a subreddit called One bag, which is a community I didn't even know about, that's all about minimalist travel. So that's like that's cool, right? That's that's relevant. Like those guys probably know what they're doing. And then I also found a search result in buy it for life, which as you can imagine is people discussing high quality things that hopefully will last the rest of their lives, right? Not cheap beard trimmers like ours that keep breaking. Um and the key part of this is there's no financial incentive for these people, right? Like they're they're not linking you out to products with commissions, they're just communities discussing these things openly and they might not always be right, but there is this voting system and so good things tend to rise to the top there. So if I were to do a search like this, would I want to look for the highest voted result? Is that how it works? Yeah, I mean that is a good place to start at least and beyond that to usually there will be responses to that top voted thing and people will discuss things back and forth, pointing out downsides or suggesting alternatives in these comment threads.

Hmm, wow.

I mean, I think I've noticed this happening more and more that Reddit has been part of my searching for information and it's really kind of makes a ton of sense.

Um and so that's very cool.

But I never thought of it as something so practical is looking for a beard trimmer.

It was always like a piece of information or people discussing. Yeah, totally, totally. And that's the sort of other domain of search.

Right? Is this category that's called informational Search and read. It is really good for that too. So again, sort of starting from the beginning here, it's like you could go to Youtube and look for a tutorial about how to take care of your succulents and often you'll get like a pretty good results, you know? But if you're just watching one video, you're getting like one person's opinion and like review websites, they're also usually monetized. So there could be like a conflicting reason that they're, you know, making all these videos right? They might not really be experts. And you know, that obviously can work against your need to find impartial and useful advice. Right?

So I assume this could take us back to reddit where we could actually like some evaluation of even these videos. Yeah, exactly. And it's basically the same process as before. Right?

You search for a few keywords, you limit your search by putting site colon reddit dot com.

And I would argue that as good as Reddit can be for product reviews, it's even better for expert advice because there's so many of these tiny niche subreddit about everything.

So take succulents for instance, there's a huge gardening subreddit, a smaller indoor gardening one and an even smaller one that's just for succulents and for some types of succulents, they even have their own whole community, which to me is crazy, but kind of beautiful too.

So this all makes sense to me because you're really relying on human knowledge and voting and all the types of things that we've counted on since Web 2.0 came about.

Um but my question is like, you know, you can type a search into search bar on reddit, so why are you using Google to search?

That's a great question too.

I mean I've tried that, believe me, I've tried that a lot of times, but red, it's built in search is in my opinion a mess.

And it does offer various ways to search.

So it kind of looks sophisticated like you can search within a single subreddit and you can sort results by, you know, the age of the post or the vote count on it.

But ultimately search ability has never really been the company's focus and it shows when you actually try to search the site.

So by using google, you're combining this really powerful search engine with this really remarkable wealth of personal knowledge, right?

You're cutting out all the bad results of like random review websites and just going straight to the site that often has good information and of course it won't work for everything.

But it's a starting point for so many of my searches these days.

And I should point out though that the sort of third major category of search is something that google actually does do very well and that's called a navigational search basically, if you know what you're looking for, you're looking for a particular website or a person plugging that into google is often a great way to find them, right?

Yeah, I mean this is what we all do, but for these informational and transactional queries, I do suggest starting with google plus reddit and then just going from there, wow, this is great. This is like real news, you can use stuff right here. Yeah. So I'm going to give the hybrid search a shot, I'll let you know how it goes. Excellent. Excellent.

Yeah, I mean, yeah, rarely are we so practical in our, in our episodes. Um but yeah, hopefully people find this useful.

Um I should mention one other thing before we go.

And that's that for some people read it will set off a bit of a red flag because the site is a hole is a very mixed bag.

It's sort of like its own version of the World wide web with warts and all.

So for people who have a negative association with Reddit going in, I hear you and you're right, there's a lot of toxicity on that website.

But if you search hard enough and avoid the bad neighborhoods, you can find great little communities.

To duly noted.

I have noticed the wards, I also have noticed really nice communities are subreddit is beautiful, I must say exactly, I like that.

I like to hang out in the blankie subreddit, you know. And I also just think the front page is really good and I'm you know, that's and and people are funny.

You'll find a lot of good stuff on there too.

But it's duly noted like let you know like just be careful out there, have your wits about you and try to find the best information possible.

All right, well thank you so much, kurt, this is great, 99% Invisible was produced this week by Law Madan edited by kelly Prime and executive produced by Delaney Hall.

Original music by Swan real sound mixed by american. Entre fact checking by Liz Boyd kurt Coulson is our digital director team includes Vivian Lay, Jason de leon chris berube Christopher, johnson, Emmett, Fitzgerald, martin, Gonzalez, Jacob maldonado, medina joe Rosenberg, Sophia clouds and me roman mars Special thanks this week to Adam Rodgers for being adam Rogers and a warm welcome this week to our fall intern Olivia green.

We are part of the stitcher and Sirius XM podcast family, now headquartered six blocks north in the Pandora building in beautiful uptown Oakland California.

You can find the show and join discussions about the show on facebook.

You can tweet me at roman mars in the show at 99 p. I. Org on instagram and read it to.

You can find links to other stitcher shows I love as well as every past episode of 99 p. I at 99 p I dot org.