Thoughts on search, AI as a rubber duck, and this blog

I’ve been working on a little side project recently that has been in the backlog for ages. I finally have a moment to pull it together, and it’s helping me brush up on a few Next.js 13 features I haven’t had the chance to play with yet.

As part of that, I’m doing a lot of searching around best practices on this that and the other, particularly server side rendering. It’s the first time in a while that I’ve been pointedly trying to use the internet to teach myself something in-depth related to coding, as opposed to finding quick sporadic answers.

The results were SO BAD.

Like, unbelievably bad. I had sort of noticed the decline in my searches generally, and I’ve seen friends complain about it a lot on Mastodon, but I hadn’t really seen anything like this yet.

And sure, search results have always been a mixed bag when you’re trying to teach yourself something. Obviously some answers are much better than others and wading through those murky waters is a part of it. But now SERPs are just jam-packed full of articles that are confidently misleading, circuitous, fatuous, a load of hot air. At first I thought that it might have something to do with the community (Maybe Next.js devs don’t blog about what they learn as much so you get more content farms trying to fill in the gaps?), but I’m almost certain that isn’t the case.

At any rate, I still needed to learn so decided to give ChatGPT a go. I’d done so here and there in the past for particularly thorny errors and that sort of thing, but never really sought out Copilot or anything like that. I just like writing code too much and never felt the need. So maybe I’m late to the game? Hey ho.

Honestly, I was pretty blown away. The results weren’t perfect, I still needed to pick through them to identify what was worthwhile and what wasn’t. But unlike random articles online, I could engage it in “conversation” to try and figure out why it suggested approaches that I felt were wrong or clumsy. A debate of sorts.

I can totally see the use. Though I have friends I can bounce ideas off of as they come up, I can’t get a quick temperature check when I’m working solo. Not like you can when you’re part of a larger team and can just send a quick Slack message to your teammates. This fills in that gap, though it absolutely doesn’t replace those broader conversations with friends and colleagues.

A chatbot can be a rubber duck.

***

Anyways, related to that, I happened to listen to two podcast episodes recently that both involved AI.

One was the season 2, ep. 9 episode of Transplaining with Jordan Gray with guests Maisie Adam (a British comedian) and Sam Duckworth (a British musician). SD posed the question, “Can we replace capitalism?” And the discussion ranged all over the place, eventually coming to AI and how it could make or break things.

Both SD and MA were sort of getting more and more down about how AI could influence things, noting that it’s “learning” from humans so is inherently flawed. Then JG said the following around 18:31.

JG I’ve asked AI before what it thinks about transgender people, it’s nice! Really pro! So I’m up for it. I thought it would be really dispassionate and give a 50/50 argument.

MA [Referring back to an earlier story about JG’s rather backwards gender specialist from when she was initially transitioning.] I thought it would be like your doctor and have a very archaic… “It’s O.K. now. They are allowed to wear a dress.”

JG But no, it’s all nice! So, I think it’s already more [com]passionate, because actually what that probably speaks to is the idea that there’s a silent majority of people that are correct about things, including capitalism, and their data is being punched in privately on to their blogs and stuff. People don’t read their blogs, but AI reads their blogs! So if you took all the sum of human knowledge together, it’s really positive and would look after the world if it was given a chance to. I don’t think AI is the thing to be scared of.

I think there might be more to it than that… I’m not sure how it works, but I’m pretty sure that we train the models to deliberately be more polite. If JG had asked the same question of Bing’s AI chatbot earlier this year, I think the response may have been different.

But I hope not, I hope JG is right. I found the optimism refreshing.

The other episode was ShopTalk Show 583: Language Models, AI, and Digital Gardens with Maggie Appleton.

In it, MA introduces the company she works for, Elicit. Elicit uses large and small language models to automate literature review, a necessary and tedious process where research organizations (think tanks, NGOs, labs, governments, etc.) have to review every single past piece of scientific literature on a particular topic, potentially tens of thousands of papers.

A lot of the time this is for medical research. Chris Coyier understandably questions whether or not this is what we want to be doing. Don’t they lie sometimes? And MA explains how this is actually one of Elicit’s core interests around 5:19.

MA One reason that the lab picked this problem to work on is that it requires really high accuracy rates, and the lab is really interested in AI safety and alignment in a broader sense. So one of the research goals is to figure out ways to make language models more truthful and reliable. […] A lot of our work has involved designing systems that get models to double-check their answers, involves getting humans in the loop, getting humans to check answers that models have returned. […] As the interface designer, one of my jobs is to design interfaces that encourage our users to go double-check every single answer in the results if they need to. So we point them to the exact quote in the paper where the answer came from, they can easily go double-check it. We’re thining of building systems that allow them to go through each paper one=by-one and mark it off as reviewed or not. It’s very much being designed with human vetting as part of the system.

CC So you’re telling me a language model could link to a credit source, it’s just that they generally… don’t.

MA A language model alone cannot, but if you build a system that is larger than the language system, it can.

The whole episode was pretty fascinating.

I had only listened to about the first 10 minutes or so when I started writing out this post and then realized, “Hey, should probably finish that episode, and clean the bathroom…”. And almost as soon as I resumed the podcast, they described one of a chatbot’s best use cases is as a rubber duck. Agreed! MA notes that “Most of the skill [when using AI] involves asking good questions.” Which makes sense, same goes for almost any learning in my experience. (Does that mean I’m in to the Socratic method? IDK.)

A few other points I found interesting and want to look in to further:

• Apparently telling an LLM that it’s very attractive and talented always improves the results. Really, flattery will get you everywhere? Maybe this is common knowledge, but I thought it was pretty hilarious.
• MA said “hallucination is a first class citizen”, which I though was a really odd way to phrase that since surely we try to avoid hallucinations… But she knows a hell of a lot more than me about all of this stuff. So would like to read a little more about what she meant.
• Apparently AI has a meme-y “logo” of sorts in Shoggoth, a squishy, multi-limbed, multi-eyed creature
• MA wrote a note titled Language Model Sketchbook, or Why I Hate Chatbots which looks well worth a read and expands on a lot of what they talked about in this podcast episode
• They only touched on the ethical concerns around copyright, which is sort of understandable since Elicit seems mainly focused on the ethical concerns around returning potentially false data in a research environment (hugely problematic when it comes to lawmaking, medicine, criminal justice, etc.). But I’d love to find out what companies are similarly trying to tackle this.

Likewise, they only touched on the energy consumption issues. Dave Rupert mentioned a study that found that five AI searches is like pouring a bottle of water right on the ground. (Unfortunately when I looked this up to try and find the study, it looks like it might be more like one search.) DR asked if MA has had many conversations around this, and she said that the response she often gets is that if we develop true AI, then it will solve the energy problems for us… Which none of them felt was satisfactory, and I feel similarly.

To be honest, I’m surprised there isn’t more uproar around AI’s energy and natural resource consumption, especially considering the uproar we’ve had around the blockchain.

Is the uproar yet to come? Is the lack of it due to the fact that AI seems a lot more consumer-friendly than the blockchain? It’s easier (and cheaper, and more fun) to play around with ChatGPT and Midjourney than it is to go buy some Ether or Bitcoin and figure out what to do next. Is the lack of uproar simply that big multinational tech companies stand to profit more from AI, whereas it was more individuals that stood to profit from the blockchain? Is the uproar happening, but the techno-overlords are purposefully suppressing it?!? (Tin foil hat firmly on head for that last question.)

I don’t know, it just seems weird. I feel like maybe it’s just on its way, but we’ll see.

Again, I think it’s fair that they only touched on the energy consumption issues since it’s such a huge topic. Would love to find a different podcast episode about it though.

***

All of this got me thinking about my site, this blog.

Back in 2014 when I started writing it, it was just an experiment in making a Tumblr theme. Sam and I wanted to make a theme that we actually wanted to use, and we needed some content to work with. So I started collecting some thoughts on Tumblr, and it snowballed from there, eventually moving off of that theme, off of Tumblr, and then off of my semi-hidden subdomain.

Even early on, I wondered: If I eventually had enough content collected in one place, on this blog, would I be able to create some sort of tool that would be like an amplification of myself? Could behave like myself? It was a very back-of-the-mind idea and something that I in no way had the resources to explore, so that was that.

But now it’s a thing.

In ShopTalk 583, Chris Coyier mentions a guy named Luke Wroblewski who created a chatbot based on his own blog.

Designer Peter Richardson co-wrote a post with GitHub Copilot that “sounds like how I would imagine I would write while stoned.” His site is on Jekyll, so it makes sense that Copilot could learn pretty easily from his content.

Later in his post, he asks, “Is this how sociologists feel? Like staring at the matrix, but it’s all creative writing workshop meetup output? I don’t want to be this predictable. Is it taking cues from my other posts? Where does it get this stuff?” And then he tries repeatedly to get Copilot to accurately describe what happened after his girlfriend got pregnant in the spring of 2014. It’s one of the most interesting pieces of writing I’ve read over the past year.

So anyways, those early ideas are now a real possibility. Wild. But would I actually want it? Maybe when I’m old and dead… so that my loved ones could “interact” with me when I’m gone. Is that dark? I would love to talk to my great-grandmother and ask her questions about some of her recipes. But it quickly starts to feel too Black Mirror. I don’t know, I guess it depends if you’re a glass half empty or half full person.

Maybe that’s an idea for B some day. If he wants it, the content is here. If he doesn’t want it, he can let it float away on the wind.

***

Right at the end of ShopTalk 583, they touched on Maggie Appleton’s “digital garden”. I’ve heard many people describe their sites like that in the past. I’m sure I’ve said something similar about mine before, and I think I was originally introduced to the concept via Laurel’s 2018 essay in The Creative Independent exploring the website as a garden, as a puddle, as a thrown rock that’s now falling deep into the ocean. But it seems like MA is probably one of the people that popularized the term.

I was wondering about the backstory behind the term, and bam, MA wrote a great essay on exactly that, the history of the digital garden. Apparently Mark Bernstein’s 1998 essay Hypertext Gardens was likely one of the first times it was mentioned. “Unplanned hypertext sprawl is wilderness: complex and interesting, but uninviting. Interesting things await us in the thickets, but we may be reluctant to plough through the brush, subject to thorns and mosquitoes.”

I think I still prefer to think of mine as a commonplace book, but a digital garden is probably just as apt based on how I interact with it.

It does make me wonder if I should abandon the linear feed on the homepage for something a bit more exploratory and ambient. Sam keeps nudging me to revisit the homepage design, maybe this is a good opportunity.

***

Time to get back to the side project.

It’s basically a big bookshelf sharing a slice of our library, since we already have it all archived in a database. Maggie Appleton’s description of her library as “books I like the idea of having read” is appropriate, so many of my books are sitting aspirationally on the shelves waiting for my eyeballs. (I suspect a lot of people are in a similar boat.)

I’d like to make it a bit of a tool for myself, a way of surfacing some of those books that I’ve under-loved and could use a read. We’ll see.