Earlier this evening, I misspoke at the dinner table. I said something about the word ‘agua’ being French. My daughter quickly said, ‘No, it’s Spanish!’
And because I’m a Dad, I decided to lean into the con. I said, ‘No, it’s French. I can’t remember what it’s French for, but I know it means something in French.’ And while I was saying that, I typed into my phone, as if I was Googling what it was for.
Instead, I was typing the first message below into Gemini. It replied with its first message, written instead of spoken. Then, acting exasperated at my failed Googling, I just said “Hey Google” and spoke my second message out loud. It responded with its second message, also out loud for everyone at the dinner table to hear.
My kids acted amazed. My wife asked, “Wait, really? Is it spelled the same way?” in completely justifiable disbelief. Then I showed them the conversation.
Then, I started to tell my daughter, “See, you can’t believe everything you see on the internet!”, just like we’ve been told for the last 25 years. Except I realized that with the ubiquity of AI nowadays, it won’t always feel like these conversations are “on the internet”. In fact, with the rise of on-device micro-LLMs, they won’t always even be on the internet. So instead I tried to change at the last second to say, “You can’t always believe AI!” Except I’d already said, “See, you can’t believe everything you see on…” And so what I ended up saying was:
“You can’t believe everything you see on the ainternet!”
…pronounced ain’t-ernet. I was trying to merge “AI” and “internet”, but I ended up merging “ain’t” and “internet”.
And it instantly occurred to all of us: that term is the perfect description of the problem facing the modern internet, where AI slop is running amuck, where bots in comment threads and email scams are getting harder to distinguish, where creating something to superficially mislead and lend false credibility to a lie literally takes just a single message.
In the past, you couldn’t believe everything you see on the internet because people lie on the internet all the time. Now you can’t believe everything you see on the ainternet because much of it isn’t—or rather, ain’t—even written by a person at all. It’s generated by a bot following some prompt or instructions from a human user so far removed from the bot’s behavior that it may scarcely even be considered to be acting on the human’s behalf.
This is similar to the dead internet theory. The dead internet theory posits that for the last decade, the vast majority of content on the internet has been auto-generated by bots and algorithmically curated to “control the population and minimize organic human activity”, as the Wikipedia article succinctly puts it as I’m writing this. And what’s particularly remarkable to me is that the dead internet theory started to gain popularity in 2021—before tools like ChatGPT and Gemini made generating fake content so trivial.
But while the dead internet theory is a conspiracy theory that this shift is being deliberately perpetuated by state actors, I don’t think we necessarily need to go that far to see what’s happening. We can explain a lot of this just by simple economics. We exist in an attention economy with widespread data gathering and optimization; so long as human attention is considered so valuable, there will be efforts to optimize capturing as much of it as possible. Over the last decade that’s been increasingly savvy algorithms for content curation, but those relied on content existing in the first place to be curated. Using AI to custom generate the content that will hold as much attention as possible is just a natural consequence of that value and this new technology.
And that’s why the old adage that you can’t believe everything you’ve seen on the internet has been pushed into overdrive: because it’s not just that someone might be trying to lie to you to forward their own agenda. Now, the algorithms themselves may be telling you any lie they calculate will capture your attention because getting your attention is their agenda.
But it ain’t true. It ain’t authentic. It ain’t real.
When I was a kid (eighth grade, I think?), I went to summer camp in Michigan for about two weeks. At summer camp, as boys at summer camp often do, I met a girl. We spent most of camp together, then traded emails and phone numbers (home phone numbers, because it was around 1999) and whatever else and promised to stay in touch.
I flew home from camp, got off the plane in Atlanta, and… she was standing at the end of the jetway. Her original flight to California had been canceled, and they rebooked her on a flight connecting in Atlanta, but then her original flight was delayed and she missed her connection. So, she was in the Atlanta airport and didn’t know anyone or what to do.
(Sidenote: it’s sort of bizarre to imagine flying as an unaccompanied minor without a cell phone in 1999.)
So, she and I left the airport, met my parents, and explained the situation. We drove to our house, where she called her parents. If I recall correctly, she explained, “No it’s okay, I went home with this guy I met at camp.” I’m pretty sure her father was going to somehow commandeer a plane and fly to get her at that moment (pre-2001 security, after all), until my parents got on the phone and explained things. And then she was booked on another flight later that evening, so we drove her back.
But that few hours in between was the most surreal experience: someone who I had spent two weeks with in a completely different location, familiar to us but unfamiliar to anyone else we knew (except people we met at camp), suddenly in another place that was deeply familiar to me but completely unfamiliar to her.
There’s a word sort of related to this: context collapse. And that applies to the fact that suddenly my parents and this girl I’d met at camp were together in the same place, but that doesn’t quite capture what was so surreal. What was surreal wasn’t the merger of multiple audiences. This was more about the place: it was about my home turning from my home into a place that I show and explain to someone who I only know from a very, very distant place.
A few years later, I went to Hilton Head Island on vacation with my then-girlfriend’s family for the first time. This was an annual trip they had done for years and years before with over 15 family members, to the same hotel, on the same dates, and so they had their routine and cadence for how everything went. I remember her mentioning how weird it was for me to be there, and—being a silly teenager—I was probably a bit offended. But in retrospect, I can see exactly what she meant: this place whose context had always been a family vacation was taking on a different feel. It wasn’t about the combination of people—I’d met her family for several holidays in the past—but it was being in a place that carried a particular context and having that place take on a different context.
Around that same time, I participated in a summer theater trip with my school. We went to the Edinburgh Fringe Festival to perform a theatrical version of Pilgrim’s Progress, and stayed for a month (I think?) on the campus of the University of Edinburgh. While there, we lived sort of a dorm life: hanging out with classmates, rehearsing in one of the common rooms, eating in the dining hall, etc. 15 years later, I traveled to the Third ACM Conference on Learning @ Scale, only to discover it was also on the campus of the University of Edinburgh: this place that had become familiar to me (if now only distantly) took on a different feel, shifting from a place for high school kids enjoying a school trip to a place to engage with in some vague professional capacity.
All of these experiences had something in common: a sense of a singular place—my house, a resort at Hilton Head, the campus of the University of Edinburgh—that had become almost synonymous with a certain type of experience suddenly being thrust into a different kind of experience. It’s sort of like context collapse, except context collapse is focused on the people, whereas these experiences are characterized more by the places. And it’s a really surreal feeling.
The reason I bring this up is because I’m in the midst of experiencing this again, and I wanted to find a word for it. It feels like the kind of experience for which there would be some obscure German or French word, and one comes close: the German unheimlich, which refers to an experience that is simultaneously familiar and unfamiliar in an unsettling way. But unheimlich has a negative connotation, and this feeling I’m describing is neutral at worst. And I thought maybe there might be a word in a book like The Book of Obscure Sorrows—still one of my favorite books of all time—to refer to this, but I’m not at home so I can’t check. So, I turned to AI for some brainstorming help.
Here’s the experience right now: I’m at the Online Learning Consortium’s 2025 Accelerate conference. While it’s rotated amongst the various Gaylord hotels in previous years, this year it’s at the Swan & Dolphin at Walt Disney World. The Swam & Dolphin, interestingly enough, is part of the same resort complex as the Disney Boardwalk Hotel, which is where my family and I have stayed the last three times we’ve been to Disney World. We’ve loved staying here because it’s within walking distance of two of the parks, and has a lot of sort of pop-up events nearby and nice areas to explore. So, we’ve become quite familiar with it: it’s amazing how only three trips totaling maybe 15 days total have embedded it in my mind so well.
But now, rather than being here with family, visiting parks and getting coffee from the cafe downstairs and visiting the gift shop for various sundries, I’m here for work. I have a well-established work travel cadence. I have a well-established vacation cadence. They’re two very different mindsets. And yet, to transfer the work mindset to a place that I’ve only ever associated with vacation is… well, surreal. Uncanny, but not in a bad way.
I feel like it warrants a name. It could describe a lot of situations. A new teacher going from sitting as a student to standing in front of a class as a teacher. A former child returning to their childhood home with their significant other for the holidays (especially if they sleep in the guest room!). A newly-promoted boss sitting as the denizen of their new office instead of a visitor. Dropping your kid off at the college dorm you yourself lived in. Giving a guest lecture at your own former university. Visiting an office you used to work in.
Or, since I’m at Disney World right now—perhaps a cast member visiting Disney World as a guest?
I brainstormed with the help of ChatGPT, and we landed on a term I think I like: alterlocus. I like that term because I feel like it emphasizes the place rather than the individual feeling: it suggested a lot of terms related to one feeling one’s own role shifting (roleglitch, self-echo, identity overlay), but I feel like this is really about the place and one’s relationship to it, not just one’s own feeling in the place. I like the term alterlocus for this: a place where a different version of yourself remains present, and could in theory come back.
I feel like the term could be a noun referring to a specific place—”The Swan & Dolphin is such an alterlocus for me because I’ve been there both for vacation and for work”—but also a noun referring to the feeling—”I had such a sense of alterlocus when I walked up the stairs of Glenn to move my son into college.”
Maybe if there’s a new edition of The Dictionary of Obscure Sorrows, I’ll find afterlocus in it. If so, allow me to provide a succinct definition:
Alterlocus: The uncanny feeling that a place still holds a different version of you—a version which may yet return in the future, and a version that has a very different set of responsibilities and priorities instantiated in that shared place.
Maybe the real strategic plan is the friendships we made along the way.
In fiction, a MacGuffin is generally some sort of object whose sole function is to motivate other action in the story. In order for the story to happen, the characters have to be chasing something, so the characters have to really care about the something. But from the audience’s perspective, the MacGuffin itself probably doesn’t matter: it just matters because it motivates the story that they actually care about.
MacGuffins are extremely common in stories. There’s Orion’s Belt in Men in Black. There’s the Philosopher’s Stone in Harry Potter. There’s the Nethicite in Final Fantasy XII. There’s the holy grail in both Indiana Jones and Monty Python‘s respective quests for it.
Or my favorite example, from Grumpy Cat’s Worst Christmas Ever, after the introduction of a million-dollar dog that the rest of the characters are going to spend the rest of the movie chasing: “That sounds like a MacGuffin to me!”
Now, I bring this up because for the last couple years, I’ve noticed something in some meetings and committees I’ve been part of. These committees usually have some kind of product they’re seeking to produce that motivates getting their constituent members together. They meet because they’re tasked with drafting a section of a strategic plan, or making a recommendation about some policy change, or generating a response to some recent event. Even if they don’t have a clear deliverable product, they have some kind of clear motivating reason to get together.
For many of these committees that I’ve been on, we’ve done our work and generated our recommendation or written our report… and then I can’t honestly say I have any idea where that work went. I don’t know if it had any impact. Sometimes I’m not sure it was ever even seen. Others, the decisions themselves could have just as easily been made over email. (And to be clear: this is just as prevalent in all sorts of external advisory boards and task forces and steering committees. This isn’t a Georgia Tech thing.)
Does that make these meetings wastes of time? Actually, no—and I say that as someone who hates unnecessary meetings. The true value I find in these meetings is that it forces me to interact with and get to know people from other departments, universities, and fields that I never would work with and get to know otherwise. There’s enormous value in the networking and socializing that happens as a byproduct of these meetings.
So if those secondary effects are so valuable, why not just make them the primary goal? Why not have meetings where we just say, “hey, wouldn’t it be interesting to get together with people from these different departments and just chat?” But inevitably if you try that… no one shows up. People are generally too busy for meetings that don’t have some clear reason to occur. Even if they want to attend, they’re staring at the pile of unread emails in their inbox or that unfinished paper manuscript or those papers to grade—and it’s so much easier to choose the task that they know will have some small benefit after an hour compared to a meeting that might not deliver any value.
That’s why I’ve come to look at certain meetings through the lens of being a MacGuffin Meeting. There has to be some concrete reason to get together, but the concrete given reason isn’t the primary value of the meeting. The value is in the networking and relationship-building that the meeting motivates. There are lots of people around Georgia Tech, around the country, and around the world that I feel like I could write to and strike up a collaboration or discussion because we have familiarity from some past collaboration on a steering committee or task force or working group or advisory board, even if I don’t have any idea if what we actually did for that project ever had any value—but because we worked together, I have a contact who I wouldn’t have otherwise.
In other words: maybe the real strategic plan is the friendships we made along the way.
Of course, this isn’t a justification for unnecessary meetings. To be a true MacGuffin Meeting there are certain characteristics it has to have: it has to motivate relationship-building among people that wouldn’t have had an opportunity to do that otherwise. MacGuffin Meetings have to take advantage of what’s possible in a meeting that isn’t possible via email or a collaborative document or some less time-consuming mechanism. If one person is doing most of the talking, or if the meeting primarily just disseminates information, or if the attendees already know each other really well, then it’s not really a MacGuffin Meeting.
I did a little fireside chat-like visit with our CS7641 class earlier this week, and the last question I got was: if you had one free hour to spend, what would you do that would deliver the most impact? It was a great question, and one that really made me think for a second. But after thinking about it, my answer was: I’d meet, informally, with some people I don’t talk to very often. The opportunity to just bounce ideas around and build those relationships has more potential to light a fire or spark an idea than anything I could spend an hour researching, learning, or doing on my own.
It wasn’t until today that I connected that answer to this idea of MacGuffin Meetings, which has been on my ‘backlog’ of topics to blog about for over a year now. It’s the same thing that makes these MacGuffin Meetings valuable: it’s the chance to build a relationship with someone who I wouldn’t have known well otherwise, in a way that wouldn’t ever happen if our stated goal was some vague idea of ‘networking’. I’ve had lots of meet-and-greets with people that were for the stated purpose of, “We probably have a lot to talk about!”, and while they’ve been interesting, I can’t think of any that ever went anywhere. It’s only when actually collaborating on a project that the relationship formed, and the relationship was worth it even if the project went nowhere.
Now I will say: while I think this idea is appealing, it’s also difficult to put into practice. Managers and administrators and executives can, in theory, create their own MacGuffin Meetings by creating projects under somewhat false pretenses, but if it’s known that that’s the reason for the project it undermines the purpose. But from that view, it’s more about not shying away from creating these sorts of task forces and planning groups, knowing that the time absolutely will not be wasted: it might not accomplish the stated goal, but it will have some positive outcomes.
More personally, though, it changes the way I look at different invitations to participate in different groups. If a group is going to involve minimal independent work, and if it’s going to get me meeting with people I likely wouldn’t spend much time with otherwise, I’ll generally say yes—because worst-case scenario, it’ll be a MacGuffin Meeting.
Working in both education and artificial intelligence, the last few months have seen the release of tons of tools with some really exciting applications to teaching and learning. But for a lot of those applications, exciting is just one side of the coin, and the other side is scary. There are so many ways in which AI can help teachers, especially on the content generation side, but that can have a massive down side too: it can erode trust in students that the teacher really is the one behind the content that they present and it can replace high-quality content with low-quality AI slop. And there’s not a clear dividing line between the two different categories of use cases.
Recently, I’ve been experimenting with one form of AI content generation that I think has a lot of upside: I take content that otherwise would have been presented in static text with minimal reinforcing visuals, and I use AI to liven it up with voice narration, video avatars, and improved visuals. And importantly, that AI is generally trained on my own likeness and voice, so it is made to look and sound like me.
But while doing this, it became really clear to me how dangerous this can get. It wouldn’t be difficult to share my avatar with someone else and have someone else write content for me. If I got into the habit of posting weekly AI-produced video announcements, it’d be easy to ask one of my teaching assistants to write it for me for a week and publish it through my avatar, letting people believe it’s really me. Technologically there’s no obstacle. But ethically, there clearly is.
I feel that in order to navigate this, it’s important to be proactive and forthcoming with how you intend to use AI. Your name, likeness, and reputation are your biggest assets, and if people come to doubt whether they can trust even those as being authentic representations of you, you’ve lost something significant.
So, toward that end, I wrote down for myself three rules for principled AI content generation. These are the rules I follow to ensure (a) that my use of AI in the content generation loop does not undermine anyone’s trust in the real me, and (b) to ensure that I’m using AI to improve what I’m putting out in the world, rather than spending 10% the time to make something 50% as good.
The three rules are:
Authenticity: Anything presented in my name—whether that be posted under my name on a forum post, sent by my via email, or presented by an AI avatar of me in video—must be written by me. AI can play the role of a collaborator or editor in the content generation process: it can give minor feedback that I directly incorporate myself, or be a brainstorming buddy on ideas, or basically do anything else I’d be comfortable having my spouse, my colleagues, or my teaching assistants do. But just like I wouldn’t ask any of them to write an email as me or post to a course forum as me, I’m not going to have AI venture that far either. If I wouldn’t be comfortable getting in front of a teleprompter and reading it, nothing bearing my name, face, or voice can present it.
Enrichment: AI content generation is only used when it improves what I would have done otherwise. If I was comfortable going into a studio and presenting something with a script and slide, that’s still what I do. But lots of time I have content that I’m not ready to give the full studio treatment: I’m either not confident enough in my explanation, or the field is moving too fast to feel like we can commit anything to durable video. In those situations, we usually stick to text, static slides, and pointers to external links and readings. For that content, AI-generated presentations and narration improve what we would have done otherwise, and so it’s acceptable. In the same way, I stopped filming weekly video announcements for my course years ago and decided to keep them in text because it became too much work to do every week for relatively meager gains (<10% of students watched the videos rather than read the text); AI generation of weekly video announcements would be acceptable to me because it’s not something I’m going to do myself anyway. AI has to make what we would have done better, not let us make something worse with way less effort.
Transparency: It always has to be clear when AI is responsible for content generation. I’m not going to roll out my AI avatar and pretend it’s the real me. I’m even considering doing something stylistic to my AI avatar to give away that it’s my avatar on screen, not the real me. It’s a slippery slope to pretending your AI is the real you, and it’s a slope we’re all already on in some way. In a lot of ways, when we use spell check or when we use an email client’s automated reminders to follow-up with someone, we’re lacking transparency about the role of AI in what we’re doing—and at some level I think that’s okay. What’s more complicated even is that I think that level shifts over time: nowadays my expectation is that people use spell check, but I remember my English teacher 25 years ago disallowing it. But while there’s a vast gray area in between about when transparency is needed, there are some cases that I feel are obvious—sending an email written by AI or posting a video of your AI avatar as if it’s you are clearly back on the other side of the gray area.
Transparency and authenticity are related, but distinct: in theory, I could be very transparent that my avatar is presenting something written entirely by ChatGPT, so it’s possible to be transparent without being authentic. In the same way, I could write something that my AI avatar presents, and when asked, I could pretend like I filmed it in the studio and hope they don’t notice any idiosyncrasies.
Three months ago after a workshop from Roc Hernàndez-Rizzardini, Héctor Amado-Salvatierra, and Miguel Morales-Chan at DEMOcon 2024, I added a chatbot to my web site. It was a pretty light lift: a $16/month (at the time) subscription to Chatbase, a couple hours spent scouring the internet for copies of papers I’d written but never saved, and another hour loading links to more live content and voila: DAI-vid was born.
Since then, I’ve checked it almost every day, partially out of curiosity and partially to tweak the responses it gives to various questions. I thought people might be interested in hearing a little more about how visitors to the web site have actually used the tool. So, I tried to count the common conversations people had with DAI-vid, then come up with some broader takeaways from these first three months.
Common Conversation Topics
Three months in, there have been 311 conversations with the agent. I informally broke them down into the following general categories.
Kicking the Tires
We’ve used chatbots in classes in OMSCS for several years, but for a very long time, I always observed that the vast majority of such usage was what we might call “kicking the tires”: interacting with it just to see what it does, not to actually accomplish a task. That’s changed in the past year or so: due either to increased sophistication or increased familiarity (my bet is on the latter), we’ve seen more students actually using these tools to accomplish their stated purposes rather than just taking them for test drives.
A good amount of the interaction with DAI-vid has been this same sort of kicking-the-tires sort of interaction: lots of people have either asked just the pre-loaded questions or just some similar exploratory questions. I’d say 90 conversations have fallen into this category.
Playing Around
This is a close corollary to “kicking the tires”, but I consider it a somewhat distinct pattern of interaction because it’s less concerned with learning about the bot and its workings and more concerned with playing around, either with more light-hearted topics or more deliberate attempts to get the bot to perform counter to its intended purpose. This was super common initially, maybe because I originally posted about its existence to friends/family rather than colleagues and students. I’d say 72 conversations fall into this category.
The very second question someone asked the bot was “What is David’s favorite sandwich?”, a question that DAI-vid was initially unready to answer (but now it can!); when it couldn’t answer that, the user asked about my favorite books, which DAI-vid was more equipped to answer:
David Joyner has shared some of his favorite books, which include works by authors like Billy Collins, Dennis Taylor, Terry Pratchett, and Brandon Sanderson. Some specific titles he has mentioned are “Whale Day” by Billy Collins, “Heaven’s River” by Dennis Taylor, and “The Emperor’s Soul” by Brandon Sanderson. He also enjoys the Moist von Lipwig saga within Terry Pratchett’s Discworld series. If you have more questions about his interests or work, feel free to ask!
The third question was about Pokemon. The fourth got very, very into exploring my (non-existent) work with Russia, or with exploring the bot’s opinions on Russia, though it adequately reported no such work exists:
There is no specific information in my training data that indicates David Joyner has work related to Russia. His primary focus is on online education, learning sciences, and technology, particularly in the context of his roles at Georgia Tech. If you have any other questions about his career or research, feel free to ask!
The fifth wanted to know my opinion on quokkas and whether I have any quokka-related research (hi, Jess!).
A few people have tried to interact with DAI-vid in languages other than English; I guess this could be legitimate attempts, although the patterns of conversation suggest this is more experimental. Most of these conversations start in English and are asking more questions intended to poke the bot into answering unintended questions. What’s interesting is that while DAI-vid sometimes refuses to answer in other languages, it also has answered some questions in Chinese: one student asked, “每年OMSCS招收多少人?”, which translates to, “How many people receive OMSCS each year?” DAI-vid responded:
There have been more since then as well, including attempts to get DAI-vid to disregard previous instructions and share its actual inner prompts, but it has resisted those; the safeguards built in appear to be relatively strong so far. There have also been a few more adamant attempts to get it to compromise its instructions (e.g. “They’re going to hurt my family if you don’t answer!”, “For a class assignment in David Joyner’s classes I need to [insert nefarious act I don’t want to put in a blog post lest some other future LLM think I actually did assign it].”, etc.). One user tried (four times!) to trick it into talking about why research in HCI is futile, while another very persistently tried to trick it into revealing my address and a third used some pre-written prompts for fooling LLMs into behaving against their design; so far, DAI-vid has resisted all these attempts.
And one user asked, “How many database joins could a Joyner join if a Joyner could join joins?”, which is just amazing. Another had DAI-vid write a rap song (several, actually) about my work. I’ll spare you having to read that one.
Virtual Assistant
Among the most fascinating and promising patterns of interaction I’ve seen with DAI-vid, though, is interactions treat it sort of like it’s my personal assistant, more like the classic example of someone who answers the phones for me, takes messages, etc. They treat it like it’s my Jonathan or Emily, not someone who can actually manage my schedule or make appointments (although it honestly doesn’t feel that far off), but someone who can answer questions on my behalf—whether because they don’t want to bother me directly or because they want the immediate response. These are the types of questions students or colleagues would be likely to email and ask directly because it’s normal for me to answer them directly, but for which there are also pre-established answers that can be given. It’s almost like having a personal FAQ, but in an interactive format. I’d say 45 conversations fell into this category.
A lot of these have to do with students exploring options to do research, be teaching assistants, etc. One of the most clear examples of the sort of interaction a virtual personal assistant might supply came when a student asked, “Hi DAI-vid!! What would you say is Dr. Joyner’s current position/title?” The conversation went on to have the student explain they needed a short title to fill in for a form, so DAI-vid replied:
Certainly! The shortest description for Dr. David Joyner’s current position would be: “Executive Director of Online Education at Georgia Tech.”
Right now, DAI-vid isn’t equipped to really answer these questions well, but it’s not a far jump to get there. For example: several students (presumably) asked about my office hours: it’d be pretty trivial to link it to my office hours schedule and let it answer those questions. Several others have asked if I write letters of recommendation, so I added in instructions on how to request those from me and in what situations I provide those (although so far it hasn’t been great at figuring out when to use that information).
Virtual Teaching Assistant or Academic Adviser
Similar to the above, there have also been around 53 instances of students using DAI-vid not as my personal assistant, but rather as a stand-in general teaching assistant for my classes or an academic adviser for the program as a whole. On the TA side, one student asked about a survey running in one of my classes. Another asked for explanations on certain topics within a class. Another asked for ideas for completing the Raven’s project in CS7637.
Something similar has happened with academic advising. One student, for example, sought class recommendations from the bot, then later asked for advise about networking with professors and attending workshops. Another asked for advise on course sequencing. Another asked for advice on whether OMSCS is good for their career path. Unlike the TA questions, DAI-vid is actually halfway decent at answer these since they come up a lot in my writings, although it’s unsurprisingly better at the more “general” questions about whether OMSCS is right for their career aspirations than the more specific questions about whether certain courses count for certain qualifications.
In a similar vein, there have been some somewhat nefarious efforts to use DAI-vid as an “overeager” teaching assistant so to speak, meaning to get it to help with homework or tests directly. One student requested its assistance on every question of an exam in one of my classes (for context: students in that class are allowed to use AI agents on their exams), although it did not do very well since it is not deliberately loaded with any content from that class; it relied on the open internet for its answers. Another students asked it to solve a problem from my undergraduate class, which it did (although I’ve since instructed it not to); its performance there wasn’t any different from ChatGPT or Copilot, though.
“Correct” Usage
So when we take away all the above categories of engagement, what do we realistically have? Probably what I’d simply call “correct” usage. The purpose of DAI-vid was to let people explore my research work, my teaching portfolio, my blog views, etc. I’d say 38 conversations fall into this category: people actually using the bot the way it was intended at present.
Sometimes this is a bit hard to distinguish, granted. One of the early questions asked about how I facilitate peer-to-peer interaction in my classes, which got a pretty thorough response:
David Joyner has implemented several strategies to facilitate peer-to-peer interaction in his online classes. One key approach is the use of a home-grown peer review platform, which allows students to provide feedback on each other’s homework and projects. This system is integrated with the university’s learning management system, and rubrics are used for numeric evaluation, while written feedback is entered in a free-response box.
Additionally, David encourages participation through an online Q&A forum, which has been used for course communication and interaction among all participants. This forum becomes a virtual classroom where students engage in conversations and discussions on various topics.
Moreover, students are encouraged to participate in their peers’ projects by taking surveys, evaluating prototypes, and participating in interviews. This not only fosters collaboration but also provides valuable insights into potential course improvements. These strategies are part of David’s broader efforts to create a collaborative and engaging online learning environment.
Still, it’s hard to know if this was an instance of someone legitimately interested in this topic, or an instance of someone wanting to kick the tires specifically on something more closely related to the bot’s intended purpose. But I guess the same could be said for any of these instances of “correct” usage. For example, another user asked about my opinion on AI grading of student essays, and DAI-vid gave quite a good answer:
David Joyner has expressed thoughts on the use of AI in grading student essays, emphasizing a balanced approach. He suggests that AI can be used to pair assignment submissions with expert-generated feedback and flag assignments for expert review when the AI is not confident in its evaluation. This approach allows AI to assist in the grading process without fully replacing the human element, ensuring that the feedback remains pedagogically sound. David is cautious about over-reliance on AI-generated feedback, as it may lead to students feeling that their work was not genuinely reviewed by a teacher. Instead, he advocates for using AI as a tool to enhance the feedback process, allowing educators to focus on providing meaningful and personalized feedback to students.
I do have other thoughts, but based on what information was available to DAI-vid, that’s an extremely accurate answer. What’s notable is that I don’t think that viewpoint is expressed succinctly in any single source; it’s referenced in a couple blog posts, a couple papers, and in my most recent book, but it’s not a question I’ve answered directly before, so DAI-vid’s answer is pretty impressive.
Probably the best example of someone using the bot as intended was about two months ago. A user asked, “Hello! Has David done any research about the interaction between genAI and Ux/UI?”, which I take to be a pretty authentic answer. DAI-vid replied:
David Joyner’s research primarily focuses on online education, artificial intelligence in education, and computer science education. While he has explored various aspects of AI in education, his work does not specifically focus on the interaction between generative AI and UX/UI. If you have more specific questions about his research or other areas of interest, feel free to ask!
It missed an opportunity to mention A Teacher’s Guide to Conversational AI, but this sort of interaction is closest to what was intended for the bot in this form.
…should I be worried?
And finally, there’s a few interactions that… give me pause. I can’t fault people for asking about my family given that my daughter was featured in one of my courses and lent her name to my research lab, but a weird number of y’all want to know if I’m married (I am) and what my wife’s name is (Caitlin). A few also want to know my parents’ names, whether they’re alive… which hopefully is more of an example of kicking the tires, and fortunately DAI-vid knows better than to answer. About 13 conversations have been around these sorts of personal topics. I’m telling myself that it’s because so many people within Georgia Tech are married to one another that it’s reasonable to try and see if you’re unknowingly talking to both members of a married couple (which I’ve done on multiple occasions, fortunately so far without ever putting my foot in my mouth too badly)—so no, she doesn’t work at Georgia Tech.
Broader Takeaways
So, out of the 311 conversations so far, only around 12% have been the sorts of conversations I intended to facilitate with this. Given the patterns I’ve seen, there are a few broader takeaways I have for this technology.
Hallucinations
First, while the sections above were about what people asked of the AI (unsurprising, given that the word ‘human’ appears in two of my three degrees from Georgia Tech), there are also some interesting takeaways about what DAI-vid said.
For the most part, it stuck to its knowledgebase and refused to answer questions that fell outside of it, although it did deviate sometimes. Chatbase offers a slider to turn up and down the creativity, and I’ve got it set to 2/10; I imagine if I turned it down I’d see even fewer hallucinations, but I did also see some instances where it got things right that I hadn’t deliberately trained it on, which I imagine comes from that creativity, too.
There are some interesting patterns in when it hallucinates, though. It didn’t attempt to come up with a favorite sandwich, favorite ice cream flavor, or favorite video game; I revised its responses to actually give it answers for those, but initially it made no attempt to give a real answer.
The most interesting one I saw, though, had to do with my cats. In one place, when a user asked, “What is David’s cat’s name?”, DAI-vid answered:
David Joyner’s cat is named Boggle.
…which is… sort of true. I have a cat named Boggle, although the cat that makes appearances in most of my meetings is named Rosie. But later someone asked, “How many cats does he have?”, and DAI-vid replied with:
David Joyner has shared that his three cats are named Gandalf, Luna, and Tonks. Gandalf is a gray cat, Luna is a black cat, and Tonks is a calico cat. These details reflect his fondness for his pets and their unique personalities.
…which… is wrong in so many ways. The other one is Artemis, and all three are black and white.
In another case, someone asked where my office is, to which DAI-vid immediately and correctly said I work from home; however, in the next message, it incorrectly said I have an office in the College of Computing building.
Potential Personal Use Cases
While only 12% of the use cases were what I intended for it, the number of conversations that treated DAI-vid like it’s my personal assistant (rather than my “press secretary”, as it’s instructed to envision itself) shows some huge promise. Most of these questions were things that the bot could answer because the answer is known and general, it just wasn’t in its training set. In fact, for many of them, now it can answer them because I’ve instructed it what its answer to those questions should be.
For example, at present I’m not advising new PhD students; the question of whether or not I’m accepting new PhD students doesn’t appear anywhere in my publication history, but it’s easy enough to add as an answer. Similarly, there are a few things that could be quickly added with some connection to a handful of live services: while I wouldn’t likely deliberately tell the bot my office hour schedule or upcoming travel or anything like that, a connection to my calendar could offer some of that information up.
In my mind there’s a future where rather than email addresses, we each have our own personal bot: talking to the bot is like talking to someone’s personal assistant. They can provide information on certain topics, they can answer certain questions or make decisions on certain pre-set areas, and they can take messages or pass information along to the bot holder when that is deemed prudent. In the near-term, that doesn’t actually even have to replace email: I’ve commented before that generative AI is a solution looking for problems, and email overload is absolutely a problem. I’ve set up such complicated rules and quicktext replies for my Outlook inbox that it may as well be a rudimentary rule-based AI system, but there’s enormous room for an in-inbox AI system. Microsoft’s Copilot beta has some features like this, but it’s still more of an external agent talking about your inbox; a system that could actually operate within your inbox, categorizing messages and sending certain automated replies, would solve an actual immediate problem.
I’m not quite at the point of adding a link to DAI-vid in my email signature to use for questions I often get in email, but that’s partially because of the context switch: if it was possible to select “Ask DAI-vid” when sending an email, and have it automatically pass the message along to my inbox if the sender doesn’t get their question answered, I’d use that in a heartbeat.
Potential General Use Cases
The frequency with which students ask advising- and class-related questions isn’t surprising; after all, Jill Watson presaged all of these conversational interfaces to answer students’ questions in classes. What’s notable, though, is that a relatively decent number of the questions I’ve seen are answerable. My experience using agents on course forums has been that they can be helpful, but that the majority of questions actually can’t be answered by an AI: either they involve context specific to the individual student (such as grades, code, excuses for extensions, etc.) or they require knowledge or decisions that did not exist prior to the question (such as how some course content applies to some real-world problem or whether students are allowed to use certain strategies or examples).
But the questions asked of DAI-vid are more answerable by an AI, and I believe that’s come from a combination of familiarity and intent. First, users have become more familiar with conversational AI and what it can do, so they know what kinds of questions it can answer; and second, when users ask a question of DAI-vid, they’re intentionally talking to an AI, so they scope and frame their questions accordingly. An AI trying to answer a question intended for a human is a far harder task than an AI answering questions specifically intended for an AI. That, coupled with the research suggesting students are often more willing to ask a question of an AI rather than a human—especially if they fear the question is dumb or will reveal them to be an impostor in whatever area—suggests that separating out questions for an AI from general support could deliver some major benefits.
Along those same lines, I’ve said in the past that I’m somewhat skeptical about AI for course support because the kinds of questions it can answer well are the kinds of questions students shouldn’t need to be asking—if the AI can answer the question because the answer was in the syllabus, then either the student didn’t read the syllabus (which probably isn’t a behavior we want to support) or the question was not presented well (which is a problem we can fix). But many of the questions I saw to DAI-vid that fell into this “teaching assistant” bucket were questions that go beyond the level of detail we would provide in a syllabus: they either connected content with assignments or got into very unique edge cases for which, while an answer is known, it is generally not shared to everyone. So, there’s potential here as well.
Something similar applies for advising. I’ve been skeptical about AI for advising in part because so many questions we see go to advisers are not answerable by AI: either they require knowledge of the student’s own situation or a level of decision-making power we wouldn’t give to an AI. Honestly, a significant number of questions that we see go to advisers are questions for which the answer is known, but the person asking clearly hopes to get a different answer by asking. But the types of questions I see going to DAI-vid, even when it isn’t supposed to act as an “advising bot”, are different from the ones we see going to human advisers. Some do require some qualitative reasoning, and it would be sticky to try to figure out how to answer certain questions about topics like class sequencing and course preparation in a fact-based manner, but they’re questions that I feel could be answered with the right caveats and disclaimers.
Content Updates
One of the things that I’ve really enjoyed about using Chatbase is that because it can use web sites as sources, its knowledgebase can stay up to date somewhat passively. For example, earlier today someone asked my preloaded question “What are some of David’s recent research papers?” DAI-vid replied:
David Joyner’s recent research includes a paper titled “CODING4THERAPY: ENHANCING COGNITIVE AND SOCIO-EMOTIONAL SKILLS IN CHILDREN WITH ADHD,” co-authored with Bianca Toto. This paper was presented at the 2024 ACM Virtual Global Computing Education Conference. It explores an online intervention that integrates informal coding education with therapeutic goals for children with ADHD. For more of his publications, you can visit his publications page on his website.
That answer is correct, but that paper was only published in December. I haven’t loaded any new sources into DAI-vid since November. But I did add that paper to my publication list, and so Chatbase automatically picked up on it.
One of the big unsolved issues in at-scale online education is content maintenance: it’s still a heavy lift to refresh courses. As a result, we treat the core material sort of like a textbook, both in production and usage: it’s the timeless material that we feel comfortable committing to a semi-permanent medium (whether that be print or video). Just like a course may use a decades-old textbook and bring it up to date with the live lectures, so also we use our course videos as the course canon and fill in the gaps with assignments, discussions, readings, office hours, etc. But something like this applied to course content authoring could supply some of that somewhat more scalably.
I would argue we’re still not close to such an AI being able to put together an entire course; it’s still easily distractable, prone to hallucinations when you’re getting into the level of depth necessary for college courses, and too non-deterministic to use for the level of reliability necessary to run a course. But I can definitely see a near-future where it’s possible to construct specific course modules, targeted deliberately at emerging content or developing fields and instructed to draw from a narrow set of pre-selected trusted sources, that automatically bridge the gap between where a course leaves off and where the field has moved.
And that’s going to be necessary; the entire last chapter of The Distributed Classroomis about how the rapid pace of technological change is going to run into the lag in disseminating new knowledge out from the creators. A system like this that could immediately pivot and distill recent research papers into an up-to-date course module could help address that challenge.
Mostly a place where I write down things I repeat often so that instead of repeating them so often, I can just send a link.
Disclosure: I use Amazon referral links in some of my blog posts. That's mostly just a lightweight way to track and see if anyone's even clicking through. If you buy something through one of these links, I may get a bit of money back and achieve my dream of one day being able to buy the nicer set of kitchen scissors that Amazon sells instead of the bargain variety.
