LIVE: Ambient Agents and the New Agent Inbox ft. Harrison Chase

Contents

Konstantine Buhler: Everything we’re about to talk about is what’s next. And the first person up here is Harrison Chase, our very own Harrison Chase, who’s been at every AI Ascent. He’s been talking about agents longer than anyone I know. We’d love to have you come up here and talk about agents, in particular ambient agents, this new concept that you guys at LangChain are bringing to the world.

[applause]

Ambient agents

Harrison Chase: Thanks for that intro and excited to be chatting. My name is Harrison, Co-Founder, CEO of LangChain. We build developer tools to make it as easy as possible to build agents.

A lot of the agents that we’ve seen being built so far are what I would call “chat agents,” so you interact with them through a chat interface, send them messages, they run and respond, and that’s great. They’re great for a lot of purposes. But one of the concepts that I’m really excited about is the concept of “ambient agents.” So what is an ambient agent? The way that I like to define an ambient agent is ambient agents listen to an event stream and act on it accordingly, potentially acting on multiple events at a time.

And so what are the differences between this and normal agents? So there’s a few. One, what are the triggers? So it’s no longer a human coming in and sending a message, it’s an event that happens in the background. How many of these can be running? With chat, you can usually only interact with one agent at a time. Maybe you open a few windows and you have a few running at the same time, but it’s generally one. With ambient-style agents, because it’s listening to these events, it’s however many events are happening in the background, so it can be a far bigger type of number.

Another interesting point is the latency requirements around it. So with chat, you message, you expect some response back pretty quickly, or you get bored and you go to another website or something like that. Because these ambient agents run in the background, they’re triggered by events, they can run for a lot longer period of time before you need a response in any shape or form. So there’s generally much less strict latency requirements.

And then lastly, I think it’s interesting to think about the UX of these agents. So for these chat agents, it’s mostly chatbots. That’s a pretty familiar interface by now. I think there’s a little bit of a question of how do you interact with these agents that are in the background, because they are running without you knowing that they’re running. But as I’ll talk about in a little bit, it’s still really important for you to interact with them in some form.

So just to make this concrete, an example of an ambient agent could be an email agent that listens to emails coming in and acts on them accordingly, and maybe tries to respond or maybe tries to schedule meetings or maybe pings you or pings other people on the team. So that’s kind of like a concrete example of one type of ambient agent that we’re seeing.

So why ambient agents? I think they’re interesting for a few reasons. First, they let us scale ourselves. So if you interact with a chat agent, it’s generally one to one, you’re doing one thing at a time. When you have these ambient agents, there can be thousands of them running in the background, and so that just lets us scale our impact a lot more.

Two, they can get at kind of like more complex operations. So when you’re interacting with a chat agent, because of the latency requirements, it’s generally a simpler operation that it’s doing. So you might have the human send a message, it goes to the chatbot, the agent, it responds right away. Maybe it calls a tool, maybe two tools. The more tools it calls, the longer it takes to run. You can’t do that. With ambient agents, because you don’t have this as strict latency requirement, you can call a ton of tools and do more and more complex operations. You can add in other steps as well, so you can add in explicit planning or reflection steps and generally build up the complexity of the agents that you’re building.

Ambient does not mean fully autonomous

One thing that I really want to highlight is ambient does not mean fully autonomous. So I still think it’s really important that we are able to interact with these ambient agents. And there’s a few different interaction patterns that we see people building towards. So one is approving or rejecting certain actions that these agents want to do. If you want to have an ambient agent that’s potentially giving refunds to customers who are emailing in, definitely when it starts, you’re going to want to have a human in there approving some of those things.

Second is a more advanced option of this: editing the actions that they do. So maybe they suggest something, you don’t want to approve or reject it, but you want to explicitly edit it and have it do that.

Third, these agents can get stuck kind of like halfway down, and so there should be an ability for you to answer questions that they might have, just like you would answer questions of a co-worker if they’re working on a deep problem or something like that.

And then fourth, because these agents take a lot of steps, it might be very useful for you to go back to the 10th out of 100 steps or something like that, interact with it there, modify what it’s doing, give it some feedback. And so this is what we call “time travel.” And facilitating this is a cool new interaction pattern we see.

So there’s a few reasons that having this human in the loop is important. First, it just gives better results. So if you think about deep research, which isn’t exactly an ambient agent, but it is a long-running agent, there’s a period of time up front where it asks you some clarifying questions to go back and forth, and that generally helps produce way better results than if it just went off whatever your initial kind of like question or statement was. And so having this human in the loop in the form of deep research, asking these clarifying questions in the form of ambient agents, there’s different types of patterns, this just gets better results. It also helps build more trust. So if you’re doing explicit actions like giving or sending payments or approving things, having the human in the loop just builds more trust.

And then third—and this is maybe the most subtle one—is I think it helps a lot with the memory of the agent. So when I’m talking about memory, I’m talking about learning from user interactions. If you don’t have the user interacting with the agent, then there are no user interactions to learn from. And so having this human in the loop helps inform a lot of the memory things that you want to be building into the agent so that it can do better in the future.

The agent inbox

And so with this importance of the human in the loop, I think it’s interesting to think about what a good UX for this might look like. This is one thing that we’ve kind of built as a prototype at LangChain, which is the concept, we call it an “agent inbox.” It’s an inbox for your agent to send things to. You can see when it requires actions, you can see some descriptions. If you click into a row, you can then see a more detailed description of what’s going on, what explicitly it wants approval for, or whether you want to respond to it. And there’s a few different interaction patterns here.

Talking a little bit very briefly about some of the things that we’re building that we think help with this, we’ve paid a lot of attention in LangGraph, which is our agent orchestration framework, to make it good at ambient agents. In particular, we’ve paid a ton of attention to the persistence layer that backs it. This enables a lot of these human interaction patterns because basically you can run your LangGraph agent, you can stop at any point in time and the entire state as well as previous states are persisted. And so then you can have all the human in the loop interaction patterns. You can wait for a second, a day, an hour, however long, have the user come in, see the state, modify it, go back to previous states, things like that.

We’re spending a lot of time right now on LangGraph platform as infrastructure for running these agents. These agents are often way more long running. They’re often bursty because they’re triggered by events. So you could get thousands of events at a time, so you need to be able to scale up. And they’re flaky in nature, not just because of typical software things, but also because of this human in the loop pattern. You want to be able to correct mistakes.

And then finally we’re building LangSmith as well for these agents. They’re really long running, they can often mess up, they’re doing more complex things. Having visibility and observability into what they’re doing is really, really important. As a concrete example of this, one of the things that I built on the side is an email agent. So if you’ve emailed me in the past year or so, it’s drafted a response or sent a calendar invite. It’s still human in the loop. I use the agent inbox all the time. It’s open source and on GitHub, so if you want to see how all these components come together and what I think is a pretty cool and unique and hopefully glimpse of what’s next, I would encourage you to check it out. And with that I will hand it off.

[applause]