# 10 2026-08-08 AI Agents Final Session

course: Module 5 — AI Agents & Agentic Frameworks
module: Module-5-AI-Agents-Agentic-Frameworks
date: 2026-08-08
type: transcript
video_url: https://personal-learn.armco.dev/files/_Recordings/Module-5-AI-Agents-Agentic-Frameworks/10_2026-08-08_AI_Agents_Final_Session.mp4

---
[00:26:43] They have integrated QUI with Copilot Studio and they have created.
[00:26:45] No, when I say glue.ai studio itself, so it's more like a low-code, no-code platform.
[00:26:52] Right, so yeah.
[00:26:53] Ah, I got it, I got it, so they have created like a, they have created like a copilot studio for QAI so that
[00:26:59] Correct.
[00:27:00] People like business users who are who are not coding, not programmatically.
[00:27:02] Yes, yes.
[00:27:04] Uh, so, you know, savvy, they can play around here.
[00:27:16] Yes.
[00:27:17] Correct, correct, because they have a lot of data also with them, right, in the form of enterprise-grade data, in the form of Excel sheets and all, right? Some data documents, so they can play around that, because for every application, we don't need a agentic solution or AI solution at the enterprise level.
[00:27:20] Yes now now what do you have any answer like uh I am asking this question so that everybody also, you know.
[00:27:29] Also, here from…
[00:27:31] Not only from me, but also from you, as you are also doing.
[00:27:34] So do you have any answer? Why do you think that Landgrav is mostly preferred in the industry?
[00:27:39] Do you have any answer? Do you feel that you found that this could be the answer? Anything that comes from you, you're within.
[00:27:46] Don't have to search and tell me, but anything within yourself.
[00:27:49] Where you felt Landgraf is
[00:27:52] So like mature library, I would say that, which has covering a lot of functionalities, that is one.
[00:28:00] Huh.
[00:28:01] say that and it is evolved very vastly from last two to three years if you look at this, right, with the more capabilities and on that perspective.
[00:28:08] Hmm.
[00:28:09] What we found that as an enterprise level, the library supports which is available at the community is much faster than other than that.
[00:28:20] It will be helpful in that perspective.
[00:28:21] Hmm… Yeah, yeah, yeah.
[00:28:23] So, if you see the thing about LangGraph is, Landgraph is
[00:28:30] very native. Very native as in, like, it is.
[00:28:34] Very less abstraction is there, very simple. Okay, Langraph codes are very simple. It is very easy to draw mental mapping in your mind.
[00:28:42] Okay, LangRap. In queue AI, what happens? You use very Crew AI heavy syntaxes.
[00:28:47] Okay, so in LangGraph, if you see, there are only three things you are importing from LangGraph.
[00:28:55] From LangGraph, you are just importing three things. State, state graph, start, and end.
[00:28:59] Start and end is very simple, just starting and ending.
[00:29:01] It is the only state graph that you are importing.
[00:29:04] Okay, REST, everything is Python. Python or LangChain? REST everything is Python only.
[00:29:10] So, LangGraph is more closer towards Python compared to Crew AI.
[00:29:15] In Crew EA, the amount of imports that you had to do was very heavy in nature.
[00:29:21] Due to which, due to which your.
[00:29:23] You feel a little, ah.
[00:29:26] You feel a little chained by Crevi, chained as in like
[00:29:30] For this, also, you have to look for an option. For this also, you have to look for another option.
[00:29:34] Everything you might have a functionality in Creview AI, you might not have it.
[00:29:39] If you have it, it is more compatible to use that. If you don't have it, then you'll have to build one.
[00:29:43] Okay, so it becomes very QEI ecosystem.
[00:29:46] Heavy. So, due to which
[00:29:49] People who are entirely in QVI, for them it is good.
[00:29:53] But since this is more Pythonic in nature.
[00:29:55] This is more close to our Python. If something is written in Python, it can be nicely used in LangGraph.
[00:30:00] So, that is why in Landgrav, it is very simple.
[00:30:02] Okay, just if you look, you have already built this agent, sequential agent. How much import statements did you use? Only state graph.
[00:30:10] Staten entho is just starting an ending point. That is very easy to understand also. Only state graph, guys. And state graph is the only thing that you ever, ever imported.
[00:30:18] You do not have any tool, nothing, nothing is required. You can make your tool, you can use the LangChain tools.
[00:30:23] Okay, or you can build your tools, all those things. But in Creavy, it is very crew AI centric.
[00:30:29] You have to write role, goal, and access.
[00:30:32] A role goal backstory, then in task you will have to write the description, you have to give the prompt and everything, right?
[00:30:40] So, for that, there are special syntaxes that you'll have to follow. You saw that flow thing in Rang Graph also, in Query UI. That was very, very
[00:30:48] Lang, uh, that was very crew AI-centric also.
[00:30:51] Very difficult to understand, very difficult to remember also.
[00:30:54] But LangGraph is more friendlier towards
[00:30:56] Coders, okay? So Bhalay.
[00:30:59] The learning curve is high in
[00:31:03] In LangGraph, it's high because you need to be more Python in nature.
[00:31:08] You just cannot.
[00:31:10] No crew AI syntax and
[00:31:13] achieve it. In QI, what happens is,
[00:31:15] It is less of a Python mode of a crew.
[00:31:17] Okay, like it is Python only, but…
[00:31:20] Since Crew has built wrappers on top of them, so if you just learn them also you can build.
[00:31:25] Agents, but in LangGraph, you have to know Python also.
[00:31:29] As a whole. Yes, Sivanj.
[00:31:33] A small query. As you mentioned is that, land graph is more popular in the industry. I just want to check if the… most of the enterprise is either go via Google, Gemini.
[00:31:46] Or anthropic one, so if they are using Google Suites or Gemini, why don't they will go with Vortex API instead of Langgraph?
[00:31:55] Vertex API.
[00:31:57] Vertex API is ADK.
[00:31:58] Yeah, what I said, they have on their Google ADK, yeah, yeah.
[00:32:02] ADK is not scaling well. It is not where I have heard ADK two years back it.
[00:32:09] It doesn't even come in the top 3 agentic application.
[00:32:15] Okay, okay.
[00:32:16] Got it. Shivaj, like you I know you are asking this as a question because you somewhere saw ADK.
[00:32:21] Or maybe you saw the mention that ADK is also there.
[00:32:24] But etiquette.
[00:32:25] Because I am on the customer-facing side, and I am seeing most of the enterprise are either using Google Gemini or an Anthropic Cloud, so if they are using that particular suit, so they will choose one of the tools from that suit
[00:32:35] No, no, no. Their LLM is different, and their agentic framework is different. Your LLM.
[00:32:42] can be powerful. Their LLM can be super good because Gemini.
[00:32:46] If you look at Google, Google is the first one that created transformer model.
[00:32:49] Okay, and, you know, Anthropic is anyways became the legend of it.
[00:32:55] Yeah
[00:32:56] Okay, so their agent can be very powerful. Agent has a separate story, sorry, their LLM has a separate story.
[00:33:00] Their agentic framework has a separate story.
[00:33:03] Got it, so their agentic doesn't have to be similarly, equally powerful as them.
[00:33:08] Correct. OpenAI never forced, uh
[00:33:11] Recently, I had heard OpenAI also coming with a framework. I'm forgetting. OpenAI also has a framework. OpenAI is also coming up with a
[00:33:18] agentic framework. See, I have not cracked it so well.
[00:33:22] Okay, but they are saying, like,
[00:33:23] They will crack it. But Landgraff has cracked it. See, LangGraph has exceptionally done.
[00:33:28] In this part, LangChain, basically, LangChain has specially worked in this.
[00:33:32] Google and all of them have worked on their…
[00:33:34] A little impart more as of now at least.
[00:33:37] Okay, so…
[00:33:39] It doesn't mean that LLM is powerful so that their agentic framework will also be powerful. Got it? That's the point I'm trying to make.
[00:33:46] She went.
[00:33:47] Understood, ma'am. One more related question. So, let's say their security team has approved Google Gemini or Anthropic
[00:33:54] And they are in combination, they are going to use Langchain or Langgraph. Is it okay to use secure device because most of the financial or BFSI domains looking everything there insecure in nature
[00:34:09] So, using LangChain with the combination with Google or Entropic is okay, or you…
[00:34:13] It's, it's a, it's normal because see, LangGraph Langchain is not.
[00:34:19] API, na, it is not… it is a framework that is installed on your system and it is used. It is not transporting any data out of you.
[00:34:26] Okay, the problem is, the problem sometimes comes is this part.
[00:34:33] Oh, wait, where is it? This part.
[00:34:35] Yeah.
[00:34:36] The LLM call that you are making is through this chat OpenAI.
[00:34:39] Now, where is… what have you imported? Chat OpenAI from LangChain.
[00:34:43] So, many companies, let's say I am working with one of the healthcare company.
[00:34:48] Uh, so.
[00:34:50] So basically, I work with, let me tell you why I keep on saying this, but let me tell you, I work with
[00:34:55] The top healthcare company in the world, that is Pfizer.
[00:34:58] Okay, so with Pfizer is one of my retainer client I work with.
[00:35:02] So Pfizer.
[00:35:04] They do not have, they do not have any.
[00:35:08] Third-party SDK to call their LLM. LLM is purely called using Fox API.
[00:35:15] VOX API.
[00:35:17] Okay, Vox CPA. Now, Vox API is vetted by Pfizer, already done by their security team and given to us.
[00:35:22] So this you choosing before you choosing the security and all these things, it is already chosen and given to you.
[00:35:29] Okay, it is told. So this part where we are calling a third-party API.
[00:35:33] to call that AI. This is the question of security only.
[00:35:38] Rest of the thing, rest of the framework has no.
[00:35:41] Question. Goddessivansh.
[00:35:45] Yeah.
[00:35:46] So your LLM object that you are creating, LLM client object.
[00:35:49] That for that, what you are using is very important.
[00:35:52] Are you using a Langchain?
[00:35:55] Chat OpenAI SDK, or are you using just OpenAI SDK?
[00:36:00] Or are you using Vox or Vertex or
[00:36:03] Uh, let's say Amazon Bedrock.
[00:36:05] Or, uh, all these things as you are.
[00:36:08] Okay, to call this part becomes very important, this part, where you use your service account and you call the LLM and.
[00:36:14] And these things is already taken before you. Let's say when I.
[00:36:19] Started working on
[00:36:21] the agentic project.
[00:36:25] I asked the question, where is the subscription key and all? They told Vox API you have to use Vox API. I had never heard of it. I what is this Vox API? What is Vox, Vox, Vox? They are telling.
[00:36:34] Then I saw Voxy page just like OpenAI only.
[00:36:36] Okay, instead of Chat OpenAI, it is like VoxLLM.
[00:36:40] Uh, and you call like that, VoxLLM, you import and you call it.
[00:36:44] It's a third-party API. Through that only you can make LLM call.
[00:36:47] Like in Oracle, only Cohere was allowed.
[00:36:50] Okay, Cohere was allowed when I was there. Now, I don't know, like, in two years what they have done with the Stargate and all.
[00:36:56] I don't know how much they have evolved.
[00:36:59] But that time only Cohere was allowed. So this thing is, this region is taking little, little.
[00:37:04] People above you. Okay, already done for you.
[00:37:08] You cannot come and pitch also that why are you not doing this? What are you not doing this?
[00:37:11] But the framework is your hand.
[00:37:13] is your choice.
[00:37:15] Got it.
[00:37:18] Okay.
[00:37:19] And she wants 1 more thing, this is for everybody.
[00:37:22] The point is, let's say you decide ADK.
[00:37:24] Okay, you decide ADK because let's say before this, I worked as a fractional developer in a startup for three months. Okay, it is.
[00:37:32] It's a very early stage startup between Oracle and between my
[00:37:38] founder journey. I worked during my founder journey only, I got a chance to mentor a startup.
[00:37:42] and work as a fractional developer to mentor their AI team. And AI was just booming that time.
[00:37:47] almost one and a half years back, okay, after I left Oracle. So.
[00:37:51] They were a…
[00:37:54] Full like.
[00:37:56] What to say, they were.
[00:37:58] They were a full pro of Gemini, Gemini, Gemini.
[00:38:03] Reason?
[00:38:05] Their CTO was a Google developers expert.
[00:38:08] Their software development entire team was Google Developers Expert. I think they have created the team from Google Developers Expert only.
[00:38:14] So, everybody, every other day, they are absent, and they used to go to Google Developers some event.
[00:38:20] Okay, they are absent, not absent, as in they are absent in the stand up.
[00:38:24] And they used to come maybe during night.
[00:38:27] Just write and call it, it's a startup, na, so fully remote culture and all those things, you can work from anywhere, so people are working from
[00:38:33] Singapore, like, Indian only Indians, they are at different, different places working from there.
[00:38:38] So they used to fully pitch all these things, Google ADK, Vertex API,
[00:38:42] And Vertex AI, all these things they used to pitch. Okay.
[00:38:45] So that, what I'm trying to say, that it depends on that company relationship with that enterprise-level cloud.
[00:38:50] Okay, many people.
[00:38:53] are very friendly with certain cloud. Let's say.
[00:38:55] Uh, Pfizer Don Trust Azure or anything.
[00:38:58] They only trust Vox API for their LLM call.
[00:39:01] Okay, this is one example. I worked with
[00:39:05] Along with this Pfizer client, I had another client known as Voodoo.
[00:39:09] Okay, it's a London-based 10 years old company.
[00:39:12] Over there also they purely used OpenAI, OpenAI, OpenAI.
[00:39:17] So they were a startup, they never had any.
[00:39:19] Uh, cloud systems or anything, they use purely OpenAI, OpenAI SDK.
[00:39:23] So this thing.
[00:39:25] It depends on the company. Like, there is no pro and con to this.
[00:39:30] Many people would like to switch to…
[00:39:31] Another cloud, but since the entire infrastructure has been already created with 1 cloud.
[00:39:36] that switching becomes very difficult. That's why infra is where the money is.
[00:39:40] Okay, if you crack one infra client, let's say you have built an infra product, infrastructure as a service.
[00:39:45] Uh, that is your software, AWS, let's say.
[00:39:49] And if you have cracked one client, it is very difficult for that client to leave you and join the other.
[00:39:53] Uh, services. It's very difficult.
[00:39:56] If once they have stayed for 2-3 years also, even 2-3 years.
[00:40:00] Also, there is a huge dependency that comes up on that infrastructure.
[00:40:03] People don't easily switch to others.
[00:40:06] Okay, so that is why some people will prefer.
[00:40:09] ADK, some preparer will prefer Landgraph.
[00:40:12] Some people will prefer QAI.
[00:40:14] But in the industry, mostly line graph is preferred because LangGraph is little cloud agnostic and
[00:40:21] It is very, very Pythonic in nature. That is the thing.
[00:40:25] So, can we say that as OpenAI or OpenRouter is safe to use or secured in nature? I mean, to know outbound
[00:40:30] Open router, uh, see the kind of membership subscription that I take, I personally.
[00:40:36] That security is not guaranteed. The one that we take, I take, you take it for free.
[00:40:41] So definitely your data is going outside. Okay.
[00:40:44] I, let's say, pick.
[00:40:45] So, customer is always worried to put an extra cost, as if we say you need to go with an enterprise
[00:40:51] license, put more cost to get the API key for OpenRouter
[00:40:57] And their apprehensions, right?
[00:40:59] Yeah, yeah, open router, uh, you will not see, uh, big level enterprises using, you will see.
[00:41:04] So, what is the best combination to go use with Langgraph? I mean, when we propose as an architect to them is what is the best combination
[00:41:10] Both Azure, both Azure and AWS.
[00:41:14] Both Azure and AW.
[00:41:15] And if they are not have a Microsoft Shop
[00:41:19] Then AWS.
[00:41:22] Is the way to go. And if then also then
[00:41:24] It doesn't work, then worst case, I have seen worst case people go into GCP.
[00:41:28] The acceptance of GCP is the lowest I have seen.
[00:41:31] With these framework.
[00:41:33] Like GCP LLM is still there.
[00:41:36] Okay, but other tools of GCP is still the lowest.
[00:41:39] Compared to AWS and Azure.
[00:41:43] Okay.
[00:41:44] Okay. Uh-huh.
[00:41:45] Thank you. Thank you so much.
[00:41:46] But, uh, but but but again, this is my basis of understanding, like, these data keeps on changing.
[00:41:52] Okay, Azure have recently killed it very well. Recently, like, with the AI.
[00:41:56] Thing with their OpenAI acquisition and with their OpenAI partnering.
[00:42:00] And all these things, they have really built the AI side of things very well.
[00:42:04] Okay, but before that, they were also struggling like GCP only they at the same level. So these data keeps on changing.
[00:42:10] Okay, within two months, you might see that Google has surpassed certain things, okay?
[00:42:17] So yeah, so that is the thing.
[00:42:18] Yeah, we are evolving
[00:42:20] Huh, sorry?
[00:42:21] We are evolving, I mean, every day.
[00:42:22] Yes, yes, yes, you are evolving and it is definitely, you shouldn't.
[00:42:26] Take 1 fact forever. Like, I told you one fact. Okay, you shouldn't.
[00:42:31] Yeah.
[00:42:32] By hearted, like, that is the only way, oh, sir told me 1, sir, 1 teacher, Anirban, told me in 2020.
[00:42:39] Seeks that in August that he told me that this is the best.
[00:42:42] After 2 years,
[00:42:44] I don't know which will be relevant, ok, MCP was nowhere 2 years back.
[00:42:49] Okay, MCP came and changed all the simple tool calling, all the tool calling things, everything.
[00:42:56] MCP. So I don't know. MCP although is not an AI concept. It's more of a API concept.
[00:43:01] Rather than. But anyways, uh, it changed. It evolved because of AI now.
[00:43:06] So it changed. So that's why.
[00:43:08] I never thought when I was doing tool calling, I never thought that even a
[00:43:13] Even, uh, evolution is required in this space, ok, because I'm not an API guy anyway, so
[00:43:19] Which side I find more dry eyes, I find more drive with.
[00:43:23] Let's say now researchers Andrich Karpathi let's
[00:43:27] is coming up, I told you, you know, vectorless RAG, he's coming up. Those side I'm more interested, where the concept is more about
[00:43:32] embeddings and all these concepts, like LLM,
[00:43:36] model and these things. But now what has happened, Naga is LLM, the underlying AI is all hiding under your API.
[00:43:43] API call. Okay, and people forget that sometime.
[00:43:47] All agentic engineers who are aspiring to become agentic engineers, they forget the core.
[00:43:51] What is the core of this agentic application? Is the LLM itself, and that is hiding under API call.
[00:43:56] And we will, let's make it so minuscule, they think that.
[00:44:01] So that's what I'm trying to say, that
[00:44:04] That is more powerful. The LLM itself is more powerful.
[00:44:10] To learn than these frameworks, these frameworks are very temporary, this is not permanent thing.
[00:44:14] It's very temporary. Today, this is there, tomorrow something else will be coming.
[00:44:21] Yes.
[00:44:22] A similar way the REST API has, or is a kind of a wrapper of all transaction management and operations
[00:44:24] Yeah.
[00:44:25] If everything is underlying. So this is…
[00:44:26] Yeah, underlying, so everything has been abstracted by that API. LLM call is also like that. Everything, the powerful, the most powerful thing.
[00:44:35] Yeah.
[00:44:36] That is LLM is again coming under API only. So that's where the software developers are, you know, very feeling very powerful. Now AI is under control.
[00:44:42] Okay, previously when I was an AI engineer, I used to feel outcasted.
[00:44:49] by this space that, oh, you are only doing R&D, R&D, and all these things, okay.
[00:44:54] But they never used to understand AI as a whole. But now, since.
[00:44:58] AI is now available through LLM call.
[00:45:00] they feel that they also know AI.
[00:45:02] Okay, so…
[00:45:04] That is the thing. That is the thing.
[00:45:06] But there are more to that than just knowing the LLM call. And that is.
[00:45:11] how the model learns and all those things. Anyways, we will.
[00:45:13] I think we have a lot of chance to learn about model tuning.
[00:45:17] We'll learn about fine tuning. I will show you one day how to fine-tune a model.
[00:45:21] Yes, Hemant.
[00:45:22] Sure.
[00:45:23] Uh, so whenever I just have, like, one question. So, it's from a previous class, and how we were discussing in the language graph, right? So, please have it inlet, while we are using the line graph, we need to make sure that the state is, uh, like, we need to carry the state.
[00:45:38] And, uh, we have the latest connection, and we are trying to carry the workflow execution, right?
[00:45:43] But one thing, like, by making all these things and passing through LLM, right?
[00:45:47] So, it's more of, like, a deterministic flow, right? Like, uh, just…
[00:45:52] Hmm.
[00:45:53] Having the probabilistic and deterministic nature, right? It's going to the word in this order, right? Like, just having an LLM call with the control flow.
[00:45:58] Then what exactly we can say? It's, like, autonomous agent or, uh, something…
[00:45:59] Hmm.
[00:46:03] independent agent kind of thing. So…
[00:46:05] Certainly.
[00:46:06] Yeah, it is a deterministic thing only. So, uh, there are agents where you require deterministic step.
[00:46:12] Inside the deterministic, there could be probabilistic intelligence that is involved.
[00:46:16] Okay, let's say…
[00:46:18] Let's say if.
[00:46:21] You want to create, let's say, for example,
[00:46:24] Marketing, ah.
[00:46:28] SEO, uh, description, SEO heavy SEO description at scale.
[00:46:32] You want to build a tool like that.
[00:46:34] Okay, that given the product name and you will do this. The example that we discussed, okay.
[00:46:40] So, in that case, the entire flow is deterministic.
[00:46:44] Okay, the only place, the only places where you can have probabilistic nature is the marketing part, audience, features, SEO.
[00:46:53] Final.
[00:46:56] Well, these things like got it, Hemant.
[00:46:58] So, under a deterministic flow, the flow is deterministic, but internal all the brain.
[00:47:04] To do this, to do the next step, you require the LLM call.
[00:47:09] Got it. So, it is a deterministic flow with
[00:47:14] probabilistic agents.
[00:47:15] Got it.
[00:47:17] Yeah.
[00:47:18] Now, in LangGraph, there is something known as Create React Agent.
[00:47:22] Uh, that I will show you. It is, uh, just like…
[00:47:26] Your.
[00:47:28] lasering and acting, right? Yeah.
[00:47:30] The, sorry?
[00:47:31] Reasoning and acting tool.
[00:47:33] Hi, this is an app tool, it is just like your, uh
[00:47:37] I'm forgetting that. The one that we have done last week, last to last week, create agent, create agent on Langchen Us on a
[00:47:42] you know.
[00:47:43] You provide LLM, you provide all your prompts together, automatically it creates the agent. Now, in Langchen that you saw.
[00:47:48] Yep.
[00:47:49] Same feature is there in LangGraph also.
[00:47:52] Just that in line graph, that agent that will be created.
[00:47:55] will internally follow a LangGraph workflow only.
[00:47:58] Just that that land graph is a black box.
[00:48:01] Got it. So, there is something known as create react agent.
[00:48:06] Create React Agent Lang Graph. If you do this.
[00:48:09] This is the same. This is a pre-built agent. This will build agent for you. You give the prompt.
[00:48:14] You give the prompt, all the tools that you will have.
[00:48:17] And you give the prompt like this, the model, the tools.
[00:48:21] While these things, it will automatically create the agent for you.
[00:48:24] And this is non-deterministic. This is again your LangChain agents, kind of like LangChain agents only.
[00:48:32] Yep, blur.
[00:48:33] What it? It is at the hope of your prompt that it will do work. So there are
[00:48:36] Use cases where you required that kind of a thing.
[00:48:39] Where you do not want agentic flow, you do not want a
[00:48:42] A deterministic flow. You want a non-deterministic flow. There are… there could be components inside your flow where you… this part you want little.
[00:48:50] Non-deterministic. That also can happen. You have a very big kind of a agentic flow.
[00:48:55] Outside that, one small part you require non-determinism. Some 3-4 steps, you want non-deterministic.
[00:49:02] Okay, that part, only that, those agents you can create using create react agent.
[00:49:07] Okay, like that.
[00:49:12] So, that is the thing, but anyways, lies, guys, let's come back to hierarchical agent, uh, uh, hierarchical agent parallel.
[00:49:18] branches, looping, edges,
[00:49:20] All these things. I don't know how many of you have seen the code.
[00:49:23] Uh, have you all seen this code?
[00:49:38] No, right? So that means…
[00:49:41] No
[00:49:42] Not seen yet, okay. Okay, so last code, this must we have done last week only, uh, the single.
[00:49:48] point agent, uh, like, one direction.
[00:49:50] Sequential Agent. Now we will do.
[00:49:52] uh, parallel agent, okay. So,
[00:49:55] Let's look at Parallel agent, how it is constructed using LandGraph.
[00:49:58] Okay, simple again from LangGraph, we import start graph, start and end.
[00:50:03] Rest everything, again, Python and Langchain only.
[00:50:06] Uh, we call the LLM first, and we take the open router key.
[00:50:16] Just a moment…
[00:50:24] partially share that link also.
[00:50:30] This link
[00:50:33] We have a workspace here.
[00:50:38] Hmm.
[00:50:41] This was shared last week also. It is done till here.
[00:50:44] Still here, we have already done it last week.
[00:50:48] Okay, so we'll start from here.
[00:50:49] Okay, so first of all, guys, now.
[00:50:53] We are doing…
[00:50:55] Okay, wait.
[00:51:00] Huh. So, over here, guys, we are basically
[00:51:04] First, we have a basic description.
[00:51:06] Okay, we have a basic description.
[00:51:08] Which is a single node, which is a node normally starting off basic description.
[00:51:12] And that basic description will branch out.
[00:51:16] into basic to features, basic to audience, basic to SEO.
[00:51:21] Okay, that is what will happen over here, so you have a basic description.
[00:51:24] Which is, you generate short product description, write a brief description of state product name.
[00:51:29] And basic description like this, uh, and whatever is the final output, that is the basic description.
[00:51:35] Now that is your basic description that you create.
[00:51:38] After this, you add something known as parallel edges.
[00:51:42] Parallel edges means C.
[00:51:45] Normally, it is a function only. The function is.
[00:51:48] List key and features benefits based on the basic description that you got.
[00:51:52] And respond. Similarly,
[00:51:55] Create, identify audience.
[00:51:57] Based on the basic description and respond.
[00:52:00] Similarly, generate SEO keywords based on basic description and response. So as you can see, all of the 3
[00:52:06] Nodes or three functions, depends on basic description.
[00:52:11] Okay, now merge. Let's look at merge.
[00:52:15] Uh, now look at the merging thing combined.
[00:52:17] Features, followed by
[00:52:20] The features that you generated.
[00:52:24] the target audience you generated and the SEO keywords you generated. So all these things, all these functions, you are merging it.
[00:52:29] So first, you start it from a process,
[00:52:32] Branched yourself, branched into 3.
[00:52:35] Three stages and then you are merging it back.
[00:52:38] You are merging back. That is what we are doing.
[00:52:41] Create a compelling marketing message using the combined all of this.
[00:52:45] So these are the three nodes.
[00:52:47] These are, uh, uh, sorry, not three nodes, five functions.
[00:52:53] Now, you might ask, like, how do you create the 5 function? Just now, it's just a… it's just a function only, like, how do you make sure that.
[00:52:59] the edges are connected. Now we'll go to the edge part.
[00:53:02] Okay, let's look at edge over here. Let's
[00:53:07] You know, look at H over here.
[00:53:16] Hmm. Okay.
[00:53:19] So, now we have a final polish.
[00:53:24] description as well. Final polities description, which is, which is taking the marketing message that you created by emerging.
[00:53:29] And taking the final, and then you have a evaluate function as well.
[00:53:33] And then you have an improved description. Can you
[00:53:36] See, till now, guys, always you have seen every agent is LLM powered.
[00:53:41] But I told you that LLM agent.
[00:53:44] Is not mandatory that they have to be, uh.
[00:53:47] I know they have to be LLM powered. They can be non LLM powered as well.
[00:53:51] This is a non-LLM-powered agent.
[00:53:55] So this node that we have created is non-LLM powered, there is no node.
[00:53:59] There is no LLM, LLM called.
[00:54:01] It is just a random function.
[00:54:04] that I have created to return a score.
[00:54:06] Ideally, you should have taken.
[00:54:08] I'll let him only to judge a score, but this is like a variety I'm giving you.
[00:54:12] A variance in the entire code, so that you
[00:54:15] can feel satisfied or get a closure that
[00:54:19] Yes, nodes can be built without LLM call as well. And this is that node.
[00:54:24] Which you are building without the LLM call.
[00:54:26] There is no LLM call in this.
[00:54:29] LLM calls this avoided.
[00:54:31] Okay.
[00:54:33] Now, let's go ahead.
[00:54:36] And…
[00:54:39] Let's build all the edges.
[00:54:41] Okay, so first you create an object of state graph, pass that state state dictionary that you have created.
[00:54:48] Workflow, add the nodes.
[00:54:50] First is basic note, features node, all the nodes you have added, all the, uh, what?
[00:54:56] 8 nodes you have added.
[00:54:58] Now you start building the edges.
[00:55:00] This is where you start building the flow.
[00:55:02] Addage followed by start.
[00:55:05] Too basic. Start to basic.
[00:55:08] Then, parallel. This is how you do parallel. Basic to featured, basic to audience, basic to SEO.
[00:55:14] See, this is parallel, you're fan out.
[00:55:17] Okay, now you'll go fan in.
[00:55:19] Panon is features to marketing, audience to marketing.
[00:55:22] SEO to marketing. All these things are fan in.
[00:55:26] Okay, all these three.
[00:55:29] Then you have…
[00:55:31] Marketing which goes to final.
[00:55:34] Okay, marketing will go to finance. So this is the architecture that we have followed.
[00:55:40] Basic, a start, basic description, basic to features, basic to audience, basic to SEO.
[00:55:46] Then features to marketing, audience to marketing, SEO to Marketing, and then final.
[00:55:51] And then we'll go to evaluate.
[00:55:53] Okay, see guys tell me till here.
[00:55:57] Is it a DAG or a non-DAG?
[00:56:20] What is it?
[00:56:21] So, it doesn't seem cyclic, so, uh…
[00:56:25] It should not be a drag, right?
[00:56:28] Um…
[00:56:32] DAG is acyclic.
[00:56:36] Actually.
[00:56:37] As a tag is this, I click OK.
[00:56:39] Hmm.
[00:56:43] So what do you think now?
[00:56:46] Till final.
[00:56:47] Yeah, it's a bag, yeah.
[00:56:50] It's a DAG, and why do you think?
[00:56:53] Uh, so, uh, there's a loop, right?
[00:56:59] Where is the loop?
[00:57:01] So we, we go through, uh, basic marketing and then, uh, it goes like that only, right?
[00:57:08] It's just a guess. I'm not like sure.
[00:57:14] Yeah.
[00:57:15] But I get it. I know that the book is fine. I'm just looking at your justification that what is your understanding of a DAC?
[00:57:18] Okay, what about the rest?
[00:57:24] I have told about that, right?
[00:57:25] I think it's DAG as well, because it's like, I don't know, in a single straight floor, right?
[00:57:30] It's not going back. I don't know if I'm using the right word, but…
[00:57:34] That is the right answer.
[00:57:36] It is not going back at any stage.
[00:57:38] Okay, it is not going back, it is, it is DAG, it is DAG till here, till final it is DAG.
[00:57:42] Okay, till evaluate also it is DAG only, but there is a slight change.
[00:57:47] Okay, see, it is not going back, guys. Directed acyclic graph is what you go only one side, acyclic, you don't compute a cycle.
[00:57:54] Okay, you go only one, one side, so…
[00:57:57] This is a DAG.
[00:58:00] Okay, so let's say.
[00:58:03] You have A, you have B, C.
[00:58:07] And then B and then.
[00:58:09] Oh, sorry. D.
[00:58:12] And then E. This is a DAG because you are going.
[00:58:14] Only one side, they are not going.
[00:58:16] back at any point. There is no concept of like this, you're going here and then you are completing a cycle.
[00:58:22] So there is no concept of
[00:58:26] You know, going back or completing a cycle or
[00:58:29] Something, you know, that is not there.
[00:58:32] Where you are coming back and then completing a cycle, that thing is not there. There's no concept of looping.
[00:58:38] Okay, so that thing is what we are breaking over here.
[00:58:42] Over here.
[00:58:44] is what we'll do next.
[00:58:47] Look at this edging technique.
[00:58:48] See, till here we have done, we have done, all these things are DAG, you saw till here is DAG.
[00:58:53] Now, guys, look at the loop section.
[00:58:57] Final, we'll go to evaluate first.
[00:58:59] Okay, then I am adding a conditional edge.
[00:59:04] What is a conditional edge? Conditional edge is evaluate.
[00:59:08] Will evaluate as the starting point.
[00:59:11] Evaluate will.
[00:59:13] Where will it go? From evaluate, where will I go?
[00:59:17] From evaluate.
[00:59:19] First, I will check if my state is improve or not.
[00:59:23] If my state is improve.
[00:59:25] Okay, if my state is improved, uh, sorry.
[00:59:28] If the quality score, if the quality score
[00:59:31] is below 6.
[00:59:33] In the evaluate, there is something known as quality score.
[00:59:37] If is less than 6,
[00:59:39] Then my state is called as improve.
[00:59:42] Else my state.
[00:59:44] Is called as end. So, this state is a temporary dictionary, a lambda variable.
[00:59:51] which will decide whether it will become improved or it will become end.
[00:59:55] Okay, so now if the state is improved.
[00:59:59] If the state is improve, then go to improve. From evaluate, you go to improve, so
[01:00:04] After evaluate, you go to Improve.
[01:00:07] Okay, that is only checked over here.
[01:00:11] If this is not improve, if it is end.
[01:00:14] So then end means end.
[01:00:16] End means that end that you have imported at the top.
[01:00:20] Katam.
[01:00:24] Got it.
[01:00:27] So, this is that end.
[01:00:28] So, if you, if you, you know
[01:00:30] strike end, then finish it off.
[01:00:33] Okay, or else if you strike improve, go to improve.
[01:00:37] So now what happens if you go to improve?
[01:00:40] Look at this improve, look at this structure, guys.
[01:00:43] Again, go back here, huh.
[01:00:46] He went to evaluate. From there, either you go to end or improve. If you go to improve,
[01:00:51] From improve where you are going, you are going back to evaluate again, look at improve, now look at this flow.
[01:00:59] Look at this. After improve valid, where will it go? From improve, you have to show the path, na.
[01:01:04] Or else that improved, you doesn't have out outside, like, you don't know where to go.
[01:01:10] From improve, it will again go back to evaluate.
[01:01:12] So, improve can be an agent.
[01:01:16] Where you can ask, let's say I don't like the output, please.
[01:01:20] You know, as the LLM, please come up with a better output.
[01:01:23] So improve is that. So improve can be an agent like that.
[01:01:28] Then again, we'll go to evaluate and this process will continue.
[01:01:31] So this is where from DAG, it becomes a non-DAG.
[01:01:37] But here we didn't mention anything is what to improve
[01:01:40] Here.
[01:01:44] Your improve is above.
[01:01:47] Okay.
[01:01:48] So, improved description. Improve this product description to make it more persuasive, and you can, you can provide the score also. That score is the… the score, that non.
[01:01:54] Non-eralemic score that you have generated. You can provide the score, the last
[01:02:00] uh, score I got was… you can pass the score also over here.
[01:02:03] Instead of state final description, you can pass.
[01:02:05] state quality score also. State quality score. Quality score is also a function.
[01:02:11] It's also a valuna.
[01:02:13] state cup. So you can pass that also, that in the last thing I got the score as 7.
[01:02:18] So please improve.
[01:02:20] Okay, our last thing I got the score as 4. So please improve.
[01:02:24] Okay.
[01:02:25] Okay. Ideally, ideally you will use this
[01:02:27] With an LLM powered.
[01:02:31] evaluation. Okay, randomness, though, it is not even checking the quality. It is just giving a score.
[01:02:35] I have done this Y, so that I can show you a variety that without LLM also functions can be created, nodes can be created, nodes can be without LLM also.
[01:02:44] Okay, so that people don't come up with… keep with this thing by heart that eh.
[01:02:49] Agents are always LLMs. Okay, agents can be without LLMs. So this is a forceful creation of a non.
[01:02:55] LLMIC agent.
[01:02:58] Her node where I'm giving a random using random to give the score.
[01:03:03] Okay. Anyways.
[01:03:07] So that is what it is. Now, guys.
[01:03:14] Our workflow is built.
[01:03:16] Okay. Now let's run it.
[01:03:25] Okay, so this is running, guys.
[01:03:45] Okay, at the 1st goal only, we got a quality score of 7, so that's why it didn't run again. Usually.
[01:03:50] If you run it, you can see the output first might come quality score 4 or 5.
[01:03:55] And then again, it will run.
[01:03:58] And again, it will go through this entire process.
[01:04:01] And then for me, this time it came once, you know, just in the one run only, but let me run it again.
[01:04:39] Again, it came, quality score.
[01:04:42] 7 only. Let us see again.
[01:04:44] I know you all won't be able to run it so many times, because you do not have the power of
[01:04:50] So much of API tokens also, I get it.
[01:04:52] You can run it once. I think few of you will get a quality score less than seven, less than six.
[01:04:57] But another one, just one question, so right now it is just a random score, right? It's not based on what inputs are doing. It's not actually evaluating the quality, no?
[01:05:02] No, that point I told already here.
[01:05:04] Ah, okay.
[01:05:07] Yeah, see, this is again 6 only. It has to go below 6. See.
[01:05:10] hard-code it to 5 or something.
[01:05:13] Yeah, so, yes, yes, yes, if you hard code it to SVE, then it will keep on generating and my OpenAI cost will go high.
[01:05:19] Okay, that's right. So, what you can do is make it probably 4.
[01:05:23] Okay, and then, uh, then this thing…
[01:05:26] You have a higher chance because
[01:05:28] Because, uh, because less than 6, you have one number, na.
[01:05:31] At 6 and greater than 6, you have more numbers, like 6 and 7.
[01:05:35] So that's why it is not finding it.
[01:05:40] Uh, a good space to generate that number, so if you have 4, then there are higher chances that
[01:05:46] You might have 50 50 chance.
[01:05:56] See, see quality score. It didn't come out. See, output didn't come out. Now it went to evaluate.
[01:06:02] It went to evaluate, it is improving the score.
[01:06:04] Okay, C.
[01:06:07] It has improved the score, and next time it came, quality score 7. Although this score has no depend.
[01:06:12] Nothing, nothing, no connection to that evaluate.
[01:06:15] Okay, you can see it could be a possibility that after this also another time quality score 5 only came.
[01:06:20] Could be a possibility.
[01:06:23] Okay. Anyways, so this is what it is guys. So first of all, understood guys, what is hierarchical parallel all these concept guys.
[01:06:32] Any question from here?
[01:06:40] Sir, actually this type to take, it is more like validation, correct
[01:06:46] It is,
[01:06:47] Uh, typedict is, ah, not a validation. You are very close, Prakash, it is fine to think of it as a validation.
[01:06:53] Because it looks like that, because you have seen like this. So this typedict is basically
[01:06:58] is used to make a class.
[01:07:02] which behaves like a dictionary.
[01:07:05] See, this class, we have not created any function.
[01:07:08] Can you see that? Prakash?
[01:07:10] Yeah.
[01:07:11] This is a class. We have not created any function.
[01:07:14] Usually TypeDict we use for that only.
[01:07:17] Like, I have also used Typedict first time when I used LangGraph only, when I first learned LangGraph.
[01:07:21] Okay, before that, we never used it. Okay, so typedict is usually used when you want to create.
[01:07:28] a class which just has.
[01:07:29] key-value pairs, key value pairs, that's all. And that can be used as a state
[01:07:35] Dictionary, graph dictionary.
[01:07:38] Got it. So this dictionary is shared across everybody.
[01:07:41] This is like a ball that is carrying.
[01:07:44] and information, and you're playing passing the ball against.
[01:07:47] With all the agents.
[01:07:49] Yeah, pocket
[01:07:50] And every person is adding something on that.
[01:07:52] Every person is adding something on that. Let's like a Chinese whisper.
[01:07:57] Okay. You are passing.
[01:08:00] A thing by telling it onto somebody's ears and that person might be adding.
[01:08:07] by mistakely something, and that's how the Chinese whisper.
[01:08:09] grows or changes, you know, over the time.
[01:08:13] Got it, it is like state information, status, stateful, you call stateful.
[01:08:18] Okay, it is stateful. This is what makes the agent stateful.
[01:08:25] Okay.
[01:08:26] Yeah, got it.
[01:08:32] Edges, edges.
[01:08:33] Always, we need to create node, and even we need to create the edges and
[01:08:37] It just
[01:08:38] And we need to connect those, things also node to HS
[01:08:39] Hmm. So you might have an agent.
[01:08:42] Usually, our mini-projects are this only, I'm telling you, now only.
[01:08:46] It'll be a customer support chatbot.
[01:08:48] It will have.
[01:08:51] user query that will be
[01:08:54] Given to IT department, HR department.
[01:08:57] Some other admin department.
[01:08:59] From there,
[01:09:01] That question.
[01:09:03] Based on that department, we'll go to that
[01:09:06] Vector DB and will only generate answer from there.
[01:09:10] And give you the answer. So this flow that I told you just now, this flow, you can, let's say, build it using LangGraph.
[01:09:19] Now…
[01:09:20] It is rack, correct?
[01:09:22] It is a rag
[01:09:23] Huh? That, no, rag is the one of the part of it.
[01:09:25] One of the node can be RAG.
[01:09:26] Okay.
[01:09:30] Okay.
[01:09:31] Rakesh, got it? See, you, you created, today you are creating a chat bot for me.
[01:09:33] I am your, I am your ed tech company, I have come to you.
[01:09:37] And Leo that in my edtech companies there are
[01:09:40] Students who ask questions, there are
[01:09:42] Employees also who ask question.
[01:09:44] And there are admin people who also ask questions. LMS-related question.
[01:09:49] Okay, so all the student queries will go to student.
[01:09:52] All the admin-related query will go to LMS-related query will go to admin. Admin vector DB.
[01:09:58] And you can have another vector DB just for
[01:10:02] What to say.
[01:10:03] The, the other one which I mentioned, HR.
[01:10:06] Okay, then.
[01:10:08] You can have a LLM node at the top, which based on the user query, judges where to send.
[01:10:16] Got it.
[01:10:19] Okay.
[01:10:20] This is like a guided rag.
[01:10:23] Okay
[01:10:24] Before even going to the RAG, you are judging the intent of the question.
[01:10:28] Based on that intent, you send it to the right RAG. You have 3 rags.
[01:10:32] Three vector DVs.
[01:10:34] Got it.
[01:10:36] So that 1 VectorDB doesn't have every all the information together.
[01:10:42] Yes.
[01:10:43] Got it. So, this…
[01:10:44] Architecture also people create. Till now, guys, you have been only dealing with one vector DB. Let's say you have three vector DB.
[01:10:52] Many people, there is a concept known as multimodal RAG.
[01:10:58] Very popular, but it has not scratched the surface so well.
[01:11:02] Okay, one of the company product, company tool that is really doing well is layout LLM.
[01:11:09] Okay, layout, uh, sorry, layout LM.
[01:11:11] Okay, by Google, which is multimodal RAG only.
[01:11:15] Where it in multimodal drag, what happens is in a PDF there could be images as well as text.
[01:11:21] So, usually with images, our models are not so good. You saw diffusion models, that is one thing you saw at the start and that time also I told you.
[01:11:30] That diffusion model is just at a stage.
[01:11:33] of how…
[01:11:35] GPT was in 2022.
[01:11:38] It has just scratched the surface.
[01:11:40] I still now don't find AI.
[01:11:42] Video are so good, so.
[01:11:45] So, undetectable, I don't find it, I easily detect it, ok.
[01:11:49] There are certain symbols by which you easily understand, right, eye movements, then fingers.
[01:11:54] Okay, all those things, and then the texture, then the movement of the.
[01:11:59] Uh, object, or let's say the movement of the human.
[01:12:02] Okay, you automatically understand.
[01:12:04] So, AI.
[01:12:06] Image models are at a stage of 2022 now with diffusion, with diffusion, they have cracked a little bit, like with GPT.
[01:12:13] 3.5, they have cracked.
[01:12:15] Amazing thing and then actual.
[01:12:17] Good things started happening in the last two years with coding agents and all these things, where people
[01:12:22] So much of agents started happening, right.
[01:12:24] So, but in 2022 also people were just using ChatGPT as a chat mode only.
[01:12:29] Similarly, image is also at a very limited kind of a space now.
[01:12:33] So, what I'm trying to say, multimodal RAG.
[01:12:37] Is good, but it is not so good that I can boast about it.
[01:12:41] Okay, so multimodal lag is also built in this concept, Prakash.
[01:12:45] Where, based on user query.
[01:12:48] You convert that user query.
[01:12:52] And you convert that into
[01:12:54] Image embeddings, like an embedding which image models can understand.
[01:12:58] And an embedding which is text embedding like BERT embeddings.
[01:13:01] And then you send…
[01:13:03] The query into something that image model you can relate to the image model.
[01:13:08] And you can send the query, same query through the text vector DB as well. So, there are two vector DB. One is which has the images of the entire PDF. Let's say you have PDF five pages, there are three images.
[01:13:19] You have created a vector DB of those 3 images first.
[01:13:22] Got it, Rakesh.
[01:13:24] Yeah, got it, but how it will differentiate which, better DB it has to check it is with metadata, or how it will check?
[01:13:31] No, no, no, that is the, every query will be sent to both of them, Prakash.
[01:13:36] Then, at the consumption, it will be more, correct? The… it is the…
[01:13:40] Definitely, you are building a multimodal rack, you cannot put constraint on every site.
[01:13:44] You are building a multimodal rack, you will have to open up something or else how will you understand?
[01:13:53] Uh, you can, you can.
[01:13:54] Otherwise, we can keep some filtering in middle, like, with metadata.
[01:13:55] Ah, you can, you can, every embedding you convert forcefully into.
[01:14:00] Let's say forcefully into image embeddings and text embeddings and send it to them.
[01:14:06] And uh… and then.
[01:14:09] based everything like both the time.
[01:14:12] It will retrieve an image and it will try to retrieve a text also and
[01:14:16] Both will be sent to the LLM.
[01:14:20] Before coming up with an answer, okay.
[01:14:22] Got it.
[01:14:24] Okay, we'll see, correct one example
[01:14:27] Sorry?
[01:14:29] Yeah, in our classes, we'll see one example correct with multiple
[01:14:31] Uh, multimodal rack in a different way I will show you, I will try to show you 1 multimodal rack, I will.
[01:14:37] Because multimodal lag, it's little difficult to handle, like I have tried to build it multiple times using
[01:14:42] Unstructured unstructured loader. I have failed it because of GPU requirement.
[01:14:47] Okay.
[01:14:48] Okay, I will see that if similar to something like multimodal RAC can be created.
[01:14:53] Okay.
[01:14:54] Okay, not exactly like an enterprise-level multimodal rack because
[01:14:56] Usually unstructured loader that you require to load these PDF into both text and image.
[01:15:04] Usually it takes GPU access. Okay, so I tried to build the same thing in Oracle, I failed miserably, because Oracle was
[01:15:11] Oracle couldn't provide me the GPU, first of all.
[01:15:14] Okay, because of, you know, this was a R&D project and every R&D thing is an investment.
[01:15:20] Okay, first of all, paying my salary was an investment for them.
[01:15:23] Again, that GPU is also another investment. So whatever limited option you have, you do it on that.
[01:15:29] So we never approached multimodal RAG after that. I tried whatever I have with the limited.
[01:15:33] So what I'm trying to say is, after that, I explored it.
[01:15:36] But everywhere I have faced the issue of GPU only, multimodal rack. So what I will show you.
[01:15:41] He's, I will take an image separately only. I'll not take a PDF.
[01:15:46] I'll take an image separately, I'll create image embeddings.
[01:15:48] You may separately. I will take a text separately and I will create. And I will search from both of them.
[01:15:53] Get both of them and give it to the LLM, and…
[01:15:57] Give you an answer. It's just that instead of using unstructured loader, I'm just actually using an image.
[01:16:03] This is the difference I will do.
[01:16:07] Nope
[01:16:08] Because unstructured loader failed for me, rest of the things are simple.
[01:16:10] Got it.
[01:16:12] You got it.
[01:16:13] So there is a loader known as unstructured loader, which actually is used
[01:16:17] to load PDFs with images.
[01:16:19] Which will, which will help you to separate out image as well as text as well.
[01:16:24] Okay, so it converts all your PDF into HTML.
[01:16:27] And if you once convert into HTML tables, then you can easily understand that.
[01:16:32] every, uh, which table rows has this image, which table.
[01:16:35] So that way you can map. Okay, so you have a mapping.
[01:16:40] That, let's say, third…
[01:16:41] table row has this image. Fourth table row has a text.
[01:16:46] Fifth table row has image again.
[01:16:49] So, you can create a mapping, you know where to store what.
[01:16:51] So, that unstructured loader usually fails on our system.
[01:16:55] I didn't try it on Colab, although, like, with Colab's GPU have to see.
[01:17:01] But on my local, it has failed.
[01:17:03] System provided by companies, it has failed.
[01:17:06] So what I'll do is instead of using unstructured loader, I will use the actual image.
[01:17:12] and actual text and combine them together and create two vector DBs. One is an image, one is text.
[01:17:17] And once the query comes, in both of them, I will search, get the answer, give it to the LLM.
[01:17:23] And give you an answer. That is a multimodal rack.
[01:17:25] Just that I am avoiding the use of unstructured loader.
[01:17:30] Got it, Pragash.
[01:17:32] Yeah, got it, sir
[01:17:33] Yeah, it's a stopgap solution I'll build.
[01:17:35] Yeah, Jugard, Jugard, basically.
[01:17:39] Okay, so now guys, this is the point I am trying to make that, uh,
[01:17:44] There are systems where you can have these flows, there are systems where you might not have these flows.
[01:17:50] Uh, where you might have a non-deterministic, for that you use that LangChain create agent also is there.
[01:17:56] Then you can use this create react agent also. So this, when we are doing that.
[01:18:01] mini projects and then
[01:18:03] In September, we have only project, project, project.
[01:18:06] over there in one of the projects I will build this, and that way, you know, our time will also be utilized, and you will.
[01:18:11] see actual project rules.
[01:18:13] Okay, on the spot, it will build a pre-built agent for you.
[01:18:16] And automatically it will, you know, do everything.
[01:18:20] See, you just write like this, provide the tool, provide the LLM.
[01:18:23] It automatically builds the agent for you.
[01:18:25] Okay, it knows whom to call, what to call, all the same like React agent, uh, create agent.
[01:18:31] Of LangChain, just that over there.
[01:18:34] Internally, it was using LangChain expression language or whatever LangChain was internally using.
[01:18:38] And over here, LangGraph uses
[01:18:41] Graph concept, that's the thing.
[01:18:44] Okay. Okay.
[01:18:46] Now, guys, I will just take 7 to 8 minutes break. I've talked a lot.
[01:18:52] Uh, so just a 7 to 8 minutes week. I come back, guys, we will.
[01:18:55] Uh, do Lang Smith. We'll start off with Lang Smith a bit.
[01:18:58] There is one more part in this code.
[01:19:04] That we will do after Lang Smith because Lang Smith is more important.
[01:19:07] In this code, that is below this code, that is gradual interface.
[01:19:10] It is just like Streamlit. This we can do tomorrow as well.
[01:19:14] But first I will do Lang Smith, okay? Because Lang Smith requires you all to
[01:19:18] sign up and all those things as well.
[01:19:24] Okay guys, see you in 5 to 7. But guys, everybody, are you all understanding whatever we have discussed? I have
[01:19:29] I've really gone very, very slow in this land graph part.
[01:19:33] Okay, very slow. I have taken a lot of time to clear this line graph, so many doubts you all had.
[01:19:39] All those things, okay? Because Landgrav is super simple, guys. Now, do you feel, guys, Langraph is simple only?
[01:19:47] If you know Python, it is very simple.
[01:19:50] It becomes difficult for people who don't know Python, who only knows.
[01:19:54] Uh,
[01:19:56] Who have just learned Python, they feel QAI is more simple because in QAI you're writing text, no.
[01:20:02] Role goal and all those things. It's very, very English-English to you.
[01:20:07] Okay, that's fine. Okay. Anyway, see you in seven minutes, guys.
[01:28:21] Okay, guys.
[01:28:31] Any more questions?
[01:28:39] Huh. No, so Shivans…
[01:28:51] There is no concept of
[01:28:55] Manager or year deciding over see land graph.
[01:28:59] There are 3 ways you can build it.
[01:29:03] Okay, either you build it.
[01:29:05] Sequential workflow, or you build a hierarchical workflow where you are only determining the agent.
[01:29:11] Okay. Or…
[01:29:13] You build something like Create React Agent.
[01:29:16] Where your agent, there is no manager, the LLM only will take a decision.
[01:29:22] Like how we did in LangChain.
[01:29:27] Okay, got it.
[01:29:34] Or, since if you want to,
[01:29:38] You know, incorporate the same logic over here.
[01:29:42] What you can do, just like that scoring metric,
[01:29:45] You can have one LLM.
[01:29:47] And tell everything to the LLM, like give the entire state graph access to the LLM.
[01:29:53] Until, like, from here.
[01:29:56] These are the agents that I can go to.
[01:29:58] Which agent should I go to? So you can use that same Lambda kind of a thing lambda.
[01:30:03] If the agent says this, then go to this.
[01:30:06] Like that. Like how you saw improve, nah.
[01:30:10] Yes.
[01:30:11] Same thing, you just have to over here, you do not have a dedicated manager for that.
[01:30:15] You have to write that programmatically.
[01:30:18] What is she wants?
[01:30:20] That make it.
[01:30:22] Instead of making that random score.
[01:30:25] Random and non-almic, you make it LLMic.
[01:30:29] And based on that LLM, you take a decision.
[01:30:32] that outside this function node, the output will be basically a name of the next node it will go to.
[01:30:40] That also you can do.
[01:30:47] So that is one more way that you have to achieve it programmatically.
[01:30:52] That there is no dedicated agent, see, their concept of agent is not there.
[01:30:56] In the entire LangGraph, LangChain, the concept of
[01:30:59] agent, user agent,
[01:31:01] manager, agent, customer agent, all these things which is there in AutoGen.
[01:31:05] And QAI, it is not there.
[01:31:07] LangGraph is more of a graph concept.
[01:31:10] Like, in a graph, a node is just flowing.
[01:31:13] Now inside that node, how much brain you want to apply, you can apply.
[01:31:18] What is she wants, that is the difference.
[01:31:20] Didn't it?
[01:31:23] Yes, it is a workflow based on graph.
[01:31:24] Yeah, so Langgraph, we can act as a workflow and then internet
[01:31:32] Huh.
[01:31:33] Yeah, and they have some internal or low… low code, they have an engine, which is automatically taken care, so we don't need to define explicitly manager here.
[01:31:35] So, so in Crew AI, it is more of a agentic framework.
[01:31:43] This is also agentic, but there is a difference. Over there, it is designed as agent, agent, agent, agent like that.
[01:31:48] So you create agent also by using that term agent, no.
[01:31:51] In this, have you, have you seen me using the term agent in any of the 2 agents? No.
[01:31:58] So, it is ultimately a workflow that they have created to flow work. So it is a more of a workflow framework than an agentic framework.
[01:32:06] It is a workflow framework used for agentic.
[01:32:08] Use for Agentic
[01:32:09] Yes.
[01:32:11] If you see Autogen also, you will relate to this more, you, then you will say the LangGraph is the only thing that is.
[01:32:17] That is quite, quite unique.
[01:32:19] Because autogen is almost like UVI only because in autogen also you have agents, you have 2 agents talking to each other, you have 3 agents talking to each other.
[01:32:27] Okay, so over there also you have the concept of agent.
[01:32:31] Okay.
[01:32:32] Only in this land graph only, you don't have anything as agent. It is all nodes.
[01:32:35] That is talking, uh, that is passing information to the another one.
[01:32:40] And that they converted that to agentic architecture.
[01:32:46] Understorm.
[01:32:47] So, that is the principle design or fundamental of Landgraph versus Autogen versus ClearAI.
[01:32:56] Sure.
[01:32:57] Yeah
[01:33:00] Okay. Now guys, let's go to one thing that is Lang Smith. How many of you over here has used Lang Smith?
[01:33:15] Have any one of you used Langsmith?
[01:33:17] No, from my side.
[01:33:19] In the class no one. Gunjan.
[01:33:25] What is the framework you use? Deepak?
[01:33:27] Deepakadara.
[01:33:34] Custom, custom. Custom.
[01:33:35] Actually, I have built an observability stack for one of the… I mean, OCR-based image analysis custom one using the Prometheus and then Grafana
[01:33:49] Hmm.
[01:33:50] Also, for traces and logs, I use the Loki and Tempo as a data source. But dashboard wise, I built everything in Grafana
[01:33:52] Uh-huh. Okay, got it.
[01:33:54] Deepak Kadara.
[01:33:58] Have you used it for SRE or anything?
[01:34:00] Because observability is your field basically in SRE it's heavily logging.
[01:34:06] And all these things are heavy enough trace.
[01:34:10] Dorakesh, have you heard of any Lang Smith or anything?
[01:34:16] Okay. So I guess nobody has heard of it, guys. It is very interesting. So I will just share with you one.
[01:34:24] Notebook.
[01:34:26] Okay.
[01:34:28] Just give me one minute, guys. I will add all these contents also.
[01:34:43] Okay.
[01:34:46] Take it.
[01:34:50] Okay.
[01:35:05] Sharing it, just a moment.
[01:35:13] Yeah.
[01:35:20] Okay, so guys with the Lang Smith, we are entering into the space of LLM evals.
[01:35:27] LLM events, we have half we have already entered. We have done a bigger LLM evals.
[01:35:32] framework than Lang Smith, that is Ragha's.
[01:35:34] Ragas is more.
[01:35:37] Off our LLM equals than Lang Smith. Lang Smith is more of a tool.
[01:35:41] that is used for debugging, for logging and all these things.
[01:35:46] Ragas is both framework.
[01:35:48] Plus, it's a metric also, like Ragas provide you lots of metrics, right? You saw.
[01:35:53] uh, non-LLM similarity, then.
[01:35:56] Leventstein distance, then.
[01:35:59] Oh.
[01:36:01] What was that? Answer relevancy, answer faithfulness, answer correctness.
[01:36:07] Those things are metrics of ragas. So ragas itself is a metric system plus.
[01:36:13] It's a framework also. They have created a framework also. Langsmith is just a library.
[01:36:17] or just a library framework.
[01:36:19] It doesn't have any internal metric of its own. Length is just a tool.
[01:36:23] With which you can observe your agent ops and LLM ops. So this is part of LLM ops only. LLM ops, agent ops, all these things.
[01:36:30] So ops is basically this phase of evaluating.
[01:36:33] And then going back, making changes, coming back.
[01:36:36] Again, evaluating, going back, changes, coming back, so that eval thing you saw.
[01:36:40] Inside that Landgraph node, that is what ops is, ok.
[01:36:44] The ops part involves that. I'm not talking about DevOps. DevOps is different, okay? I'm talking about LLM ops and
[01:36:49] agent ops. Okay, so now this space.
[01:36:54] About lang, uh, ops is very
[01:36:58] Is very non-standardized, that's why I found no people have used language. Zero people have used LangSwidth.
[01:37:04] Many people have built.
[01:37:06] agentic, but they have not built ranks with. But does that mean those people have not done
[01:37:11] Evaluation or LLM ops, no, they have done it, they have done it using probably
[01:37:16] custom framework. They have built a custom tool of their own.
[01:37:21] Or they have used some other…
[01:37:22] old tools which were used for other tools, like Grafana was there before AI also.
[01:37:27] So, they have used Grafana, Prometheus, these kind of tools which are
[01:37:32] Uh, non AI focused tool, they were created before that, so they can be used now also.
[01:37:38] Like that. Those tools are also there, but there are tools that has been exceptionally created to do this observability and all.
[01:37:45] And for that, we use something known as Langsmith, Langfuse. These things are there.
[01:37:51] Uh, yeah, mostly…
[01:37:54] Uh, these only. There is another one that is there, that is not coming.
[01:37:59] To my mind just now…
[01:38:02] It has something to do with
[01:38:05] Something to do with some sort of an animal, it is named after an animal.
[01:38:09] Okay, that is also there. But anyways, Lang Smith, Lang Fuse are the two most popular.
[01:38:15] Once again,
[01:38:17] Many people, uh, do build these dashboards using
[01:38:22] Let's say, uh…
[01:38:24] Using plain database and Python as well.
[01:38:27] Using plain Python matplotlib as well.
[01:38:30] Okay, so now let's come to Lang Smith, guys. We will talk about this.
[01:38:34] Thing about Lang Smith and the industry usage a little more later also.
[01:38:38] But now, first of all, guys, you all come over here.
[01:38:41] And go to Langsmith.
[01:38:57] I'm a little surprised guys, nobody has used it.
[01:39:00] In other session, I have seen people, at least, they have touched it.
[01:39:05] Uh, okay. Okay, anyways, uh, this is the website.
[01:39:12] Yes, I'm sharing the link to this notebook.
[01:39:27] I have, uh…
[01:39:30] Did I add it? I'm adding it here as well.
[01:39:46] Yeah. You can take from here as well.
[01:40:02] Okay.
[01:40:08] Okay, everybody has landed onto this page.
[01:40:11] Now, don't change any region or anything, just…
[01:40:14] Sign up using your Google account.
[01:41:02] Please, guys, sign up with your Google account, please.
[01:41:11] And let me know if it is done.
[01:42:06] Okay, my Chrome all of a sudden crashed.
[01:42:14] Guys, if it is done, signing up and let me know.
[01:42:38] I am getting unexpected
[01:42:41] I know this problem will happen for few people. Let me see.
[01:42:45] I am adding a crew UI data project tracker sheet also, guys.
[01:42:49] This is the project tracker that we all will be using to submit your projects, okay? Many projects, LAK, all these things, okay.
[01:42:55] So…
[01:43:00] Let me share my screen.
[01:43:03] Okay, so this is the sheet also I'm adding over here, guys, so all your projects, GitHub.
[01:43:07] What you have added?
[01:43:10] Please add it over here, guys, instead of sending me, I should have done this last week only, but I didn't get that.
[01:43:16] Thing in my mind at that time, so.
[01:43:18] Please add all your projects inside this.
[01:43:21] Inside the project tracker, you will find a column.
[01:43:25] The column name also, I'm writing it as
[01:43:28] Uh, GitHub links.
[01:43:31] And your full name.
[01:43:34] Your full name.
[01:43:37] Okay, add all your projects over here, guys.
[01:43:39] Okay.
[01:43:45] Okay, so all your GitHub links will be over here.
[01:43:48] And a full name. This is for the crew VI. Same, we will create for other projects as well that we will do.
[01:43:53] And any project I do.
[01:43:55] Uh…
[01:43:58] That I will create a separate sheet and I will.
[01:44:01] keep the GitHub there. So I'm adding it here. Please add back. Okay.
[01:44:06] Now, Braghash, show me what is happening. Show me your screen.
[01:44:34] I'm able to
[01:44:35] Can you go back?
[01:44:40] Go back.
[01:44:45] Uh, go back, go back.
[01:44:47] No, no, just open a different page known as Lang Smith.
[01:44:52] Different tab.
[01:45:08] Is India's system very slow?
[01:45:14] Hello.
[01:45:18] Rakesh.
[01:45:23] Okay, it bent away. Okay, everybody else, others, what about others?
[01:45:31] Dun, dun.
[01:45:32] I guess majority of us are done, uh…
[01:45:33] Done. Okay.
[01:45:35] So, great. So if you are done guys, let's go to the next step.
[01:45:41] Very, very interesting.
[01:45:42] Okay, then this page, if you see the project, this is the same our product description project only.
[01:45:48] There is no installation of Lang Smith. Lang Smith is so easy to integrate with LangChain and LangSmith project… LangGraph projects that you don't do anything.
[01:45:57] Just have to provide these three lines.
[01:45:59] Into your project.
[01:46:01] Langsmith API key, you have to provide an API key that we will get it from Langsmith.
[01:46:07] Lang-Smith tracing equals to true.
[01:46:09] And length with project equals to, you give a name to this project, this project name will be shown in your Langsman dashboard.
[01:46:14] Okay, so these are the few things that you will have to show.
[01:46:17] After you do this, it will ask you for the open router key and the Lang Smith key. How do you take the Langsmith key?
[01:46:23] Go to Settings.
[01:46:26] And there is an API key. Create an API key.
[01:46:28] API key.
[01:46:30] Give a description testing, personal access token, PAT only.
[01:46:34] Create a key and copy that key somewhere.
[01:46:39] Copy that key somewhere.
[01:46:43] Copy and keep that key somewhere.
[01:46:44] Oh,
[01:46:48] Yes, yes, yes, yes, yes, yes, yes, Chandrasekhar.
[01:46:51] So, go to your Langsmith.
[01:46:54] Uh, settings…
[01:46:57] API keys.
[01:46:59] Length settings API key plus API key.
[01:47:48] And this project of LangGraph, Langsmith, have you shared that link? Because it is not in the document itself.
[01:47:55] Is it?
[01:47:56] It's there, it's there, this one. Yeah, yeah, this one. Langra plus Langsman. Aha.
[01:47:59] Okay. The length tracing one, right?
[01:48:03] Since October.
[01:48:10] Uh, Anavan, can you share this link again? I somehow lost it.
[01:48:21] No, no, this document on Google Doc.
[01:48:22] Here it is. Oh, sure, sir.
[01:48:23] I haven't lost it so.
[01:48:56] Got your API keys, guys?
[01:49:02] Got it. Now guys, start putting the API key guys.
[01:49:07] I am putting my open router key first.
[01:49:09] Then my Langsmith key.
[01:49:14] What I'm putting.
[01:49:15] Same agent I have built, no problem in this, same code, no change in the code.
[01:49:21] Same workflow.
[01:49:23] Now I run the agent.
[01:49:25] Now guys, look at Lang Smith.
[01:49:28] See, this is my screen, na? This was my screen, go to tracing.
[01:49:37] Refresh.
[01:49:43] Wait, wait, wait, wait, this is not coming up.
[01:49:54] Yes.
[01:49:56] See, 8.
[01:49:59] So you will see an entry of your project name, my project is already there because I have run this before also.
[01:50:03] Okay, so 3 days ago, only I ran it.
[01:50:05] So I'm seeing the entry. This is the entry.
[01:50:08] The start time is 8-8, just now I started now.
[01:50:11] So, he did. My project got started over there, 646 it started.
[01:50:15] See, if you click here, you can see this is your first agent.
[01:50:21] This agent has…
[01:50:23] Uh, 27 tokens, input tokens, 99 as output tokens.
[01:50:28] Total 126 tokens.
[01:50:30] Second agent is audience. This has 115 input tokens.
[01:50:35] 286 output tokens total as 401.
[01:50:38] A 9%, 21% of your token is 1,000 or 100% like that. This is the cost as per the open.
[01:50:44] GPT-4o mini. Okay, there's the costing also that is given. Every agent.
[01:50:48] And if you want to see the output after every agent, that also you can see, see 1st agent only had smart water bottle.
[01:50:54] Second agent, it got the 1st basic description.
[01:50:58] Third agent, it got…
[01:51:04] Yeah, it got the key features, this is the better, better way of looking at it.
[01:51:08] Fourth agent it got.
[01:51:10] The SEO outputs.
[01:51:12] Marketing, it got everything, like…
[01:51:15] Why to choose and everything benefits.
[01:51:18] I think then final, this is the final.
[01:51:22] It went to evaluate.
[01:51:25] Evaluate score was 5, which is less than 6.
[01:51:28] So it went to improve.
[01:51:30] It went to improve.
[01:51:32] Okay, it has improved this.
[01:51:35] It has got the improvement done.
[01:51:36] And then again it went to evaluate.
[01:51:39] The score became 6, so it got accepted.
[01:51:44] Okay.
[01:51:47] So, this is known as model tracing, guys.
[01:51:51] Agentic, agent tracing.
[01:51:55] Apart from this,
[01:52:01] See, over here, you can get a total cost, total tokens utilized.
[01:52:05] All these things, there are more things that we will see next day.
[01:52:08] You will see LangSpace evaluator also, this part.
[01:52:12] data sets and experiment. This also will see next day.
[01:52:15] This is more interesting, but this is what is known as Langsmith, guys.
[01:52:19] Same thing you can do using Langfuse as well. That also I'll show you tomorrow. So we'll show you Langfuse tomorrow, we'll see Lang Smith.
[01:52:25] Evaluators tomorrow will see gradu as well tomorrow.
[01:52:28] And after all that is done, we will
[01:52:30] Start a little bit of intro about Autogen as well.
[01:52:33] Guys, got it?
[01:52:35] Everyone.
[01:52:39] Now, uh, people who have done Grafana, like, do you relate to this, Sachin?
[01:52:48] Yeah, yeah.
[01:52:50] Yeah, in such, probably you have built evaluators also.
[01:52:54] That we will do next, ok.
[01:52:55] Uh, after this, uh, so over there you will see how you can bid LLME valves as well.
[01:53:01] So that is more utilization of Langsmith. This is just, just to see.
[01:53:05] model tracing, and how your…
[01:53:07] Each model where it is going.
[01:53:10] And at every step, because over here, guys, you are generating the output only, so you're not seeing at every step and
[01:53:16] How much token is it going?
[01:53:18] Now, you might ask, how, like, how does it decide
[01:53:21] What token everything. So, guys, remember.
[01:53:24] In the, when I was explaining RAG for the first time, I had
[01:53:25] Anirban.
[01:53:29] shown you that when you make a LLM call, you get the content, and along with that, you get some metadata as well.
[01:53:35] In that metadata, you have total number of input tokens, output tokens, everything.
[01:53:38] Those data are coming over, important coming over here.
[01:53:41] That is used by LangSpith. So your LangSpith uses that data.
[01:53:45] So, Anirban, what exactly is the use case of LangSmith, if you can explain it in, like, simple terms? Is it to track
[01:53:51] Ha, simple term, it is to track.
[01:53:54] How much tokens are you using as a company? Because
[01:54:00] Correct.
[01:54:01] People… see, you will not build one agent, right? Let's say you are an AI engineer, I am AI engineer.
[01:54:04] 3-4 AI engineers are there.
[01:54:06] You have built one.
[01:54:08] LLM API and that LLM API is used by left, right and center by everybody.
[01:54:14] Let's say you have built some usable agents.
[01:54:18] That agents are used by other people also.
[01:54:19] So, like this, your products are scaling.
[01:54:22] They're becoming bigger.
[01:54:23] Okay. Okay. Okay.
[01:54:24] Now, from where Anirban is calling this LLM,
[01:54:29] which you created okay or you have created one agent. I'm calling that agent.
[01:54:34] We need to track all these things, right?
[01:54:36] That why this call, why this LLM call suddenly spiked up the token cost.
[01:54:42] Got it, so it is an observability platform.
[01:54:45] Which is used to track how much cost, how much tokens it is being utilized.
[01:54:46] Okay.
[01:54:50] These things kind of things doesn't happen at the scale at which we are operating now, like, in the class and all, it's… you might be thinking, like,
[01:54:57] What is the use of it? But…
[01:54:59] Once you scale, once your product becomes scalable, like after 2-3 months, you will realize that, no, I need to track my tokens also.
[01:55:07] Got it. So, let's say…
[01:55:08] Okay, okay. And does this give…
[01:55:09] Sorry, yeah, go ahead.
[01:55:11] Yeah, let's say one of your LLM is too big, one of your agentic architecture too big.
[01:55:15] that one of its agent is calling all of a sudden 20,000 tokens.
[01:55:20] Okay, and you need to evaluate that.
[01:55:23] So, this is a very clear picture, it gives you that dashboard kind of a thing and more runs you do, more runs you can also see, so if I run it again now.
[01:55:31] More runs will be shown over here. Multiple runs will also appear.
[01:55:32] Okay.
[01:55:36] Got it.
[01:55:37] Okay. And does it then have a…
[01:55:38] Yes.
[01:55:40] I don't know, capability to then say, key, if something is not happening in the right way, or does it then give some…
[01:55:45] Extra insights, or this is just observability as a platform and nothing else.
[01:55:47] No, no, that is… that is observability. The insight is your decision.
[01:55:51] Okay, this will just show me this is happening. It's my headache.
[01:55:55] Ah, it's a dashboarding tool. Now dashboard ko leki kya karnai, that is up to you.
[01:56:00] Okay.
[01:56:01] Okay, and since it's a dashboarding tool, it allows me to configure alerts, etc. as well, or it's just a visual dashboard.
[01:56:05] No, no, this is, this is just tracing. After this, I will show you custom.
[01:56:08] Okay?
[01:56:09] evaluators when you write, then you will see that, uh, that, uh, that using certain evaluators, how it is behaving, like, these dashboards will come out.
[01:56:18] Okay, so like these things will come out.
[01:56:22] Got it. So…
[01:56:23] And is there any alternative for LangChain, or it's only LangChain, uh, that is one of the…
[01:56:25] Any alternative for Langsmith is Langfuse.
[01:56:28] Lang Smith, okay, Lang Fuse.
[01:56:29] Okay, lang fuels. Langfuels, it is little.
[01:56:33] Difficult compared to length, length, uh, Lang Smith. I will show you length use tomorrow, uh.
[01:56:38] So Lang Smith difficult as in Lang fuse was not created by LangChain.
[01:56:43] Okay.
[01:56:44] So, little extra lines of code, just two, three lines extra.
[01:56:47] Okay.
[01:56:48] Like langs with your sonar, it's so easy, you just
[01:56:51] Importer 3 environment variables, that's all its done.
[01:56:54] Integration is easy.
[01:56:55] Very easy with Langfuse and see, I have never worked with this thing on crew AI. Crew UI is not for us, I told you, it is for business users or not.
[01:57:00] Welcome.
[01:57:01] So I don't know how with QI, the same thing.
[01:57:04] Will occur because these things I have used always with Land graph.
[01:57:08] Okay.
[01:57:10] Okay, got it.
[01:57:11] Here, Shivans, I will take one last question, guys, then I have to wrap up for today. Tomorrow I'll take more questions. Yes, Shivansh, tell me.
[01:57:17] I think one of the questions you have answered is the pictorial representation, which you will discuss more in the next class also.
[01:57:25] Another check is, other than the token information
[01:57:30] Can we also measure as how many hits are coming on to this particular
[01:57:36] Agent
[01:57:37] How many?
[01:57:39] Hits are coming for this particular agent in the production system.
[01:57:43] how many hits are…
[01:57:46] Ah, coming to this particular agent. Yeah, yeah, yeah, you see the moment you hit multiple times now if your Langsmith tracing is on.
[01:57:54] In that project, you will have that many runs that trace count, you will have trace count.
[01:57:59] So from that…
[01:58:00] So, trace count is per user, or per request.
[01:58:01] Ha, yes, yes, it's per per agentic flow.
[01:58:07] For Agentic floor
[01:58:08] But paragentic law.
[01:58:09] So you will have runs, see this is the runs that I have done multiple times.
[01:58:15] Okay, so from there you will get an idea that how many.
[01:58:18] Times evaluate was hit, and all those things. So this is
[01:58:21] This is my all my 8pm Garan.
[01:58:24] Okay, because I am keeping a note of my
[01:58:26] Uh, last 1 day, so if you keep it last 30 days, you will be able to see how many times.
[01:58:32] You have hit particular and
[01:58:34] All these things you will be able to see, able to evaluate.
[01:58:37] Like, they see on fifth also I have it.
[01:58:40] Got it.
[01:58:42] Yeah.
[01:58:43] I think you've also answered my another question, the time slicing, so we have an option of filter of time
[01:58:50] Last 30 days, past 1 day
[01:58:51] Latency. That you have. Yes.
[01:58:53] Okay, Manishes, do people use it at the enterprise level? Very much. It is used at the enterprise level.
[01:58:59] Uh, but there are people who
[01:59:03] Don't allow it, Manish, very good question, don't allow it because it is sending your information to a third party application, which is Langspath.
[01:59:09] So in that kind of case, you create a custom…
[01:59:12] Uh, you… what you do, every LLM call, you create a database entry.
[01:59:16] Okay, of what is the token that that that thing which I was telling content dot metadata, you keep a track of that.
[01:59:17] Yeah. Yeah.
[01:59:24] Okay, and you create a database.
[01:59:25] Right, yeah.
[01:59:27] Okay, and from that database, you create a custom dashboard of your own.
[01:59:31] Let's say Sachin has done that using Grafana, you can create that using a plain.
[01:59:36] matplot only.
[01:59:37] Yeah, I want to add here, so, like, you have this open telemetry collector, through which you can
[01:59:43] Yes, Opal Telemetry connector, yes, that is also.
[01:59:44] Yeah, so that is what I used, actually, in building this complete because there are two way of instrumenting the logs, right? And logs and traces. So one is auto instrumentation and another one is manual one. So we used a manual one because we wanted a trace of
[02:00:03] start of the request to till complete of that request. So it was basically a vision-based application tracing
[02:00:10] So yeah, OpenTelemetry, I mean, is the best one
[02:00:11] Hmm. Yes, OpenTelemetry is there. Then Jaeger is there.
[02:00:17] Uh, yeah, Yagar is there.
[02:00:19] Many, many are there. Loki is there, Loki.
[02:00:23] Loki is also there.
[02:00:27] Yeah, I love logs.
[02:00:28] Yeah, Loki is specifically for the logs. The open telemetry is, like, completely for metrics, logs, and traces, completely for all three.
[02:00:32] Yeah, there are multiple, like, all these things, but the most popular one now, it is becoming length width, but yeah, length width is again not.
[02:00:41] Uh, many people will not allow it, so that time you create custom, completely custom.
[02:00:46] Everything that is allowed by your company is used. That time it becomes like that.
[02:00:53] Anyways, guys, uh, I will wind it up here, uh, you know, more questions, I'll take it tomorrow, guys. You all can today sit down, Blank Smith, you can explore a few things.
[02:01:01] I would urge this, I would urge everybody to go through Lang Smith.
[02:01:03] Play with the same agent, see what you all you can do. Like, evaluators, all those things I will show tomorrow. So, anyways, we'll do more things on Lang Smith.
[02:01:10] Let's build some custom evaluators and we'll see that.
[02:01:12] But within this tracing only in that same dashboard, if you want to click certain option, if you want to select and see.
[02:01:19] What you have done in the past, you can go and check it yourself, and you can discuss tomorrow as well.
[02:01:25] Okay. And in Langfuse, you can literally see the visualization of the flow also with the workflow where it went.
[02:01:32] Okay, in that that particular visual is also there.
[02:01:35] Okay, in a graph format.
[02:01:37] That also we will see.
[02:01:40] Okay, okay guys. So with that guys, I will wind it up here.
[02:01:45] Thank you, everyone. See you tomorrow.
[02:01:48] And we'll continue on top of this. I'll take up more questions tomorrow, guys. Uh, I'm a little late for the…
[02:01:53] Other called, thank you, thank you, thank you everyone.
[02:01:56] Thank you.
[02:01:57] Thank you.