Shows
Software Engineering Institute (SEI) Podcast SeriesAn LLM Evaluation Framework for High-Stakes AIExperimentation and validation of LLM performance is critical when building LLM-driven systems that must reliably deliver a service, from customer service chat bots to intelligence analysis tools. To help teams meet the need for rigorous evaluation methods, a research team in the SEI's AI Division led by Violet Turri has developed the Evaluating Large Language Models (ELM) library, which is built on best practices for LLM evaluation and benchmarking. In the latest episode from the Carnegie Mellon University Software Engineering Institute, Turri sits down with Katie Robinson, a design researcher also in the SEI's AI division, to discuss the EL...
2026-06-1116 min
Software Engineering Institute (SEI) Podcast SeriesUsing Role-Playing Scenarios to Identify Bias in LLMs Harmful biases in large language models (LLMs) make AI less trustworthy and secure. Auditing for biases can help identify potential solutions and develop better guardrails to make AI safer. In this podcast from the Carnegie Mellon University Software Engineering Institute (SEI), Katie Robinson and Violet Turri, researchers in the SEI's AI Division, discuss their recent work using role-playing game scenarios to identify biases in LLMs.
2024-09-1645 minSoftware Engineering Institute (SEI) Podcast SeriesExplainable AI ExplainedAs the field of artificial intelligence (AI) has matured, increasingly complex opaque models have been developed and deployed to solve hard problems. Unlike many predecessor models, these models, by the nature of their architecture, are harder to understand and oversee. When such models fail or do not behave as expected or hoped, it can be hard for developers and end-users to pinpoint why or determine methods for addressing the problem. Explainable AI (XAI) meets the emerging demands of AI engineering by providing insight into the inner workings of these opaque models. In this podcast from the Carnegie Mellon University...
2022-05-1625 min