Shows
Daily Paper CastCodeMonkeys: Scaling Test-Time Compute for Software Engineering
🤗 Upvotes: 5 | cs.LG
Authors:
Ryan Ehrlich, Bradley Brown, Jordan Juravsky, Ronald Clark, Christopher RĂ©, Azalia Mirhoseini
Title:
CodeMonkeys: Scaling Test-Time Compute for Software Engineering
Arxiv:
http://arxiv.org/abs/2501.14723v1
Abstract:
Scaling test-time compute is a promising axis for improving LLM capabilities. However, test-time compute can be scaled in a variety of ways, and effectively combining different approaches remains an active area of research. Here, we explore this problem in the context of solving real-world GitHub issues from the SWE-bench dataset. Our system, named CodeMonkeys, allows models to i...2025-01-2923 min
AI Safety FundamentalsMeasuring Progress on Scalable Oversight for Large Language ModelsAbstract:Â Developing safe and useful general-purpose AI systems will require us to make progress on scalable oversight: the problem of supervising systems that potentially outperform us on most skills relevant to the task at hand. Empirical work on this problem is not straightforward, since we do not yet have systems that broadly exceed our abilities. This paper discusses one of the major ways we think about this problem, with a focus on ways it can be studied empirically. We first present an experimental design centered on tasks for which human specialists succeed but unaided humans a...2025-01-0409 min
ArgmaxMixture of ExpertsIn this episode we talk about the paper "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" by Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean.2024-10-0854 min
Papers Read on AIOutrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerThe capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting...2023-12-1337 min
Jay Shah PodcastHow did I get into Machine Learning research? | Sara Hooker, Azalia Mirhoseini & Natasha Jacques - GoogleThree research scientists from Google share their journey about interest in Machine Learning research and how they got started with it.Watch full podcasts with each of these speakers:Azalia Mirhoseini: https://youtu.be/5LCfH8YiOv4Sara Hooker: https://youtu.be/MHtbZls2utsNatasha Jacques: https://youtu.be/8XpCnmvq49sAbout the Host:Jay is a Ph.D. student at Arizona State University, doing research on building Interpretable AI models for Medical Diagnosis.Jay Shah: https://www.linkedin.com/in/shahjay22/You can reach out to https://www...2021-06-2611 minJay Shah PodcastUsing Deep Reinforcement Learning for System Optimization & more | Dr. Azalia Mirhoseini, Google ​Azalia is a Research scientist at the Google Brain team, where she leads machine learning for systems moonshot projects. Her research interests include and not limited to exploring deep reinforcement learning for optimizing computer systems. She has a Ph.D. in Electrical and Computer Engineering from Rice University and has received many awards for her contributions including the  MIT Technology Review 35 under 35.About the Host:Jay is a Ph.D. student at Arizona State University, doing research on building Interpretable AI models for Medical Diagnosis.Jay Shah: https://www.linkedin.com/in/shahjay22/Y...2021-02-0146 min
Practical AIReinforcement learning for chip designDaniel and Chris have a fascinating discussion with Anna Goldie and Azalia Mirhoseini from Google Brain about the use of reinforcement learning for chip floor planning - or placement - in which many new designs are generated, and then evaluated, to find an optimal component layout. Anna and Azalia also describe the use of graph convolutional neural networks in their approach.Join the discussionChangelog++ members support our work, get closer to the metal, and make the ads disappear. Join today!Sponsors:Linode – Our cloud of choice and the home of Changelog.co...2020-04-2744 min
Changelog Master FeedReinforcement learning for chip design (Practical AI #87)Daniel and Chris have a fascinating discussion with Anna Goldie and Azalia Mirhoseini from Google Brain about the use of reinforcement learning for chip floor planning - or placement - in which many new designs are generated, and then evaluated, to find an optimal component layout. Anna and Azalia also describe the use of graph convolutional neural networks in their approach.2020-04-2700 min
The Data Exchange with Ben LoricaHyperscaling natural language processingIn this episode of the Data Exchange I speak with Edmon Begoli, Chief Data Architect at Oak Ridge National Laboratory (ORNL). Edmon has developed and implemented large-scale data applications on systems like Open MPI, Hadoop/MapReduce, Apache Calcite, Apache Spark, and Akka. Most recently he has been building large-scale machine learning and natural language applications with Ray, a distributed execution framework that makes it easy to scale machine learning and Python applications.Our conversation included a range of topics, including:Edmon’s role at the ORNL and his experience building applications with Hadoop and Spark.What is...2020-03-0535 minThe Data Exchange with Ben LoricaWhat businesses need to know about model explainabilityIn this episode of the Data Exchange I speak with Krishna Gade, founder and CEO at Fiddler Labs, a startup focused on helping companies build trustworthy and understandable AI solutions. Prior to founding Fiddler, Krishna led engineering teams at Pinterest and Facebook.Our conversation included a range of topics, including:Krishna’s background as an engineering manager at Facebook and Pinterest.Why Krishna decided to start a company focused on explainability.Guidelines for companies who want to begin working on incorporating model explainability into their data products.The relationship between model explainability (transparency) and security (ML th...2020-02-2736 min
Banana Data PodcastThe Future (and the now) of AI with Azalia Mirhoseini, Senior Researcher at Google BrainAI constantly promises the cutting edge. So, what’s behind the newest, hottest AI trends out there? This episode, Triveni & Will sit down with Azalia Mirhoseini, Senior Researcher at Google Brain, named on Technology Review's 35 Innovators under 35 to explore what’s really going on behind the scenes, and what’s actually overrated, underrated, and just right in the field. Azalia Mirhoseini, 35 Innovators Under 35: Visionaries (MIT Technology Review)2020-02-0125 min