Shows
Joe Carlsmith Audio(Part 1, Otherness) Extended audio from my conversation with Dwarkesh PatelExtended audio from my conversation with Dwarkesh Patel. This part focuses on my series "Otherness and control in the age of AGI." Transcript available on my website here: https://joecarlsmith.com/2024/09/30/part-1-otherness-extended-audio-transcript-from-my-conversation-with-dwarkesh-patel/
2024-09-303h 58
Dwarkesh PodcastJoe Carlsmith — Preventing an AI takeoverChatted with Joe Carlsmith about whether we can trust power/techno-capital, how to not end up like Stalin in our urge to control the future, gentleness towards the artificial Other, and much more.Check out Joe's sequence on Otherness and Control in the Age of AGI here.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Sponsors:- Bland.ai is an AI agent that automates phone calls in any language, 24/7. Their technology uses...
2024-08-222h 30
LessWrong (Curated & Popular)“Loving a world you don’t trust” by Joe Carlsmith(Cross-posted from my website. Audio version here, or search for "Joe Carlsmith Audio" on your podcast app.)This is the final essay in a series that I'm calling "Otherness andcontrol in the age of AGI." I'm hoping that the individual essays can beread fairly well on their own, butsee here fora brief summary of the series as a whole. There's also a PDF of the whole series here.Warning: spoilers for Angels in America; and moderate spoilers forHarry Potter and the Methods of Rationality.)"I come into the presence of still water..."~...
2024-07-011h 03
Joe Carlsmith AudioIntroduction and summary for "Otherness and control in the age of AGI"This is the introduction and summary for my series "Otherness and control in the age of AGI." Text version here: https://joecarlsmith.com/2024/01/02/otherness-and-control-in-the-age-of-agi
2024-06-2112 min
Joe Carlsmith AudioSecond half of full audio for "Otherness and control in the age of AGI"Second half of the full audio for my series on how agents with different values should relate to one another, and on the ethics of seeking and sharing power. First half here: https://joecarlsmithaudio.buzzsprout.com/2034731/15266490-first-half-of-full-audio-for-otherness-and-control-in-the-age-of-agiPDF of the full series here: https://jc.gatspress.com/pdf/otherness_full.pdfSummary of the series here: https://joecarlsmith.com/2024/01/02/otherness-and-control-in-the-age-of-agi
2024-06-184h 11
Joe Carlsmith AudioFirst half of full audio for "Otherness and control in the age of AGI"First half of the full audio for my series on how agents with different values should relate to one another, and on the ethics of seeking and sharing power. Second half here: https://joecarlsmithaudio.buzzsprout.com/2034731/15272132-second-half-of-full-audio-for-otherness-and-control-in-the-age-of-agiPDF of the full series here: https://jc.gatspress.com/pdf/otherness_full.pdfSummary of the series here: https://joecarlsmith.com/2024/01/02/otherness-and-control-in-the-age-of-agi
2024-06-183h 07
Joe Carlsmith AudioLoving a world you don't trustGarden, campfire, healing water. Text version here: https://joecarlsmith.com/2024/06/18/loving-a-world-you-dont-trust This essay is part of a series I'm calling "Otherness and control in the age of AGI." I'm hoping that individual essays can be read fairly well on their own, but see here for brief text summaries of the essays that have been released thus far: https://joecarlsmith.com/2024/01/02/otherness-and-control-in-the-age-of-agi
2024-06-171h 03LessWrong (Curated & Popular)[HUMAN VOICE] "On green" by Joe CarlsmithCross-posted from my website. Podcast version here, or search for "Joe Carlsmith Audio" on your podcast app.This essay is part of a series that I'm calling "Otherness and control in the age of AGI." I'm hoping that the individual essays can be read fairly well on their own, but see here for brief summaries of the essays that have been released thus far.Warning: spoilers for Yudkowsky's "The Sword of the Good.")Examining a philosophical vibe that I think contrasts in interesting ways with "deep atheism."Text version here: https...
2024-04-121h 15
Joe Carlsmith AudioOn attunementExamining a certain kind of meaning-laden receptivity to the world.Text version here: https://joecarlsmith.com/2024/03/25/on-attunement This essay is part of a series I'm calling "Otherness and control in the age of AGI." I'm hoping that individual essays can be read fairly well on their own, but see here for brief text summaries of the essays that have been released thus far: https://joecarlsmith.com/2024/01/02/otherness-and-control-in-the-age-of-agi(Though: note that I haven't put the summary post on the podcast yet.)
2024-03-2544 min
Joe Carlsmith AudioOn greenExamining a philosophical vibe that I think contrasts in interesting ways with "deep atheism."Text version here: https://joecarlsmith.com/2024/03/21/on-greenThis essay is part of a series I'm calling "Otherness and control in the age of AGI." I'm hoping that individual essays can be read fairly well on their own, but see here for brief text summaries of the essays that have been released thus far: https://joecarlsmith.com/2024/01/02/otherness-and-control-in-the-age-of-agi(Though: note that I haven't put the summary post on the podcast yet.)
2024-03-211h 15
Hear This IdeaJoe Carlsmith on Scheming AIJoe Carlsmith is a writer, researcher, and philosopher. He works as a senior research analyst at Open Philanthropy, where he focuses on existential risk from advanced artificial intelligence. He also writes independently about various topics in philosophy and futurism, and holds a doctorate in philosophy from the University of Oxford.
You can find links and a transcript at www.hearthisidea.com/episodes/carlsmith
In this episode we talked about a report Joe recently authored, titled ‘Scheming AIs: Will AIs fake alignment during training in order to get power?’. The report “examines whether advanced AIs that perfor...
2024-03-161h 51
EAG TalksScheming AIs | Joe Carlsmith | EA Global Bay Area 2024This talk examines whether advanced AIs that perform well in training will be doing so in order to gain power later — a behavior Joe Carlsmith calls "scheming" (also often called "deceptive alignment"). This talk gives an overview of his recent report on the topic, available on arXiv here: https://arxiv.org/abs/2311.08379.
Joe Carlsmith is a senior research analyst at Open Philanthropy, where he focuses on existential risk from advanced artificial intelligence. He also writes independently about various topics in philosophy and futurism, and he has a doctorate in philosophy from the University of Oxford.
...
2024-03-0651 min
Joe Carlsmith AudioOn the abolition of manWhat does it take to avoid tyranny towards to the future?Text version here: https://joecarlsmith.com/2024/01/18/on-the-abolition-of-man This essay is part of a series I'm calling "Otherness and control in the age of AGI." I'm hoping that individual essays can be read fairly well on their own, but see here for brief text summaries of the essays that have been released thus far: https://joecarlsmith.com/2024/01/02/otherness-and-control-in-the-age-of-agi(Though: note that I haven't put the summary post on the podcast yet.)
2024-01-181h 09
Joe Carlsmith AudioBeing nicer than ClippyLet's be the sort of species that aliens wouldn't fear the way we fear paperclippers. Text version here: https://joecarlsmith.com/2024/01/16/being-nicer-than-clippy/ This essay is part of a series I'm calling "Otherness and control in the age of AGI." I'm hoping that individual essays can be read fairly well on their own, but see here for brief text summaries of the essays that have been released thus far: https://joecarlsmith.com/2024/01/02/otherness-and-control-in-the-age-of-agi(Though: note that I haven't put the summary post on the podcast yet.)
2024-01-1647 minLessWrong (Curated & Popular)[HUMAN VOICE] "Gentleness and the artificial Other" by Joe Carlsmith"(Cross-posted from my website. Audio version here, or search "Joe Carlsmith Audio" on your podcast app.)"This is the first essay in a series that I’m calling “Otherness and control in the age of AGI.” See here for more about the series as a whole.)When species meetThe most succinct argument for AI risk, in my opinion, is the “second species” argument. Basically, it goes like this.Premise 1: AGIs would be like a second advanced species on earth, more powerful than humans.Conclusion: That’s scary.To be clear: this is very...
2024-01-1422 min
Joe Carlsmith AudioAn even deeper atheismWho isn't a paperclipper?Text version here: https://joecarlsmith.com/2024/01/11/an-even-deeper-atheism This essay is part of a series I'm calling "Otherness and control in the age of AGI." I'm hoping that individual essays can be read fairly well on their own, but see here for brief summaries of the essays that have been released thus far: https://joecarlsmith.com/2024/01/02/otherness-and-control-in-the-age-of-agi
2024-01-1125 min
Joe Carlsmith AudioDoes AI risk "other" the AIs?Examining Robin Hanson's critique of the AI risk discourse.Text version here: https://joecarlsmith.com/2024/01/09/does-ai-risk-other-the-aisThis essay is part of a series of essays called "Otherness and control in the age of AGI." I'm hoping the individual essays can be read fairly well on their own, but see here for brief summaries of the essays that have been released thus far: https://joecarlsmith.com/2024/01/02/otherness-and-control-in-the-age-of-agi
2024-01-0913 min
Joe Carlsmith AudioWhen "yang" goes wrongOn the connection between deep atheism and seeking control. Text version here: https://joecarlsmith.com/2024/01/08/when-yang-goes-wrongThis essay is part of a series of essays called "Otherness and control in the age of AGI." I'm hoping the individual essays can be read fairly well on their own, but see here for brief summaries of the essays that have been released thus far: https://joecarlsmith.com/2024/01/02/otherness-and-control-in-the-age-of-agi
2024-01-0821 min
Joe Carlsmith AudioDeep atheism and AI riskOn a certain kind of fundamental mistrust towards Nature. Text version here: https://joecarlsmith.com/2024/01/04/deep-atheism-and-ai-riskThis is the second essay in my series “Otherness and control in the age of AGI. I’m hoping that the individual essays can be read fairly well on their own, but see here for brief summaries of the essays released thus far: https://joecarlsmith.com/2024/01/02/otherness-and-control-in-the-age-of-agi
2024-01-0446 min
Joe Carlsmith AudioGentleness and the artificial OtherAIs as fellow creatures. And on getting eaten. Link: https://joecarlsmith.com/2024/01/02/gentleness-and-the-artificial-otherThis is the first essay in a series of essays that I’m calling “Otherness and control in the age of AGI.” See here for more about the series as a whole: https://joecarlsmith.com/2024/01/02/otherness-and-control-in-the-age-of-agi.
2024-01-0222 min
Joe Carlsmith AudioIn search of benevolence (or: what should you get Clippy for Christmas?)What is altruism towards a paperclipper? Can you paint with all the colors of the wind at once? (This is a recording of an essay originally published in 2021. Text here: https://joecarlsmith.com/2021/07/19/in-search-of-benevolence-or-what-should-you-get-clippy-for-christmas)
2023-12-2752 min
Joe Carlsmith AudioThe counting argument for scheming (Sections 4.1 and 4.2 of "Scheming AIs")This is sections 4.1 and 4.2 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1610 min
Joe Carlsmith AudioIs scheming more likely if you train models to have long-term goals? (Sections 2.2.4.1-2.2.4.2 of "Scheming AIs")This is sections 2.2.4.1-2.2.4.2 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1609 min
Joe Carlsmith AudioHow useful for alignment-relevant work are AIs with short-term goals? (Section 2.2.4.3 of "Scheming AIs")This is section 2.2.4.3 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1609 min
Joe Carlsmith AudioThe goal-guarding hypothesis (Section 2.3.1.1 of "Scheming AIs")This is section 2.3.1.1 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1619 min
Joe Carlsmith AudioDoes scheming lead to adequate future empowerment? (Section 2.3.1.2 of "Scheming AIs")This is section 2.3.1.2 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1622 min
Joe Carlsmith AudioNon-classic stories about scheming (Section 2.3.2 of "Scheming AIs")This is section 2.3.2 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1624 min
Joe Carlsmith AudioArguments for/against scheming that focus on the path SGD takes (Section 3 of "Scheming AIs")This is section 3 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1629 min
Joe Carlsmith AudioSimplicity arguments for scheming (Section 4.3 of "Scheming AIs")This is section 4.3 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1619 min
Joe Carlsmith AudioSpeed arguments against scheming (Section 4.4-4.7 of "Scheming AIs")This is section 4.4 through 4.7 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1615 min
Joe Carlsmith AudioSumming up "Scheming AIs" (Section 5)This is section 5 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1615 min
Joe Carlsmith AudioEmpirical work that might shed light on scheming (Section 6 of "Scheming AIs")This is section 6 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1628 min
Joe Carlsmith Audio"Clean" vs. "messy" goal-directedness (Section 2.2.3 of "Scheming AIs")This is section 2.2.3 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1616 min
Joe Carlsmith AudioTwo sources of beyond-episode goals (Section 2.2.2 of "Scheming AIs")This is section 2.2.2 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1621 min
Joe Carlsmith AudioTwo concepts of an "episode" (Section 2.2.1 of "Scheming AIs")This is section 2.2.1 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1612 min
Joe Carlsmith AudioSituational awareness (Section 2.1 of "Scheming AIs")This is section 2.1 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1609 min
Joe Carlsmith AudioOn "slack" in training (Section 1.5 of "Scheming AIs")This is section 1.5 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1607 min
Joe Carlsmith AudioVarieties of fake alignment (Section 1.1 of "Scheming AIs")This is section 1.1 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1617 min
Joe Carlsmith AudioA taxonomy of non-schemer models (Section 1.2 of "Scheming AIs")This is section 1.2 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1611 min
Joe Carlsmith AudioWhy focus on schemers in particular? (Sections 1.3-1.4 of "Scheming AIs")This is sections 1.3-1.4 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1631 min
Joe Carlsmith AudioFull audio for "Scheming AIs: Will AIs fake alignment during training in order to get power?"This is the full audio for my report "Scheming AIs: Will AIs fake alignment during training in order to get power?"(I’m also posting audio for individual sections of the report on this podcast, but the ordering was getting messed up on various podcast apps, and I think some people might want one big audio file regardless, so here it is. I’m going to be posting the individual sections one by one, in the right order, over the coming days. )Full text of the report here: https://arxiv.org/abs/2311.08379Summary here: http...
2023-11-156h 13
Joe Carlsmith AudioEmpirical work that might shed light on scheming (Section 6 of "Scheming AIs")This is section 6 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1528 min
Joe Carlsmith AudioThe counting argument for expecting schemers (Section 4.1-4.2 of "Scheming AIs")This is sections 4.1 and 4.2 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1510 min
Joe Carlsmith AudioSimplicity arguments for expecting schemers (Section 4.3 of "Scheming AIs")This is section 4.3 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1519 min
Joe Carlsmith AudioSpeed arguments against expecting schemers (Sections 4.4-4.7 of "Scheming AIs")This is sections 4.4-4.7 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1515 min
Joe Carlsmith AudioSumming up "Scheming AIs" (Section 5)This is section 5 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1515 min
Joe Carlsmith AudioSituational awareness (Section 2.1 of "Scheming AIs")This is section 2.1 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1509 min
Joe Carlsmith AudioOn "slack" in training (Section 1.5 of "Scheming AIs")This is section 1.5 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1507 min
Joe Carlsmith AudioWhy focus on schemers in particular? (Sections 1.3-1.4 of "Scheming AIs")This is section 1.3 and 1.4 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1531 min
Joe Carlsmith AudioA taxonomy of non-schemer models (Section 1.2 of "Scheming AIs")This is section 1.2 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-powerAudio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1511 min
Joe Carlsmith AudioVarieties of fake alignment (Section 1.1 of "Scheming AIs")This is section 1.1 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of report here: https://arxiv.org/abs/2311.08379Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-powerAudio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1517 min
Joe Carlsmith AudioThe goal-guarding hypothesis (Section 2.3.1.1 of "Scheming AIs")This is section 2.3.1.1 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1519 min
Joe Carlsmith AudioTwo sources of beyond-episode goals (Section 2.2.2 of "Scheming AIs")This is section 2.2.2 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1521 min
Joe Carlsmith Audio"Clean" vs. "messy" goal-directedness (Section 2.2.3 of "Scheming AIs")This is section 2.2.3 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1516 min
Joe Carlsmith AudioIs scheming more likely if you train a model to have long-term goals? (Section 2.2.4.1-2.2.4.2 of "Scheming AIs")This is section 2.2.4.1-2.2.4.2 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1509 min
Joe Carlsmith AudioHow useful for alignment-relevant work are AIs with short-term goals? (Section 2.2.4.3 of "Scheming AIs")This is section 2.2.4.3 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1509 min
Joe Carlsmith AudioTwo concepts of an "episode" (Section 2.2.1 of "Scheming AIs")This is section 2.2.1 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1512 min
Joe Carlsmith AudioDoes scheming lead to adequate future empowerment? (Section 2.3.1.2 of "Scheming AIs")This is section 2.3.1.2 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1522 min
Joe Carlsmith AudioNon-classic stories about schemers (Section 2.3.2 of "Scheming AIs")This is section 2.3.2 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1524 min
Joe Carlsmith AudioArguments for/against scheming that focus on the path that SGD takes (Section 3 of "Scheming AIs")This is section 3 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?” Text of the report here: https://arxiv.org/abs/2311.08379 Summary of the report here: https://joecarlsmith.com/2023/11/15/new-report-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power Audio summary here: https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power
2023-11-1529 min
Joe Carlsmith AudioIntroduction and summary of "Scheming AIs: Will AIs fake alignment during training in order to get power?"This is a recording of the introductory section of my report "Scheming AIs: Will AIs fake alignment during training in order to get power?". This section includes a summary of the full report. The summary covers most of the main points and technical terminology, and I'm hoping that it will provide much of the context necessary to understand individual sections of the report on their own. (Note: the text of the report itself may not be public by the time this episode goes live.)
2023-11-1456 min
Joe Carlsmith AudioIn memory of Louise Glück"It was, she said, a great discovery, albeit my real life."
2023-10-1521 min
Foresight Institute RadioExistential Hope Podcast: Joe Carlsmith | Infite Ethics and the Sublime UtopiaJoe Carlsmith is a writer, researcher, and philosopher. He works as a senior research analyst at Open Philanthropy, focusing on existential risk from advanced artificial intelligence. He also writes independently about various topics in philosophy and futurism and has a doctorate in philosophy from the University of Oxford.Much of his work is about trying to help us orient wisely towards humanity’s long-term future. He delves into questions about meta-ethics and rationality at the foundation, feeding into questions about ethics (and especially about effective altruism), which motivate concern for the long-term future. S...
2023-09-2949 min
80k After HoursHighlights: #152 – Joe Carlsmith on navigating serious philosophical confusionThis is a selection of highlights from episode #152 of The 80,000 Hours Podcast.These aren't necessarily the most important, or even most entertaining parts of the interview — and if you enjoy this, we strongly recommend checking out the full episode:Joe Carlsmith on navigating serious philosophical confusionAnd if you're finding these highlights episodes valuable, please let us know by emailing podcast@80000hours.org.Get this episode by subscribing to our podcast on the world’s most pressing problems and how to solve them: type ‘80,000 Hours’ into your podcasting app. Or read the tran...
2023-08-0812 min
The Flares - Podcasts#46 - Les dangers de l’IA avancée : un risque existentiel ? – avec Joe Carlsmith⚠️ Découvrez du contenu EXCLUSIF (pas sur la chaîne) ⚠️ ⇒ https://the-flares.com/y/bonus/
⬇️⬇️⬇️ Infos complémentaires : sources, références, liens... ⬇️⬇️⬇️
Le contenu vous intéresse ?
Abonnez-vous et cliquez sur la 🔔
Sommaire:
0:00:00 Introduction
0:00:36 Sponsor Farnell
0:00:59 Présentation de l'invité
0:01:36 Réalisme ou anti-réalisme morale
0:06:07 Qu'est-ce qui constitue un système d'IA puissant et agentique ?
0:11:24 Existe-t-il déjà des IA agentiques ?
0:13:20 Pourquoi l'intelligence est-elle si importante pour le futur ?
0:16:48 Pourquoi l'humanité cherche-t-elle à construire des IAG ?
0:19:25 Comment définir une IA mal alignée ?
0:27:50 La recherche de pouvoir des IA est-elle inévitable ?
0:29:37 Pourquoi l'humanité risque-t-elle de perdre le contrôle sur le monde ?
0:36:13 Quelles ont été les crit...
2023-07-151h 10
Future of Life Institute PodcastJoe Carlsmith on How We Change Our Minds About AI RiskJoe Carlsmith joins the podcast to discuss how we change our minds about AI risk, gut feelings versus abstract models, and what to do if transformative AI is coming soon. You can read more about Joe's work at https://joecarlsmith.com.
Timestamps:
00:00 Predictable updating on AI risk
07:27 Abstract models versus gut feelings
22:06 How Joe began believing in AI risk
29:06 Is AI risk falsifiable?
35:39 Types of skepticisms about AI risk
44:51 Are we fundamentally confused?
53:35 Becoming alienated from ourselves?
1:00:12 What will change people's minds?
1:12:34 Outline of different futures
1:20:43 Humanity losing touch with reality
1:27:14 Can we understand AI sentience?
1:36:31 Distinguishing real from fake...
2023-06-222h 24
The Valmy#152 – Joe Carlsmith on navigating serious philosophical confusion Podcast: 80,000 Hours Podcast Episode: #152 – Joe Carlsmith on navigating serious philosophical confusionRelease date: 2023-05-19Get Podcast Transcript →powered by Listen411 - fast audio-to-text and summarizationWhat is the nature of the universe? How do we make decisions correctly? What differentiates right actions from wrong ones?Such fundamental questions have been the subject of philosophical and theological debates for millennia. But, as we all know, and surveys of expert opinion make clear, we are very far from agreement. So... with these most basic questions unresolved, what’s a species to do...
2023-06-083h 26
80,000 Hours Podcast#152 – Joe Carlsmith on navigating serious philosophical confusionWhat is the nature of the universe? How do we make decisions correctly? What differentiates right actions from wrong ones?Such fundamental questions have been the subject of philosophical and theological debates for millennia. But, as we all know, and surveys of expert opinion make clear, we are very far from agreement. So... with these most basic questions unresolved, what’s a species to do?In today's episode, philosopher Joe Carlsmith — Senior Research Analyst at Open Philanthropy — makes the case that many current debates in philosophy ought to leave us confused and humbled. These are themes...
2023-05-203h 26
Joe Carlsmith AudioOn the limits of idealized valuesContra some meta-ethical views, you can't forever aim to approximate the self you would become in idealized conditions. You have to actively create yourself, often in the here and now. Originally published in 2021. Text version here: https://joecarlsmith.com/2021/06/21/on-the-limits-of-idealized-values
2023-05-121h 00
TYPE III AUDIO (All episodes)"Predictable updating about AI risk" by Joe Carlsmith---client: ea_forumproject_id: curatedfeed_id: ai_safetynarrator: jc---How worried about AI risk will we feel in the future, when we can see advanced machine intelligence up close? We should worry accordingly now. Original article:https://joecarlsmith.com/2023/05/08/predictable-updating-about-ai-riskNarrated by Joe Carlsmith and included on the Effective Altruism Forum by TYPE III AUDIO.Share feedback on this narration.
2023-05-091h 03
Joe Carlsmith AudioPredictable updating about AI riskHow worried about AI risk will we feel in the future, when we can see advanced machine intelligence up close? We should worry accordingly now. Text version here: https://joecarlsmith.com/2023/05/08/predictable-updating-about-ai-risk
2023-05-081h 03
Die Corona-Lüge1954 Festinger & Carlsmith's Cognitive Dissonance StudyFestinger & Carlsmith's Study
Every individual has his or her own way of evaluating their own selves and usually this is done by comparing themselves to others. This is manifested in the phenomenon called cognitive dissonance. This is further exp...
2023-04-2400 min
Die Corona-Lüge1954 Festinger & Carlsmith's Cognitive Dissonance StudyFestinger & Carlsmith's Study
Every individual has his or her own way of evaluating their own selves and usually this is done by comparing themselves to others. This is manifested in the phenomenon called cognitive dissonance. This is further exp...
2023-04-2400 minTYPE III AUDIO (All episodes)"Seeing more whole" by Joe Carlsmith---client: ea_forumproject_id: curatednarrator: jc---In my last essay, I looked at two stories (brute preference for systematic-ness, and money-pumps) about why ethical anti-realists should still be interested in ethics – two stories about why the “philosophy game” is worth playing, even if there are no objective normative truths, and you’re free to do whatever you want. I think some versions of these stories might well have a role to play; but I find that on their own, they don’t fully capture what feels alive to me about ethi...
2023-03-3052 min
Joe Carlsmith AudioExistential Risk from Power-Seeking AI (shorter version)A shorter version of my report on existential risk from power-seeking AI. Forthcoming in an essay collection from Oxford University Press. Text version here: https://jc.gatspress.com/pdf/existential_risk_and_powerseeking_ai.pdf
2023-03-1955 min
Joe Carlsmith AudioProblems of evilIs everything holy? Can reality, in itself, be worthy of reverence? Text version here: https://joecarlsmith.com/2021/04/19/problems-of-evil
2023-03-0535 minTYPE III AUDIO (All episodes)"Why should ethical anti-realists do ethics?" by Joe Carlsmith---client: ea_forumproject_id: curatednarrator: not_t3a---Ethical philosophy often tries to systematize. That is, it seeks general principles that will explain, unify, and revise our more particular intuitions. And sometimes, this can lead to strange and uncomfortable places.So why do it? If you believe in an objective ethical truth, you might talk about getting closer to that truth. But suppose that you don’t. Suppose you think that you’re “free to do whatever you want.” In that case, if “systematizing” starts getting tough and un...
2023-02-2653 min
Joe Carlsmith AudioSeeing more wholeOn looking out of your own eyes. Text version at joecarlsmith.com.
2023-02-1752 min
Joe Carlsmith AudioWhy should ethical anti-realists do ethics?Who needs a system if you're free? Text version at https://joecarlsmith.com/2023/02/16/why-should-ethical-anti-realists-do-ethics
2023-02-1653 minTYPE III AUDIO (All episodes)"Is Power-Seeking AI an Existential Risk?" by Joseph Carlsmith---client: joe_carlsmithproject_id:feed_id: ai, ai_safety, ai_safety__technical, ai_safety__forecasting narrator: pwqa: kmnarrator_time: 18h00mqa_time: 5h30m---This report examines what I see as the core argument for concern about existential risk from misaligned artificial intelligence. I proceed in two stages. First, I lay out a backdrop picture that informs such concern. On this picture, intelligent agency is an extremely powerful force, and creating agents much more intelligent than us is playing with fire -- especially...
2023-02-143h 21
Joe Carlsmith AudioIs Power-Seeking AI an Existential Risk?Audio version of my report on existential risk from power-seeking AI. Text here: https://arxiv.org/pdf/2206.13353.pdf. Narration by Type III audio.
2023-01-253h 21
Joe Carlsmith AudioOn sincerityNearby is the country they call life. Text version at: https://joecarlsmith.com/2022/12/23/on-sincerity
2022-12-231h 35
Joe Carlsmith AudioAgainst meta-ethical hedonismCan the epistemology of consciousness save moral realism and redeem experience machines? No.
2022-12-011h 02
Joe Carlsmith AudioAgainst the normative realist's wagerIf your find a button that gives you a hundred dollars if a certain controversial meta-ethical view is true, but you and your family get burned alive if that view is false, should you press the button? No.Text version here. Edited for Joe Carlsmith by TYPE III AUDIO.
2022-10-0942 min
Joe Carlsmith AudioOn infinite ethicsInfinities puncture the dream of a simple, bullet-biting utilitarianism. But they're everyone's problem. Text version here.Edited for Joe Carlsmith by TYPE III AUDIO.
2022-10-051h 25
Joe Carlsmith AudioActually possible: thoughts on UtopiaLife in the future could be profoundly good. I think this is an extremely important fact, and one that often goes under-estimated.Text version here.Edited for Joe Carlsmith by TYPE III AUDIO.
2022-10-0528 min
Joe Carlsmith AudioAgainst neutrality about creating happy livesMaking happy people is good. Just ask the golden rule. Text version here.Edited for Joe Carlsmith by TYPE III AUDIO.
2022-10-0523 min
Joe Carlsmith AudioOn future people, looking back at 21st century longtermismI find imagining future people looking back on present-day longtermism (the view that positively influencing the long-term future should be a key moral priority) a helpful intuition pump, especially re: a certain kind of “holy sh**” reaction to existential risk, and to the possible size and quality of the future at stake.Text version here.Edited for Joe Carlsmith by TYPE III AUDIO.
2022-10-0525 min
Joe Carlsmith AudioCan you control the past?Sometimes, you can “control” events you have no causal interaction with (for example, if you're a deterministic software twin).Text version here.Edited for Joe Carlsmith by TYPE III AUDIO.
2022-10-051h 17
Joe Carlsmith AudioKilling the antsIf you kill something, look it in the eyes as you do.Text version here.Edited for Joe Carlsmith by TYPE III AUDIO.
2022-10-0515 min
Joe Carlsmith AudioOn clingingHow can "non-attachment" be compatible with care? We need to distinguish between caring and clinging.Text version here.Edited for Joe Carlsmith by TYPE III AUDIO.
2022-10-0517 min
Joe Carlsmith AudioThoughts on being mortalYou can't keep any of it. The only thing to do is to give it away on purpose.Text version here.Edited for Joe Carlsmith by TYPE III AUDIO.
2022-10-0512 min
The ValmyJoseph Carlsmith - Utopia, AI, & Infinite Ethics Podcast: Dwarkesh Podcast Episode: Joseph Carlsmith - Utopia, AI, & Infinite EthicsRelease date: 2022-08-03Get Podcast Transcript →powered by Listen411 - fast audio-to-text and summarizationJoseph Carlsmith is a senior research analyst at Open Philanthropy and a doctoral student in philosophy at the University of Oxford.We discuss utopia, artificial intelligence, computational power of the brain, infinite ethics, learning from the fact that you exist, perils of futurism, and blogging.Watch on YouTube. Listen on Spotify, Apple Podcasts, etc.Episode website + Transcript here. Fo...
2022-08-091h 31
Dwarkesh PodcastJoseph Carlsmith - Utopia, AI, & Infinite EthicsJoseph Carlsmith is a senior research analyst at Open Philanthropy and a doctoral student in philosophy at the University of Oxford.We discuss utopia, artificial intelligence, computational power of the brain, infinite ethics, learning from the fact that you exist, perils of futurism, and blogging.Watch on YouTube. Listen on Spotify, Apple Podcasts, etc.Episode website + Transcript here. Follow Joseph on Twitter. Follow me on Twitter.Subscribe to find out about future episodes!Timestamps(0:00:06) - Introduction(0:02:53) - How to Define a Be...
2022-08-031h 31
The Nonlinear Library: Alignment Section(Part 2/2) Is power-seeking AI an existential risk? by Joseph CarlsmithWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.
This is part two of: Is power-seeking AI an existential risk?, published by Joseph Carlsmith.
5. Deployment
Let’s turn, now, to whether we should expect to actually see practically PS-misaligned APS systems deployed in the world.
The previous section doesn’t settle this. In particular: if a technology is difficult to make safe, this doesn’t mean that lots of people will use it in unsafe ways. Rather, they might adjust their usage to reflect the degree of saf...
2021-12-181h 10The Nonlinear Library: Alignment Section(Part 1/2) Is power-seeking AI an existential risk? by Joseph CarlsmithWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.
This is part one of: Is power-seeking AI an existential risk?, published by Joseph Carlsmith.
1. Introduction
Some worry that the development of advanced artificial intelligence will result in existential catastrophe -- that is, the destruction of humanity’s longterm potential. Here I examine the following version of this worry (it’s not the only version):
By 2070:
It will become possible and financially feasible to build AI systems with the following properties:
Advanced capability: they outperform the best huma...
2021-12-181h 29
The Nonlinear Library: LessWrong Top PostsCan you control the past? by Joe CarlsmithWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.This is: Can you control the past?, published by Joe Carlsmith on the AI Alignment Forum.Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.(Cross-posted from Hands and Cities. Lots of stuff familiar to LessWrong folks interested in decision theory.)I think that you can “control” events you have no causal interaction with, including events in the past, and that this is a wild and disorienting fact, with uncertain but possibly significant impl...
2021-12-111h 18
The Nonlinear Library: Alignment Forum Top PostsCan you control the past? by Joe CarlsmithWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.
This is: Can you control the past?, published by Joe Carlsmith on the AI Alignment Forum.
(Cross-posted from Hands and Cities. Lots of stuff familiar to LessWrong folks interested in decision theory.)
I think that you can “control” events you have no causal interaction with, including events in the past, and that this is a wild and disorienting fact, with uncertain but possibly significant implications. This post attempts to impart such disorientation.
My main...
2021-12-101h 18The Nonlinear Library: Alignment Forum Top PostsComments on Carlsmith's “Is power-seeking AI an existential risk?” by Nate SoaresWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.
This is: Comments on Carlsmith's “Is power-seeking AI an existential risk?”, published by Nate Soares on the AI Alignment Forum.
The following are some comments I gave on Open Philanthropy Senior Research Analyst Joe Carlsmith’s Apr. 2021 “Is power-seeking AI an existential risk?”, published with permission and lightly edited. Joe replied; his comments are included inline. I gave a few quick replies in response, that I didn't want to worry about cleaning up; Rob Bensinger has summar...
2021-12-051h 04
UtilitarianCreating Utopia - Joseph CarlsmithOn this episode of the Utilitarian Podcast, I talk with Joseph Carlsmith. Joseph is a research analyst at Open Philanthropy and a doctoral student in philosophy at the University of Oxford. His views and opinions in this podcast are his own, and not necessarily those of Open Philanthropy.
Our conversation has three main themes. We talk about the long-term future, including the possibility of actually creating utopia. We talk about Joseph’s work on the computational power of the brain. And we talk about meta-ethics and consciousness, including discussions of illusionism and the effects of meditation.
...
2021-07-273h 15