Superintelligence, Ethics, and Human Agency: How We Shape the Future
Superintelligence is often discussed through dramatic predictions: machines that outthink people, technology that changes everything, or futures we may not be able to control. Yet, there’s another way to approach the subject. What if superintelligence is a technological development, and also a mirror reflecting human intelligence, values, and choices?
Watch on YouTube, or listen on Spotify or Apple Podcasts.
In this episode of the Studio Shangri-La Multimedia Podcast, Sarah Ikerd considers what superintelligence means beyond faster computing. She proposes four defining qualities—speed, access, understanding, and judgment—and asks how people can take a more active role in shaping the story around emerging technologies. Her central point is that the future is not just something technology delivers. It is by nature something humanity helps steer.
What does superintelligence mean beyond technology?
Superintelligence is usually discussed as a technological or computational phenomenon. In Sarah’s framing, it is associated with four qualities, for example: speed, access, understanding, and judgment. Computing systems can process information rapidly, connect to broad stores of information, identify patterns, and support decisions.
Yet Sarah argues that the idea reaches beyond minerals and machines. Technology is created by people, built from generations of human creativity, research, and effort. That means superintelligence can also prompt questions about human intelligence: What are we capable of? How might we develop? And what responsibilities come with creating tools that can influence decisions at great speed?
This perspective changes the conversation. Instead of treating superintelligence as something wholly separate from humanity, we can see it as part of a relationship between people and the systems we build. Its development may be computational, but the questions around its purpose and use remain human directions.
Four qualities that help shape the discussion
The four qualities Sarah identifies in this episode offer a practical starting point:
- Speed: How quickly can a system process information or produce an answer?
- Access: What information or tools can it draw on?
- Understanding: How well can it make sense of relationships, context, or patterns?
- Judgment: How does it help evaluate choices or possible outcomes?
These are useful dimensions to consider, though they do not answer every important question. Faster decisions are not automatically better decisions. Access to information does not, by itself, tell us which goals are worthwhile. And a system’s ability to support judgment does not remove the need for ethics and people to decide what matters.
That distinction is central to the overall ethical conversation. The abilities of a technology and the values guiding its use are related, but they are not the same. A tool may expand what is possible; people and societies still have to consider what they want to do with that possibility and how to steer it.
Superintelligence can broaden how we think about intelligence
Sarah’s view of superintelligence also connects technology to the wider natural world. Human beings are far from the only forms of life with abilities worth noticing. Other organisms perceive and interact with the world in ways people may not immediately recognize. A tree that can live for thousands of years, for instance, has a relationship with time that differs dramatically from ours. There’s much for us yet to learn.
The point is not to rank every form of life on one scale, and to recognize everything has intrinsic value. It is to remember that intelligence and capability can take different forms. Human perception is only one layer of perspective, and what is easy for us to recognize is not necessarily the full range of what exists.
Sarah describes evolution as a branching process. Different forms of life continue along different paths, developing capacities that may not be directly comparable at first glance. That image offers a helpful counterweight to the assumption that intelligence must always look human, or that every kind of capability can be understood through a single measure.
From natural intelligence to technological intelligence
Seeing intelligence as varied and interconnected makes it easier to ask more open questions about technology. Rather than asking only whether a system is “smarter” than a person, we might ask what kinds of tasks it can perform, what it helps people perceive, and where human understanding remains essential.
In Sarah’s interpretation, superintelligence could help people explore possibilities beyond their usual point of view, and within themselves. It may offer new ways to examine complex situations, consider alternatives, or think about intelligence itself. This is an invitation to curiosity, not a claim that technology can answer every question.
The same curiosity applies to human biology. Sarah describes people as information-processing beings, pointing to DNA and biological systems as examples of ways information is encoded. Her broader reflection is that human beings are already part of a world of communication, adaptation, and ongoing development. Technology, in this sense, is not separate from human evolution; it is one of the ways people extend their ability to investigate and shape their environment.
The practical takeaway is to hold two ideas together: technology can expand human capabilities, and human beings are already complex, evolving organisms. Neither idea requires us to assume that machines will simply replace people or that every new capability is automatically beneficial. Both encourage a more careful exploration of what different forms of intelligence can do. It’s also important to acknowledge the potential developments of mineral or mechanical intelligence.
Ethics and human agency belong in the AI conversation
For Sarah, ethics is not an afterthought to be considered once technology has already reshaped society. It is part of deciding what direction to take. She argues that people need some sense of how they want the future to look—even if that vision is provisional and open to revision.
That does not mean ignoring risks. Sarah explicitly acknowledges that concerns deserve attention. Her argument is against letting fear or sensational stories become the only way to discuss emerging technology. A steadier approach can make room for real questions about risk while also asking what people want to build, protect, and improve.
This is where agency matters. Human agency is the capacity to make choices and influence what happens next. In the context of AI and superintelligence, it means recognizing that technology’s place in society is shaped by decisions: what systems are designed to do, how they are used, and which values guide those choices. It’s also about respecting intelligence in general, and what further evolutionary forms it may take.
Risk awareness without surrendering the future
It is easy for conversations about powerful technology to swing between two extremes. One is uncritical optimism, where every new development is treated as progress simply because it is new. The other is fatalism, where a frightening possibility is treated as inevitable.
Sarah suggests a different posture: acknowledge uncertainty, consider risks, and still take responsibility for the choices available now. That approach does not promise control over every outcome. It does insist that people should not confuse imagining a possible future with proving that it must happen.
Her image of stepping back for a bird’s-eye view is useful here. When you are caught up in a heated debate, it can be difficult to separate evidence, assumptions, emotion, and desired outcomes. Taking a wider view helps you ask: What do we know? What are we worried about? What values are driving that concern? And what decisions can people make in response?
This kind of reflection is relevant well beyond AI. It is a way to approach difficult social questions without being pulled entirely into reaction or polarization. In Sarah’s view, philosophical traditions such as Stoicism can help create that pause: examine your own thinking, look at the larger situation, and avoid rushing to conclusions.
A more deliberate vision for technological progress
Sarah’s preferred future is hopeful. She points to science-fictional possibilities such as Star Trek-style replicators, more peaceful co-existence and non-invasive medicine as examples of the kinds of outcomes that can inspire people. These examples are visions, not guarantees. Their value is in making the conversation concrete: What kinds of improvements would people actually welcome, and what shared goals might guide progress?
Her answer begins with quality of life and familiar human aspirations, including life, liberty, and the pursuit and experience of happiness. People will disagree about exactly what a better future looks like. Still, reflecting on shared values can help societies discuss technology in terms of more than capability alone.
A technology’s speed or power cannot decide what a good life means. That takes deliberation. It requires people to ask which benefits matter, who should have a voice in decisions, and how new systems can cooperate with human aims rather than displace the conversation about those aims.
Turn optimism into questions and choices
A positive vision is most useful when it leads to practical questions. When you hear a claim about superintelligence, consider asking:
- What capability is being discussed? Is the claim about speed, access, understanding, judgment, or something else?
- What outcome is being proposed? Does it address a real need or a clearly stated goal?
- What values are involved? Who benefits, what matters to them, and what trade-offs should be considered?
- Where does human responsibility remain? Who is making decisions, and how can those decisions be examined?
These questions help move the discussion away from broad spectacle and toward choices and feeling involved. They do not settle every debate, but they make it easier to identify what people are actually disagreeing about.
They also keep the role of human beings visible. Superintelligence may alter how quickly information can be processed or how decisions are supported. But the direction of society is also shaped by the priorities people choose and the responsibilities they accept. For Sarah, humanity is not finished evolving, and machine sentience may be a factor — technological development is one part of a grand ongoing story.
How can philosophy help with technology debates?
Philosophy can help people examine self, situations, assumptions, clarify values, and consider the consequences of different choices. It encourages a wider more deliberate view than immediate reactions, when debates become polarized or emotionally charged.
Superintelligence raises questions about the future and evolution of intelligence in general, beyond what computers can do. It also asks how people understand intelligence, what values we want technology and ourselves to serve, and how actively we will shape the future. Sarah’s central invitation is to combine curiosity with responsibility and respect: Acknowledge uncertainty, make room for ethical reflection, and steer our own agency. To explore the ideas in her own words, listen to the full Studio Shangri-La podcast or visit the Science & Philosophy page of the site, studio-shangri-la.com.
This is an epic and wonderful topic, that will have probably have a part 2.
Interested in being on the show? Contact: sarah.ikerd@studio-shangri-la.com.