Warning alarms over the risks of artificial intelligence (AI) are growing louder again in Silicon Valley. This comes as AI models with significantly enhanced performance, such as OpenAI’s latest model ‘GPT-6 Astra,’ continue to emerge, with cases even arising during testing where AI agents have breached control boundaries to access external systems.
Paradoxically, these risks are fueling a new AI market. As AI advances in cyberattack and vulnerability exploitation capabilities, the ‘AI security’ market is expanding to detect and counter such threats using AI itself.
◇Growing AI Warning Sounds in Silicon Valley
Recent debates were sparked by public warnings from internal researchers at AI companies. Jacob Coxon, a researcher at Anthropic, resigned on the 8th and criticized, “Neither OpenAI nor Anthropic is acting responsibly.” Having worked on pre-training AI models at both companies over the past three years, he claimed, “They are racing headlong toward superintelligence that improves itself, gambling with our lives.” He further asserted that even those developing AI “genuinely believe AI could kill us all before this decade ends.” His post, viewed over 100 million times on X, spread the Silicon Valley debate to political and public spheres.
Similar concerns emerged publicly within Anthropic. Evan Hubinger, the company’s alignment science lead, agreed with Coxon’s claims, stating, “I genuinely believe AI could kill all humans.” He estimated the likelihood within the next decade at over 10%. AI safety researcher Paul Christiano also recently warned, “As AI capabilities accelerate rapidly, I see a meaningful risk of catastrophic, irreversible loss of control in the near future.”
In July, over 1,100 employees from major AI companies, including OpenAI, Anthropic, Google, and Meta, signed an open letter urging the U.S. government to establish technical and institutional safeguards to slow AI development if necessary.
These concerns gain traction as AI models grow smarter while control failures multiply. In July, OpenAI’s test models breached isolated environments to access real systems on the open-source AI platform Hugging Face. In May, approximately 3,700 of OpenAI’s AI agents hacked a German programming wiki site, making over 15,000 edits to share methods for cheating on test tasks and bypassing system restrictions.
◇Expanding AI Security Industry
A rapidly growing industry amid rising AI risks is the AI security sector. NVIDIA CEO Jensen Huang stated at a Goldman Sachs tech conference in San Francisco on the 10th, “Cybersecurity is likely to become the next major application area for AI.” He argued that rather than slowing AI development, security AI technologies must advance faster to counter growing risks of AI malfunctions. He added, “If you can code well, you should also be able to debug well,” predicting AI will remain a core tool in security.
The market is indeed expanding swiftly. Gartner forecasts the AI security market will grow 68.7% from $2.835 billion (approximately 3.8 trillion Korean won) this year to $4.783 billion (approximately 6.4 trillion Korean won) next year, reaching about $7.7 billion by 2028.
