China’s AI surveillance network, DHS expansion cited as warning signs
Schneier and Penney described the emerging systems as “automated speed cameras, but on steroids” that would enforce not just speed limits but “any other rule you can imagine,” issuing fines immediately rather than weeks later by mail. They wrote that these systems combine powerful AI, public and private surveillance via real-time facial recognition and digital tracking, mass databases, and highly personalized enforcement.
The authors pointed to China as the most advanced example, writing that the country has over 600 million surveillance cameras “increasingly powered by AI and facial recognition to enforce legal and social rules.” They detailed the case of Lao Duan, a Chinese citizen who lost his job and was unable to repay loans. According to the authors, when Duan visited Beijing, the city’s AI surveillance system identified him at a major intersection and displayed his face, name, and citizen ID number on a large electronic billboard with a message that he was an “untrustworthy person.” They wrote that similar systems are being deployed across China and integrated with its online monitoring, censorship, and social credit systems.
In the United States, Schneier and Penney wrote that the Department of Homeland Security is “rapidly increasing its use of AI-based surveillance, including facial recognition and the monitoring of social media accounts” to track immigrants, dissidents, journalists, legal observers, and protesters, according to a new report they cited. They argued that while such systems are ostensibly used for security, “the real aim is often social control.”
The authors quoted Oracle CEO Larry Ellison, whose company works closely with the Trump administration, as saying: “Citizens will be on their best behavior because we’re constantly recording and reporting.” They wrote: “The chilling effects are the point.”
Penney, author of a new book titled “Chilling Effects: Repression, Conformity, and Power in the Digital Age,” outlined mechanisms that increase the scale of chilling effects: surveillance, personalization, uncertainty, and authority. These cause people to self-censor, become more conformist and compliant, the authors wrote. “The result will be a kind of supercharged societal level of chilling effects where fear, self-censorship and groupthink reign, and dissent, creativity and innovation become increasingly rare,” they wrote.
Schneier and Penney argued that AI surveillance’s long-term impact would be to halt social progress. They cited the decades-long normalization of same-sex relationships and marijuana legalization, writing that in order for those changes to happen “there had to be a counterculture that was able to experiment and eventually demonstrate to the world that morality could change over time.” To the extent AI surveillance chills such experimentation, they wrote, “social progress becomes impossible.”
The authors called for policy choices to reject AI-enhanced mass surveillance: bans on facial recognition and other identification technologies, new privacy and data protections, AI regulations curtailing invasive uses, and structural reforms to break up powerful state-technology cartels. “The chill of AI-powered mass surveillance will suffocate the very foundations of healthy democratic societies,” they wrote. “But we can still choose a different path.”