There are several problems here. If you address them, Iām happy to answer your question faithfully. But as it stands, I donāt think itās a fair comparison.
First and foremost, weāre specifically discussing an āarms race,ā not just a ārace.ā I fail to understand what qualifies an AI company to be an expert in national security, weapons, and foreign affairs. To me, an AI company is an expert in what makes them more profit. That just so happens to involve AI here, but not āarms races.ā
Second, doctors are probably not necessarily experts on vaccines. Theyāre likely to be experts on vaccine benefits, recommendation, procurement, administering, storage, ā¦ but thatās just administration and some operational knowledge. A vaccine expert (IMO) is a specialist for vaccines. Theyāre probably not working in a hospital.
A pharmaceutical company might be an expert on vaccines. But whatās the parallel to AI companies? Pharmaceutical companies research and develop actual vaccines. AI companies do not research and develop arms. The only argument I understand for this would be to say that an LLM is a weapon.
I donāt buy that an LLM is a weapon. Real competition exists, and certain military applications could become arms races, but AI as a whole is not one. Building out swaths of data centers to power more AI without a concrete public benefit isnāt something we should put up with.
You donāt benefit from the narrative that this is an arms race, not as an individual or a citizen. The benefactors here are the AI companies.
The technology can certainly be used as a weapon; it can hack systems already, and defend against hacks. This capability will only increase, and logically the better AI affords a strategic benefit. Hence, an arms race.
The rest of your questions arenāt questions or presuppose that AI canāt be used as a weapon.
In case it isnāt obvious, there are experts in this field that arenāt employed by AI.
The term āarms raceā has a definition and Iām pretty sure this still doesnāt apply. So in my opinion, even at best, this is political propaganda.
An āarms raceā is a competition over the quality or quantity of military capabilities between states in the international system.
The US and China are in an economic race. We arenāt racing to develop a particular military capability. Nobody can even name what that capability would beā¦ āmass hacking?ā Is that it? Sounds scaryā¦ I donāt buy it. It gives me the impression of fear mongering.
If it were about AI driven military capabilities, we could have probably worked with the compute we had years ago.
I think what should be obvious is the fact that ācan be used as a weaponā doesnāt justify it being an āarm.ā The fact that they can āhackā now is largely irrelevant. Itās a nice talking point that was parroted around the AI circle jerk recently, but largely unsubstantiated. I think they all caught on to the hype Mythos was getting, so they took the narrative for another spin. But even Mythos was a load of propaganda.
How do you know it wasnāt conveniently allowed to happen, to further fuel the propaganda machine? How do you know they didnāt exaggerate or leave blank important details, particularly because theyāre financially incentivized to maintain the hype?
State actors have always hacked one another. Now with a fancy tool to do automated discovery, they might be a little faster. But that speed is also arguable, because there are reports on how AI led development actually shifts the development time burden rather than resolving it.
In case it isnāt obvious, there are experts in this field that arenāt employed by AI.
Those experts in the field (that arenāt employed by AI) mostly agree with this stance, from what I can tell.
I ran the report through AI (the irony is not lost on me) and while itās really very clear this is closer to a manifesto than a unbiased report (not that it claims to be unbiased), I donāt strictly disagree with any specific point.
I have several thoughts, however:
If you donāt think having a legion of skilled hackers that never sleep is a weapon, fine-- hell, even if you tighten your tinfoil hat and reject the claim that they can even do that, if the American government and the Chinese government both believe it has this capability, and they both believe itās advantage to have the better technology, then this is an arms race. Even if theyāre both wrong about the capabilities of AI or the plausibility of AGI, its still an arms race. If the it turned out that atomic bombs werenāt feasible or possible to create, that wouldnāt have made the race to build one less of an arms race, without hindsight.
If not clear, I donāt believe a conspiracy on this scale is a realistic assumption without more data than āI donāt believe itā. It would have to spread across several competing companies throughout most, if not all, of the employees.
Like I said in the first comment, Sanders can threaten all he wants, but I just donāt believe Congress is going to take a risk on āits not really that importantā and impose restrictions on AI development. (Side note that no restrictions on AI development is not the same as no restrictions on AI use or deployment.)
I think, as I said before, that we will need a global treaty on pausing development before any one country or company pauses development, or even slows it down.
If you donāt think having a legion of skilled hackers that never sleep is a weapon, fine-- hell, even if you tighten your tinfoil hat and reject the claim that they can even do that,
Let me tighten my tinfoil hat, really quickly. First and foremost, itās great claims which require equally great evidence. The onus of evidencing (or lack thereof) ānever sleeping skilled hacking machinesā belongs to the AI companies, not me. So far itās unsubstantiated. What we really ought be discussing is the plausibility of such a machine, given what we already know.
Thatās a whole discussion in itself, which Iām happy to have. I digress, for now.
if the American government and the Chinese government both believe it has this capability, and they both believe itās advantage to have the better technology, then thisĀ isĀ an arms race.
We have not met that definition, in my opinion. I donāt believe even the US government is convinced of this. In fact, Iām doubtful anyone is fully convinced.
The Manhattan Project was an arms race between the USSR and the US, which this AI race looks nothing like. We didnāt have to rely on CEOs to tell us as much, either.
I think the US and China were once in recent time more bought into the idea that the recent LLM (and more deeply, the transformer and attention) innovations might lead to more breakthroughs with military applications. Those breakthroughs have been sparse and rather not up to par, like a theoretical never-sleeping-hacking-machine would be. I certainly donāt see any grand scale race between them, militarily speaking. So noā¦ just as (if China were) foreign competition during combustion engine development, I wouldnāt consider that an āarms race.ā Especially if the largest surface area in the field was disparate competitors in the market of business to consumer goods.
Whereās the military research programs actually treating this like a priority? DARPA for AI would be something that comes to mind, not thisā¦ would this really be the best that the US has to offer in its supposed āarms race?ā
Even if theyāre both wrong about the capabilities of AI or the plausibility of AGI, itāsĀ stillĀ an arms race. If the it turned out that atomic bombs werenāt feasible or possible to create, that wouldnāt have made the race to build one less of an arms race, without hindsight.
Iām arguing that most the reasons itās a race are political and economical. My issue is with the word āarms.ā I would still say it is not an arms race.
If not clear, I donāt believe a conspiracy on this scale is a realistic assumption without more data than āI donāt believe itā. It would have to spread across several competing companies throughout most, if not all, of the employees.
I also donāt believe in a grand conspiracy. My belief is closer to cultural stupidity having gone āoff the rails.ā
Itās propaganda, CEOs hemorrhaging cash, people buying into the propaganda, every small win gets exacerbated while loses ignored.
My position isnāt unconventional or all that unreasonable, really. Itās a growing position, per my own observations. Whatās happening isnāt right.
I think, as I said before, that we will need a global treaty on pausing development before any one country or company pauses development, or even slows it down.
I still say we should worry about ourselves first. That narrative only benefits the tech CEOs. Slowing down now means we can preserve livelihood, plan better, not blow trillions more dollars, and eventually speed back up without risk of crashing.
Right now, the current trajectory causes harm in peopleās lives. I donāt buy the argument that this is acceptable without discussion, and that AI companies should get to pick that for us through their pushed narratives.
I also donāt buy that our current trajectory is optimal for arriving at some grander finish line. If you stop and readjust, you might actually ābeat Chinaā (for whatever thatās worth).
The onus of evidencing (or lack thereof) ānever sleeping skilled hacking machinesā belongs to the AI companies, not me. So far itās unsubstantiated.
What, hypothetically, would that look like?
We have not met that definition, in my opinion. I donāt believe even the US government is convinced of this. In fact, Iām doubtful anyone is fully convinced.
Based on what? Theyāre certainly behaving in that manner.
I certainly donāt see any grand scale race between them, militarily speaking.
This should make a global treaty relatively easy, if, as you say, no party actually believes there is a threat.
Whereās the military research programs actually treating this like a priority?
Youād expect to know about them if they existed?
Iām arguing that most the reasons itās a race are political and economical. My issue is with the word āarms.ā I would still say it is not an arms are.
Are the specific semantics important?
I also donāt believe in a grand conspiracy. My belief is closer to cultural stupidity having gone āoff the rails.ā
You do if you are claiming that the known hacks from AI companies arenāt real, and that LLMs donāt even have that capability. Consider how many people would have to be in on it to sustain that fabrication.
My position isnāt unconventional or all that unreasonable, really. Itās a growing position, per my own observations.
I mean absolutely no disrespect or insult, but an antivaxxer could say that same exact thing. They likewise do not think their stance is unconventional or unreasonable, and, to our sorrow, it is a growing position.
I still say we should worry about ourselves first.
Yes, but thereās a subtle begging the question in this statement. If it is an arms race, āworrying about ourselvesā would be continuing on developing AI at a breakneck speed. Only if we assume your stance is correct does that sentence mean slowing down.
Right now, the current trajectory causes harm in peopleās lives. I donāt buy the argument that this is acceptable without discussion
Without discussion, or without coming to the conclusion you want? I think thereās quite a bit of discussion.
Evidence of a legion of skilled hackers that never sleep would look like repeatable, preferably independently verified demonstrations that an AI system can:
Find previously unknown vulnerabilities in real, unfamiliar software;
Construct and execute working exploits;
Chain multiple vulnerabilities across systems;
Adapt when an attack fails;
Maintain access and pursue objectives over long periods;
Do this end-to-end with minimal human direction;
Perform at approximately the level of skilled human penetration testers. Not merely solve capture-the-flag exercises or reproduce known exploits.
Those reports deserve independent scrutiny, particularly because some originate with interested companies. But the unresolved questions concern its reliability, generality, scalability and degree of autonomy. These are the important factors, of which most sources are silence.
I donāt think itās a coincidence that (1) they financially benefit from the hot air around AI, (2) something interesting happens with AI, (3) they speak only on the points that promote hot air, while (4) leaving out the important details. Thatās not a conspiracy any more than you going to the bathroom would be, once you need to piss. These CEOs donāt deserve the credit theyāre given as benevolent, honest actors. Theyāre money hungry and will do as their interests suggest.
That is different from alleging that the incidents were fabricated. No conspiracy is required for real demonstrations to be surrounded by exaggerated extrapolation. A model can perform an impressive autonomous exploit while still being far from a dependable, infinitely scalable army of expert hackers. Both things can be true.
Based on what? Theyāre certainly behaving in that manner.
Youāre right that my statement that the US government probably isnāt convinced was a strong one. I can dial it back.
The governmentās public behavior clearly demonstrates that it considers AI strategically and militarily important. DARPA says it has investedĀ more than $2 billion in AI for national security. The White House explicitly published a plan titledĀ āWinning the AI Raceā. The Defense Departmentās current strategy explicitly describes military AI as a race and calls forĀ āMilitary AI Dominanceā.
So, yes: there are military research programs, substantial spending, official doctrine and concrete efforts to integrate AI into warfighting. Asking āwhere is DARPA for AI?ā was a bad rhetorical question. It exists.
Consequently, Iāll revise my position:
It is defensible to describe the narrower US-China competition in military AI autonomous weapons, cyber operations, intelligence, targeting and command systems as an potential emerging arms race. What remains inadequately established is the much broader claim that all frontier-model development constitutes one unified, winner-take-all arms race.
My disbelief is that weāll ever graduate from āpotential emergingā to āactualā arms race. Therefore, I also believe there is better ways to invest our time and resources. Not to mention the humanitarian concerns.
This should make a global treaty relatively easy, if, as you say, no party actually believes there is a threat.
Not necessarily. States resist agreements because they are uncertain about future capabilities and want to preserve their options. Verification would also be extremely difficult; the same model, compute cluster and vulnerability research can support both defensive and offensive purposes.
Still, I think your larger point stands: if governments genuinely believe uncontrolled military AI is dangerous, they should be willing to negotiate reciprocal limits.
I would support targeted agreements covering things such as autonomous nuclear-launch decisions, fully autonomous lethal targeting, attacks on civilian infrastructure and particular forms of autonomous cyber operation. A treaty pausing āAI developmentā generally would be much harder to define and verify.
I would also support a local measure to slow AI development while we find a better set of training wheels. I donāt buy the concerns of an āarms raceā here. In my eyes, we have much better chances of benefiting from the ordeal if we slow down. If we donāt, the benefactors are the AI companies (particularly, at our loss too).
Youād expect to know about [military programs] if they existed?
Not all of them. Classified programs obviously exist, and absence of public evidence is not evidence of absence.
But that principle cuts both ways: secrecy cannot serve as positive evidence for any particular claim either. A public argument that an arms race exists must ultimately rely on observable budgets, doctrine, procurement, deployments, research programs and demonstrated capabilities. As noted above, some of it exists but its details are all too conveniently sparse.
Pair that with the financial incentives, thatās what I call a cultural driver for propaganda. Propaganda can boom organically just like any other movement, which doesnāt require a conspiracy.
Are the specific semantics important?
Yes, the label smuggles in policy conclusions. It also smuggles in a the rhetoric that we canāt slow down, must win, because thereās some kind of existential threat.
Calling something an āarms raceā suggests that falling behind creates an intolerable security risk, speed is inherently protective, restraint is dangerous, and ordinary regulation benefits the enemy. That is precisely how the term is being used politically.
Nevertheless, I was treating one narrow academic definition as though it controlled ordinary language. Under the broader Oxford definition, the military-AI competition may arguably qualify. I will argue that it doesnāt.
The semantic distinction still matters when someone moves from:
The US and China are competing in consequential military-AI capabilities
to:
Therefore, every restriction on American commercial AI companies threatens national security.
That conclusion does not follow automatically. General frontier-model development overlaps some with military capability, but the two are not identical.
You do if you are claiming that the known hacks from AI companies arenāt real.
That is not the claim I was trying to make. My claim is that real capabilities and incidents exist, but companies have incentives to present the strongest results as representative, extrapolate from demonstrations to future dominance, and characterize policies benefiting them as national-security necessities. All while none of it is necessarily qualifying as the risk they present it to be. Meanwhile, you take on the responsibility of paying out of your livelihood to support their endeavors ā because of the supposed risks. Well, if the risks are overplayed, then what?
That requires no coordinated conspiracy. It requires ordinary institutional incentives, selective emphasis and mutually reinforcing beliefs. Most employees would not need to deceive anyone or even agree about the geopolitical implications.
An antivaxxer could say that same exact thing.
Fair. Saying my position is āgrowingā does not make it correct.
If it is an arms race, āworrying about ourselvesā would be continuing to develop AI at breakneck speed.
Agreed: I begged the disputed question with that wording.
But even accepting that a military-AI arms race exists does not establish that breakneck, minimally regulated commercial development is the optimal response. It could favor:
Concentrated investment in defensive cybersecurity;
Hardened critical infrastructure;
Carefully tested military systems;
Limits on publishing or exporting dangerous capabilities;
Mandatory security evaluations;
Counterintelligence and model-weight protection;
Arms-control negotiations;
Slower deployment of unreliable systems.
ā¦all while more so maintaining our homes, sanctity, budgets, and privacy.
An arms race creates an argument for maintaining capability. It does not prove that maximum aggregate speed is safer than targeted development, testing and regulation.
So, why do we entertain the narratives pushed primarily by tech CEOs when it creates exactly the difference which we should seek not to create?
Without discussion, or without coming to the conclusion you want?
There is clearly substantial discussion. My complaint should have been more precise: the existence of public discussion does not mean the public has meaningful influence over the decisions, nor that the costs and benefits are distributed democratically.
I do have a preferred conclusion, and disagreement with it is not evidence that discussion is illegitimate. The question is whether āChinaā is being used to foreclose ordinary evaluation of labor displacement, energy use, surveillance, market concentration and safetyāas though the arms-race premise settles every downstream policy question.
I donāt believe this social and economical pain is worthwhile. The narrative is overblown to justify it. That doesnāt require a conspiracy.
Those reports deserve independent scrutiny, particularly because some originate with interested companies.
This is where it all gets a little tinfoil hatty for me. The timeline is that huggingface detected a hack, shut down the hack, and reported it to the FBI. Days later, OpenAI discovered that their AI was the source of the attack. Are you implying that Huggingface and OpenAI were working together, and were willing to make a false FBI report for a PR stunt? That seems like a pretty extraordinary claim.
What remains inadequately established is the much broader claim that all frontier-model development constitutes one unified, winner-take-all arms race.
When it comes go AGI, the distinction between skeptics in the field and believers (because I donāt know a word that is an antonym of skeptic) in the field isnāt that this is race, itās how long the race is going to be. Believers claim that AGI will be created within a decade, skeptics put it within 25 years. (These numbers are for example only; I havenāt kept up on the specific claims).
People that reject the possibility of AGI at all are, from my experience, using a definition of AGI that includes sentience or consciousness, but the accurate definition of AGI doesnāt mention these concepts at all.
My disbelief is that weāll ever graduate from āpotential emergingā to āactualā arms race.
Based on what?
I would support targeted agreements covering things such as autonomous nuclear-launch decisions, fully autonomous lethal targeting, attacks on civilian infrastructure and particular forms of autonomous cyber operation.
These things do not need a treaty. I would wholeheartedly support any legislation to this effect. I recall that one company, Anthropic I think, lost government contracts because it categorically refused to allow its AI to be used for public surveillance or weapon systems.
Which means the government wonted to do those things. Scary thought.
A treaty pausing āAI developmentā generally would be much harder to define and verify.
No more than if would be to police a countryās efforts to develop nuclear weapon technology.
An arms race creates an argument for maintaining capability. It does not prove that maximum aggregate speed is safer than targeted development, testing and regulation.
At the risk of being snarky, do you not understand what a race is?
I think Iāve assumed some shared knowledge that we may not have.
I focused on the current capability of AI in regard to hacking to demonstrate that it could be used offensively, but the danger that warrants a global pause in development isnāt strictly the ability to hack, though it is entwined with that concern.
The danger (many consider an existential danger) is creating an LLM that is better at developing LLMs than even the best humans. Once that happens, it becomes a positive feedback loop. If we allow it to help make a better version of itself (and I think we both agree that the incentives, both financial and strategic, would push them to do so) then we lose almost all control over AI.
Note that being better at improving LLMs is not AGI. That is to say that we can reach this point without arriving at anyoneās definition of AGI-- the artificial intelligence does not need to be general for this to happen.
Another assumption I made is that we both understood that no one on this planet really understands why LLMs work as they do. Chatting using natural language and receiving a coherent natural language answer is an emergent property of LLMs that we do not fully understand.
The āarms raceā is based on the pretty solid assumption that AGI is possible using current transformer technology and that the first group to get to a place where their LLM is better than any human at improving LLMs will be the first to AGI. A side effect is that the skills that would make an LLM better at improving LLMs, also makes them better at finding and using exploits in software.
Iām not much concerned on whether people might consider the ability to improve itself as an āarmā and therefore comfortable calling this an arms race. The threat is arguably more dangerous to humanity than nuclear bombs could ever be.
Thatās why we need a global treaty to halt AI development.
Are you familiar with AI 2027? As far as I know none of the people who worked on that are employed by AI companies. (Though I think some might have been in the past.)
Thereās a really excellent third-party YouTube Video that I recommend, that walks through the narratives in the paper.
Those reports deserve independent scrutiny, particularly because some originate with interested companies.
This is where it all gets a little tinfoil hatty for me. The timeline is that huggingface detected a hack, shut down the hack, and reported it to the FBI. Days later, OpenAI discovered that their AI was the source of the attack.
I donāt know how to explain this without risk that I might be getting so pedantic that it comes off as patronizing. Iām not trying to, anyway.
HuggingFace detected an attack, took corrective action, and reported it. Days later, OpenAI released their statement. Thatās typical behavior regardless of whether Iām right or not.
My point is that CEOs are financially incentivized. Look at what drives investment into AI companies ā itās a lot of hype. Hence the concern that weāre in an āAI bubble.ā The ābubbleā would be speculative investments based on hype.
My statements come from marrying the ideas that (1) CEOs are financially incentivized and (2) the value of their companies are current being driven by hype rather than demonstrated value. Therefore, what naturally leads is that you have CEOs with a financial incentive to maintain hype around the topic.
Are you implying that Huggingface and OpenAI were working together, and were willing to make a false FBI report for a PR stunt? That seems like a pretty extraordinary claim.
No, not at all. Iām implying (as I thought my prior questions made clear) that this could have been allowed to happen merely to demonstrate that it can happen. That might, for example, include having a āsandboxā (that isnāt properly air gapped) while doing ātestingā (on penetration capabilities).
My argument could be inferred to suggest that OpenAI wouldnāt have cared if it were HuggingFace, Google, or some other actor.
This, to me, comes across much to the same tune as Mythos was being given by Anthropic. They touted its ability to find exploits in software, though it didnāt quite live up to scrutiny. The tout had an impact on stock valuation, though. Same with having Mythos pulled from foreign use for ānational securityā reasons, then only dropping Fable as the consumer facing version.
You can argue that these actions make sense when thereās an actual concern. Sure. My argument is that Anthropic, OpenAI, and others benefit from the concern, financially speaking. They want more concern. They want people to believe the shit they spew, because that maintains the value of their companies. They know that, which is why they frame these events as they do.
This isnāt a tinfoil hat conclusion. Itās a belief in uncoordinated, unethical behavior due to financial incentives. Furthermore, Iām pissed off that it caused so much more harm within communities than would be necessary. This is not an extraordinary claim. As I began in my earlier comment, itās closer to ācultural stupidity.ā
ā-
I will have to give time to the rest of what you said. You make remarks insinuating that I dont know certain things, but honestly I in many cases would like to argue the point. Itās not that I ādonāt know,ā itās that I believe differently than you do.
Iāll need to figure out how to respond without turning this into a Thomas Hobbes vs. John Bramhall situation.
There are several problems here. If you address them, Iām happy to answer your question faithfully. But as it stands, I donāt think itās a fair comparison.
First and foremost, weāre specifically discussing an āarms race,ā not just a ārace.ā I fail to understand what qualifies an AI company to be an expert in national security, weapons, and foreign affairs. To me, an AI company is an expert in what makes them more profit. That just so happens to involve AI here, but not āarms races.ā
Second, doctors are probably not necessarily experts on vaccines. Theyāre likely to be experts on vaccine benefits, recommendation, procurement, administering, storage, ā¦ but thatās just administration and some operational knowledge. A vaccine expert (IMO) is a specialist for vaccines. Theyāre probably not working in a hospital.
A pharmaceutical company might be an expert on vaccines. But whatās the parallel to AI companies? Pharmaceutical companies research and develop actual vaccines. AI companies do not research and develop arms. The only argument I understand for this would be to say that an LLM is a weapon.
I donāt buy that an LLM is a weapon. Real competition exists, and certain military applications could become arms races, but AI as a whole is not one. Building out swaths of data centers to power more AI without a concrete public benefit isnāt something we should put up with.
You donāt benefit from the narrative that this is an arms race, not as an individual or a citizen. The benefactors here are the AI companies.
The technology can certainly be used as a weapon; it can hack systems already, and defend against hacks. This capability will only increase, and logically the better AI affords a strategic benefit. Hence, an arms race.
The rest of your questions arenāt questions or presuppose that AI canāt be used as a weapon.
In case it isnāt obvious, there are experts in this field that arenāt employed by AI.
The term āarms raceā has a definition and Iām pretty sure this still doesnāt apply. So in my opinion, even at best, this is political propaganda.
The US and China are in an economic race. We arenāt racing to develop a particular military capability. Nobody can even name what that capability would beā¦ āmass hacking?ā Is that it? Sounds scaryā¦ I donāt buy it. It gives me the impression of fear mongering.
If it were about AI driven military capabilities, we could have probably worked with the compute we had years ago.
I think what should be obvious is the fact that ācan be used as a weaponā doesnāt justify it being an āarm.ā The fact that they can āhackā now is largely irrelevant. Itās a nice talking point that was parroted around the AI circle jerk recently, but largely unsubstantiated. I think they all caught on to the hype Mythos was getting, so they took the narrative for another spin. But even Mythos was a load of propaganda.
How do you know it wasnāt conveniently allowed to happen, to further fuel the propaganda machine? How do you know they didnāt exaggerate or leave blank important details, particularly because theyāre financially incentivized to maintain the hype?
State actors have always hacked one another. Now with a fancy tool to do automated discovery, they might be a little faster. But that speed is also arguable, because there are reports on how AI led development actually shifts the development time burden rather than resolving it.
Those experts in the field (that arenāt employed by AI) mostly agree with this stance, from what I can tell.
I linked a paper in one of my earlier replies. Hereās another by a different author: https://tnsr.org/2021/06/debunking-the-ai-arms-race-theory/
ā¦ and, I just found a table. Hereās 7 more I guess.
I ran the report through AI (the irony is not lost on me) and while itās really very clear this is closer to a manifesto than a unbiased report (not that it claims to be unbiased), I donāt strictly disagree with any specific point.
I have several thoughts, however:
If you donāt think having a legion of skilled hackers that never sleep is a weapon, fine-- hell, even if you tighten your tinfoil hat and reject the claim that they can even do that, if the American government and the Chinese government both believe it has this capability, and they both believe itās advantage to have the better technology, then this is an arms race. Even if theyāre both wrong about the capabilities of AI or the plausibility of AGI, its still an arms race. If the it turned out that atomic bombs werenāt feasible or possible to create, that wouldnāt have made the race to build one less of an arms race, without hindsight.
If not clear, I donāt believe a conspiracy on this scale is a realistic assumption without more data than āI donāt believe itā. It would have to spread across several competing companies throughout most, if not all, of the employees.
Like I said in the first comment, Sanders can threaten all he wants, but I just donāt believe Congress is going to take a risk on āits not really that importantā and impose restrictions on AI development. (Side note that no restrictions on AI development is not the same as no restrictions on AI use or deployment.)
I think, as I said before, that we will need a global treaty on pausing development before any one country or company pauses development, or even slows it down.
Let me tighten my tinfoil hat, really quickly. First and foremost, itās great claims which require equally great evidence. The onus of evidencing (or lack thereof) ānever sleeping skilled hacking machinesā belongs to the AI companies, not me. So far itās unsubstantiated. What we really ought be discussing is the plausibility of such a machine, given what we already know.
Thatās a whole discussion in itself, which Iām happy to have. I digress, for now.
We have not met that definition, in my opinion. I donāt believe even the US government is convinced of this. In fact, Iām doubtful anyone is fully convinced.
The Manhattan Project was an arms race between the USSR and the US, which this AI race looks nothing like. We didnāt have to rely on CEOs to tell us as much, either.
I think the US and China were once in recent time more bought into the idea that the recent LLM (and more deeply, the transformer and attention) innovations might lead to more breakthroughs with military applications. Those breakthroughs have been sparse and rather not up to par, like a theoretical never-sleeping-hacking-machine would be. I certainly donāt see any grand scale race between them, militarily speaking. So noā¦ just as (if China were) foreign competition during combustion engine development, I wouldnāt consider that an āarms race.ā Especially if the largest surface area in the field was disparate competitors in the market of business to consumer goods.
Whereās the military research programs actually treating this like a priority? DARPA for AI would be something that comes to mind, not thisā¦ would this really be the best that the US has to offer in its supposed āarms race?ā
Iām arguing that most the reasons itās a race are political and economical. My issue is with the word āarms.ā I would still say it is not an arms race.
I also donāt believe in a grand conspiracy. My belief is closer to cultural stupidity having gone āoff the rails.ā
Itās propaganda, CEOs hemorrhaging cash, people buying into the propaganda, every small win gets exacerbated while loses ignored.
My position isnāt unconventional or all that unreasonable, really. Itās a growing position, per my own observations. Whatās happening isnāt right.
I still say we should worry about ourselves first. That narrative only benefits the tech CEOs. Slowing down now means we can preserve livelihood, plan better, not blow trillions more dollars, and eventually speed back up without risk of crashing.
Right now, the current trajectory causes harm in peopleās lives. I donāt buy the argument that this is acceptable without discussion, and that AI companies should get to pick that for us through their pushed narratives.
I also donāt buy that our current trajectory is optimal for arriving at some grander finish line. If you stop and readjust, you might actually ābeat Chinaā (for whatever thatās worth).
deleted by creator
What, hypothetically, would that look like?
Based on what? Theyāre certainly behaving in that manner.
This should make a global treaty relatively easy, if, as you say, no party actually believes there is a threat.
Youād expect to know about them if they existed?
Are the specific semantics important?
You do if you are claiming that the known hacks from AI companies arenāt real, and that LLMs donāt even have that capability. Consider how many people would have to be in on it to sustain that fabrication.
I mean absolutely no disrespect or insult, but an antivaxxer could say that same exact thing. They likewise do not think their stance is unconventional or unreasonable, and, to our sorrow, it is a growing position.
Yes, but thereās a subtle begging the question in this statement. If it is an arms race, āworrying about ourselvesā would be continuing on developing AI at a breakneck speed. Only if we assume your stance is correct does that sentence mean slowing down.
Without discussion, or without coming to the conclusion you want? I think thereās quite a bit of discussion.
Evidence of a legion of skilled hackers that never sleep would look like repeatable, preferably independently verified demonstrations that an AI system can:
We have evidence meaningfully approaching parts of that. DARPAāsĀ AI Cyber ChallengeĀ demonstrated autonomous vulnerability discovery and remediation. OpenAI reported that models conducting an evaluationĀ escaped the intended research environment, chained vulnerabilities, and accessed Hugging Faceās production infrastructure. Anthropic and Meta have also reported āagenticā cyberattacks.
Those reports deserve independent scrutiny, particularly because some originate with interested companies. But the unresolved questions concern its reliability, generality, scalability and degree of autonomy. These are the important factors, of which most sources are silence.
I donāt think itās a coincidence that (1) they financially benefit from the hot air around AI, (2) something interesting happens with AI, (3) they speak only on the points that promote hot air, while (4) leaving out the important details. Thatās not a conspiracy any more than you going to the bathroom would be, once you need to piss. These CEOs donāt deserve the credit theyāre given as benevolent, honest actors. Theyāre money hungry and will do as their interests suggest.
That is different from alleging that the incidents were fabricated. No conspiracy is required for real demonstrations to be surrounded by exaggerated extrapolation. A model can perform an impressive autonomous exploit while still being far from a dependable, infinitely scalable army of expert hackers. Both things can be true.
Youāre right that my statement that the US government probably isnāt convinced was a strong one. I can dial it back.
The governmentās public behavior clearly demonstrates that it considers AI strategically and militarily important. DARPA says it has investedĀ more than $2 billion in AI for national security. The White House explicitly published a plan titledĀ āWinning the AI Raceā. The Defense Departmentās current strategy explicitly describes military AI as a race and calls forĀ āMilitary AI Dominanceā.
So, yes: there are military research programs, substantial spending, official doctrine and concrete efforts to integrate AI into warfighting. Asking āwhere is DARPA for AI?ā was a bad rhetorical question. It exists.
Consequently, Iāll revise my position:
It is defensible to describe the narrower US-China competition in military AI autonomous weapons, cyber operations, intelligence, targeting and command systems as an potential emerging arms race. What remains inadequately established is the much broader claim that all frontier-model development constitutes one unified, winner-take-all arms race.
My disbelief is that weāll ever graduate from āpotential emergingā to āactualā arms race. Therefore, I also believe there is better ways to invest our time and resources. Not to mention the humanitarian concerns.
Not necessarily. States resist agreements because they are uncertain about future capabilities and want to preserve their options. Verification would also be extremely difficult; the same model, compute cluster and vulnerability research can support both defensive and offensive purposes.
Still, I think your larger point stands: if governments genuinely believe uncontrolled military AI is dangerous, they should be willing to negotiate reciprocal limits.
I would support targeted agreements covering things such as autonomous nuclear-launch decisions, fully autonomous lethal targeting, attacks on civilian infrastructure and particular forms of autonomous cyber operation. A treaty pausing āAI developmentā generally would be much harder to define and verify.
I would also support a local measure to slow AI development while we find a better set of training wheels. I donāt buy the concerns of an āarms raceā here. In my eyes, we have much better chances of benefiting from the ordeal if we slow down. If we donāt, the benefactors are the AI companies (particularly, at our loss too).
Not all of them. Classified programs obviously exist, and absence of public evidence is not evidence of absence.
But that principle cuts both ways: secrecy cannot serve as positive evidence for any particular claim either. A public argument that an arms race exists must ultimately rely on observable budgets, doctrine, procurement, deployments, research programs and demonstrated capabilities. As noted above, some of it exists but its details are all too conveniently sparse.
Pair that with the financial incentives, thatās what I call a cultural driver for propaganda. Propaganda can boom organically just like any other movement, which doesnāt require a conspiracy.
Yes, the label smuggles in policy conclusions. It also smuggles in a the rhetoric that we canāt slow down, must win, because thereās some kind of existential threat.
Calling something an āarms raceā suggests that falling behind creates an intolerable security risk, speed is inherently protective, restraint is dangerous, and ordinary regulation benefits the enemy. That is precisely how the term is being used politically.
Nevertheless, I was treating one narrow academic definition as though it controlled ordinary language. Under the broader Oxford definition, the military-AI competition may arguably qualify. I will argue that it doesnāt.
The semantic distinction still matters when someone moves from:
to:
That conclusion does not follow automatically. General frontier-model development overlaps some with military capability, but the two are not identical.
That is not the claim I was trying to make. My claim is that real capabilities and incidents exist, but companies have incentives to present the strongest results as representative, extrapolate from demonstrations to future dominance, and characterize policies benefiting them as national-security necessities. All while none of it is necessarily qualifying as the risk they present it to be. Meanwhile, you take on the responsibility of paying out of your livelihood to support their endeavors ā because of the supposed risks. Well, if the risks are overplayed, then what?
That requires no coordinated conspiracy. It requires ordinary institutional incentives, selective emphasis and mutually reinforcing beliefs. Most employees would not need to deceive anyone or even agree about the geopolitical implications.
Fair. Saying my position is āgrowingā does not make it correct.
Agreed: I begged the disputed question with that wording.
But even accepting that a military-AI arms race exists does not establish that breakneck, minimally regulated commercial development is the optimal response. It could favor:
ā¦all while more so maintaining our homes, sanctity, budgets, and privacy.
An arms race creates an argument for maintaining capability. It does not prove that maximum aggregate speed is safer than targeted development, testing and regulation.
So, why do we entertain the narratives pushed primarily by tech CEOs when it creates exactly the difference which we should seek not to create?
There is clearly substantial discussion. My complaint should have been more precise: the existence of public discussion does not mean the public has meaningful influence over the decisions, nor that the costs and benefits are distributed democratically.
I do have a preferred conclusion, and disagreement with it is not evidence that discussion is illegitimate. The question is whether āChinaā is being used to foreclose ordinary evaluation of labor displacement, energy use, surveillance, market concentration and safetyāas though the arms-race premise settles every downstream policy question.
I donāt believe this social and economical pain is worthwhile. The narrative is overblown to justify it. That doesnāt require a conspiracy.
This is where it all gets a little tinfoil hatty for me. The timeline is that huggingface detected a hack, shut down the hack, and reported it to the FBI. Days later, OpenAI discovered that their AI was the source of the attack. Are you implying that Huggingface and OpenAI were working together, and were willing to make a false FBI report for a PR stunt? That seems like a pretty extraordinary claim.
When it comes go AGI, the distinction between skeptics in the field and believers (because I donāt know a word that is an antonym of skeptic) in the field isnāt that this is race, itās how long the race is going to be. Believers claim that AGI will be created within a decade, skeptics put it within 25 years. (These numbers are for example only; I havenāt kept up on the specific claims).
People that reject the possibility of AGI at all are, from my experience, using a definition of AGI that includes sentience or consciousness, but the accurate definition of AGI doesnāt mention these concepts at all.
Based on what?
These things do not need a treaty. I would wholeheartedly support any legislation to this effect. I recall that one company, Anthropic I think, lost government contracts because it categorically refused to allow its AI to be used for public surveillance or weapon systems.
Which means the government wonted to do those things. Scary thought.
No more than if would be to police a countryās efforts to develop nuclear weapon technology.
At the risk of being snarky, do you not understand what a race is?
I think Iāve assumed some shared knowledge that we may not have.
I focused on the current capability of AI in regard to hacking to demonstrate that it could be used offensively, but the danger that warrants a global pause in development isnāt strictly the ability to hack, though it is entwined with that concern.
The danger (many consider an existential danger) is creating an LLM that is better at developing LLMs than even the best humans. Once that happens, it becomes a positive feedback loop. If we allow it to help make a better version of itself (and I think we both agree that the incentives, both financial and strategic, would push them to do so) then we lose almost all control over AI.
Note that being better at improving LLMs is not AGI. That is to say that we can reach this point without arriving at anyoneās definition of AGI-- the artificial intelligence does not need to be general for this to happen.
Another assumption I made is that we both understood that no one on this planet really understands why LLMs work as they do. Chatting using natural language and receiving a coherent natural language answer is an emergent property of LLMs that we do not fully understand.
The āarms raceā is based on the pretty solid assumption that AGI is possible using current transformer technology and that the first group to get to a place where their LLM is better than any human at improving LLMs will be the first to AGI. A side effect is that the skills that would make an LLM better at improving LLMs, also makes them better at finding and using exploits in software.
Iām not much concerned on whether people might consider the ability to improve itself as an āarmā and therefore comfortable calling this an arms race. The threat is arguably more dangerous to humanity than nuclear bombs could ever be.
Thatās why we need a global treaty to halt AI development.
Are you familiar with AI 2027? As far as I know none of the people who worked on that are employed by AI companies. (Though I think some might have been in the past.)
Thereās a really excellent third-party YouTube Video that I recommend, that walks through the narratives in the paper.
I donāt know how to explain this without risk that I might be getting so pedantic that it comes off as patronizing. Iām not trying to, anyway.
HuggingFace detected an attack, took corrective action, and reported it. Days later, OpenAI released their statement. Thatās typical behavior regardless of whether Iām right or not.
My point is that CEOs are financially incentivized. Look at what drives investment into AI companies ā itās a lot of hype. Hence the concern that weāre in an āAI bubble.ā The ābubbleā would be speculative investments based on hype.
My statements come from marrying the ideas that (1) CEOs are financially incentivized and (2) the value of their companies are current being driven by hype rather than demonstrated value. Therefore, what naturally leads is that you have CEOs with a financial incentive to maintain hype around the topic.
No, not at all. Iām implying (as I thought my prior questions made clear) that this could have been allowed to happen merely to demonstrate that it can happen. That might, for example, include having a āsandboxā (that isnāt properly air gapped) while doing ātestingā (on penetration capabilities).
My argument could be inferred to suggest that OpenAI wouldnāt have cared if it were HuggingFace, Google, or some other actor.
This, to me, comes across much to the same tune as Mythos was being given by Anthropic. They touted its ability to find exploits in software, though it didnāt quite live up to scrutiny. The tout had an impact on stock valuation, though. Same with having Mythos pulled from foreign use for ānational securityā reasons, then only dropping Fable as the consumer facing version.
You can argue that these actions make sense when thereās an actual concern. Sure. My argument is that Anthropic, OpenAI, and others benefit from the concern, financially speaking. They want more concern. They want people to believe the shit they spew, because that maintains the value of their companies. They know that, which is why they frame these events as they do.
This isnāt a tinfoil hat conclusion. Itās a belief in uncoordinated, unethical behavior due to financial incentives. Furthermore, Iām pissed off that it caused so much more harm within communities than would be necessary. This is not an extraordinary claim. As I began in my earlier comment, itās closer to ācultural stupidity.ā
ā-
I will have to give time to the rest of what you said. You make remarks insinuating that I dont know certain things, but honestly I in many cases would like to argue the point. Itās not that I ādonāt know,ā itās that I believe differently than you do.
Iāll need to figure out how to respond without turning this into a Thomas Hobbes vs. John Bramhall situation.