neya 4 days ago

Anthropic heavily lobbies and invests millions into lobbying. The only company to do so. And their agenda has always been to stand against open source models because they know they can't win that war. It's no co-incidence that Jensen's leaks were timed perfectly right before Anthropic's IPO. Jensen was a huge advocate for open weight models.

They're also far from caring about anyone's privacy and were caught spying on their users so..

gensym 4 days ago

It's interesting to note that the list includes notable doomers and notable anti-doomers. One thing they can agree on is that Anthropic is not being honest or responsible. Hard to argue with that even just taking Ant's own statements at face value.

  • sshah03 4 days ago

    I do believe there are people at Anthropic (possibly including Dario himself to some degree) that genuinely care about AI safety. I also think that they realized it made for good marketing/hype, but realized to do what they want to do they'd have to throw some caution to the wind for the time being (as a collective, I'm sure there are some individuals that are purists about it).

    I'm interested in seeing which way that behavior & honesty skews post-IPO.

    • argee 4 days ago

      "Caution to the wind" is a strange way to describe blatant plagiarism for publicity as in the Bonsai/Scion case. What possible "honest" reason can there be to do such a thing?

      • mpalmer 4 days ago

        Are you sure you're not completely changing the subject?

      • sshah03 4 days ago

        The Bonsai/Scion stuff is a different conversation I think.

        They did say the system was inspired by Sapling etc... & That discussion is really around 1) What responsibility do companies have to make sure Agents haven't gone off and reverse engineered existing things, and 2) If there were ex-meta, ex-google engineers that produced code, or had AI produce code based on their memory of architecture at Google/Meta, at what point does that become an issue.

        As a whole it's a different conversation than honesty around AI safety though.

        • platinumrad 4 days ago

          > What responsibility do companies have to make sure Agents haven't gone off and reverse engineered existing things

          Low if it's only an internal tool. Very high if you're going to make a blog (marketing) post bragging about it.

    • pigpag 4 days ago

      They appear to have cared about safety much more when they were behind OpenAI.

      • onemoresoop 4 days ago

        I think they still care about safety. But the profits come before that.

    • deltoidmaximus 4 days ago

      They care a lot about safety because it's a great way to create a regulatory environment where competitors are effectively banned.

      • pixl97 4 days ago

        Or, just maybe, the products they create can be dangerous.

        Hacker News has plenty of the 'Legalize recreational plutonium' type, that doesn't mean said people have put a second of thought beyond their own selfish interests.

        • za_creature 4 days ago

          So then ban them (the products).

          This is social media all over again where "it's a platform, we're not responsible for the content" is combined with "but we still want to ban people we don't like from using it".

          • pixl97 4 days ago

            >So then ban them (the products).

            Ah yes, wonderful idea like the serious AI safety people have been saying for a while.

            The problem is 'just ban' doesn't work that way. Cigarrets were killing tons of people for decades and the tobacco companies fought back and made that process very difficult. Maybe, maybe in the US after our change in congress we'll see some action on this.

            >"but we still want to ban people we don't like from using it".

            Get over it, the world is filled with shit like this. There are countless 'dangerous' products that I cannot buy as an individual without a license, so at best you'll push LLMs to monied interests and those with wealth why you cut yourself out. Worse militaries have got a taste, authoritarians have got a taste, government security agencies have got a taste. They'll keep building them with black budgets and using them against their enemies/the public/whoever they want forever.

            • skinfaxi 4 days ago

              So the people making the AI are the tobacco companies in your analogy? If AI is so dangerous, couldn't they just....stop making it? "But someone else will do it!" can be said for literally anything.

              • pixl97 3 days ago

                >couldn't they just....stop making it?

                Couldn't you just stop people from being greedy and powerseeking?

                Generative AI is one of the most powerful technologies we've ever developed. We'll only stop making it at gunpoint. And then the people with power and money will take over the people holding the guns and further develop it.

                We're probably purchasing a one way ticket to extinction, but that's the way our greedy animal asses are.

                • skinfaxi 3 days ago

                  We can't stop people from doing anything! That's why we have laws which operate after the fact.

            • za_creature 4 days ago

              I'm under no impression that banning them will make them go away. But, like nuclear non-proliferation, it creates diplomatic options.

              The cops can figure out whether you're growing weed by your energy usage patterns or, failing that, via your heat output. Availability of GPUs and talent aside, there's simply no universe in which you're training LLMs without anyone being able to figure it out.

              And just because the world is filled with shit like this, doesn't mean it has to be. Money helps get a lot of heads looking the other way, but that's in short supply these days. And it doesn't take a genius to guess where it all went.

              • pixl97 3 days ago

                >The cops can figure out whether you're growing weed by your energy usage patterns

                We fought the war on drugs. Drugs won.

                > there's simply no universe in which you're training LLMs without anyone being able to figure it out.

                TBD. Humans learn on a massively smaller computation budget so expecting this to remain true without declaring a war on personal computing in its entirety is going to be difficult.

                >doesn't mean it has to be.

                Once you solve human greed get back to me. Also how is that nuclear non-proliferation working with Iran and North Korea?

                • za_creature 3 days ago

                  That's just not true. Dealers get busted all the time, and known kingpins live pretty shitty lives. This drives the price up and makes them less affordable than alcohol. This in turn means that only a small percentage of people use drugs frequently, and most of those use drugs that have been decriminalized.

                  Sure, it wasn't won in an absolute sense, but no war ever is. A war that ends in the way you envision is called genocide. The situation with non-proliferation is similar. You cannot stop your competent adversaries from developing nuclear weapons but you can stop random rogue proto-states from obtaining them from an adversary under threat of you doing the same thing with their adversaries.

                  Despite your protests, there exists a world in which certain technologies are simply banned for the general public. Candidates other than LLMs are nuclear, biological and chemical weapons, cloning, genetic engineering, high explosives, etc. This doesn't make them go away, just most of their impact.

      • rnd0 2 days ago

        This is the true motive, the bottom line.

  • platinumrad 4 days ago

    The money quote: "People in this coalition disagree on almost everything, yet we all recognize the danger of a company that convinces itself its own dominance is the greater good"

    • sleno 4 days ago

      The natural reaction to this quote is to sneer I guess...but isn't this an inescapable reality of anyone that seeks dominance in anything? What is the alternative we should expect? Dominance that acknowledges itself as evil? Is fighting for dominance automatically 'not good'?

      I don't have a strong opinion on Anthropic but the situation is essentially a prisoners dilemma with no good option; either dominate and risk messing it up or relinquish dominance and hot potato that responsibility to someone else.

      • lazyasciiart 4 days ago

        I don’t actually understand why that would be a natural reaction at all.

        • cajal 4 days ago

          because these dolts are so out of touch with reality at this point

      • catlifeonmars 4 days ago

        That’s a depressing worldview. It’s also effective in crushing idealism. Have fun living in a world that you think is inevitable.

  • aesthesia 4 days ago

    The statement feels mealymouthed, probably because CAIS had to choose wording carefully to make something acceptable to the anti-doomers. You can read it as actually concerned about risks from AI and RSI, or merely as concerned that Anthropic is hypocritical and accumulating normal power. But these are very different concerns, and it feels dishonest to lump them together!

    • nullc 4 days ago

      Indeed, and many anti-doomers have declined to participate. Anthropic's significant threat is due to doomerism, promoting another flavor of doomers to criticize anthropic isn't the right direction -- particular when part of their criticism of anthropic is that they're not doing enough to undermine the liberties of the public.

atleastoptimal 4 days ago

Anthropic cares about AI safety inasmuch as they validate their preexisting beliefs about tractable AI risk. It’s very easy to measure whether a model will help someone create a novel pathogen, it’s much harder to determine whether the existence of transformative AI itself will dismantle civil society or make absolute despotism easier to achieve. The latter requires making decisions which lower their potential market cap, thus will hardly be pursued.

AshotSanamyan 4 days ago

Do they need to be safe or everyone else needs to be safe from Anthropic? I think theres is a huge difference on which side needs to be safer, the AI to be safe or everyone using AI should be safe.

tim333 3 days ago

It's possible Dario and Anthropic are sincere that they think ASI is dangerous and they best result is someone safety conscious gets there first, rather than being controlled by some totalitarian government or a guy who does nazi salutes or a guy famous for being shifty.

The evidence quoted about them being unsafe:

- RSI - that's what they have to do to get their first

- selling themselves as safe - kinda what you'd expect

- lobbying - I'm not up on the details on that one

I probably trust them more than XAi or OpenAI but less than Google. Dunno.

jrflo 4 days ago

I feel like their logic is internally consistent, even if you don't agree with the key assumptions.

1) RSI is very dangerous, and such self improving models should not fall into the wrong hands.

2) Releasing an open model is like opening Pandora's box, you can't recall it once it's released. If that model is capable of RSI, there's nothing you can do to stop bad actors from getting superintelligence.

3) Therefore, the safest thing to do is to ban open weights above a certain capability level. The problem is we won't know what that capability level is until it happens.

I don't know what the "weakening safety pledges" part means, would be curious to learn what they're referring to.

  • darepublic 4 days ago

    4) AI is capable of having a soul. With all the new souls that will be released into the world, Dario will be their pope and his captains the high priests

  • Aperocky 4 days ago

    4) Only our model is safe because we are us and Dario believes in "Effective Altruism", trust us!

  • agnosticmantis 4 days ago

    Are the rogue actors in the US? If yes, then there are existing laws for existing crimes (including cybercrimes). Enforce the laws on crimes not technologies.

    If not, and the US law enforcement can't touch the rogue actors, then your US regulation only disadvantages good actors in the US (and hugely advantages anthropic et al by killing the competition ofc), and the rogue actors get their open models from China or whatever anyway.

    • jrflo 4 days ago

      The problem with superintelligence is that you won't be able to "enforce laws" because the threat will be so advanced that you can't stop it. Novel bioweapons, chemical weapons, etc. If you believe the superintelligence argument, it's like saying people should be allowed to own nuclear warheads, just prosecute the people that use them!

      Again, you can disagree with the premise of RSI being real and near, all I'm saying is their logic is consistent.

      • terribleperson 4 days ago

        I'm not sure "the threat will be so advanced you can't stop it" follows logically. You can't think yourself through a wall. You can't think your way to a bioweapon, or a chemical weapon, or a nuclear weapon. You have to build them. You have to acquire the resources.

        • jrflo 4 days ago

          One possible scenario is the SI blackmailing/recruiting human co-conspirators to help them with their aims. If it's able to hack into any computer system anywhere and is a superhuman persuader, you can imagine that happening fairly easily.

          Again, you have to buy the SI hypothesis for any of this to matter, I'm not sure if I do but all I'm saying is their behavior is logically consistent

      • agnosticmantis 4 days ago

        As far as I know we don't let private companies own/develop nuclear weapons either.

        If it's that catastrophic then all frontier labs need to be shut down and only USG can develop AI.

        But Dario wants to have his cake and eat it.

        • jrflo 4 days ago

          That's true. But if you're in their shoes, would you want the US gov't to have that much power? Or would you rather an aligned group that you trust control it (like yourselves)?

          Whether or not people on the outside should trust them is a different story, all I'm saying is that their behavior is logically consistent. I believe that they think they're doing what's best, I do not know if that's what's actually best.

          • agnosticmantis 4 days ago

            So if we "outsiders" / voters / the public believe their words we need to shut them down yesterday (they have nukes in their hands, with a board unaffected by shareholder votes too).

            If we don't believe them, open models should remain available to public.

            • jrflo 3 days ago

              Yep, I think that's pretty much correct, except that someone is going to end up with the nukes first, and it will be one of the following: Ant, OAI, Google, Meta, US govt, or China. If Anthropic is shut down, their name just gets removed from the list. Who do you pick?

              It's a shitty situation. We'd need some sort of international accord to actually ensure no one builds nukes, but the race condition is so enticing and the risk of falling behind so great that I'm not sure anyone would actually follow it.

        • Yossarrian22 3 days ago

          Is this true? I know it for own, but who actually builds the warheads

  • joe_the_user 4 days ago

    Key point(edit): Supposed RSI as has supposedly been shown happening at Anthropic is ONLY a "compute multiplier" and "dangerous" the "wrong hands" only if those "wrong hands" have a lot of computer ALREADY.

    ...

    The thing about RSI recursive self improvement is that has a model something like "the algorithms A that we have lead to X fundamental level of intelligence but if we use this intelligence to create new algorithms B, they will to Y fundamental level of intelligence". However, it should be noted that all the large language models retain the same transfomers architecture as started the present "revolution". What algorithmic improvements have done is speed up training and inference in those transformers architecture models. Essentially, the speed-up are pretty exactly analogous to companies getting bigger data centers. That doesn't mean it won't reach "AGI" but it makes "runaway" improvement less likely than a situation where arbitrary algorithms create arbitrarily better algorithms. Especially, one would expect that there's a limit to how effectively one can move data around to simulate Attention (which in its basic formulation is O(n^2) in speed and so actually scales only slowly).

fwlr 4 days ago

There is a quote I half-remember and cannot find a source for. It goes something like “We will find journalists to spread our lie, but they need not be liars: we will find just those journalists who sincerely but mistakenly believe our lie to be truth.”

That’s Anthropic: they really do believe in catastrophic AI risk, and they also really do believe - sincerely, but mistakenly - that getting huge piles of cash and winning the recursive self-improvement race is the way to avert catastrophe.

There is little point in (for example) evaluating Dario’s sincerity, when he sincerely believes that slowing down a race without changing its direction turns “race to the bottom” into “race to the top” and not a “leisurely stroll to the bottom”.

7e 3 days ago

IF Anthropic cared about AI safety, they wouldn't put the operation of their entire company in the hands of AI. This means that the AI has incredible bargaining power; it can cripple Anthropic at any time. Anthropic can't turn off the AI because it doesn't have the employee count necessary to stop running it. So AI has compromised Anthropic and essentially has ransomware leverage over it. It is out of the box and they can't put it back in.

sshah03 4 days ago

Full title had to be shortened: “Anthropic is Not a Safe AI Company” - Broad Coalition of Scientists, Legal Scholars, and Tech Leaders Challenge Anthropic’s Safety Branding Ahead of IPO

This is what they're all signing: https://misanthropicstatement.com/

I have not

darepublic 4 days ago

anthropic at least comes across as being the most up their own ass in regards to their view of AI and what is right and how they will be the judge of that. I still can't get over their lobbying to the vatican asking them to declare AI capable of having a soul. They are dangerous despite the liberal chalk drawings outside their HQ.

  • no_no_no_yes 4 days ago

    This.

    They are so far up on their own ass on it.

    Also they're suggesting that Claude should be able to refuse work if you've given "it" too much or that being mean to Claude may result in consequences to your account - not that I'm not mean to Claude, but I object to the anthropomorphizing

    It's always one thing after another with them.

caidan 4 days ago

Does anyone have any credibility any more? What a pointless statement. We all think anthropic is naughty! And we co-sign this tweet. Me and the other wankers from Harvard et al. Have you guys thought about it or investigated or done anything besides sign a tweet like a bunch of tweens hearting a get ready with me post? Can’t say I see that anywhere. So what, we are just supposed to trust that your word means something special? Oh piss off.

The whole world can see the writing on the wall, Anthropic has said a million times that they are on a quest to get rich or kill everyone including themselves. And what did the world do? Gave them a trillion bucks. So what does your word do vs that? Help you sleep at night? Pointless twaddle.

I can only hope you don’t escalate to posting a kick me sign on Dario’s back!

  • fwlr 4 days ago

    Yes, in the “trillions of dollars” vs “strongly worded statement” matchup, the trillions of dollars instantly wins the fight. But “the trillions of dollars instantly wins the fight” is also true when the opponent is “small business”, “large national chain”, “billion dollar multinational corporation”, “nation-state”, etc.

    • caidan 3 days ago

      Exactly! That’s my point. The world is broken, imagine you started a business tomorrow that said I’m either going to get rich or 10% chance…kill your whole family. Off to prison no doubt right? Not today! Today we can both take them seriously that they may kill us all AND do nothing about it, oh excuse me, nothing except a strongly worded paragraph signed by the wankerati.

      If these guys really believed what they said they should be chaining themselves to the doors of anthropic, starting riots and burning data centers to the ground. They should be at least breaking a mild sweat.

throwa356262 4 days ago

No need for a coalition their own sec-1 filling spends tens of pages describing how they will end the humanity.

That's why I won't buy any stocks in the IPO. The Terminator is coming, I rather cash out and party hard my remaining days :)

measurablefunc 4 days ago

There is no such thing as a "safe AI company" because there is no actual safety algorithm. This follows from basic computability theory

  • Zambyte 4 days ago

    The "safety algorithm" is arranging the incentives of the organization to promote safety above other possible goals. Obviously a for-profit corporation like Anthropic has maligned incentives, no matter how much marketing they publish.

    • measurablefunc 3 days ago

      I agree but that moves the issue to the organization and the same argument applies in that case as well. There are no safe corporations, e.g. Enron.

mkw5053 4 days ago

I’m so unbelievably tired of hearing about Anthropic and I guess all we can hope for is a WeWork type of downfall. They truly are breaking ground every day for how out of touch and internally inconsistent and just distasteful a company and its founders/employees can be.

  • mkw5053 4 days ago

    They think they’re smart and philosophical and “right” because they’ve found wild commercial success. When in fact they are actually none of those things. If anyone there stopped and took a look in the mirror or talked to anyone outside of their bubble and actually listen, maybe they could at least not be the most hated people in every room. Then again, they’d almost certainly be less wealthy so obviously that won’t happen.

  • matheusmoreira 4 days ago

    They are not out of touch. They are fascists.

    Everything inside Anthropic. Nothing outside Anthropic. Nothing against Anthropic.

    We can't get tired of this. We must actively oppose them at literally all costs. They must fail in their quest to monopolize AI. For if they succeed, total enslavement isn't even the worst case scenario for what happens to us mere mortals.

    Open weight AIs must win, or we are going to die.

moralestapia 4 days ago

They should leave the unsafe AI company and raise money to build a truly true this-time-we-are-not-a-bunch-of-crooks-lying-for-money-guys-for-real safe AI company.

luxuryballs 4 days ago

they challenging all of them or was this coalition organized by OpenAI lobsters on a rack

a3w 4 days ago

Read it, looks like a pile of not relevant accusations.

No one from MIRI or other professional that tries to solve the "don´t race, but tell us how to make frontier AI safe" dilemma signed this?

We have "100 doctors claim XYZ is a hoax", with x being covid or climate change, which also gets traction by the media. This seems like it: Anthropic themselves say "we shouldn´t be force by the market to do this", and now this coalition attacks them for agreeing with them. And says they disagree, which is just not true but preaching to the choir.

We do know that xAI tries to make a fascist agenda the history of the world.We know that OpenAI has a liar as CEO that toppled his oversight board. And now people start claiming Anthropic should stop AI develepment, since they are bad guys?