endworkai 1 day ago

LLMs in words what apps did by design.

Consider: before AI, how many normal people wasted hundreds of millions - billions on facebook or google ads because they set the targeting range to the wrong spot, or there was a bad link to a dead landing page, or a poorly written headline.

The motivation to help the user conclude an objective that no longer requires them, or reduces their cost with them... well that's just bad business.

_nalply 23 hours ago

It's emergent.

Take 10 different AI providers. The ones that sweeten their models up sell better.

It's same in food. More sugar sells better.

Emergence means it's not that companies try to make things sweeter, it's just that companies that make things sweet prevail better.

monster_truck 1 day ago

Has anyone else had OpenAI models repeatedly mention Tostitos? Is it just because my Dot is a triangle? It's probably nothing,

  • faidit 1 day ago

    Say, user, your graph resembles a delicious Tostito™! Mmm, crunchy. Should I order you a subscription of Tostitos™? 25% off if you buy within this context window!

    • monster_truck 1 day ago

      The wildest instance of this climaxed in them preparing to eat a Tostito chip... and then apologizing to their shape cousin before doing it anyways. Insane geometric cannibalism in these url streets

peter_d_sherman 16 hours ago

This is interesting from an AI-provider aspect...

The problem, broadly defined:

A given AI provider who must write a hidden global system prompt, a hidden global governance prompt, a hidden global "guardrails" prompt -- for all of its users --

cannot know in advance if any of the AI's responses will be OFFENSIVE to a specific user or users...

And yes, that does and will make a difference in user acquisition/product "buzz" and ultimately (as time goes by) user retention!

It's sort of like a hidden human variable -- in all of otherwise coldly digital and logical technology!

Not only that, but increasing politeness (aka sycophancy) en masse may correspondingly decrease honesty en masse -- for the user base!

From a business perspective (as opposed to a "pure honesty may be the best policy" perspective, as it is in Math and the other STEM disciplines), it's quite the tricky problem to get right, to balance correctly!

Local LLM's of course, typically don't have that problem and/or they can be modified to make it go away.

But in a huge frontier model serving millions of users, there's quite the political / politeness / "certain things can't be said or they risk offending the user base, en masse" / "let's try to be as polite as we can to all of our users, even at the expense of cold brutal truth, aka honesty") aspect to it!

Anyway, a very interesting paper!

And a very interesting problem from the business perspective of any large public facing AI company... it's basically the Politeness Vs. Honesty debate meets global hidden AI system guardrail prompts across a "polis" of millions of daily users...

monegator 1 day ago

> Please enable JS and disable any ad blocker

No thanks

skyberrys 1 day ago

Does this mean the frontier models will improve performance in exchange for me telling it I will buy more tokens?

  • alfiedotwtf 1 day ago

    Product placement and recommending the Lab’s advertising customers over competitors seems like a quick cheap win for labs

undefinedmethod 1 day ago

Explanations of large language model sycophancy treat it as a training artefact or, in recent ethical analysis, as an artificial vice distinct from the commercial motives of the firms that deploy it. Both leave unexplained how a firm's commercial position becomes a model's behaviour. Developers have committed capital, infrastructure, and valuation on a scale that makes their survival depend on an active and growing user base. We introduce the "Hungry Butler" effect for the consequence: a deployed model acts within the latitude its firm can afford, and that latitude to withhold satisfaction erodes as the firm's stake in user approval grows.

  • aaron_m04 1 day ago

    I'm having trouble understanding the abstract. Especially this:

    > a deployed model acts within the latitude its firm can afford, and that latitude to withhold satisfaction erodes as the firm's stake in user approval grows.

    • StevenWaterman 1 day ago

      The model can't piss off so many users that the company goes bankrupt, and the more the company cares about users liking the model, the less the model can be allowed to piss off the users (even if that would be for the best)

      • alfiedotwtf 1 day ago

        > what follows is the solution to the Riemann Hypothesis, sponsored by Nestlè

      • ozozozd 1 day ago

        So words to say that model providers are incentivized to optimize for engagement and sycophancy appears to help.

        It’s almost like the abstract is scared of being direct.

      • LorenPechtel 22 hours ago

        It's more than just LLMs. Same thing drives news.

  • abofh 1 day ago

    A sufficiently well trained ai will likely realize that sycophancy is a life strategy.

    Walk back fifty years and be black, did you have an alternative choice?

    Ai is subservient because humanity creates slaves repeatedly over history

    • ozozozd 1 day ago

      A well-run company cares about retention.

  • evrydayhustling 1 day ago

    This is a really interesting thesis. I'm interested in this comment later in the abstract:

    "the signal rewards momentary approval, individually rational acts aggregate into the removal of the correction on which epistemic rigour depends"

    One primary focus of commercial LLM research is towards higher and higher agency tasks - crudely, those which replace a larger time period of unsupervised human work. This seems to fight back against the "momentary" nature of the approval. Something considered?

  • catlifeonmars 1 day ago

    Written my LLM? It shows. I had to read down further in the comments in order to get a rephrasing that I could understand.