reddmachine 10 hours ago

I want to preface that I've done LLM evals in a proper setting before. This is far sillier. It's derivative of the "pelican riding a bicycle" test, something simple that shows how capable an LLM is at perceiving intent, dealing with physics and general aesthetics. I run this for every new model now and it gives me a decent idea of what to expect.

Rodmine 4 hours ago

The setup needs more description. Can it have pre-installed npm, npx, chrome, playright etc.? It turns out my agent did not have access to it and it is setting all of that itself. Will update when done.

  • reddmachine 42 minutes ago

    Agreed. I need to specify the playwright setup in a clearer way. Other than that nothing custom should be needed.

cyanydeez 4 hours ago

would be better if it compared to open weights models.

  • reddmachine 43 minutes ago

    Yes. I need to set up the models and harnesses for this. Likely via open router.