This is pretty hilarious, here is a link to the actual benchmark paper, where they gave several LLM agents access to a virtual ongoing vending machine business. Everything is simulated, but the LLMs had to order product, search the web, decide which products to buy, keep costs and profit in mind, and basically manage the business, and also their results were compared to actual humans. Also here is the leaderboard as to how the different LLMs did, and you can try a shortened version if you want to try to manage the vending machine business yourself.
That is much much funnier. It’s willing the entire simulation out of existence. It’s attempting magic. That’s existential dread.
Yeah that was another choice quote for sure. We make the LLMs in our own image, and maybe thats not the best idea.