I don’t trust ChatGPT/GPT-4 for much to begin with, but this study is not great. From Ars Technica’s article on the same topic (with emphasis added by me):
While this new study may appear like a smoking gun to prove the hunches of the GPT-4 critics, others say not so fast. Princeton computer science professor Arvind Narayanan thinks that its findings don’t conclusively prove a decline in GPT-4’s performance and are potentially consistent with fine-tuning adjustments made by OpenAI. For example, in terms of measuring code generation capabilities, he criticized the study for evaluating the immediacy of the code’s ability to be executed rather than its correctness.
“The change they report is that the newer GPT-4 adds non-code text to its output. They don’t evaluate the correctness of the code (strange),” he tweeted. “They merely check if the code is directly executable. So the newer model’s attempt to be more helpful counted against it.”
AI researcher Simon Willison also challenges the paper’s conclusions. “I don’t find it very convincing,” he told Ars. “A decent portion of their criticism involves whether or not code output is wrapped in Markdown backticks or not.” He also finds other problems with the paper’s methodology. “It looks to me like they ran temperature 0.1 for everything,” he said. “It makes the results slightly more deterministic, but very few real-world prompts are run at that temperature, so I don’t think it tells us much about real-world use cases for the models.”
Can I pick a PC? x86 is retro, right? /joke
But seriously, probably the PS2. Mainly because it’s the only console I got as a kid and also because it’s the last console before games and consoles started wanting to phone home over the Internet. I have PS3 games that I’m pretty sure are permanently hampered or unplayable because their servers are offline, but I feel confident I can still boot any PS2 game I own and play it without issues.