Post by @jonny@neuromatch.social

I don't know how much louder anyone can scream "the endgame of AI is a dependency trap for whole life surveillance"
https://openai.com/index/gpt-6-astra/

https://openai.com/index/gpt-6-astra/

20 Replies

9 Boosts

Leonardo Fontenelle

@leonardof@bertha.social 54 min. ago

@jonny for a moment, I thought it was something from OpenAI about GTA 6

CubeRootOfTrue

@CubeRootOfTrue@mathstodon.xyz 2 hr. ago

@jonny Nah, they will move on, head out into space, colonize the galaxy, etc. Timescales won't matter to them

bit

@bit@ohai.social 2 hr. ago

@jonny Which they will use to rip everyone off on any purchases through the service. Already studied, and already happening.

https://www.pcmag.com/news/shop-around-study-finds-google-ai-mode-shows-higher-prices-than-search

Laura

@landelare@mastodon.gamedev.place 2 hr. ago

@jonny If this AI is so good, it should've generated a website where the charts aren't broken.

The layout is corrupt, and the image doesn't make sense. Most of the picture is black, with the header "Terminal-Bench Science 0.1", and there appears to be a horizontal axis with API cost values on it ranging $0-$80, and some markers that say GPT-6 Astra, GPT-5.6 Sol, and Claude Fable 5. At the bottom of the image, "Terminal-Bench Science 0.1 tests whether" can be seen in italics.

Paul Barker

@pbarker@social.afront.org 3 hr. ago

@jonny Also: What is it with AI companies and incredibly heavyweight webpages which slow my laptop* to a crawl? They all seem to love them.

*: Laptop (Core Ultra 5 225U, 16 GiB RAM) is not a potato so the issue is on them.

craignicol

@craignicol@glasgow.social 5 hr. ago

@jonny I love how they put in a presentation about their Gaia model that doesn't exist, knowing full well that CEOs and consultants are going to be citing it

jonny (nonvenomous)

@jonny@neuromatch.social 5 hr. ago

Its neat how openai has caught up with Google's brand strategy from like 2016 though

xinit ☕

@xinit@mastodon.coffee 4 hr. ago

@jonny
Don't be evil?

jonny (nonvenomous)

@jonny@neuromatch.social 4 hr. ago

@xinit
Google assistant

jonny (nonvenomous)

@jonny@neuromatch.social 5 hr. ago

This is such a great x axis. This must look awesome if you don't know anything

resolution rate for terminal-bench 0.1 on y axis, API cost on x axis.

stars indicate gpt-6 at 55-65%, between $10-25. vertically above fable 5.1, 25-55% between the same price range.

GPT-5.6 and Claude opus 5 are single points around 25% and$25, fable 5 is 20% for $70 — resolution rate for terminal-bench 0.1 on y axis, API cost on x axis. stars indicate gpt-6 at 55-65%, between $10-25. vertically above fable 5.1, 25-55% between the same price range. GPT-5.6 and Claude opus 5 are single points around 25% and$25, fable 5 is 20% for $70

Spawnie

@spawnie@mastodon.social 1 hr. ago

@jonny the y axis as well, percentage points but truncated between 20 and 70 hmmm yes seems good graphmaking

Splitradix

@splitradix@assemblag.es 4 hr. ago

@jonny Q: What did the x-axis say to the y-axis? A: "They're all plotting against me!"

tkissing

@tkissing@mastodon.social 5 hr. ago

@jonny "70% success rate is fine. Humans make mistakes too. Just the other day I saw a toddler put a round peg into a square hole."

*cries*

Albert ARIBAUD Ⓜ

@aaribaud@piaille.fr 5 hr. ago

@jonny I am indeed curious to know what for them is a "resolution rate". I *suspect* it is the number of answers per time unit or some such.

jonny (nonvenomous)

@jonny@neuromatch.social 5 hr. ago

@aaribaud
Meaningless benchmark metric

jonny (nonvenomous)

@jonny@neuromatch.social 5 hr. ago

the adjoining graph measuring "output tokens" when everyone knows goddamn well that means nothing when there is at least one hidden slush fund category "reasoning tokens" that is the one that's actually inflating

jonny (nonvenomous)

@jonny@neuromatch.social 5 hr. ago

I was trained in psychology, I am familiar with the lineage of developing measurement instruments named "Gary's Perfect Intelligence Metric" and the test is just rotating a bunch of plus signs. I live in the united States and am aware of bills named "The Great Infinite Jello Glitch" that legalizes stripmining in orphanages. So naturally I love AI company press releases.

~12 more replies

jonny (nonvenomous)

@jonny@neuromatch.social 5 hr. ago

> "GPT-6 score 99% on the AGI test
> looks inside AGI test
> the arcade pack CD-ROM you'd shake out of a cereal box in 2003

2 media files

[bar plot of GPT-6 at 99%, opus 5 at 30% score. score is undefined. output tokens are undefined.]

ARC-AGI-3 tests how well agents learn as they solve unfamiliar interactive tasks. GPT‑6 Astra saturates the eval, scoring 99.9%. The average human tester scored 48%. GPT‑6 Astra was measured with our responses API harness⁠, which better reflects real-world performance than the original benchmark harness, which discards past reasoning and past messages. With this harness, we estimate Sol would score in the ballpark of ~30%. — [bar plot of GPT-6 at 99%, opus 5 at 30% score. score is undefined. output tokens are undefined.] ARC-AGI-3 tests how well agents learn as they solve unfamiliar interactive tasks. GPT‑6 Astra saturates the eval, scoring 99.9%. The average human tester scored 48%. GPT‑6 Astra was measured with our responses API harness⁠, which better reflects real-world performance than the original benchmark harness, which discards past reasoning and past messages. With this harness, we estimate Sol would score in the ballpark of ~30%.

literally a gameboy looking color matching game where you flip cards to match a color grid that is printed on the board itself. like. "red color square says make these squares around it red" level shit.

jonny (nonvenomous)

@jonny@neuromatch.social 5 hr. ago

We know we have reached true AGI when a brute force machine can reach 100% on a game that has a small number of discrete states with like 4 inputs without disclosing anything about the testing conditions

jonny (nonvenomous)

@jonny@neuromatch.social 4 hr. ago

So hang on. The screenshot test says I can pay 10 cents for a robot to move my mouse to a button?

That can't be right, it must be 10 cents to run the whole benchmark, there are only 1500 samples per run.

But the output tokens are all less than 1000... So either it can move my mouse with fewer than 1 token on average... Or...... 10 cents to move my mouse is really what is being shown here.............

3 media files