Real-SWE Benchmark — Specific LabsUnder 10 min: 70 failed (71.4%) and 28 passed (28.6%), out of 98 rollouts.10 min or longer: 398 failed (73.4%) and 144 passed (26.6%), out of 542 rollouts.Gemini 3.8 Flash: 31.2% · $2.50; Gemini CLIGPT-5.6 Sol: 16.2% · $2.65; Codex CLIMuse Spark 1.3: 23.8% · $2.74; Muse CodeGrok 4.6: 23.8% · $3.44; Grok Build; incomplete usage, actual cost may be higherKimi K3: 18.8% · $3.90; Kimi Code; incomplete usage, actual cost may be higherGPT-6 Astra: 33.8% · $4.67; Codex CLIGLM 5.3: 28.8% · $5.12; Claude CodeFable 5.1: 38.8% · $6.96; Claude CodeGemini 3.8 Flash: 31.2% · $2.50; Gemini CLIGPT-5.6 Sol: 16.2% · $2.65; Codex CLIMuse Spark 1.3: 23.8% · $2.74; Muse CodeGrok 4.6: 23.8% · $3.44; Grok Build; incomplete usage, actual cost may be higherKimi K3: 18.8% · $3.90; Kimi Code; incomplete usage, actual cost may be higherGPT-6 Astra: 33.8% · $4.67; Codex CLIGLM 5.3: 28.8% · $5.12; Claude CodeFable 5.1: 38.8% · $6.96; Claude CodeEntitlement overage lines · Fable 5.1: 34kMulti-region sweep · Fable 5.1: 30kTax jurisdiction · Fable 5.1: 78kAPI token metering · Fable 5.1: 95kAPI keys & environments · Fable 5.1: 71kS3 datastore measurement · Fable 5.1: 62kCustomer identity migration · Fable 5.1: 67kBilling schedule migration · Fable 5.1: 26kLinearizable scan · Fable 5.1: 86kAnalytics stream reducer · Fable 5.1: 88kEntitlement overage lines · GPT-6 Astra: 13kMulti-region sweep · GPT-6 Astra: 13kTax jurisdiction · GPT-6 Astra: 24kAPI token metering · GPT-6 Astra: 31kAPI keys & environments · GPT-6 Astra: 32kS3 datastore measurement · GPT-6 Astra: 22kCustomer identity migration · GPT-6 Astra: 25kBilling schedule migration · GPT-6 Astra: 15kLinearizable scan · GPT-6 Astra: 33kAnalytics stream reducer · GPT-6 Astra: 29kEntitlement overage lines · Gemini 3.8 Flash: 78kMulti-region sweep · Gemini 3.8 Flash: 67kTax jurisdiction · Gemini 3.8 Flash: 95kAPI token metering · Gemini 3.8 Flash: 134kAPI keys & environments · Gemini 3.8 Flash: 102kS3 datastore measurement · Gemini 3.8 Flash: 106kCustomer identity migration · Gemini 3.8 Flash: 97kBilling schedule migration · Gemini 3.8 Flash: 70kLinearizable scan · Gemini 3.8 Flash: 106kAnalytics stream reducer · Gemini 3.8 Flash: 88kEntitlement overage lines · GLM 5.3: 68kMulti-region sweep · GLM 5.3: 53kTax jurisdiction · GLM 5.3: 141kAPI token metering · GLM 5.3: 177kAPI keys & environments · GLM 5.3: 125kS3 datastore measurement · GLM 5.3: 121kCustomer identity migration · GLM 5.3: 90kBilling schedule migration · GLM 5.3: 58kLinearizable scan · GLM 5.3: 172kAnalytics stream reducer · GLM 5.3: 169kEntitlement overage lines · Grok 4.6: 7kMulti-region sweep · Grok 4.6: 3kTax jurisdiction · Grok 4.6: 12kAPI token metering · Grok 4.6: 15kAPI keys & environments · Grok 4.6: 16kS3 datastore measurement · Grok 4.6: 13kCustomer identity migration · Grok 4.6: 20kBilling schedule migration · Grok 4.6: 6kLinearizable scan · Grok 4.6: 261kAnalytics stream reducer · Grok 4.6: 315kEntitlement overage lines · Muse Spark 1.3: 36kMulti-region sweep · Muse Spark 1.3: 43kTax jurisdiction · Muse Spark 1.3: 67kAPI token metering · Muse Spark 1.3: 152kAPI keys & environments · Muse Spark 1.3: 104kS3 datastore measurement · Muse Spark 1.3: 71kCustomer identity migration · Muse Spark 1.3: 76kBilling schedule migration · Muse Spark 1.3: 38kLinearizable scan · Muse Spark 1.3: 141kAnalytics stream reducer · Muse Spark 1.3: 137kEntitlement overage lines · Kimi K3: 30kMulti-region sweep · Kimi K3: 9kTax jurisdiction · Kimi K3: 39kAPI token metering · Kimi K3: 69kAPI keys & environments · Kimi K3: 44kS3 datastore measurement · Kimi K3: 32kCustomer identity migration · Kimi K3: 66kBilling schedule migration · Kimi K3: 19kLinearizable scan · Kimi K3: 71kAnalytics stream reducer · Kimi K3: 55kEntitlement overage lines · GPT-5.6 Sol: 12kMulti-region sweep · GPT-5.6 Sol: 8kTax jurisdiction · GPT-5.6 Sol: 22kAPI token metering · GPT-5.6 Sol: 31kAPI keys & environments · GPT-5.6 Sol: 25kS3 datastore measurement · GPT-5.6 Sol: 25kCustomer identity migration · GPT-5.6 Sol: 24kBilling schedule migration · GPT-5.6 Sol: 13kLinearizable scan · GPT-5.6 Sol: 37kAnalytics stream reducer · GPT-5.6 Sol: 30k
