DeepSWE: el benchmark que demuestra que el 32% de los veredictos de SWE-Bench Pro eran erróneos

Este artículo sintetiza información publicada originalmente por swebench.com. Para el contexto completo, las declaraciones originales y los detalles que no hemos incluido, consulta la fuente indicada.

Resumen: DeepSWE: el benchmark que demuestra que el 32% de los veredictos de SWE-Bench Pro eran erróneos

El contexto

No models selected. Use quick select or go back to select models in the first column.

Qué ha pasado

SWE-bench Verified is a human-filtered subset of 500 instances; use the Agent dropdown to compare LMs with mini-SWE-agent or view all agents [ Post ].

Detalles

SWE-bench Multilingual features 300 tasks across 9 programming languages [ Post ].

Each entry reports the % Resolved metric, the percentage of instances solved (out of 2294 Full, 500 Verified, 300 Lite & Multilingual, 517 Multimodal).

[11/2025] Introducing CodeClash, our new eval of LMs as goal (not task) oriented developers! [ Link ]

[07/2025] mini-SWE-agent scores 65% on SWE-bench Verified in 100 lines of python code. [ Link ]

[05/2025] SWE-smith is out! Train your own models for software engineering agents. [ Link ]

Fuente original

Lee el artículo completo en swebench.com.