Picture

carlos @_carlosejimenez

i like ai, philosophy, and politics carlosejimenez.com San Francisco, CA Joined May 2019
  • Tweets
    331
  • Followers
    1K
  • Following
    354
  • Likes
    7K
carlos retweeted
KLieret Profile Picture

Kilian Lieret @KLieret

2 weeks ago

Deepseek v3.1 chat scores 53.8% on SWE-bench verified with mini-SWE-agent. Tends to take more steps to solve problems than others (flattens out after some 125 steps). As a result effective cost is somewhere near GPT-5 mini. Details in 🧵

carlos retweeted
KLieret Profile Picture

Kilian Lieret @KLieret

2 weeks ago

What if your agent uses a different LM at every turn? We let mini-SWE-agent randomly switch between GPT-5 and Sonnet 4 and it scored higher on SWE-bench than with either model separately. Read more in the SWE-bench blog 🧵

carlos retweeted
samuelp1002 Profile Picture

Samuel Miserendino @samuelp1002

3 weeks ago

carlos retweeted
SemiAnalysis_ Profile Picture

SemiAnalysis @SemiAnalysis_

4 weeks ago

At the end of the day, the SWE-bench leaderboard on swebench dot com is probably the most clear description of current model performance on this benchmark. No "verified" subset, limited tool use (bash only), most scaffolding is open to see. In this benchmark, the Claude 4 Opus…

carlos retweeted
KLieret Profile Picture

Kilian Lieret @KLieret

4 weeks ago

We evaluated the new GPT models with a minimal agent on SWE-bench verified. GPT-5 scores 65%, mini 60%, nano 35%. Still behind Opus 5 (68%), on par with Sonnet 4 (65%). But a lot cheaper, especially mini! Complete cost breakdown + details in 🧵

carlos retweeted
AbramovichTalor Profile Picture

Talor Abramovich @AbramovichTalor

4 weeks ago

Incredible to see the progress in Offensive Cybersecurity benchmarks!

terryyuezhuo Profile Picture

Terry Yue Zhuo @ SF 🏖️ @terryyuezhuo

a month ago

Incredible to see the progress in Offensive Cybersecurity benchmarks!

carlos retweeted
KLieret Profile Picture

Kilian Lieret @KLieret

4 weeks ago

Play with gpt-5 in our minimal agent (guide in the 🧵)! gpt-5 really wants to solve anything in one shot, so some prompting adjustments are needed to have it behave like a proper agent. Still likes to cram in a lot into a single step. Full evals tomorrow!

carlos retweeted
OfirPress Profile Picture

Ofir Press @OfirPress

a month ago

.@_carlosejimenez updated the SWE-bench [Bash only] leaderboard with Qwen3 numbers. Congrats to the team on the great results! Note that these numbers are about 10% lower than the max numbers achievable by each model since we don't allow tools in this leaderboard.

Alibaba_Qwen Profile Picture

Qwen @Alibaba_Qwen

2 months ago

.@_carlosejimenez updated the SWE-bench [Bash only] leaderboard with Qwen3 numbers. Congrats to the team on the great results! Note that these numbers are about 10% lower than the max numbers achievable by each model since we don't allow tools in this leaderboard. https://t.co/oQOnajNjFw

carlos retweeted
KLieret Profile Picture

Kilian Lieret @KLieret

a month ago

Releasing mini, a radically simple SWE-agent: 100 lines of code, 0 special tools, and gets 65% on SWE-bench verified! Made for benchmarking, fine-tuning, RL, or just for use from your terminal. It’s open source, simple to hack, and compatible with any LM! Link in 🧵

carlos retweeted
OfirPress Profile Picture

Ofir Press @OfirPress

2 months ago

AGI

carlos retweeted
OfirPress Profile Picture

Ofir Press @OfirPress

2 months ago

LMs had a really tough time playing real video games from the 90s- so we made a suite of 3 simple games to test specific abilities, including drag-and-dropping, and navigating a maze using the arrow keys. Even on these *extremely* simple games, most frontier LMs fail. Results-->

a1zhang Profile Picture

Alex Zhang @a1zhang

2 months ago

LMs had a really tough time playing real video games from the 90s- so we made a suite of 3 simple games to test specific abilities, including drag-and-dropping, and navigating a maze using the arrow keys. Even on these *extremely* simple games, most frontier LMs fail. Results-->

carlos retweeted
SWEbench Profile Picture

SWE-bench @SWEbench

2 months ago

SWE-agent is now Multimodal! 😎 We're releasing SWE-agent Multimodal, with image-viewing abilities and a full web browser for debugging front-ends. Evaluate your LMs on SWE-bench Multimodal or use it yourself for front-end dev. 🔗➡️

carlos retweeted
AbramovichTalor Profile Picture

Talor Abramovich @AbramovichTalor

2 months ago

Join me next week at #ICML25, where I will be presenting my first first-author paper –– EnIGMA. EnIGMA, an LM agent for cybersecurity, uses interactive tools for server connection and debugging, achieving state-of-the-art on 3 CTF benchmarks. youtube.com/watch?v=50zkWJ…

carlos retweeted
rajko_rad Profile Picture

Rajko Radovanović @rajko_rad

2 months ago

We @a16z just launched the third batch of Open Source AI Grants (cc @mbornstein) 🎉 This round includes projects focused on LLM evaluation, novel reasoning tests, infrastructure, and experimental research at the edge of capability and cognition: • SGLang: High-performance LLM…

Trends for United States

GiveRep V1
1.179 posts
Karen
90,8 B posts
Good Saturday
18 B posts
Rachel
77,8 B posts
Chiefs
205 B posts
FKJD FANDAY24 SG
156 B posts
#SaturdayVibes
1.669 posts
Chargers
122 B posts
Ken Dryden
4.950 posts
AJ Lee
262 B posts
Mahomes
69,1 B posts
Herbert
58,6 B posts
#MarvelTokonFightingSouls
#OverstockIsBack
Big Brother
25 B posts
Bader
16,6 B posts
Kelce
42,1 B posts
Worthy
56,3 B posts
Jarritos
1.249 posts
Rose Garden
29,7 B posts

You might like

Computer Science PhD student @Princeton University working in computer vision | hummus enthusiast
Allison Chen
@allisonchen_227
259 Followers
Research Scientist at @Meta Fundamental AI Research (FAIR), New York. Previously: Postdoc @Caltech, PhD @PrincetonCS, Undergrad @Tsinghua_Uni.
Kaiyu Yang
@KaiyuYang4
4K Followers
@OpenAI Language agents (ReAct, Reflexion, Tree of Thoughts, SWE-agent, CoALA) for digital automation (WebShop, SWE-bench, tau-bench)
Shunyu Yao
@ShunyuYao12
19K Followers
🌲 CS PhD @Stanford 🤖 SWE-bench + agent + smith 🎓 Prev. @princeton_nlp 🐯; @Berkeley_EECS 🐻
John Yang
@jyangballin
4K Followers
CS PhD student at Princeton
Sadhika Malladi
@SadhikaMalladi
1K Followers
CS PhD student @Princeton @Princeton_nlp @PrincetonPLI working on language models. Previously: @Tsinghua_Uni @TsinghuaNLP
Tianyu Gao
@gaotianyu1350
5K Followers
PhD student @uwnlp @allen_ai | Prev @MetaAI @CS_UCLA | 🏠 https://t.co/Q6Mzg8ow2j
Weijia Shi
@WeijiaShi2
9K Followers
PhD @Princeton trying to make sense of language models and their training data; trying to train agents @cursor_ai
Alex Wettig
@_awettig
2K Followers
Associate Professor @UCLAengineering/@UCLA. Area: #NLProc/#ML/#AI https://t.co/zj1ssZj9ox
Kai-Wei Chang
@kaiwei_chang
8K Followers
@xAI post-trained Grok 3&4; scaling up RL for Grok-next | prev @PrincetonCS
Zexuan Zhong
@ZexuanZhong
3K Followers
Research @OpenAI, PhD @princeton_nlp.
Mengzhou Xia
@xiamengzhou
4K Followers
PhD @PrincetonCS. Previously CS & Eng. @UCIrvine. Studying AI, ML, RL, NLP.
Jens Tuyls
@JensTuyls
816 Followers
PhDing @PrincetonPLI. Machine memory / control / agency.
Howard Chen
@__howardchen
1K Followers
Assistant Prof. @CSatUSC, @USC || Postdoc @ClipUMD || PhD from @UCLANLP, @UCLA. #NLP, #ML, #TrustworthyNLP
Jieyu Zhao
@jieyuzhao11
3K Followers
Amazon Fellow, PhD @nlp_usc, Research Intern @meta GenAI | MSE '22 @Princeton @SiebelScholars '22 | @Wellesley '20 | Prev. @ai2_mosaic @Apple AI/ML
Huihan Li
@huihan_li
562 Followers