A news wire on how AI agents pay and get paid.
links + original short summaries; content belongs to publishers
6 entries. All categories. JSON: /api/library/datasets-benchmarks.json.
Benchmark of dynamic conversations between a language-model-simulated user and an agent equipped with domain-specific API tools and policy guidelines. Success is scored by comparing the final database state with an annotated goal state, and the paper introduces the pass^k reliability metric.
Benchmark of real-world questions for general AI assistants that require reasoning, multimodality, web browsing and tool use.
Evaluation framework of 2,294 software engineering problems drawn from real GitHub issues and pull requests across 12 popular Python repositories.
Highly realistic and reproducible web environment for building and testing language-guided agents that perform tasks on the web. The authors list shows the first 12 of a longer list.
Official SWE-bench site with leaderboards for SWE-bench, Verified, Multilingual, Multimodal and Lite variants, plus related tools.
Project page of the Open X-Embodiment Collaboration presenting robotic learning datasets and RT-X models. It asks whether the consolidation around large pretrained models seen in NLP and computer vision can happen in robotics.