A Chinese company has introduced a dynamic range of AI performance standards.

Development of the benchmark at Hongshan began in 2022, following the significant success of ChatGPT, as an internal tool for assessing which models merit investment. Since then, led by partner Gong Yuan, the team has steadily expanded the system, collaborating with outside researchers and professionals to help refine it. As the project grew more sophisticated, they opted to release it to the public.

Xbench approached the problem with two distinct systems. One resembles traditional benchmarking: an academic test that evaluates a model’s proficiency across various subjects. The other mimics a technical interview round, assessing the real-world economic value a model may deliver.

Xbench’s methods for assessing raw intelligence currently comprise two components: Xbench-ScienceQA and Xbench-DeepResearch. ScienceQA is not a radical shift from existing postgraduate-level STEM benchmarks like GPQA and SuperGPQA. It features questions covering fields from biochemistry to orbital mechanics, crafted by graduate students and verified by professors. Scoring accounts for not only the correct answer but also the reasoning process that leads to it.

DeepResearch, on the other hand, emphasizes a model’s capability to navigate the Chinese-language web. Ten subject-matter experts devised 100 questions in music, history, finance, and literature—questions that cannot be easily searched online but require thorough research to answer. Scoring values the breadth of sources, factual consistency, and a model’s acknowledgment of insufficient data when applicable. An example from the collection is, “How many Chinese cities in the three northwestern provinces border a foreign country?” (The answer is 12, with only 33% of tested models answering correctly.)

Researchers expressed a desire to add further dimensions to the test—such as evaluating how creative a model is in problem-solving, how well it collaborates with other models, and its reliability. The team has committed to refining the test questions quarterly while maintaining a balanced data set.

To evaluate models’ readiness for real-world applications, the team worked with experts to create tasks modeled on actual workflows, starting with recruitment and marketing. For instance, one task requires a model to source five qualified battery engineer candidates and justify each selection. Another task asks it to match advertisers with suitable short-video creators from a pool of over 800 influencers.

Additionally, there are hints of upcoming categories, including finance, legal, accounting, and design, with question sets for these areas yet to be shared publicly.

Currently, ChatGPT-o3 ranks first in both of the existing professional categories. For recruiting, Perplexity Search and Claude 3.5 Sonnet take second and third place, respectively. In marketing, Claude, Grok, and Gemini demonstrate strong performance.

“It is really difficult for benchmarks to encapsulate elements that are hard to quantify,” remarks Zihan Zheng, the lead researcher on a new benchmark called LiveCodeBench Pro and a student at NYU. “But Xbench represents a promising start.”