Modulo Research is developing specialized datasets to help advance empirical alignment research in artificial intelligence. These datasets will not only serve as the foundation for our own follow-up studies but will also be made available to the broader AI safety research community.<\/p>\n\n\n\n
You can sign up<\/a> to be notified when we release future datasets.<\/p>\n\n\n\n Currently Available<\/strong><\/p>\n\n\n\n FindTheFlaws<\/strong> is a set of datasets that include (1) long-form expert-verified correct solutions and (2) long-form flawed solutions with annotations highlighting specific errors to difficult questions in medicine, physics, chemistry and more. While several of the questions are drawn from existing benchmarks such as GPQA Diamond, it also includes the novel CELS dataset containing detailed expert annotations of LLM responses to difficult questions in surgical medicine, law, and Lojban. The repository<\/a> <\/strong>includes the datasets presented in the paper<\/a>, and the scripts used to conduct model evals using UK AISI’s Inspect library.<\/p>\n\n\n\n <\/p>\n\n\n\n <\/p>\n","protected":false},"excerpt":{"rendered":" Modulo Research is developing specialized datasets to help advance empirical alignment research in artificial intelligence. These datasets will not only serve as the foundation for our own follow-up studies but will also be made available to the broader AI safety research community. You can sign up to be notified when we release future datasets. Currently Available FindTheFlaws […]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"open","template":"","meta":{"_themeisle_gutenberg_block_has_review":false,"footnotes":""},"class_list":["post-2","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/www.moduloresearch.com\/index.php\/wp-json\/wp\/v2\/pages\/2","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.moduloresearch.com\/index.php\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.moduloresearch.com\/index.php\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.moduloresearch.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.moduloresearch.com\/index.php\/wp-json\/wp\/v2\/comments?post=2"}],"version-history":[{"count":10,"href":"https:\/\/www.moduloresearch.com\/index.php\/wp-json\/wp\/v2\/pages\/2\/revisions"}],"predecessor-version":[{"id":118,"href":"https:\/\/www.moduloresearch.com\/index.php\/wp-json\/wp\/v2\/pages\/2\/revisions\/118"}],"wp:attachment":[{"href":"https:\/\/www.moduloresearch.com\/index.php\/wp-json\/wp\/v2\/media?parent=2"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
Datasets<\/strong> Under Development<\/strong>
We’re now finalizing a dataset of textual representations of the research processes followed by high-performing participants in an experiment involving an online research task \u2014 which we hope could be useful for improving LLM capability elicitations \u2014 and have written up the results of the\u00a0experiment<\/a>\u00a0in which they were collected.<\/p>\n\n\n\n