The escalating complexity of neural networks, exemplified by models like GPT-4, has magnified the difficulty in comprehending their intricate workings. With the increasing sophistication and size of these models, conventional human oversight alone is insufficient to grasp their behavior. To tackle this challenge head-on, researchers from MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) have embarked on a groundbreaking initiative. They've introduced an innovative approach that leverages AI models to conduct experiments on diverse systems, shedding light on their behavior in a manner that surpasses traditional human-centric methodologies.

 

At the core of this pioneering strategy is the "automated interpretability agent" (AIA), designed to emulate a scientist's experimental processes. These AI agents plan and execute tests on various computational systems, spanning from individual neurons to entire models, producing explanations in diverse forms. Unlike existing interpretability procedures that passively classify or summarize examples, the AIA actively engages in hypothesis formation, experimental testing, and iterative learning, refining its understanding of other systems in real time.

 

The researchers at CSAIL have introduced the "function interpretation and description" (FIND) benchmark, a pivotal component in their approach. FIND serves as a test bed for functions resembling computations inside trained networks, accompanied by detailed descriptions of their behavior. A significant challenge in evaluating the quality of descriptions of real-world network components has been the lack of ground-truth labels or descriptions of learned computations. FIND addresses this issue by providing a reliable standard for evaluating interpretability procedures, enabling comparisons of AIAs with other methods in the literature.

 

For instance, FIND incorporates synthetic neurons mimicking the behavior of real neurons within language models, including selectivity for specific concepts such as "ground transportation." AIAs are granted black-box access to these synthetic neurons and design inputs to test their responses. Through autonomous hypothesis generation and testing, AIAs can uncover behaviors that might be challenging for human scientists to detect. The benchmark then serves as a yardstick to evaluate the capabilities of AIAs against established methods.

 

Sarah Schwettmann, a co-lead author of the paper and a research scientist at CSAIL, underscores the advantages of this approach. She notes, "The AIAs' capacity for autonomous hypothesis generation and testing may be able to surface behaviors that would otherwise be difficult for scientists to detect. It's remarkable that language models, when equipped with tools for probing other systems, are capable of this type of experimental design."

As language models continue to captivate the tech world, CSAIL's team recognizes their potential to serve as the backbone for generalized agents in automated interpretability. Schwettmann emphasizes the multifaceted nature of interpretability, acknowledging that there is no one-size-fits-all approach. However, the AIAs, built from language models, could provide a general interface for explaining other systems, synthesizing results across experiments, and even discovering new experimental techniques. To evaluate the effectiveness of AIAs and existing interpretability methods, the researchers introduced an innovative evaluation protocol along with the FIND benchmark. This protocol involves direct comparisons between AI-generated estimations and original, ground-truth functions for tasks requiring code replication. For tasks involving natural language descriptions of functions, a specialized "third-party" language model is trained to assess the accuracy and coherence of AI-generated descriptions compared to ground-truth function behavior.

 

Despite the promising progress made by AIAs, the evaluation using FIND reveals that fully automating interpretability remains a challenge. While AIAs outperform existing methods, they still struggle to accurately describe almost half of the functions in the benchmark. Tamar Rott Shaham, co-lead author of the study, points out that AIAs, while effective in describing high-level functionality, often overlook finer-grained details, particularly in function subdomains with noise or irregular behavior. Addressing this issue, the researchers experimented with guiding the AIAs' exploration by initializing their search with specific, relevant inputs, resulting in enhanced interpretation accuracy.

 

Looking ahead, the CSAIL team envisions developing nearly autonomous AIAs capable of auditing other systems, with human scientists providing oversight and guidance. Advanced AIAs could potentially generate new kinds of experiments and questions beyond human scientists' initial considerations. The goal is to expand AI interpretability to include more complex behaviors, such as entire neural circuits or subnetworks, and predict inputs that might lead to undesired behaviors. The work from MIT's CSAIL marks a significant step forward in the realm of AI research, striving to make AI systems more understandable and reliable. As the field continues to evolve, the combination of automated interpretability agents and carefully designed benchmarks like FIND holds the promise of unraveling the mysteries hidden within the complex neural networks of advanced AI models.