[INTERVIEW] The provisions governing Foundation Models in the upcoming AI Act: questions to Rishi Bommasani

1 December 2023

By Giulia Geneletti

In this second interview on the EU AI Act trilogues and the specific provisions covering Foundation Models (find the first one here), Giulia Geneletti interviews Rishi Bommasani, Society Lead at the Stanford Center for Research on Foundation Models (CRFM) and PhD Researcher at Stanford Computer Science.

The interview explores Rishi’s insights on the ongoing EU AI Act policy process coming from a transatlantic perspective, the concern of balancing responsibilities and liability across developers and downstream deployers, the need and meaning of transparency requirements and a comparative analysis of the EU AI Act and Biden’s Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence.

Let’s start with a framing overview. What is your general assessment on how the EU AI Act trilogues are attempting to regulate Fondation Models?

I see regulation of Foundation Models as mandatory for a well-designed AI Act: foundation models are at the epicenter of AI, the very basis for public awareness of AI, and are pivotal to the future of AI. Failing to regulate Foundation Models is not an option under any sensible regulatory approach.

Despite the general political consensus and momentum on covering FMs in the EU AI Act (with exception made for the recent Franco-German-Italian proposal), the concrete provisions are still highly debated. Do you think FMs should be included in the risk-framework approach designed by the EU AI Act? Should FMs be covered by horizontal requirements outside of the risk-framework?

I don’t think at this stage that trying to shoehorn FMs into the risk framework makes sense, because the principle for introducing FMs is they diverge from the application-centric approach of determining risk. As Kai Zenner, Head of Office and Digital Policy Adviser of MEP Axel Voss, put it, foundation models have the potential to be deployed for countless intended purposes, many not initially foreseen in development. Thus, the standard product safety approach the EU AI Act is based on cannot be applied to foundation models.[1]

In turn, I see the best compromise at the moment as transparency requirements for foundation models, with additional burdens for models that are high-impact and that could precipitate systemic risk.

Some relevant metrics to define high-impact of FMs would include: [2]

The number of customers or entities paying for the foundation model – The latest Council mandate from Nov 28 mentions 10k business users as an example of this.
The number of downstream AI applications.
The number of downstream high-risk AI systems.
The aggregate number of users across all downstream AI systems.
The aggregate number of users across all downstream high-risk AI systems.
The number of high-risk areas (e.g. categories listed under Annex III of the EU AI Act [3]) covered by downstream high-risk AI systems.
The number of queries to the foundation model.

Seeing the complexity of defining, allocating and implementing effective provisions that would differ according to the size of the FMs and their downstream applications, how to strike the right and safest balance of responsibility/liability between developers of AI systems and downstream application deployers?

Placing all burdens on downstream application deployers is myopic: foundation models play an outsized role in the AI value chain and their developers often wield asymmetric bargaining power to shape contracts.

I think the key is there must be a balance. Having all burdens under the AI Act shouldered by those downstream is not prudent: many of the key levers to address risks are at the foundation model level. To get to a virtuous state, information-sharing and transparency along the value chain is essential: having those downstream operating blind of the models they are building upon is unequivocally a bad outcome.

What about Open Source and FMs release strategy?

The EU should consider release strategy and, specifically, the interests of open model developers in arriving at a compromise. However, my sense is the Act is trending towards a state where the requirements for all foundation models would be the very transparency requirements that open model developers support. For example, the Open Future Foundation – Amsterdam-based Organization advocating for openness in the European Union’s digital policy framework – has been labeling self-regulation as ineffective and stressing that the lack of mandatory regulation of FMs would be detrimental both to open-source approaches to AI and society as a whole. According to the Foundation, a proportionate and fair approach for Open-Source AI would require a tiered system where basic obligations like transparency and documentation would apply to all FMs (training methods, data used, and resource consumption), while more extensive obligations would be reserved for larger-scale, commercially operated models. [4]

The Foundation Model Transparency Index developed by your institution, Stanford Center for Research on Foundation Models, relieved that no major FMs developer is currently close to providing adequate transparency as required by the EU AI Act, revealing a fundamental lack of transparency in the AI industry.

Many AI providers and downstream developers (especially SMEs) accuse the AI Act of requiring highly burdensome compliance. How concretely and technically hard do you consider this compliance to be? What do you believe are the biggest challenges for developers/deployers when it comes to implementation and to building new systems of accountability, transparency and trustworthy governance?

I see two areas where transparency is vital, because the current status quo is such striking opacity.

The first is on data labor: model developers consistently disclose little, if anything, about those whose labor creates the vital data for building their models (e.g. workers who constitute the Human element in Reinforcement Learning from Human Feedback (RLHF) [5] or who concretely conduct content moderation). Reporting by the likes of Karen Hao [6] and Billy Perrigo [7] has confirmed the poor worker conditions at present. If we are to ensure companies are held accountable and treat workers with appropriate dignity, transparency is an obvious precondition.
The second is on downstream impact: foundation models, much like social media platforms, are intermediary technologies. Foundation models power myriad applications. In turn, the fact we know astonishingly little about how even the most widely used foundation models (e.g. GPT-4, DALL-E 3, Claude 2, Llama 2, PaLM 2) are used is very concerning. In particular, if we knew the extent to which these models were used for entertainment, for rote tasks, for medicine, for relationship advice, or so forth, we would be able to better envisage sound policy.

Both matters show that foundation models orient fledgling supply chains: modern society is predicated on complex supply chains that warrant transparency at a minimum.

On the effective burden of compliance – my view is the contrast here should be understood as comparing (i) the compliance burden of downstream entities, including SMEs given the exclusion of FM-related transparency requirements vs. (ii) the compliance burden of the same entities plus the compliance burden on FM developers given the inclusion of FM-related transparency requirements. In my judgment, FM-related transparency requirements will impose burden on FM developers but reduce burden on downstream developers.

Consequently, I see placing burdens on FM developers as sensible and the appropriate distribution of responsibility along the value chain. I would be very glad to see FM developers like Mistral – that claim compliance burdens are too onerous – to be precise: which specific requirements are too burdensome, how are they estimating the cost of compliance with those requirements, and why can they not bear these costs?

I do not see significant underestimation on the side of policymakers: my experience is that all the key negotiators in the Council, Commission, and Parliament have a remarkably clear understanding of what the costs and benefits of transparency/disclosure for FMs are.

Finally, in our work analyzing the Parliament position in June or our Transparency Index in October (see above), we paint a clear picture that companies are not uniformly transparent on various matters, with ample room for improvement. However, this is precisely the impetus for legislative action: regulation can set the minimum standards for transparency. Especially against a backdrop of transparency decreasing over time (the ecosystem was much more transparent in 2018 or 2020 than in 2023), action from the EU could correct the status quo, better reflecting how this technology should advance the public interest and not just the commercial interest of its developers. As Commissioner Breton put forcefully on current lobbying: “There is no longer any point in giving a voice to those who defend their particular interests at European level, as we listened to them a lot in the preparatory phase. Now, we are going to bring the general interest to bear”. My hope is his colleagues and countrymen in parts of the Council will look in the mirror and remind themselves of this.

Taking now into consideration both sides of the Atlantic, it’s very interesting to compare the efforts undertaken by the EU and the US Administration in delivering policy solutions around a political moment of discussion on AI Governance. Biden’s Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence of October 2023 spurred analysis considering American efforts more intentional than we have experienced in the past in addressing technological harms and concentration of powers. What is your view on the EO and how do the two texts impact the ongoing process and discussions on the global governance of FMs?

I have contributed to several analyses of the EO at this point: [Full Analysis] [Key Statistics] [Open Models] [Implementation Tracker].

Overall, the key point is the Executive Order does not do much to shape the current status quo on foundation models, perhaps barring some protein-related models, given its impacts are felt by models exceeding 10^26 FLOPs (Floating Point Operations Per Second – a measure of a computer’s performance, especially in the context of tasks that require a lot of numerical calculations. This gives an idea of the computational intensity and the technological capability needed to train advanced AI models.). That is, notably, 5x the amount of compute GPT-4 is estimated to have been trained on. So, I see the EO as mainly playing a signaling role, and being the basis for information-gathering (e.g. The US Department of Commerce will prepare reports on both content provenance and open models in the coming months). This is very different from the AI Act, which is obviously legislation that aims to more directly shape the landscape of AI.

I think global governance needs both mechanisms over time. Executive action as in the EO helps provide pathways and move quickly, but ultimately legislative backing is what will provide the enduring stability for society to navigate this new technology.

What are the scenarios of harms and risks coming from FMs that worry you the most? Do you believe upcoming legislation will be able to fully cover them on time for the alerts rising for the great number of elections upcoming in 2024?

2024 is of course the year of elections: if I recall correctly, 2 billion hit the polls including 9 of the 10 most populous countries. Therefore, disinformation is naturally a significant concern. I do not expect any current policy proposal or approach to fully insulate us from having disinformation of various forms influence elections. In fact, the current excitement over various watermarking approaches, which are very technically nascent, may lull us into false security. If we are really to address AI-generated disinformation, the best route forward would entail robust coordination between the creators of generative AI and platforms that would disseminate this AI-generated content.

Beyond that, I think we perpetually misunderstand how digital technologies transform our lives. If we think of the Internet and social media, sometimes there are problems of malicious use. For example, how data sharing errors yielded the Cambridge Analytica scandal. Or sometimes the risks are of day-to-day issues at scale, for example the soaring concerns of mental health especially among young women that are exacerbated by modern social media. Today, we see both classes of concerns manifesting in AI discourse: some worry about catastrophic misuse of AI, while others lament the lack of progress on well-established problems of bias and privacy. But to me, the nature of these generational technologies is that simply life before and after them is different. One could try to write out all the pros and all the cons of the Internet, but the basic fact is life is different. I worry we have not thought through how our society will be different, and that even for us who do we won’t anticipate all the shifts. For example, significant economic shifts at all scales: geopolitical macroeconomic power to individual job experiences.

Finally, I also do worry about market concentration as a specific concern. The salient challenge of the last 20 years in technology policy is that little, if anything, can hold Big Tech to account. We should be very deliberate to not intensify the concentration of power.

Footnotes

[1] Zenner Kai. OECD AI Policy Observatory. “A law for foundation models: the EU AI Act can improve regulation for fairer competition”. (July 2023). https://oecd.ai/en/wonk/foundation-models-eu-ai-act-fairer-competition

[2] See more here: Bommasani Rishi. Stanford HAI & Stanford Center for Research on Foundation Models. “Drawing Lines: Tiers for Foundation Models“ (November 2023). https://crfm.stanford.edu/2023/11/18/tiers.html

[3] Annexes to the Proposal for a Regulation of the European Parliament and of the Council Laying Down Harmonized Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts. (April 2021). https://eur-lex.europa.eu/resource.html?uri=cellar:e0649735-a372-11eb-9585-01aa75ed71a1.0001.02/DOC_2&format=PDF

[4] Keller Paul, Warso Zuzanna. Open Future Foundation. “THE EU SHOULD NOT TRUST AI COMPANIES TO SELF-REGULATE”. (November 2023). https://openfuture.eu/blog/the-eu-should-not-trust-ai-companies-to-self-regulate/

[5] Reinforcement Learning from Human Feedback is a sophisticated approach to train AI models (namely advanced language models like OpenAI’s GPT-4) that integrates human feedback into the Reinforcement Learning process, a type of machine learning where an agent (like a robot or a software) learns to make decisions by performing actions in an environment (experimentation) and receiving feedback in the form of rewards or penalties. RLHF helps in aligning AI behavior more closely with human values and expectations and it is especially useful in scenarios where defining a clear, objective reward function is difficult.

[6] Hao Karen, Seetharaman Deepa. The Wall Street Journal. “Cleaning Up ChatGPT Takes Heavy Toll on Human Workers”. (July 2023). https://www.wsj.com/articles/chatgpt-openai-content-abusive-sexually-explicit-harassment-kenya-workers-on-human-workers-cf191483

[7] Perrigo Billy. Time. “Exclusive: OpenAI Used Kenyan Workers on Less Than $2 Per Hour to Make ChatGPT Less Toxic”. (January 2023). https://time.com/6247678/openai-chatgpt-kenya-workers/

Rishi Bommasani – Society Lead at the Stanford Center for Research on Foundation Models (CRFM) and PhD Researcher at Stanford Computer Science

Giulia Geneletti – Research Assistant at Sciences Po Chair Digital, Governance and Sovereignty