# SERI MATS Application - Georgios Kaklamanos

**Interdisciplinary AI safety - Problem 1**

> Read and summarize “Pragmatic AI Safety” three, on complex systems, and four, on capabilities externalities. Bonus points if you provide substantive discussion or disagreements with some points.

## Complex Systems for AI Safety [PAIS#3]

### Introduction

In this post, the authors present a "Systems Approach" to AI Safety. After they introduce the field of Complex Systems, they provide an overview of the general factors that contribute to a system (e.g. systemic/sociotechnical factors, safety culture, personalities of top AI researchers, etc). Finally, they argue that AI Safety can be modeled as a complex system, present some insights on what we can get from it, and close by making the case for more diversification on the agendas and our interventions.

### Complex Systems

Complex Systems consist of many components whose interactions result in emerging behaviors of the collective, that can't be predicted by analyzing each component individually. They are too organized to model them with statistics and their dependencies brake a lot of statistical assumptions.

On the safety aspect, instead of focusing on one system and trying to think how to make that safer, we need to abstract, study multiple systems and identify what made them safer. Applying this to AI, we shouldn't think of AI Safety independently. We need to move one level of abstraction up, think of the whole AI Safety ecosystem, and focus on how to make that safer.

### Improving Contributing Factors

We can impact a system through direct or indirect interventions. Direct interventions are produced by causal, short, and deterministic paths. They are intentional, easy to quantify and produce quick results. Indirect impact involves secondary effects, non-deterministic factors, late feedback, and uncertainty of the results. In complex systems, we should account for and focus on indirect interventions, since they could have a significant impact (e.g. AI Safety field building -> Increase in resources -> More researchers -> more problems solved). Still, we should be careful with theories of the impact that are less related to x-risk, because they might be prone to Goodharting.

Systemic factors that tend to increase AI-Safety are improved epistemics, proactivity about tails risks, expanded moral circles and keeping misaligned / malevolent actors out of power. Similarly, the authors list some sociotechnical factors that seem relevant to make high-risk technological systems safer. Examples of these are rules and regulations, social/productivity/competition pressure, reduction in inspection and preventative maintenance, building defense in depth, and reducing alarm fatigue, etc. The most important factor however is improving the "Safety Culture".

Establishing a safety would make it easier for researchers to accept and formalize arguments about risk, without having the discussions backfire or suffer social costs. Safety culture could be enhanced by thinking more about failure (especially black swan events) and a reluctance to oversimplify interpretations. Additional approaches would be committing to resilience and organizational structures that allow fast travel of information. Nevertheless, we should be careful of how quickly we'd try to shift the Overton window.

Another key factor is the characteristics of top AI researchers. It seems highly probable that directions of future advanced AI systems would be set by a very small number of the top-AI researchers, and most of them are not sympathetic to AI Safety. While training more safety-conscious people to be top researchers might be easier, we still need to increase the "buy-in" among existing ones, especially including researchers in China.

Finally, some other causes contribute to the neglectedness of the topic. Corporations focus on short-term goals and due to techno-optimism, there is an inherent distaste for discussing the risks. In the political sphere, AI Safety competes with more popular causes like Climate Change, and the lack of technical background makes it even harder to raise and understand those issues.

### Complex Systems for AI Safety

Deep learning exhibits many hallmarks of complex systems: highly distributed functions, numerous nonlinear connections, self-organization, adaptivity, feedback loops, scalable structures, and emergent capabilities. Due to that, we can use complex systems not only as explanatory but also as predictive models for various problems. From the \`\`Systems Bible'', we know that systems develop goals of their own the instant they come into being (e.g. instrumental convergence thesis) and intrasystem goals come first (e.g. mesa optimization). The crucial variables would be discovered by accident, and the failure modes can't be predicted from its structure. Since humans are unreliable, any system that depends on humans would also be unreliable. Still, since a complex system that works is invariably found to have evolved from a simple system that works, we should view this as a top priority to focus on making today's simpler systems safer.

### Diversification

We should have multiple approaches to the problem, not just one. Safety research failed to do this in the past when it focused on RL and neglected DL. Since AI Safety is an area with high uncertainty, it is most important to improve the virtues of the system. Diversification however doesn't mean that we shouldn't be discretionary about ideas and individuals do not necessarily need to have a diverse portfolio.

The impact is long-tailed and in a diverse portfolio, the vast majority of the impact will likely be dominated by a few grants. Since the tails aren't known, we shouldn't focus on targeted interventions in the tails, but instead, have broad interventions that have a sufficient chance of being in the tails.

### Review / Thoughts on [PAIS#3]

#### For the Sociotechnical factors section:
- The authors mention: "The following sociotechnical factors (compiled from Perrow, La Porte, Leveson, and others) tend to influence hazards:". However, they leave unclear if they influence positively or negatively.
- e.g. for the "Incentive structures within the organization, such as benefits to delivering quickly or retaliation for whistleblowing."
- This can backfire, as people would fear retaliation when they do something wrong so they'd avoid reporting it.
- Also pressures to deliver might lead people to "cut corners" on security procedures as it already happens today in many software development teams.

#### Productivity pressures, or pressure to deliver quickly.

- Past research focusing on functional fixedness on a participant's problem-solving capabilities ([e.g. candle problem](https://en.wikipedia.org/wiki/Candle_problem)), has shown that strong incentives and pressure can increase productivity and results on problems with known solutions. However when we're dealing with unknown problems that require more "creative" solution, such an appoach is counter-productive and actually decresases productivity.
- This is also the argument presented in the [TED Talk The puzzle of motivation, by Dan Pink](https://www.youtube.com/watch?v=rrkrvAUbU9Y).

#### On Improving Safety Culture:

- Although I agree with the approach of focusing on the culture of the system (it is also the most impactful leverage point on [Donella Meadows classification](https://www.donellameadows.org/wp-content/userfiles/Leverage_Points.pdf)), the authors of this paper miss some necessary parameters that need to be enabled before such a culture becomes widespread.
- The first is to create a culture of trust and remove the "blame culture".
- At the [DevOps Handbook](https://www.goodreads.com/book/show/26083308-the-devops-handbook), they present the three ways of DevOps. The third way focuses on the principles of continual learning", and eliminating the blame culture is one of the top priorities.
- DevOps and specifically Ops, had to deal with building and administering complex and distributed systems for several years, so there could be a lot of practices that could be imported into the ML Safety community (e.g. recommendations on monitoring techniques, [Chaos Engineering](https://principlesofchaos.org/), etc).
- Specifically, removing the culture of blame means that when (an inevitable failure) happens, we shouldn't focus on who to blame, but on how that happened and how to prevent future occurrences.
- The second is the requirement that people become more familiar with and experience failure themselves.
- Although failures in the research process are quite common, people don't talk openly about it. [Uri Alon is tries to change this](https://www.youtube.com/watch?v=F1U26PLiXjM), by being open on all the ways his projects failed, and changed directions, until he managed to make a significant discovery, but it's not enough.
- In a culture where mistakes and failures are stigmatized, people who notice initial signs of their system failing, might try to hide it in fear of associating the failure of the system with personal mistake. Thus failures would remain unseen and accumulate, until the system collapses.

## Perform Tractable Research While Avoiding Capabilities Externalities [PAIS#3]

In this post the authors focus on two essential properties of important research: tractably produce tail impact and avoid creating capabilities externalities.

### Strategies for Tail Impact
#### Processes that generate long tails and step changes

To make serious progress, researchers should focus on maximizing their probability of being in the tail of research ability. Some processes that generate tail impacts are: multiplicative processes, preferential attachment, and the edge of chaos.

##### Multiplicative processes

When we're dealing with variables whose values are normally distributed, and the process is additive, we operate on Mediocristan. However, when there are nonlinearities or multiplicative processes, we operate in Extremistan: outcomes will be dominated by combinations of variables. This also means that if one variable is close to zero the total output is also very small. This mindset can also be used to select aptitude of individual researchers, but also the combinations of teams.

##### Preferential Attachment

The Matthew Effect states that to those who have, more will be given. This implies that researchers need to be acutely aware that it helps a lot to do very well early in their careers if they want to succeed later.

##### Edge of Chaos

The edge of chaos is the space between an ordered area and a chaotic area. This is the zone of proximal development (and research). We avoid total chaos where there is no tractability and solutions that are impossible to find, but also avoid complete order where there is no progress to be done.

Easy access to edge of chaos is by designing some metrics, or keeping a list of open projects that don't work now, but might work in the future.

#### Managing Moments of Peril

In moments of peril and turmoil, it the chances that people would have extreme reactions increase. We should be ready for such events and make sure that the safety ideas that we'll present would be as simple and as time-tested as possible.

#### Getting in early

Thinking about safety from early on, increases the chances that it is implemented. Also early actions reduce the costs, since there won't be any need to change existing systems. We could also acrue more benefits by compounding effects and stable trends.

#### Scaling laws

One objective of AI safety research should be to improve scaling laws of safety relative to capabilities. For safety metrics, we need to move as far along the scaling law as possible, which requires researchers and sustained effort. However the most important factor are ideas.

#### Don't let perfect be the enemy of good

We should accept that we will not get things perfect, so we should be content with good enough.

### Problems with asymptotic reasoning

#### Goodhart's law

While many people are using Goodgart's law (especially in EA / LW communities), it very often is misinterpreted and presented in a much stronger version.

##### Counteracting forces

In dynamical systems, a lot of times there are counteracting forces that help balance a situation. So we could also try and build counter-measures to Goodhart's law. Similarly to increase safety we could build systems such Some examples of counteracting systems include artificial consciences, AI watchdogs, lie detectors, filters for power-seeking actions, and separate reward models.

##### Rules vs Standards

Rules are clear, objective, and knowable beforehand. Standards are complex, sometimes vague, and require to more case-by-case interpretation. DL systems can model standards, so we could use intelligent systems to evaluate other intelligent systems.

##### Goal refinement

Goodhart's law applies to proxies, not terminal values. So as our ability to optimize and refine our goals increase, the approximation error would decrease.

#### Limitations of research based on hypothetical superintelligence

Research focusing on hypothetical superintelligence shouldn't be the only agenda since it encourages work in areas that are less tractable, the assumption of superintelligence eliminates an entire class of interventions that might be needed, third ti assumes that there would be a single superintelligence, and finally it ignores sociotechnical factors.

#### Instead Improve cost/benefit variables

And alternative to the previous approach is to try and make incremental steps and focus on cost/benefit analysis.

### Safety / Capabilities tradeoff

Safety and capabilities are linked. A more capable system might be able to understand what humans consider harmful, but also have greater ability to cause harm. While it isn't always easy to do, researchers should focus on improving safety, without resulting on externalities that improve general capabilities.

Machine ethics should be preferred to tasks preferences and safety should achieved through evolution, rather than only trying to arrive at safety through intelligent design.

### Review / Thoughts on [PAIS#3]

#### On Multiplicative Processes

- Strongly agree on considering multiple factors for the selection of researchers. As the problems start to span multiple fields the ability to navigate in those differences becomes quite important.

#### Preferential Attachment

- The Halo Effect is also in effect to a number of these situations.

#### Edge of Chaos

- Zone of proximal development: This is practically the state of [Flow](https://en.wikipedia.org/wiki/Flow_(psychology)).

#### Don't let perfect be the enemy of good

- While I do understand this sentiment (and it's the repeated message) from a tutor in a project management course I'm attending, it is my impression that a lot of people don't even go to good, but remain on the MVP. Thus we sacrifice quality for quantity.

#### Goodhart's law

- And we also need to remember that even though we obsever goodhart's law, it isn't the same as a physical law, and there can be systems that are not affected by it.

#### General Comments

- Worth mentioning factors that could help researchers make impact are building resilience to failure and avoiding burnout.