My mainline prediction scenario for the next decades. My mainline prediction * : * LLMs will not scale to AGI. They will not spawn evil gremlins or mesa-optimizers. BUT Scaling laws will continue to hold and future LLMs will be very impressive and make a sizable impact on the real economy and science over the next decade.  * there is a single innovation left to make AGI-in-the-alex sense work, i.e. coherent, long-term planning agents (LTPA) that are effective and efficient in data sparse domains over long horizons.  * that innovation will be found within the next 10-15 years * It will be clear to the general public that these are dangerous  * governments will act quickly and (relativiely) decisively to  bring these agents under state-control. national security concerns will dominate.  * power will reside mostly with governments AI safety institutes and national security agencies. In so far as divisions of tech companies are able to create LTPAs they will be effectively nationalized.  * International treaties will be made to constrain AI, outlawing the development of LTPAs by private companies. Great power competition will mean US and China will continue developing LTPAs, possibly largely boxed. Treaties will try to constrain this development with only partial succes (similar to nuclear treaties).  * LLMs will continue to exist and be used by the general public * Conditional on AI ruin the closest analogy is probably something like the Cortez-Pizarro-Afonso takeovers. Unaligned AI will rely on human infrastructure and human allies for the earlier parts of takeover - but its inherent advantages in tech, coherence, decision-making and (artificial) plagues will be the deciding factor. *  The world may be mildly multi-polar.  * This will involve conflict between AIs. * AIs very possible may be able to cooperate in ways humans can't.  * The arrival of AGI will immediately inaugurate a scientific revolution. Sci-fi sounding progress like advanced robotics, quantum magic, nanotech, life extension, laser weapons, large space engineering, cure of many/most remaining diseases will become possible within two decades of AGI, possibly much faster.  * Military power will shift to automated manufacturing of drones &  weaponized artificial plagues. Drones, mostly flying will dominate the battlefield. Mass production of drones and their rapid and effective deployment in swarms will be key to victory.   Two points on which I differ with most commentators: (i) I believe AGI is a real (mostly discrete) thing , not a vibe, or a general increase of improved tools. I believe it is inherently agenctic. I don't think spontaneous emergence of agents is impossible but I think it is more plausible agents will be built rather than grown.  (ii) I believe in general the ea/ai safety community is way overrating the importance of individual tech companies vis a vis broader trends and the power of governments. I strongly agree with Stefan Schubert's take here on the latent hidden power of government: https://stefanschubert.substack.com/p/crises-reveal-centralisation Consequently, the ea/ai safety community is often myopically focusing on boardroom politics that are relativily inconsequential in the grand scheme of things.  *where by mainline prediction I mean the scenario that is the mode of what I expect. This is the single likeliest scenario. However, since it contains a large number of details each of which could go differently, the probability on this specific scenario is still low. 

There is a mystery which many applied mathematicians have asked themselves: Why is linear algebra so over-powered? An answer I like was given in Lloyd Trefethen's book An Applied Mathematician's Apology, in which he writes (my summary): Everything in the real world is described fully by non-linear analysis. In order to make such systems simpler, we can linearize (differentiate) them, and use a first or second order approximation, and in order to represent them on a computer, we can discretize them, which turns analytic techniques into algebraic ones. Therefore we've turned our non-linear analysis into linear algebra.

EA organizations frequently ask for people to run criticism by them ahead of time. I’ve been wary of the push for this norm. My big concerns were that orgs wouldn’t comment until a post was nearly done, and that it would take a lot of time. My recent post  mentioned a lot of people and organizations, so it seemed like useful data. I reached out to 12 email addresses, plus one person in FB DMs and one open call for information on a particular topic.  This doesn’t quite match what you see in the post because some people/orgs were used more than once, and other mentions were cut. The post was in a fairly crude state when I sent it out. Of those 14: 10 had replied by the start of next day. More than half of those replied within a few hours. I expect this was faster than usual because no one had more than a few paragraphs relevant to them or their org, but is still impressive. It’s hard to say how sending an early draft changed things. One person got some extra anxiety because their paragraph was full of TODOs (because it was positive and I hadn’t worked as hard fleshing out the positive mentions ahead of time). I could maybe have saved myself one stressful interaction if I’d realized I was going to cut an example ahead of time Only 80,000 Hours, Anima International, and GiveDirectly failed to respond before publication (7 days after I emailed them).  I didn’t keep as close track of changes, but at a minimum replies led to 2 examples being removed entirely, 2 clarifications and some additional information that made the post better. So overall I'm very glad I solicited comments, and found the process easier than expected.   

Something I've been thinking about lately: For 'scarcity of compute' reasons, I think it's fairly likely we end up in a scaffolded AI world where one highly intelligent model (that requires much more compute) will essentially delegate tasks to weaker models as long as it knows that the weaker (maybe fine-tuned) model is capable of reliably doing that task. Like, let's say you have a weak doctor AI that can basically reliably answer most medical questions. However, it knows when it is less confident in a diagnosis, so it will reach out to the powerful AI when it needs a second opinion from the much more intelligent AI (that requires more compute). Somewhat related, there's a worldview that Noah Smith proposed, which is that maybe human jobs don't actually end up automated because there's an opportunity cost in giving up compute that a human can do (even if the AI can do it for cheaper) because you could instead use that compute for something much more important. Imagine, "Should I use the AI to build a Dyson sphere, or should I spread that compute across tasks humans can already do?"

Implicitly, SAEs are trying to model activations across a context window, shaped like (n_ctx, n_emb). But today's SAEs ignore the token axis, modeling such activations as lists of n_ctx IID samples from a distribution over n_emb-dim vectors. I suspect SAEs could be much better (at reconstruction, sparsity and feature interpretability) if they didn't make this modeling choice -- for instance by "looking back" at earlier tokens using causal attention. Assorted arguments to this effect: * There is probably a lot of compressible redundancy in LLM activations across the token axis, because most ways of combining textual features (given any intuitive notion of "textual feature") across successive tokens are either very unlikely under the data distribution, or actually impossible. * For example, you're never going to see a feature that means "we're in the middle of a lengthy monolingual English passage" at position j and then a feature that means "we're in the middle of a lengthy monolingual Chinese passage" at position j+1.  * In other words, the intrinsic dimensionality of window activations is probably a lot less than n_ctx * n_emb, but SAEs don't exploit this. * [Another phrasing of the previous point.] Imagine that someone handed you a dataset of (n_ctx, n_emb)-shaped matrices, without telling you where they're from.  (But in fact they're LLM activations, the same ones we train SAEs on.). And they said "OK, your task is to autoencode these things." * I think you would quickly notice, just by doing exploratory data analysis, that the data has a ton of structure along the first axis: the rows of each matrix are very much not independent.  And you'd design your autoencoder around this fact. * Now someone else comes to you and says "hey look at my cool autoencoder for this problem. It's actually just an autoencoder for individual rows, and then I autoencode a matrix by applying it separately to the rows one by one." * This would seem bizarre -- you'd want to ask this person what the heck they were thinking. * But this is what today's SAEs do. * We want features that "make sense," "are interpretable." In general, such features will be properties of regions of text (phrases, sentences, passages, or the whole text at once) rather than individual tokens. * Intuitively, such a feature is equally present at every position within the region.  An SAE has to pay a high L1 cost to activate the feature over and over at all those positions. * This could lead to an unnatural bias to capture features that are relatively localized, and not capture those that are less localized. * Or, less-localized features might be captured but with "spurious localization": * Conceptually, the feature is equally "true" of the whole region at once. * At some positions in the region, the balance between L1/reconstruction tips in favor of reconstruction, so the feature is active. * At other positions, the balance tips in favor of L1, and the feature is turned off. * To the interpreter, this looks like a feature that has a clear meaning at the whole-region level, yet flips on and off in a confusing and seemingly arbitrary pattern within the region. * The "spurious localization" story feels like a plausible explanation for the way current SAE features look. * Often if you look at the most-activating cases, there is some obvious property shared by the entire texts you are seeing, but the pattern of feature activation within each text is totally mysterious.  Many of the features in the Anthropic Sonnet paper look like this to me. * Descriptions of SAE features typically round this off to a nice-sounding description at the whole-text level, ignoring the uninterpretable pattern over time.  You're being sold an "AI agency feature" (or whatever), but what you actually get is a "feature that activates at seemingly random positions in AI-agency-related texts." * An SAE that could "look back" at earlier positions might be able to avoid paying "more than one token's worth of L1" for a region-level feature, and this might have a very nice interpretation as "diffing" the text. * I'm imagining that a very non-localized (intuitive) feature, such as "this text is in English," would be active just once at the first position where the property it's about becomes clearly true. * Ideally, at later positions, the SAE encoder would look back at earlier activations and suppress this feature here because it's "already been accounted for," thus saving some L1. * And the decoder would also look back (possibly reusing the same attention output or something) and treat the feature as though it had been active here (in the sense that "active here" would mean in today's SAEs), thus preserving reconstruction quality. * In this contextual SAE, the features now express only what is new or unpredictable-in-advance at the current position: a "diff" relative to all the features at earlier positions. * For example, if the language switches from English to Cantonese, we'd have one or more feature activations that "turn off" English and "turn on" Cantonese, at the position where the switch first becomes evident. * But within the contiguous, monolingual regions, the language would be implicit in the features at the most recent position where such a "language flag" was set.  All the language-flag-setting features would be free to turn off inside these regions, freeing up L0/L1 room for stuff we don't already know about the text. * This seems like it would allow for vastly higher sparsity at any given level of reconstruction quality -- and also better interpretability at any given level of sparsity, because we don't have the "spurious localization" problem. * (I don't have any specific architecture for this in mind, though I've gestured towards one above.  It's of course possible that this might just not work, or would be very tricky.  One danger is that the added expressivity might make your autoencoder "too powerful," with opaque/polysemantic/etc. calculations chained across positions that recapitulate in miniature the original problem of interpreting models with attention; it may or may not be tough to avoid this in practice.) * At any point before the last attention layer, LLM activations at individual positions are free to be "ambiguous" when taken out of context, in the sense that the same vector might mean different things in two different windows.  The LLM can always disambiguate them as needed with attention, later. * This is meant as a counter to the following argument: "activations at individual positions are the right unit of analysis, because they are what the LLM internals can directly access (read off linearly, etc.)" * If we're using a notion of "what the LLM can directly access" which (taken literally) implies that it can't "directly access" other positions, that seems way too limiting -- we're effectively pretending the LLM is a bigram model.