Extended Development Record — The Gary Marcus Audit

OK, my thoughts on the current draft are below. To be honest, I’m not sure how much, if any, of this can be integrated into a coherent essay. Let me know what you think. Consider revising “The more serious critique is not that Marcus is wrong because he is irritating. It is that he may be technically right in some places and still culturally ineffective in others.”, to something closer to “The more serious critique is not that Marcus is wrong because he is irritating. It is that he may be technically right and still culturally ineffective.” Line 33 needs clarification. Consider revising “ The narrower claim is about update power among builders, adopters, investors, and AI-adjacent readers already inclined to dismiss him.”, to something closer to “ The narrower claim is about his warnings completely lacking relevance to the vast majority of builders, adopters, investors, and AI-adjacent readers who have long since moved on.” Line 41 - 43 - We need to drop or modify these lines. I don’t have any suspicion to retire here. I don’t think Gary has ever scolded ordinary users for finding the tools useful - at least to the best of my knowledge. Line 51 - consider changing to something closer to “But does the warning matter when everyone seems to have already accepted to the warning?” It’s like yelling at car manufacturers to install seatbelts. True, but I think they’ve already received the message…” Re: Line 55, the technical issues here are fascinating and the crux of this essay and we shouldn’t just glass this over. Gary’s critiques LLMs, but what alternatives does he support? Neuro-symbolic processing, right? But what is this exactly? How is this different from what LLMS actually do? And What is the evidence to support these Gary’s claims? Are current frontier labs pursuing any of Gary’s recommendations or something similar? Or different than what Gary is recommending but robust and architecturally distinct from LLM’s? If yes, what is the architecture exactly? And we can’t just label/provide a name for any techniques beyond LLMs that frontier AI labs are pursuing, we need to try, in simple terms, to describe/specify exactly what these advanced (non LLM) techniques are doing differently and how they are distinct from LLM processing. In simple terms whenever possible, what techniques/architecture are frontier labs employing and pursuing that might be in full alignment with Gary Marcuses claims about the need for Neuro-symbolic AI? Unless we make the advanced (non-LLM) AI architecture explicitly clear, for all readers will know, all of this “reasoning work, reinforcement learning, test-time thinking, multimodal reasoning, interpretability, alignment, red-teaming, agent autonomy, increasingly capable models, cognitive evaluations, multimodal agents, robotics, planning, and tools, world models, and modular architectures” is just “labels” that obscure the underlying architecture that could still be based on LLM inference. And while we are at it, the crux of the disconnect between Gary’s claim that “LLMs are not the path to AGI” and the government limiting access to Mythos level models made by Anthropic who’s CEO, Dario Amodei, leans hard on scaling laws for increasingly general cognitive capabilities. So there seems there is, at least on the surface, a major disconnect between Gary’s claims that LLMs are unreliable and not a path to AGI *and* the government simultaneously limiting access to a SOTA model because it is so advanced that it presents a major security threat when this SOTA model is made by Anthropic who’s CEO is leaning hard on LLMs and scaling laws. The core question here is, how much of Anthropic’s Mythos AI capacity is due to scaling high parameter LLMs and how much of it is due to other underlying architecture/techniques? If it’s all or mostly LLMs, Gary’s claims seem to be on shaky ground. Moreover, what is the evidence for continuing scaling and number of parameters and LLMs? Could increasing the number of parameters into the trillions and somehow increasing synthetic training data eventually produce true AGI (that isn’t spiky intelligence the way current AI is? Re Line 61: “But even those examples are not clean "LLMs alone" claims. Altman immediately talks about agents joining the workforce and iterative deployment. Amodei talks about powerful models, policy, national strategy, and general cognitive capability, not a bare next-token machine magically crossing into trustworthy general intelligence with no surrounding machinery.” This entire section seems like a non-sequitur to me. Saying “But even those examples are not clean "LLMs alone" claims.”, immediate followed by “agents joining the workforce and iterative deployment, powerful models, policy, national strategy, and general cognitive capability” looks like a category error to me because it doesn’t address the underlying architecture doing all of these things. Agents, iterative deployment, powerful models, policy, notional strategy, and general cognitive capability” could be powered by LLM AI, non-LLM AI, or a combination of LLM and non-LLM AI. In line 67, the following line is kind-of vague, we need to be more explicit here “Marcus may be arguing against a public story that labs benefit from even when their internal research maps are more complicated.” what public story are we referring to here? And how are their internal research maps more complicated? And same issue with this line “he hype does not need to be the private lab doctrine to shape funding, products, customers, and policy.” - what exactly is the price lab doctrine and how does it “shape funding, products, customers, and policy.”? Line 75: Maybe this is a good live claim, but I think it’s complicated by the recent banning of Mythos by the government - in that if an agent is so capable that it is a major national security threat, “trusting” the AI may increasingly make sense at even at core institutional, government, financial, legal, and medical levels - in terms of trusting the AI to “make high IQ decisions” may make sense, so it’s perhaps becoming less about “trusting” the AI to avoid errors with Mythos level AI - although I’m sure errors have not been completely eliminated here - and maybe never will - and more about integrating AI that is “trusted to make high IG/generally good/and possibly better than most human” into these high level/core/foundational institutions *without adequate governance & alignment*. Regarding line 93, I think the recent banning of Mythos access complicates the reliability critique. If the agent is so capable that it presents a national security threat, how bad can it’s reliability be? not a bare next-token machine magically crossing into trustworthy general intelligence with no surrounding machinery. Lines 99 - 121 may actually undermine Marcus’s claims about LLM’s being unable to achieve AGI (with government limiting Mythos access because it’s capabilities represent a security threat) and support his stance on the need for better governance and alignment - if, indeed, he is routinely calling for better governance and/or alignment. Line 131 & 133 - with Mythos level models, I think these claims are weakening (or it sure looks like it is weakening to me). Line 135 - this is kind of vague - isn’t broad performance directly related to dependable general intelligence? Line 137 - I might sharpen this to something like “priced in more economic benefit certainty” Line 139 - Yes, while this is exactly and obviously true, it also represents a view that is undermined with the ongoing advancement in SOTA models, especially the mythos class models. Moreover, has Gary ever predicted a ceiling performance level for LLM based AI? Did he ever envision an LLM based AI model that was capable enough to be considered a national security threat? Has he ever specified an “upper level of performance” and been consistent over time with these predictions? (In other words, are any of his claims about LLM performance even falsifiable? Have they ever been falsifiable? Or is he, and has he always been playing the moving goalpost game, and every time a new capability milestone is achieved, he just points at the remixing (ever shrinking) limitations and gleefully exclaims “AGI not achieved!”?

Agree, please proceed.

Line 25, consider revising to something closer to: Those examples show a side of Gary’s public persona that - to me anyway - seems out of place for someone with his obvious intellectual gifts/talents. Line 41, consider sharpening to something closer to: So what does the warning mean once the culture has already acknowledged it, mostly ignored it, and moved on? Line 59, regarding this section, “OpenAI talks about reinforcement learning, test-time thinking, tools, multimodality, and agentic use. Google DeepMind talks about agentic frameworks for robotics and models that reason before acting. Meta, through LeCun's research program, is explicitly pursuing world models and modular architectures.”,it seems like we may be making a category error here again with listing modalities that may, in fact, rely - at lest in part - on underlying LLM technology. So if any or all of the following is clearly unrelated to underlying LLM AI, we need to state so explicitly because it not apparent (to me at least) that this is true “reinforcement learning, test-time thinking, tools, multimodality, and agentic use, agentic frameworks for robotics, and models that reason before acting, world models and modular architectures.”. Moreover, if any of these modalities do rely on underlying LLM technology, they should not be included in a section claiming to show modalities that do not rely on LLM technology. Line 61 - I’m not sure what this sentence is trying to do, exactly, it’s going in too many directions, and “are being sold as if they have earned dependable general intelligence” isn’t a claim I’ve heard Marcus (or anyone else for that matter) make. Maybe I’m wrong, I just haven’t heard this. And overall the sentence is just hard to understand. * *If* - Marcus means that LLM-centered systems, even with scaffolding, are being sold * *as if* - they have earned dependable general intelligence * *the critique* - what critique were we talking about now? - is still alive. Line 67 - Is there a difference between solving the failure and routing around the failure in terms of real world performance? I’m not sure there is. Wouldn’t either solution adequately solve the problem in terms of real world performance/execution? Also would consider saying “the failures Marcus describes” instead of “the failure Marcus cares about”. Line 69 - What is the model in this section? The coding agent? And what is the environment here? The symbolic machinery? Something else? How does it push back? Line 73 - Would perhaps expand on how changing externally grounded real world context overtime is one of the biggest challenges here. No matter how brilliant the initial AI “output”/“response” is, changing external real world context can make the initial output irrelevant. Line 81 - We would need to verify all of my Marcus related claims that follow (verify he actually said these things), but I would consider changing “If the claim is that LLM-centered systems cannot become strategically important, economically transformative, or dangerous enough for state power to care about, Mythos makes that claim look shaky. A model does not become the subject of export-control conflict because everyone agrees it is merely autocomplete.” to something closer to “If the claim is that LLM-centered systems cannot become strategically important, economically transformative, or dangerous enough for state power to care about, Mythos makes that claim look shaky. A model does not become the subject of export-control conflict because is unreliable, operating on shallow pattern recognition, a sophisticated game of Mad Libs, subject to pervasive hallucinations & brittle reasoning, and a glorified memorization machines subject to the “dead end” scaling trap.” Line 89 - I would change to something closer to “The system may be powerful enough to matter and unaligned/ungrounded in reality enough to require governance. Capable enough for national-security concern does not automatically mean full institutional alignment. A model more capable than most humans presents governance and alignment challenges and risks to the institutions in which it is deployed.” And even if some ever shrinking unreliability/inconsistency remains in Mythos level AI, cyber attackers can still be dangerous because attackers just can retry, filter failures, and chain partial successes. And in medicine, law, finance, government, or education, the tolerance for hidden errors - no matter how small - is different.” Line 93 - would tweak towards something like “while still not being aligned, governed, secure, or 100% dependable enough for the authority it is being handed.” Line 127 - Need to clarify “hype” in references to what, “abstinence” from what, “dismissal” of what, and “surrender” to what. Line 133, This is a lot to absorb, much less remember. Maybe we could just say something closer to (or add this at the end of this section), “or, at the very least, just write down your views on anyone or anything and ask a SOTA AI model for an adversarial critique”. The results may surprise and will almost certainly enlighten you. (And maybe we should mention something here about why this can be such a helpful technique - because critiquing, by definition, finding flaws in a position/claim, is much easier than finding the correct “answer”/“output” that is actually grounded in reality (as opposed to a collapsed synthesis of statistical related letters that is completely disconnected from reality), so humans and AI can find the flaws more easily. And this is probably one of the biggest upsides and underutilized advantages of AI > the capacity for adversarial critique Line 139 - Consider tweaking to something closer to “He may be culturally ineffective - at least with some people who think like I do. He may also be right about enough that dismissing him says more about my filters than his claims.

Line 1 needs to be more consistent with Intuition Issue 2, so something closer to “I want Gary Marcus to be wrong” or “I would like to write him off, ignore his claims as irrelevant, unimportant, and most gratifyingly, just plain wrong. Line 13, we shouldn’t directly reference the intuition piece in the Signal & Noise piece. Each issue should essentially stand own it’s own, but work together if read in sequence. Consider modifying to something closer to “but do my wishes hold up under scrutiny?” Line 19: Whether or not I trust Marcus is irrelevant. I don’t think he makes stuff op or posts things he doesn’t believe. I think he is internally consistent and has, when viewed within the context he is operating in, probably complete integrity. The issues is he annoys the **** out of me so I would love to dismiss everything he says, write him off as completely misguided, stuck in the past, and relish the schadenfreude of watching his predictions fade into irrelevance. << Come to think of it, maybe everything I just said is good material to integrate into Intuition Issue 2 - please look into this too. Regardless, I’m not sure what you could use in this transition besides trust, but it’s not really accurate or relevant to say “I do not trust Marcus”. Maybe we could do away with this line altogether if we can’t find a suitable alternative to trust here. Line 21: change “adoption momentum” to something closer to “perceived political affiliation” or just “politics”. LIne 135: - 141 - We can’t end this section with a laundry list of things people can do to audit their views. It may be true, but it is complete unhelpful. They will forget 100% of this a picosecond after they read it. Ideally we would distill this into one or two, or at most 3 absolute maximum things people could do to adversarially critique their own views, but honestly it’s hard to do from inside one’s own head due to motivated reasoning and filtering, etc, etc, and this is where the adversarial critique function of AI can be genuinely useful because it doesn’t have all this motivated reasoning and filtering that make self-critique so difficult for humans. So my suggestions here is to put 1 - 3 ways that people can self critique in the beginning of this section - along with all the reasons this is so challenging and often ineffective (motivated reasoning and filtering, etc) and end with the AI assisted adversarial critique.

OK. We are very close. Please make the following adjustments and then lets move to the next steps. Line 25, would tweak to something closer to: “they may sound brilliant, rejuvenating, gratifying, funny, bracing, unfair, vindicating, petty, or exhausting.” Line 113: let’s tweak to something closer to “I still do not want Marcus as my oracle for AI because his rhetoric often makes this reader do avoidable cleanup.” And “That style may give his opponents permission to ignore him cheaply.”

Excellent, but before we proceed, since we are publishing two issues together (Influent #2 and Signal & Noise #16), and since both of these issues directly reference a public figure and make claims about his views, Let’s prepare a prompt to do the Adversarial critique for both Intuition #2 and Signal & Noise #16 at the same time. I know this is somewhat redundant with checks we’ve already run, but since these issues involve Gary Marcus, let’s try not to misrepresent his views and do a modified more rigors adversarial critique that covers the following 4 things: * Are we misrepresenting Gary Marcuses publically stated claims in any way? * Are we misrepresenting the current state of AI and/or its technological underpinnings in any way? * Are our issues internally coherent and coherent across both issues (Issue 2 and Issue 16)? * Our normal Light NIR for refining prose, etc.

Agree with one question. Can we come up with a better title for Issue 16 than "The warning after adoption"? This title is accurate, but also incredibly vague and not linked to Intuition #2 in any way. Could we use "The Gary Marcus Audit" for Issue 16?

Actually, what do you think about the following: "The Gary Marcus Adversarial Audit" - gives readers a "heads up" about where I'm coming from and perhaps also makes the landing better (in defense of Gary's core messaging). With your deck/subtitle included. What do you think?

I see your point. How about "The Gary Marcus Audit An adversarial test of the critic I want to dismiss" so the title doesn't give away the final landing. What do you think?

Perfect. Let's proceed to the next steps.

Excellent, now we still need to prepare the world behind the words material for both essays, including "the conversation behind this" content and links, a brief statement to include in both issues with a link to the other issue (so they are cross linked effectively), and the audio material for each issue. Can you start with a brief statements for each issue that informs the reader of the corresponding issue and links to it?

Excellent. Can you prepare the audio for both essays, and the world behind the words material for both essays, including "the conversation behind this" content, and the links for placement inside the Issues for "the conversation behind this" content?