• 0 Posts
  • 139 Comments
Joined 3 years ago
cake
Cake day: September 24th, 2023

help-circle




  • Funny I’ve seen the opposite. I want it to rewrite half the codebase to do things properly. A human shouldn’t do that because of the effort, but it’s no issue for AI.

    However it very often says “that’s too big of a change so I’ll do this workaround instead” which is not what I want at all.

    But I agree they sometimes implement things that don’t actually make sense to do. Mostly you can solve that by just asking how to program / code works at a high level without needing to read every line.


  • Yeah so Opus 5.5 (which is what I use basically 100% of the time currently) gets 100% anyway. As do most of the other frontier models.

    Would you care to show some evidence, or is it a vibe?

    It’s an very conservative estimate based on my experience. Almost all open source software these days keeps their docs in the same Git repo as their source. I haven’t checked these, but go and look at e.g. LLVM, Git, libpng, SDL, Qt, etc. if you don’t believe me.

    Show us the evidence.

    Nah, too much work and you wouldn’t believe me anyway. I’ve seen it with my eyes though. Hell, frontier models are happy to reverse engineer binaries without prompting in order to get things done. Reading a bit of source code is nothing to them. (Does it violate a “no reverse engineering” license agreement if AI does it of its own volition lol…)

    Would you like to find this out at runtime

    No I’m not dumb enough to use dynamic typing.


  • I can’t say I agree with this.

    AI writes shallow tests.

    It definitely did in the past. But not any more. It’s far better at writing tests than humans and I haven’t seen a single nonsense test out of Opus 5.5.

    AI writes overly complex ccode

    This one I will kind of give him. AI does seem to like being super paranoid - writing tons of checks for things that we don’t really need to check, and adding config flags for absolutely everything. You can just tell it not to do that though. Annoying that you have to, but it does work.

    Reliance on AI decreases human understanding

    Yes but does it matter? Does your manager understand the codebase? Does his manager?

    AI moves too quickly to maintain a high quality bar

    It’s too quick for thorough human code review definitely, but I don’t think that means you can’t have a high quality bar. There are other ways to ensure a high quality product. Traditional QA testing for example.


  • The moat in CoDoc is that the documentation ships inside the package, version-matched, offline, deterministic. Can you tell me what proportion of software libraries today ship complete AI-optimized documentation within the library’s artifact? I’ll wait.

    Wow what a moat. I would say maybe 50% of software already does that. For example 100% of Rust crates ship the docs with the code. In any case I don’t think this really matters because AI will happily skip the docs and just figure things out from the source code, which is usually better anyway.

    Vercel’s evals

    Using which model? They don’t say? Huh.

    A 27B on your laptop with the right context can beat a frontier model that’s hallucinating your v5 API signatures.

    Frontier models very rarely hallucinate anymore, and they’re actually really good at writing tests now so I really doubt this.






  • Ha yes I always thought this is how assembly should look. Although why not go further - instead of move a, b which is ambiguous about the move direction, why not just copy 99% of other programming languages: b = a.

    Also it’s a real disappointment that even new ISAs like RISC-V have the same “assembly must be unreadable” brain disease. auipc? Come on. You can’t even claim that they want a strict 1:1 assembly:instruction language because they added a ton of pseudoinstructions. Even add can turn into add or addi.

    Anyway although AI means you can easily make a tool like this, I think it also means there is not really any point to it anymore. Oh well.