• 0 Posts
  • 28 Comments
Joined 3 years ago
cake
Cake day: July 2nd, 2023

help-circle





  • That’s a lot, so I’ll be forthright that I’ll probably be giving a relatively terse and gappy response, but that’s more because of my own limitations as a mobile user.

    Fundamentally every test unit, integration, ui, manual is still the same thing:

    Given X do Y expect Z.

    Mocks, fakes, whatever you want to call them, they’re fundamentally the same concept too: “given some idealized version of P”

    All you’ve done is pushed where that line sits. If I update my company writer to prospend all my company names with “corp.”, and my company read API test seeds without it and something at your read service layer strips “.”, if I forget to update the “given” in my read test seed data, that bug stays invisible.

    It’s the same class of issue. We just kicked the can down the road.

    You could do writes and reads in the same test, but even then you’re still kicking the can down the road that your front end is calling the API correctly at all. Maybe that the buttons even render to an end user given their authority.

    One might then conclude, then, that everything should be an automated UI test.

    But would it catch a bug where the bottom collapsed to 1PX? Is that part of the test too? Maybe manual testing is the real way to go.

    A further step back why do we even test anything at all? To provide some ultimate guarantee that everything works perfectly at all times? Is that the gate for a PR? A deployment to prod?

    There are various gates, at different levels of certainty, and there is an economic scaling for the different levels of assurances. AND they scale with the overall complexity of the system.

    And that isn’t a bug, that’s a feature.

    It’s ALL behavioral contracts, up and down the pyramid.

    There is a bang, there is a buck. Get the most bang for your buck. The distribution will vary by complexity and structure of your project.

    That being said, I’ve said it before and I’ll say it again: int and UI tests make it very very easy to write fucking garbage code, the VALUABLE architectural pushbacks on poorly patterned code that unit tests avail is AT LEAST as much value to the overall tech debt accumulation of a project as the results themselves.

    Please don’t read this as me shitting on int tests. It’s me trying to impress that unit tests are worth writing, and justifications for forgoing them usually are fundamentally driven by an underlying code smell. Sometimes the smell is “our architecture is already so fucked that writing a meaningful unit test would involve re architecting large swathes of our system”.

    And in that case I’d say “it is what it is”, recalibrate the testing posture towards int tests, and accept that is the reality.

    But I would NOT conclude on the blanket statement that unit tests SHOULDN’T be at the bottom of the testing period generally. Just that I’d inherited or failed to maintain a strong system design, and take the opportunity to reflect on how things came to that state for the purposes of becoming wiser.

    Addendum:

    Even int tests can reveal insane system architecture, just at a different layer and scale. I worked on a system at a fortune 500 company where the architecture was too far gone to even have meaningful integration tests. They were too far gone to even have automated front end tests. All material testing had to be done by hand. They had engineered their system with that staggering level of incompetetence. At the risk of making a slippery slope argument… It’s a slippery slope.


  • Awesome and well articulated response. Love it.

    There are a few threads to pull here:

    Is an integration test the best way to test the full e2e behavior of an api call? Yep.

    Now, depending on the size (and architecture) of your system, the full test suite of into tests could take like, 10 hours.

    And even then, you’re STILL probably mocking something (direct seeding bombs into the db). Full setup of all life cycles of all your entities using their associated entities (bombs, wires, wire clippers, bomb diffusers, their rank, their bosses, their diffusion history, etc…)

    So they’re expensive, and if you really look you’re probably still cheating some steps anyways.

    Another thread to pull on is what are you mocking in your tests to begin with, and if they’re not resulting in valuable test results then that goes back to one of my original points being that you might have suboptimal patterns in your production code.

    Like, what logic governs bomb diffusion and where does it live? Are you mocking the right thing? If you have to mock the wrong thing… WHY? The ritual of writing those tests give you the first whiff of a potential smell.

    I tell my guys not to test implementation: test behaviour. Test contract adherence. If that’s hard, no problem, message me let’s review it. Maybe a behavior they’re trying to test legitimately should be an int test. More often than not, their design is demanding to mock something that should be decomposed further, and then composed with dependency injections that are where the units that should ACTUALLY be mocked are the things that get mocked and injected, so the unit tests are actually testing the BEHAVIORS we want to pin rather than implementations. Composition and single responsibility principal are what enable unit tests to pull their weight.

    Edit: I’m so passionate on the subject that if you scribbled up a scratch example and pinged me a github link I’d genuinely review it.



  • Curious how bad they’d have to be to land you at that position.

    If they’re hard to write, it’s kinda a red flag about the overall code itself. Forcing people to write them tends to force people to use better patterns just so the associated tests aren’t a bitch.

    And they pin behavior more than strict correctness, in my experience, but that’s a valuable signal. Canaries in the mine.

    You can be much more bold when you’ve got strong coverage.

    So I do admit some of the benefits might feel intangible… But boy howdy you’ll miss them if you move to a shop that never bothered. You’ll see the dumbest fucking patterns and be kinda handcuffed to do anything good.



  • I expect it would take about 15 minutes with the language definition to simply write a static tool (via an agentic coder, why not, just to prove the point) to evaluate the rules of the language, translate it to another (normal) language. Llm does the work there, then runs it through a mechanic to go the opposite direction.

    Identification that this has occurred becomes an evaluation of the prose “obscured” values are embedded in.

    I get the intent. I just think the testing methodology answers a very specific question, and forms a conclusion that over claims.


  • I think you completely missed the point. They’re not saying genz should be the adults in the room.

    They’re surprised that a generation who never drew breath without the internet came of age without the wisdom they hypothesized such a life would avail.

    You gotta understand: earlier generations couldn’t trivially verify things. Anyone they ever interacted with were the people like… On their street. Diversity of thought was informed by limited contact. The internet, at one point, appeared to promise a radical solution to both. The optimism for the future, as a result, was essentially unbounded.

    It’s not fair to ask gen z to save anyone. I don’t think its what buddy was saying, but if it was, it was unfair.

    I think buddy was coming to the very late realization that the optimism of the transformative nature of the internet was misplaced. The hypothesis just didn’t pan out. You can give a human for their entire lives the ability to get the answer to any question, verify anything they’re told, and meet and talk to people with diverse viewpoints that may challenge what they’re exposed to based on proximity…

    … And ultimately they’re just about as dumb as anyone else.

    And I get the impression that, more than anything else, is what makes them sad.






  • Windex007@lemmy.worldtoFuck AI@lemmy.world•Pull the plug already
    link
    fedilink
    arrow-up
    1
    arrow-down
    6
    ·
    29 days ago

    I don’t know the full context of the claim, but if I could even rebuild its own CLI executable to be able to run on the device and had network access, “Claude” (model actually runs on anthropic severs) could operate pretty comfortably on a smart device.

    So, I think it just depends on what you “count” as self replicating. I’d count it. And it’s already easily at that point now.


  • Windex007@lemmy.worldtoFuck AI@lemmy.world•Pull the plug already
    link
    fedilink
    arrow-up
    2
    arrow-down
    1
    ·
    29 days ago

    I mean, it already can? I have a lab with a subnet of VMS. I I got an agent to install across all of them. It was like “BTW I didn’t, but there is an active CVE that would allow me to access the host and do the same directly on the hypervisors”.

    Like, wow, cool. Thanks for the heads up.

    Which is actually true. I DID appreciate the heads up. Patched the offending issue. But still… It implies anthropic could flip a “go rogue” switch on their end.



  • The answers you’ve already gotten are the best. Commit patterns. By far.

    But the code itself has a distinct shape: impeccable application of pattern. Essentially perfect.

    Which then is juxtaposed by bizzare, compounding, and amplifying design choices, as if some invisible rule or assumption was invented and became a self imposed constraint to be worked around… with every iteration pointing at the previous as justification for it’s correctness for applying it the next.

    Neither in isolation is weird, it’s the marriage of the two. It’d be like watching a pro driver making millisecond perfect RPM matching shifts and then struggle to parallel park. Like… It doesn’t make sense. It doesn’t appear in nature.

    It’s normal to see a gap for human devs, although it’s typically inverted (better evidence of justification rational design choices, worse application of patterns). But it’s the MAGNITUDE of that gap (and ALWAYS favoring application of pattern) that gives it away.

    Sufficiently supervised processes raise the level of design coherence to roughly that of the operator. So I’m sure I could be bamboozled in those cases. Not sure exactly sure how everyone draws the line for what constitutes “vibe-coding”.

    Those are almost scarier though, because the level of QA or user testing would sit at non existent I’m sure.