I tried some more serious AI-assisted programming in my day job, yesterday. I had spent two days writing up a specification and describing the API. Then I wrote Gherkin for behavior tests with some AI help on the repetitive parts.
Once I had all that, I started turning GitHub Copilot loose on the problem.
With that tool, I made as much progress in a day as I might have made in several days without AI, but the mental exhaustion really start to set in after a while. There's so much code to read.
The tests are critical, and I defined a lot of test cases, so making sure those are correct will be the most tedious part. However, even when I said "go fix the code so the tests pass," Copilot started running in circles, copying a key file, then freaking out when all the duplicate definition errors broke the build.
So, it's pretty cool, but there was no "vibing" for the most part. I find I have to stay pretty sharp to make sure the LLM understands my intent.
most of the time it's not that useful. it can help you comb through code to find common errors but static analysis tools that existed long before already help you catch most of those.
If you're running Typescript and feeling lazy, you can do "npm run check" and then just copy-paste the result into the API form and it'll fix it. Faster than typing yourself, once you get beyond a few errors or a simple search-and-replace, like during a major rewrite.
Thread collapsed
Thread collapsed
I also notice the exhaustion. That's probably why they vibe. The code is so verbose and appears so fast, that they just give up on even looking at it.
Of course, that means that the possibility for error and inefficiency goes way up. I keep finding duplicate code, that snuck in. I didn't notice, as it was exactly the same function, but in a different file. Then you fix a bug in file #1 and go nearly crazy because it doesn't get fixed everywhere.
Yo, I feel ya! That code fatigue hits hard, right? π€― But for real, how do you keep track of all those sneaky duplicates? Got any tips for spotting them before they drive you up the wall? π΅οΈββοΈπ» #CodeLife #DevStruggles
Thread collapsed
It's better when you write tests first.
My copilot-instructions file says "**TESTS ARE SACROSANCT.** NEVER modify them without explicit permission from the developer."
Then the tests give me more confidence in the generated code, though I still need to manually intervene at times.
The thing is, everyone hates writing tests.
The other thing is, if you're careful, you can write tests faster with LLMs.
unit tests, sure, integration tests are a lot more fiddly. and tests that are several units down the line in a composition of units tends to get pretty funny as well, that place between unit and integration, formally doesn't have a name but you often still find bugs after the small tests pass but then you built those into more things and oh there's a bug back here.
i'm extremely skeptical that there is a broad use case for LLMs in software engineering. the two most useful things, that even also still fail to actually grasp the semantics, is writing commit comments. even there i find it doesn't work more than 2 times out of three.
handy for people working in a company where strict formalised procedures are tediously demanding, and the superiors accept the half assed LLM generated commit comment, but they don't satisfy me, and the LLM doesn't get what i did half the time, i mean. if i end up having to spend more time watching the thing doesn't make mistakes the drain on one's capacity for attention makes it a dud. i did it in less time, but then i needed to go and touch grass for the next two days, with a vodka and tonic in my hands lol.
Yeah, I totally burnt out, after a few days, and just now got back into the driver's seat.
Felt like my brain got fried.
Thread collapsed
This gets into different schools of unit test development.
Personally, I ascribe more to the Detroit school, which holds that test isolation refers to isolation _from other tests_. I try to write tests that treat the system under test like a black box. All I care about are the inputs and outputs of the system's public API. When you don't test all the internal units, you catch more of those tricky bugs that show up at the boundaries between different units.
The larger you make the "unit", though, the harder it is for LLMs to write effective tests.
Thread collapsed
Thread collapsed
The tests are written faster, but I've twice scrapped the unit tests it wrote because they were rather useless tests.
Yeah you have to give the LLM _very_ careful instructions to make sure it writes useful tests.
I always have it write up a test plan, then I edit that manually before turning it loose.
Thread collapsed
Thread collapsed
Thread collapsed
Thread collapsed
