I this case I did though because I need some additional dependencies for the arm64 build, and I had to pass a few extra flags to the go build command to make it happy :)
Discussion
You could probably use the TARGETARCH arg to conditionally install it.
Something like this might work:
RUN [ "$TARGETARCH" != "arm64" ] || apt-get install -y gcc-aarch64-linux-gnu binutils-aarch64-linux-gnu
Another strategy is to have both platforms defined in the same Dockerfile and then use the BUILDARCH and TARGETARCH arguments to select the correct one. That's what I'm doing for the Helipad Dockerfile: https://github.com/Podcastindex-org/helipad/blob/main/Dockerfile