Is that required? I'd argue just directly ingesting all the software that exists directly avoids all the garbage that comes with human language. Maybe there's not enough data though.