I don't really have a solution but more or less commiserating here. If I see any decent solutions, I'll let you know because they absolutely do not respect robot.txt files.