Pam,
Her father was indeed a Poet, one of Britain's finest. He was also a scoundrel who married for money, was violent with his wife, drunken, constantly in debt, and a horrible womanizer. He had several illegitimate children including one with his half-sister. He was, at the time in Victoria's England, infamous.
Ada's mother, Annabella was a math whiz. Byron derided her by calling her his "Princess of Parallellograms". Annabella decided early on to keep Ada away from poetry. She feared that her father's insanity would infect he daughter. So Ada was tutored in math from a very young age.
This worked out well in the end. Indeed one of her later tutors, recommended by Babbage, was none other than Augustus De Morgan. But the early tutoring wasn't always about math. At 17 Ada had an affair with one of her tutors. This was discovered and quickly covered up. Victorian society was a voracious rumor mill, and the daugher of the infamous Byron was always under scrutiny.