Wrong. Male means that you are of the phenotype that produces a specific type of gamete, called the male gamete, small gamete, or sperm. Female means that you are of the phenotype that produces the other type of gamete, called the female gamete, large gamete, or ovum. One of each gamete is required for human reproduction. No human can produce both gametes.
Some men are more masculine and some are more feminine. Some women are more feminine and some are more masculine. One's gender (masculine traits versus feminine traits) is a social construct and is an independent dimension from one's sex. Saying that men should only be masculine and that women should only be feminine, and/or that gender stereotypes define them as men and women, is sexist and regressive.
I will agree with you that the shape of your genitalia is not strictly relevant to your sex, male or female. Intersex people do exist albeit very rare, and the shape of their genitalia do not conform to the norms of their sexual phenotype. However, if you have complete male genitalia and produce sperm, you are a man and will always be a man even if you amputate your penis and scrotum and get silicone implanted chesticles. If you have a womb and produce eggs, you will still be a woman even if you mutilate your breasts and vulva and take testosterone to make your voice deep, grow facial hair and develop "male pattern baldness", which really should be called testosterone baldness since it is the testosterone causing it, not the fact of being male.