Sorry for being a party-pooper, but no, it's nowhere near that simple.
A) What some women say
I know two women who were non-violently sexually abused at a very young age, neither of which got the authorities involved or went into therapy, and they both independently say pretty much the exact same thing about the trauma and its consequences.
What's most striking is the similar language they use about the experience itself, saying for instance that it was "like an electric shock", or that those memories are so powerful that they really stand out among all other memories of the time. They both say the men, their fathers in both cases, didn't realize that they were causing harm precisely because the girls "seemed OK with it" (this is partly why they forgave them). They never cried or resisted.
They describe it as having "burned off" (I'd say "overused" or something similar) those neural connections. The underlying hypothesis is that such powerful stimulation—yes, sex is powerful, I do agree with that much—that the brain wasn't ready for meant that the brain down-regulated to compensate, which eventually made all sexual stimuli feel uninteresting and boring. It may be that all the talk about brain development being insufficient for sexual activity, while it's patently false when it comes to adolescents, does apply to very young children.
Now, how harmful is it to permanently prevent a person from truly enjoying sex? My mother says she doesn't care, but of course, she doesn't know what she's missing. The other woman is more resentful about it.
B) What my moral framework says
My preferred moral theory is
a version of preference utilitarianism with some added sophistication. The general idea is that what is morally good is what satisfies the greatest number of "informed" or "grounded" preferences, for the greatest number of people. Preferences are "grounded" when they are the result of inferences that are true (in a universal, objective sense), or at least believed to be true (in a personal, pragmatic sense).
If they are to be morally relevant, preferences must be understood as much more complex than "pleasure and pain", and definitely more complex than the outward expression of such feelings. Many people may hate being tickled yet appear to enjoy it (I've got one in mind). A masochist may prefer what you would view as pain. Many parents, in many cultures, may claim that a child should suffer to some extent to become "stronger", although I'd personally mostly disagree.
Even the simplest understanding of those matters shows that it isn't as simple as looking at the infant's momentary reaction. To give an extreme but very real example, parents at the turn of the 19th century would give opium to infants to soothe them. This caused addiction in several of them at a time where opiate dependency was ill-understood. Of course, the opium pleased the infant. Was it a good idea to give it? We must look at the long-term outcomes, not merely the immediate results.
Here's an even better example: if I give my kid lots of delicious junk food that they love, they'll be happy at the time, but the long-term consequences will be catastrophic.
To take into account such scenarios, the moral framework I subscribe to gives moral value not just to preferences before their fulfillment, but also
a posteriori, so that the aim of parenting isn't just to satisfy the child's preferences at the time (and in the case of infants, none of their preferences are "grounded" anyway), but that when they are capable of looking back at their infancy and then their childhood, they consider that what they experienced was preferable to the alternatives.
In other words, if you could demonstrate that people who experienced sexual activity as babies end up looking back at such experiences positively, I could consider it acceptable. In the absence of such evidence, and indeed considering that there is substantial evidence against it, I would err on the side of caution and forbid it altogether, similar to how I feel about incest generally.