I believe it has more to do with the brain development and power imbalance. Children are still developing and they don't have morals, they are still learning about how the world is and the evils of the world. If a child gets sexually abused their entire childhood, they aren't gonna be able to tell if that's normal or not, they are going to believe that this happens to every kid and that's it's just normal parenting. Later in life, this can create mental health problems to the child and also make a cycle of abuse. When it comes to power imbalance, the child is much less privileged than the adult. Children are much weaker than adults both mentally and physically. Society has a hierarchy which puts the child below the adult. The adult can hit the child, the adult can force a child to wear whatever they want, the child doesn't owe anything in the home so the parents are allowed to break any tows they want, the child can't deny a hug and etc. This hierarchy makes the power imbalance between child and adult worse, which is used to justify the possible lack of the child's consent.That makes sense; the most likely reason why sex with younger children is more often perceived as a negative experience is simply that the child is less likely to want sex, and so they are also less likely to consent, and so this argument would apply less to the ones with higher libido. But I'm not totally sure.
Now, I personally believe that there are better forms of dealing with this, but I believe these are the main reasons.
