Honestly, I wouldn't do it. To be fair, this is coming from someone who couldn't imagine missing the majority of my child's early life. Sure, once you get through, your life would be more manageable, but you'll never get those years of your toddler back. If that's your top priority, which it sounds like to me in your posts and responses, I'd move on to something else. Keep in mind though, there are surgical subspecialties with very reasonable hours, even in residency, so maybe you should consider those of you really want to be in the OR.
Also, since it's neurosurg, I'm going to tell you about a friend of mine that always planned to be a neurosurgeon. After 3.5 years outside of the state he grew up in, away from his family, it was time to apply for residency. He also liked Neuro a lot, so he applied to both NS and Neuro, with most of his programs in his home state. When it came time to submit his ROL, he decided it was more important to him to be home than the difference between NS and Neuro, so he ranked NS, Neuro, NS, Neuro, etc.
In the end he matched Neuro in his home state at a program he really liked, and he managed to spend the last couple years of his father's life at home with him. Looking back, he is really glad not only that he ranked the programs the way he did, but that he matched Neuro. My point is that sometimes you have to figure out what's most important to you (and it might be NS). For most people it's not one field or bust. They can be easily happy in their job doing something else. Medicine is versatile.
You could also do what my friend did, and delay your decision until you actually go to those programs on your interviews, talk to NS residents outside of your own program, and then submit your ROL. Obviously, it's not the easiest thing applying to 2 different specialties, but people do it all the time. Maybe you just need some more time to decide.
EDIT: Just to be clear, I'm saying you need to prioritize what you want. If you can't see yourself being nearly as happy doing something else, then do NS. Obviously don't choose something else and then blame your family, that's not fair to them or you. This is your decision, and yours alone (given that your wife is on-board with what you want). Pick what you feel is most important and don't look back.