During what times? It would be difficult to have required work hours during the weekdays unless it's after 6 pm as you can have required things come up on the weekday mornings. Everyone will have primary clinic all day on Wednesday every other week, and you'll have hospital mornings every other week on a different weekday than Wed. Also, random things can come up. For instance, they made our tests on Thursday mornings this block and that came kind of out of the blue.
I work about 4-8 hours a week, but it's usually on the weekends, very, very early in the morning or later in the evening. I'm fine with it, and the money does help. My job is also not stressful in the slightest. You could probably do it, and if you're finding that it's hard to balance, quit your job. You could let your employer know that's a possibility. It's honestly hard to predict how well you're going to adjust to med school -- it really depends on you.