I'm not a doctor, nor chemist, but I can't see any harm in it as long as he didn't bite it open and swallow anything. Children eat foreign objects all the time, and although it's not good, it's seldom serious. What's a 4-year-old doing putting things in his mouth anyway? I'd worry about a dental or possible emotional problem that might need addressing before worrying about poisoning from the battery contents. In my experience, kids don't do that much beyond teething months and early toddlerhood. By 4, he shouldn't be interested in doing that, in my opinion. It should move on to worrying about skateboard injuries, safe handling of matches and fire, and of course gun dangers & respect. I'm not trying to insult the boy or his parents; just pointing out a concern.
Charles