Am I being un-reasonable?

I am a single father of two children a boy 12 and a girl 7. My ex and I split custody of them every other week. Her live in boyfriend has been disciplining our son. I have a problem with this is my ex is usually there when this happens. I was raised to believe that that as my and her job/responsibility. I have sat down with both her and the boyfriend twice now to discuss this and was told that he understood and that it was her job. Now, when I picked them up from her yesterday, my son told me that the boyfriend had made him run around the the house till his back hurt so bad that he was crying, he was then told by the boyfriend that if he sat down he would him, now all of this is while his mother was there. Of course now I am pissed, I text my ex and got no response so I asked the boyfriend, his response was "if I would discipline my son, he would not have to!" This discipline was brought about due to my son yelling for his mother while er mother was asleep(they live with her). I told the boyfriend that this was not what we all agreed to and if he had actually ed my son I would press charges( as my father, a judge, said I could already do if I wanted. So, I ask, am I being unreasonable in that I expect their mother t do HER job when they are there and not to delegate it to her boyfriend?