Morality almost by definition is subjective. while pain and pleasure are just for the moment and do not matter in the long term besides the fact that they are things that change people's minds.

What really matters in ethics seems to be how you change other people regardless if this is pain or pleasure. society should consider things wrong that change other people into something that will make them cause harm.

like if some drunk husband is battering his kids I may not care about those kids, however, I do care about the fact that most likely when they will grow they may kill me.

it looks like this is how society operates in general because nobody really cares about crimes itself, what matters is to create the illusion that crime does not pay.

If you were robbed by someone nobody really cares about you however what everyone does care is that this thief who robbed you will change your mind (or the mind of those who care about you) in a way that you will start doing same. so this Is why I have to protect you against robbers and pretend that I am trying to catch the criminal just to prevent you from becoming criminal yourself.
In situations when you actually can't harm me I will not care about what happens to you at all. just like we do not care what people do to each other behind our state borders.