I havent seen them/him anymore ever since he left me.. so I have no clue if it upsets me. She has been stalking me for a while on different message boards so SHE upsets me.
I think that to start a relationship with your ex's brother, even to entertain the flirting, is a really horrible idea that will go up in flames eventually.
You wish you met his brother first but that's not reality. Focusing on "what if" is a waste of time that will get you nowhere except stuck.
Dial it back with his brother, this is bad drama/karma that you will regret. There are other men in the world. Maybe if a few years pass and you don't see your ex as "pathetic" anymore, you can revisit this idea if you can be civil, but your ex will always be his brother and right now I don't see anything good coming from this situation.