Yes, I think you're being unreasonable.
I think you're trying to dictate/judge the relationship this guy has with his kids. 17 year old girls do weird things that their fathers will never understand, even though they try to be good dads by taking them shopping, to concerts, etc. You're nitpicking his family situation (that existed long before you were in his life) way too much, and if you want to get along with a man who also has children, and his own way of parenting, you will have to learn to live with that or leave if it bothers you too much.
I don't think it's unreasonable for a dad to have alone time with his kids. You're not their mother, you're not married to him. You haven't even been dating him a very long time. I think you should back off a little bit and just let him be the dad he needs to be.
If you aren't comfortable with this situation you do not need to tolerate it. When you were describing it something does seem a little off, but then again she's 17 yo and they can be a little off sometimes. I think before you can do anything you have to decide if you can accept what is. If you can't accept it then that's your answer, you can't accept it so you need to leave the relationship.
But if you want to be with him you'll probably have to give him some room here. If all the other kids are comfortable, that's great. It doesn't mean the older kids will behave like the younger kids. If you don't put yourself in competition with her for his attentions, then it'll probably resolve itself. If you do put yourself in competition with her while trying to maintain a committed relationship to him, you've created a no-win situation for at least one of you. If you want to be included, accept the offer to be included even if it comes after the fact. If you want the three of you to do something else to bond, suggest it. If you want to get closer to her, come up with a way to do that. I don't think you can get him have some sort of talk with her that is going to suddenly make her accept you. You can talk to your SO and figure out ways you can be a team in figuring out how to have a blended family with one resistant family member.
I think you may be over reacting... Teens are always difficult to deal with, whether you are the parent, or dating the parent. Your boyfriend is probably just trying to be there for her, spending time with her to be a positive influence. If he is worried over her behavior lately, then he would especially want to be there for her, hoping to lead her in a better direction. He probably wants to fix her awkwardness with you, but is doing so in his own way, on his own time. Which is his right as her parent.
Rather than try to compete with her, or allowing yourself to feel left out (like a girl who is sad she is not invited to the slumber party), you should focus on what you can do to lessen this tension between you two
This is not his child. She's the daughter of someone he could not have been with all that long, years ago. This relationship is suspicious.
I think that maybe you should have him leave the house where your own children live until it's sorted out. I guess you could check out websites or see a counselor for professional guidance. Any errors in judgement should lean to protecting your children. This was way too soon to move in with him, you didn't know him well enough. Good luck.