How do I deal with my boyfriend's brother?

My boyfriend & I have been dating for 9 yrs and he supports me while I go to school. I am not currently working and when his half brother needs to go to town for a doctors appt., I've taken him. One time he made a vulgar comment in front of dying dad that when my boyfriend isn't around we can (then he grinded his hips). I just stared at him in disbelief. Another time taking him to the doctor he said that if the cops in the small town he lives in saw my "jugs" they would arrest me. I ignored him. All I do is bitc* at my boyfriend about him and he tells me that his brother has a weird since of humor and I'm being a drama queen. My boyfriend has a high power executive job and works a lot, I still help with his brother because they lost their parents and he has nobody. Friday, he had a seizure and can't drive until he goes to the neurologist. He told me the appt is Wed at 2:40. We were talking and he said to me that we have a lot in common and we should hook up. Then, he called and left a VM Sunday saying the day and freakin time of the appt. My boyfriend said there was nothing wrong with the message, that I'm taking his brother that has a weird sense of humor too serious and he'll deal with it. Why do I feel guilty?