Yeah, they rhyme for me. But I understand what you mean. I can say them differently in my head, and some of the accents here do distinguish between them.
But it's very unnatural for me. To pronounce poor differently my whole mouth just feels weird. "Pooh-err".