thorn in your side
also thorn in the side of someonesomeone or something that continually causes problems for you:
Money problems have been a thorn in our side since the day we got married.
Health inspectors are a thorn in the side of most restaurants.