If the black phantoms linger after becoming disconnected from the servant (if you go offline, etc), there is an invisible timer that gets set to 10 minutes. Wait out the 10 minutes, and you'll receive the "disasters are gone..." message automatically and the black phantoms will dissappear.
However, the catch is that this timer resets itself every time the player dies, rests at a bonfire, or accesses a loading screen. So if you keep dying or resting at a bonfire, you can prolong the Gravelord infection indefinitely. I know of a few examples where someone thought the game bugged and gave them a permanent infection, but it turns out they were just resetting the timer by accident.