I'm not trying to excuse my fellow forest hunters. The 2v1 invader gankers are as bad as the forest golden boys. But we all expect the worst, so people probably think you've set up a trap. Even when they learn that there isn't another phantom, it's hard to shake the feeling that the host is going to do something cheesy.
If your going to fight in the forest, you basically have to learn to deal with cheap losses and move on. You also have to learn the art of fighting 2v1 or 3v1. As others have said, it's a totally different skill set then dueling. A small piece of advice: when you see that two phantoms have invaded, don't stand out in the open near the cliff. You need to retreat to a bottleneck where you can dispatch one of the phantoms. Alternatively, you can feign a retreat, switch directions last minute, and pivot bs a phantom. You can also use the trees to make it difficult for one phantom to get behind you while you fight the other phantom. There is a lot of good advice all over these boards relating to anti-forest gank tactics. This advice applies equally to hosts who are getting ganked.