Residents of a posh royal town say a "nightmarish" invasion of flies which swarm their homes each summer has become "absolute hell" and is "ruining people's lives".