Family Karma's Shaan Patel Reveals He Was Just 'Being Me' When Brian Called Him 'Annoying'