'There's almost a sadness': Harry Styles has found fame to be isolating at times