Exploring the field of affective computing is important for understanding how humans think and interact with each other. Personality computing focuses on methods of performing the automatic detection of human traits which compose their personality. Using the Five Factor Model of Personality as a measure to describe a subject's personality, temperament, and psyche, this work employs a multimodal model to perform automatic personality recognition on speech. We employ the use of speaker and phone disentanglement in speech representation learning, a technique known to be effective in emotion recognition, to predict scores for personality traits trained on the UDIVA dataset and outperform current methods that use visual features.