Who Defines a Good Answer? Demographic-Conditioned Pluralistic LLM Alignment with Preference Uncertainty and Privacy-Aware Personalization

Abstract

Large language model alignment usually treats human feedback as if it expressed one stable population objective, although judgments of helpfulness, style, safety, and values differ across people and contexts. This study tested whether demographic conditioning and warm-start personalization improved prediction of pairwise response preferences in PRISM Alignment 2024. PRISM links detailed surveys from 1,500 participants to 8,011 live conversations, 68,371 scored model responses, and text-level language, privacy, and moderation annotations. After English-language and personally identifiable information filters, the analysis formed 54,293 unequal-score comparisons within the same interaction. Whole conversations were assigned within participant to training, validation, and test sets, yielding 36,041, 8,966, and 9,286 pairs. A global logistic preference model was compared with partial-pooled demographic residual models and a personalized stack based on prior preference history and nine stated-preference dimensions. Temperature scaling quantified preference uncertainty, subgroup audits measured worst-group predictive performance, and randomized response traced an event-level local privacy-utility frontier. On held-out conversations, the global model achieved AUC 0.693, negative log-likelihood 0.625, and expected calibration error 0.014. Demographic conditioning produced comparable discrimination (AUC 0.692) and slightly lower calibration error (0.013), but personalization reduced AUC to 0.682 and increased negative log-likelihood to 0.634. The personalized-global AUC difference was -0.0106, with a 95% conversation-cluster bootstrap interval of [-0.0163, -0.0052]. Privacy perturbation changed AUC by less than 0.001 across epsilon 0.5-8 because observed-history personalization contributed little test utility. Preference uncertainty depended strongly on rating margin: global AUC rose from 0.547 for 1-9-point differences to 0.855 for 50-99-point differences. The results show that access to demographic and individual data does not automatically produce better alignment. Population conditioning requires strong regularization, independent calibration, subgroup evaluation, and explicit privacy scope.