Analyzing the fluid velocity gradients in a Lagrangian reference frame provides an insightful way to study the small-scale dynamics of turbulent flows, and further insight is provided by considering the equations in the eigen frame of the strain-rate tensor. The dynamics of the velocity gradient tensor is governed in part by the anisotropic pressure Hessian, which is a non-local functional of the velocity field. This anisotropic pressure plays a key role in the velocity gradient dynamics, for example in preventing finite-time singularities, but it is difficult to understand and model due to its non-locality and complexity. In this work, a gauge symmetry for the pressure Hessian is introduced, such that when the gauge is added to the original pressure Hessian, the dynamics of the eigen frame variable remain unchanged. We exploit this symmetry to perform a rank reduction on the three-dimensional anisotropic pressure Hessian, which, remarkably, is possible everywhere in the flow. The dynamical activity of the newly introduced rank-reduced anisotropic pressure Hessian is confined to two dimensional manifolds in the three-dimensional flow and exhibits striking alignment properties with respect to the strain-rate eigen frame and the vorticity vector. The dimensionality reduction, together with the striking preferential alignment properties, leads to new dynamical insights for understanding and modeling the role of the anisotropic pressure Hessian in three-dimensional turbulent flows.