Many common optimization algorithms can be interpreted as the interconnection of linear systems and operator nonlinearities (e.g. gradient descent, ADMM, Douglas-Rachford). This dynamical systems formulation allows for the analysis and synthesis of optimization algorithms by solving robust control problems. In this work, we form a link between convergence properties of nonlinear optimization algorithms and the regulation theory of linear dynamical systems. We show that the linear internal model principle in control theory applies in this specific nonlinear setting. This permits us to structurally factorize convergent optimization algorithms into the cascade of a parameter-dependent controller and a parameter-independent internal model. The regulation-theoretic interpretation of algorithm convergence extends to posing convergence properties for optimization algorithms operating in dynamic environments (e.g., time delays, channel memory, switching, or crosstalk). Multicriteria analysis and design of these algorithms proceeds by solving convex semidefinite programs subject to these structural constraints.