Isolating the role of building block shape for self-assembly and packing provides insight into the ordering of molecules and the crystallization of colloids, nanoparticles, proteins, and viruses. We investigated a large group of polyhedra whose phase behavior arises solely from their anisotropic shape. At intermediate packing density, our results demonstrate a remarkably high propensity for thermodynamic self-assembly and structural diversity. We show that from simple measures of particle shape and local order in the fluid, the assembly of a given shape into a liquid crystal, plastic crystal, or crystal can be predicted. Towards higher density, packing considerations dominate. Good packings can often be distinct from what is observed to assemble from the disordered state. It turns out that structural complexity is particularly high, when entropy and geometric constraints compete with locally dense configurations.