“The moment any of them gave up on the difficult work of living with their neighbors--and all of the compromise, frustration, and delay that inevitably entailed--they risked losing everything.”