Do we need a physical office address to form a nonprofit?

Not necessarily. Many nonprofits start with a mailing address, but you still need a reliable address for state notices and a registered agent address where required.