What Happens When All Neural-Network Weights Start at Zero?

What happens if every weight in a neural network is initialized to zero? Explain why the answer depends on the architecture and activation functions. Focus on a multilayer network with several interchangeable hidden units, and state the bias initialization when using that example. Contrast the result with a linear or logistic model that has no hidden layer.

Explain the learning limitation caused by identically initialized hidden units and discuss whether all gradients must be zero in every possible network. Your answer should distinguish failure to break symmetry from the specific zero-gradient behavior of a particular activation and loss implementation.