General:
* A cluster of branches
* A thicket of branches
* A tangle of branches
Specific:
* A bough of branches (referring to large, heavy branches)
* A sprawl of branches (referring to widely spread branches)
* A canopy of branches (referring to branches forming a dense overhead cover)
Ultimately, the best collective noun depends on the specific image you are trying to convey.