Depending on how much Decay you have accumulated, all your party members will have their All Elemental RES, Physical RES, and Max HP decreased. These stats will decrease more in accordance with the amount of Decay accrued.

If you remain within Withering Zones after you fully accumulate Decay, all your party members will fall.

You must use the power of the Dendrograna to clear the Withering away. Only via this method can you destroy the Withering Branches that spread this corrosive power. Only when all the Withering Branches within a Withering Zone are destroyed can the Tumor of the Withering at the beating heart of the area be destroyed.
Most Withering Zones can be destroyed in this way, but...

Within the Withering Zones, Candles of Life and Flames of Life may appear and help to decrease the accumulation of Decay.
Prepare for the worst before entering the Withering Zones.