Triboar attack

Hey! My player are in Triboar and i’ve made the decision to split the attack in two, the giants that goes for the vonindod fragment and the orogs that goes for the townfolk. The party went to defend the citizens and leave the giants digging. They succedeed, but now i’m not sure if i’m still going to use the giants and the orcs. I don’t want the players to think that the siege was to easy for them.

What would you guys have done? Continue the siege but the giants are trying to leave with the fragment or make them already left during the players focusing on the orogs and magmins?