You do not have to follow directions given by a flagger in a construction zone.

Be prepared to reduce your speed and use caution when directed to do so by a sign, flagger, or law enforcement officer. Failure to do so is illegal and is dangerous to those working in the construction zone.
DMV Writen Test