You should drive on the shoulder to pass a car:

Passing on the right is permissible only if it is possible to do so without driving off the roadway. Never pass another vehicle on the shoulder because the other driver will not expect you to be there and may pull off the road.
DMV Writen Test