The time certainly does make a difference as prices go up after about 9.30 in the evening. It shouldn't cost you more than 15 - 20 dinars, and probably quite a bit less if you manage to haggle. Going to Sousse (which is not quite so far) in April I got charged 20 dinar as it was late at night and I couldn't be bothered to stand my ground, but going back for a morning flight the driver had the meter on and it showed just under 6 dinars.