Houston is an average of 50 ft above sea level. The mandatory evac is only for suburbs south of town and on the water (i.e., Clear Lake, Friendswood) and low-lying areas right now, AFAIK.
That said, many of my friends and family are evacuating (I'm in Austin) because the thought of being without power in 100 degree post-Rita weather is not too appealing. Many have small children or are pregnant, so it makes sense to err on the side of caution.
Drive times are crazy. My sister (in HOU) reports that a colleague of hers took 17 hours to get to Dallas (normally 4), and that drive time from Galveston to downtown are 6 hours.