For a first camera, consider used equipment. KEH, Adorama, and others sell fully functional, inspected bodies with a ~6 month warranty.
$800 should comfortably get you a complete weather kit.
You can get an older, but still very competent camera for $200 or less. Canon, Nikon, Minolta, whatever.
Specifically, look for a camera that has 'live view' manual focusing. This is handy when shooting lightning at night, when the autofocus system won't work worth beans. Water-sealed controls are also a 'nice' feature for a storm camera.
Toss in a super-wide zoom. Tokina 12-24 gets good reviews, but there are plenty of others to consider.
A middle-range zoom or 35/50mm fixed lens.
Tripod and head. None of this "K-Mart Tripod" crap. Get a ~$100 item.
Cable release, a few flash cards, etc.
And you're good to go!
Down the road, when you know what specific features you want in a camera, you can upgrade if you feel the need.