Sub-tropical storms are storms that have both tropical, and non-tropical characteristics. For instance, the storm might have convection more typical of a tropical cyclone, but be cold core (relatively cold air aloft, as opposed to warm air of normal tropical cyclones). They are relatively common in October/November in the North Atlantic (maybe 1 every other year, I'm guessing). They are also fairly common in the eastern North Pacific, north and east of Hawaii, although they are almost never officially warned on. They are also rarely encountered in the Mediterranean Sea, and possibly other ocean basins (though I can't recall seeing any in other places, except for the hurricane in the South Atlantic in March 2004, which started off subtropical). I'm sure others would have more detailed information.
Extratropical cyclones are storms that have transitioned into regular mid-latitude cyclones, with a frontal system, and also get their energy from temperature differential (i.e. jet stream), as opposed to latent heating with normal tropical cyclones.