I'm wondering when does this approximation hold, and when does it not? For example, is there a certain size scale where you can generally use this?
rho is density
u-east west wind
v-north south wind
w-up down wind
This is the incompressible mass continuity equation. One has to be careful here, because the actual incompressibility condition is that the material derivative of density following the motion is zero. That is, density doesn't change following a given air parcel (i.e. d(rho)/dt = 0). The result of this is that the 3D divergence of the velocity field also is zero (the first part of your equation above: del dot V = 0.
The incompressibility condition does not hold in general for atmospheric flows, since air is compressible (it's density can change significantly due to pressure changes, especially at high speeds or over large vertical length scales, due to the stratification of the atmosphere). However, for small vertical length scales, such as over the depth of the boundary layer, and provided the wind speeds are significantly smaller than the speed of sound, the incompressibility assumption is actually a very accurate approximation and is used extensively in models of the boundary layer. In such a flow, density can still vary spatially, or change in time at a given fixed point.
The anelastic approximation is more often used when vertical length scales are large (such as over the depth of the entire troposphere), since density changes appreciably in the vertical and vertical motions can no longer be assumed to be incompressible. Instead of del dot V = 0, we write del dot (rhobar*V) =0, where rhobar is the "base state density" which is only a function of height. Thus, it is like a less restrictive form of incompressibility, where compressible motions are only allowed in the vertical. This is an excellent approximation for many atmospheric flows (even, especially if your purpose is qualitative understanding, for thunderstorms).
It is convenient in meteorology to use pressure as a vertical coordinate rather than geopotential height. In this case the continuity equation becomes as follows:
du/dx + dv/dy + d(omega)/dP = 0
where d( )/d( ) is a partial derivative
u and v are evaluated on isobaric surfaces
omega is defined as dP/dt and represents vertical motion
Correct, but this can be misleading in the context of the original question because the continuity equation in pressure coordinates as you have written is actually *not* incompressible. It is in fact the full continuity equation (d(rho)/dt = del dot V) re-written in pressure coordinates. The advantage to doing so is that density disappears explicitly from the equation, making it look a lot like the incompressible continuity equation, but it is still there, hiding