So many variables.
There will always be debates about categories of classification for tornadoes, hurricanes and similar weather related events were structures, and lives are affected. There are continuing research, politics, economics, and other factors that go into such assessments. Different people assess different storms, and standards change. I think the thing to remember in classifications as high as (E)F4 or (E)F5 is that both are catastrophic events, and that each tornado has unique circumstances that warranted it's designations. Also remember that there is a certain amount of variability in these assessment give the amount of time, expertise, and data available for evaluating such ratings. There are just so many variables it is nearly impossible to compare one tornado to the next as too why one got a F4 versus another getting an F5.
Now...since I have somewhat of a design background I'll speak to that. Structures come in all kinds of materials, structures, connections, etc. There are different shapes, sizes, designs and more that can significantly change how one structure compares to another. For example....in traditional dwellings in tornado alley you may find a well built farm house made of more modern built materials like pine wood studs, plywood sheathing, vinyl siding and plywood and asphalt shingle roofing. Very common design. Now if was a couple of decades ago it likely won't have the best foundation connects. Probably simple anchor bolts to pier foundations or CMU (block wall) perimeter foundation. They are usually held with a washer and bolt over the "top plate" which is a single 2x6 or larger length of wood. And then simple nailing and old fashion floor truss hangers are used. But in more modern structures built more recently, some homes are adding more anchors like those used in hurricane zones (aka the Dade County Code). These can make a significant change in the appearance of tornado damage after the affect and may not at first glance seem totally justified by the given rating. One of my more famous examples of this was with the (local to me) LaPlata, Maryland tornado that was heavily debated, but in the end it was difference in building types that gave a more confident justification to it's rating.