No, that one relies on some properties of the binary representation of floating point numbers and makes only sense when there's a reasonably fast fpu present. On an 68000 it would be much slower than directly computing it with fixed point arithmetic, let alone using a table-based approach.
The Wikipedia article on this topic is actually quite good:
https://en.wikipedia.org/wiki/Fast_inverse_square_root
A possible way to speed up coordinate calculations is to exploit object symmetries, e.g. your sphere is highly symmetric, so that there's a lot less independent coordinate values than for a set of random points. This allows you to precompute some of the products in the matrix multiplication: If your coordinate vectors (x,y,z) are e.g. (a,a,a),(a,-a,a),(-a,-a,a),(-a,a,a),(a,a,-a),(a,-a,-a),(-a,-a,-a),(-a,a,-a) for a simple cube, you only have to precompute the product of every rotation matrix element with 'a' and carry out the additions and subtractions. So in that case only nine multiplications instead of 216 for random coordinates. For a generic 3D-engine you'd probably need a bit of clever coordinate transformation ping pong to make it work.