Alright, had some spare time to tinker with the rendering again, and now the stream runs at a nice steady 25fps (around 1.6-1.9 frames). The rendering is now done horizontally, with the previously added double pixel blitter pass modified to also rotate the buffer back 90 degrees.
There's still room for improvement in the wall rendering (for example, the code slices could be modified to draw longwords where possible). But I think the next would be to make the raycasting real time for interactivity, although this will probably not be happening any time soon
The usual preview is available
here