Thanks for the explanation. Indeed that makes my idea look quite slow. So back to the drawing board.
Although in theory the idea is fascinating, because the previous frame always approximately contains the same image that we are going to draw next. So if the old data could be somehow used to build the current frame, without too many reads and comparisons, then something like this might be useful.
But right now it's just an interesting theory, and I don't know how to make it work.