If you average in the 15-20 second range, I would personally advise not to worry too much about it yet, except for the fact that you should definitely try to prioritize front slots. If your back slots do end up being the unsolved ones, don't be too afraid to simply rotate.
However, I often find myself solving a pair that had a piece in a back slot, and then realizing I have no idea how I knew that piece was there. I'm guessing I have accustomed myself to subconsciously taking notes of where all (or at least most) of my F2L pieces are. What helps out a ton as well is knowing all the places a certain piece is not, to the point where there is only one place left for it to be. You then only need the orientation of the piece, for which you don't need to look at the back of the cube (theoretically, not even at the piece itself). I have no idea how I do this without pauses (admittedly, sometimes with pauses), because it sounds extremely hard and 'thought-extensive', but I guess it just comes naturally with practice, as most things do.
