In paired-end sequencing, the library preparation yields a set of fragments, and the machine sequences each fragment from both ends; for example if you have a 300bp contiguous fragment, the machine will sequence e.g. bases 1-75 (forward direction) and bases 225-300 (reverse direction) of the fragment.
In mate-pair sequencing, the library preparation yields two fragments that are distal to each other in the genome and in the opposite in orientation to that of a mate-paired fragment.
The three read orientation categories are forward reverse (FR), reverse forward (RF), and reverse-reverse/forward-forward (TANDEM). In general, paired-end reads tend to be in a FR orientation, have relatively small inserts (~300 - 500 bp), and are particularly useful for the sequencing of fragments that contain short repeat regions. Mate-pair fragments are generally in a RF conformation, contain larger inserts (~3 kb), and enable sequence coverage of genomic regions containing large structural rearrangements. Tandem reads can result from inversions and rearrangements during library preparation.
Here is a more illustrative example:
FR: 5' --F--> <--R-- 5' (in slang called "innie" because they point inward)
RF: <--R-- 5' 5' --F--> (in slang called "outie" because they point outward)
TANDEM: 5' --F--> 5' --F--> or <--R-- 5' <--R-- 5'
The figure below illustrates this graphically along with the SAM flags that correspond to the FR and RF configurations.
For detailed explanations of library construction strategies (for Illumina sequencers) and how read orientations are determined, please see: