Replication foci are generated by many viruses to concentrate and localize viral DNA synthesis to specific regions of the cell. Expression of the HPV16 E1 and E2 replication proteins in keratinocytes results in nuclear foci that recruit proteins associated with the host DNA damage response. We show that the Brd4 protein localizes to these foci and is essential for their formation. However, when E1 and E2 begin amplifying viral DNA, Brd4 is displaced from the foci and cellular factors associated with DNA synthesis and homologous recombination are recruited. Differentiated HPV-infected keratinocytes form similar nuclear foci that contain amplifying viral DNA. We compare the different foci and show that, while they have many characteristics in common, there is a switch between early Brd4-dependent foci and mature Brd4-independent replication foci. However, HPV genomes encoding mutated E2 proteins that are unable to bind Brd4 can replicate and amplify the viral genome. We propose that, while E1, E2 and Brd4 might bind host chromatin at early stages of infection, there is a temporal and functional switch at later stages and increased E1 and E2 levels promote viral DNA amplification, displacement of Brd4 and growth of a replication factory. The concomitant DNA damage response recruits proteins required for DNA synthesis and repair, which could then be utilized for viral DNA replication. Hence, while Brd4 can enhance replication by concentrating viral processes in specific regions of the host nucleus, this interaction is not absolutely essential for HPV replication.