Heart assembly requires input from two populations of progenitor cells, the first and second heart fields (FHF and SHF), that differentiate at distinct times and create different cardiac components. The cardiac outflow tract (OFT) is built through recruitment of late-differentiating, SHF-derived cardiomyocytes to the arterial pole of the heart. The mechanisms responsible for selection of an appropriate number of OFT cells from the SHF remain unclear. Here, we find that cell adhesion molecule 4 (cadm4) is essential for restricting the size of the zebrafish OFT. Knockdown of cadm4 causes dramatic OFT expansion, and overexpression of cadm4 results in a greatly diminished OFT. Moreover, cadm4 activity limits the production of OFT progenitor cells and the duration of their accumulation at the arterial pole. Together, our data suggest a role for cell adhesion in restraining SHF deployment to the OFT, perturbation of which could cause congenital OFT defects.