Background: Whether the association of height with cancers differs by insulin-like growth factors has not been fully elucidated. Therefore, this study aimed to investigate the sex-specific associations between height and 24 site-specific cancers and to assess whether the association differed by IGF-1. Methods: In total, 414,923 participants from the UK Biobank prospective cohort study were included. The association of height (per 5-cm increment) with incidence and mortality from 24 cancer sites was investigated by using Cox proportional hazard models. Results: The median follow-up was 6.0 years. In men, height was positively associated with incidence risk of all-cause cancer and at five sites (lung, lymphatic, leukaemia, non-Hodgkin lymphoma and melanoma). In women, it was associated with breast, melanoma, lymphatic, non-Hodgkin lymphoma and all-cause cancer. The association was stronger in women than men for all-cause cancer incidence. The strength of the association did not differ by IGF-1 concentration. Conclusions: Adult height was associated with risk of several cancer sites. However, some of these associations were sex-specific. There was no strong evidence to support IGF-1 moderating the association between height and cancer.