Worse socioeconomic conditions (e.g., having a migration background, lower income, and lower educational level) may lead to barriers for cancer prevention and early detection. Community-based initiatives providing tailored and accessible information are found to increase cancer screening participation and improve health outcomes for people with challenging socioeconomic conditions. To inform such initiatives, more research is needed from the perspective of those working with them. This article focuses on the Swedish “peer advisor project” in which the Regional Cancer Centre Stockholm Gotland (RCC) collaborates with community-based peer advisors to reduce cancer inequities in areas with challenging socioeconomic conditions. We aim to investigate the perspectives of people working with the peer advisor project in relation to challenges they face, strategies they use, and forms of impact they perceive their work to have. We used a participatory action research approach, involving 12 peer advisor representatives and three RCC representatives. Underlying data (interviews and participant observations) were qualitatively analyzed. Findings are presented in relation to three themes: (1) Peer advisors bridge a gap between the cancer care system and communities through dialogues with communities and RCC representatives. (2) Peer advisors navigate culturally based sensitivities through efforts to create safe spaces for discussing cancer prevention and early detection. (3) Peer advisor and RCC representatives described forms of impact in relation to personal contexts, community contexts, RCC’s organizational context, and societal context. Sustainability and structural challenges may limit the peer advisor project to create impact. We discuss that peer advisors may act as cultural brokers through dialogues with communities and the cancer care system, thereby increasing an understanding of communities’ contexts and needs. Findings from this study can inform development and implementation of similar community-based peer-to-peer initiatives in other contexts. More research is needed to investigate the long-term impact of the project including community-based perspectives.