Introduction

Despite dramatic declines in HIV incidence in Eastern and Southern Africa there were 450,000 new infections in 20231, the majority of which occurred among members of the general population2. UNAIDS 2030 targets for ending AIDS may be missed and HIV prevention programmes require strengthening1. Accurate targeting of prevention requires good information on who is most at risk of acquiring HIV and would help achieve these targets while being cost-effective to deliver.

Studies have explored risk factors for HIV acquisition in the general population across sub-Saharan Africa3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27. Generally, these found higher incidence in women compared to men and variations in incidence by age3,11,14,22,27. The most commonly identified risk factors for HIV infection were: being unmarried (in five populations)6,10,11,15,18,20,21,22,26,28, having multiple partners (in four populations)11,14,18,20,21,22,26,28,29, having a sexually transmitted infection (STI) (in six populations)6,11,12,14,15,19,20,23,28,30,31,32 and alcohol use (in two populations)6,11,15,20,21,26,33 (see Supplementary Material Appendix 1 for detailed description of literature review). However, comparison of risk factors across different populations is hampered by differences in analytical approaches between the studies, in risk factors evaluated, period of study and the age ranges.

Understanding drivers of new infections is essential for effective prevention. Structural determinants and more proximate individual-level factors are relevant and some will be amenable to change. To identify potential risk factors for this analysis we devised an analytical framework (Supplementary Fig. 1), initially based on the proximate determinants framework set out by Boerma and Weir34 adapted for an individual-based risk factor analysis and supported by our literature review. We start with the sequence of events needed for a susceptible person to become infected with HIV followed by a representation of how both the proximate and more distal risk factors influence different stages in that sequence. A risk factor may exert a differing degree of influence on different stages which might partly explain the varying results obtained in different analyses.

The sequence begins with a susceptible index person seeking, then meeting, a potential new sexual partner who may have HIV and may be viraemic. If they have sex, and do not use a method of HIV prevention, the susceptible person may become infected. If this does not happen, and the person remains susceptible, they may continue the partnership and the cycle will repeat. During the partnership the partner may become infectious for HIV and place the index person at risk. Within our framework we have identified the distal and proximate, measurable factors and which events in the sequence they may influence.

The prevalence of infectious individuals in the population and the chances of a susceptible person encountering an infectious one will greatly modify the effects of risk behaviours, such as sex without using a condom. The prevalence of infectious individuals depends on other people’s risk behaviour, treatment uptake and adherence.

We investigated individual and community-level risk factors for incident HIV infections among adults in the general population in Kenya, Malawi, South Africa, Tanzania, Uganda and Zimbabwe. Kenya, Malawi, Tanzania and Zimbabwe are on track to achieve the UNAIDS 2030 target for reductions in incidence whilst South Africa has made moderate, and Uganda only limited progress towards this goal35. To understand the contribution of distinct, and ideally modifiable, behaviours and characteristics, we created variables that aligned with our analytical framework and were independent of other aspects of behaviour and individual circumstances (see Table 1). We looked for associations which were generalisable across settings in both the direction and magnitude of effect.

Table 1 List of measures identified in the conceptual framework with their corresponding definition in the data available for analysis
Full size table

In this study, we find that education, marital status, male circumcision, new sexual partners, types of partner, prevalence of untreated HIV infection in the community and community partner acquisition rates are associated with HIV incidence. We show that the prevalence of risk factors and the nature of their associations with HIV incidence differs by age and sex.

Results

Data available

Between 2005 and 2016, 99,353 initially HIV-negative individuals aged 15 to 49 were observed and tested for HIV at least once following an initial negative HIV test (43,434 men and 55,919 women). They contributed 349,201 person years (58% from women) and there were 4,612 seroconversions (72% among women). The largest study, was uMkhanyakude (23,794 people and 2,876 serconversions) and the smallest Ifakara (1,451 people) (Table 2).

Table 2 Description of study population and prevalence of potential risk factors
Full size table

Data on sexual partners and recent behaviour was available for almost 90,000 person years. Coverage was incomplete, because data were not sought for all periods in all studies, and around 90% of respondents contributed time to the unknown category for these variables. There was less person time without behaviour data for men (e.g., 65% for number of partners) than for women (72%).

Prevalence of potential risk factors

The distributions of person-years, by age and sex, for each of the potential risk factors are shown in for the pooled cohort in Table 2 and separately for each study, by sex, in Supplementary Data 1 and 2. These are described in detail in Supplementary Methods.

Most respondents lived in rural areas, or in small urban or peri-urban areas within a predominantly rural setting. Around a quarter of people reported moving house during the period of observation. People with primary education contributed approximately 60% of person-time.

Men and women differed in their behaviour and this also varied by age.

Young men spent 60% of their time with no partner, 30% with one and 10% with multiple partners. 95% of their time was prior to marriage. Young men also spent the least time in age disparate partnerships: 73% of their time in partnerships was with partners aged within 5 years of their own age and less than 1% was spent with partners where the age gap was more than 10 years. New partnerships were most common among young men who spent 23% of their time in partnerships that were less than a year old (Table 2 and Supplementary Data 1).

Older men spent the most time in partnerships, with only 10% of time without a partner, and the most in multiple partnerships (27% of time). A quarter of their time was prior to first marriage or cohabitation. Of their time spent in partnerships, 46% was with a partner within 5 years of their own age and 21% was with partners where the age gap was more than 10 years. New partnerships accounted for 20% of older men’s time in partnerships (Table 2 and Supplementary Data 1).

Young women spent 48% of their time without a partner and 51% was with one partner. 75% of time was spent unmarried and 22% with a married or cohabiting partner. Age differences between partners were not universal and for 54% of young women’s time in partnerships, the age gap was less than 5 years. 11% of young women’s time was in the first 12 months of a partnership (Table 2 and Supplementary Data 2).

Older women spent 15% of their time with no partners and 84% with one partner. Around 60% of time was spent married or cohabiting with a partner, except in uMkhanyakude where most of the time was spent unmarried. Althought much of the time spent in partnerships was with partners similar in age (43%), substantial portions were with partners who had a 5 to 10 year age gap (28%) and those where the age gap was more than 10 years (17%). Just 6% of older women’s time was spent in the first year of a partnership (Table 2 and Supplementary Data 2).

Information on partnership type was not regularly collected in every study. Where available, the proportion of time spent in regular partnerships was similar by age and sex, ranging from 22% to 30% of person-time in the pooled data (Table 2 and Supplementary Data 1 and 2). Older men spent a similar amount of time with casual partners (21%) but women and younger men reported less (10-12%).

Male circumcision status was also inconsistently collected but, where known, the proportion of young men’s person-time post-circumcision ranged from 6% in Masaka to 92% in Ifakara. In total, 14% of the time contributed by young men for this analysis was known to be post-circumcision, 45% of the time where status was known. For older men, 18% of time was post-circumcision, representing 63% of the time for which status was known.

Differences between the studies were observed in marital status, numbers of partners, types of partner, new partnerships, and condom use (see Supplementary Data 1 and 2). The variation observed across the studies was not predictable, in that these aspects of behaviour did not differ in the same way between studies.

Associations with HIV acquisition in pooled data

Table 3 gives the incidence rates (per thousand person years) derived from multiple imputation and crude hazard ratios (HR).

Table 3 Crude HR and 95% CI for HIV seroconversion in pooled data, by age and sex, adjusted for study
Full size table

The adjusted HR from the preferred multivariate models are shown in Table 4 and Supplementary Data 7–10 present the comparison multivariate models. The different versions of the models gave broadly similar results. Models including the prevalence of untreated infection and calendar year did not show an effect of both; we opted to include untreated prevalence in our preferred model, since it is the more proximate determinant of infection. For each sex and age group, this was model 4 in Supplementary Data 7–10, which included a fixed effect for study, the prevalence of untreated infection and the other risk factors but did not include age and calendar year.

Table 4 Results from models of HIV incidence for each age and sex group, for 2005-16 inclusive, showing adjusted HR and 95% CI
Full size table

In the crude analysis, associations with calendar year, prevalence of untreated infection, having a regular partner and having a new partner were seen for men and women in both age groups. Incidence was lower in 2013-16 compared to 2005-08 with crude HR ranging from 0.60 (95% CI 0.46-0.79) in older women to 0.70 (95% CI 0.61-0.81) in younger women (Table 3). Having new, regular or casual partnerships compared to all other partnership configurations (including no partner), were risks for HIV acquisition. There was strong evidence that the community prevalence of untreated HIV infection was associated with higher incidence hazards in all groups and this was the only association that remained apparent for men and women in both age groups in the adjusted for analysis (Table 4). The effect was strongest for young men where a one percentage point increase in this prevalence was associated with a 7% increase in the hazards of HIV acquisition (Adj. HR 1.07, 95% CI 1.05-1.09). The association with the prevalence of untreated infection was not seen in the models which included age and calendar time (Supplementary data 7 to 10). It is correlated both with age and with calendar time and, when included in the same models, these variables more efficiently explain the variation in the outcome.

In the crude analysis, characteristics of partnerships were consistently associated with acquisition risk for men and women in both age groups but once adjusted for other factors these associations differed by sex and by age. In the crude, there was strong evidence for an association with regular partners in all groups with HR ranging from 1.75 to 3.17. For casual partners the strongest evidence was for young women (crude HR 1.57 95% CI 1.22-2.02) and older women and men had only weak evidence for a more modest association (crude HR range 1.27-1.42). Strong evidence for associations with new partnerships were seen in all groups with HR ranging from 1.5 for men to 2.5 for older women (Table 3).

Young men

For young men, having moved house in the last year was associated with increased risk of HIV acquisition (crude HR 1.68 95% CI 1.11-2.55) but this was not seen in the adjusted model after adjusting for partnership factors. In both crude and adjusted models there was weak evidence that secondary education was protective (HR 0.72 95% CI 0.49-1.06) (Table 4).

The strong protective effects of being unmarried (adj HR 0.38 95% CI 0.23-0.61) and being circumcised (adj HR 0.38 95% CI 0.21-0.70) were the only individual-level factors for which strong evidence of association with HIV acquisition was seen in the crude and adjusted analysis.

In the crude analysis, young men with no partners experienced a lower hazard of HIV acquisition than men with one partner (crude HR 0.62 95% CI 0.43-0.91) while those who had multiple partners (crude HR 2.32 95% CI 1.46-3.70) had higher risk. There was no evidence for an association with the age gap between partners. After adjustment, there was no evidence for the association of partner type with acquisition of HIV by young men with the adjusted HRs tending towards 1. There was weak evidence for a diminished association with multiple partners (Supplementary Data 9).

Older men

For older men, in the crude analysis there was weak evidence for increased risk of HIV acquisition following a house move (crude HR 1.30 95% CI 0.94-1.81) but, after adjustment, this could have been due to chance. A lower hazard of HIV acquisition among men with primary education in the crude analysis reversed, following adjustment, to a declining trend with increasing education but these associations may have been due to chance.

In the crude analysis, current marriage was protective for older men, with increased risks for the never and formerly married men but, once adjusted for having a partner, only the formerly married had higher risk of HIV acquisition (adj HR 1.41 95% CI 1.06-1.86). The HR for having multiple partners were positively associated with HIV acquisition but ths may have been due to chance (Supplementary Data 10). HIV acquisition hazard decreased as the age gap with partners increased, but this may also have been a chance finding (Table 3). Among older men, inconsistent condom use was associated with higher hazard of HIV acquisition in the crude analysis (HR 1.91 95% CI 1.35-2.69) but not in the adjusted. Unlike for younger men, there was no evidence for an association with male circumcision.

In the adjusted model having a regular partnership remained associated with risk for older men (Adj. HR 1.82 95% CI 1.24-2.68, Table 4) but the effect of new partnerships was diminished and the confidence intervals were wide (Supplementary Data 10).

Young women

For young women, having moved house in the last year showed a crude association with increased risk of HIV acquisition (crude HR 1.29 95% CI 1.04-1.60) which disappeared once adjusted for other factors. Secondary education was protective when compared with primary (adj. HR 0.74 95% CI 0.59-0.92). Being formerly married was associated with increased risk of HIV acquisition compared with current marriage (adj. HR 2.71 95% CI 1.63-4.50). Young women in casual partnerships had higher HIV incidence compared to young women with other partnership types (including no partner) (adj. 1.82 95% CI 1.38-2.39). Regular partners were also a risk for young women (adj. HR 1.70 95% CI 1.39-2.08). The association with new partnerships remained for young women (adj. HR 1.35 95% CI 1.07-1.72) but the increase in risk with higher number of partners that was seen in the crude did not persist in the adjusted analysis. In the crude analysis there was a suggestion that HIV acquisition hazard increased as the age gaps between partners increased but this may have been due to chance and there was no evidence for this association in the adjusted model.

Older women

For older women, having moved house in the last year was associated with increased risk of HIV acquisition (adj HR 1.50 95% CI 1.19-1.88). Unlike for younger people, among older women secondary education was associated with a higher chance of HIV acquisition (adj HR 1.25 95% CI 1.01-1.53). The risk of HIV acquisition in older women who were married was around half that experienced by those never or formerly married. The association with new partnerships remained for older women (adj HR 1.87 95% CI 1.29-2.71). Inconsistent condom use was associated with acquisition of infection among older women (adj HR 2.06, 95% CI 1.59-2.67).

In the crude analysis, there was a trend of increasing HIV acquisition hazard with higher number of partners which was not seen in the adjusted model. There was weak evidence that older women with a larger age gap between their partners had lower hazards of HIV acquisition (crude HR 0.65 95% CI 0.41-1.01), the opposite effect to that observed among young women, but this did not persist after adjustment for other characteristics of the partners.

Associations between community level factors and HIV incidence

Community rates of partner acquisition (CPAR) were available in Karonga, Kisesa and Rakai, and were highest for young men, and higher for men than for women. Community partner loss rates (CPLR) were lower than acquisition rates, except for older men (Table 2) and were not associated with incidence. CPAR among peers showed a crude association with incidence for men and older women (Table 3) and men’s PAR was associated with risk for older women. In the adjusted models, men’s PAR remained associated with HIV acquisition for older men and older women (Supplementary Data 11 and 12). The effect of a one percentage point in the men’s partner acquisition rate was a 3% increase in the HIV incidence hazard for both older women and older men (adjusted HRs 1.03, 95% CI 1.004-1.05 and 1.03, 95% CI 1.01-1.04, respectively).

Sensitivity analyses

To assess whether the results were unduly influenced by any one study, we repeated the models above for each study and did a meta-analysis to combine the estimates leaving one study out each time. The results in Supplementary Table 2 show the estimates obtained are not determined by any single study.

To see if the risk factors were different in the most recent period we repeated the preferred models for data from 2013-16 only and restricted to the studies which had data for this period: Kisesa, Masaka, Rakai, uMkhanyakude and Kisumu (Supplementary Data 13). For older men, the HR were similar to the model using all years but the confidence intervals were wider. New partners did not show an association for older women but other results were similar. For young men and women the only difference was that the confidence intervals for casual and regular partner estimates widened to include one.

Discussion

This multi-country study combined eight open cohort studies that followed almost 100,000 people for 349,140 person years, observed 4,617 new infections and collected detailed demographic and behavioural data. This large harmonised dataset facilitated an extensive exploration of risk factors for HIV incidence in the general population.

For young women, former marriage and having regular, casual and new partners were associated with increased risk of acquiring HIV while secondary education was protective. For older women, being unmarried, having changed residence in the last year, having a new partner and inconsistent condom use were associated with risk and tertiary education was protective. For young men, never being married and being circumcised were associated with lower risk and no behavioural risk factors were identified. In contrast, there were few protective factors for older men, among whom former marriage and having a regular partner increased the risk. The community prevalence of untreated HIV infection was associated with HIV acquisition among men and women in both age groups. The rate of partner acquisition by men in the community increased the risk of HIV acquisition among older men and women. The relative paucity of identified risk factors for men compared to women suggests that there were either unmeasured factors which affect HIV incidence among men or the correlation between reported behaviour and risk was weaker among men. For young men, the strong association with the prevalence of untreated infection, combined with the lack of behavioural risk factors may indicate that their risk is affected by characteristics of partners besides their number and relationship. A similar pattern of association by sex was observed in the PopART trial in Zambia and South Africa36

We found associations that were consistent across the different studies and also observed variations, by sex, age and study in the prevalence of risk factors. We have added information for populations in Malawi where risk factors in the general population have not previously been described. All of these had been identified as risks by other studies, and many have been used as components of HIV risk scoring algorithms but this is the first time they have been compared in a single analysis and shown to be consistently associated with risk in different populations37. The effects of some risk factors varied by age and/or sex, which matters where prevention activities are targeted a priori by age and sex. Former marriage was a risk for all except young men. Only the prevalence of untreated infection was associated with acquiring infections for both men and women and for both age groups, as seen elsewhere38,39. New partnerships were a risk for women but not men.

The effect of CPAR on incidence shows that other people’s behaviour can have a demonstrable effect on individual risk. This has also been reported elsewhere with a correlation between CPAR and HIV prevalence seen using national level data and results from the PopART trial showed men’s behaviour at the community level to be predictive of incidence among women36,40. Communities with higher CPAR will have a denser sexual network, with new partnerships more closely spaced in time, increasing STI risk41,42.

We did not find associations with multiple partners, which had been seen in other studies6,9,14,18,20,21,22,26,28,43. This may be because the risk of multiple partnerships is experienced at the beginning of the most recently acquired partnership and we controlled for this with variables describing new and casual partnerships11,14,18,20,21,22,26,28,29.

Whilst effective against HIV infection, condom use was difficult to incorporate into these risk factor models, partly because it was not common. In an analysis of the general population, where most people have very low levels of HIV risk, condom use can be a marker of risky behaviour even in a generalised epidemic setting. Our results indicate condom use reported by older women was insufficient to offset the increased risk they experienced. It suggests that at least one person in the partnership was aware of a risk of HIV transmission and the need for prophylaxis, which could make members of this group open to using PrEP. However, the older women may not be the instigators of condom use, or in a position to obtain and use PrEP and they would also be protected by effective treatment of the male partners.

We did not find universal associations between different types of partnership and acquisition of infection. This may be an artefact of our analytical approach. We had a separate category for new partnership, which encompassed many of the casual partnerships and was generally predictive of incidence. Fewer people were in casual partnerships than regular so power to detect an association was lower. Semantic differences in the terms used for partner type, due to translation and mapping local descriptions onto standard terms, might have led to some misclassification of partner types.

This highlights a problem of labelling particular types of partnership as those which present a greater risk of HIV transmission. Partnerships frequently evolve over time, and what begins as casual (and new) can progress to something regular and onwards to marriage. This complicates assessment of the effect of condom use. Where condoms are used, it is most often at the start of the partnership and they are frequently discontinued as the relationship matures, particularly if childbearing is intended. If couples do not test and disclose before discontinuing condom use then adequate condom use in the initial, casual, stage may postpone HIV transmission until later, when the partnership has become more stable. We know that condom use levels vary by type of partnership and we observed differences by study, age and sex (Table 1 and Supplementary Data 1 and 2) so these might contribute to the differences in partnership risks44. The level of risk in different partnerships may be further modified by patterns of partner acquisition for people in partnerships wherein concordant negative partnerships can become serodiscordant. The number and type of existing partners affect the chances of acquiring a new partner. People with partners are less likely than single people to acquire a new partner and the extent of the difference varies by sex, country and the type of partnership40. Differences in the extent of mutual monogamy between marital, regular and casual partnerships will affect incidence risk, unless this is offset by condom use or PrEP. Our analyses were adjusted for partner acquisition and condom use at the individual-level but we did not capture population-level differences in partnership characteristics. Selection effects caused by differences in partnership behaviours might also explain the different effects of circumcision for younger and older men. Our results suggest that older men who are circumcised have a different risk profile to younger men, one that has not been captured in our analysis. Additionally, we were unable to distinguish medical from traditional circumcision, and older men may be more likely that younger men to have undergone a procedure that is less effective for HIV prevention.

The importance of new partners and the largely protective effect of marriage, also seen by others6,9,10,15,18,21,22,28,43, implies that people navigate periods of increased risk before arriving into the relative safety of a marriage. The protective effect of marriage may be overstated by this, and other, similar analyses. Discordant couples who have sex before marriage and do not take steps to prevent transmission of infection are likely to become seroconcordant. Marriages that start out, or become serodiscordant, will likewise remain serodiscordant for only a short period in the absence of prevention measures. Work from Rakai showed that a large proportion of new infections could be linked to an incident infection in a household contact7. Therefore, people included in an incidence analysis (who are HIV-negative) and initially observed while currently married are likely to have an HIV-negative partner or be taking effective steps to prevent HIV transmission from a positive partner. We considered looking only at recent marriages to investigate this but did not have sufficient statistical power.

The prevention response to any association with partner type would be similar: the use of condoms or PrEP, testing and disclosure and effective treatment for PLHIV. Our results show that using one universal definition of a ‘risky’ type of partnership as the trigger for intervention could fail to capture people before they are exposed to risk. It could be more useful to think about how people navigate transitions in their partnerships and what is needed to make these transits safe, rather than focussing on people who have already spent some time in a particular category of partnership. Finding out how to identify people who are about to make risky transitions could be useful, for example, the substantial increase in risk associated with a new partnership would be best addressed before the person entered into the partnership, not after they have done so.

Harmonisation and secondary analysis of disparate sources of data inevitably introduces some limitations and can be especially complicated for sexual behaviour, where reporting is subjective and may be affected by social desirability bias. Different survey frequencies and incomplete participation in every survey round led to person time being classified as “unknown” for many aspects of behaviour and the extent of this varied between studies and over time. The different sizes and durations of the studies meant some contributed more data than others. These differences could have affected the results as availability of data was correlated with the study and calendar time, which were also associated with HIV incidence. We mitigated against this by including a fixed effect for study in all models and by conducting a sensitivity analysis. In some models, some of the “unknown” categories retained an association with HIV incidence, possibly because people with missing data were different (e.g., due to work or travel patterns) or because there was residual confounding by study or calendar period. Sensitivity analysis showed the results were robust to the omission of each study, and there was not sufficient study heterogeneity to vitiate the pooled estimates. Our population-level measures were estimated based on data for the entire study area, which may have masked geographic heterogeneity in risk. Without good information on the extent of mixing between residents of different areas of the studies it was not possible to produce estimates that were meaningfully geographically stratified. By using the average across the whole area our measures may be less precise but, given the small size of most of the studies, and the detection of plausible associations we believe this to be a minor limitation.

Using harmonised data meant the omission of some information on wealth, alcohol use and sexual practices which have been shown to have important associations with HIV acquisition3,6,11,15,20,21,26,33,36,43. We do not think their omission has affected our results. Wealth is difficult to harmonise as the locally relevant measures are not easy to translate across contexts. We did not attempt to do this because we expected the effects of wealth on behaviour, and thus HIV risk, to differ between studies, as has been seen elsewhere40. Most studies had some information on alcohol use and specific sexual practices but these were for different time periods. They are very proximate factors which may change with each encounter and some of the information is imprecise, which introduces noise in the associations when they are included in regression analysis of whole populations over a long timescale. Proximate measures such as these, and condom use and PrEP, are difficult to capture at this scale. There was little PrEP available in the the study populations at the time of data collection. How the availability of PrEP has affected the associations we have shown will depend on how these are correlated with access to and uptake of PrEP. Particularly for women our risk factors speak of periods when women might have less agency and power; in those moments PrEP could be transformative and might remove that excess risk, but those same factors might also hinder its use.

The socially-conditioned and subjective nature of self-reported data on sexual behaviour is a perennial problem. We believe these data to be high-quality because there are high levels of trust between the study communities and researchers in these long-running projects and, because each study is bespoke to a small area, the enquiries are tailored to the local population45. Differences in the local meaning ascribed to descriptions of partners, alongside social desirability bias, may have led to the quite large differences in the prevalence of reported partner types. Finding coherent associations between several hypothesised risk factors and HIV incidence shows the measures, even if inaccurate, are capturing differences in HIV risk.

The associations identified here related to new infections that occurred between 2005 and 2016 and the risks we have identified may be less important today. Treatment was available for this entire period though availability was limited in some studies before 2010 and treatment eligibility expanded over time. Sensitivity analysis limited to 2013-16 data suggests that the risk factors in that period were similar to those identified for the whole period. Many were identified as risks in the late 1990s and early 2000s, at very different phases of the epidemic. This consistency over time could mean that they remain important going forwards.. An analysis of slightly more recent PHIA data found only two risk factors for women, living in a subnational area with high HIV-1 viremia and having a non-cohabiting partner but this was based on a small number of recent infections which may have precluded identification of other risks46. There are no studies with recently collected data on HIV risk factors and their association with documented HIV seroconversion in the general population in sub-Saharan Africa. This is likely explained by changing research priorities and funding, meaning more recent data are not available for most of the ALPHA cohorts, and due to the length of time it takes to harmonise data for multicountry studies. Initiatives such as the African Population Cohorts Consortium will increase the speed with which results can be produced47.

We observed a large number of respondents, but they came from just eight districts in six countries so may not represent the experience elsewhere in these countries, or in other countries in the region. That the results are stable across the different studies, which represent a diverse range of peoples, gives us confidence that these findings can be more widely applicable.

Our results raise the possibility that the timing of prevention intervention relative to behaviour might be important. The importance of new partnerships and former marriage suggests that the liminal periods around partnerships’ beginning and end could be the start of periods of increased risk. The identification of community level risk factors risk point to the utility of using community characteristics to target prevention, perhaps before individual behaviour escalates risk, and even of altering these characteristics to ameliorate that increased risk. Other authors have shown substantial age and spatial variation in the prevalence of risk factors48, that people cycle in and out of periods of increased risk49, and the utility of community level measures to target prevention46 and these findings reinforce theirs. This could be worthwhile in contexts, such as mature generalised epidemic settings, where the “high risk” behaviours are not atypical behaviour but things most people experience or do at some point. Our measures of both untreated prevalence and partner acquisition can be calculated from surveys, such as DHS. Continued effective prevention, in the face of falling incidence, requires better targeting to reach those most in need whilst remaining cost effective and these could be two new approaches to improve that targeting. Demographic and social changes, such as urbanisation, greater mobility, and increasing use of the internet and social media may change the profile of the population at higher risk, which emphasises the need for continued monitoring of HIV incidence and sophisticated and evolving assessment of risk, alongside sustained prevention, diagnosis and treatment.

Methods

Study design

We used open cohort data from eight independently run general population cohort studies which are members of the Network for Analysing Longitudinal Population-based HIV/AIDS data on Africa (ALPHA): Ifakara and Kisesa in Tanzania, Karonga in Malawi, Kisumu in Kenya, Manicaland in Zimbabwe, Masaka and Rakai in Uganda and uMkhanykaude in South Africa50. We used harmonised data from these studies covering people observed between 2005 and 2016, the last date at which most studies collected HIV data.

Ethical approval for this data harmonisation and analysis was granted by the ethics committee of the London School of Hygiene and Tropical Medicine (number 15472) and each study’s data collection and analysis was approved by the relevant local committees. Data used in this analysis are not publically available but may be made available to bona fide researchers upon successful application to each study’s data access committee.

Participants

Study participants were included in the incidence cohort if they had participated in the study’s HIV testing at least twice whilst resident in the area and their earliest result was HIV negative. In this analysis we used data from people who were aged between 15 and 49.

Procedures

Several times a year, with the exception of Manicaland51, each study collected demographic information from household heads in all consenting households within their study areas. For eligible adults, after obtaining informed consent HIV tests and individual surveys, which included questions on HIV-related risk behaviours and socio-demographic characteristics, were carried out at regular intervals, typically between one and three years apart.

Each study is autonomous and data were not standardised prior to collection. Data were harmonised taking into account variation between study sites and over time (see Supplementary Methods for further details).

We identified and constructed a list of common explanatory variables and prioritised those which captured a discrete aspect of behaviour or personal characteristics and were available for most participants. Table 1 shows the measures we reviewed and those we included. We developed a population-level measure to describe the prevalence of untreated HIV infection in people of the opposite sex, resident within the study area,who were in the age range to be sexual partners, constructed using HIV prevalence, ART coverage and the distribution of age gaps between partners. We calculated other population-based measures of risk describing partnership dynamics: the partner acquisition rates40 and rates for the dissolution of partnerships. For both rates, the denominator was person time in the year before the survey, expressed per 100 person years. The numerators were, for acquisition, having had sex with a new partner in the last year and, for dissolution, the ending of a partnership (dated to the time of last sex) with a person the respondent had sex with at some point in the year before the survey. We estimated these rates for peers of the same sex (from the same study and calendar period and similar in age) and also for opposite sex potential partners.

Statistical analysis

The outcome was HIV seroconversion, defined as an HIV positive test in a participant who was resident at the time of the test and who had previously tested negative for HIV. We used multiple imputation to assign a seroconversion date in the interval between the last negative and first positive test dates with a uniform distribution and fitted piecewise exponential regression models to the survival time elapsed between the date of the first negative test and the date of the last negative test or seroconversion52. For all estimates we used 70 imputations and combined the estimates using Rubin’s rules53. Piecewise exponential models were used because they allowed the shape of the hazard function to vary between imputations and captured that variation in the terms for age group.

Subjects had missing data for portions of their follow up time because either information was not collected in every round, participants missed one, or more, survey rounds or due to a small amount of non-response to particular questions. When information was not available, we retained the person time in the analysis and assigned the relevant period of person time to an “unknown” category. People who remained HIV negative were censored at the time of their most recent HIV test.

We wanted to assess if the effects of some risk factors differed by age and sex so analysed four groups of respondents: young men aged 15-24, older men aged 25-49, young women aged 15-24 and older women aged 25-49. Participants whose follow-up time spanned the two age groups contributed to both groups, i.e., they were censored at their 25th birthday in the younger age group and entered the older age group as they turned 25.

We described every risk factor for each age group, study and calendar period using the distribution of person-years for categorical variables and the mean for continuous variables. We estimated the study-specific crude hazard ratio (HR) for new HIV infections and made the same estimates using data pooled from all studies, adjusted for study. We examined the crude HRs for each risk factor and, where study-specific effects appeared to differ based on both HR and p-values, fitted interaction terms to assess effect modification by study.

Risk factors with a consistent pattern of association in the study-specific analyses were included in adjusted models fitted to the data pooled from all studies for each age and sex group. We focussed primarily on the HRs, rather than their associated p-values, since some studies lacked power for some comparisons. Where study-specific HR were different, we included them and captured the heterogeneity of effects between the studies by fitting terms for risk factor-study interactions and retained those with small p-values for the interaction terms. We included a fixed-effect for study because HIV incidence differed between studies. We fitted three piecewise exponential models including age and calendar year: 1) with a fixed effect for study, allowing each study to have a different background incidence rate; 2) with an age and study interaction to relax the assumption of proportional hazards and allow the hazard function to behave differently over age in the different studies and 3) with an age and study interaction plus an interaction between study and prevalence of untreated infection in the opposite sex. We fitted a fourth model which omitted the age and calendar year variables. We hypothesised that age and calendar year are not direct determinants of HIV incidence and by omitting them we aimed to reveal the more proximate determinants of HIV incidence. Lastly, we reviewed the results from these four models for each age and sex group and judged which was the most parsimonious model- the preferred model- for each group.

Information on partnership dynamics (the numbers and timings of partnerships) was available only for Karonga, Kisesa and Rakai. The analysis was repeated with these studies including the measures of partnership dynamics.

We conducted two sensitivity analyses. To assess the influence each study had on the results from the preferred models we fitted study-specific versions of those models, carried out a meta-analysis on each risk factor estimate and then repeated the meta-analysis using the leave-one-out approach. We also fitted the preferred models to data from 2013-16 only, to see if risk factor associations were different in that period.

Stata version 18.0 was used throughout the analysis.

The funder of the study had no role in study design, data collection, data analysis, data interpretation, or writing of the report.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.