Footnote 4 of the Gallo document states:
“The statistical picture of AIDS in Africa, consequently, is a communal projection based on very rough estimates of HIV positives culled from select and small samples, which are extrapolated across the continent using computer models and highly questionable assumptions.
The vast majority of HIV/AIDS estimates for African countries are based on the results of periodic HIV antibody surveys using blood samples taken from a cross-section of pregnant women attending public antenatal clinics (ANC surveys). In fact, virtually every estimate regarding HIV/AIDS that has ever been put forth by the WHO or UNAIDS is based on mathematical and computer projections using data from these surveys. As Gallo et al. suggest, of all the countries in Africa, surveillance data emanating from South Africa (SA) is recognized to be among the best. In this regard, it is important to note that SA’s 2004 ANC survey was based on a sample representing only 0.035% of the entire population (i.e., 16,064/46.6 million). As such, Farber's characterization of this as a "select [pregnant women too poor to afford private care] and small [only about 1 sample for every 3,000 citizens] sample" is anything but a "false" statement. Furthermore, HIV antibody prevalence in this survey was determined using a single ELISA screening assay that has never received FDA approval for marketing in the United States. As such, Farber’s further characterization of these prevalence estimates as “very rough,” may be debatable, but certainly cannot summarily be characterized as either false or misleading.
Gallo et al provide no reference for their statement that: “Large samples of people with HIV have been taken in a number of countries including Kenya, Botswana, Uganda and South Africa,” however I believe they must be referring to so called ‘population-based’ surveys that have recently been conducted in several African countries. The advantage of such surveys is that they sample persons of both genders, and in some cases of all ages and therefore do not need to project prevalence estimates from ‘select’ populations onto other population groups using ‘questionable assumptions.’ Indeed, it is because we now have data from national population-based surveys conducted in Mali, Zambia, Zimbabwe, South Africa, Niger, and Burundi, that we know many of the assumptions used to project prevalence estimates from ANC surveys onto other population groups were indeed inaccurate. In fact, in response to this new data, the WHO and UNAIDS have recently published a document entitled, “Reconciling Antenatal clinic-based surveys and population-based survey estimates of HIV prevalence” (WHO/UNAIDS, August 2003), which specifically addresses these inconsistencies.
As Gallo et al emphasize, two of these population-based surveys have been conducted in SA (i.e., the “two countrywide household surveys”). Specifically, researchers affiliated with the Nelson Mandela/Human Sciences Research Council conducted these surveys in 2002 (HSRC 2002)1 and 2005 (HSRC 2005)2 using weighted national samples representative of all persons aged 2 years and older.3 HIV antibody prevalence estimates for these surveys were based on samples representing only 0.019% and 0.034% percent of the entire population of SA for these years, respectively.4 Furthermore, the first of these surveys relied on the use of a single screening assay using saliva samples, while the second relied on a testing algorithm using only two ELISA screening assays; and neither survey confirmed a single result-let alone a representative cross-section of samples-using FDA required, and CDC recommended, Western Blot (WB) testing. As such, Galo et al's characterization of these surveys in SA as, “better than most industrialised countries,” is baffling to say the least.
In fact, even a cursory comparison of data between these two surveys reveals gross inconsistencies that cannot be explained by epidemiological arguments alone. For example, if we are to believe the results of these surveys, then we have to accept that HIV antibody prevalence among white South Africans dropped from 6.2% in 2002 to only 0.6% in 2005; representing an unbelievable 90% reduction [i.e., (0.062-0.006)/0.062 = 0.90] in overall prevalence for these individuals. Given that the only way HIV prevalence in a demographic group can drop is through death; and that observations from prevalence cohorts in Africa would suggest that only about 10% of these individuals could be expected to die off in any given year (in the absence of treatment); and that the vast majority of whites in SA are either covered by insurance schemes, or wealthy enough to afford treatment, in the first place; such an observation can only lead to the conclusion that there are some serious problems with the data from at least one, and perhaps both, of these surveys. It is further important to note that the discrepancy detailed here is anything but anecdotal.
For example, prevalence among coloured South Africans likewise dropped by 69% over this three-year period (i.e., from 6.9% to 1.9%). And on the provincial level, antibody prevalence among persons of all ages in Gauteng (GT), Northern Cape (NC), and Western Cape (WC) dropped by 27%, 36%, and 82%, respectively, over this three-year period (i.e., from 14.7% to 10.8%, 8.4% to 5.4%, and 10.7% to 1.9%, respectively). Even if we were to assume that there were no new infections over this period, drops of this magnitude defy explanation. It is also interesting to note that while antibody prevalence among sexually active males aged 20-24 was observed to drop by 45% (i.e., 22% to 12.1%), prevalence among arguably less sexually active males aged 50-54 soared by 184% (i.e., from 5.0 to 14.2%). In summary, to characterize data from these surveys as anything other than “very rough,” would be disingenuous.
Gallo et al go on to assert: “The most widely used computer model [the ASSA2003] used to determine the size of South Africa's epidemic closely matches the prevalence calculated in the latest countrywide household survey [i.e., the HSRC 2005 survey].” Apparently, the implication here is that if a model can accurately forecast HIV prevalence into the future, the “assumptions” that were used to make this prediction must be accurate; and that Farber's characterization of such assumptions as “highly questionable” must therefore be considered to be inaccurate.
Specifically, Gallo et al are referring to the fact that the above referenced ASSA2003 model predicted that the overall HIV prevalence for persons aged 2 and older in SA would be 11.3% in 2005,5 which is indeed very close to the 10.8% observed in the HSRC 2005 survey. However, it is important note that this model is calibrated to reproduce the patterns of HIV prevalence observed in all surveys conducted in SA through the end of year 2003; including the population-based HSRC 2002 study, and another national population-based survey conducted among youth in 2003.6 Furthermore, by the time this prediction was put forth, it was already known that the epidemic in SA had stabilized, and that the overall HIV prevalence in 2005 could therefore be expected to be very similar to the 11.4% observed in the HSRC 2002 survey. But more importantly, when we look at predictions from this model according to race, age, gender, and province, we see that they are in reality dramatically different from what was observed in the HSRC 2005 survey.
For example, as compared to the survey results, the ASSA2003 model overestimated prevalence among sexually active (age 15-49) coloureds, whites, and asians, by 63%, 171%, and 215%, respectively (i.e., modeled vs. observed: 5.2% vs. 3.2%, 1.4% vs. 0.5%, and 3.1 vs. 1.0%, respectively); and for sexually active adults of all races, the model overestimated prevalence in kwaZulu/Natal (KZN), GT and WC, by 19%, 38%, and 154%, respectively (i.e., modeled vs. observed: 26.1% vs. 21.9%, 21.7% vs. 15.8%, and 8.1 vs. 3.2%, respectively). And just as the model dramatically overestimated prevalence in some demographic groups, it likewise underestimated prevalence in other groups. For example, the HSRC survey found HIV prevalence among children aged 2-14 to be 130% higher than predicted by the model (i.e., model vs. observed: 1.4% vs. 3.3%), and for male youth aged 15-19, prevalence was found to be more than 10-fold higher than predicted (i.e., model vs. observed: 0.3% vs. 3.2%) And finally, for sexually inactive females aged 55 and older, HIV prevalence was observed to be 463% higher than predicted (i.e., model vs. observed: 0.6% vs. 3.5%).
In summary, the fact that this model accurately predicted the overall national prevalence to be observed in the 2005 HSRC survey (i.e., 11.3% vs. 10.8%), when it was already known that the prevalence had more or less stabilized, and that national prevalence in 2002 was observed to be 11.4% (i.e., the HSRC 2002 survey), falls well short of remarkable. The further observation, however, that this prediction is nothing more than the net result of massive overestimates in some population groups, taken in combination with equally massive underestimates in others, renders the contention that predictions from this model “closely matches” what was observed in the year 2005 survey completely without merit. In fact, based on the gross discrepancies between modeled and observed prevalence as detailed above, one can only conclude that the assumptions this model uses to make prevalence predictions should indeed be considered “highly questionable.” Alternatively, one could argue that the survey results themselves lack credibility and should not be used to judge the model.
Gallo et al suggest that the quality of HIV/AIDS surveillance data is SA is also supported by analyses of “death certificates.” Specifically, the authors are likely referring to the fact that several studies have indicated the annual number of reported deaths in SA has skyrocketed over the past several years, in a manner that some contend supports mortality projections from models that have been calibrated to available HIV prevalence data. Commenting on one of these studies, Gallo et al emphasize: “Statistics South Africa (2005) counted South African death certificates between 1997 and 2002 and found a 57% increase in mortality;” and they go on to emphasize that, “most of this increase is accounted for in young adults.” Indeed, if we examine data from this report7 a bit more closely, we see that reported deaths among the “sexually active” (ages 15-49) increased markedly faster over this period, coming in at 105% as compared to only 57% for persons of all ages.
That such increases in reported deaths in SA have occurred is not disputed. At issue is the cause; namely, are we seeing more HIV-related deaths, as claimed by the experts, or more complete recording of deaths, as a result improved death registration? Gallo et al – as well as others8 – contend that “only a small portion [of the increases] can be accounted for by improved death registration;” however, no one has yet provided a reference to any study that details any methodology or data used to arrive at this conclusion.9 And in fact, the referred to Statistics South Africa (Stats SA) report itself states: “The registration of deaths has improved significantly over the last decade as a direct result of efforts to improve the vital registration system.”7 Furthermore, according to earlier reports from Stats SA, death registration in post-apartheid SA was found to be “grossly incomplete,”10 with only about 37% of all rural deaths being recorded by 1996. Given that Census 1996 found nearly half of the population to be living in such areas, the significace of this cannot be ignored. This is particularly the case when one considers there is near universal agreement that completeness of death registration (CDR) in SA had reached 90% or higher by 2002. The consequence of this shift on observed increases in reported death can be expected to be profound.
For example, if it is indeed the case that CDR in rural SA was as low as 37% in 1997, and that it subsequently increased to 90% by 2002, this shift alone would result in a 143% increase in reported deaths [i.e., (0.90-0.37)/0.37 = 1.43, or 143%] from these areas, even in the absence of any actual increase in mortality whatsoever. As such, it is no wonder Stats SA warns that it is not possible to use data in their report to estimate actual changes in mortality rates, “without first adjusting for the incompleteness of [death] reporting.”7 As such, until researchers critically address this politically charged issue, it is impossible to say what proportion of the observed increases in reported death over this period are due to HIV/AIDS, as opposed to simple improvements in death registration in the years following the reintegration of the majority black population back into the SA's vital registration machinery.
What is possible, however, is to compare the observed increases in reported deaths at the provincial level, to other data available at the provincial level, in order to see what factors might be associated, or correlated, with these increases. A correlation between two variables is best revealed by calculating the Pearson correlation coefficient (r), the square of which (r2) tells us how well a change in one variable (e.g. smoking rates) explains, or predicts, changes in another variable (e.g., death rates). Although this is a useful tool, it should be emphasized that no correlation, even if perfect (i.e., r2 = 1.0, or 100%), is sufficient to declare a causal link between two variables. Rather, strong correlations, or a lack thereof, can only serve to either support, or discount, such proposed links; thereby guiding future research efforts.
For example, if it were indeed the case that that the observed increases in death over this period were the result of HIV/AIDS, we would expect to see the largest increases precisely in those provinces with the highest HIV prevalence. In fact, according to outputs from the ASSA provincial model in use at the time the Stats SA report was released, provincial death rates for persons of all ages from 1997-2002 are predicted to increase in near perfect concordance with relatively higher provincial HIV prevalence. Specifically, according to this model, relatively higher provincial HIV prevalence as observed in the 1997 ANC survey accounts for 88% of the overall increase in all-age death rates over this period (i.e., r = 0.94, r2 = 0.882; or 88.2%). A similar comparison using HIV prevalence as determined in the year 2002 ANC survey is likewise equally strong (r = 0.94, r2 = 88%).
However, when we repeat this comparison using real-life data from the Stats SA report, we see an entirely different picture. In fact, the correlation between the observed increases in reported death at the provincial level (expressed as rates) and HIV prevalence as determined in the 1997 ANC survey is feeble at best (r = 0.34, r2 = 11%). The same correlation using data from the 2002 ANC survey is weaker yet (r = 0.21, r2 = 4.5%). And completely contrary to expectations, year 2002 provincial HIV prevalence as determined in the arguably more representative population-based HSRC study, fails to predict any of the observed increases in reported mortality over this period (r = 0.03, r2 = 0%). In other words, HIV can explain at best only about 11% of the observed increases.
If, on the other hand, it were the case that that the observed increases in death over this period were the result improvements in death registration, we would expect to see the largest increases occurring precisely in those provinces with the most room for improvements in registration. In other words, we would expect to see the largest increases in those provinces with the lowest estimated CDR at the beginning of this period. Fortunately, Stats SA has also provided us with estimates of CDR at the provincial level for 1996,10 thereby allowing us to make just such a comparison. In this case, relatively lower provincial death registration in 1996 predicts a remarkable 77% of higher subsequent increases in reported death from 1997-2002 (i.e., r = -0.87, r2 = 77%). This observation strongly supports the possibility that the observed rise in reported deaths in SA over the past several years has indeed been strongly influenced by improvements in death reporting.
This possibility is further supported by the fact that these increases just happen to coincide with the 1998 launch of widespread governmental reforms aimed specifically at improving death registration, particularly in rural villages and townships. Among other things, these reforms included the introduction of a user-friendly death notification form;11, 12 radical simplification of reporting procedures; rapid expansion of Home Affairs satellite offices; installation of computers and internet connections at data collection points; grassroots campaigns to encourage registration; workshops for Traditional Healers and Herbalists; the solicitation of Tribal Courts, midwives and chiefs to assist in vital registration at the community level; and finally, a welfare project that enabled undertakers to tender for the burial costs of poor families, provided that such deaths were properly registered.13
As we have seen, shortly thereafter, reported deaths began to rise rapidly; with the vast majority of the subsequent increases occurring precisely in those provinces target by the above mentioned registration campaigns. It is also worth noting that the above observations (i.e., weak to non-existent correlations between observed increases in death and provincial HIV prevalence, and powerful correlations with relatively lower CDR at the beginning of this period) hold true even when the above analysis is repeated using data confined to the sexually active age groups. For a thorough review of the results and methodology used to perform these comparisons, see Richards. 14
In summary, a critical analysis of the observed increases in reported death in SA over the past several years does not support the suggestion that these increases serve to validate HIV prevalence estimates. In fact, modeled mortality predictions based on these prevalence estimates do not even come close to approximating what was actually observed by Stats SA's analysis of actual death notification forms. Specifically, a comparison of all age increases in death rates at the provincial level from 1997-2002 as predicted by the ASSA2003 model, to those observed in the Stats SA report, reveals that the modeled and observed increases are completely unrelated (i.e., r = -0.06, r2 = 0.3%). The further observation that the observed increases are at best only weakly related to provincial HIV prevalence, and powerfully related to relatively lower provincial CDR at the start of this period, strongly supports the suggestion that the vast majority of the observed increases are the result of improved death reporting in response to successful registration campaigns launched by the government in 1998. This being the case, one can only conclude that either the prevalence estimates used to calibrate the ASSA2003 model, or the underlying assumptions used by the model, are seriously in error.
In conclusion, the possibility that HIV/AIDS surveillance data, as well as modeled projections based on these data, should be regarded as “very rough” is supported by the fact that the vast majority of these estimates are based on data from ANC surveys, which in turn are based on very small number of samples taken from a highly select population. The further observation that recently conducted population-based surveys in several African countries have revealed errors in the assumptions used to project ANC data onto other population groups, validates this concern. And while demographers from UNAIDS and the WHO have incorporated what has been learned from these new surveys into their modeled projections, consecutive population-based surveys conducted in SA reveal massive discrepancies that call data from these surveys into question as well. The further observation that the most highly regarded demographic model – which is calibrated to this data – fails to even come close in predicting either future HIV prevalence, or even past increases in provincial death rates, supports the suggestion that all conclusions based on HIV prevalence estimates in Africa should be regarded with abundant caution. This is particularly the case when one considers that demographic data from SA is justifiably recognized as the best on the continent.