Description and Application of an Accuracy Assessment Method for Gap Analysis Models

Jill A. LaBram¹, Amanda E. Peck¹, and Craig R. Allen²

¹South Carolina Cooperative Fish and Wildlife Research Unit,Department of Aquaculture, Fisheries and Wildlife, Clemson University,
Clemson, South Carolina

²USGS, South Carolina Cooperative Fish and Wildlife Research Unit,Department of Aquaculture, Fisheries and Wildlife, Clemson University,
Clemson, South Carolina

Introduction

Gap Analysis uses literature-based information on vertebrate habitat affinities to determine areas of high species richness.  Predictive models for vertebrates are created based on literature and expert review to predict species' occurrences and overall richness of vertebrate diversity.  However, these models need validation based on fieldwork to assess their accuracy. 

Accuracy assessment of animal spatial models is crude and poorly developed and requires quantification of both commission and omission errors.  Boone and Krohn (2000) found that the most common accuracy assessment method used for GAP models is to compare the predicted species for an area such as a U.S. National Park or National Wildlife Refuge to checklists of breeding species available for such areas.  Omission errors (occurrence when absence was predicted) are relatively easy to document, but commission errors (absence when occurrence was predicted) are more difficult to determine.  These different error types may have weighted costs associated with the ecological “value” of the species in terms of conservation priorities.  In a model used to define protected areas, failure to correctly predict positive locations (omission error) may be more “costly” than commission errors (Fielding 2002). 

Recent accuracy assessments of GAP vertebrate models have stressed the importance of separation of actual commission errors (species is not present on the site) from apparent errors (incomplete field inventories falsely omit the true species occurrence on the site) and an a priori species ranking of occurrence, placing common, density-dependent species above rare ones in terms of the likelihood of the model being correct (Boone and Krohn 1999, Schaefer and Krohn 2002).  Boone and Krohn (1999) developed a multivariate method to correct commission errors in GAP models by predicting how likely a species would be seen in future surveys, called Likelihood of Occurrence Ranks.  They showed that variables such as size of survey site, duration of surveys, natural history of the species, and quality of species distribution models influence the validity of accuracy assessments.   

The development of vertebrate monitoring programs allows for the validation of model predictions.  Long-term sampling decreases errors associated with spatial and temporal variability in animal-habitat use and increases the odds of detecting rare species, thus monitoring programs provide ideal data for assessing the accuracy of GAP models. 

Our goals are (1) to describe our methods of accuracy assessment, (2) to assess the accuracy of the South Carolina Gap Analysis Project’s (SC-GAP) vertebrate models in predicting reptile and amphibian (herpetofauna) and mammal species richness as compared to capture-based models, and (3) to determine the spatial correspondence between the nodes of highest richness for herpetofauna and mammals separately and combined.  

Methods

Study Area - Our study sites were located within the 78,000-hectare Savannah River Site (SRS) near Aiken, South Carolina.  The site was closed to the public in 1951, and the USDA Forest Service planted pine seedlings on former crop and pastureland, beginning in 1952, as an initial forest restoration effort.  By 1963, about 90% of the area was covered by young forests (Golley et al. 1965). 

SRS Land Cover - We modified an existing digital land cover classification (Imm 1997) by grouping similar land cover classes into seven cover types: bottomland hardwood, swamp-edge, mixed forest, hardwood slope, planted pine, Carolina bay, and sandhill. 

GAP Land Cover - A habitat-based, 27-class raster land cover map with a resolution of 30 meters was produced by Clemson University and South Carolina Department of Natural Resources personnel using a combination of remote sensing image interpretation and ground-truthing from Landsat TM imagery dating from 1991-1993. 

The Savannah River Site was clipped from the SC-GAP coverage.  The SRS area included 22 of the 27 GAP land cover classes, but only 10 of those were natural terrestrial classes (swamp, bottomland/floodplain forest, closed-canopy evergreen forest/woodland, needle-leaved evergreen mixed forest/woodland, pine woodland, dry deciduous forest/woodland, mesic deciduous forest/woodland, dry mixed forest/woodland, mesic mixed forest/woodland, and wet evergreen).

The GAP classification differed from the SRS classification.  Therefore, we created a crosswalk table that converted GAP land cover into SRS land cover to provide comparison between the two classifications using SRS land cover as the base map.  SRS land cover also was cross-walked into GAP land cover using the GAP land cover as the base map.

Vertebrate Sampling- We trapped herpetofauna and small mammals at five replicates of each of the seven land cover types in the fall of three years (1999-2001).  Small mammals were sampled utilizing Sherman live traps, tomahawk traps, and pitfall-drift fence arrays.  Herpetofauna was sampled using pitfall-drift fence arrays and visual captures for all three years, with the addition of funnel traps, cover boards, and PVC pipes in 2001.

SRS Sample-based Model - We built presence/absence habitat-association models only for the species that were most abundant over three years, including four reptile species, seven amphibian species, and six mammal species.  Because we focused on terrestrial species, our capture data only apply to a 200-meter swamp-edge buffer rather than the entire swamp land cover type at SRS.  We limited our assessment to common species and set a criterion that captures within a given land cover must account for > 5% of the captures for a species to be considered present in that land cover.  A key assumption is that this sample-based model reflects the “real” distribution of both presence and absence of species, because it is based on actual trapping data for the most abundant species.

GAP Model - GAP-generated habitat affinities for herpetofauna and mammals were determined primarily from literature review.  These animal-habitat associations were cross-walked into the SRS land cover.  This information was used to build a matrix of species x land cover for the seventeen species for which we had adequate data.  These species were predicted to be present or absent in each land cover type, using both the SRS and GAP land cover classifications as our base maps.

Species Richness - Composite species richness maps for herpetofauna, mammals, and both taxa combined were produced by adding the individual species maps to produce a composite map of overall sample-based richness for the SRS land cover and predicted richness for the GAP land cover.

Spatial Correspondence - We compared the GAP predictive model to our sample-based model, using both the SRS and GAP land cover classifications as our base land cover maps to determine spatial correspondence of species richness.  The SRS sample-based richness model was subtracted from the GAP-predicted richness model.  Values of zero occurred where the levels of species richness between GAP-predicted and SRS captures were equivalent.  High positive values occurred where GAP-predicted species richness was high relative to SRS capture richness (GAP commission errors), and high negative values occurred where SRS capture richness were high relative to GAP-predicted species richness (GAP omission errors) (Allen et al. 2001b).

Commission and omission errors also were calculated for individual species, including the percentage of area and percentage of land cover agreed upon by GAP prediction and capture success.  An area or land cover type was considered in spatial agreement between the GAP and SRS models if both predicted the species to be either present or absent within that land cover type or area.  This was calculated using both land cover classifications as base maps:

A = (Σ (AA)/TA)*100

L = (Σ (LA)/TL)*100

where A is percent agreement, AA is area of agreement of the base map, TA is total area of the base map, L is percent land cover agreement, LA is land cover agreement, and TL is total number of land cover types (SRS=7, GAP=10).  These values are calculated using both land cover base maps.  To find the omission errors, we added the area where a species was present but not predicted to occur:

O = (TOA/TA)*100

LO = (TO/TL)*100

where O is percent omission error, TOA is the total omission area, TA is the total area of the base map, LO is the percent land cover omission error, TO is the total land cover types omitted, and TL is total number of land cover types.  To find the commission errors, we added the area where a species was predicted to occur but not present:

C = (TCA/TA)*100    

LC = (TC/TL)*100     

where C is percent commission error, TCA is the total commission area, TA is the total area of the base map, LC is the percent land cover commission error, TC is total land cover types with commission error, and TL is the total number of land cover types. 

Nodes of Highest Richness - The explicit focus of gap analyses are not single species, but the identification of areas of high species richness.  Therefore, we determined the correspondence between nodes of highest richness (top 20%) (Allen et al. 2001b) for each taxon.  To qualify as the top 20%, 5 of the 6 mammal species, 9 of the 11 herpetofauna species, or 14 of the 17 total species must be present in a land cover type.          

Results

SRS Land Cover as a Base Map - Species richness based on our monitoring program varied from 4 to 10 species per land cover type for herpetofauna, from 1 to 6 species for mammals, and from 5 to15 species for the two groups combined.  GAP-predicted species richness ranged from 6 to 11 species for herpetofauna, from 2 to 6 species for mammals, and from 8 to 17 for the two combined.

There was spatial correspondence of overall herpetofauna species richness between species captured and those predicted by gap analysis in the planted pine land cover.  The swamp-edge land cover showed actual species richness higher than predicted.  Predicted species richness was higher than actual species richness in the remaining land cover types, ranging from two more species to seven more species (Figure 1). 

There was spatial correspondence of overall mammal species richness between species captured and those predicted by gap analysis only in the swamp-edge land cover.  Captured species richness was higher than predicted in the planted pine and Carolina bay classes.  Predicted species richness was higher than captured in the remaining four land cover types.

There was no spatial correspondence between sampling and GAP models for mammals and herpetofauna combined.  Actual species richness was higher than predicted in the planted pine, swamp-edge, and Carolina bay classes.  Predicted species richness was higher than captured in bottomland hardwood, mixed, hardwood slope, and sandhill land cover types.  The commission error rates were higher than omission error rates (Table 1).

GAP Land Cover as a Base Map- Herpetofauna species richness based on our monitoring program varied from 4 to 10 species per land cover type, while mammal species richness varied from 1 to 5 species, and combined captured species richness varied from 5 to 15 species.  GAP-predicted herpetofauna species richness varied from 1 to 11 species, predicted mammal species richness ranged from 2 to 6 species, and combined predicted species richness ranged from 3 to 16 species.

There was spatial correspondence of herpetofauna species richness between species captured and those predicted by gap analysis in closed-canopy evergreen mixed forest/woodland.  Captured species richness was higher than predicted in only the swamp-edge and wet evergreen classes.  Predicted species richness was higher than captured in the remaining land cover types (Figure 2).

Figure 2.  Spatial correspondence of species richness using GAP land cover.

There was spatial correspondence of mammal species richness in the swamp-edge land cover only.  Captured species richness was higher than predicted in two land cover types, while predicted species richness was higher in the remaining seven land cover types, ranging from one to four more species.

For overall richness, there was no spatial correspondence for any land cover type.  Captured species richness was much higher than predicted in the wet evergreen, closed-canopy evergreen mixed forest/woodland, and swamp-edge classes.  The predicted species richness was higher than actual richness in the remaining land cover types.  The commission error was higher than omission error for both the area and land cover calculations, with the exception of percent agreement area for mammals (Table 2).

Nodes of highest richness - Using the SRS land cover base map, five of the seven land cover types were predicted to be within the node of highest richness, but only one land cover type (swamp-edge) qualified based on captures.  Mammals were predicted to have five species-rich land cover types, while only three occurred (bottomland hardwood, Carolina bay, and swamp-edge).  Herpetofauna was predicted to have six species-rich land cover types, while only one (swamp-edge) occurred (Table 3). 

In the GAP base map, eight of the ten applicable land cover types were predicted to be species-rich, while only one (swamp-edge) qualified based on captures.  Mammals were predicted to have seven species-rich land cover types, while only three occurred (swamp-edge, bottomland floodplain, and wet evergreen).  Herpetofauna were predicted to have eight species-rich land cover types, while only one (swamp-edge) occurred (Table 4).

Discussion

Conserving areas of high species richness is the most efficient and cost-effective way to retain maximal biological diversity (Scott et al. 1987).  The high commission rate we documented suggests a need to refine the GAP vertebrate modeling process.  However, given that Gap Analysis is a tool for predicting vertebrate distributions for use in conservation planning, Edwards et al. (1996) argue that commission error is preferred over omission error.  High omission error could possibly lead to the exclusion of species from conservation plans.  The best assessment of a model's accuracy is to test it with some independent data.  Therefore, we modeled species within the SRS area that were commonly captured to test the SC gap analysis.  South Carolina was under drought conditions for the duration of this study, which may have affected species abundance and trappability.  Animals captured within a land cover class harboring < 5% of the total individuals of that species were assumed to be transient in that land cover class, which could lead to an additional source of commission error.  For example, captures of 20 eastern narrow-mouthed toads in Carolina bays and 23 in bottomland hardwood sites were insufficient (i.e., 5% of 515 captures = minimum of 26 animals) for inclusion of these land cover types as occupied by the species.  On the other hand, if a certain land cover patch was located between a breeding site and the resident land cover type, it could be a secondary habitat for that species.  Failure to detect a species on a site may simply be due to trapping difficulty, natural rarity, or spatial or temporal variability in habitat use rather than the absence of the animal. 

Different classification schemes aggregate differently within and among land covers.  Thus, converting between classification systems can increase the commission and omission errors of the models.  For herpetofauna and mammals combined, the range between predicted and captured richness was greater (by six species) when cross-walking SRS land cover into GAP land cover than when converting GAP to SRS.  There were several land cover classes that were not clearly delineated in the SC gap analysis, which may have led to failure of animal-habitat associations to predict occurrence within the correct spatial area.  For example, SC-GAP could not reliably separate the land cover types of swamp and bottomland hardwood.  Also, none of the 194 Carolina bays (786 ha) known to occur on the SRS area were present on the SC-GAP map; therefore, we could not include that class in the GAP-based model.

One way to improve vertebrate models is to determine the sources of commission errors.  Two possible sources are that habitat associations may be incorrect, or species models are too simplistic.  If the former is the case, monitoring and sampling programs can provide information with enough spatial and temporal breadth to refine associations and hence improve models.  In the latter case, models can be improved utilizing current knowledge that blends landscape ecology and population viability.  Inclusion of landscape metrics may improve species models and give the user more confidence in management decisions based on output of the models.  For example, Allen et al. (2001a) incorporated minimum critical area criteria into species models to reduce commission errors arising from considering an animal as present in a patch too small to support a population of that species.  Most likely, commission errors propagate from a combination of these sources.  Further refinement of the vertebrate modeling process will improve the accuracy of predictive models.

Literature Cited

Allen, C.R., L.G. Pearlstine, and W.M. Kitchens.  2001a.  Modeling viable mammal populations in gap analyses.  Biological Conservation 99:135-144.

Allen, C.R., L.G. Pearlstine, D.P. Wojcik, and W.M. Kitchens.  2001b.  The spatial distribution of diversity between disparate taxa: Spatial correspondence between mammals and ants across south Florida, USA.  Landscape Ecology 16:453-464.

Boone, R.B., and W.B. Krohn.  1999.  Modeling the occurrence of bird species: Are the errors predictable?  Ecological Applications 9:835.

Boone, R.B., and W.B. Krohn.  2000.  Predicting broad-scale occurrences of vertebrates in patchy landscapes.  Landscape Ecology 15:63-74.

Edwards, T. C., Jr., E. T. Deshler, D. Foster, and G. G. Moisen.  1996.  Adequacy of wildlife habitat relation models for estimating spatial distributions of terrestrial vertebrates. 

Conservation Biology 10:263-270.

Fielding, A.H.  2002.  What are the appropriate characteristics of an accuracy measure?  Pages 271-280 in J.M. Scott, P.J. Heglund, M.L. Morrison, J.B. Haufler, M.G. Raphael, W.A. Wall, and F.B. Samson, editors.  Predicting species occurrences.  Island Press, Washington, DC.

Golley, F.B., J.B. Gentry, L.D. Caldwell, and L.B. Davenport, Jr.  1965.  Number and variety of small mammals on the AEC Savannah River Plant.  Journal of Mammalogy 46:1-18.

Imm, D.  1997.  ArcView classification of the landcovers of the Savannah River Site.  USDA Forest Service, Savannah River Institute, New Ellenton, South Carolina.

Schaefer, S.M. and W.B. Krohn.  2002.  Predicting vertebrate occurrences from species habitat associations: improving the interpretation of commission error rates.  Pages 419-427 in J.M. Scott, P.J. Heglund, M.L. Morrison, J.B. Haufler, M.G. Raphael, W.A. Wall, and F.B. Samson, editors.  Predicting species occurrences.  Island Press, Washington, DC.

Scott, J. M., B. Csuti, J. D. Jacobi, and J. E. Estes.  1987.  Species richness: a geographic approach to protecting future biological diversity.  BioScience 37:782-788.