Brian D. Greenhill, Michael D. Ward and Audrey Sacks

The separation plot package (current version at allows users to visually compare the fits of binary models, both in and out of sample. We present a visual method for assessing the predictive power of models with binary outcomes. This technique allows the analyst to evaluate model fit based upon the models’ ability to consistently match high-probability predictions to actual occurrences of the event of interest, and low-probability predictions to nonoccurrences of the event of interest. Unlike existing methods for assessing predictive power for logit and probit models such as Percent Correctly Predicted statistics, Brier scores, and the ROC plot, our “separation plot” has the advantage of producing a visual display that is informative and easy to explain to a general audience, while also remaining insensitive to the often arbitrary probability thresholds that are used to distinguish between predicted events and nonevents. We demonstrate the effectiveness of this technique in building predictive models in a number of different areas of political research.

CRISP, a suite of programs to aid CRISis Prediction.

Michael D. Ward

The CRISP R package is an updated version of the earlier ICEWS package developed for DARPA. The package allows for in-sample and out-sample predictions of five types of events of interests (EOI) –Insurgency, Rebellion, Domestic Crisis, Ethnic and Religious Violence, and International Crisis– within 167 countries. The EOI data are generated by utilizing news articles from over 75 electronic regional and international news sources and machine-coding these events using the SERIF software program. We provide tools for analyzing and visualizing predicted probabilities of risk based on a data set that includes a large set of explanatory variables.

The project was initially funded by the Information Processing Technology Office of the Defense Advanced Research Projects Agency was focused on 25 Pacific countries, and provided an integrated crisis early warning system  for decision makers in the U.S. defense community. The holding grant is to the Lockheed Martin Corporation, Contract FA8650-07-C-7749. The current research effort includes 167 countries, updates it to the present, and expands the kind of data and models employed as well. The current effort was partially in support of ONR contract N00014-12-C-0066 to Lockheed Martin's Advanced Technology Laboratories.

If you are interested in licensing this software, contact


Jacob M. Montgomery, Florian Hollenbach, and Michael D. Ward

The EBMAforecast package (current version at allows users to increase the accuracy of forecasting models by pooling multiple component forecasts to generate ensemble forecasts. It includes functions to fit an ensemble Bayesian model averaging (EBMA) model using in-sample predictions, generate ensemble out-of-sample predictions, and create useful data visualizations. Currently, the package can only handle dichotomous outcomes or those with normally distributed errors, although additional models will be added to the package in the coming months. Missing observations are allowed in the calibration set, but models with many predictions missing are penalized.