The IPUMS API Program
The IPUMS API Program is the home for all API-related activity at IPUMS across all of our data products. This page outlines the overall API portfolio and a high-level view of our future roadmap.
Much of this page describes future goals for our API Program. Please understand that much of what is described here doesn’t exist yet.
IPUMS API Community Forum
If you need help at any time in your work with the IPUMS API, please join us in the IPUMS API area of the IPUMS user forum.
Understanding IPUMS API Organization
IPUMS APIs are organized into a hierarchy as follows:
Asset Type -> Type of Data -> Product
- Metadata -> Microdata -> IPUMS USA
- Data Extract -> Aggregate -> IPUMS NHGIS
Our API URL structure reflects this hierarchy. Thus, the API at https://api.ipums.org/data-extract/microdata/usa is used to create data extracts of IPUMS USA microdata.
Each of these components is explained in more detail below.
IPUMS APIs allow access to two primary types of assets: data and metadata. IPUMS APIs are organized into metadata and data APIs like so:
IPUMS Metadata APIs - APIs which provide metadata about IPUMS data. These APIs are used for data discovery and availability exploration as well as for retrieving codes, names, and other bits of information which are necessary to form requests to our Data APIs.
IPUMS Data APIs - APIs which provide IPUMS data. IPUMS Data APIs are further divided into two flavors:
Data Extract APIs - APIs which receive data requests from API users and submit them for background processing. The data is not returned right away, rather, API users monitor the status of their request, and when requests are completed data can be retrieved. This is the model which most closely resembles the website experience with our IPUMS data products and produces the same exact kinds of extracts that website requests do (complete with codebook, syntax files, and so on).
Data Streaming APIs - APIs which return data immediately. Most appropriate for smaller amounts of microdata or tabulated data, and for applications which support real-time data browsing.
APIs for Different Types of IPUMS Data
The IPUMS data collection contains many different types of data. IPUMS data includes:
- Microdata - Records represent individual persons and households or sets of attributes that nest under individual persons or households (e.g. activities, injuries).
- Aggregate Data - Records representing aggregate calculations (e.g. sums, medians, averages, max, min) for a given geographic area (e.g. counties, states, census tracts).
- Spatial Data - Rasters, vectors, point data representing spatial entities (e.g. political boundaries, locations, latitude/longitude).
Generally speaking, each type of data has its own API, though we try to follow as many of the same endpoint names and conventions across the APIs. For example, IPUMS USA and IPUMS International both provide microdata and would use the Microdata Data Extract API (once International data is available via API), while two separate APIs are needed to interact with IPUMS USA and IPUMS NHGIS, as the latter product provides aggregate data and therefore uses the Aggregate Data Extract API. Both the Microdata Data Extract API and the Aggregate Data Extract API follow the overall design and pattern we’ve established for Data Extract APIs.
Differences Between IPUMS Products
With the IPUMS API Program, we aim to make APIs which are as consistent as possible across our data products. Unlike our website portfolio, where each data product has its own website, our APIs are intended to be used across our data products as much as possible.
For example, our Data Extract APIs have been intentionally made as similar as possible across all types of IPUMS data. The same overall pattern can be used to create extracts from IPUMS USA or IPUMS NHGIS. The details of the request naturally are different for the two use cases, but both the Microdata and the Aggregate Data Extract APIs implement the same API endpoints - they only differ in the details of the JSON payload that specifies your extract request. Otherwise, the workflow of creating an extract is very similar for each product.
Specifying which product you’d like to use merely requires passing the appropriate product name in the URL and then providing the proper extract request payload for your chosen product. Any product-specific details the user needs to be aware of are covered in the sample workflows and in the API reference documentation.
These are some thoughts about our future roadmap. These are plans, not promises, and plans will almost certainly change as we gain more experience and user feedback. But here’s what we’re currently thinking!
In the winter and spring of 2019 we are focused on releasing a beta of our Data Extract API which will support a subset of IPUMS USA and IPUMS NHGIS extract requests. To achieve this goal, we will also need to develop partial Metadata APIs to expose sample and variable metadata (IPUMS USA) as well as dataset and data table metadata (IPUMS NHGIS), as that metadata is needed in order to construct requests to the Data Request API. When ready, we will start with a very small, invite-only beta program with hand-selected users that will help us prove out the model and implementation before we open it to a wider beta.
Medium term directions will partially depend on user feedback from the initial beta program. However, we anticipate the following work areas:
- expanding the beta program to more users
- adding support to the Data Extract API for more of the operations that are available via the website
- investigating whether we can add more products’ data to the Data Extract API
- increasing the scope of the Metadata API to support more use cases
Further into the future, we hope to bring real-time data streaming APIs to fruition.
The IPUMS API Program is spearheaded by the IPUMS API Program Support Team. We hang out on the IPUMS user forum’s API board. Email feedback to firstname.lastname@example.org and we’ll reply as soon as we can!