The IPUMS API Program
The IPUMS API Program is the home for all API-related activity at IPUMS across all of our data products. This page outlines the overall API portfolio and a high-level view of our future roadmap.
Much of this page describes future goals for our API Program. Please understand that much of what is described here doesn’t exist yet.
IPUMS API Community Forum
If you need help at any time in your work with the IPUMS API, please join us in the IPUMS API area of the IPUMS user forum.
IPUMS APIs allow access to two primary types of IPUMS assets: data and metadata. IPUMS APIs are organized into metadata and data APIs like so:
IPUMS Metadata APIs - APIs which provide metadata about IPUMS data. These APIs are used for data discovery and availability exploration as well as for retrieving codes, names, and other bits of information which are necessary to form requests to our Data APIs.
IPUMS Data APIs - APIs which provide IPUMS data. IPUMS Data APIs are further divided into two flavors:
Data Extract APIs - APIs which receive data requests from API users and submit them for background processing. The data is not returned right away, rather, API users monitor the status of their request, and when requests are completed data can be retrieved. This is the model which most closely resembles the website experience with our IPUMS data products and produces the same exact kinds of extracts that website requests do (complete with codebook, syntax files, and so on).
Data Streaming APIs - (Do Not Exist Yet) APIs which return data immediately. Most appropriate for smaller amounts of microdata or tabulated data, and for applications which support real-time data browsing.
APIs for Different Types of IPUMS Data
The IPUMS data collection contains many different types of data. IPUMS data includes:
- Microdata - Records representing individual persons and households or sets of attributes that nest under individual persons or households (e.g. activities, injuries).
- Aggregate Data - Records representing aggregate calculations (e.g. sums, medians, averages, max, min) for a given geographic area (e.g. counties, states, census tracts).
- Spatial Data - Rasters, vectors, point data representing spatial entities (e.g. political boundaries, locations, latitude/longitude).
Although we try to follow as many of the same endpoint names and conventions across APIs, we do not always have one unified API for all IPUMS products. For example, IPUMS USA and IPUMS International could both be served via the IPUMS Microdata Data Extract API (once said data is available via API), while IPUMS NHGIS requires use of a separate IPUMS NHGIS Data Extract API. This largely has to do with differences in how these datasets are structured and organized. Nevertheless, both the IPUMS Microdata Data Extract API and the IPUMS NHGIS Data Extract API follow the overall design and pattern we’ve established for Data Extract APIs.
Differences Between IPUMS Products
With the IPUMS API Program, we aim to make APIs which are as consistent as possible across our data products. Unlike our website portfolio, where each data product has its own website, our APIs are intended to be reused across our data products as much as possible.
For example, our Data Extract APIs have been intentionally made as similar as possible across all IPUMS products. The same overall pattern can be used to create extracts from IPUMS USA or IPUMS NHGIS. The details of the request naturally are different for the two use cases, but both the Microdata and the NHGIS Data Extract APIs implement the same API endpoints - they only differ in the details of the JSON payload that specifies your extract request. Otherwise, the workflow of creating an extract is very similar for each product.
Specifying which product you’d like to use merely requires passing the appropriate product name as a query parameter and then providing the proper extract request payload for your chosen product. Any product-specific details the user needs to be aware of are covered in the sample workflows and in the API reference documentation.
These are some thoughts about our future roadmap. These are plans, not promises, and plans will almost certainly change as we gain more experience and user feedback. But here’s what we’re currently thinking!
In early 2020 we will be focused on releasing a beta of our Microdata Data Extract API which will support a subset of IPUMS USA extract requests. To achieve this goal, we will also need to develop partial Microdata Metadata APIs to expose sample and variable metadata, as that metadata is needed in order to construct requests to the Data Request APIs. If you are interested in USA microdata via API, please consider joining our beta program for early access.
Medium term directions will partially depend on user feedback. However, we anticipate the following work areas:
- expanding the API user base
- adding support to the Microdata Data Extract API for more of the operations that are available via the website
- investigating whether we can add more products’ data to the Data Extract APIs
- increasing the scope of the Metadata APIs to support more use cases
- aggregate data streaming APIs
Further into the future, we hope to bring more real-time data streaming and data tabulation APIs to fruition.
The IPUMS API Program is spearheaded by the IPUMS API Program Support Team. We hang out on the IPUMS user forum’s API category. You can also email feedback to firstname.lastname@example.org and we’ll reply as soon as we can!